Automated Tester

.science

This content shows Simple View

June 2015

Applitools Eyes: Easy Automated Visual Checking

Applitools Eyes allows us to easily integrate automated visual testing into our existing BDD Solution. Having automated visual testing is a big plus because whilst our automation may be good at asserting if elements are visible on the page or if some certain text is present, it doesn’t tell us that it has appeared in the right place.

Automated visual testing will capture baseline images to make future checks against.

To quote another article: “For example, a single automated visual test will look at a page and assert that every element on it has rendered correctly. Effectively checking hundreds of things and telling you if any of them are out of place. This will occur every time the test is run, and it can be scaled to each browser, operating system, and screen resolution you require.”

If you run this in parallel then you have a powerful testing beast on your hands.

Put another way, one automated visual test is worth hundreds of assertions. And if done in the service of an iterative development workflow, then you’re one giant leap closer to Continuous Delivery. — Dave Haeffner

As with everything on this blog it is tailored around C# .Net but the following works in a similar way with other bindings like Java, Python, Ruby and JS.

How it’s set up in the Solution for C# .Net

First off we need a Nuget package for Eyes:

Nuget_Eyes

Then in our TestRunContext Class we need to instantiate it as part of our window setup:

using Applitools;

private static Eyes eyes;
public static void WindowSetup()

{
    Driver.Navigate().GoToUrl("about:blank");
    Driver.Manage().Window.Maximize();

// This is your api key, make sure you use it in all your tests.
    eyes = new Eyes {ApiKey = "API KEY GOES HERE"};

// Start visual testing with browser.
// Make sure to use the returned driver from this point on.
   Driver = eyes.Open(Driver, "Application name", "Test Name", new System.Drawing.Size(1024, 768));
}

We call on the following in our tests to perform a check of the page, passing in a string to be displayed in Applitools:

public static void EyeCheck(string pageName)
        {
            // Visual validation point
            eyes.CheckWindow(pageName);
        }

And then close the Eyes at the end of our checks:

public static void CloseEyes()
        {
 // End visual testing. Validate visual correctness.
            eyes.Close();
            eyes.AbortIfNotClosed();
        }

Rather than storing the Selenium instance in the Driver variable, we’re now storing it in a local browser variable and passing it into eyes.Open — storing the WebDriver object that eyes.Openreturns in the Driver variable instead.

This way the Eyes platform will be able to capture what our test is doing when we ask it to capture a screenshot. The Selenium actions in our test will not need to be modified.

Before calling eyes.Open we provide the API key. When calling eyes.Open, we pass it the Selenium instance, the name of the app we’re testing (e.g., “Test Suite“), and the name of the test (e.g., “Test Name“) in the above example the part that says “new System.Drawing.Size(1024, 768)” is specifying the view port size of the browser.

For mobile testing it is not relevant, since the device size will be used, but for web testing, it is highly recommended to configure a size in order to ensure that the result of the test will be consistent even when you run it on different machines with different screen resolutions/sizes.  You can also paramterise it and run on several sizes in case of responsive design, but in general it is recommended to set it explicitly.

Now we’re ready to add some visual checks to our test by calling the following in our step definitions.

Page.PerformEyeCheck(pageName);

With eyes.CheckWindow() working in the background we are specifying when in the test’s workflow we’d like Applitools Eyes to capture a screenshot (along with some description text).

NOTE: These visual checks are effectively doing the same work as the pre-existing assertion (e.g. where we’re asking Selenium if a success notification is displayed and asserting on the Boolean result) — in addition to reviewing other visual aspects of the page. So once we verify that our test is working correctly we can remove this assertion and still be covered.

We end the test with eyes.Close. You may feel the urge to place this in teardown, but in addition to closing the session with Eyes, it acts like an assertion. If Eyes finds a failure in the app (or if a baseline image approval is required), then eyes.Close will throw an exception; failing the test. So it’s best suited to live in the test itself.

NOTE: An exception from eyes.Close will include a URL to the Applitools Eyes job in your test output. The job will include screenshots from each test step and enable you to play back the keystrokes and mouse movements from your Selenium tests. You can also determine the level of strictness to use or exclude areas of your page from being checked.

When an exception gets thrown by eyes.Close, the Eyes session will close. But if an exception occurs before eyes.Close can fire, the session will remain open. To handle that, we’ll need to add an additional command to our teardown.

eyes.AbortIfNotClosed(); will make sure the Eyes session terminates properly regardless of what happens in the test.

Now when we run the test, it will execute locally while also performing visual checks in Applitools Eyes.

Applitools Eyes

Further Example:

https://eyes.applitools.com/app/tutorial.html



Parallel Test Execution with Specflow

Running Specflow scenarios in Parallel is a tricky problem to solve due to the nature of how tests are written. Achieving this is a lot simpler if we write Selenium tests in a unit style fashion but as this isn’t the case for Specflow with its Gherkin syntax we will look at executing them with a powershell build script against a Browserstack grid.

Thanks to this great article from Kenneth Truyers, it has really helped me to achieve this. We will look at executing parallel Specflow tests against a BrowserStack grid.

Set Up

In our Initialisation method we need to specify our remote driver whilst passing in our varying desired capabilities in Config.

private static void SetupCloudDriver()
        {
            var capabilities = new DesiredCapabilities();

            capabilities.SetCapability(CapabilityType.Version, ConfigurationManager.AppSettings["version"]);
            capabilities.SetCapability("os", ConfigurationManager.AppSettings["os"]);
            capabilities.SetCapability("os_version", ConfigurationManager.AppSettings["os_version"]);
            capabilities.SetCapability("browserName", ConfigurationManager.AppSettings["browser"]);

            capabilities.SetCapability("browserstack.user", ConfigurationManager.AppSettings["browserstack.user"]);
            capabilities.SetCapability("browserstack.key", ConfigurationManager.AppSettings["browserstack.key"]);
            capabilities.SetCapability("browserstack.local", false);
            capabilities.SetCapability("browserstack.debug", true);

            capabilities.SetCapability("project", "Project Name");

            Driver = new RemoteWebDriver(new Uri(ConfigurationManager.AppSettings["browserstack.hub"]), capabilities);
            Driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(1));
            ScenarioContext.Current["driver"] = Driver;
        }

Config for Cross Browser spin up

One of our config files might look like so:

<xml version="1.0" encoding="utf-8">
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings>
        <add key="browser" value="Safari" xdt:Transform="Insert"/>
        <add key="os" value="osx" xdt:Transform="Insert"/>
        <add key="version" value="8" xdt:Transform="Insert"/>
        <add key="os_version" value="Yosemite" xdt:Transform="Insert"/>
    </appSettings>
</configuration>

This tells Browserstack what system to spin up and run the tests against for one parallel instance, multiple config files with other systems are needed so they can all be executed depending on your requirements. Furthermore we need to set these up in Configuration Manager so that each config picks up the tests when the solution is built.

What This Enables

We can select what environment we wish to run against, run the test and see it appear in Browserstack (not in parallel) or run it locally using our own setup.

Bstack_1

So now we have essentially got a single test running in the cloud (and our config still allows us to run locally if we choose), next we need to kick off a bunch of these against different systems.

The Build Script

We use a PowerShell build script to achieve the parallel part of this, here it is:

$solution = "Your.Testing.Solution.sln"

function Get-SolutionConfigurations($solution)
{
        Get-Content $solution |
        Where-Object {$_ -match "(?&lt;config&gt;\w+)\|"} |
        %{ $($Matches['config'])} |
        select -uniq
}

$frameworkDirs = @((Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\MSBuild\ToolsVersions\12.0" -Name "MSBuildToolsPath32")."MSBuildToolsPath32",
                        "$env:windir\Microsoft.NET\Framework\v4.0.30319")
    for ($i = 0; $i -lt $frameworkDirs.Count; $i++) {
        $dir = $frameworkDirs[$i]
        if ($dir -Match "\$\(Registry:HKEY_LOCAL_MACHINE(.*?)@(.*)\)") {
            $key = "HKLM:" + $matches[1]
            $name = $matches[2]
            $dir = (Get-ItemProperty -Path $key -Name $name).$name
            $frameworkDirs[$i] = $dir
        }
    }

    $env:path = ($frameworkDirs -join ";") + ";$env:path"

@(Get-SolutionConfigurations $solution) | foreach {
      Write-Host "Building for $_"
    msbuild $solution /p:Configuration=$_ /nologo /verbosity:quiet
}
 
 New-Item "$(get-location)\packages\specflow.1.9.0\tools\specflow.exe.config" -type file -force -value "&lt;?xml version=""1.0"" encoding=""utf-8"" ?&gt; &lt;configuration&gt; &lt;startup&gt; &lt;supportedRuntime version=""v4.0.30319"" /&gt; &lt;/startup&gt; &lt;/configuration&gt;" | Out-Null

@(Get-SolutionConfigurations $solution)| foreach {
    Start-Job -ScriptBlock {
        param($configuration, $basePath)

        try
        {
            &amp; $basePath\packages\NUnit.Runners.2.6.4\tools\nunit-console.exe /labels /out=$basePath\nunit_$configuration.txt /xml:$basePath\nunit_$configuration.xml /nologo /config:$configuration "$basePath/Your.Testing.Solution/bin/$configuration/Your.Testing.Solution.dll"
        }
        finally
        {
            &amp; $basePath\packages\specflow.1.9.0\tools\specflow.exe nunitexecutionreport "$basePath\Your.Testing.Solution\Your.Testing.Solution.csproj" /out:$basePath\specresult_$configuration.html /xmlTestResult:$basePath\nunit_$configuration.xml /testOutput:nunit_$configuration.txt
        }

    } -ArgumentList $_, $(get-location)
}
Get-Job | Wait-Job
Get-Job | Receive-Job

It obviously needs tweaking in areas to point to your solution but what the hell is it actually doing? It is running msbuild against each config file and executing the test suite. Our Safari based config gets built and executed against BrowserStack as do any other configurations. A test report for each config/ system is then generated to let you know the outcome of that particular run.

Running the script should allow you to see parallel test execution against your desired systems in Browserstack:

Bstack_3

Generated Feedback and Reporting

Reports generated after look similar to the below:

Bstack_2

What Next?

You will undoubtedly encounter issues with Browserstack seeing your internal test environments, this is something that you will need to consider. Consult the Browserstack documentation online regarding tunnelling or running Browserstack locally.

We can also throw in some automated visual checking to really get the ball rolling with Continuous Delivery. If you have some well set up Applitools Eyes base images then visual checking on many systems at once is potentially worth thousands of checks.



Automation with AutoIt

This is a tool primarily used for automating windows applications but can also be used in conjunction with Selenium to deal with any windows dialogue boxes that Selenium on its own cannot manipulate. It can also interact with Active X dialogues. Generally speaking it is bad practice to implement this tool in your framework as it will only work with windows but if for example you find yourself on a project where you have little or no option it can be very useful, I have personally used this for a legacy application that was windows only and relied heavily on ActiveX but I would not recommend it for anything else!

Selenium on it’s own can do a lot with it’s builder actions and abilities to handle (to an extent) alert boxes, so don’t bring this into your solution unless you really need it. This is good for legacy applications that are not very browser friendly.

First off, download AutoIt:

www.autoitscript.com

AutoItX

Rather than go through the more traditional route of authoring VB Script in Auto IT’s Scite Editor (Explained below) we can import AutoItX – DLL/COM control which features a C# assembly. You still need to install AutoIt and also add the assembly to your references in Visual Studio.

Once setup we can start authoring C# with the goodies contained in AutoItX, lets dig in with an example.

public static void HandlePrintDialogue()
        {
            au3.WinWait(AiWindows.PrintWindow, "", 10);
            au3.WinActivate(AiWindows.PrintWindow);
            au3.Send(AiKeystrokes.Enter);
        }

This code handles IE’s print dialogue window after the user has clicked a print button on the page or pressed CTRL + P. To begin with we are waiting for a window with the title “Print” to appear on screen within 10 seconds. Please bear in mind the window you are looking for may be names differently in different versions of windows.

au3.WinWait(AiWindows.PrintWindow, "", 10);

Once found we are activating that window to be be manipulated in some way.

au3.WinActivate(AiWindows.PrintWindow);

Then we send an enter key press because with careful observation we know Enter defaults to “Ok” and the print job will be started.

au3.Send(AiKeystrokes.Enter);

You can do a lot more with AutoItX including making assertions on the content of windows dialogues, handling downloads and more.

Please refer to the following for a full list of functions: https://www.autoitscript.com/autoit3/docs/functions.htm

Scite Editor

Open the Script editor and type your VB Script to deal with the dialogue/ issue in question. In this example we are only sending keystrokes to the active window if it is titled as “Login”.

If Winactive ("Login") = 0 Then
   WinActivate("Login")
EndIf

Send ("Username")
Send ("{TAB}")
Send ("Password")
Send ("{ENTER}")

We can save this and compile it, AutoIt will create an *.exe which we can execute from our step definition in the selenium code:

public void HandleLoginPrompt()
        {
            Runtime run = Runtime.getRuntime();
            Process pp = run.exec("C:\\PathToFile\\AILoginScript.exe");
        }

Run the test and see the results.



Load Testing made easy with BlazeMeter & JMeter

Continuing for a moment with load testing lets have a look at Blaze Meter…

What the tools are

  • BlazeMeter: A Chrome App that allows you to record journeys through a system and export a load test plan to JMX Format. You will need to sign up for an account. BlazeMeter.com
  • JMeter: A load testing tool that can run files in JMX format. JMeter.Apache.org

How to create a load test plan

If installed correctly you should see the following logo in Chrome BlazeMeter

When clicked the BlazeMeter panel is displayed, allowing you to input a test plan name and change a few other basic parameters such as the type of system doing the load test and the origin of the traffic.

BlazeMeter_Panel

All you need to do is hit record and perform the actions you wish to load test.

When you have completed this you can export the captured JSON into a JMX file and download it by clicking the .JMX button on the panel.

Open and edit with JMeter

Open your newly downloaded test plan in JMeter, change the threads/ ramp up and loop count to whatever is desired.

Next add the appropriate listeners to your plan and hit Run. Job done.

BlazeMeter_JMX




top