Automated Tester

.science

This content shows Simple View

Selenium

Parallel Test Execution with Specflow

Running Specflow scenarios in Parallel is a tricky problem to solve due to the nature of how tests are written. Achieving this is a lot simpler if we write Selenium tests in a unit style fashion but as this isn’t the case for Specflow with its Gherkin syntax we will look at executing them with a powershell build script against a Browserstack grid.

Thanks to this great article from Kenneth Truyers, it has really helped me to achieve this. We will look at executing parallel Specflow tests against a BrowserStack grid.

Set Up

In our Initialisation method we need to specify our remote driver whilst passing in our varying desired capabilities in Config.

private static void SetupCloudDriver()
        {
            var capabilities = new DesiredCapabilities();

            capabilities.SetCapability(CapabilityType.Version, ConfigurationManager.AppSettings["version"]);
            capabilities.SetCapability("os", ConfigurationManager.AppSettings["os"]);
            capabilities.SetCapability("os_version", ConfigurationManager.AppSettings["os_version"]);
            capabilities.SetCapability("browserName", ConfigurationManager.AppSettings["browser"]);

            capabilities.SetCapability("browserstack.user", ConfigurationManager.AppSettings["browserstack.user"]);
            capabilities.SetCapability("browserstack.key", ConfigurationManager.AppSettings["browserstack.key"]);
            capabilities.SetCapability("browserstack.local", false);
            capabilities.SetCapability("browserstack.debug", true);

            capabilities.SetCapability("project", "Project Name");

            Driver = new RemoteWebDriver(new Uri(ConfigurationManager.AppSettings["browserstack.hub"]), capabilities);
            Driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(1));
            ScenarioContext.Current["driver"] = Driver;
        }

Config for Cross Browser spin up

One of our config files might look like so:

<xml version="1.0" encoding="utf-8">
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings>
        <add key="browser" value="Safari" xdt:Transform="Insert"/>
        <add key="os" value="osx" xdt:Transform="Insert"/>
        <add key="version" value="8" xdt:Transform="Insert"/>
        <add key="os_version" value="Yosemite" xdt:Transform="Insert"/>
    </appSettings>
</configuration>

This tells Browserstack what system to spin up and run the tests against for one parallel instance, multiple config files with other systems are needed so they can all be executed depending on your requirements. Furthermore we need to set these up in Configuration Manager so that each config picks up the tests when the solution is built.

What This Enables

We can select what environment we wish to run against, run the test and see it appear in Browserstack (not in parallel) or run it locally using our own setup.

Bstack_1

So now we have essentially got a single test running in the cloud (and our config still allows us to run locally if we choose), next we need to kick off a bunch of these against different systems.

The Build Script

We use a PowerShell build script to achieve the parallel part of this, here it is:

$solution = "Your.Testing.Solution.sln"

function Get-SolutionConfigurations($solution)
{
        Get-Content $solution |
        Where-Object {$_ -match "(?&lt;config&gt;\w+)\|"} |
        %{ $($Matches['config'])} |
        select -uniq
}

$frameworkDirs = @((Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\MSBuild\ToolsVersions\12.0" -Name "MSBuildToolsPath32")."MSBuildToolsPath32",
                        "$env:windir\Microsoft.NET\Framework\v4.0.30319")
    for ($i = 0; $i -lt $frameworkDirs.Count; $i++) {
        $dir = $frameworkDirs[$i]
        if ($dir -Match "\$\(Registry:HKEY_LOCAL_MACHINE(.*?)@(.*)\)") {
            $key = "HKLM:" + $matches[1]
            $name = $matches[2]
            $dir = (Get-ItemProperty -Path $key -Name $name).$name
            $frameworkDirs[$i] = $dir
        }
    }

    $env:path = ($frameworkDirs -join ";") + ";$env:path"

@(Get-SolutionConfigurations $solution) | foreach {
      Write-Host "Building for $_"
    msbuild $solution /p:Configuration=$_ /nologo /verbosity:quiet
}
 
 New-Item "$(get-location)\packages\specflow.1.9.0\tools\specflow.exe.config" -type file -force -value "&lt;?xml version=""1.0"" encoding=""utf-8"" ?&gt; &lt;configuration&gt; &lt;startup&gt; &lt;supportedRuntime version=""v4.0.30319"" /&gt; &lt;/startup&gt; &lt;/configuration&gt;" | Out-Null

@(Get-SolutionConfigurations $solution)| foreach {
    Start-Job -ScriptBlock {
        param($configuration, $basePath)

        try
        {
            &amp; $basePath\packages\NUnit.Runners.2.6.4\tools\nunit-console.exe /labels /out=$basePath\nunit_$configuration.txt /xml:$basePath\nunit_$configuration.xml /nologo /config:$configuration "$basePath/Your.Testing.Solution/bin/$configuration/Your.Testing.Solution.dll"
        }
        finally
        {
            &amp; $basePath\packages\specflow.1.9.0\tools\specflow.exe nunitexecutionreport "$basePath\Your.Testing.Solution\Your.Testing.Solution.csproj" /out:$basePath\specresult_$configuration.html /xmlTestResult:$basePath\nunit_$configuration.xml /testOutput:nunit_$configuration.txt
        }

    } -ArgumentList $_, $(get-location)
}
Get-Job | Wait-Job
Get-Job | Receive-Job

It obviously needs tweaking in areas to point to your solution but what the hell is it actually doing? It is running msbuild against each config file and executing the test suite. Our Safari based config gets built and executed against BrowserStack as do any other configurations. A test report for each config/ system is then generated to let you know the outcome of that particular run.

Running the script should allow you to see parallel test execution against your desired systems in Browserstack:

Bstack_3

Generated Feedback and Reporting

Reports generated after look similar to the below:

Bstack_2

What Next?

You will undoubtedly encounter issues with Browserstack seeing your internal test environments, this is something that you will need to consider. Consult the Browserstack documentation online regarding tunnelling or running Browserstack locally.

We can also throw in some automated visual checking to really get the ball rolling with Continuous Delivery. If you have some well set up Applitools Eyes base images then visual checking on many systems at once is potentially worth thousands of checks.



Load Testing with Selenium and BrowserMob Proxy

In days of yore, Selenium had the capability to capture network traffic and manipulate it. This was taken out of later versions as the mantra for the project was to mimic user actions.

We can still harness this capability and do some other cool things along the way such as blacklisting or whitelisting certain services, simulate slow connections, write data and produce metrics.

How To

We can only harness this capability by spinning up our tests via a proxy and that is where our first tool comes in; BrowserMob Proxy. There is also a write up on it here and which coincidentally I lifted some of the code from: Ada The Dev – BrowserMob Proxy, the blog of one of the .Net Gurus of BrowserMob.

When we are setting up our start up method we need to instantiate the proxy and then spin up the browser.

// Supply the path to the Browsermob Proxy batch file
            Server server = new Server(@"C:\Users\user.name\Desktop\BMP\bin\browsermob-proxy.bat");
            server.Start();
            Client client = server.CreateProxy();
            client.NewHar("Load Test Numbers");
            var seleniumProxy = new Proxy { HttpProxy = client.SeleniumProxy };
            var profile = new FirefoxProfile();
            profile.SetProxyPreferences(seleniumProxy);

            // Navigate to the page to retrieve performance stats for
            Driver = new FirefoxDriver(profile);
            Driver.Navigate().GoToUrl("http://www.google.co.uk");

After we have started the proxy we are instructing the client to create a new HAR file called “Load Test Numbers”. A HAR file is a Http Archive. More information can be found on that here: Har File Spec v1.2.

After navigating to Google we need to get the performance statistics and then view them in some way. This is easy to do in debug mode, looking into variables or writing the content out to the console.

// Get the performance stats
         HarResult harData = client.GetHar();

 // Do whatever you want with the metrics here. Easy to persist
 Log log = harData.Log;
            Entry[] entries = log.Entries;

            var file = new System.IO.StreamWriter("c:\\test.txt");
           
            foreach (var entry in entries)
            {
                Request request = entry.Request;
                Response response = entry.Response;
                var url = request.Url;
                var time = entry.Time;
                var status = response.Status;
                Console.WriteLine("Url: " + url + " - Time: " + time + " Response: " + status);

                file.WriteLine("Url: " + url + " - Time: " + time + " Response: " + status);
            }

            file.Close();

In the code above we a getting select elements from the HAR (URL, Time and Status Code) and writing them out to the console and also a text file.

Viewing the entire HAR

We can capture the JSON generated in the HAR file and write out the HAR to disk. Using this HAR we can use another tool to view it and it’s metrics. We can also convert the HAR to a JMX file which creates for us a JMeter Test plan.

Once we retrieve the performance stats we need to serialise the content:

// Get the performance stats
            HarResult harData = client.GetHar();

            var json = new JavaScriptSerializer().Serialize(harData);
           
            Log log = harData.Log;
            Entry[] entries = log.Entries;

            var file = new System.IO.StreamWriter("c:\\test.har");
            file.WriteLine(json);

This writes a serialised HAR file out to file. If we paste the content into HAR Viewer we magically get a whole bunch of useful metrics:

HAR1

HAR2

Converting to JMX

This is simple to do, as with pasting your HAR data into HAR Viewer. You can paste the same JSON in Flood.IO Har2JMX and download the JMX. Now simply load into JMeter, change the Thread number, ramp up, loops and then add desired listeners and you are ready to roll.

Unfortunately there is currently no way of programatically converting HAR2JMX in C#. The Flood.IO code is open source however.

BlazeMeter_JMX

Exporting a HAR straight from the Browser

To do this you need FireBug and NetExport. Once installed you can save all network traffic logged in FireBug to a HAR format and even view in firebug by pressing F12. If you look under the Net tab. Selecting Export >> Save As will get you your HAR.

HAR3



Structuring Projects with BDD, Specflow and Selenium in C#

Typically when I create a Specflow BDD solution, BDD elements are separated into Feature Files, Hooks, Step Definitions and Pages. These elements are separate folders or subsets of folders within the solution.
Code begins with the feature files written in Gherkin, feature files contain Scenarios which themselves execute test cases. Sometimes a scenario is just one test case or can be many iterations of a particular action. An example is listed below:

Scenario Outline: Sitemap Links
 And I click the sitemap link 'LinkName'
 Then I am taken to the sitemap page for 'LinkName'
Examples:
 | LinkName              |
 | Contact Us            |
 | Terms and Conditions  |
 | Privacy Policy        |

Generally we perform a set of actions and then assert the outcome at the end, in the scenario above we are searching for links in the sitemap section of the footer, selecting a result from a list and then asserting that we are taken to that particular page.

It helps greatly if these features are written in collaboration. In an ideal world features should be authored in a “three amigos” style with the project Business Analyst and Product Owner.

Gherkin generates Step Definitions which can be used across a set of feature files. It makes sense to group common ones together in one big Step Definition file for actions such as navigating to the site or logging in/out of the application. More feature specific steps are kept in separate files.

It is our aim to keep the code in step definitions short and clean, we do this by wrapping up selenium code and putting these methods in to the Pages .cs files. For example:

Given I have navigated to Test Environment

Generates the following Step Definition:

[Given(@"I have navigated to (.*)"]
public void GivenIHaveNavigatedTo(string website)
{

}

From here in one of the common Pages .cs files we have created the following method:

public void NavigateTo(string url)
{
this.context.Driver.Navigate().GoToUrl(url);
}

We try to keep the Selenium code wrapped up in these statements to keep the Step definitions easy to write. With this method in place our step definition looks like this:

[Given(@"I have navigated to (.*)"]
public void GivenIHaveNavigatedTo(string website)
{
Page.NavigateTo(URLS.TestEnvironment1)
}

Organising Tests
Tests are organised with @Tags which make groups of or individual tests easy to find in MS Test. They also have other important uses in hooks and can be used to execute a set of tests on a CI build.

For example all Sanity tests are labelled with @_Sanity this makes the whole suite of sanity tests easy to find and execute. If this tag was passed to a CI build then rather handily just the sanity tests would run on an overnight build rather than a lengthier @_Regression suite.

What’s with the underscore? Well that just keeps your tags at the top of the MS Test list if you happen to be using that test framework, depending on your preference of viewing them.

Hooks

Hooks are used primarily for the test run set up and teardown. Before a run we want to instantiate a driver and spin a browser up before any scenarios are automated, otherwise it would go nowhere fast. Similarly at the end of a test run we want to kill the browser off.

You can be a bit more clever with hooks and execute bits of code before or after certain features, steps, scenarios or scenario blocks. In order to keep an easy to understand codebase these must be used sparingly. This is because debugging can be harder and feature files can have all sorts of things happening that aren’t specified in English and that’s not what BDD is intended for.

For more information on hooks please visit: https://github.com/techtalk/SpecFlow/wiki/Hooks

Pages

This section of classes is where we keep all of the wrapped up Selenium code, which is a bit nasty. We wrap them up into helper methods to make our step definitions easier to write as described previously. This code should be kept clean and refactored and should encapsulate common practices and apply sensible programming practices such as DRY (Don’t Repeat Yourself).

The Selectors Page is where we keep a large number of constants, this is so that if an Id changes on the System Under Test you will only ever need to change it once to fix your broken test(s).

Rules

We also keep in mind a set of rules  when structuring the code which are:

  • No Selenium Code in Step Definitions
  • No Project Specific Methods in PageBase, use MainPage for these
  • All By.Criteria to be defined and referenced in selectors page
  • URLs and base URLs that aren’t in Gherkin must be constants
  • Create triptych Given/When/Then step definitions where necessary
  • Always copy generated methods to clipboard
  • No Thread.Sleeps in any step definitions, wait for something instead
  • Regions, regions, regions – use regions in code to keep it nice and tidy
  • “That” is a banned word in Gherkin
  • Replace all auto generated variable names with something sensible
  • All assertions have the optional message parameter populated except for AreEqual
  • Split Step Definitions out into sensible groupings/ filenames

Diagram

BDD Solution Architecture
BDD Solution Architecture

Base Solution

You can find a base solution here.

Wrap Up

There is a lot more you can do to improve this solution like putting in a clever environment selector, addition of Sauce Labs or BrowserStack and also Applitools Eyes. The way this is build is designed to give you scalability and organisation to cope as your test suite grows but you have to be strict with yourself and keep in mind the BDD Rules set out or things can get messy very quickly.




top