Automated Tester

.science

This content shows Simple View

BrowserStack

Parallel Test Execution with Specflow

Running Specflow scenarios in Parallel is a tricky problem to solve due to the nature of how tests are written. Achieving this is a lot simpler if we write Selenium tests in a unit style fashion but as this isn’t the case for Specflow with its Gherkin syntax we will look at executing them with a powershell build script against a Browserstack grid.

Thanks to this great article from Kenneth Truyers, it has really helped me to achieve this. We will look at executing parallel Specflow tests against a BrowserStack grid.

Set Up

In our Initialisation method we need to specify our remote driver whilst passing in our varying desired capabilities in Config.

private static void SetupCloudDriver()
        {
            var capabilities = new DesiredCapabilities();

            capabilities.SetCapability(CapabilityType.Version, ConfigurationManager.AppSettings["version"]);
            capabilities.SetCapability("os", ConfigurationManager.AppSettings["os"]);
            capabilities.SetCapability("os_version", ConfigurationManager.AppSettings["os_version"]);
            capabilities.SetCapability("browserName", ConfigurationManager.AppSettings["browser"]);

            capabilities.SetCapability("browserstack.user", ConfigurationManager.AppSettings["browserstack.user"]);
            capabilities.SetCapability("browserstack.key", ConfigurationManager.AppSettings["browserstack.key"]);
            capabilities.SetCapability("browserstack.local", false);
            capabilities.SetCapability("browserstack.debug", true);

            capabilities.SetCapability("project", "Project Name");

            Driver = new RemoteWebDriver(new Uri(ConfigurationManager.AppSettings["browserstack.hub"]), capabilities);
            Driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(1));
            ScenarioContext.Current["driver"] = Driver;
        }

Config for Cross Browser spin up

One of our config files might look like so:

<xml version="1.0" encoding="utf-8">
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings>
        <add key="browser" value="Safari" xdt:Transform="Insert"/>
        <add key="os" value="osx" xdt:Transform="Insert"/>
        <add key="version" value="8" xdt:Transform="Insert"/>
        <add key="os_version" value="Yosemite" xdt:Transform="Insert"/>
    </appSettings>
</configuration>

This tells Browserstack what system to spin up and run the tests against for one parallel instance, multiple config files with other systems are needed so they can all be executed depending on your requirements. Furthermore we need to set these up in Configuration Manager so that each config picks up the tests when the solution is built.

What This Enables

We can select what environment we wish to run against, run the test and see it appear in Browserstack (not in parallel) or run it locally using our own setup.

Bstack_1

So now we have essentially got a single test running in the cloud (and our config still allows us to run locally if we choose), next we need to kick off a bunch of these against different systems.

The Build Script

We use a PowerShell build script to achieve the parallel part of this, here it is:

$solution = "Your.Testing.Solution.sln"

function Get-SolutionConfigurations($solution)
{
        Get-Content $solution |
        Where-Object {$_ -match "(?&lt;config&gt;\w+)\|"} |
        %{ $($Matches['config'])} |
        select -uniq
}

$frameworkDirs = @((Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\MSBuild\ToolsVersions\12.0" -Name "MSBuildToolsPath32")."MSBuildToolsPath32",
                        "$env:windir\Microsoft.NET\Framework\v4.0.30319")
    for ($i = 0; $i -lt $frameworkDirs.Count; $i++) {
        $dir = $frameworkDirs[$i]
        if ($dir -Match "\$\(Registry:HKEY_LOCAL_MACHINE(.*?)@(.*)\)") {
            $key = "HKLM:" + $matches[1]
            $name = $matches[2]
            $dir = (Get-ItemProperty -Path $key -Name $name).$name
            $frameworkDirs[$i] = $dir
        }
    }

    $env:path = ($frameworkDirs -join ";") + ";$env:path"

@(Get-SolutionConfigurations $solution) | foreach {
      Write-Host "Building for $_"
    msbuild $solution /p:Configuration=$_ /nologo /verbosity:quiet
}
 
 New-Item "$(get-location)\packages\specflow.1.9.0\tools\specflow.exe.config" -type file -force -value "&lt;?xml version=""1.0"" encoding=""utf-8"" ?&gt; &lt;configuration&gt; &lt;startup&gt; &lt;supportedRuntime version=""v4.0.30319"" /&gt; &lt;/startup&gt; &lt;/configuration&gt;" | Out-Null

@(Get-SolutionConfigurations $solution)| foreach {
    Start-Job -ScriptBlock {
        param($configuration, $basePath)

        try
        {
            &amp; $basePath\packages\NUnit.Runners.2.6.4\tools\nunit-console.exe /labels /out=$basePath\nunit_$configuration.txt /xml:$basePath\nunit_$configuration.xml /nologo /config:$configuration "$basePath/Your.Testing.Solution/bin/$configuration/Your.Testing.Solution.dll"
        }
        finally
        {
            &amp; $basePath\packages\specflow.1.9.0\tools\specflow.exe nunitexecutionreport "$basePath\Your.Testing.Solution\Your.Testing.Solution.csproj" /out:$basePath\specresult_$configuration.html /xmlTestResult:$basePath\nunit_$configuration.xml /testOutput:nunit_$configuration.txt
        }

    } -ArgumentList $_, $(get-location)
}
Get-Job | Wait-Job
Get-Job | Receive-Job

It obviously needs tweaking in areas to point to your solution but what the hell is it actually doing? It is running msbuild against each config file and executing the test suite. Our Safari based config gets built and executed against BrowserStack as do any other configurations. A test report for each config/ system is then generated to let you know the outcome of that particular run.

Running the script should allow you to see parallel test execution against your desired systems in Browserstack:

Bstack_3

Generated Feedback and Reporting

Reports generated after look similar to the below:

Bstack_2

What Next?

You will undoubtedly encounter issues with Browserstack seeing your internal test environments, this is something that you will need to consider. Consult the Browserstack documentation online regarding tunnelling or running Browserstack locally.

We can also throw in some automated visual checking to really get the ball rolling with Continuous Delivery. If you have some well set up Applitools Eyes base images then visual checking on many systems at once is potentially worth thousands of checks.




top