Automated Tester

.science

This content shows Simple View

PowerShell

Setting Up Allure with NUnit in TeamCity

This article is a follow up from my previous article in setting up Allure locally with NUnit. I made a brief mention of the Allure TeamCity plugin but encountered difficulties in getting it to work and didn’t have much time to look into getting around it.

Lets get stuck in with a brief recap/ some initial steps to take.

  • Install NUnit 2.6.4 and the compatible NUnit adapter into your allure bin/addins folder
  • Edit the config.xml in your bin/addins folder to write out the XML to a known location (I’m using C:\AllureXML)

Generated Allure XML

  • Run some tests in the GUI or NUnit Console and generate some XML, you should check XML is being generated before continuing
  • Generate a report with the Command Line Interface (this will verify your JAVA_HOME system variable is present and correct)

Allure TeamCity Setup and Configuration

  • Install the plugin and restart the build server
  • Copy the allure-commandline.zip from latest release to the <TeamCity Data Directory>/plugins/.tools – No server restart needed for this step.
  • Your build needs to be generating XML, we tested this earlier
  • Open your build config settings, create a new build step for the Allure Report
  • Specify a relative path to your XML folder (in this example it is ../../../AllureXML)
  • Use the given example for test reports or similar

Allure Report Build Step

  • Edit your build step running the tests – I have not been able to get it to work with the NUnit runner type (it just won’t generate any XML) however, we can run a script to spin up the NUnit Console and run it that way.
  • Choose PowerShell as the runner, x86 bitness
  • Script: Source code
  • The script:
&amp; 'C:\Program Files (x86)\NUnit 2.6.4\bin\nunit-console-x86.exe' /framework=net-4.0 C:\BuildAgent\work\a0569b5caa0eb74d\Automation\Tests\Example.Automated.Selenium.Tests\Example.Automated.Selenium.Tests\bin\Release\Example.Automated.Selenium.Tests.dll

 

Test run build step for Nunit and Allure

It is a good idea to test the script on the build machine locally so you can ensure it is correct, before putting it into your TeamCity build step.

Notice a couple of things:

  • We run the x86 bit console as specified in the powershell run mode in the build step
  • We have passed in  “/framework=net-4.0”, this is important and in line with current documentation.

This should now, when run, kick off the tests via the console, generate the necessary XML and publish the artifacts in TeamCity giving you a report, which is great for retrieving daily summaries of builds you run overnight which is a great health check on your continuous integration builds.

Further Help

Please keep these installation notes at the forefront of your mind if you encounter difficulties:

If you wish to add different parameters to your tests you can find a full list below or by running  “nunit-console-x86.exe /help”

NUNIT-CONSOLE [inputfiles] [options]

Runs a set of NUnit tests from the console.

You may specify one or more assemblies or a single project file of type .nunit.

Options:

/fixture=STR               Test fixture or namespace to be loaded (Deprecated) (Short format: /load=STR)
/run=STR                   Name of the test case(s), fixture(s) or namespace(s) to run
/runlist=STR               Name of a file containing a list of the tests to run, one per line
/config=STR                Project configuration (e.g.: Debug) to load
/result=STR                Name of XML result file (Default: TestResult.xml) (Short format: /xml=STR)
/xmlConsole                Display XML to the console (Deprecated)
/noresult                  Suppress XML result output (Short format: /noxml)
/output=STR                File to receive test output (Short format: /out=STR)
/err=STR                   File to receive test error output
/work=STR                  Work directory for output files
/labels                    Label each test in stdOut
/trace=X                   Set internal trace level: Off, Error, Warning, Info, Verbose
/include=STR               List of categories to include
/exclude=STR               List of categories to exclude
/framework=STR             Framework version to be used for tests
/process=X                 Process model for tests: Single, Separate, Multiple
/domain=X                  AppDomain Usage for tests: None, Single, Multiple
/apartment=X               Apartment for running tests: MTA (Default), STA
/noshadow                  Disable shadow copy when running in separate domain
/nothread                  Disable use of a separate thread for tests
/basepath=STR              Base path to be used when loading the assemblies
/privatebinpath=STR        Additional directories to be probed when loading assemblies, separated by semicolons
/timeout=X                 Set timeout for each test case in milliseconds
/wait                      Wait for input before closing console window
/nologo                    Do not display the logo
/nodots                    Do not display progress
/stoponerror               Stop after the first test failure or error
/cleanup                   Erase any leftover cache files and exit
/help                      Display help (Short format: /?)



Parallel Test Execution with Specflow

Running Specflow scenarios in Parallel is a tricky problem to solve due to the nature of how tests are written. Achieving this is a lot simpler if we write Selenium tests in a unit style fashion but as this isn’t the case for Specflow with its Gherkin syntax we will look at executing them with a powershell build script against a Browserstack grid.

Thanks to this great article from Kenneth Truyers, it has really helped me to achieve this. We will look at executing parallel Specflow tests against a BrowserStack grid.

Set Up

In our Initialisation method we need to specify our remote driver whilst passing in our varying desired capabilities in Config.

private static void SetupCloudDriver()
        {
            var capabilities = new DesiredCapabilities();

            capabilities.SetCapability(CapabilityType.Version, ConfigurationManager.AppSettings["version"]);
            capabilities.SetCapability("os", ConfigurationManager.AppSettings["os"]);
            capabilities.SetCapability("os_version", ConfigurationManager.AppSettings["os_version"]);
            capabilities.SetCapability("browserName", ConfigurationManager.AppSettings["browser"]);

            capabilities.SetCapability("browserstack.user", ConfigurationManager.AppSettings["browserstack.user"]);
            capabilities.SetCapability("browserstack.key", ConfigurationManager.AppSettings["browserstack.key"]);
            capabilities.SetCapability("browserstack.local", false);
            capabilities.SetCapability("browserstack.debug", true);

            capabilities.SetCapability("project", "Project Name");

            Driver = new RemoteWebDriver(new Uri(ConfigurationManager.AppSettings["browserstack.hub"]), capabilities);
            Driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(1));
            ScenarioContext.Current["driver"] = Driver;
        }

Config for Cross Browser spin up

One of our config files might look like so:

<xml version="1.0" encoding="utf-8">
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings>
        <add key="browser" value="Safari" xdt:Transform="Insert"/>
        <add key="os" value="osx" xdt:Transform="Insert"/>
        <add key="version" value="8" xdt:Transform="Insert"/>
        <add key="os_version" value="Yosemite" xdt:Transform="Insert"/>
    </appSettings>
</configuration>

This tells Browserstack what system to spin up and run the tests against for one parallel instance, multiple config files with other systems are needed so they can all be executed depending on your requirements. Furthermore we need to set these up in Configuration Manager so that each config picks up the tests when the solution is built.

What This Enables

We can select what environment we wish to run against, run the test and see it appear in Browserstack (not in parallel) or run it locally using our own setup.

Bstack_1

So now we have essentially got a single test running in the cloud (and our config still allows us to run locally if we choose), next we need to kick off a bunch of these against different systems.

The Build Script

We use a PowerShell build script to achieve the parallel part of this, here it is:

$solution = "Your.Testing.Solution.sln"

function Get-SolutionConfigurations($solution)
{
        Get-Content $solution |
        Where-Object {$_ -match "(?&lt;config&gt;\w+)\|"} |
        %{ $($Matches['config'])} |
        select -uniq
}

$frameworkDirs = @((Get-ItemProperty -Path "HKLM:\SOFTWARE\Microsoft\MSBuild\ToolsVersions\12.0" -Name "MSBuildToolsPath32")."MSBuildToolsPath32",
                        "$env:windir\Microsoft.NET\Framework\v4.0.30319")
    for ($i = 0; $i -lt $frameworkDirs.Count; $i++) {
        $dir = $frameworkDirs[$i]
        if ($dir -Match "\$\(Registry:HKEY_LOCAL_MACHINE(.*?)@(.*)\)") {
            $key = "HKLM:" + $matches[1]
            $name = $matches[2]
            $dir = (Get-ItemProperty -Path $key -Name $name).$name
            $frameworkDirs[$i] = $dir
        }
    }

    $env:path = ($frameworkDirs -join ";") + ";$env:path"

@(Get-SolutionConfigurations $solution) | foreach {
      Write-Host "Building for $_"
    msbuild $solution /p:Configuration=$_ /nologo /verbosity:quiet
}
 
 New-Item "$(get-location)\packages\specflow.1.9.0\tools\specflow.exe.config" -type file -force -value "&lt;?xml version=""1.0"" encoding=""utf-8"" ?&gt; &lt;configuration&gt; &lt;startup&gt; &lt;supportedRuntime version=""v4.0.30319"" /&gt; &lt;/startup&gt; &lt;/configuration&gt;" | Out-Null

@(Get-SolutionConfigurations $solution)| foreach {
    Start-Job -ScriptBlock {
        param($configuration, $basePath)

        try
        {
            &amp; $basePath\packages\NUnit.Runners.2.6.4\tools\nunit-console.exe /labels /out=$basePath\nunit_$configuration.txt /xml:$basePath\nunit_$configuration.xml /nologo /config:$configuration "$basePath/Your.Testing.Solution/bin/$configuration/Your.Testing.Solution.dll"
        }
        finally
        {
            &amp; $basePath\packages\specflow.1.9.0\tools\specflow.exe nunitexecutionreport "$basePath\Your.Testing.Solution\Your.Testing.Solution.csproj" /out:$basePath\specresult_$configuration.html /xmlTestResult:$basePath\nunit_$configuration.xml /testOutput:nunit_$configuration.txt
        }

    } -ArgumentList $_, $(get-location)
}
Get-Job | Wait-Job
Get-Job | Receive-Job

It obviously needs tweaking in areas to point to your solution but what the hell is it actually doing? It is running msbuild against each config file and executing the test suite. Our Safari based config gets built and executed against BrowserStack as do any other configurations. A test report for each config/ system is then generated to let you know the outcome of that particular run.

Running the script should allow you to see parallel test execution against your desired systems in Browserstack:

Bstack_3

Generated Feedback and Reporting

Reports generated after look similar to the below:

Bstack_2

What Next?

You will undoubtedly encounter issues with Browserstack seeing your internal test environments, this is something that you will need to consider. Consult the Browserstack documentation online regarding tunnelling or running Browserstack locally.

We can also throw in some automated visual checking to really get the ball rolling with Continuous Delivery. If you have some well set up Applitools Eyes base images then visual checking on many systems at once is potentially worth thousands of checks.




top