Automated Tester

.science

This content shows Simple View

Test Reporting

Specflow Reports with NUnit3

This post will look at setting up Specflow reports to work with NUnit3. When you install Specflow via NuGet, there is a tools folder which contains specflow.exe – if we run a batch process after the build on the CI server which points to the NUnit3 TestResult.xml, then we can generate a report and reference it as an artefact in TeamCity (and similar CI software).

But there is a catch, which isn’t (at the time of writing) explained in the Specflow reporting documentation. Currently Specflow (v 2.0.0) cannot interpret the NUnit3 TestResult output, so to counter this restriction we need to configure NUnit3 to output its TestResult.xml in the NUnit2 format. A full list of NUnit3 console options can be found here.

nunit3-console.exe –where cat==SmokeTests&&cat!=ExcludeFromCI
–out=TestResult.txt
–result=TestResult.xml;format=nunit2 C:\Path\To\Acceptance.Tests.dll

With this in place it’s now just a case of rigging up the Specflow command to point to the Test Result data and telling it to produce a report and specifying where to output the HTML report.

specflow.exe nunitexecutionreport Your.AcceptanceTest.csproj /testResult:TestResult.xml
 /out:MyResult.html

Once you have the TestResult.xml and the Result output generating where you want, it’s easy to wrap the command up into a batch file and call it from a build step after your tests have executed. Make this output an artefact and then you have metrics in place for your nightly and/ or other test builds.

Demo Specflow Report



Setting Up Allure with NUnit in TeamCity

This article is a follow up from my previous article in setting up Allure locally with NUnit. I made a brief mention of the Allure TeamCity plugin but encountered difficulties in getting it to work and didn’t have much time to look into getting around it.

Lets get stuck in with a brief recap/ some initial steps to take.

  • Install NUnit 2.6.4 and the compatible NUnit adapter into your allure bin/addins folder
  • Edit the config.xml in your bin/addins folder to write out the XML to a known location (I’m using C:\AllureXML)

Generated Allure XML

  • Run some tests in the GUI or NUnit Console and generate some XML, you should check XML is being generated before continuing
  • Generate a report with the Command Line Interface (this will verify your JAVA_HOME system variable is present and correct)

Allure TeamCity Setup and Configuration

  • Install the plugin and restart the build server
  • Copy the allure-commandline.zip from latest release to the <TeamCity Data Directory>/plugins/.tools – No server restart needed for this step.
  • Your build needs to be generating XML, we tested this earlier
  • Open your build config settings, create a new build step for the Allure Report
  • Specify a relative path to your XML folder (in this example it is ../../../AllureXML)
  • Use the given example for test reports or similar

Allure Report Build Step

  • Edit your build step running the tests – I have not been able to get it to work with the NUnit runner type (it just won’t generate any XML) however, we can run a script to spin up the NUnit Console and run it that way.
  • Choose PowerShell as the runner, x86 bitness
  • Script: Source code
  • The script:
&amp; 'C:\Program Files (x86)\NUnit 2.6.4\bin\nunit-console-x86.exe' /framework=net-4.0 C:\BuildAgent\work\a0569b5caa0eb74d\Automation\Tests\Example.Automated.Selenium.Tests\Example.Automated.Selenium.Tests\bin\Release\Example.Automated.Selenium.Tests.dll

 

Test run build step for Nunit and Allure

It is a good idea to test the script on the build machine locally so you can ensure it is correct, before putting it into your TeamCity build step.

Notice a couple of things:

  • We run the x86 bit console as specified in the powershell run mode in the build step
  • We have passed in  “/framework=net-4.0”, this is important and in line with current documentation.

This should now, when run, kick off the tests via the console, generate the necessary XML and publish the artifacts in TeamCity giving you a report, which is great for retrieving daily summaries of builds you run overnight which is a great health check on your continuous integration builds.

Further Help

Please keep these installation notes at the forefront of your mind if you encounter difficulties:

If you wish to add different parameters to your tests you can find a full list below or by running  “nunit-console-x86.exe /help”

NUNIT-CONSOLE [inputfiles] [options]

Runs a set of NUnit tests from the console.

You may specify one or more assemblies or a single project file of type .nunit.

Options:

/fixture=STR               Test fixture or namespace to be loaded (Deprecated) (Short format: /load=STR)
/run=STR                   Name of the test case(s), fixture(s) or namespace(s) to run
/runlist=STR               Name of a file containing a list of the tests to run, one per line
/config=STR                Project configuration (e.g.: Debug) to load
/result=STR                Name of XML result file (Default: TestResult.xml) (Short format: /xml=STR)
/xmlConsole                Display XML to the console (Deprecated)
/noresult                  Suppress XML result output (Short format: /noxml)
/output=STR                File to receive test output (Short format: /out=STR)
/err=STR                   File to receive test error output
/work=STR                  Work directory for output files
/labels                    Label each test in stdOut
/trace=X                   Set internal trace level: Off, Error, Warning, Info, Verbose
/include=STR               List of categories to include
/exclude=STR               List of categories to exclude
/framework=STR             Framework version to be used for tests
/process=X                 Process model for tests: Single, Separate, Multiple
/domain=X                  AppDomain Usage for tests: None, Single, Multiple
/apartment=X               Apartment for running tests: MTA (Default), STA
/noshadow                  Disable shadow copy when running in separate domain
/nothread                  Disable use of a separate thread for tests
/basepath=STR              Base path to be used when loading the assemblies
/privatebinpath=STR        Additional directories to be probed when loading assemblies, separated by semicolons
/timeout=X                 Set timeout for each test case in milliseconds
/wait                      Wait for input before closing console window
/nologo                    Do not display the logo
/nodots                    Do not display progress
/stoponerror               Stop after the first test failure or error
/cleanup                   Erase any leftover cache files and exit
/help                      Display help (Short format: /?)



Gallio Test Runner

Gallio (sometimes known as MB Unit or Gallio Icarus) is a test runner which to me has two distinct advantages over MsTest and NUnit.

Parallel test execution support and built in test reports, although NUnit does have parallel support with PNUnit.

How to set it up

  • Download and install Gallio from here.
  • In your Visual Studio Project navigate to Project > [project name] Properties from the drop down menu.
  • Click Debug
  • Set startup program to Gallio, by default it should be somewhere like: C:\Program Files\Gallio\bin\Gallio.Icarus.exe
  • In Start Options > Command Line Arguments paste in the path of your projects DLL housing the tests you want to execute
  • Save
  • Hit Start when you want to run or debug your tests

Gallio_Setup

Gallio GUI

This looks very similar to NUnit but with some extra buttons and menus, the only ones you will really care about are start/ stop/ debug and test reports.

Gallio_GUI

The reports aren’t anywhere near as nice as that you would find in Allure, but they are easily generated and easy to view.

Gallio_Report

Bootnote: In Gallio Icarus select menu Tools -> Options, select page “Preferences”, set “Test Runner Factory” to IsolatedAppDomain or Local to get the debugger to work.



Test Reports with Visual Studio, NUnit and Allure

Getting some decent test reporting can be painful when working with Visual Studio and particularly with MS Test as the framework as newer versions of Visual Studio don’t appear to produce a *.trx file anymore.

Using NUnit we can generate some fancy graphs and stats using the Allure framework with NUnit.

  • Firstly we need to download and install NUnit: www.NUnit.org
  • Secondly we need the Allure NUnit adaptor, follow instructions here: Allure Framework Adaptor for NUnit
  • The Allure Command Line Interface is needed, get it from here: Allure CLI
  • Java JDK needs to be installed and specified correctly in your JAVE_HOME variable in your environment

Java_Home

Allure produces an XML file which can use that to generate a HTML report, there is also an adaptor for MS Test but it’s an extra hoop to jump through as it converts a *.trx into a XML which you then have to generate.

The NUnit adaptor generates the XML file to a directory when your tests are run with NUnit.

To generate the report run cmd in your Allure.bat directory and run the command:

allure report generate –v 1.4.1 path/to/directory/with/xml/files

Note: It is important you DO NOT paste this command in – type it out the long way – You can paste in the path to the XML files though.

You should see the following message:

Successfully generated report to [//Path to generated report]

Go to the directory and open the HTML report in Firefox as other browsers will not work with some of the Ajax requirements locally.

This is your generated report! You might also want to publish it as an artefact in your Team City CI build, with the Allure Team City Plugin.

NOTE: If you are using Specflow with NUnit be sure to install nuget package NUnit.Runners. If this breaks your build, go download the binary version of NUnit (http://www.nunit.org/index.php?p=download) and copy over both nunit.core.dll – nunit.core.interfaces.dll  to be referenced in your solution. This will ensure NUnit sees your [AfterTestRun] hook and perform all necessary teardown tasks.

SummaryxUnitTimeline    GraphsNo Defects

 

Collating multiple sets of results into one report

If you have the need to merge multiple sets of results into a single report this can be done easily by copying the generated XML files out to another directory, running another set of tests and then copying the previous files back over.

When you generate the report from the directory with two sets of XML files, Allure will mash it all together into one report.

You could also try Running Multiple Assemblies with NUnit but this is not recommended in my opinion.




top