Automated Tester

.science

This content shows Simple View

Allure

Setting Up Allure with NUnit in TeamCity

This article is a follow up from my previous article in setting up Allure locally with NUnit. I made a brief mention of the Allure TeamCity plugin but encountered difficulties in getting it to work and didn’t have much time to look into getting around it.

Lets get stuck in with a brief recap/ some initial steps to take.

  • Install NUnit 2.6.4 and the compatible NUnit adapter into your allure bin/addins folder
  • Edit the config.xml in your bin/addins folder to write out the XML to a known location (I’m using C:\AllureXML)

Generated Allure XML

  • Run some tests in the GUI or NUnit Console and generate some XML, you should check XML is being generated before continuing
  • Generate a report with the Command Line Interface (this will verify your JAVA_HOME system variable is present and correct)

Allure TeamCity Setup and Configuration

  • Install the plugin and restart the build server
  • Copy the allure-commandline.zip from latest release to the <TeamCity Data Directory>/plugins/.tools – No server restart needed for this step.
  • Your build needs to be generating XML, we tested this earlier
  • Open your build config settings, create a new build step for the Allure Report
  • Specify a relative path to your XML folder (in this example it is ../../../AllureXML)
  • Use the given example for test reports or similar

Allure Report Build Step

  • Edit your build step running the tests – I have not been able to get it to work with the NUnit runner type (it just won’t generate any XML) however, we can run a script to spin up the NUnit Console and run it that way.
  • Choose PowerShell as the runner, x86 bitness
  • Script: Source code
  • The script:
&amp; 'C:\Program Files (x86)\NUnit 2.6.4\bin\nunit-console-x86.exe' /framework=net-4.0 C:\BuildAgent\work\a0569b5caa0eb74d\Automation\Tests\Example.Automated.Selenium.Tests\Example.Automated.Selenium.Tests\bin\Release\Example.Automated.Selenium.Tests.dll

 

Test run build step for Nunit and Allure

It is a good idea to test the script on the build machine locally so you can ensure it is correct, before putting it into your TeamCity build step.

Notice a couple of things:

  • We run the x86 bit console as specified in the powershell run mode in the build step
  • We have passed in  “/framework=net-4.0”, this is important and in line with current documentation.

This should now, when run, kick off the tests via the console, generate the necessary XML and publish the artifacts in TeamCity giving you a report, which is great for retrieving daily summaries of builds you run overnight which is a great health check on your continuous integration builds.

Further Help

Please keep these installation notes at the forefront of your mind if you encounter difficulties:

If you wish to add different parameters to your tests you can find a full list below or by running  “nunit-console-x86.exe /help”

NUNIT-CONSOLE [inputfiles] [options]

Runs a set of NUnit tests from the console.

You may specify one or more assemblies or a single project file of type .nunit.

Options:

/fixture=STR               Test fixture or namespace to be loaded (Deprecated) (Short format: /load=STR)
/run=STR                   Name of the test case(s), fixture(s) or namespace(s) to run
/runlist=STR               Name of a file containing a list of the tests to run, one per line
/config=STR                Project configuration (e.g.: Debug) to load
/result=STR                Name of XML result file (Default: TestResult.xml) (Short format: /xml=STR)
/xmlConsole                Display XML to the console (Deprecated)
/noresult                  Suppress XML result output (Short format: /noxml)
/output=STR                File to receive test output (Short format: /out=STR)
/err=STR                   File to receive test error output
/work=STR                  Work directory for output files
/labels                    Label each test in stdOut
/trace=X                   Set internal trace level: Off, Error, Warning, Info, Verbose
/include=STR               List of categories to include
/exclude=STR               List of categories to exclude
/framework=STR             Framework version to be used for tests
/process=X                 Process model for tests: Single, Separate, Multiple
/domain=X                  AppDomain Usage for tests: None, Single, Multiple
/apartment=X               Apartment for running tests: MTA (Default), STA
/noshadow                  Disable shadow copy when running in separate domain
/nothread                  Disable use of a separate thread for tests
/basepath=STR              Base path to be used when loading the assemblies
/privatebinpath=STR        Additional directories to be probed when loading assemblies, separated by semicolons
/timeout=X                 Set timeout for each test case in milliseconds
/wait                      Wait for input before closing console window
/nologo                    Do not display the logo
/nodots                    Do not display progress
/stoponerror               Stop after the first test failure or error
/cleanup                   Erase any leftover cache files and exit
/help                      Display help (Short format: /?)



Test Reports with Visual Studio, NUnit and Allure

Getting some decent test reporting can be painful when working with Visual Studio and particularly with MS Test as the framework as newer versions of Visual Studio don’t appear to produce a *.trx file anymore.

Using NUnit we can generate some fancy graphs and stats using the Allure framework with NUnit.

  • Firstly we need to download and install NUnit: www.NUnit.org
  • Secondly we need the Allure NUnit adaptor, follow instructions here: Allure Framework Adaptor for NUnit
  • The Allure Command Line Interface is needed, get it from here: Allure CLI
  • Java JDK needs to be installed and specified correctly in your JAVE_HOME variable in your environment

Java_Home

Allure produces an XML file which can use that to generate a HTML report, there is also an adaptor for MS Test but it’s an extra hoop to jump through as it converts a *.trx into a XML which you then have to generate.

The NUnit adaptor generates the XML file to a directory when your tests are run with NUnit.

To generate the report run cmd in your Allure.bat directory and run the command:

allure report generate –v 1.4.1 path/to/directory/with/xml/files

Note: It is important you DO NOT paste this command in – type it out the long way – You can paste in the path to the XML files though.

You should see the following message:

Successfully generated report to [//Path to generated report]

Go to the directory and open the HTML report in Firefox as other browsers will not work with some of the Ajax requirements locally.

This is your generated report! You might also want to publish it as an artefact in your Team City CI build, with the Allure Team City Plugin.

NOTE: If you are using Specflow with NUnit be sure to install nuget package NUnit.Runners. If this breaks your build, go download the binary version of NUnit (http://www.nunit.org/index.php?p=download) and copy over both nunit.core.dll – nunit.core.interfaces.dll  to be referenced in your solution. This will ensure NUnit sees your [AfterTestRun] hook and perform all necessary teardown tasks.

SummaryxUnitTimeline    GraphsNo Defects

 

Collating multiple sets of results into one report

If you have the need to merge multiple sets of results into a single report this can be done easily by copying the generated XML files out to another directory, running another set of tests and then copying the previous files back over.

When you generate the report from the directory with two sets of XML files, Allure will mash it all together into one report.

You could also try Running Multiple Assemblies with NUnit but this is not recommended in my opinion.




top