Automated Tester

.science

This content shows Simple View

Reporting

Specflow Reports with NUnit3

This post will look at setting up Specflow reports to work with NUnit3. When you install Specflow via NuGet, there is a tools folder which contains specflow.exe – if we run a batch process after the build on the CI server which points to the NUnit3 TestResult.xml, then we can generate a report and reference it as an artefact in TeamCity (and similar CI software).

But there is a catch, which isn’t (at the time of writing) explained in the Specflow reporting documentation. Currently Specflow (v 2.0.0) cannot interpret the NUnit3 TestResult output, so to counter this restriction we need to configure NUnit3 to output its TestResult.xml in the NUnit2 format. A full list of NUnit3 console options can be found here.

nunit3-console.exe –where cat==SmokeTests&&cat!=ExcludeFromCI
–out=TestResult.txt
–result=TestResult.xml;format=nunit2 C:\Path\To\Acceptance.Tests.dll

With this in place it’s now just a case of rigging up the Specflow command to point to the Test Result data and telling it to produce a report and specifying where to output the HTML report.

specflow.exe nunitexecutionreport Your.AcceptanceTest.csproj /testResult:TestResult.xml
 /out:MyResult.html

Once you have the TestResult.xml and the Result output generating where you want, it’s easy to wrap the command up into a batch file and call it from a build step after your tests have executed. Make this output an artefact and then you have metrics in place for your nightly and/ or other test builds.

Demo Specflow Report



Pickles Reports – Living Document Generation

Adding Pickles to your Project is really useful for presenting your Gherkin based tests in an easy to read, searchable format with some funky metrics added in. If you get your CI build to publish the output as an artifact then on each build you will always have up to date current documentation of what is being tested.

You can configure it numerous ways, via MSBuild, PowerShell, GUI or Command Line.

In this article I will be setting up via command line and running a batch file as a build step in TeamCity. First off we need to Install Pickles Command Line via NuGet. NUnit 3 is my test runner.

Once built we can start making use of it by running the executable with our desired parameters:

  • –feature-directory: Where your Feature Files live, relative to the executable
  • –output-directory: Where you want your Pickles report to generate
  • –documentation-format: Documentation format (DHTML, HTML, Word, Excel or JSON)
  • –link-results-file: Path to NUnit Results.xml (this will allows graphs and metrics)
  • –test-results-format: nunit, nunit3, xunit, xunit2, mstest, cucumberjson, specrun, vstest

Together it looks like the below, which is put into a batch file and called in a closing build step.

.\packages\Pickles.CommandLine.2.5.0\tools\pickles.exe –feature-directory=..\..\Automation\Tests\Project.Acceptance.Tests\Features^ –output-directory=Documentation^ –link-results-file=..\..\..\TestResult.xml –test-results-format=nunit –documentation-format=dhtml

I use the NUnit format over NUnit3 because I have set my NUnit3 runner to output NUnit2 formatted results, this is so Specflow can consume the same output and produce more metrics on the build. With the results file hooked in you can get some whizzy graphics:

pickles_report

Pickles is a great tool for BDD and should  help bridge that gap between “the Business” and IT. A full sample report can be found here. You can find extensive documentation here for the various ways to set up Pickles.

 




top