Automated Tester

.science

This content shows Simple View

BDD

Specflow Reports with NUnit3

This post will look at setting up Specflow reports to work with NUnit3. When you install Specflow via NuGet, there is a tools folder which contains specflow.exe – if we run a batch process after the build on the CI server which points to the NUnit3 TestResult.xml, then we can generate a report and reference it as an artefact in TeamCity (and similar CI software).

But there is a catch, which isn’t (at the time of writing) explained in the Specflow reporting documentation. Currently Specflow (v 2.0.0) cannot interpret the NUnit3 TestResult output, so to counter this restriction we need to configure NUnit3 to output its TestResult.xml in the NUnit2 format. A full list of NUnit3 console options can be found here.

nunit3-console.exe –where cat==SmokeTests&&cat!=ExcludeFromCI
–out=TestResult.txt
–result=TestResult.xml;format=nunit2 C:\Path\To\Acceptance.Tests.dll

With this in place it’s now just a case of rigging up the Specflow command to point to the Test Result data and telling it to produce a report and specifying where to output the HTML report.

specflow.exe nunitexecutionreport Your.AcceptanceTest.csproj /testResult:TestResult.xml
 /out:MyResult.html

Once you have the TestResult.xml and the Result output generating where you want, it’s easy to wrap the command up into a batch file and call it from a build step after your tests have executed. Make this output an artefact and then you have metrics in place for your nightly and/ or other test builds.

Demo Specflow Report



Pickles Reports – Living Document Generation

Adding Pickles to your Project is really useful for presenting your Gherkin based tests in an easy to read, searchable format with some funky metrics added in. If you get your CI build to publish the output as an artifact then on each build you will always have up to date current documentation of what is being tested.

You can configure it numerous ways, via MSBuild, PowerShell, GUI or Command Line.

In this article I will be setting up via command line and running a batch file as a build step in TeamCity. First off we need to Install Pickles Command Line via NuGet. NUnit 3 is my test runner.

Once built we can start making use of it by running the executable with our desired parameters:

  • –feature-directory: Where your Feature Files live, relative to the executable
  • –output-directory: Where you want your Pickles report to generate
  • –documentation-format: Documentation format (DHTML, HTML, Word, Excel or JSON)
  • –link-results-file: Path to NUnit Results.xml (this will allows graphs and metrics)
  • –test-results-format: nunit, nunit3, xunit, xunit2, mstest, cucumberjson, specrun, vstest

Together it looks like the below, which is put into a batch file and called in a closing build step.

.\packages\Pickles.CommandLine.2.5.0\tools\pickles.exe –feature-directory=..\..\Automation\Tests\Project.Acceptance.Tests\Features^ –output-directory=Documentation^ –link-results-file=..\..\..\TestResult.xml –test-results-format=nunit –documentation-format=dhtml

I use the NUnit format over NUnit3 because I have set my NUnit3 runner to output NUnit2 formatted results, this is so Specflow can consume the same output and produce more metrics on the build. With the results file hooked in you can get some whizzy graphics:

pickles_report

Pickles is a great tool for BDD and should  help bridge that gap between “the Business” and IT. A full sample report can be found here. You can find extensive documentation here for the various ways to set up Pickles.

 



Recording Videos of your Selenium Tests

It is good practice to take a screen grab of test failures and review them later, particularly if you have some bizarre wait or timeout error that can be difficult to pin down but sometimes even these can lead you down the wrong path or not give enough context to what went wrong. This led me to explore being able to capture videos of running tests that can be scrutinised at a later time, much like you get when you run tests in SauceLabs or BrowserStack.

I created a helper class called RecordVideo and pulled down a NuGet package called Microsoft Expression Encoder. You will also need to install Microsoft Expression Encoder 4.

Having set up a start (and stop) recording method we can easily capture tests in and write them out to file with the help of our set up or specflow hooks.

Here’s how:

using System;
using Microsoft.Expression.Encoder.ScreenCapture;

public class RecordVideo
{
private static readonly ScreenCaptureJob vidRec = new ScreenCaptureJob();

public static void StartRecordingVideo(string scenarioTitle)
{
string timestamp = DateTime.Now.ToString("dd-MM-yyyy-hh-mm-ss");
vidRec.OutputScreenCaptureFileName = @"C:\Test Videos" + scenarioTitle + " " + timestamp + ".wmv";
vidRec.Start();
}

public static void EndRecording()
{
vidRec.Stop();
vidRec.Dispose();
}
}

We can now call these in our setup/ teardown (in my case I’m using Scenario Hooks) and use the test case name for the file title along with a timestamp.

[BeforeScenario]
public void BeforeScenario()
{
RecordVideo.StartRecordingVideo(ScenarioContext.Current.ScenarioInfo.Title);
objectContainer.RegisterInstanceAs(new PageContext(TestRunContext.Driver));
}
[AfterScenario]
public void AfterScenario()
{
if (ScenarioContext.Current.TestError != null)
{
TakeScreenShot.SaveScreenShot(TestRunContext.Driver, TestFailurePath + ScenarioContext.Current.ScenarioInfo.Title + ".jpeg");
}

RecordVideo.EndRecording();
}

Be sure to put something in place to clear the directory out of videos on each run or you might find yourself running out of disk space quite quickly.



API Testing with Specflow

I’m a big fan of Specflow/ Cucumber, it’s great for creating tests to run against a front end with webdriver but what if we want to keep that same plain English BDD approach and apply it against an API for automated API Testing? It’s certainly possible and this article will cover the basics to get you up and running.

I am going to assume you have a framework set up and in place, you can even use a webdriver based solution as described previously, although you may wish to run it with a headless driver.

I’m also going to add a SOAP endpoint to my solution and write a test in Gherkin, this endpoint is the publicly available Global Weather SOAP endpoint available here:

http://www.webservicex.com/globalweather.asmx?wsdl

This WSDL generates 2 service operations, GetCitiesByCountry (to get a list of all the cities for any country) and GetWeather (to get weather for any city/country combination).

Let’s add the WSDL somewhere we can use it in Visual Studio. If you right click your project and go to Add > New > Service Reference, you will be presented with a dialogue in which you can paste in the WSDL, click Go and give it an appropriate name.

Add Service Reference for API Testing

Click OK, you should see a new Service References folder in your solution. Lets begin formulating a test for our API Testing suite.

@API
Scenario Outline: Get Cities
Given I query the API
Then I can retrieve <City> from <Country>

Examples:
| City        | Country        |
| Reims       | France         |
| London      | United Kingdom |
| Agra        | India          |
| Porto Santo | Portugal       |
| Cancun      | Mexico         |
| Pretoria    | South Africa   |
| Melbourne   | Australia      |

We’re going to need a helper class which I will call ‘GeoHelper’ where we can reference the endpoint and then make some methods we can call from the step definition. Below I am passing a string value to the ‘GetCitiesByCountry’ extension on the endpoint and returning the result. Then I have created another helper method which takes that result and asserts the city is present in the response.

public class GeoHelper
{
     private readonly GlobalWeatherSoap globalWeather = new GlobalWeatherSoapClient();

     public string GetCitiesByCountry(string country)
     {
          var x = globalWeather.GetCitiesByCountry(country);
          return x;
     }

     public void CheckCityNameIsPresent(string countryName, string cityName)
     {
          Thread.Sleep(200);
          Assert.IsTrue(GetCitiesByCountry(countryName).Contains(cityName));
     }
}

So our step definition calls this like so:

private readonly GeoHelper geo = new GeoHelper();

[Then(@"I can retrieve (.*) from (.*)")]
public void ThenICanRetrieveFrom(string cityName, string countryName)
{
     geo.CheckCityNameIsPresent(countryName, cityName);
}

And that in brief is how we can leverage the use of Specflow for API Testing.



Interacting with Dropdown lists with Selenium in C#

This article briefly explains how to, or at lease how I interact with dropdown menus in C#. Generally speaking I try to pass through the identifier and the value to select to a helper method which carries out the task.

Lets have a look at a typical step definition, first we’re waiting for the element to appear on screen and then selecting the option by value.

[When(@"I select the organisation (.*)'")]
public void WhenISelectTheOrganisation(string orgName)
{
Page.WaitForElementByClass(ClassName.Organisation);
Page.SelectFromDropDownByClass(ClassName.Organisation, orgName);
}

So lets look at SelectFromDropDownByClass:

public void SelectFromDropDownByClass(string dropDownClass, string dropdownValue)
{
new SelectElement(Context.Driver.FindElement(By.ClassName(dropDownClass))).SelectByText(dropdownValue);
}

We’re creating a new instance of Webdrivers’ SelectElement and passing through the class name and text of the item in the dropdown, keeping the identifier variable ensures we can reuse the same method elsewhere in the solution, you can interact and select items in other ways from dropdowns and also use other identifiers as necessary such as Id, CssSeleector and XPath.




top