Automated Tester

.science

This content shows Simple View

March 2015

Data Driven Specflow Testing

What is it?
Specflow has a plugin called Specflow + Excel to allow you to hoard data in a spreadsheet and run tests using the data contained within.

Example

  • Download Specflow + Excel from Nuget into your solution.
  • Write a normal looking Scenario Outline such as below and notice that there are no items in the example table and also notice the @Tag above the examples table; this is referencing the file we want to use that houses all of our data:
Feature: GoogleTest
In order to have data driven tests in specflow
As a tester
I want to be able to plug all of my data into a spreadsheet

@mytag
Scenario Outline: GoogleTest
Given I have navigated to Google
And I have entered  into the search field
When I press enter
Then the result containing  are shown on screen

@source:GoogleTest.xlsx
Examples:
| SearchTerm | Result |
  • Write out your Step Definitions as usual, no special changes or requirements are needed
  • Create a new Excel Spreadsheet called ‘GoogleTest.xlsx’ and include this in your solution
  • Enter your corresponding parameter values
SearchTerm Result
fish Official Fish site
foo bar foobar2000
specflow SpecFlow – Cucumber for .NET

These will now appear as tests in your test runner!

Specflow Excel



Gherkin DSL Overview

Gherkin is a Domain Specific Language for bridging the communication gap between “The Business” and “IT”.

Feature for the software are written in English (or any other language for that matter) and then translated into actionable tests using a simple Given, When, But and Then. It has a simple format to follow, let’s have a look with an example.

Feature: Login
 In order to make use of my account
 As a user
 I want to be able to log into my account

Gherkin feature files always start with Feature: “Title of your feature” followed by a few lines detailing what the feature is and what it is doing. This way we know what we are testing.

Background:
       Given I have navigated to 'WebsiteName'
       When I click the link 'Login'

Background is not mandatory but is useful if you have some common steps, this gets read or executed before the main guts of the test.

Scenario: Login link goes to login page
       Then I am taken to the Login Page

The main guts of the test: Scenario defines what exactly is being tested followed by more Given, When, But & Then steps to ultimately come to an assert-able outcome.

We can also substitute English words for parameters in which we can pass in many different words. This is called a Scenario Outline and the parameters are defined with Examples in a table format.

Scenario Outline: Invalid Login Credentials
       And I am taken to the Login Page
       When I input my username as 
       And I input my password as 
       Then I should see the error 

       Examples:
       | Username             | Password | Error                                   |                                             
       | test.one@website.com | abcdefg  | Password or email address is incorrect. |


Test Reports with Visual Studio, NUnit and Allure

Getting some decent test reporting can be painful when working with Visual Studio and particularly with MS Test as the framework as newer versions of Visual Studio don’t appear to produce a *.trx file anymore.

Using NUnit we can generate some fancy graphs and stats using the Allure framework with NUnit.

  • Firstly we need to download and install NUnit: www.NUnit.org
  • Secondly we need the Allure NUnit adaptor, follow instructions here: Allure Framework Adaptor for NUnit
  • The Allure Command Line Interface is needed, get it from here: Allure CLI
  • Java JDK needs to be installed and specified correctly in your JAVE_HOME variable in your environment

Java_Home

Allure produces an XML file which can use that to generate a HTML report, there is also an adaptor for MS Test but it’s an extra hoop to jump through as it converts a *.trx into a XML which you then have to generate.

The NUnit adaptor generates the XML file to a directory when your tests are run with NUnit.

To generate the report run cmd in your Allure.bat directory and run the command:

allure report generate –v 1.4.1 path/to/directory/with/xml/files

Note: It is important you DO NOT paste this command in – type it out the long way – You can paste in the path to the XML files though.

You should see the following message:

Successfully generated report to [//Path to generated report]

Go to the directory and open the HTML report in Firefox as other browsers will not work with some of the Ajax requirements locally.

This is your generated report! You might also want to publish it as an artefact in your Team City CI build, with the Allure Team City Plugin.

NOTE: If you are using Specflow with NUnit be sure to install nuget package NUnit.Runners. If this breaks your build, go download the binary version of NUnit (http://www.nunit.org/index.php?p=download) and copy over both nunit.core.dll – nunit.core.interfaces.dll  to be referenced in your solution. This will ensure NUnit sees your [AfterTestRun] hook and perform all necessary teardown tasks.

SummaryxUnitTimeline    GraphsNo Defects

 

Collating multiple sets of results into one report

If you have the need to merge multiple sets of results into a single report this can be done easily by copying the generated XML files out to another directory, running another set of tests and then copying the previous files back over.

When you generate the report from the directory with two sets of XML files, Allure will mash it all together into one report.

You could also try Running Multiple Assemblies with NUnit but this is not recommended in my opinion.



Structuring Projects with BDD, Specflow and Selenium in C#

Typically when I create a Specflow BDD solution, BDD elements are separated into Feature Files, Hooks, Step Definitions and Pages. These elements are separate folders or subsets of folders within the solution.
Code begins with the feature files written in Gherkin, feature files contain Scenarios which themselves execute test cases. Sometimes a scenario is just one test case or can be many iterations of a particular action. An example is listed below:

Scenario Outline: Sitemap Links
 And I click the sitemap link 'LinkName'
 Then I am taken to the sitemap page for 'LinkName'
Examples:
 | LinkName              |
 | Contact Us            |
 | Terms and Conditions  |
 | Privacy Policy        |

Generally we perform a set of actions and then assert the outcome at the end, in the scenario above we are searching for links in the sitemap section of the footer, selecting a result from a list and then asserting that we are taken to that particular page.

It helps greatly if these features are written in collaboration. In an ideal world features should be authored in a “three amigos” style with the project Business Analyst and Product Owner.

Gherkin generates Step Definitions which can be used across a set of feature files. It makes sense to group common ones together in one big Step Definition file for actions such as navigating to the site or logging in/out of the application. More feature specific steps are kept in separate files.

It is our aim to keep the code in step definitions short and clean, we do this by wrapping up selenium code and putting these methods in to the Pages .cs files. For example:

Given I have navigated to Test Environment

Generates the following Step Definition:

[Given(@"I have navigated to (.*)"]
public void GivenIHaveNavigatedTo(string website)
{

}

From here in one of the common Pages .cs files we have created the following method:

public void NavigateTo(string url)
{
this.context.Driver.Navigate().GoToUrl(url);
}

We try to keep the Selenium code wrapped up in these statements to keep the Step definitions easy to write. With this method in place our step definition looks like this:

[Given(@"I have navigated to (.*)"]
public void GivenIHaveNavigatedTo(string website)
{
Page.NavigateTo(URLS.TestEnvironment1)
}

Organising Tests
Tests are organised with @Tags which make groups of or individual tests easy to find in MS Test. They also have other important uses in hooks and can be used to execute a set of tests on a CI build.

For example all Sanity tests are labelled with @_Sanity this makes the whole suite of sanity tests easy to find and execute. If this tag was passed to a CI build then rather handily just the sanity tests would run on an overnight build rather than a lengthier @_Regression suite.

What’s with the underscore? Well that just keeps your tags at the top of the MS Test list if you happen to be using that test framework, depending on your preference of viewing them.

Hooks

Hooks are used primarily for the test run set up and teardown. Before a run we want to instantiate a driver and spin a browser up before any scenarios are automated, otherwise it would go nowhere fast. Similarly at the end of a test run we want to kill the browser off.

You can be a bit more clever with hooks and execute bits of code before or after certain features, steps, scenarios or scenario blocks. In order to keep an easy to understand codebase these must be used sparingly. This is because debugging can be harder and feature files can have all sorts of things happening that aren’t specified in English and that’s not what BDD is intended for.

For more information on hooks please visit: https://github.com/techtalk/SpecFlow/wiki/Hooks

Pages

This section of classes is where we keep all of the wrapped up Selenium code, which is a bit nasty. We wrap them up into helper methods to make our step definitions easier to write as described previously. This code should be kept clean and refactored and should encapsulate common practices and apply sensible programming practices such as DRY (Don’t Repeat Yourself).

The Selectors Page is where we keep a large number of constants, this is so that if an Id changes on the System Under Test you will only ever need to change it once to fix your broken test(s).

Rules

We also keep in mind a set of rules  when structuring the code which are:

  • No Selenium Code in Step Definitions
  • No Project Specific Methods in PageBase, use MainPage for these
  • All By.Criteria to be defined and referenced in selectors page
  • URLs and base URLs that aren’t in Gherkin must be constants
  • Create triptych Given/When/Then step definitions where necessary
  • Always copy generated methods to clipboard
  • No Thread.Sleeps in any step definitions, wait for something instead
  • Regions, regions, regions – use regions in code to keep it nice and tidy
  • “That” is a banned word in Gherkin
  • Replace all auto generated variable names with something sensible
  • All assertions have the optional message parameter populated except for AreEqual
  • Split Step Definitions out into sensible groupings/ filenames

Diagram

BDD Solution Architecture
BDD Solution Architecture

Base Solution

You can find a base solution here.

Wrap Up

There is a lot more you can do to improve this solution like putting in a clever environment selector, addition of Sauce Labs or BrowserStack and also Applitools Eyes. The way this is build is designed to give you scalability and organisation to cope as your test suite grows but you have to be strict with yourself and keep in mind the BDD Rules set out or things can get messy very quickly.




top