Automated Tester

.science

This content shows Simple View

C#

Visualising test data with ElasticSearch & Kibana

You probably already get some kind of metrics out of your tests, whether you get stats from your build, Specflow reports and other build artefacts. This article will show you how to send your data to ElasticSearch and visualise the stats with it’s sister product Kibana.

There are several alternative ways you can achieve the same thing here, you may prefer to use another visualisation tool such as Graphite or Grafana… or you may want to store your metrics in something like Influx, so lets keep an open mind… I chose the ELK stack for no particular reason!

So lets get started, you’re going to need to be running ElasticSearch on your machine or a VM.

I’ll leave it to you to install rather than detail it here, the documentation is detailed and there are several ways, choose what’s best for you.

When you have the services running, check you can access them. Assuming you kept the default ports:

  • http://192.168.xxx.xxx:9200/

elasticSearch

  • http://192.168.yyy.yyy:5601/

Kibana

ElasticSearch

So how do we add useful test data? Let’s have a think about what you want to capture, maybe things like… :

  • Test Name
  • Test Result (Pass/ Fail)
  • The Server it ran against
  • The elapsed time of the test
  • The Feature File the test belongs to
  • A name for the test run (I call this the test run context)

Some of this stuff is easier to get hold of than others, depending on whether or not you run tests in parallel but lets start with our hooks or setup/teardown.

As we want to know how long our test takes to run, let’s create a stop watch in the [BeforeScenario] hook:

var sw = new Stopwatch();
sw.Reset();
sw.Start();

Unsurprisingly, we should stop it after the test is complete.

sw.Stop();

What about the rest of the info? As I’m using NUnit3 to run tests and Specflow, it should be easy to get at the test goodies we want to visualise and include these in our [AfterScenario] Hook:

var testRunTime = sw.ElapsedMilliseconds;
// Convert the time so it can be easily consumed later
var convTime = Convert.ToInt32(testRunTime);
var testName = TestContext.CurrentContext.Test.Name;
var testResults = TestContext.CurrentContext.Result.Outcome.Status.ToString();
var featureFile = FeatureContext.FeatureInfo.Title;

I tend to keep info like the server name in config so we can just pull that out of there.

var appSettings = ConfigurationManager.AppSettings;
var serverName = appSettings.Get("ServerName");

Tip: If you are running concurrently and hitting circular dependencies or similar, you can inherit your hook class from Specflows ”Steps”.

We have now have our info, if you debug you can see the variables being populated, however we are not yet sending it to our instance of ElasticSearch, let’s do that now.

Have a look in your [BeforeTestRun] hook and lets make a connection to ElasticSearch which in essence is just HttpClient call:

public class TestRunHooks
{
public static HttpClient EsClient;

[BeforeTestRun]
public static void BeforeTestRun()
{
// Setup ElasticSearch client
EsClient = ESClient.Create();

...
}

...
}

Here we call a class ESClient to do the heavy lifting:

public class ESClient
{
public static HttpClient client;

public static HttpClient Create()
{
// Get the info for your ES instance from config (http://192.168.xxx.xxx:9200/)
var appSettings = ConfigurationManager.AppSettings;
var elasticUrl = appSettings.Get("elasticSearchUrl");
client = new HttpClient()
{
BaseAddress = new Uri(elasticUrl),
Timeout = TimeSpan.FromMilliseconds(500)
};
return client;
}

public static async Task<HttpResponseMessage> PostESData(HttpClient client, object test)
{
try
{
// Post the data as Json to an index of your choice in ElasticSearch!
return await client.PostAsync("/qa/apitests", new StringContent(JsonConvert.SerializeObject(test), Encoding.UTF8, "application/json"));
}
catch (Exception)
{
return null;
}
}

// The object to post in our async call to ElasticSearch
public class TestResult
{
public string name;
public int elapsedTime;
public string result;
public string testRunContext;
public string featureFile;
public string serverName;
public string date = DateTime.UtcNow.ToString("yyyy/MM/dd HH:mm:ss");
}

// Call this from your hook with the data we have gathered in our [AfterScenarioHook]
public static void PostToEs(string testName, string outcome, int convTime, string featureFile, string serverName)
{
// Test conext Id is just a unique name for the test run, generate it how you like!
var testRunContext = TestRunHooks.TestContextId;
var client = ESClient.client;

var testResult = new TestResult() { name = testName, result = outcome, elapsedTime = convTime, testRunContext = testRunContext, featureFile = featureFile, serverName = serverName};
var result = PostESData(client, testResult).Result;
}
}

Don’t forget to dispose of your client after the test run:

[AfterTestRun]
public static void AfterTestRun()
{
EsClient.Dispose();

...
}

All going well (No Firewall issues etc) your post call to ElasticSearch should return a 201 status code and ElasticSearch now has Data! Moving on to Kibana…

Kibana

If we click on ‘Discover’ in Kibana and we have the correct date/time range selected from the upper right hand corner, we should see the raw data sent to ElasticSearch. Alternatively, you can perform a Lucene query (don’t worry, you won’t need to know this syntax in depth to make good use of it!) such as…

result:”Passed”

This returns all tests that passed for the selected time period, based on the data we have pushed into ElasticSearch.

Now that we have data we can create visualisations based on it! Keeping with the query above let’s visualise our passing tests in varying forms:

gauge 
When you have enough visualisations you can drop them all onto a dashboard and share with your team.

Dashboard

Note: If you want to lock down Kibana functionality and give team members their own login, you will need to install the x-pack addon.

Finally, and as eluded to at the start – once you have your metrics in ElasticSearch or Influx or whatever… (there are a few out there) then you are not limited by what tool to visualise with. I’d like to build on what is outlined here to compare results of runs, trends, drilldown to failures etc although I am not there yet 🙂

Happy graphing!



Proxying UI Automation to OWASP ZAP

Quick disclaimer: I’m not a security expert, pen tester or ZAP expert but that doesn’t mean to say we should ignore security. A cheap way of adding a layer of security testing is to take your existing Selenium automation and proxy them through OWASP Zed Attack Proxy. So lets get started.

This is not a guide on how to use OWASP Zap and will not go into great configuration detail.

  • Download, install and start OWASP ZAP (Requires Java) either locally or on a VM
  • Install FoxyProxy which is a popular browser plugin. You can get it for Chrome or Firefox.
  • Install root level certificate

In ZAP go to Options > Local Proxies and set the Address and Port as desired. In this article I am running ZAP on a VM so I put the address of the VM in and set the port to 8999. Add an additional proxy of localhost:8999.

We can test this with FoxyProxy, setup a proxy to point to the VM on port 8999 and you should see the ZAP API front end interface.

Zap API Frontend

Next go to the API options in ZAP and add the VMs IP address to the list of addresses permitted to use the API. For the purposes of this article, I have also disabled the API key required to perform commands.

Now we can hit that API front end without the use of FoxyProxy.

So lets setup a connection to our ZAP instance in code and point tests via the proxy. Lets start by adding the OWASPZAPDotNetAPI nuget package.

We’ll connect to the ZAP service at the start of our test run in the hook:

public static ClientApi Zap;

[BeforeTestRun]
public static void BeforeTestRun()
{
// Note if you are using an API key, pass it in here instead of null
Zap = new ClientApi("192.168.xxx.xxx", 8999, null);

...
}

Now lets configure a proxy and pass it to the driver:

var options = new ChromeOptions(); var options = new ChromeOptions();

options.AddArgument("start-maximized");
options.AddArguments("disable-infobars");

// ZAP Proxy (Passive Scan) - ZAP should already be invoked.

var proxy = new Proxy();
options.Proxy = proxy;
proxy.Kind = ProxyKind.Manual;
proxy.IsAutoDetect = false;
proxy.HttpProxy = "192.168.xxx.xxx:8999";
proxy.SslProxy = "192.168.xxx.xxx:8999";

options.AddArgument("ignore-certificate-errors");
options.Proxy = proxy;
var timespan = TimeSpan.FromMinutes(3);
_driver = new RemoteWebDriver(new Uri(gridHub), options.ToCapabilities());

If we run a test, we should see traffic being generated in the ZAP History tab:

Zap Traffic

Great, we’re almost there. Now it’s time to generate a report – you can do this manually by using the API UI in core/other/html report (http://192.168.xxx.xxx:8999/UI/core/other/htmlreport/) but as we have a connection in code to the API lets do it there, after the test run:

public static void WriteZapHtmlReport(string path, byte[] bytes)
{
File.WriteAllBytes(path, bytes);
}

[AfterTestRun]
public static void AfterTestRun()
{
HookHelper.WriteZapHtmlReport(ReportDirectory + "_PassiveScanReport.html", Zap.core.htmlreport());
Zap.Dispose();

Here we take the byte array from the zap.core api and write it to a HTML file somewhere on disk, giving you a handy passive scan report.

Zap Report

Limitations

Be aware that any test that uses ZAP needs to have exclusive use of it- so to generalise, it’s better to keep all the browser based tests sequential. This includes tests that use ZAP as a proxy, not just the scanning tests. These words were robbed from here and are very true having spent a couple of hours see if the above would work with concurrency.



Appium Mobile Emulation

Getting started with Appium requires a few installs and some configuration, so let’s get to it.

This article assumes you already have Java installed and set to the PATH environment variables.

What you need:

Android Studio

Once installed you need to look under Tools > Android > SDK. There are several components we need. Namely SDK Tools, SDK Build Tools and SDK Platform Tools.

Android SDK Install

Once setup you will need to add ANDROID_HOME to your PATH environment variable and point it to your main SDK folder. Note: A reboot may be required for Windows to pick up the change.

Android HOME

Visual Studio Emulator for Android

Now let’s look at Visual Studio Emulator for Android, once installed it should hopefully list a bunch of android API device profiles you can either run (if HyperV is enabled). Start one up and you should be able to interact with the virtual device. Note: If you look in Hyper-V Manager you can see the running emulator and tweak memory usage etc.

Device Profiles

HyperV Manager

If we navigate to our sdk tools we can also see the device running by running 

adb devices -L

 and this will list the device name for our Appium setup. In this instance the device is donatello.

Tip: If you happen to want to install an App you can do so with the command: adb install <path\to\yourapk.apk>

Appium Desktop

Next download Appium and click the Android icon to configure the server, input the relevant fields and device name then click the start button to run the server.

Appium Config

Installing an SSL Certificate on the device if required

As we are interacting with a website rather than an App for the purposes of this article, it can be useful to store SSL Certs on the devices when connecting. Using Visual Studio Emulator for Android we can setup and SD Card.

Emulator Options > SD Card

Pull from SD card first to a temp folder in order to create the directory structure first, then place the SSL Certificate in an easy to remember place and push to the SD card. Under Settings > Privacy in the Android OS we can then add a trusted certificate to the device.

When doing this, Android will force you to create a passcode for the device, do so and keep it easy to remember.

How do I find Android UI Elements for my tests?

Back over in Android SDK land…. in Sdk/tools/bin there is a batch file called uiautomatorviewer.bat. Open this in a text editor and find the line:

call "%java_exe%" "-Djava.ext.dirs=%javaextdirs%" "-Dcom.android.uiautomator.bindir=%prog_dir%" -jar %jarpath% %*

Then replace it with:

call "%java_exe%" "-Djava.ext.dirs=%javaextdirs%" "-Dcom.android.uiautomator.bindir=C:\DEV\androidSDK\tools" -jar %jarpath% %*

Making sure to set the binding directory to where your SDK tools are installed.

Then run uiautomatorviewer.bat whilst your emulator is running. Clicking the Android screenshot icon (second one from the left) will paint a view of what is currently on the emulator screen, you can then hover over buttons etc in order to find locators and anything else you might need.

uiautomatorviewer

Please note, your emulator API must be level 17 and upwards.

Time to Code

We can now write automation to run via this emulator with the usual webdriver goodies. Install the Appium Webdriver NuGet package and create the driver with the desired capabilities.

DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.SetCapability("device", "Android");
capabilities.SetCapability(CapabilityType.Platform, "Windows");
capabilities.SetCapability("deviceName", "donatello");
capabilities.SetCapability("platformName", "Android");
capabilities.SetCapability("platformVersion", "6.0.0");
capabilities.SetCapability(MobileCapabilityType.BrowserName, "BROWSER");
capabilities.SetCapability(CapabilityType.AcceptSslCertificates, true);

driver = new AndroidDriver<AndroidElement>(new Uri("http://127.0.0.1:4723/wd/hub"), capabilities, TimeSpan.FromSeconds(180));

Note that browserName “BROWSER” will start the devices’ native Android browser.

We can now start the automation of the site, starting with some very crude Selenium here… :

driver.Navigate().GoToUrl("https://mySecureUrl.com");

Thread.Sleep(1000);
var element = driver.FindElement(By.Id("Email"));

element.Click();
element.Clear();
element.SendKeys("username@mySecureUrl.com");

var element2 = driver.FindElement(By.Id("Password"));
element2.Click();
element2.Clear();
element2.SendKeys("mySecurePassword");
element2.SendKeys(Keys.Enter);

Hopefully this is a an easy enough to follow guide to get you up and running using Appium in your c# dev environment, these are the steps I took to being playing with Appium in more depth. Happy testing!



Use Selenium to Automate Windows with WinAppDriver

If you have ever found yourself in the position that you needed to automate a Windows application as part of your day to day automation testing you might well have come across AutoIt or gone down the CodedUI path. Now however, those good folk at Microsoft have made WinAppDriver which makes use of all the usual Selenium goodies… from the horses mouth:

Windows Application Driver is a service to support UI Test Automation of Windows Applications. The service design subscribes to the Mobile JSON Wire Protocol standard. If you’ve been looking for better support for using Appium to test Windows Applications then this service is for you!

The GitHub page has some good code samples to get you started (so I won’t bore you with any of mine), such as manipulating the Windows calculator. As you will see though the driver is still in beta but has come a long way recently and to help my foray into using it it now supports WCF applications.

Finding locators is pretty easy with the tool they suggest (Inspect) which is not too dissimilar to Spy++ if you have ever used that. Further to that I have found that the accessibility id locator works best for me (so far anyway!).

After installing the WinAppDriver executable, you can also run remotely which looks good but it does seem to only like being run on Windows 10 so you may need to take that into account if you don’t have the resources already in place.

If you decide to give it a whirl and encounter problems I would have a look at the issues page – chances are someone else has encountered it and a fix may already be underway.

Now obviously with WinAppDriver, this is only ever going to work with windows so I wouldn’t ever go recommending that this is used in the same project as your usual Selenium web based tests.

Until next time, have fun experimenting with WinAppDriver.



Testing Rest Endpoints with RestSharp and Specflow

In an earlier post I set how to use Specflow to test SOAP endpoints but what about REST API’s?

Given the nature of building URLs to interact with them it should be a cinch!… And it was. By leveraging use of the RestSharp library I was easily able to manipulate an example Rest interface over at http://www.thomas-bayer.com/sqlrest/.

So lets have a look at how:

Each method will be calling on the endpoint we initially set (the base URL), so lets wire in some tests with Gherkin and define some step definitions.

Scenario Outline: RestSharp Get Customer Record
Given I have an example endpoint http://www.thomas-bayer.com/sqlrest/
When I search for customer record 0
Then the result contains customer <ID>
Then the result contains customer <First Name>
Then the result contains customer <Last Name>
Then the result contains customer <Street>
Then the result contains customer <City>

Examples:
| ID | First Name | Last Name | Street | City |
| 0 | Laura | Steel | 429 Seventh Av. | Dallas |

Scenario: Post Product Price
Given I have an example endpoint http://www.thomas-bayer.com/sqlrest/
And I update the price of Product 1 to 10.00
When I search for Product 1
Then the price is 10.00
And I reset the price of product 1 to 24.8

We are passing the endpoint down to our helper here but I would normally put it in the App.config to keep things configurable.

Next up I created a Helper called RestHelper which will deal with all the setup, GET and POST requests etc. We need to define the endpoint and assign it to a commonly accessible place.

public class RestHelper
{
public RestClient endpoint = null;

public RestClient SetEndpoint(string endpointUrl)
{
endpoint = new RestClient(endpointUrl);
return endpoint;
}

Note: Your endpoints will most likely require some authentication of some kind which I am not covering. However I was easily able to add cookie authentication with a little tweaking.

Next lets create some other useful helper methods like GET and POST:

public string GetQuery(string query)
{
var request = new RestRequest(query, Method.GET);
IRestResponse response = endpoint.Execute(request);
var content = response.Content; // raw content as string
return content;
}

My POST method is simply to update the price of a product but you could rework to whatever you need:

public void UpdatePrice(string query, string price)
{
var request = new RestRequest(query, Method.POST) { RequestFormat = DataFormat.Xml };
var body = ("<resource><PRICE>" + price + "</PRICE></resource>");
request.AddParameter("text/xml", body, ParameterType.RequestBody);
endpoint.Execute(request);
}

In our Step definitions we just need to new up the Helper and pass in the right query, we are also assigning the Rest query result in a place that can be accessed by multiple step definitions:

private readonly RestHelper Rest = new RestHelper();

private string queryResult = null;

[Given(@"I have an example endpoint (.*)")]
public void GivenIHaveAnExampleEndpoint(string restEndpoint)
{
Rest.SetEndpoint(restEndpoint);
}

[When(@"I search for customer record (.*)")]
public void WhenISearchForCustomerRecord(string customerNo)
{
queryResult = Rest.GetQuery("CUSTOMER/" + customerNo + "/");
}

[Given(@"I update the price of Product (.*) to (.*)")]
[Then(@"I reset the price of product (.*) to (.*)")]
public void GivenIUpdateThePriceOfProductTo(int productNo, string newPrice)
{
Rest.UpdatePrice("PRODUCT/" + productNo +"/", newPrice);
}

And so on… If you want to have a play around with the code you can clone it from here and have fun testing REST APIs!




top