Automated Tester

.science

This content shows Simple View

Applitools

Visual testing with BackstopJs

There are many visual testing tools out there these days. Previously I have covered Applitools which without a doubt is great product with awesome people who contribute a lot to the testing community. I have also looked at Galen which is a more of a DOM layout tester. In this article we will have a look at BackstopJs, which is the closest opensource project I’ve come across that matches what Applitools do with regard to the way visual regressions are presented to the user.

So, in a nutshell:

BackstopJS automates visual regression testing of your responsive web UI by comparing DOM screenshots over time.

Setup

The good thing about Applitools is I can just dump in a line of code to already existing tests and see it do it’s thing. We have to start from the ground up here but fortunately the project maintainers have done a fantastic job of making everything streamline, so let’s go.

I’m a windows user in my day to day so lets have a look at setting this up. Note, you will require some pre-reqs in order to get started:

BackstopJs has very good documentation, so I’ll try not to just repeat what that covers. Rather, give a quick flavour of what BackstopJs can do and how to overcome any issues I had during set up.

There was only one of those actually, as I am looking at Local installation I edited my PATH environment variable to include backstop.cmd, so just had to add the following parameter: C:\Users\username\AppData\Roaming\npm\

Workflow

It’s important to remember the workflow here, we need to call Backstop Init before we run our baseline tests.

  • Backstop Init (from your project directory)
  • Backstop test (get our base images)
  • Backstop approve (set the base images)
  • Backstop test – run the tests and compare against the baseline

BackstopJs Commands in action

To hit the ground running I recommend cloning this sample project and just tweaking the settings to meet your needs.

If we edit/ tweak ‘backstop.json’ we can easily dump new scenarios into the Json array once we are familiar with it, add or modify viewport sizes and exclude elements on the page that may be much more variable than others.

Output

In the output we get a cool slider in the visual diff inspector, a scrubber for checking out differences and other useful info in a nicely styled report, tests are also very fast to execute.

Visual ChangeBackstopJs Report

 

This was great for quickly spinning up tests against 25 odd branded log in screens generating 75 test cases covering three different resolutions. Yes, it’s not the same as emulating or against physical devices but nonetheless useful.

Digging a little deeper

What if we do want to log in though? The recommended ‘engine’ is some selenium-esque coding which is required in the OnReady.js file for the specified engine. You can use others but here we’re looking at the recommended option of Puppeteer.

If you’re familiar with Selenium then Puppeteer is a breeze and also has very good API documentation. This is for use with Chromium only (makes sense as it’s created by the Google dev tools team).

To perform a login for example, we add some Puppeteer code:

module.exports = async (page, scenario, vp) => {
console.log('SCENARIO > ' + scenario.label);
await require('./clickAndHoverHelper')(page, scenario);

// add more ready handlers here...
await page.focus('#Email')
await page.keyboard.type('email@someDomain.com');
await page.focus('#Password')
await page.keyboard.type('secretPassword');
await page.keyboard.press('Enter');
const navigationPromise = page.waitForNavigation();

await navigationPromise; // The navigationPromise resolves after navigation has finished

page
.waitForSelector('#homeSupport > h3:nth-child(8) > a')
.then(() => console.log('First URL with image: ' + currentURL));
for (currentURL of ['https://www.someUrl.com'])
await page.goto(currentURL);
await page.waitFor('#homeSupport > h3:nth-child(8) > a');
};

If you require an area to be excluded from the test such as some dynamic panel with variable data in it, we can hide DOM elements in the config (backstop.json).

Why not give BackstopJs a try? It could save you a lot of time.



Applitools Eyes: Easy Automated Visual Checking

Applitools Eyes allows us to easily integrate automated visual testing into our existing BDD Solution. Having automated visual testing is a big plus because whilst our automation may be good at asserting if elements are visible on the page or if some certain text is present, it doesn’t tell us that it has appeared in the right place.

Automated visual testing will capture baseline images to make future checks against.

To quote another article: “For example, a single automated visual test will look at a page and assert that every element on it has rendered correctly. Effectively checking hundreds of things and telling you if any of them are out of place. This will occur every time the test is run, and it can be scaled to each browser, operating system, and screen resolution you require.”

If you run this in parallel then you have a powerful testing beast on your hands.

Put another way, one automated visual test is worth hundreds of assertions. And if done in the service of an iterative development workflow, then you’re one giant leap closer to Continuous Delivery. — Dave Haeffner

As with everything on this blog it is tailored around C# .Net but the following works in a similar way with other bindings like Java, Python, Ruby and JS.

How it’s set up in the Solution for C# .Net

First off we need a Nuget package for Eyes:

Nuget_Eyes

Then in our TestRunContext Class we need to instantiate it as part of our window setup:

using Applitools;

private static Eyes eyes;
public static void WindowSetup()

{
    Driver.Navigate().GoToUrl("about:blank");
    Driver.Manage().Window.Maximize();

// This is your api key, make sure you use it in all your tests.
    eyes = new Eyes {ApiKey = "API KEY GOES HERE"};

// Start visual testing with browser.
// Make sure to use the returned driver from this point on.
   Driver = eyes.Open(Driver, "Application name", "Test Name", new System.Drawing.Size(1024, 768));
}

We call on the following in our tests to perform a check of the page, passing in a string to be displayed in Applitools:

public static void EyeCheck(string pageName)
        {
            // Visual validation point
            eyes.CheckWindow(pageName);
        }

And then close the Eyes at the end of our checks:

public static void CloseEyes()
        {
 // End visual testing. Validate visual correctness.
            eyes.Close();
            eyes.AbortIfNotClosed();
        }

Rather than storing the Selenium instance in the Driver variable, we’re now storing it in a local browser variable and passing it into eyes.Open — storing the WebDriver object that eyes.Openreturns in the Driver variable instead.

This way the Eyes platform will be able to capture what our test is doing when we ask it to capture a screenshot. The Selenium actions in our test will not need to be modified.

Before calling eyes.Open we provide the API key. When calling eyes.Open, we pass it the Selenium instance, the name of the app we’re testing (e.g., “Test Suite“), and the name of the test (e.g., “Test Name“) in the above example the part that says “new System.Drawing.Size(1024, 768)” is specifying the view port size of the browser.

For mobile testing it is not relevant, since the device size will be used, but for web testing, it is highly recommended to configure a size in order to ensure that the result of the test will be consistent even when you run it on different machines with different screen resolutions/sizes.  You can also paramterise it and run on several sizes in case of responsive design, but in general it is recommended to set it explicitly.

Now we’re ready to add some visual checks to our test by calling the following in our step definitions.

Page.PerformEyeCheck(pageName);

With eyes.CheckWindow() working in the background we are specifying when in the test’s workflow we’d like Applitools Eyes to capture a screenshot (along with some description text).

NOTE: These visual checks are effectively doing the same work as the pre-existing assertion (e.g. where we’re asking Selenium if a success notification is displayed and asserting on the Boolean result) — in addition to reviewing other visual aspects of the page. So once we verify that our test is working correctly we can remove this assertion and still be covered.

We end the test with eyes.Close. You may feel the urge to place this in teardown, but in addition to closing the session with Eyes, it acts like an assertion. If Eyes finds a failure in the app (or if a baseline image approval is required), then eyes.Close will throw an exception; failing the test. So it’s best suited to live in the test itself.

NOTE: An exception from eyes.Close will include a URL to the Applitools Eyes job in your test output. The job will include screenshots from each test step and enable you to play back the keystrokes and mouse movements from your Selenium tests. You can also determine the level of strictness to use or exclude areas of your page from being checked.

When an exception gets thrown by eyes.Close, the Eyes session will close. But if an exception occurs before eyes.Close can fire, the session will remain open. To handle that, we’ll need to add an additional command to our teardown.

eyes.AbortIfNotClosed(); will make sure the Eyes session terminates properly regardless of what happens in the test.

Now when we run the test, it will execute locally while also performing visual checks in Applitools Eyes.

Applitools Eyes

Further Example:

https://eyes.applitools.com/app/tutorial.html




top