#visual validation

And you guessed right, Electron uses HTML, JavaScript and CSS to build desktop applications. The vision of same codebase for web, mobile and desktop applications has never been closer. When it comes to automation, it was always a chase after the recent platforms, frameworks and technologies, and here again, we are closer than ever to use same automation for all.

Technically speaking, you can automate Electron applications as if they were standard web applications using Selenium Chrome Driver.

So How to Start?

Let’s use the Electron api demos app which can be downloaded here.

First thing to notice, chrome developer tools are already there.

Use alt+cmd+i (or F12 on windows) to bring up the dev tools. Now you can locate elements in the HTML and build css selectors for your own selenium test.

What's Next?

Let’s set up a test in Junit using Java and Selenium. It will be required to set up the path to the actual binary of the application in ChromeOptions object. The rest is as in any other selenium test:

package com.applitools.Full.Electron; import org.junit.After; import org.junit.Before; import org.junit.Test; import org.junit.runner.RunWith; import org.junit.runners.JUnit4; import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import org.openqa.selenium.WebElement; import org.openqa.selenium.chrome.ChromeDriver; import org.openqa.selenium.chrome.ChromeOptions; import org.openqa.selenium.remote.DesiredCapabilities; import java.util.List; @RunWith(JUnit4.class) public class ElectronApiAppTest { private WebDriver driver = null; @Before public void setup() { ChromeOptions opt = new ChromeOptions(); opt.setBinary("/Users/yanir/Downloads/Electron API Demos.app/Contents/MacOS/Electron API Demos"); DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability("chromeOptions", opt); capabilities.setBrowserName("chrome"); driver = new ChromeDriver(capabilities); if (driver.findElements(By.id("button-about")).size() > 0) driver.findElement(By.id("button-about")).click(); } @Test public void test() throws InterruptedException { driver.findElement(By.id("get-started")).click(); List elements = driver.findElements(By.className("nav-button")); for (WebElement element : elements) { element.click(); } } @After public void teardown() { driver.quit(); } }

Now we are good to go. You will notice that, unlike regular web tests, the test is running much faster. That’s due to the fact that the entire application is hosted on the same machine so there is no network latency involved.

Take it to the Next Level: Visual Validation

If you never used visual validation - this is the time. Visual validations will add a layer of coverage to all the visible elements, so you will enjoy a significant reduction in assertions coding and maintenance. Let’s use Applitools to demonstrate how easily it can be achieved (some code ahead):

package com.applitools.Full.Electron; import com.applitools.eyes.selenium.Eyes; import org.junit.After; import org.junit.Before; import org.junit.Test; import org.junit.runner.RunWith; import org.junit.runners.JUnit4; import org.openqa.selenium.By; import org.openqa.selenium.WebDriver; import org.openqa.selenium.WebElement; import org.openqa.selenium.chrome.ChromeDriver; import org.openqa.selenium.chrome.ChromeOptions; import org.openqa.selenium.remote.DesiredCapabilities; import java.util.List; @RunWith(JUnit4.class) public class ElectronApiAppTest { private WebDriver driver = null; private Eyes eyes = new Eyes(); @Before public void setup() { ChromeOptions opt = new ChromeOptions(); opt.setBinary("/Users/yanir/Downloads/Electron API Demos.app/Contents/MacOS/Electron API Demos"); DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability("chromeOptions", opt); capabilities.setBrowserName("chrome"); eyes.setApiKey("MY_APPLITOOLS_API_KEY"); driver = new ChromeDriver(capabilities); if (driver.findElements(By.id("button-about")).size() > 0) driver.findElement(By.id("button-about")).click(); } @Test public void test() throws InterruptedException { driver = eyes.open(driver, "Electron API Demos app", "Electron app Smoke test"); eyes.checkWindow("Home page"); driver.findElement(By.id("get-started")).click(); List elements = driver.findElements(By.className("nav-button")); for (WebElement element : elements) { element.click(); eyes.checkRegion(By.cssSelector("main.content"), element.getAttribute("id"), true); } eyes.close(); } @After public void teardown() { driver.quit(); eyes.abortIfNotClosed(); } }

The result will appear in Applitools dashboard:

If something changes in the screens of your the application - it will be discovered by Applitools Eyes visual validation.

For further reading and reference on Applitools go to applitools.com website. To familiarize yourself with Selenium, head to http://www.seleniumhq.org, as well as expert blogs, such as Elemental Selenium by Dave Haeffner.

HAPPY TESTING

How to set up your first visual test

How to do visual testing with Selenium

How to integrate visual testing with Sauce Labs

How to ensure cross-browser and cross-device compatibility

How to analyze results - for faster actionable insights

Watch it now - you can find Greg's slide deck HERE:

You ever been in this position? You’ve been testing a web page, found multiple problems with it. Those problems get fixed, you test a bit more, and declare yourself done. You then sit down with the designer to review the implementation, and immediately: “Wrong! Wrong, that’s wrong”. “That’s the wrong font”. “Ahhh, that’s not the right blue”. “That margin is too small”. I’m sure you’ve heard similar things. So why didn’t we find these issues? Of course there are infinite reasons, but I want to explore a common one: Inattentional Blindness.

“Inattentional blindness, also known as perceptual blindness, is a psychological lack of attention that is not associated with any vision defects or deficits. It may be further defined as the event in which an individual fails to recognize an unexpected stimulus that is in plain sight” (Wikipedia)

For the most common example used when talking about inattentional blindness, watch this video and tell me how many time the team in white pass the basketball. Or watch the amazing colour changing card trick by Richard Wiseman.

Simply put: we don’t always see everything that is directly in front of us. We blind ourselves based on lots of factors, In my opinion, one of the most common in testing is being what we’re actually testing for, i.e. where's our focus. Some testers simply don’t know, they would just claim to be testing. Others would make claims to be testing functionality, other perhaps visually, or many of the others things to chose from. Either or, we will always struggle to spot all the issues.

Designers spend a large percentage of their time, well, designing. They’re emerged in the tool of their choice, analysing mock ups and doodling on white boards. Tweaking here, tweaking there. A few pixels to the right, a few pixels to the left. A darker colour here, a lighter colour there. They dream about designs, just like some testers dream about the application as a whole... (or is it just me then?). When they finally look at the finished product, they are armed with so much more tacit knowledge than us, so they immediately start targeting areas that perhaps they had more discussion around, or simply areas that were concerning them the most.

Now I’m not saying that we can’t find the same issues that designers find, of course we could, but we would need some of that tacit knowledge that the designers have. We could build up this knowledge for ourselves, by talking to the designers, asking them questions about the design, for example; What was tricky to design? Which areas have more changes than others? And then use this to structure some testing solely focused on the visuals, taking advantage of available tools.

However, my preferred choice is to work alongside the designers, to ensure we get the best of all the minds. Test as a pair. Time doesn’t always allow such activities to take place, so on a previous project I decided to take advantage of some tools.

We had some basic UI checks in place for the majority of screens on our application, and would tend to create checks for new screens as they were created. I decided to incorporate a screenshot into this process. The check would take a screenshot after several key steps, and save it out to a shared drive. We would then collate those screenshots and review them as a pair. This approach saved us a lot of time, as we didn’t have to recreate the scenarios in real time to do the visual review. This worked well, but we started to wonder if this could be improved.

Time was the shortcomings of this approach: finding a time where we could both be present was sometimes difficult, therefore we would fallback to a direct comparison to the previous version, then I would send any noticeable changes to the designer for her to review. Reducing her workload from all of them, to the ones I believe required her attention. This was far from ideal, as it falls back to the previous problem: it relied on me to spot things, something I’d gotten better at due to all the pairing, but I still didn’t have that designer's eye.

I started to think that there must be some tools out there that could automatically compare these images for me. I was very much aware of diff tools, so I looked for one with a specific focus on images, this lead me to ImageMagick. I integrated ImageMagick into the process, and I would run the comparison tool on the two images, which in turn would return me a mathematical percentage on the difference of the two images. This improved the process, as it would draw my attention to the differences, however, I started to get a lot of false negatives, as the comparison it produced was pixel-perfect. So when the tool reports a 0.5% difference, I still have to investigate, as that 0.5% could potentially have been a serious issue, for example: a problem with the company's logo. However, in the majority of the time, it was a different I couldn’t even spot, even though the tool was reporting a difference, and sometimes, issues that even the designer couldn’t even spot!

However, our process was vastly improved with the addition of tools. Simply adding the screenshots and diff tool into our process did speed our goal up. Sure it wasn’t perfect, but it overcome the skill gap between myself and the designer when it comes to spotting visual issues. It also reduced the time we had to spend on this activity, only getting together to look closer at some screens when the tool reported changes. Should point out that for new screens we still paired a lot in the first few iterations, as there was no baseline image for us to compare against.

Nowadays? Well as you could probably guess with me writing on this blog, I do visual checking on my products, and my current context is actually mobile. I still do the same process, however I take advantage of the many tools available now to help us tackle this problem. Those tools do the screenshotting, comparing and result reporting seamlessly for us. Beyond seamlessly though, in most cases it’s only a few lines of code to get up and running.

So, if are you experiencing a similar problem with your visual testing, and looking at implementing visual checking to keep on top of it, take a look at the tools in this post, give them a go. I can tell you that implementing automated visual checking tools has vastly improved our approach to testing, and more importantly: it’s dedicated to checking the visual layer, which is what my designers actually care about, amongst other things, of course.

More posts by Richard Bradshaw:

"Automated Form Validation Checking"

"Checking More Things, That's a Good Thing, Right? "

*************

Author: Richard Bradshaw. You can read more about Richard’s view on automation on his blog. He is also an active tweeter, where you can find him as @FriendlyTester.