Cover Image of the Story

Visual testing: obstacle that every tester had so far..

It’s a common obstacle that every tester has, to check the UI manually. The tester must have a keen eye, in order to point out the tiny alignment changes of the front-end web elements. And it’s a test case that misses the test automation scope as it is impossible to do with traditional test automation aspects. In order to address this problem, visual testing came up with Image Processing!

What is Visual Testing?

Visual testing is how you ensure that your app appears to the user as you intended! It is the automated process of detecting and reviewing visual UI changes.

In today’s world, in the world of HTML, web developer create pages that appear on a mix of browsers and operating systems. Because HTML and CSS are standards, frontend developers want to feel comfortable with a ‘write once, run anywhere’ approach to their software. Which also translates to “Let QA sort out the implementation issues.” QA is still stuck checking each possible output combination for visual bugs — definition by Applitools

Hope now you have an idea what visual testing is.. right? Then..

Why do Visual Testing?

Because visual issues happen more frequently than you might expect and it’s a test case that’s missing in your test automation scope. Let’s have a look on the following issue from a sponsored Instagram ad..

visual issue (mis-aligned web elements) in an instagram sponsored advertisement
visual issue (mis-aligned web elements) in an instagram sponsored advertisement

OMG..! This is in production! How did they miss it? They would probably have run test automation and all the elements and texts are there in the DOM, so the test has passed. But visually, can you let this pass?

Even if you run unit tests and integration tests each time you’re deploying, visual bugs can still make their way into production. Functional tests can’t detect changes caused by rendering subtleties or CSS changes–visual testing does. That’s why we must move on with Visual Testing.

Is visual testing for pages or components?

Visual testing can apply to both components and full pages. Both are useful in different situations, but rendering full pages ensures that the actual UI that your customers see is appearing correctly, encapsulating how components are rendered and interact with one another.

Visual Testing Tools

When you read about the visual testing tools you will come across these keywords, so let’s check what those means first.

Is a rendering of a web page or component–including each individual responsive width and browser rendering–counts as a snapshot.

Is a snapshot of a web page or component marked as the expected visual look & feel.

Snapshots are compared to baselines to identify relevant visual changes between the two. If a snapshot is different from the compared baseline, it has a visual diff.

By the time I’m writing this story, there are two famous tools for this subject area. Namely:

  1. Percy.io
  2. Applitools Eyes

Percy.io

Percy.io is one of the first commercial visual testing tools that emerged in this subject. It was founded in 2011. Percy highlights the visual changes in your UI that are introduced by each pull request. Percy is a tool in the In-Browser testing category of a tech stack.

  • Percy uses pixel by pixel comparison to validate snapshots with the baseline image
  • Offers a free version
  • Price: $1,449.00
  • Provides technical support through email
  • Provides a pretty detailed documentation
  • Provides integrations to Jenkins, GitHub, CircleCI
  • Supports Chrome & Firefox browser rendering

Applitools Eyes

Applitools Eyes delivers the next generation of test automation powered by AI assissted computer vision technology known as Visual AI. Applitools was founded in 2012.

  • Uses AI-powered computer vision algorithms to simulate the human eye and ignore small imperceptible differences to avoid potential false positives
  • Offers free version & free trials
  • Price: is determined on a case by case basis
  • Offers 24/7 live support, online support
  • Provides training through detailed documentation, webinars, Live online sessions and in-person sessions
  • Provides integrations to Jenkins, GitHub, LT Browser
  • Provides accessibility testing on colour contrast issues
  • Supports Chrome, Edge, Firefox, Safari, IE & mobile browsers of iOS and Android devices

Applitools Ultrafast Test Cloud represents the next generation of cross browser testing. With Ultrafast Grid, you run your functional & visual tests once locally and it instantly renders all screens across all combinations of browsers, devices, and viewports. This is all done with unprecedented security, stability, and speed, and with virtually no setup required.

Ultrafast Grid supports Chrome, Firefox, Safari, Edge, and IE. Advanced mobile web support is available via Safari on iOS for simulation on a variety of Apple devices. Android devices are emulated using Chrome emulation.

Users have the option to run on classic runner or the ultra-fast grid..

How single browser execution happens with ultra-fast grid

What happens when visual grid or the ultra-fast grid is used, Applitools SDKs upload DOM Snapshots — not screenshots — to Applitools Ultrafast Grid. Snapshots include all resources needed to render a page (HTML, etc.) and are much smaller than screenshots, so they are uploaded much faster. The snapshot is then sent to the containers in the cloud where the snapshot is rendered on a browser to execute the visual inspection.

Also you have the option to execute the tests on multiple browsers at the same time using paralellization. Applitools Ultrafast Grid runs screenshot jobs in parallel by farming them out to a grid of hundreds of browsers in the cloud that generate screenshots of web pages for all the different combinations of browser types,viewport sizes, simulated, and emulated devices you request. This lets you test responsive web pages and components on a range of page widths corresponding to phones, tablets, laptops, and external monitors.

How parallel execution happens in ultra-fast grid

Here is a demo on how this amazing inspection happens within miliseconds:

Ultra-fast grid uses Containers — not virtual machines. Running code on multiple virtual machines in the cloud takes a lot of time and is flakey due to virtualization overhead. Applitools Ultrafast Grid uses containers to render web pages on different browsers in a much faster and more reliable way.

Snapshot (not screenshots) caching: Applitools Ultrafast Grid does not always upload a snapshot for every page. If a page’s resources didn’t change, Ultrafast Grid doesn’t upload them again. Since most page resources don’t change from one test run to another, there’s less to transfer, and upload times are measured in milliseconds.

Comparison: Percy vs Applitools Eyes

When it comes to the comparison between the two tools, the focus goes to the following:

  • Comparison mechanism: Percy uses pixel by pixel comparison while Applitools uses Visual-AI for the comparison. Applitools is way ahead of lesser false-positives on this aspect due to AI algorithms.
  • Browser compatibility: Percy only supports Chrome & Firefox while Applitools supports a huge range of browsers and versions.
  • Setup & Configuration: When it is compared to Percy, Applitools is way more easier to setup and configure to any testing framework.
  • Customer support: Applitools offers a significant support than Percy to help customers learn and use the tool
  • Extra features: Applitools offers accessibility testing on colour contrast
  • Efficiency & Accuracy: Applitools presents an accuracy of 99.9995% and Visual AI is..
Empirical Evidence of stats when using Visual-AI
Empirical Evidence of stats when using Visual-AI
Detailed competitive analysis of the features of Percy and Applitools

So as of my opinion and what I have experienced, Applitools Eyes is way ahead, out of the two tools since it’s rich with many features and provides more efficiency and accuracy.

That’s it for the story, I hope you got a proper understanding on What Visual Testing is, and detailed comparison on the tools Percy & Applitools Eyes. Don’t forget to express your ideas in the comments section below & Follow my profile if you think it’s worth.

Let’s meet again on how to integrate both these with a Selenium & a Cypress test automation framework..

Associate Lead QE @ Nagarro