Quality Engineering for Web UI

Quality is a shared responsibility within the team. Historically, unit tests have always been responsibility of developers while all other types of tests, along with CI/CD integration, were responsibility of QA engineers. Over time, software testing has evolved becoming more sophisticated, but also more understood and structured. New terms have permanently entered our vocabulary - integration testing, regression, end-to-end (e2e) tests, smoke, load, stress, usability tests, etc. For UI development, however, unit tests are not sufficient. It is just impossible to cover all possible user interactions with your app by unit tests. What then should UI developers do to insure quality of the code and, more importantly, what tools are now available to help with this task? This is the question I want to explore in this article.

How it All Started

Nine months ago, my manager came up to me and said: "I have some news... We do not have QA anymore... From now on, you are completely responsible for the quality of your own code." At that very moment time seemed to have stopped and I vividly imagined a "doom's day" coinciding with our next release date. It was inevitable... and there was no time to prepare for it. I thought of all doom's day movies were machines were taking over the world, the ruins and chaos that followed, the savage-looking people running around trying to find rest, avoiding some mean, crazy AI that wanted to dominate their world and control their minds. But I made my best effort to snap back to reality. Then my manager said a few other things that got me thinking. He said that it is a trend in the industry to move from QA towards QE (Quality Engineering) and developers all over the spectrum realize that in order to write better software you have to create it with Test Automation in mind. Having intimate knowledge and power to change the code gives an incredible advantages in writing tests over anyone who comes from outside. I can write/change code in a way that makes it more testable, and for UI it goes far beyond adding ids to key elements. So, developers should write tests for their code and these tests should become part of team's Test Automation suite. After leaving office that day, one phrase was echoing in my mind "Quality Engineering." My manager suggested to start with TDD to see if this can be used as part of UI development cycle to improve quality.

TDD for UI Testing

TDD starts with a great idea: write a test first then write code to satisfy it, repeat. It is simple and attractive. There is a host of use cases where it is an excellent approach, but in case of UI it is hard to implement effectively. I, too, struggled to apply TDD in my UI development. So, I am suppose to write a test that checks if a div is present on the page, then create code to insert that div to satisfy the test. Then, write a test to see if this div has a certain class, and then write code to add that class... etc. What exactly am I testing? Am I testing browser's ability to insert divs and assign classes without breaking? Does it need a test? It is something that my future code can break? It does not make much sense.... and it seemed too granular too. If I follow TDD I will be spending 90% of my coding time writing tests that test how browser inserts divs and will have no time to work on the actual application. I have also discovered that there are different schools of TDD thought - classical Detroit (aka Chicago school), mockist London, as well as various flavors of TDD such as BDD and ATDD/STDD. Each one of them solves a particular pain point within the TDD approach, but all of them require developer to write the test first and then the code to satisfy it. It is not my intention in this article to argue TDD's pros and cons, as there is a host of articles about it on the web already. I shall leave you with a short quote from wikipedia.org:

Monkeys for UI Testing

So, if I cannot effectively use TDD, then where do I start? It is a good question, but I think it is an easy one. Since UI stands for User Interface and it is meant for humans, I should write tests as if it was a human who interacts with my app. How does a human interact with it? Well, with a keyboard and a mouse of course! (for the sake of the argument, I will consider touchscreen interactions same as mouse interactions and voice commands as text typed in a text box). Who else can effectively use a keyboard and a mouse - a monkey of course. And they have a huge advantage over humans in testing - they do not mind doing repetitive and boring tasks once you teach them. So, my job as an engineer is to teach a monkey (write a script) to click and type where I want, and to make sure the results are as expected. (disclaimer: no monkey or any other animal were hurt in preparation of this article) Now, I am getting somewhere. I have decided that I will write tests that interact with my application as a human/monkey. Therefore, I need a tool that should be able to do 3 things: Inevitably, I have limited myself to what kind of tests I can write. I will be writing tests that are at the same time integration, functional and e2e, which for simplicity sake I will call e2e tests. These tests are integration tests because users interact with application that is fully integrated. They are functional because I want to cover only one single functional unit per test. And they are end-to-end tests because from the user's standpoint, the action is only meaningful when it produces desired outcome. I click "Add New" button, fill out the form, click "Save", form gets saved, and it returns me to the list where I see my new record. I realize that these tests are not purely UI tests, as they test the entire flow. Though, it is possible to mock API responses, I decided not to do that as I see more problems with this approach in a long run. In an active project APIs change frequently and I have no desire to keep my mocks updated each time back-end mutates.

Available Tools

JavaSciprt testing tools are plentiful (see An Overview of JavaScript Testing in 2020). Some libraries concentrate on a single aspect. For example, Mocha is a test runner, Chai is an assertion framework, etc., but I would like to have all-in-one solution that allows me to write e2e tests in JavaScript. My major choices are: Since I am web UI engineer and my primary language is JavaScript, I will only consider tools that are inherently JavaScript and, luckily, there is a host of them. I will also evaluate tools by the following criteria:

1. Expressive API
2. Rich API
3. Quick Dev Cycle
4. Extensible
5. Easy to Debug/Set Breakpoints
6. Pause/Resume Step Through Tests
7. Integrated into Dev Cycle
8. Good Docs/Community
9. Run in Browser
10. Run Heedlessly
11. Cross Platform/Browser
12. Capable of CI/CD Integration
13. Capable of Parallel Execution

I decided not to use Selenium because in its core it is a Java tool. Its architecture is heavily influenced by Java mentality that is often alien for JavaScript developers. I decided to eliminate Puppeteer and Playwright because these great projects are general purpose tools. They are not specifically designed for test automation. I decided against very popular and glorified Jest because it is best suited for unit tests not for e2e tests. Besides, Jest works in a simulated DOM, which always raised my eye brow and lowered my excitement about it. For similar reason I eliminated Jasmine as it is a BDD testing framework, and a very popular one, but I need framework for e2e tests. Now, I am left with these choices, Cypress, Nightwatch, Nightmare, Taiko, TestCafe and WebDriver.io. To break the tie, I will write same test in all of them to see which tool better suites my criteria. In general, I think all of them are quite similar in what they do.

Picking the Winner

I am going to write a simple test that fills out a short form, clicks the submit button, waits for the form to be saved. Then, I will evaluate all the contestants by the criteria I laid out above.

Cypress

it('Fill out the form, click submit', () => { cy.visit('http://localhost/test.html') .get('#userName') .type('Vitali') .get('#comment') .type('This text is types one character at a time') .get('#submit') .click() .route2('/test.html').as('save') .wait('@save') }) Dislikes:

• Makes your application run in an iframe

Nightwatch

it('Fill out the form, click submit', (browser) => { browser .url('http://localhost/test.html') .setValue('#userName', 'Vitali') .setValue('#comment', 'This text is not typed, but set to the element.') .click('#submit') .pause(2000) .end() }) Dislikes:

• There is a timeout that cannot be avoided
• Could not wait for a resource to load
• Unclear/confusing documentation
• Lacking in features and community support
• Emulation of events is difficult or impossible
• Hard to debug

Nightmare

const Nightmare = require('nightmare') const nightmare = Nightmare({ show: true, typeInterval: 10 }) nightmare .goto('http://localhost/test.html') .type('#userName', 'Vitali') .type('#comment', 'This text is typed one character at a time.') .click('#submit') .wait(2000) .end() .then(result => { console.log('All Good') }) .catch(error => { console.error('Search failed:', error) }) Dislikes:

• There is a timeout that cannot be avoided
• Could not wait for a resource to load
• Confusing documentation
• Lacking in features and community support
• Emulation of events is difficult or impossible
• Hard to debug
• Limited browser support

Taiko

const { openBrowser, goto, write, click, intercent, into, closeBrowser } = require('taiko') (async () => { await openBrowser() await goto("http://localhost/test.html") await write('Vitali', into($("#userName"))) await write('This text is not typed, but set.', into($("#comment"))) await intercept('test.html', {}) await click(link('#submit')) await waitFor(5000) await closeBrowser() })() Dislikes:

• Syntax is not expressive
• Syntax is repetitive
• Lacking in features and community support
• There is a timeout that cannot be avoided
• Could not wait for a resource to load
• Emulation of events is difficult or impossible
• Hard to debug
• Only chromium browsers (DevTools Protocol)

TestCafe

import { Selector, RequestLogger } from 'testcafe'; let url = 'http://localhost/test.html' const pageWait = RequestLogger(/test\.html/) fixture .disablePageCaching('Getting Started') .page(url) test.requestHooks(pageWait) ('Fill out the form, click submit', async t => { await t .typeText('#userName', 'Vitali') .typeText('#comment', 'This text is typed.') .click('#submit') .expect(pageWait.count(() => true)).eql(2) }) Dislikes:

• Syntax is not expressive
• Cumbersome architecture
• Syntax is rigid
• Injects scripts into your app, pollutes global scope
• Slow dev cycle

WebDriver.io

it('Fill out the form, click submit', () => { browser.url('http://localhost/test.html') browser.$('#userName').setValue('Vitali') browser.$('#comment').setValue('This text is not typed, but set to the element.') browser.setupInterceptor() browser.expectRequest('GET', 'test.html', 200) browser.$('#submit').click() browser.pause(3000) // BAD JOJO, I know, but is there another way browser.assertRequests() }) Dislikes:

• There is a timeout that cannot be avoided
• Syntax is repetitive
• Syntax is rigid
• Emulation of events is difficult or impossible
• Hard to debug

Conclusion

There is no tool that I would be crazy about and super helpful for UI developers (I am thinking of creating one :), but looks like Cypress is by far most suited for UI e2e tests. It has very expressive API, feature rich, extensible, great documentation and community support. There are a few things I do not like about it, but it is a subject for another article. Here are my scoring table:

	Cypress	Nightwatch	Nightmare	Taiko	TestCafe	WebDriver.IO	Max Possible
Expressive API	5	3	3	0	0	1	5
Rich API	5	2	1	1	3	4	5
Quick Dev Cycle	3	1	1	1	0	1	5
Extensible	4	1	1	1	1	1	5
Easy to Debug/Set Breakpoints	2	0	0	0	0	1	5
Pause/Resume Step Through Test	3	0	0	0	0	0	5
Integrated into Dev Cycle	1	0	0	0	0	0	5
Good Docs/Community	5	2	1	2	5	5	5
Run in Browser	1	1	0	1	1	1	1
Run Heedlessly	1	1	1	1	1	1	1
Cross Platform/Browser	2	3	1	1	3	3	3
Capable of CI/CD Integration	2	2	1	1	2	2	2
Capable of Parallel Execution	2	1	1	1	3	2	3
TOTAL	36	17	11	10	19	22	50

A word to QA engineers

I am becoming a strong advocate that developers should write automated tests, however, it does not eliminate the need for QA engineers. Each team should have QA engineers to help with following areas:

• Second pair of eyes. If you look at your "creation" for a long time and interact with it, you inevitable fall in love with what you've created, event it were a Frankenstein. At the very least you get used to it and often miss something obvious. This is where a fresh look helps.
• Systematic approach to testing. QA engineers are trained to look for bugs, so they know most problematic areas, such as, user input validation, security flaws, taking a comprehensive look at the entire software, etc. This help is invaluable for developers.
• Overall Test Automation. To make sure that all aspects of automation is covered, someone needs to take a bird's eye view and write tests where they are missing.
• Management of Test Automation. If developers will be contributing to Test Automation suite, someone needs to make sure that all the test from all developers are working fine.
• Requirements validation. Requirements come from users, often in the form of user stories. These stories are interpreted by developers into a working software. Someone needs to look at the software and validate if it actually corresponds to user requirements.
• Educate developers about testing. It could be overwhelming to do development and test automation in parallel. Having extra resources to educate and help team members with software automation is a huge help.
• CI/CD integration/maintenance. Someone still need to make sure Jenkins is working properly.

Resources

• An Overview of JavaScript Testing in 2020
• Why regression testing should be a shared responsibility
• Who is responsible for the quality of application testing? 10 reasons why the bug hits the production
• Differences between TDD, ATDD and BDD
• Which E2E testing framework to use for JS-based client applications?
• Which JavaScript UI Testing Framework to Use in 2020?
• Unit vs E2E Testing for Vue.js

User Comments

Other Articles

• Release Notes for w2ui 2.0
  April 26, 2023
• Quality Engineering for Web UI
  October 5, 2020
• Info Bubbles in the Grid
  December 12, 2015
• W2UI 1.4 Released
  July 21, 2014
• HTML5 Input Types Alternative (Part 2)
  July 17, 2014
• Folder Structure for Single Page Applications
  July 7, 2014
• HTML5 Input Types Alternative
  January 22, 2014
• Nested CSS Rules
  November 22, 2013
• W2UI 1.3 Released: Goodbye Pagination, Hello Infinite Scroll
  October 29, 2013
• JavaScript Grid with One Million Records
  July 5, 2013
• W2UI 1.2 Released
  May 4, 2013
• JavaScript W2UI 1.1 Released
  February 9, 2013
• Using ANT to Build and Deploy JavaScript Projects
  October 27, 2012
• CSS Scoping
  August 27, 2012
• Speech Bubbles in Pure CSS
  June 2, 2012
• A Variation of JavaScript Module Pattern
  May 25, 2012