Visual regression testing

The visual regression testing is done with backstopjs. For a proper and unified testing environment the complete test run is done in Docker. This approach allows us to run the same test locally and in the CI and expect the same results.

What is a golden sample

A golden sample is the reference screenshot of a component, e.g. the primary button. Every time the test runs, it will compare the golden sample against the new screenshot that will be taken by backstopjs. When the difference is below the given threshold, the test will pass.

Tools

The following tools are integrated to run the visual-regression test cases:

  • backstopjs as the test runner
  • Docker as the test environment

Build docker image

To run the test cases locally, the developer first needs to build the docker image from the Dockerfile. This file is located in the folder visual-regression-test.

The following command can be used to build the docker image for local testing.

DOCKER_BUILDKIT=1 docker build --target=dev-img -t visual-regression-test:latest .

The image that is used in the Azure CI pipeline can be build with.

DOCKER_BUILDKIT=1 docker build --target=ci-img -t visual-regression-test:latest .

Update ci docker image

To update the docker image that is used in the azure pipeline the following steps needs to be followed:

  • build new ci image 
    DOCKER_BUILDKIT=1 docker build --target=ci-img -t visual-regression-test:latest .
  • tag image 
    docker tag visual-regression-test uiciregistry.azurecr.io/frok/frok-ci:node12-lts-backstop-5

For the following steps azure cli is required and permissions to push to the registry

  • login into azure 
    az login
  • login into azure ui/ci registry 
    az acr login --name uiciregistry
  • push image to registry 
    docker push uiciregistry.azurecr.io/frok/frok-ci:node12-lts-backstop-5

How to run

To run the test cases locally, the following steps needs to be done:

  • yarn run build to build the project
  • yarn run serve to serve the build
  • yarn run test:visual:run:atoms to run the test cases for atoms
  • yarn run test:visual:run:molecules to run the test cases for molecules
  • yarn run test:visual:run:organisms to run the test cases for organisms
  • yarn run test:visual:run:[atoms|molecules|organisms] -- --filter="COMPONENTNAME" to run cases for a specific component only.

To run the test cases in a CI environment, the command yarn run test:visual:run:ci:[atoms|molecules|organisms] can be used.

How to create new test cases

To create new test cases for components, the following steps should be done:

  • create a new test specification file or extend an already existing one in ./visual-regression-test/tests/*
  • for new test specification file choose the correct level, e.g. atoms
  • name the new test specification file <component-name>.spec.js
  • add new test cases to the file
  • require the newly created test in ./visual-regression-test/backstop.config.[atoms|molecules|organisms].js

Naming of the test scenarios

The label is required and is the name of each test case. The schema for naming test cases is:

<component-level> - <component-name> - <component-variant> - <component-state>

The possible <component-state> are default, hovered, pressed. This pattern is derived from the css custom properties.

Example test specification file

module.exports = [
  {
    label: 'Atoms - Button - Primary label - Default',
    url: 'http://host.docker.internal:9000/atoms/button/primary-label',
    selectors: ['.a-button'],
    misMatchThreshold: 0,
    requireSameDimensions: true,
  },
  {
    label: 'Atoms - Button - Primary label - Hovered',
    url: 'http://host.docker.internal:9000/atoms/button/primary-label',
    selectors: ['.a-button'],
    hoverSelector: '.a-button',
    misMatchThreshold: 0,
    requireSameDimensions: true,
  },
  {
    label: 'Atoms - Button - Primary label - Pressed',
    url: 'http://host.docker.internal:9000/atoms/button/primary-label',
    selectors: ['.a-button'],
    activeSelector: '.a-button',
    misMatchThreshold: 0,
    requireSameDimensions: true,
  }
]

For explanation of the different key:value pairs of each scenario, please head over to the backstopjs documentation.

To trigger hover states you need to pass a hoverSelector or hoverSelectors array.

To trigger active states you need to pass a activeSelector or activeSelectors array.

Special Triggers

Scenarios like 'click and then hover' or 'drag' won't be handled by Backstop Js, but by Puppeteer. All those special custom scenarios can be found in the folder ./visual-regression-test/helper/*.

How to handle changes

When there are planed changes to a component, that will affect the design the following steps should be followed:

  • create a new branch
  • develop the new features
  • run visual regression test locally
  • only the target component test cases should fail
  • push to the server
  • design review should check the changes to the component on the preview server
  • design gives feedback if the changes are correct
  • development update the golden sample for the component
  • run visual regression test locally
  • all test cases should pass
  • push update to the server
  • test cases on the server should also pass

How to approve changes

To approve and update a new golden sample, the following commands can be used:

  • yarn run test:visual:approve:[atoms|molecules|organisms] to approve them all.
  • yarn run test:visual:approve:[atoms|molecules|organisms] -- --filter="COMPONENTNAME" to approve only one component.