Automated visual testing with Percy

2021-06-18

I shared my last post with some friends, and soon found out that the images and videos didn’t load on Safari. What better time than to introduce an overly complicated automated testing process to catch these problems in the future!

I first checked my site in BrowserStack. Sure enough, it didn’t display correctly on any iPhones or Safari on MacOS.

I’d heard of Percy before, but never used it. It takes snapshots of your site, and provides a visual tool for comparing differences as you make changes. Seems like a good way to make sure I don’t break the site as I inevitably try out new stuff.

Running Percy locally

Using WebdriverIO

Percy is just a visual comparison tool, not a testing framework. It supports many frameworks, including WebdriverIO which I’m familiar with, so let’s start with that:

# initialise npm in my blog project (nooo! I didn't think it would happen so soon :( )
npm init
# add webdriver
npm install @wdio/cli
npx wdio init
# modify package.json to make the test script = wdio ./wdio.conf.js
# check it works:
npm test

All good. Now replace the example test wdio created with just opening a few pages on my site running locally:

const baseUrl = 'http://localhost:1313';

async function click(selector) {
  const elem = await $(selector);
  await elem.waitForDisplayed();
  await elem.click();
}

describe('stuff', () => {
  it('goto homepage', async () => {
    await browser.url(baseUrl);
  });

  it('goto blog', async () => {
    await browser.url(baseUrl);
    await click('=Blog');
  });
});

In one terminal:

hugo server

In another terminal:

npm test

OK. This should be enough to take a couple of snapshots with Percy. Install it:

npm i @percy/cli @percy/webdriverio

Add a percySnapshot at the end of each test, then run:

export PERCY_TOKEN=my_secret_token
npx percy exec -- wdio ./wdio.conf.js

All my pages are new at this point, so there’s no changes to compare:

example of a new snapshot display on Percy's UI

Using Percy CLI

As usual, after setting up a complicated way to do something, I discovered that there’s an easier process. For static sites like this one, the Percy CLI can take snapshots of all published content. Let’s try that:

# publish everything
hugo
# snapshot all html files, and send to percy
npx percy snapshot public/

This is much easier than using webdriver! In the bin!

Also, I’ve just realised that the only browsers currently supported by Percy are Chrome, Edge and Firefox. Derp. So much for using it to detect problems on Safari. Ah well, I’ve come this far, and it may still come in handy. Let’s get it running in GitHub instead of locally. Add the Percy token to the GitHub repo: Go to the repo, then: settings -> secrets -> new secret.

Then, create an action in .github/workflows:

name: Percy
on: [push]
jobs:
  snapshot:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: get Node.js
        uses: actions/setup-node@v1
        with:
          node-version: 12.x
      - run: npm ci
      # don't need to do a hugo build here - I do that locally, for now at least
      - run: npx percy snapshot public/
        env:
          PERCY_TOKEN: ${{ secrets.PERCY_TOKEN }}

Push the changes. If all’s gone well, the action will upload the site’s snapshots to Percy, where changes can be reviewed:

example of comparing two versions of a page in Percy's UI

The Percy UI shows a list of pages, with visual changes highlighted in red. You can toggle the highlighting on/off with the ’d’ key, which makes finding and comparing changes easy.

Next steps

OK. I’m tired, and as usual, have learned a bunch of stuff without solving my original problem. Time to actually fix things for Safari :) I’ll just manually test things in BrowserStack for now.

It’s possible to set up easy navigation to/from Percy from PRs on GitHub. This means PRs will show the Percy review status, which presumably goes green when I approve all the changes. I can’t be bothered setting that up right now, but the docs for setting that up are at Percy docs: source code integrations.

Epilogue - why wasn’t Safari showing images and videos?

It looks like only very recent versions of Safari support webp and webm formats, which I was using. I’ve since converted them to jpg and mp4 (with h264 video).

References