Assertions in Automated Testing: Assertion Consolidating

No matter what role you have with creating automated testing (Unit, Integration, End to End, etc), we can all agree that the goal is shooting for full code coverage so that we can ensure our applications always work as expected. This includes performing assertions on the results of API calls, values and states of components on a UI, or countless other scenarios. This means verifying multiple values in a test. A common practice to accomplish this is writing a single assertion for each value.

const result = await getVehicle({ id: 1 });
expect(result.make).toEqual('Chevy');
expect(result.model).toEqual('Colorado');
expect(result.year).toEqual(2022);
expect(result.trim).toEqual('LT');
expect(result.color).toEqual('Summit White');

While this technique works, we have a problem. By default, most automated testing frameworks will stop execution with a failing assertion and no other assertions will run. To take the example above, if expect(result.make).toEqual('Chevy') fails, everything stops.

Even worse, we wouldn’t know if other assertions pass/fail until the first assertion passes. If others do fail, then we would start the process all over again.

A general best practice is to aim for one assertion in a test. The example above could be broken into multiple tests, but that would be overly verbose with a test for each property value. Not to mention, that would stray far from any code reuse potential. Fortunately, there are strategies to consolidate multiple assertions to provide a full picture of all of the results within a single test.

The examples below are demonstrated using Jest as the testing framework, but most other testing frameworks should provide similar capabilities.

Use object matcher assertions

The example above but now using an object matcher assertion:

const expected = {
    make: 'Chevy',
    model: 'Colorado',
    year: 2022,
    trim: 'LT',
    color: 'Summit White'
}
const result = await getVehicle({ id: 1 });
expect(result).toMatchObject(expected);

By using the toMatchObject() matcher, if multiple property values are different, you will get a consolidated result. The following is an example result output if both model and trim values do not match.

expect(received).toMatchObject(expected)
    - Expected  - 2
    + Received  + 2
      Object {
        "color": "Summit White",
        "make": "Chevy",
    -   "model": "Colorado",
    -   "trim": "LT",
    +   "model": "Silverado",
    +   "trim": "LTD",
        "year": 2022,
      }

With this consolidated result, all failures are identified with a single run of the test! This simple change saves software/quality engineers time and reduces CI/CD cycles which deliver software faster.

Each testing framework will have multiple assertion/matcher functions for objects that will perform their assertion slightly differently. The `toMatchObject()` was selected here as it performs its assertion in a way that is similar to multiple individual assertion statements. It asserts only a subset of properties by checking only what is provided in the `expected` object.

Use a custom matcher

Object matchers definitely help to consolidate assertions when you are just checking property values, but what if you have a series of common assertions that verify more than just property values? With Jest you can create your own custom matcher by using expect.extend(). This provides the ability to create very robust assertions that can consolidate any number of other assertions to provide effective code reuse and can ultimately improve the readability of the code within tests.

The following code defines our custom matcher.

expect.extend({
    toMatchVehicle(vehicle, expected) {
        if (!vehicle instanceof Vehicle) {
            return {
                message: () => 'The expect() parameter must be an instance of Vehicle to perform toMatchVehicle() assertion.',
                pass: this.isNot
            };
        }
        const failures = [];
        if(!this.equals(vehicle.make, expected.make)) {
            failures.push(`Expected vehicle make: "${expected.make}" - Actual: "${vehicle.make}"`);
        }
        if(!this.equals(vehicle.model, expected.model)) {
            failures.push(`Expected vehicle model: "${expected.model}" - Actual: "${vehicle.model}"`);
        }
        if(!this.equals(vehicle.year, expected.year)) {
            failures.push(`Expected vehicle year: "${expected.year}" - Actual: "${vehicle.year}"`);
        }
        const pass = failures.length === 0;
        if(pass) {
            return {
                message: () => 'Expected vehicle not to match, but it did.',
                pass: true
            };
        } else {
            return {
                message: () => {
                    let msg = `The vehicles did not match:n`
                    failures.forEach((failure) => {
                        msg += `- ${failure}n`
                    })
                    return msg
                },
                pass: false
            }
        }
    }
});

The use of the custom matcher assertion.

const expected = new Vehicle({
    make: 'Chevy',
    model: 'Colorado',
    year: 2022
});
const result = await getVehicle({ id: 1 });
expect(result).toMatchVehicle(expected);

Finally, an example output if there are failures.

The vehicles did not match:
    - Expected vehicle make: "Chevy" - Actual: "Ford"
    - Expected vehicle year: "2022" - Actual: "2021"

Admittedly, this custom matcher example is verbose with rather simplistic checks. With that said though, it does demonstrate a couple of the major benefits of creating a custom matcher.

• Ability to perform multiple types of checks within a single assertion
• Ability to customize the failure messages when an assertion fails

These benefits ultimately lead to improved debugging with fewer and shorter code iteration cycles.

Summary

We explored the major drawback of utilizing multiple narrowly scoped assertions. We then demonstrated how object matcher and custom matcher assertions are great alternatives. Hopefully these techniques can help you to author more effective tests and reduce your test iterations. Happy testing!

FloQast is growing and we are looking for great software engineers and SDETs. Checkout our current open positions!

Kerry Doan

Kerry is a Senior Software Development Engineer in Test at FloQast on the QE Core team helping to bring common QE solutions for use by all FloQast engineering teams. When not working, Kerry enjoys hanging with family, mountain biking, and football.