David Ruttka

Engineering | Gaming | Improv | Storytelling | Music

Invest in Test

| Comments

Seeing Red

Can you imagine having this level of objective feedback within five seconds of making your change? I can, because despite my being lazy about it in recent years, my teams and I have enjoyed the benefits of investments in test.

Can you imagine merging this type of change and only finding out how bad you’ve broken things after it’s in production and impacting your business? I can, because due to my being lazy about it in recent years, my teams and I have suffered the pains of throwing things over the wall.

Why Test

I’ve given conference talks on all of this, but the React Native docs say it better than I ever could. Check out the Why Test section.

Assumptions

Here are a few things that I’ve been guilty of saying.

Works on one platform, no need to test the other.

And from browser compatibility to PhoneGap plugins to React Native components, I’ve been wrong countless times.

The change I made to this shared component looks great when I verify the bug I was trying to fix. Surely it’ll be fine for all other consumers.

Of course, even when I think I’ve added nothing but new functionality driven by optional parameters with sensible back-compatible defaults, I’ve been wrong countless times.

There’s no way we could get here in null or empty set conditions, so we don’t need to guard against those conditions.

LOLOLOLOLOLOL

I’ve done a custom sort function enough times to know the difference between returning -1 and 1, and I’d certainly never forget to return 0 when the items should be treated as equal.

I have become the physical embodiment of shame.

Two Birds, One Stone

I recently shared my silly little side project, Phasekeeper. As we’ve continued using this at home, I’ve become frustrated that the end screen didn’t sort the players in descending order.

I’ve also wanted to create a reduced, streamlined example to demonstrate the return on investment with automated testing.

Let’s go!

Exploding components

Before I tackled the sorting, I had something else on my mind.

There’s no way we could get here in null or empty set conditions, so we don’t need to guard against those conditions.

I cast no stones, as I’d need a pitchback to direct them at myself, but I’ve lost count of how many times a component explodes while accessing foo.bar while foo is falsey.

In Phasekeeper terms, I’ll admit this is contrived. The flow of Phasekeeper does ensure that we wouldn’t get to the End Screen without players. Future me looks back at this and says, “That’s what you thought at the time.” Present me says, “Why not take this opportunity to demonstrate how we’d test and fix it anyway?” Past me says, “Listen to them.”

Now, we have a few options to dodge the investment in automated tests:

  • Could I have checked this manually?
  • Spotted it during code review?
  • Just been a better @^#*! developer who guards against every possible condition, no matter how unlikely?

Probably at least two of those three, sure. But in less than twenty minutes I fixed the bug with an objective data point. The collection of that data point is repeatable to ensure we don’t regress with future changes:

Getting things sorted

Next, I committed a change to test and fix the sorting of players. Note that this test arranges the state so that all players had completed Phase 10. More on that later.

So far, so good. We’ll render the End Screen with a test state on our context, and now we know

  • The Congratulations text is presented
  • It uses the correct name
  • Players are listed in the order of their scores (remember, in Phase 10, a lower score is better)

Whoops

Do you remember when I said

I’ve done a custom sort function enough times to know the difference between returning -1 and 1…

I promise you, I did not do this on purpose for the sake of a contrived demonstration. I actually, with two decades of experience, got the sort return values backwards. Again.

Here’s the commit where I fix the sorting to account for the condition where not all players complete Phase 10. For those who don’t play this game often… this is not an edge case!

It’s not as clear in this commit as in the Exploding Component commits because I didn’t commit the failing state first, but I did

  • write the new tests first
  • run the tests, fully expecting to see everything green
  • watch the new tests fail
  • facepalm myself on the reversed -1 / 1 return values
  • swap the -1 and 1 returns
  • watch the new tests pass

It’s important to note that I’m not just getting an objective data point that I’ve fixed the condition I was concerned about. What did I get as a bonus?

  • I didn’t have to click through my UI to manually check this result
  • Which means I got the answer in less than three seconds, not minutes
  • AND ALL PREVIOUS TESTS ALSO STILL PASS WITHOUT ME MANUALLY RETESTING THEM

That’s right! I don’t have to go do a manual regression pass to make sure that all those previous scenarios still behave as intended.

Mileage Varies

As always, everyone and every team is different. You may get different returns on this investment.

For me, these tests have become executable specifications that provide a more efficient and more objective evaluation of the code. It took me less time to write and run them than it would have taken to run through the UI for each combination.

We now have a permanent record of which combinations we’ve checked, and which ones we haven’t.

And that’s just the initial investment. Now that the harness exists, it will be even more efficient to assert other interesting combinations if the need arises… all the while avoiding the need to repeat manual regression passes against the previously tested combinations.

All in under three seconds.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 PASS  src/components/EndScreen.test.tsx
  EndScreen
    with no players
      √ it renders an error view (25 ms)
    with all players completing Phase 10
      √ has congratulations text (7 ms)
      √ congratulates the player with the lowest score (4 ms)
      √ lists all players (4 ms)
      √ lists the winner first (3 ms)
      √ sorts the other players with the lowest scores first (3 ms)
    with only some players completing Phase 10
      √ has congratulations text (4 ms)
      √ congratulates the player that completed Phase 10 with the lowest score (4 ms)
      √ lists all players (7 ms)
      √ lists the winner first (5 ms)
      √ sorts the other players in order of completed phase, then lowest score (4 ms)

Test Suites: 1 passed, 1 total
Tests:       11 passed, 11 total
Snapshots:   0 total
Time:        2.596 s, estimated 3 s
Ran all test suites.

Comments