Context

I decided to write this blog post after dealing with some disabled tests in a project I recently worked on. Those tests were there for almost three years. I did not write them, but the day-to-day circumstances put them in front of me.

It was a Kotlin codebase; therefore, I had a type checker working for me. The problem could be even worse if those tests did not compile, but as expected, I was challenged (by two seconds) by the usual cognitive load attached to figuring out why such tests were disabled in the first place.

For almost three years, those tests had been kept up-to-date by automatic code refactors but have yet to review. Those tests never delivered any value. Hence, I did the obvious thing and deleted them. Fearlessly. No time wasted trying to figure out how to make them work.

Such a situation made me think: why do people still keep around tests that never run?

A glorified form of commented code

I think I have a possible answer, taking in consideration 10+ years working as Software Engineer. It relates to the same reason why people insist on keeping commented source code around, especially in a world where the status quo is using a Version Control System like Git in all professional software projects.

I talk here about the false promise one makes to himself/herself, something like:

“I will return to this code as soon as possible, to make things right.”

There are many reasons why people opt to disable tests, and all of them boil down to the same point: it is a quick workaround when an Engineer doesn’t understand why a test fails but he/she wants to remove a blocker without finding a proper solution to it, and without giving up what he/she already has.

Since the “almost working” test is around, the author assumes it will be fast and easy to resume the work when going back to that part of the codebase later. Since that test never executes, it can stay around without harming anyone.

Now, I invite you to replace “test” by “commented code” in the excerpt above, and you’ll get my whole point. A disabled test is just as a glorified form of code commented out in source files, and the proof is that Engineers often fail to elaborate about real advantages of skipping a test execution compared to commenting out that same test.

Disabled tests could also be interpreted as an incarnation of dead code, although I see this one happening non-intentionally during the Software Development Lifecycle. Regardless of the point of view, by disabling a test or commenting out a code snippet one brings no real value to the project. On the other hand, there is room for issues.

As a first example, cognitive load is almost guaranteed when returning to a disabled test. The more time passes, the more severe the cognitive hit, either due requirement changes, context changes, people changes, etc.

In addition, I follow Kent Dodds’s argument , and I see disabled tests promoting information hiding in the same way commented snippets do. Likewise, keeping useless tests around is another stone contributing to more broken windows in a project, since they effectively cause cluttering and misleading to a test suite.

An easier and better way

If disabled tests are just another form of commented code, both should be treated in the same way in terms of good practices to adopt in Software projects.

It’s been nearly a decade since I educate peers - especially entry-level Engineers - about commenting out code and how it represents bad hygiene in terms of Software Quality. Every time I face commented snippets in a Pull/Merge Request, I block that PR review until that code is appropriately changed or entirely removed.

I think disabled/skipped tests should face the same zero-tolerance policy.

That should happen in typical code reviews (Pull Requests) or pair/mob programming sessions. Even better would be having an automation enforcing such a policy, either on top of quality checks driven by the build system or tasks executed as part of the Continuous Integration process. When a robot says what everyone must follow there is no room to pointing out fingers or discussing personal tastes.

If you are missing a test case for any reason, check out Git history to learn if such test case was considered before or just implement it right away. If you can’t make it work, remove it completely and track a missing test case in your project tracker, so a priority is assigned to it later.

Just say no to disabled tests. Your future self and your future peers thank you.