Contemplating automated testing? Calculate your ROI first

In software development, speed has become the defining characteristic of product launches. It sets the standard we can’t really escape anymore.

It places demands on market players both old and new: release now, fix the issues later. Launch the feature today or watch your competitors outpace you. Meet the deadline or lose a vitally important partner/client. 

These are all thoughts and ideas most product managers, CTOs, and team leads have had (or heard). In part, I think the culprit for this state of affairs has been automation itself, as it has streamlined so many tech processes across industries and has set client and market expectations.

For software products, however, automation can also be the saving grace, offering a unique opportunity to go both fast and steady.

I’m talking, of course, about automating tests to ensure both velocity and product stability.

The pitch is: why choose between quality and speed when automated testing can ensure quality at high speed? I know that’s quite the mouthful, but it’s essentially true.

Now, many companies, product managers, CTOs, and CEOs are hesitantly standing on the doorstep of that ‘automation ride’ and wondering: should I go in? Is it worth it? Will I spend more than I get back?

The truth is… automated testing isn’t that different from any other business investment. It all comes down to calculating the returns — here’s how I do it.

The basics of test automation ROI calculation 

First and foremost, let’s look at the basic method of ROI calculation. The most commonly used formula for automated testing is the following:

ROI for testing automation = Cost savings / Investment cost 

OR

(Investment value – Investment cost) / Investment cost  

However, each company will have to take into account their own specifics, pre-existing problems as well as natural trends in software development to apply this calculation correctly.

To begin with, cost savings can be calculated in various different ways, based on your knowledge of company processes, expenditures, and the testing automation tools in question.

For example, the go-to method of counting savings is to consider how much Quality Assurance time is saved by automation. If implementing automated testing saves us Z hours of QA each month, it’s easy enough to calculate the X cost of those work hours versus the Y cost of implementing automated testing. Most people see it as a simple matter of inserting X and Y into the first variant of the equation.

If we assume that the cost of implementing automated testing is 500 units and we save 100 units each month on lessened QA time, we’ll break even in 5 months. Beyond that point, we have a constantly increasing positive ROI.

Next, the investment category of the equation can be vastly different in each specific implementation case. Some companies opt to develop their own automation framework, while others prefer to find external partners that specialize in this area or have ready-made tools.

Needless to say, depending on whether your team has experience in developing and maintaining automated testing frameworks, the costs (and cost-effectiveness) can vary wildly. 

Another “phantom” point to factor into your analysis is what happens if you don’t invest.

Opportunity costs of manual testing

No discussion about any sort of ROI can really be complete without considering the potential financial or organizational losses you might suffer if you don’t invest in new tech. “The cost of lost opportunity”, as it were.

To correctly evaluate the opportunity (i.e. investing in a streamlined process), you have to consider the downsides of the commonplace alternatives. Here’s a brief rundown of the possible lost opportunity costs of sticking with manual testing:

A slow-down of multiple company processes

The average speed of automated testing is about five times that of its manual counterpart. This impacts your delivery velocity, deadlines, team dynamics, stress, and employee burn-out.

That’s only half the issue though. Any in-house department influences other departments. The company is a living, breathing organism.

A slowdown in development or production can also grind other departments to a halt: sales, marketing, auditing, and others. It can place a strain on HR to manage employee happiness or to recruit new developers/testers that couldn’t handle the crunch.

Organizational micromanagement 

As with anything that’s done in-house and by humans, there’s always a need for organizational management, adjustments, meetings, briefings, and employee one-on-ones.

If your managers don’t have to deal with that yet, it’s only a question of time. As soon as you scale up, you’ll have to get involved in managing the human side of your teams under the immense pressure of delivery velocity.

Manual testing increases this micromanagement strain since you’re offloading even more technological complexity onto living people that will require support to stay productive.

Lowering of delivery output due to regression tests

As Blake Norrish accurately points out in his blog on “the Regression Death Spiral,” testing isn’t only about verifying that one new feature. There’s also regression testing.

Consider what happens when you’ve been working for years on a software product. Dozens of engineers and developers leave, new ones arrive. It’s a multi-layered cake of code created by different people.

Each manual test is a laborious process of making sure the new feature doesn’t break any or ALL of the previous ones, created over the ENTIRE period of development.

Ignoring this idea can often lead to disastrous consequences (like a new patch that breaks the fundamental functions of the entire product).

What happens when your team has to do continuous regression testing of each new feature? That’s right — everything slows down painfully. Burn-out, stress, and missed deadlines ensue. Automated testing is the widely accepted solution to avoid both slowing down your testing AND avoid new features breaking old systems.

Overall, only the stakeholders within the company itself can determine whether you need to implement test automation at any given time. However, none of us can run from innovation forever and not ultimately fall behind.

Common mistakes in ROI calculation 

As with any new endeavor, making the calculated decision to adopt new tech is only half the journey. How you approach the implementation process and whether you’re wise enough not to repeat the mistakes of others is likely to determine your ROI as well.

Thankfully, the industry has made enough strides in the direction of automation for most of the common pitfalls to be glaringly obvious. Let’s take a look:

Ignoring ALL manual testing scenarios

Automated testing doesn’t fit every single scenario and process. A lot of the value of automated testing stems from running these tests multiple times and not every scenario requires automation.

If your development is diverse in its processes and products/features, you may have to run manual tests alongside automated tests. Being discerning where to apply one or the other will save you money.

Only short-term or narrow ROI calculations

While it’s completely fine to first consider the initial cost and return of implementing test automation, it’s also necessary to evaluate the long-term ROI for your company.

I’ve already outlined the potential problems with manual in-house testing. No doubt you have your own insights into these (and others) since you know your team well.

Will automated testing solve more issues than just QA time in the long run? Will it open up new opportunities? Will it improve the quality of development and product launches? These are all things to consider in long-term ROI calculations.

Ignoring test maintenance

Automated tests are still code. They’re still processes that need to be maintained and updated if you want them to provide value for years. This is a factor you need to investigate in your ROI equation.

Is your team ready to handle the maintenance? Or do you have a partner/outsourcing firm that’s proficient in this? How do you wish to handle maintenance in the long-term?

Ignoring records/documentation

This is where you address the ‘knowledge leakage’ parameter in automated testing ROI calculations. It’s a short and vital directive:

Document everything you can.

There are few events as damaging as losing a key employee that kept all the knowledge in their head and having to re-engineer numerous complex cases. Automation can help here as well, as there is less of a dependence on human expertise (though it is still obviously necessary).

Depending on your product, organizational structure, and management style, the above considerations may move up or down in priority. You may even encounter entirely new pitfalls as the technological environment changes. Yet keeping these points in mind will help you stay flexible and vigilant.

Advanced test automation ROI parameters 

The basic formula for ROI calculation on automated testing (ROI = Cost savings / Investment cost) is almost universally applicable. However, its components may vary depending on your company. Furthermore, the metrics, cases, and best practices to pay attention to may also be different. 

There are six parameters/variations of ROI calculations for automated testing, which may be stand-alone for your company or integrate into one bigger equation:

• Automation of new tests — this is the most standard case scenario, handled by the simplest “QA time saved” formula.
• Automation of older tests — I discussed regression testing, this is a best practice use case to seriously consider.
• All-around environment testing — knowing how your product handles in various environments may be key, considering most companies seek to diversify in this area. How much will you save on avoiding bugs across different environments?
• Defect leakage — the number of bugs “leaking” from development into production. Many experts consider this to be the most impactful parameter since it can do the most damage when affecting customers (and consequently, your business).
• Reuse or redundancy of tests — there’s little reason to build a new test when an existing one can be repurposed. Modular scripts are one way to mitigate this, but there are others. Consider the savings aspect here.
• Knowledge leakage — consider the average tenure of a QA engineer. Then think about what happens if he/she leaves. What’s the cost of time spent on re-training a new one and re-engineering lost manual test cases? How can automation reduce your costs here?

These can be parameters in your larger calculation or specific cases if you choose to implement automation in one area first. This leads us to the topic of the general best practices in the industry and how to ensure that the ROI formula actually corresponds to reality via correct tech integration.

Test automation implementation best practices

An important factor in calculating ROI and preparing to integrate a new company process is knowing the best practices of the industry, following which may directly or indirectly affect your costs and returns.

The two rules of thumb are: gradual introduction and long-term planning.

Calculating ROI on paper is one thing, but making sure that the company organically adopts a new technology is quite another. So, what are the three main points to consider during actual implementation?

1. Don’t automate every single process right away

Rushing to automate every single test from day one is the other end of the extreme. As I mentioned before, manual testing has its place.

Consider carefully the pipeline of your automated transformation and what kind of balance your company has when it comes to testing needs. Which of the advanced ROI parameters listed above is the most vital for you? Where should you start?

2. Understand that every test will become a regression test eventually

The danger of ignoring ROI on prior test automation is that it leaves a blind spot in your company: you’re not taking the future of your current tests into account.

The best practice for this is to integrate new tests as an element of your regression testing process right away.

3. Evaluate the timeline (waterfall vs. shift-left model)

The shift-left testing method has been hailed as the saving grace of companies mired in expensive manual testing via the old waterfall model (where development and testing are done sequentially).

Shift-left is based on simultaneous development and testing via automation and agile practices, ensuring better quality and considerable time savings. In most cases, this is the go-to direction for most development and QA teams.

Conclusion

Automated testing has the potential to take most companies to a new level of efficiency. In today’s competitive markets, speed and quality of delivery are vital factors of survival and success.

The synergy between human expertise and automated tools is what defines the most dominant market players. ROI calculations remain the main method of determining how to approach test automation, but managers must never forget that it’s not a cut-and-dry formula.

At the end of the day, your own internal resources and company structure, choice of automation partner, client needs, and other factors have a serious impact on your long-term returns.

So, be cognizant of your company specifics, find trustworthy contractors, and value your team and development process health above all.