Test Automation ROI

Recently I was working on a ROI (Return-on-Investment) Model for Test Automation. It was quite interesting to note that there are not too many "ready-made-excels" out there for this purpose. The ones that exist come mainly from automation tool manufectures. So I went to creating my own model, resulting with interesting notions.

Test Automation ROI Ingredients

Expenditure side

1. NRE (Non-Recurring Effort / Expense) - buying tool license, training people, raising the environemnt and infrastructure
2. Implementing the initial testing scenarios
3. Maintaining the testing scenarios and the environemnt and infrastructure
4. Adding new scenarios and new required features to the environemnt and infrastructure
5. Usually requires better trained testing engineers to maintain the automated testing system

Revenue side

1. In the long run, may be able to reduce manpower (less manual testing is needed)
2. Reduce number of escaped defects
3. Increase development confidence, allowing more features per release, thus less releases per year
4. Shorter test and development cycles and improved TTM (Time-to-Market)
5. Enhances the abilities of the testing team ("soft" revenue)

Influencing Factors

When coming to real life numbers it appears that it is very tricky to achieve positive ROI. And even if there is positive ROI the break even point comes after at least one year of investment, and in most cases two years. It's a simple case of big lump sum investment for future expected revenues.

The ROI is strongly related to the amount of bugs detected in the system in general and specifically in the field, number of existing manual testing engineers and the number of releases per year:

• A system with very low amount of bugs that are mostly found at testing room and not in the field, would make it harder for automation to have positive ROI, as the potential of reducing escaped defects is low
• If the current existing number of manual testing engineers is small, potential of revenues in reducing manpower is low. However, if number of escaped defects is high automation can assist in lowering it
• Low number of releases per year would also make it harder to achieve positive ROI, as the lump sum investment is made for less cycles of use per year

The ROI is strongly related to the changes in the system:

• Need for new scenarios and features in the environemt along the year means more investment in maintaing the testing system up-to-date
• The lifetime of an automated scenario before it becomes broken or irrelevant because of changes in the system is very important

On an "average" system with some major new features each year, about 4 major releases per year and about 20 major escaped defects per year, following can be found:

• In order to properly support and maintain a test automation project, you would probably need the same amount of people as you had before with manual testing, all along. The new features in your product requires new testing scenarios as well as new features in the test automation environment
• The major revenues stem from reducing number of major releases (adding more features per release) and from reduced number of defects in field
• It takes about 18 months to return the investment

Projects that are not stabilized yet and have too many changes would probably not earn from end-to-end test automation, although automated regression based on UnitTesting can still assist.

Module Separation

Module separation is one of the oldest tricks in software engineering. As it is too complicated to attack a big problem, we split it and attack it in pieces. The pieces should eventually work together. We don't want to have two teams digging a tunnel from two sides of the mountain, missing the meeting point. In the tunnel case you get two tunnels which may have some value, with software you will just have a dysfunctional messy system.

The engineering task, some would say the art, of defining the modules and the interactions between them, is crucial for the success of the system. Do we need our modules to be stateful? Do we want synchronous or asynchronous communication? What would be the messages language between the modules, would it be programming API or a communication layer? On top of which protocol the messages would travel? And on top of all, which modules should we define? and for each module, which modules it shall be aware of and which it shall be totally agnostic and unaware of?

Usually we would like the communication between modules, i.e. the public interfaces that a module should expose, to be as little as possible. Operations or sub-operations that require much communication would be preserved in the same module while operations or sub-operations that require lesser amount of communication can be split. We should also like to have some hierarchy between modules so that not every module would have to know all the others, preventing cyclic dependencies is usually an architectural requirement.

So, what is the definition of a module?

Trying to create a definition of a module, or even better, a subscription, is not easy. Recently I was attending a session by Rick Kazman the co-author of "Software Architecture in Practice". Rick referred to this question, defining a module as "the set of tasks belonging to a unite topic, managed and built by an independent team of people." It is interesting to note that Rick's definition focus on the organizational aspects more than on architectural aspects of "what is a module". Rick argues that this goes well with Conway's Law stated by Melvin Conway in 1968, arguing that the required communication between two teams has a direct relation to the required communication between the modules they develop.

Create your own module

A module should have a clear definition of roles and responsibilities, both technical (what the module does) and organizational (who is responsible of defects found at this module). It should have also a clear and concise public interface. A module would usually be deployed as one unit, but in many cases several modules would be deployed as a unit together. Different programming languages would give modules different semantics, syntax and deployment characteristics, yet still some programming languages may not have the notion of modules in the language itself. And, not less important, a module shall be clearly defined as a set of tasks managed and built by an independent team of people.

How to do a 2000 pieces puzzle in a team



Doing a huge puzzle in a big team, does increasing the size of the team assist? Is there a number at which we will have a negative marginal contribution? ("Mythical Man Month"). Can you create "modules" here? This question about puzzle building is inspired by Chris Rupp's talk at JAOO 2008 (see video capture here).

It's a very nice exercise for a management course. I would think of the following design: (a) dispatcher team - goes through the pieces and assign them to the other teams; (b) teams by colors and shades, e.g. Dark Blue team would deal with the sea part while the Light Blue team would deal with the sky; (c) frame team, getting all the pieces that shall compose the frame. Each team is a module, so we have a dispatching module, a few color building modules, by colors and shades, and a frame building module.

In case of resource shortage we can settle without the dispatching team and let the building teams themselves do the dispatching. Number of people will define the separation into colors for the color building teams, but even if we would have 1,000 people we would not define 10 shades of blue, as the interaction between the teams (I got this blue piece but it seems unrelated to me) would be too exhausting.

Shall module definition be influenced by external factors?

We saw in the puzzle example that to some extent, number of available people would influence the number of shade teams that we may define. Expertise and skill set of our resources, as well as their geographical disposition shall also play a role here, but we will leave this for a later post on the relations between organizational structure and architecture.