Test Automation ROI

Recently I was working on a ROI (Return-on-Investment) Model for Test Automation. It was quite interesting to note that there are not too many "ready-made-excels" out there for this purpose. The ones that exist come mainly from automation tool manufectures. So I went to creating my own model, resulting with interesting notions.

Test Automation ROI Ingredients

Expenditure side

1. NRE (Non-Recurring Effort / Expense) - buying tool license, training people, raising the environemnt and infrastructure
2. Implementing the initial testing scenarios
3. Maintaining the testing scenarios and the environemnt and infrastructure
4. Adding new scenarios and new required features to the environemnt and infrastructure
5. Usually requires better trained testing engineers to maintain the automated testing system

Revenue side

1. In the long run, may be able to reduce manpower (less manual testing is needed)
2. Reduce number of escaped defects
3. Increase development confidence, allowing more features per release, thus less releases per year
4. Shorter test and development cycles and improved TTM (Time-to-Market)
5. Enhances the abilities of the testing team ("soft" revenue)

Influencing Factors

When coming to real life numbers it appears that it is very tricky to achieve positive ROI. And even if there is positive ROI the break even point comes after at least one year of investment, and in most cases two years. It's a simple case of big lump sum investment for future expected revenues.

The ROI is strongly related to the amount of bugs detected in the system in general and specifically in the field, number of existing manual testing engineers and the number of releases per year:

• A system with very low amount of bugs that are mostly found at testing room and not in the field, would make it harder for automation to have positive ROI, as the potential of reducing escaped defects is low
• If the current existing number of manual testing engineers is small, potential of revenues in reducing manpower is low. However, if number of escaped defects is high automation can assist in lowering it
• Low number of releases per year would also make it harder to achieve positive ROI, as the lump sum investment is made for less cycles of use per year

The ROI is strongly related to the changes in the system:

• Need for new scenarios and features in the environemt along the year means more investment in maintaing the testing system up-to-date
• The lifetime of an automated scenario before it becomes broken or irrelevant because of changes in the system is very important

On an "average" system with some major new features each year, about 4 major releases per year and about 20 major escaped defects per year, following can be found:

• In order to properly support and maintain a test automation project, you would probably need the same amount of people as you had before with manual testing, all along. The new features in your product requires new testing scenarios as well as new features in the test automation environment
• The major revenues stem from reducing number of major releases (adding more features per release) and from reduced number of defects in field
• It takes about 18 months to return the investment

Projects that are not stabilized yet and have too many changes would probably not earn from end-to-end test automation, although automated regression based on UnitTesting can still assist.