Test automation is a key focus area of every IT organization’s application ecosystem strategy. Who wouldn’t want top-notch quality coupled with blazing-fast test cycle times, reduced overall efforts and the added bonus of eliminating human error? For an organization just starting its test-automation journey, it seems like a magical, fool-proof strategy guaranteed to be successful and reap quick benefits. It’s just an incremental version of basic manual functional testing. What could go wrong?
Ask any organization knee-deep into their enterprise-wide test automation journey, and they will tell you the reality on the ground is very different. Test automation is like an iceberg. Most organizations only account for the part visible above the surface. They seem to be either delaying when they’ll address the part below the surface, or perhaps they’re unaware of it entirely. Of the many reasons for a disappointing test-automation journey, the most common and important is believing that simply investing in a tool or platform will take an organization to the promised land. Test automation is a multi-faceted journey. Although the tool or the platform is a key component, how companies use it to traverse this journey is equally important. Test automation is a long process that requires a patient, multiyear approach to see benefits across the enterprise.
To better understand and appreciate this journey, we have gamified it by comparing it to the “Snakes and Ladders” board game. Below is the correlation we have considered:
- The board in its entirety is the automation journey, peppered with snakes (pitfalls) and ladders (best practices).
- Each square/block on the board is an important step in a company’s journey toward its test-automation goal
- Ladders are key steps companies should take to see nonlinear benefits in the future
- Snakes are the roadblocks companies encounter during the journey that have the potential to bring them back to square one due to shortcuts taken or key steps overlooked (or unknown)
For a successful and seamless automation journey, companies should consider some of the key aspects below while blueprinting their strategy. There are snakes to avoid and ladders to help in each.
Automation Platform Costs
- Snake to avoid: Organizations often seem to take a “stone age” approach to factoring tool costs into their strategy. While they consider the high initial cost of a superior automation platform to be a roadblock, they may not have carried out a comprehensive Business Value Assessment to address the ROI and benefits over time. From our experience, organizations tend to move toward creating a framework of their own that integrates with other automation tools that seem more cost-effective (e.g. open-source or commercial). Unfortunately, the high degree of integration and varied protocols make these frameworks extremely brittle, and they tend to crash frequently, making automation more of a hindrance than a blessing.
- Ladder to help climb the board faster: Rather than building a framework from scratch using highly skilled talent – which can be difficult to find in today’s competitive markets anyway – companies should focus on end-to-end technology-compatible automation platforms that augment the existing manual test engineers. This saves on investments in training and recruiting an entirely new skillset. Additionally, this makes automation easy to adopt enterprise-wide, whether it be custom web applications, commercial off-the-shelf (COTS) products or cloud-native applications. The ROI in these scenarios is easier to realize, and automation is perceived more as a boon than a necessary evil.
Script Maintenance and Related Overheads
- Snake to avoid: Consider desktop and mobile web apps, the most-common test-automation tools for which are often open-source in nature. These tools can bring down costs considerably, but they have drawbacks that can severely impact ROI and schedule adherence. Dynamic webpages, for example, cause many issues while using open-source tools, because the backlog of scripts that must be developed and maintained keeps increasing, which leads to significant testing efforts. Eventually, companies will hit a roadblock and realize that the scripting efforts are unable to reduce the overall testing efforts.
- Ladder to help climb the board faster: A comprehensive tool fitment analysis framework is a good start to any automation journey. If we continue the previous example of dynamic webpages in agile or pseudo-agile projects, the tool’s maintenance efforts are a critical factor to be considered during ROI calculations. If traditional open-source or COTS tools prove inefficient for such scenarios, companies should explore AI/intelligence-driven platforms that can update their object repositories automatically, thereby eliminating the need to maintain scripts at all.
Optimum Level of Automation Coverage. 50%? 75%? 100%?
- Snake to avoid: Quite simply, 100% automation is NOT an organization’s nirvana state. Many organizations believe that, “If I my automation coverage is 60%, I am still 40% inefficient!” and move toward a scenario in which they have automated almost every test case. This decision is not wise. As organizations move toward extremely high levels of automation, there is a linear increase in money spent on tool licenses, as well as the people needed to carry out scripting. For example, one client spent a large amount of effort and money to move from 70% to 90% automation and maintain that level. Ironically, the client spent more on that 20% than it had spent going from 0% to 70% automation in the first place, eventually realizing a negative ROI. Higher levels of automation may not always result in positive ROI. It is okay to have low automation numbers as long as the ROI continues to increase and the company achieves its maximum coverage percentage.
- Ladder to help climb the board faster: Dynamically tracking ROI and technical debt, coupled with techniques such as impact-based testing, are keys to avoid over-automating test suites. Businesses should always remember Brooks’s law and the law of diminishing returns when investing in an IT-related activity. Another dimension organizations should explore is identifying the automatable test-suite set in the first place to enhance coverage. Using historical analyses, the target number of artefacts to be automated can be further reduced, thereby enabling a high degree of automation at a much lower effort and cost than before. There are also algorithms available that can perform impact assessment by analyzing code-level coverage and identifying new changes. This intelligence can help with smarter automation coverage, targeting only the impacted code module-based test cases and providing high confidence without compromising risk coverage.
Test automation is a multifaceted journey that organizations must tackle diligently while remaining mindful of long-term benefits, as well as the future technology roadmaps of IT and the business. One benefit of the test automation board above is that there’s no rolling of the dice required to advance. Companies are empowered to pick the number every turn, and the smaller the number the better.
