Welcome
How Does Your Organization’s Testing Stack Up?
With increases in product complexity, testing can’t keep pace. Based on our study, we found that 50% or more of projects and designs are either complex, multilayer systems or integrated systems, yet only 16% of testing processes use AI for integrating complex test suites.
Our short quiz assesses how mature your organization’s approach to testing is. Are you a Leader, Transitioner, or Late adopter? How do you compare to your peers and what recommendations do we have based on your responses?
Questions
What software test approach does your company use today? (Select one.)
Questions
What method does your company use for analytics today (as part of your testing process)? (Select one.)
Questions
How do you identify and resolve defects, bugs, and issues as part of your validation and testing process? (Select all that apply.)
Questions
What level of coverage would you say your test routines include for each of the following? (Select one per row.)
Results Overview
To strike the balance between perfect testing and the business realities of increasingly complex product suites, limited resources, and the pressure of time-to-market, our assessment evaluated participants across four key disciplines:
Current Testing Approach
Is your organization’s testing approach manual, automated, or a mix?
Current Analytics Approach
Is your organization conducting basic data aggregation? Or is it using AI for complex test suites?
Issue Identification And Resolution
How is your organization identifying and resolving defects, bugs, and other issues?
Test Routine Coverage
Is your organization conducting too many tests? Too few tests? Or just enough?
How does your company compare to your peers? What can you do to mature your testing approach in the future? Register to see your personal results and recommendations.
Recommendations
Late Adopter
Transitioner
Leader
Your maturity result:
You Scored As A Late Adopter
Your testing maturity is in the beginning stages, which is behind most of your peers.
- Your company has a highly manual approach to testing.
- Your data aggregation and analysis systems are in the beginning stages. Either you do not aggregate the data at all, or you are overwhelmed with large amounts of disconnected data and are unable to derive key insights for decision-making.
- You are either significantly over or under testing, and you are not using all the test processes you should to identify and resolve issues.
Findings And Comparisons
You need to transition your organization’s testing strategy to include more automation and AI. While this is a journey, the time to take first steps is now, as most of your peers are already well on their way.
- Over the next three years, only 37% of Late Adopters plan to have at least more than half of their testing automated, compared to 95% of Leaders. Reinforce the cost savings of automation with your leadership teams. Automation doesn’t just reduce overall testing costs; it also frees up resources to focus on other tasks. Currently, 55% of your fellow Late Adopters report that the challenges they face with their current testing systems cause increased expenses for the organization.
- You are likely overwhelmed with data that you are not able to use. In fact, 56% of Late Adopters collect and archive data for each test, rather than using it to make and execute data-driven decisions. AI can help integrate your testing results. Using intelligent, integrated testing provides better (but not necessarily more) testing coverage as well as the ability to support agile product development and release.
Recommendations And Next Steps
Your team should set goals of testing “the right amount” — as opposed to over or under testing.
- Fight test creep. Ninety-one percent of Late Adopters report that the length of time to test is now somewhat or much more than the past couple of years given new complexities that have been introduced. Ninety-two percent also say the number of tests is somewhat or much more than just a few years ago.
- Question you testing status quo. Even though Late Adopters face mostly manual testing and are significantly over or under testing, 77% of Late Adopters (compared to just 39% of Leaders) claim to be very satisfied with their current testing system. Ignorance is not bliss! Your testing system needs an overhaul as you work towards getting your tests more automated and the amount of testing you do in the “just right” spectrum.
You Scored As A Transitioner
Your score means your testing maturity is climbing, and it is on par with many of your peers. However, you still have significant strides to make to reach maturity.
- Your company has an approach to testing that is partially manual and partially automated.
- Your data aggregation and analysis systems are transitioning. Your organization uses aggregated data for analytics and reporting, but it has neither fully integrated complex test suites nor has it accomplished predictive analytics with the data.
- You are either slightly over or under testing, and you are not using all of the test processes you should to identify and resolve issues.
Findings And Comparisons
You need to transition your testing strategy to include full automation and AI over the coming years. While this is a journey that you have likely been on for some time, there are additional steps you can take to take to catch up with Leaders:
- Over the next three years, only 77% of Transitioners plan to have more than half of their testing automated, compared to 95% of Leaders. It’s worth noting that this is improved over the mere 37% of Late Adopters!
- Reinforce the cost savings of automation with your leadership teams. Automation doesn’t just reduce overall testing costs, but it also frees up resources to focus on other tasks. Currently, 45% of your fellow Transitioners report that the challenges they face with their current testing systems cause increased expenses for the organization.
- Increased AI can help integrate your testing results. While you are likely ahead of your Late Adopter peers who are mostly just collecting and archiving data for each test, you have strides to make before reaching the fully mature position of making and executing data-driven decisions that continuously improve testing and products. Using intelligent, integrated testing provides better (but not necessarily more) testing coverage and the ability to support agile product development and release.
Recommendations And Next Steps
Your team should set goals of testing “the right amount” — as opposed to over or under testing:
- Fight test creep. Sixty-seven percent of Transitioners report that the length of time to test is now somewhat or much more than the past couple of years, given new complexities that have been introduced. Seventy-six percent also say that the number of tests has increased somewhat or much more than just a few years ago.
- Question your testing status quo. Even though Transitioners face somewhat manual testing and are over or under testing, 87% of Transitioners are moderately or very satisfied with their current testing system. Your testing system should be working with you, not against you, as you aim to get your tests fully automated and the amount of testing you do in the “just right” spectrum
You Scored As A Leader
Your score means your testing maturity is ahead of your peers:
- Your company has a mostly or fully automated approach to testing.
- Your data aggregation and analysis systems are in the mature stages. You use your aggregated data for predictive analytics and integration with complex test suites.
- You are most likely testing the right amount, i.e., not testing too much nor too little, and you are using most or all of the test processes you should to identify and resolve issues.
Findings And Comparisons
Your mature testing strategy either includes full automation or full automation and AI combined. While this is a journey that you have likely been on for some time, there are some additional steps you can take to ensure your organization remains in its leadership position:
- Over the next three years, 95% of Leaders plan to be mostly or fully automated. This is significantly improved over the mere 37% of Late Adopters, but they are working hard to catch up. Periodically review the state of your automation technology platform.
- Continue to emphasize to leadership the business benefits of automation cost savings and the flexibility it provides. Automation doesn’t just reduce overall testing costs; it also frees up resources to focus on other tasks. Constantly use your data to reinforce this business decision to make an ever-evolving business case for maintaining and expanding AI and automation.
- Increased AI can help constantly integrate your testing results as landscapes change. While many of your peers are just collecting and archiving data for each test (i.e., letting it sit on a shelf), you have made strides to make and execute data-driven decisions that continuously improve testing and products. Increased AI can help constantly integrate your testing results as landscapes change. Maturity in this space is dependent on your organization’s ability to remain agile. Using intelligent, integrated testing provides better (but not necessarily more) testing coverage and the ability to support agile product development and release.
Recommendations And Next Steps
Your team should set goals of testing “the right amount” — as opposed to over or under testing:
- This is no time to rest on your laurels. The landscape is constantly changing and evolving, and new complexities are constantly being introduced. Only the most agile organizations will continue to adapt to these changes.
- Fight test creep. Even though many Leaders are not significantly over or under testing, 36% of Leaders still report that the length of time to test is now somewhat or much more longer than it was in the past couple of years, given new complexities that have been introduced. Sixty-two percent also say the number of tests is somewhat or much more than just a few years ago. This demonstrates that there is no time to rest on your laurels. The landscape is constantly changing and evolving, and new complexities are constantly being introduced. Only the most agile organizations will continue to adapt to these changes.
- Question your testing status quo. Eighty-seven percent of Leaders are moderately or very satisfied with their current testing system. Continue to constantly evaluate your testing system to ensure that it is working with you, not against you, as you aim to keep your tests fully automated and the amount of testing you do in the “just right” spectrum.
Next Steps
Read the research
Thank you for taking the time to complete this assessment. Click here to read the full Forrester report commissioned by Keysight.
Ready to get started?
Discover AI and Software Test Automation
Ready to explore the Eggplant platform and see how it works?
Try AI assisted software testing to ensure rapid release of your applications
View your detailed results
Methodology And Disclaimer
Methodology And Disclaimers
Methodology
Methodology
In this study, Forrester conducted an online survey of 406 test operations decision-makers at organizations in North America, EMEA, and APAC to evaluate current testing capabilities and to hear their thoughts on investing in automation (including AI). Questions provided to the participants asked about their organizations’ current testing environments, future investments, challenges, and expected outcomes from testing automation. The study was completed in December 2021.
Disclaimers
Although great care has been taken to ensure the accuracy and completeness of this assessment, Keysight and Forrester are unable to accept any legal responsibility for any actions taken on the basis of the information contained herein.