Project Context
The Health Insurance Marketplace relies on a complex array of software that must be updated, integrated, and thoroughly tested each year prior to Open Enrollment. This occurs in an ever-changing regulatory environment, yet leaves little room for error when validating the massive amount of code in an extremely tight timeframe. Over 12,000 functional tests and 25,000 regression tests, along with many other test types, must be conducted. Since the testing needs far exceed the time available using traditional testing methods, there is a critical need to find more efficient and effective approaches. Most importantly, this testing must deliver maximum effectiveness, as the nation's entire Open Enrollment depends on how accurately and completely the Marketplace systems are validated.
Challenge
Modernization of Client’s Existing/New systems.
The Healthcare agency provides critical healthcare options and coverage to 21 million consumers through a complex eco-system of information systems. The Agency needs the capability of quickly and accurately testing and ensuring the proper functions and integrity of all interacting systems during their various stages of evolvements and upgrades. The time taken to discover bugs and defects was slow due to the time required to execute manual tests.
Solution
Automation was implemented to assist with essential process testing, aiming to decrease the time required for results reporting. By embracing the "Automation-First" strategy and fostering a culture around it, the Team encouraged everyone to find and pursue opportunities for automation and optimization. Utilizing our understanding of the ecosystem we are tasked with testing and validating, we created an innovative tool for automated regression testing to reduce testing durations and eliminate human errors.
Key Results & Positive Outcomes
CMS avoided $2.9 million in costs by implementing an innovative automated testing environment using Node.js and the Codecept.js library.
This resulted in a 50-fold increase in test execution speed compared to legacy automation, and a 100-fold increase compared to manual testing.
Most notably, testing effectiveness rose from 85% to over 95% of defects found in independent testing versus production, all within one contract year.
Additionally, by freeing up tester time, test coverage increased by 21%.
The high-performance Node.js and Codecept.js based automation continues to enable significant improvements in speed, effectiveness, and coverage for CMS systems validation.