Is automated testing one of your pain points?  Have you ever wondered how to get from where you are to where you need to be?  Do you lie awake at night wondering “Why am I paying for that automation team and what am I getting out of it”?  If any of these have crossed your mind, then you need to come see where we’ve been, where we are now, and where we’re going.   We will share our journey along organization, technology, and process.   You will walk away with some concrete advice that you can apply to your own efforts. At Intuit, we have a long history of pursuing test automation.   We started with a centralized automation team, utilizing “click and record” methods, and an ad hoc process that could best be described as “automate anything you can”.  The results we obtained were, in hindsight, predictable.  We ended up with a large number of automated tests that were fragile, difficult to run, and expensive to maintain.  On top of that, we had no idea what the coverage level of the tests was.  While the benefits in terms of testing and ROI were unclear, we would still characterize this stage in our evolution as the “Renaissance”.  We realized that we needed to invest in automation because the manual testing emphasis of the past was inefficient and didn’t scale.

Our next stage of evolution, the “Enlightenment”, we recognized that our automation efforts wouldn’t scale. Without a change in organization, technology, and process, we were doomed to failure and regression to the “Dark Ages” (i.e. a total reliance on manual testing).  During this stage, we focused our efforts in two distinct areas:  infrastructure and process.   We decentralized the test ownership and made huge investments in reducing false negatives.  We enabled developers to run tests against their personal builds. And, we put focused efforts on improving the automation infrastructure.  These actions resulted in major advances in the mindset of the organization – automation, while still fragile, was considered a critical component of the process.  Decision-makers waited for automation results before declaring confidence.  Some developers wouldn’t check in their code unless they could run and pass the automated tests on their personal builds. At this point, we would consider ourselves in the “Industrial Revolution” stage.  We have successfully moved to a centralized infrastructure team with federated testing.  We have moved from an attitude of “blame the test first” for failures to “what went wrong with the code?”  In the last year, we have gone from 50% of the automated tests requiring triage down to nearly 0%.  We are running large automated test suites against multiple code branches on a daily basis.  We know what the tests cover and use code metrics to add coverage in the most important areas.  Developers write unit tests that integrate with our automation system and that become part of the testing suites.  Shipping a product or even promoting a code branch is considered inconceivable without running comprehensive automated tests with flawless results. 

What are our results so far? 
We have decreased our testing cycles, increased efficiency, and increased developer productivity.  We’ve matured from an organization where our QA team executed tests over and over to one where they write new tests and actively work with the developers to build quality in.  We have happier engineers, less rework, more agility to market, higher quality, and scalable development teams.  And, we have helped Intuit become America’s most admired software company and a Great Place To Work©.Where are we going next?  What does the “Post-Industrial Revolution” look like?  Come and attend our talk and we're happy to share! 

---