I’ve recently come across a situation where the expectations for automation wasn’t very clearly defined/understood, and it made me wonder how we had arrived at that place. These were pretty bright folks, and that they didn’t truly know what the purpose of automation was, and what benefits it could provide sort of stunned me. When I took a moment to step back and think about it though, it kind of made sense. Since I am sure there are others asking the same question, or are just wondering how to effectively communicate this, here is my breakdown of what automation is (and isn’t).
Automation at it’s very core should do one of these two simple things; it should either save you time, or it should save you money, or, ideally, it does both. Notice I did not say it should make the Dev’s or QA folks lives easier…this is more often than not a pleasant side-benefit, but it should not be why you are automating. If that is the only outcome, then you need to re-think why you are automating. The cost of automation is very expensive, and if it is not saving you time and/or money, then you should push the big pause button until you figure out what you really are trying to accomplish.
Here is a scenario that I recently had to decide whether to continue automating. We have a suite of tests that a certain team needs to run on a regular basis, which seems like an obvious candidate for automation. When pulling together the details, it was determined that this set of scripts would be run every 2 weeks, and that it would be necessary to run those scripts for ~2 years. On the surface, this seems like an ideal automation candidate, but a closer look reveals the ugly side of automation. Each script in this package takes approximately 24 hours (roughly 3 days) to automate (on average, some are less, others more), and on average it took 15 minutes per script to run manually. That means in order to get the invested time of script creation back, a given script would have to run 96 times before any investment would start to pay off. If the script is only slated to run every 2 weeks for 2 years, that is only 52 execution times, so roughly half the amount needed. Even if the script was ran every week, that would mean that by the time you started getting a return on your investment, it would be time to shut those scripts off. In addition to this, you also have to take into account the maintenance of a given script…over the life of a script, that may end up being as much as your initial investment! We got a lot of pushback when we stated that we would not be automating any additional things for this specific team, as it really cut down on their time when doing their testing. What they couldn’t see was that it actually was costing us money to create these scripts over their lifespan, because we were never able to break even by the time the scripts were outdated and deprecated from the suites. Yes, we were making their life easier now, but we were paying for it pretty heavily.
Taking the time to figure out exactly what you are trying to accomplish with your automation initiative is so incredibly important, because without that knowledge, it is very easy to fall into this trap. Automation should be used as a tool, and like any other tool, it is better at some tasks than it is at others.
- Automating workflows
- Validating core functionality
- Creating repetitive content
Not Good For
- Code quality
- Build validity
- Visual inspection (can’t replace eyeballs)
Obviously, these are very broad strokes, and there can certainly be other things that fit into each of these, but this will give you a good idea of what to expect. Here’s the other thing you should consider…every company and product is different. What may be a great automation candidate in one place, may not be something you would do at another place. It pays to do your homework before you jump into the automation waters.