It has been a terrible two weeks for me. I lost my last living grandparent and my workplace drama culminated in the latest episode of humiliating and ham-fisted attempts to micromanage my work because The People Who Matter can’t seem to grasp the reality that test automation is really really hard.
I am demoralized today. I have hesitated to write about the ‘people’ element involved in developing a robust automation framework because this tends to be volatile and subjective content, but my work week totally blew, so I am going to do what has become a time-honored tradition since The InterWubs became a mainstream pastime — I am going to bitch about it online and talk some shit. When I joined my current workplace about three years ago, the state of UI automation was less mature than it is now. There were a lot of teams who were just starting out and there were some teams who had more mature tooling.
At this time, a complete overhaul of a big system was also just beginning, which meant disparate teams would have to collaborate together in ways that had never occurred at this company before. There was a sense of urgency about getting this ‘Automation’ thing going because there was so much to test and manual regression testing is slow and expensive. The various teams of SQA engineers were trying to get a handle on the automation and some were succeeding, but many teams were not. I was newly hired and excited about finally getting a job where I could spend more time writing code than running the same old boring manual regression tests over and over again.
I was assigned to a complex web application that saddles many domains across the company. Both the UI and the data/business logic aspects of this application are complex. It’s the kind of meaty automation challenge that gets a weirdo nerd like me pumped up and ready to code. My boss had recruited a team of senior employees and we wrote a large suite of test cases for this application. Then we started to investigate how we could approach the automation. We decided to go with the Page Object Model design pattern that is the gold standard for Selenium UI test automation. We evaluated some other frameworks, but, surprisingly, none of them were actually using this design pattern, so we started writing our own framework. Although I am biased, I will state that the code was sophisticated and high-quality stuff that was up to the task of being a great API for writing automated tests against the rich and complex UI of the application we were responsible for.
Then along came a high-level ‘architect’ who decided that everyone had to use the same ‘framework’. I won’t go into the ridiculous lack of consensus about what the term ‘framework’ meant because it’s a distraction. Let’s just say, there was no consensus on what this term meant. He took a look at the some of the various tools that some teams had built which they called ‘frameworks’ and then decided to declare one of them the ‘winner’. Then he decreed that all teams must use the one he had declared the winner. On the surface, this isn’t necessarily a bad idea and if the person making the decision actually knew how to evaluate the candidate frameworks, this could have ended well. The losers would have felt slighted and upset, naturally, but technical people generally can be persuaded to accept standards and tools they didn’t originally choose or build for themselves if they are good.
The key ingredient in making the above scenario end well is the word ‘good’. In my opinion, the following things must be true for a test automation framework to be good:
- It should be built with industry standard tools, not someone’s pet bleeding edge thing that is new and sexy, but not mature and certainly not in widespread use
- It should be built with tools that you can easily recruit talent to use and extend, not tools which have few experts you have to struggle to find and hire
- It should be built with tools that your current employees are mostly familiar with
- It should be built with tools that are well-documented and well-supported
- It should be built with a well-designed and properly articulated architecture so that its components are maintainable, reusable, and scalable
- The separate components of the system should be cleanly separated and use APIs to communicate with each other
- There should be no hardened dependencies on specific test management or defect management systems
Now that we have this settled, let’s return to the high-level ‘architect’ who decided to require all the QA teams to use the same automation framework. A person without good programming or software design skills can handle the first four items on the list. The last three items need to be evaluated by someone who does. Guess which category homeboy belonged to.
The framework he declared the ‘winner’ was just Not. Very. Good. It was actually one of the frameworks my team evaluated when we were trying to figure out what we were going to do with our automation work. We quickly passed on this one because it was very poorly designed and had none of the characteristics we were looking for. It didn’t help that the person who wrote it clearly did not have much familiarity with the WebDriver API. When I investigated how other teams were using it after Sir Architect declared it the winner, I noticed that a lot of them included it as a dependency in their POMs, but didn’t actually use anything in it. I think this was mostly because the design was so bad that it was nearly impossible to understand unless you methodically walked through every line of code. I eventually did this myself, and I have to say, I am a stronger person for the experience.
My co-worker and I couldn’t bear to dump our system which worked great for us and adopt this other system which totally didn’t work and would require us to shoehorn some ugly-ass spaghetti code into our test code in order to ‘use’ a framework that did absolutely nothing for us. We went on writing our page objects and incrementally automating some steps in our suite of manual end-to-end tests. Meanwhile, my boss decided to become an individual contributor and a member of our team who never even wanted to manage was given responsibility for our whole team. She had recently been hired and had not even had time to settle in. After she took over, Sir Architect kept asking if we were doing our automation using The Chosen Framework.
I should stop here and explain that our new manager, unlike the one who decided to stop managing, was not someone who had strong programming and software design experience. So, she put a lot of pressure on us to deliver lots of real good automation real quick. The problem with people who don’t ‘do’ software is that they tend to think that test automation is easy and when they hear the word ‘framework’ they think, “Magic system which shits a pound of automation gold every hour with little to no effort and so little technical expertise required that even a preschooler could automate the whole internet with it.”
We tried to explain that in order to automate the regression suite for our application, we needed to build models for all the UI components so that we could interact with them programmatically. We also tried to explain that the data and business logic domain was also something that needed to be modeled so that we could define and feed test data into automated tests instead of hard-coding it right there in the test methods. We tried explaining that The Chosen Framework contained some badly designed mechanism for getting a reference to a WebDriver instance of some desired type and that was basically all there was. All the rest of the necessary tooling, namely the models, were not provided with The Chosen Framework. Not only that, the Chosen Framework required that we use a ‘base’ page object which seemed to be super confused about whether it was a generic page object or a misguided effort to wrap some basic WebDriver functionality in a way that was inferior to WebDriver’s own native interface.
It. Just. Did. Not. Sink. In. Part of the problem was that other teams had managed by some miracle to tease some automated tests out of The Chosen Framework. We couldn’t seem to make our manager or Sir Architect understand that these tests didn’t do very much and that they were for very simple applications with maybe a page or two with a very limited data domain. Our application is not that type of application. It has lots of dynamic UI elements, numerous dialogs, menus, and data entry forms along with some complicated multistep state changes that have to be tracked and verified along the way. Not only that are a lot of validation rules on the data that are in play as well as a complicated user and role access permission domain to make it even more fun. And then there is the collection of features and behavior related to migrating customer artifacts from the older application that our application is supposed to replace. I also cannot neglect to mention the oh so very special test environments which have been featured in earlier episodes of “Why You Don’t Have Good Automation”.
A subset of these complications should be enough to send any self-respecting automation engineer over the edge, but throw in the need to periodically explain your failure to deliver some sweet, sweet automation for this mess to people who just refuse to understand that a badly designed mechanism for getting a WebDriver instance is not going to result in Automation Jesus marching triumphantly down our cubical-lined aisles and you have yourself a heaping helping of Soul-Destroying Is This Really My Life Cognitive Dissonance. We had a test suite of some 40 manual tests with roughly 15-30 steps apiece that we were running on two versions of IE, the latest version of Firefox on Windows 7 and OSX, the latest version of Chrome on Windows 7 and OSX in addition to Safari. At one point, our manager just declared that if we could not complete the automation for this test suite in three weeks, we had to dump our work and switch over to The Chosen Framework. This was after saying time and again, that the Chosen Framework provided no interfaces for interacting with the UI, nor did it provide any efficiencies for building the interfaces.
Fast forward two years. My team falls back under my original boss who has decided to manage again. My team now includes the person who built The Chosen Framework who has been promoted to Principal SDET. It is announced that we will collaborate with some other QA engineers in our organization to built a new framework and the designer of The Chosen Framework, V1, will ‘architect’ it. We spend a year and a half building various components while our Principal SDET regularly becomes very concerned about the directory structure of the repositories and the very large amount of code which some people are writing because large amounts of code is ‘heavy’ and that is bad. Oh, and ‘the communication protocol will be a POM dependency’. Somehow we write something that some of can use and there is an effort under way by our upper management to shove our framework down the throats of every other business unit in the company. A framework which none of them helped build or design and does not include anything from their now mature and fully-featured frameworks that are mostly better than ours. You can imagine how fast they all are running from the directive to adopt The Chosen Framework, V2.
I have finally been given space and time to do the automation after two years of relentless and never-ending manual testing stints all the while hearing, “Where is all that good automation?!!!! We can’t afford to hire manual testers. Just do it in your spare time!” Now I have about 200 automated tests of which only a couple are flappers which need some tweaking to make stable. Nevertheless, the new QA director has decided that we need to ‘start making some progress’ and a close working relationship and some coding ‘help’ from the creator of The Chosen Framework, V1, is just the solution to this problem.
It almost made me want to quit and take up goat-herding as a profession after pouring acid over my laptop and taking a shit right on top of my workstation out of spite.
So, folks, this is the latest edition of “Why You Don’t Have Good Automation.” Automation frameworks are a serious technical challenge and you need to put technically sophisticated people in charge of it. Also, don’t try to shove a tool down the throats of people who know better than you do what they need. If you’re looking to have your entire company on a single framework, you need to involve all the teams in requirements gathering. You also need to incorporate the best of their tooling into it. Also, if you don’t want to drive your automation staff into the arms of your competitors, don’t keep asking why they aren’t done every two days.