I got a bit sidetracked with the Brixen posts that I forgot to spend some more time explaining Why You Don’t Have Good Automation.
I am tired today. I spent the day dividing my time between way too many tasks, one of them boring as hell, but which should make it possible to dynamically put together a test run with Zephyr for Jira using keywords. The keywords correspond to group names on the automated tests for each test case. It’s a giant pain in the ass to add all these labels to the Zephyr test case and then add them as group names to the test methods, but it’s necessary to make what I want to do possible.
In addition to this mind-numbing data entry horror, I also spent an hour or two trying to figure out how to run these automated tests on the production system. I have never set up automation for the production system. My suite runs every night on whichever of our two test systems that isn’t currently crapping its pants and painting the walls with its own feces. I have some configuration files containing a list of test users and test accounts and I have a neat little set up routine that pulls this test data out of the config files and delivers it to tests running in parallel so that no single user login is used by more than one thread at a time. Behavior on our awful test environments gets even more awful and unpredictable if a user is logged in from multiple browser instances.
I still get random errors resulting from a user being logged in more than once because I am not the only person running tests on the test system. There are hundreds of people who use these environments and everyone has access to the same user accounts that I do. So, shit happens. Shit happens a lot less since I set up my tests to ensure that at least they don’t inadvertently use the same login credentials, but I don’t control the accounts that other engineers are using for their tests.
But I digress. The production system is another can of worms. I need a way to store the login credentials and deliver them to the tests in a secure way. Storing it all in a plain text configuration file and checking it into source control is a pile of bad idea topped with steaming fresh dog turds. It would violate PCI security standards six ways from Sunday. I’m getting some help from the engineer who wrote the original and now retired automation suite that used to run on production. He had a database running on a managed VM that stored the credentials and used a config file to specify the database access credentials. The config file was actually a stub because checking the DB credentials into source control in a plain text file is no better than checking in the login credentials themselves. A human being had to check out the repo and add the DB credentials to the config file. Then they could trigger the tests which used the DB credentials from the config file to log into the database to retrieve the user login credentials.
I found out that I can’t use the same strategy because a DB running on a managed VM isn’t kosher anymore due to PCI security standards. I have to find some other way to deliver the production login credentials for automated tests to use. Did I mention that I am not an expert in security standards? And there is no documented process for dealing with this problem? A few months ago, our entire organization was tortured with a giant PCI training effort in which we had to watch lots of videos and take lots of quizzes to prove we had been trained in PCI security awareness. I guess developing procedures and systems for dealing with practical, real-life problems just like this one and training us in that was too much to ask for.
Another part of my workday was devoted to speaking to an account executive at Sauce Labs in order to get a cost estimate for their services for the four teams in my particular mini-organization. I think he was excited up until he found out that I had no budget power and that I actually don’t know what the management class here is willing to spend. I was handed the task for getting an estimate for using them in an offhand way in the middle of a meeting that was supposed to be about something else two weeks ago.
This Sauce Labs POC task has been a hot potato that got passed around for the last year. A couple managers had this task sitting on their plates for weeks and every week during our status meetings, it just kept getting pushed off. Eventually, we stopped talking about it. Then some engineers on one of our teams actually did a trial with them and did a great presentation on the experience. I was excited because I thought something was actually going to happen at that point, but then I never heard another word about it until I said we needed Sauce Labs in a meeting two weeks ago because I was pissed at the lack of infrastructure for Selenium test suites. Suddenly the task of doing the same thing that has already been done by others was given to me. So, I am doing what others already did, which sadly, will lead to the the same likely result. Nothing will happen and a year from now, some other over-worked person will be told to do it again.
So, you’re wondering, “What does this have to do with why I don’t have some of that real good automation?!” One thing I have learned in all my years of working in quality assurance is that ‘quality’ is not a thing. It’s actually a living system made up of processes, people and tools. It’s also a ungodly complex system in a large organization. Automation is only one piece of it. And unfortunately for you, it’s a piece that depends largely on the smooth functioning of almost every other piece of the system. How many times have you acknowledged that there is a problem somewhere in your system that relates to quality that resulted in thousands of hours of lost productivity? Shitty test systems? Shitty or non-existent documentation? Shitty or non-existent approach to generating and managing test data? Shitty, fractured approach to tools and infrastructure procurement and maintenance?
If your approach to any or all of these deficits has been to click your heels and wish for your own personal Automation Savior to swoop in and drop a load of some real good automation on you like Rapunzel spinning gold out of straw, I would like to point out that you need to provide the straw first. Providing the straw involves the procurement of arable land, seeds, fertilizer and labor to grow and harvest it, along with a transportation network to get it you in good condition. Because Rapunzel ain’t gonna spin no gold out of moldy, wet, rotting straw with rat droppings all over it. ‘Kay?