I am exhausted today. I have been consumed by a tide of work that a team twice our size would not be able to competently manage. It started with a monster new feature that can not be quickly tested via our functional UI regression suite. It requires real end-to-end verification. Normally, we only need to create configurations which use features with the web application for configuration management. For this feature, we have to verify that the configuration is applied correctly once it goes active. This involved a journey of discovery that highlighted the insane lack of coordination and communication between our particular organization and the rest of the company at large.
First, we did not know how to do this end-to-end testing because we have never done it before. Second, we were under the mistaken impression that there was no apparatus for doing this kind of testing in our two test environments. We believed this because we have been told repeatedly that there was no such thing since we were hired. The entire development team and their managers also believed this to be true. A co-worker and I began trying to master this unknown art of end-to-end testing by contacting the team which develops and tests that part of the network. During this first meeting in which the two of us probably sounded as intelligent as brain-damaged turkeys riding the short bus to school, we discovered that both of our test environments did in fact have the networks for doing this end-to-end testing. Not only that, they have had them for FIFTEEN YEARS. It’s impossible to understand that our entire organization could share the delusion that this part of the system was not present in either test environment for so long. I still don’t have an explanation for it. The only thing I can say is that inter-group and inter-department communication is abominable.
We also discovered that another business unit was nearly done with building a system which allows them to go to a website, fill out a simple data entry form and press a button which spins up test VMs on demand that function as a nice little sandbox testing environment for testing for this part of the end-to-end pipeline in isolation. It is well on its way to being part of a continuous deployment testing apparatus where there is little to no human intervention required to spin it up and do tests. I am astonished that my entire organization seemed to be under the delusion that this was an impossible pipe dream. The sheer embarrassment of talking to one team after another about their testing which is so much better than ours and their systems knowledge, which is so much more comprehensive than ours, and their test automation which is so much better than ours has broken something in me. It didn’t help I calculated that it was going to take a month to design, write and execute these test cases. A month in which I would not be writing any code.
Also weighing on me is the task of writing all the functional test cases for the entire UI. There is a hiring freeze and getting new requisitions is akin to an act of god. We have two principal-level employees who want nothing to do with this task. And who would? It’s tedious and unrewarding work to write functional UI test cases. Since I have been tagged as the person who couldn’t get any automation done for three years, I figured it was better to bite the bullet and be the SQA lead even if I am not getting paid to do that role. I’m not even getting paid enough to do my current role. This decision was the genesis for an unpleasant discovery regarding my company’s IT support infrastructure, which is that there really isn’t one. You’d think that a company with 6000 employees would have gotten around to hiring a 24/7 team to support and administer things like the build system, the source code control system and the defect tracking and project management systems. But, sadly, this is not true.
Our JIRA installation started out as a rebel application running off the director of engineering’s desktop computer because a cadre of new employees refused to use Bugzilla. Naturally, because there was no formal introduction of JIRA as a tool, there were no knowledgeable professionals who controlled the adoption. The result is an explosion of project templates with a ton of totally unnecessary custom fields and overly complicated workflows. An organization full of supposedly amazing engineers should be able to design a system that cleanly separates the work of product design, development, testing and customer support, but what we got was a system full of templates and workflows that was designed by a drunken squirrel on acid. Eventually, it got too big to be a rebel application running on the director of engineering’s desktop computer. It did not get big enough, however, to merit a 24/7 dedicated support team. It had gotten big enough to have a part-time support team who were all doing other jobs before they got stuck with the responsibility for managing and administering this monstrosity.
This is where I entered the picture with my quixotic notions of quickly configuring three projects — one for the APIs I wrote for our test automation framework, one for the domain models I have written using the APIs, and one for the QA team to use for defining and managing work related to testing the customer configuration application. I had dreams of using these three projects to define and manage the workload of writing all the formal test cases for the application and the catalog of work that must be done to automate them. This seemingly simple task has been going on since early February. I am in the final stages of getting the correct workflow defined for the Zephyr test tickets. This customized workflow has been most painful part of the process. For everything else, I was able to use the default, out-of-the-box Jira workflow and fields or the agile-style workflow and fields that the team building the new front end uses. The Zephyr test cases were another matter. I wanted a workflow that would accommodate test case design and development for manual tests and automated tests.
I wanted a design phase followed by a review phase. Then, if the test case is to be automated, there is a development phase followed by a code review phase, followed by a testing phase. If manual, the ticket is ‘complete’ after review. If an automated test case successfully gets through testing, it is ‘complete’. I also wanted a workflow that could accommodate sending the ticket back for re-work if review, code review or testing revealed issues as well as a workflow that could handle flagging a test case for needed updates because of changes in the behavior of the application. Granted, this is not a simple workflow design and it didn’t help that I did not initially understand the JIRA terminology or means of representing a workflow. The other problem was that the number of custom issue statuses and fields had completely gotten out of control during the wild west period for our JIRA installation. Apparently, this can cause serious performance degradation in JIRA, so they were no longer allowing any new statuses to be defined. I was thankfully able to choose good statuses from the list of existing ones. That fact that there are about four different statuses signifying that something is ready for development or testing just shows how important it is for a company to get out in front of the problem of building IT support infrastructure instead of chasing after the cowboys who just get tired of waiting for a modern tool chain.
In the midst of this mess, I also took on the task of implementing localization support for my page object library. Our California team requested this feature before Christmas. There are eleven possible languages that a user can select when browsing our customer portal. It took 3 weeks of late evenings and working weekends to pull off locale-agnostic page objects. I also chose to do a major refactor of the DSL API I had written because it badly needed to be simplified and scaled down. I think the system works well. Now all the page objects refer to enumerated constants which represent these localized identities. There is some under the hood tooling that they use for mapping locale-sensitive identifiers to the enumerated constants. It was the best I could do given the short time frame for building it. It was a huge task that burned me completely out.
So, I looking at this job and I have come to the conclusion that nothing is going to change. I have been complaining about the fact that the traditional SQA work and the automation work need to be separate roles since I joined this company nearly four years ago. Nothing has changed. I’m still stuck with work I am not interested in doing and having to burn the candle at both ends to do the work I actually want to do. So, in short, I am looking for a new job as we speak. I finally realized that this company which is supposedly full of amazing engineers is not actually interested in applying solid engineering principles to the test automation. They aren’t even interested in applying solid SQA practices because they severely under-resource the function to the point of absurdity. I keep thinking back to my first real SQA job with a medical device company. The size and scope of the application was smaller, yet we had two managers, four people on the backend and six people on the front end. That’s half again the size of my current team. When I think about this, I realized how totally delusional my company is about the work burden and the complexity of not only testing these applications, but also building robust test automation for them.