Category: Automation

Treading Water Is Too Much Work

I honestly meant to post about the daily insanity of my workplace about a dozen times over the last two months. The only problem was that I was working all the time. Something amazing happens every time I step off the Endless Manual Testing Train To Perdition. I always have a giant pile of technical debt to address and I have to work overtime for weeks to catch up on the deferred maintenance. The current task on the pile is implementing an interface for a REST API to query the customer configuration management application for configurations meeting necessary criteria for automated tests. I didn’t want to use the UI to search for these configurations because the test systems are still a steaming pile of crap. Using the UI introduces a ton of new opportunities for their horrible performance to blow a test execution run to smithereens. Plus, these tests already take long enough to run that any means I have at my disposal to optimize the execution time should be put to work.

The API belongs to an ecosystem of APIs the company developed for programmatically interacting with our services and systems. There is a tool available in the The Chosen Framework v2.0 for executing requests against these APIs, but the design is not to my liking. Given the political shit storm that always surrounds any attempt to alter the design or function of any of the services in The Chosen Framework v2.0 and the total lack of staffing to support and maintain the framework, I decided to secretly write my own tool. There is an overly complex authentication system for using any of these APIs, so I had to build out my model for the access control group and user admin business domain to include the credentials needed for authentication. I have a penchant for wanting all my test data to live outside my test suite source code, so everything needed to be serializable/deserializable from both XML and JSON because who am I to dictate which storage format a person should use in order to employ my tools for their own purposes?

One thing lead to another, and suddenly, I had 6 different repositories for tools I have written to model parts of the business domain and as well as services I built on top of those models to automate interactions with the products I test. Not to mention that the number is set to grow because I have plans for additional services for even more automation goodness. In short, this collection of tools and services has become a total beast. It is really an automation framework in its own right. I finally managed to get a Stash project dedicated to them, so I moved everything there. As soon as I did this, I started to get nervous about getting caught for not using the services in The Chosen Framework v2.0. QA Director has been on the warpath against people who don’t want to use the services in the The Chosen Framework v2.0. I believe that the failed effort to shove it down the throats of other business units in the company is a sour and somewhat embarrassing memory for him, especially because several people on our team have turned their own noses up at various tools in framework. I can only imagine what the reaction will be if I get caught for building a rogue framework.

I confessed to my boss today that I was guilty of coding without a license. He asked me why I didn’t use the existing tool and I explained my reasons, and he said I should just add my tool to The Chosen Framework v2.0. This meant I had to explain that this would mean adding other tools to it, including the business objects and page objects that I had been asked to remove last year. He seemed confused and said he would address the issue in Some Very Important Discussions taking place next week among QA Director, the other managers and Principal SDET regarding the future of The Chosen Framework v2.0. I don’t think he realizes yet what a clusterfuck this whole thing has been. I would prefer to have nothing to do with The Chosen Framework v2.0 and continue building out my secret framework, but I know this is never going to happen.

The summer was a busy affair. It started out with a massive fail by the new organization that has taken over support for development tools and infrastructure. They officially took over maintenance of Stash in June and immediately buggered system the first time they did routine maintenance on it. For some reason, the most recent backup available was a month old. Thankfully, I had the most recent version of all my work saved locally, but some teams lost work. I am very happy to see that we are getting something that looks a lot like a real IT support staff, but these handoffs are a bitch. At the very least, before doing anything to the source control system, one should make a backup and then make a couple copies of that backup and store each of them in different locations. They handled the recovery process well, though, with good communication and organization of the effort to glue Humpty Dumpty together again.

As part of my job duties over the summer, I helped Junior SDET Who Should Be Senior mentor our summer intern from MIT. This was one of the most pleasant and enjoyable experiences I’ve had. It’s a lovely experience to work with one of the smart ones who learns quickly and is able to extrapolate abstract principles from a few basic examples and work independently. I highly recommend hiring MIT students as interns. I think a lot of them could easily drop out of school and start working as developers or even start their own companies and be wildly successful.

QA Director has been on everyone’s case about work from home policies and has lately taken up the cause of monitoring what we are doing on our computers at any given time. At one point, he asked Junior SDET Who Should Be Senior to talk to our intern about not playing games or watching any videos while he is in the office. QA Director claimed that ‘someone’ had complained to him about it, but I wouldn’t put it past him to lie about that so that he doesn’t come off as the ‘bad guy’ with a bad case of Hall Monitor Syndrome. I personally think that people who are productive and hit their milestones should be left alone, but I’m not the boss. Right after this, he claimed that ‘someone’ complained about the whole QA team being absent from the office on the first Friday after a giant release date. This was first and foremost, incorrect because I was in the office that day. I’m just too short for anyone to see my head over the top of my cube. Second, several people took vacation after that horrific release cycle, one of us couldn’t work that day because their new H1B visa wouldn’t be in effect until the following Monday, another of us was out sick and the rest were working from home on Friday because that was the day they had chosen for their one day a week allowance.

This of course, ignited a tense and difficult team meeting in which many of us expressed anger and resentment because developers seem to do whatever they want when they want and we are the only team that ever gets called to account for our whereabouts. QA Director became very defensive and sort of pissed at us and then he said he was going to tell the person who complained to mind their own business. Which is what he should have done in the first place. There’s nothing more demoralizing than a boss or director who won’t stand up for their team. Our team manager seems to be getting really tired of all this and has said he is not going to get into the habit of taking attendance because we are all grown adults and he would prefer to focus on the work and whether it is getting done or not. Although I am really pissed that my former boss was fired, I like the new manager a lot. He’s focused on the work and getting the developers to work together with us as a team, which is nice.

I found a senior developer role elsewhere in the company that is with one of the business units that is turning a profit. I have an informational interview with the hiring manager next week. Hopefully, this will be the right fit because it would be really great if I could continue working with the company in a part of it that is actually making some money. I really like the technology here and working with a core piece of it would be a great opportunity for me to gain some new experience and skills. I am still working on acquiring the knowledge and skills to get into data science or machine learning. I underestimated the amount of time that would take. There’s a lot of math. SO MUCH MATH. I built an Anki flashcard deck for the MIT discrete class I studied through MIT OCW. After finishing up the book and the lectures, I started using it study the material. For the first two weeks, it felt like it wasn’t doing anything for me, but now I have started to see cards which are easy for me to answer. The other benefit is that there are many proofs that I ‘know’, but understand more fully as a result of repeatedly reviewing them. Our summer intern is taking the class this fall, so I shared the deck publicly, and it appears to have taken off like wildfire because it has been downloaded 41 times so far. Hopefully, Anki will catch on at MIT and students will make decks for their other classes and post them publicly so I can use them myself.

I also took on the task of identifying the root cause of some TestNG defects related to parallel test execution. These defects have been making it impossible to implement the ability to log my tests to a separate log file by thread without advanced jujitsu configuration and hacking in Logback. If the tests would just freaking execute in the right thread in the expected fashion, this would be trivial, but all these random extra threads get spawned because a new thread worker is inappropriately spawned when you have test method dependencies. TestNG works as expected when test methods don’t depend on each other. Actually, group and test method dependencies are notoriously buggy when trying to use the various parallel execution modes offered by TestNG. It was quite a ride debugging that code base, but I found the bug in the source code and now I am trying to figure out how to fix it. It seems to be difficult to get a pull request accepted in TestNG (for good reason), but I want this bug fixed so bad, I am willing to jump through whatever hoops they want.

Basically, I have a lot of irons in the fire all at once. Hopefully, switching to a role in a profitable part of the company will put me in a less over-burdened role. I am sick of working on an absurdly understaffed team with people in positions of technical leadership who shouldn’t be there because they can’t code or understand basic software architecture design principles. Also, I really want to work for an organization that doesn’t treat me or my team like a bunch of middle schoolers who need a hall pass and a reason to be absent for a bathroom break.

Share This:

Wherein I Give Up Due To Fatigue and Not Giving a Shit Anymore

I am exhausted today. I have been consumed by a tide of work that a team twice our size would not be able to competently manage. It started with a monster new feature that can not be quickly tested via our functional UI regression suite. It requires real end-to-end verification. Normally, we only need to create configurations which use features with the web application for configuration management. For this feature, we have to verify that the configuration is applied correctly once it goes active. This involved a journey of discovery that highlighted the insane lack of coordination and communication between our particular organization and the rest of the company at large.

First, we did not know how to do this end-to-end testing because we have never done it before. Second, we were under the mistaken impression that there was no apparatus for doing this kind of testing in our two test environments. We believed this because we have been told repeatedly that there was no such thing since we were hired. The entire development team and their managers also believed this to be true. A co-worker and I began trying to master this unknown art of end-to-end testing by contacting the team which develops and tests that part of the network. During this first meeting in which the two of us probably sounded as intelligent as brain-damaged turkeys riding the short bus to school, we discovered that both of our test environments did in fact have the networks for doing this end-to-end testing. Not only that, they have had them for FIFTEEN YEARS. It’s impossible to understand that our entire organization could share the delusion that this part of the system was not present in either test environment for so long. I still don’t have an explanation for it. The only thing I can say is that inter-group and inter-department communication is abominable.

We also discovered that another business unit was nearly done with building a system which allows them to go to a website, fill out a simple data entry form and press a button which spins up test VMs on demand that function as a nice little sandbox testing environment for testing for this part of the end-to-end pipeline in isolation. It is well on its way to being part of a continuous deployment testing apparatus where there is little to no human intervention required to spin it up and do tests. I am astonished that my entire organization seemed to be under the delusion that this was an impossible pipe dream. The sheer embarrassment of talking to one team after another about their testing which is so much better than ours and their systems knowledge, which is so much more comprehensive than ours, and their test automation which is so much better than ours has broken something in me. It didn’t help I calculated that it was going to take a month to design, write and execute these test cases. A month in which I would not be writing any code.

Also weighing on me is the task of writing all the functional test cases for the entire UI. There is a hiring freeze and getting new requisitions is akin to an act of god. We have two principal-level employees who want nothing to do with this task. And who would? It’s tedious and unrewarding work to write functional UI test cases. Since I have been tagged as the person who couldn’t get any automation done for three years, I figured it was better to bite the bullet and be the SQA lead even if I am not getting paid to do that role. I’m not even getting paid enough to do my current role. This decision was the genesis for an unpleasant discovery regarding my company’s IT support infrastructure, which is that there really isn’t one. You’d think that a company with 6000 employees would have gotten around to hiring a 24/7 team to support and administer things like the build system, the source code control system and the defect tracking and project management systems. But, sadly, this is not true.

Our JIRA installation started out as a rebel application running off the director of engineering’s desktop computer because a cadre of new employees refused to use Bugzilla. Naturally, because there was no formal introduction of JIRA as a tool, there were no knowledgeable professionals who controlled the adoption. The result is an explosion of project templates with a ton of totally unnecessary custom fields and overly complicated workflows. An organization full of supposedly amazing engineers should be able to design a system that cleanly separates the work of product design, development, testing and customer support, but what we got was a system full of templates and workflows that was designed by a drunken squirrel on acid. Eventually, it got too big to be a rebel application running on the director of engineering’s desktop computer. It did not get big enough, however, to merit a 24/7 dedicated support team. It had gotten big enough to have a part-time support team who were all doing other jobs before they got stuck with the responsibility for managing and administering this monstrosity.

This is where I entered the picture with my quixotic notions of quickly configuring three projects — one for the APIs I wrote for our test automation framework, one for the domain models I have written using the APIs, and one for the QA team to use for defining and managing work related to testing the customer configuration application. I had dreams of using these three projects to define and manage the workload of writing all the formal test cases for the application and the catalog of work that must be done to automate them. This seemingly simple task has been going on since early February. I am in the final stages of getting the correct workflow defined for the Zephyr test tickets. This customized workflow has been most painful part of the process. For everything else, I was able to use the default, out-of-the-box Jira workflow and fields or the agile-style workflow and fields that the team building the new front end uses. The Zephyr test cases were another matter. I wanted a workflow that would accommodate test case design and development for manual tests and automated tests.

I wanted a design phase followed by a review phase. Then, if the test case is to be automated, there is a development phase followed by a code review phase, followed by a testing phase. If manual, the ticket is ‘complete’ after review. If an automated test case successfully gets through testing, it is ‘complete’. I also wanted a workflow that could accommodate sending the ticket back for re-work if review, code review or testing revealed issues as well as a workflow that could handle flagging a test case for needed updates because of changes in the behavior of the application. Granted, this is not a simple workflow design and it didn’t help that I did not initially understand the JIRA terminology or means of representing a workflow. The other problem was that the number of custom issue statuses and fields had completely gotten out of control during the wild west period for our JIRA installation. Apparently, this can cause serious performance degradation in JIRA, so they were no longer allowing any new statuses to be defined. I was thankfully able to choose good statuses from the list of existing ones. That fact that there are about four different statuses signifying that something is ready for development or testing just shows how important it is for a company to get out in front of the problem of building IT support infrastructure instead of chasing after the cowboys who just get tired of waiting for a modern tool chain.

In the midst of this mess, I also took on the task of implementing localization support for my page object library. Our California team requested this feature before Christmas. There are eleven possible languages that a user can select when browsing our customer portal. It took 3 weeks of late evenings and working weekends to pull off locale-agnostic page objects. I also chose to do a major refactor of the DSL API I had written because it badly needed to be simplified and scaled down. I think the system works well. Now all the page objects refer to enumerated constants which represent these localized identities. There is some under the hood tooling that they use for mapping locale-sensitive identifiers to the enumerated constants. It was the best I could do given the short time frame for building it. It was a huge task that burned me completely out.

So, I looking at this job and I have come to the conclusion that nothing is going to change. I have been complaining about the fact that the traditional SQA work and the automation work need to be separate roles since I joined this company nearly four years ago. Nothing has changed. I’m still stuck with work I am not interested in doing and having to burn the candle at both ends to do the work I actually want to do. So, in short, I am looking for a new job as we speak. I finally realized that this company which is supposedly full of amazing engineers is not actually interested in applying solid engineering principles to the test automation. They aren’t even interested in applying solid SQA practices because they severely under-resource the function to the point of absurdity. I keep thinking back to my first real SQA job with a medical device company. The size and scope of the application was smaller, yet we had two managers, four people on the backend and six people on the front end. That’s half again the size of my current team. When I think about this, I realized how totally delusional my company is about the work burden and the complexity of not only testing these applications, but also building robust test automation for them.

Share This:

Some More Reasons Why You Don’t Have Good Automation

It has been a while since our last edition of Why You Don’t Have Good Automation. I am hopeful today. On Monday, I did a presentation on the reasoning and purpose behind my detailed labeling of test cases. The gist of the presentation is that placing emphasis on end-to-end testing as if it were the only way to do automated testing was going to lead us to a limited return on the investment we put into writing all that sweet, sweet automation infrastructure. Once I replaced the older smoke test suite, I was happy to see the dev manager and my boss go all gooey-eyed over it, but I am bothered by the false sense of security that it gives. I am not saying that it is useless, but it has limited potential.

I am disillusioned where end-to-end testing is concerned. It is hideously difficult to build everything you need to implement the plumbing to even get to the point where you can do one end-to-end test. I have been through a couple iterations of this nightmare. The first time was during the Frameworks Wars when my co-worker and I toiled mightily to automate a suite of forty or so manual tests. There was just so much to do and we were slammed over and over again every few weeks with an avalanche of manual testing work for the application. We were also new to the company and not all that familiar with the surrounding, flaky systems that our application interacts with for each of these end-to-end scenarios. Combine this with our total lack of organized training with the some very unreasonable expectations, we were set up to fail and fail hard.

At the time, both my co-worker and I were operating under the same limited perspective as everyone else. End-to-end testing was where it was at and if you weren’t doing that, you were not doing Good Automation. So, we coded and coded, all the while being asked, “Where is all that sweet, sweet good automation?” We actually built a fairly extensive library of page objects which later became of the basis of the current page object library, but because we didn’t have the plumbing to get all the way from A to Z, it amounted to nothing in the eyes of the non-coding people around us. We made some mistakes such as thinking we could not throw our code out there unless it was ‘done’ and leaving the results reporting to the end. We should have had a parade up and down the hallways once a week, given tech talks and otherwise tooted our horns relentlessly. Our unpolished code was better than most of the test code I had seen up to that point. If we had a simple HTML report with green and red pass and fail indicators, I think the non-coding cadre would have believed we were actually doing something.

Our biggest mistake, however, was trying to do end-to-end testing first. If we had broken those long end-to-end test cases up into smaller, atomic test cases that could be strung together in end-to-end scenarios or run as standalone tests, we would have been able to report each week that we had automated twenty-five new test cases. It’s very easy to automate a test that verifies that a dialog opens when you click its accessor. It’s easy to automate a test that verifies that the title on the dialog is correct. It’s easy to automate a test that verifies that the dialog is dismissed if you close, cancel or submit it. Since our application had a gazillion dialogs, we could have written an entire test suite for verifying the basic behavior of dialogs. It’s not easy to verify that a given type of user can log in, access the appropriate accounts, create a configuration, fill out a complicated data entry form, save their configuration and then make it active. There are just so many interactions with the UI along the way, the chances that some flaky thing will happen to short-circuit the test are too high. Maybe one of the dialogs gets stuck, or a page gets stuck and doesn’t load. Given the sorry state of our test environments, this kind of problem arises every other test run, especially during periods of active and intense development. I won’t go into the details, but one of the systems that the application must interact with at the end of the long process for making a configuration go live is the bane of my existence. Imagine what it is like to kick off a test run that has a lower bound on it of ninety minutes to go from start to finish and it fails at the VERY. LAST. STEP. when that final sub-system in the whole pipeline chokes because of a common environmental issue.

The advantage of writing lots and lots of small tests that get into your app quick, verify a few small things and get out quick is that the chain of interactions with the UI is so much shorter. There are fewer opportunities for an unstable test environment to screw up your test results. And if a test does fail? WHO CARES. You just retry it a few times to see if it passes on a subsequent attempt. It’s easy to retry because this isolated little test lacks all those dependencies that a step in a fifty step end-to-end scenario would have. And why do you have to verify everything in one end-to-end chain? Why is it not sufficient to find a way to verify all the little steps in isolation? I’m not saying that you shouldn’t have some end-to-end test automation. There is a place for this kind of test automation. It’s just not terribly comprehensive. I know that sounds crazy. It’s an end-to-end test. By definition, that’s comprehensive, right? Nope. Nope. Nope. It’s anything but comprehensive. It represents one, very narrow and defined pathway through your system. If you want a comprehensive test suite and you want it to deliver results sometime in the next decade, you need a test suite that is composed of thousands of atomic little tests that are scattered across your system’s functionality.

This brings me back to the beginning — Why all these labels on the test cases? This is a way for categorizing all the tests so that it is easy to search for them. When you write a test suite of atomic tests, you will end up with hundreds, if not thousands of tests. Hence, you need to be able to easily find test cases. Second, these labels make it possible to dynamically specify at commit time which tests the developer would like to run. The end-to-end tests, given their limited coverage, long running time, and relative instability are not helpful for this scenario. The developer wants relevant, fast feedback. We also don’t have a giant Selenium grid at our fingertips, so we have to be conservative about how we use these resources. If there are eight commits in a day, the suite of end-to-end tests would overwhelm our resources very easily.

So, in short, if you are a regular worshipper at the idol of the end-to-end test automation god, you won’t have good automation. This god is a fickle liar who tucks you into bed at night with his minion, the Smoke Test Teddy Bear. Smoke Test Teddy Bear whispers sweet nothings into your ear, telling you if the end-to-end test suite passes, all is good. Don’t worry about the thousands of other possible user interactions with your application that it _didn’t_ cover. They are inconsequential seeds of evil doubt that the faithful believers must banish from their thoughts. Shut the closet door on those demons!

Share This:

In Which I Bitch and Moan a Bit More

First, I would like to bitch about C#. I finally translated the whole state Brixen Java bean package into C# and both the C# and Java versions are fully unit-tested. Yay! Second, C# needs something like Lombok _ASAP_. I did not fully understand The Giant Teh Suck So Much that it is to write all this freaking boilerplate for ToString(), Equals() and GetHashCode() over and over again until I enjoyed not doing it for Java classes for the last year. It’s glorious to open up a class file and Not. See. That. Shit. I am bummed there is no way to do the decorator pattern in C# as I have done it in Java. That little beauty of a design pattern _really_ cuts down on boilerplate. And Equals() for collections such as Dictionary, the equivalent of Map for Java, totally blows because it does reference equality, not ‘contains the same shit’ equality like Java Map implementations do. Seriously… what the fuck? Who thought that was a good idea? This took far far longer than I thought it would. I thought I would be done with the C# translation before Christmas and that I would be blazing through the Python implementation by now. Writing unit tests is tedious and time-consuming, but having caught some bugs with them, I am happy I have committed to doing them in tandem with the C# translation. But enough about that. Brixen marches onward and the source is available here. Looks like someone forked my repo a while back. I am flattered.

Today, I am _not_ gruntled. Far from it. You probably would have guessed that from the mere fact that there is a new post today. I had my annual performance review and the result was good, but not spectacular. I feel satisfied with this outcome and it is what I expected. This was a rocky year for me, and I did a few things which embarrassed the QA director. I publicly contradicted his goals and strategy on a fairly high-level initiative he was pushing, and I chewed out the Principal SDET over email for checking crappy page objects into the core of the automation framework without so much as a word with me over it. So, the chickens are coming home to roost for me. I have always had a mouth that got me into trouble before my rational brain could stop me. I have a habit of just expressing things in a way that others find too blunt and not diplomatic enough. I also have a temper.

Today was one of the days in which my rational brain was too slow out of the gate to catch up with my mouth. A junior developer is taking on the task of translating the regression suite I wrote to run against the new, totally rebuilt front end. I expected this task to be difficult for her. The regressions suite tests the application that has the biggest, most complicated UI in the entire customer portal. That shit is hard and complicated and the page object modeling is a non-trivial challenge. Today, she called a code review that included me, the Principal SDET and two other team members. In the midst of the code review, the question came up about labels I have attached to the test cases in JIRA which are reproduced on the test method as group names. I have a longer term plan for using those labels and the group name mappings as a way to dynamically select methods for specific functionality on the fly so that a developer can trigger a build that selects tests that cover the functionality they touched in their commit. Principal SDET claimed I had agreed to take the labels off the Jira test cases and the test methods. Problem is, I have near perfect memory for conversations going back months, and I knew that I had not agreed to do that. In fact, she and the QA director had backed down from it when it last came up. I disagreed with this assertion and before I knew it we were shouting at each other over the phone and she was refusing to explain why the presence of these labels was harmful to anyone.

After a chat with the QA director, I will be doing a presentation next week to explain why I did this and where it is going and then put it forth to the team to decide to chuck this or not. If they decide to chuck it, then I will copy all my work elsewhere and continue implementing a parallel test suite that uses this plumbing and present it to the development team when it is ready because this whole thing was meant to help them anyway. I have no idea where this came from and it is not the first time an issue which had been settled has been raised again from the dead where I am told I have to change how I did something without getting a clear explanation for why I need to. I am getting… weary of this. Principal SDET has a habit of honing in on trivial, non-architectural things which harm no one and making a giant fucking deal out of them. Test data storage formats are a giant bugaboo and apparently everything must and should be stored in a Java Properties file even though Java Properties files are flat, key/value pairs and there is no tooling I know of or that she can point me to which will allow me to suck in a Java Properties file containing serialized data that represents collections of objects and their state and instantiate collections of POJOs with it. At one point, the directory structure of the framework was a giant burning issue. If someone checked something into the wrong directory or tried to use Maven modules to manage multiple, related projects, they were not following ‘architectural’ guidelines. I have no idea what important architectural implications any of this has since they have NO EFFECT on how any particular subsystem interfaces with any other subsystem.

The code review barely touched on anything related to actual form and function of code. I don’t know if it is worth it in the long haul to stick around in this role. I like the challenge of the stuff I work on, but this perennial micromanagement and drama is getting really old.

Share This:

Long Time, No Post

My translation of the existing Java codebase for Brixen into C# continues to slowly move forward, hampered mostly by long hours for my job and the inevitable craziness of the holiday season. I have not yet encountered anything that I have been unable to duplicate in C#. I have learned how to implement extension methods and how to use generics in C# which is awesome.

I am struggling to replace an old smoke test suite for the product I support before I leave for Christmas vacation. I feel like every victory I have is rewarded with a new and difficult obstacle. Every time I successfully clear one hurdle and need to take the test suite to the next level, there is some new, difficult problem to solve that takes a week to deal with. Meanwhile, the QA Director keeps asking, “Where is that sweet sweet automation you promised us!” It’s hard to explain to people who don’t code the difficulties and challenges of re-tooling something to get around things like defects in the open source tools you’re using, or building out infrastructure to support things like configurable suite options and test data feeding. It always sounds like I’m making excuses for my pitiful lack of amazing progress in producing Teh Automation Suite of Amazingness and Wow-dom.

One thing that annoys the shit of me is that it seems that all prior progress I have made is forgotten as soon as a week goes by without new test cases getting automated. It doesn’t matter that I managed to build infrastructure that allows me to execute the exiting test cases across multiple scenarios as opposed to one. It doesn’t matter I had to stop and produce something for the test automation framework that we are trying to shove down the throats of the other business units. It doesn’t matter that I had to implement a work around for a TestNG defect. All that matters is that new test cases were not automated that week. The fact that I have automated almost 75 percent of the test cases covered by the old test suite I am replacing in addition to automating test cases that the old suite didn’t cover is quickly forgotten.

The QA director has taken to the habit of stopping by my desk and publicly ‘joking’ that I emailed him the night before to tell him that I was already done with the smoke test. Or that I told him I would be done early. Nagging that is disguised as good-natured teasing is more annoying that direct nagging could ever be. Seriously, it just makes me want to spend the next hour Christmas shopping on Amazon for my kids out of spite. The agreed upon deadline for finishing the smoke test replacement effort was still TWO WEEKS AWAY when the nagging began. It doesn’t help that we have all been told to expect lower bonuses this year and that my bonus is riding largely upon the successful completion of this project before I leave on vacation. It isn’t worth working the kind of hours I do to put up with that crap. Plus, all this overtime and stress is getting in the way of moving forward with Brixen which is far more interesting and fun to work on.

Another giant pain in my ass is that the principal SDET and Architect of the automation framework we are trying to shove down the throats of the other business units tried to quietly sneak in some page objects from her old framework into the new one because her junior engineer didn’t want to spend 10 minutes asking me some questions about how to follow the far superior approach I busted my ass to develop. Instead of making a real effort to convert to the new framework, he just wanted to be lazy and just do it the way he had gotten used to doing it. I tried to point out the irony of our effort to make the other business units switch over to our way of doing things while allowing an engineer on our team to avoid doing exactly that, but the principal SDET played dumb and pretended that my point was totally incomprehensible. This lead to a demand for a code review of the core classes in the page object API by the principal SDET.

I wrote a few Wiki instructions for how to use the core classes. For example usages of the API, I used the page objects I had built for the customer portal to support my automation. At one time in the recent past, I had advocated that the API as well as the page objects built with it should both be part of the framework, but the principal SDET insisted that only the API should be in it. So, at her insistence, I moved them out of the framework project. During the code review, the members of the West Coast team who were taking part interrupted me as I was in the process of explaining some of the examples on the Wiki page to ask, “This LoginPage class you show here…. where is it?” So, we had an unscheduled discussion about the new features they wanted in the the LoginPage class and the other page object classes for navigating through the customer portal. I would feel more smugly satisfied if I had not just signed myself up for even more work.

Share This:

Yet More Reasons Why You Don’t Have Good Automation

It has been a terrible two weeks for me. I lost my last living grandparent and my workplace drama culminated in the latest episode of humiliating and ham-fisted attempts to micromanage my work because The People Who Matter can’t seem to grasp the reality that test automation is really really hard.

I am demoralized today. I have hesitated to write about the ‘people’ element involved in developing a robust automation framework because this tends to be volatile and subjective content, but my work week totally blew, so I am going to do what has become a time-honored tradition since The InterWubs became a mainstream pastime — I am going to bitch about it online and talk some shit. When I joined my current workplace about three years ago, the state of UI automation was less mature than it is now. There were a lot of teams who were just starting out and there were some teams who had more mature tooling.

At this time, a complete overhaul of a big system was also just beginning, which meant disparate teams would have to collaborate together in ways that had never occurred at this company before. There was a sense of urgency about getting this ‘Automation’ thing going because there was so much to test and manual regression testing is slow and expensive. The various teams of SQA engineers were trying to get a handle on the automation and some were succeeding, but many teams were not. I was newly hired and excited about finally getting a job where I could spend more time writing code than running the same old boring manual regression tests over and over again.

I was assigned to a complex web application that saddles many domains across the company. Both the UI and the data/business logic aspects of this application are complex. It’s the kind of meaty automation challenge that gets a weirdo nerd like me pumped up and ready to code. My boss had recruited a team of senior employees and we wrote a large suite of test cases for this application. Then we started to investigate how we could approach the automation. We decided to go with the Page Object Model design pattern that is the gold standard for Selenium UI test automation. We evaluated some other frameworks, but, surprisingly, none of them were actually using this design pattern, so we started writing our own framework. Although I am biased, I will state that the code was sophisticated and high-quality stuff that was up to the task of being a great API for writing automated tests against the rich and complex UI of the application we were responsible for.

Then along came a high-level ‘architect’ who decided that everyone had to use the same ‘framework’. I won’t go into the ridiculous lack of consensus about what the term ‘framework’ meant because it’s a distraction. Let’s just say, there was no consensus on what this term meant. He took a look at the some of the various tools that some teams had built which they called ‘frameworks’ and then decided to declare one of them the ‘winner’. Then he decreed that all teams must use the one he had declared the winner. On the surface, this isn’t necessarily a bad idea and if the person making the decision actually knew how to evaluate the candidate frameworks, this could have ended well. The losers would have felt slighted and upset, naturally, but technical people generally can be persuaded to accept standards and tools they didn’t originally choose or build for themselves if they are good.

The key ingredient in making the above scenario end well is the word ‘good’. In my opinion, the following things must be true for a test automation framework to be good:

  1. It should be built with industry standard tools, not someone’s pet bleeding edge thing that is new and sexy, but not mature and certainly not in widespread use
  2. It should be built with tools that you can easily recruit talent to use and extend, not tools which have few experts you have to struggle to find and hire
  3. It should be built with tools that your current employees are mostly familiar with
  4. It should be built with tools that are well-documented and well-supported
  5. It should be built with a well-designed and properly articulated architecture so that its components are maintainable, reusable, and scalable
  6. The separate components of the system should be cleanly separated and use APIs to communicate with each other
  7. There should be no hardened dependencies on specific test management or defect management systems

Now that we have this settled, let’s return to the high-level ‘architect’ who decided to require all the QA teams to use the same automation framework. A person without good programming or software design skills can handle the first four items on the list. The last three items need to be evaluated by someone who does. Guess which category homeboy belonged to.

The framework he declared the ‘winner’ was just Not. Very. Good. It was actually one of the frameworks my team evaluated when we were trying to figure out what we were going to do with our automation work. We quickly passed on this one because it was very poorly designed and had none of the characteristics we were looking for. It didn’t help that the person who wrote it clearly did not have much familiarity with the WebDriver API. When I investigated how other teams were using it after Sir Architect declared it the winner, I noticed that a lot of them included it as a dependency in their POMs, but didn’t actually use anything in it. I think this was mostly because the design was so bad that it was nearly impossible to understand unless you methodically walked through every line of code. I eventually did this myself, and I have to say, I am a stronger person for the experience.

My co-worker and I couldn’t bear to dump our system which worked great for us and adopt this other system which totally didn’t work and would require us to shoehorn some ugly-ass spaghetti code into our test code in order to ‘use’ a framework that did absolutely nothing for us. We went on writing our page objects and incrementally automating some steps in our suite of manual end-to-end tests. Meanwhile, my boss decided to become an individual contributor and a member of our team who never even wanted to manage was given responsibility for our whole team. She had recently been hired and had not even had time to settle in. After she took over, Sir Architect kept asking if we were doing our automation using The Chosen Framework.

I should stop here and explain that our new manager, unlike the one who decided to stop managing, was not someone who had strong programming and software design experience. So, she put a lot of pressure on us to deliver lots of real good automation real quick. The problem with people who don’t ‘do’ software is that they tend to think that test automation is easy and when they hear the word ‘framework’ they think, “Magic system which shits a pound of automation gold every hour with little to no effort and so little technical expertise required that even a preschooler could automate the whole internet with it.”

We tried to explain that in order to automate the regression suite for our application, we needed to build models for all the UI components so that we could interact with them programmatically. We also tried to explain that the data and business logic domain was also something that needed to be modeled so that we could define and feed test data into automated tests instead of hard-coding it right there in the test methods. We tried explaining that The Chosen Framework contained some badly designed mechanism for getting a reference to a WebDriver instance of some desired type and that was basically all there was. All the rest of the necessary tooling, namely the models, were not provided with The Chosen Framework. Not only that, the Chosen Framework required that we use a ‘base’ page object which seemed to be super confused about whether it was a generic page object or a misguided effort to wrap some basic WebDriver functionality in a way that was inferior to WebDriver’s own native interface.

It. Just. Did. Not. Sink. In. Part of the problem was that other teams had managed by some miracle to tease some automated tests out of The Chosen Framework. We couldn’t seem to make our manager or Sir Architect understand that these tests didn’t do very much and that they were for very simple applications with maybe a page or two with a very limited data domain. Our application is not that type of application. It has lots of dynamic UI elements, numerous dialogs, menus, and data entry forms along with some complicated multistep state changes that have to be tracked and verified along the way. Not only that are a lot of validation rules on the data that are in play as well as a complicated user and role access permission domain to make it even more fun. And then there is the collection of features and behavior related to migrating customer artifacts from the older application that our application is supposed to replace. I also cannot neglect to mention the oh so very special test environments which have been featured in earlier episodes of “Why You Don’t Have Good Automation”.

A subset of these complications should be enough to send any self-respecting automation engineer over the edge, but throw in the need to periodically explain your failure to deliver some sweet, sweet automation for this mess to people who just refuse to understand that a badly designed mechanism for getting a WebDriver instance is not going to result in Automation Jesus marching triumphantly down our cubical-lined aisles and you have yourself a heaping helping of Soul-Destroying Is This Really My Life Cognitive Dissonance. We had a test suite of some 40 manual tests with roughly 15-30 steps apiece that we were running on two versions of IE, the latest version of Firefox on Windows 7 and OSX, the latest version of Chrome on Windows 7 and OSX in addition to Safari. At one point, our manager just declared that if we could not complete the automation for this test suite in three weeks, we had to dump our work and switch over to The Chosen Framework. This was after saying time and again, that the Chosen Framework provided no interfaces for interacting with the UI, nor did it provide any efficiencies for building the interfaces.

Fast forward two years. My team falls back under my original boss who has decided to manage again. My team now includes the person who built The Chosen Framework who has been promoted to Principal SDET. It is announced that we will collaborate with some other QA engineers in our organization to built a new framework and the designer of The Chosen Framework, V1, will ‘architect’ it. We spend a year and a half building various components while our Principal SDET regularly becomes very concerned about the directory structure of the repositories and the very large amount of code which some people are writing because large amounts of code is ‘heavy’ and that is bad. Oh, and ‘the communication protocol will be a POM dependency’. Somehow we write something that some of can use and there is an effort under way by our upper management to shove our framework down the throats of every other business unit in the company. A framework which none of them helped build or design and does not include anything from their now mature and fully-featured frameworks that are mostly better than ours. You can imagine how fast they all are running from the directive to adopt The Chosen Framework, V2.

I have finally been given space and time to do the automation after two years of relentless and never-ending manual testing stints all the while hearing, “Where is all that good automation?!!!! We can’t afford to hire manual testers. Just do it in your spare time!” Now I have about 200 automated tests of which only a couple are flappers which need some tweaking to make stable. Nevertheless, the new QA director has decided that we need to ‘start making some progress’ and a close working relationship and some coding ‘help’ from the creator of The Chosen Framework, V1, is just the solution to this problem.

It almost made me want to quit and take up goat-herding as a profession after pouring acid over my laptop and taking a shit right on top of my workstation out of spite.

So, folks, this is the latest edition of “Why You Don’t Have Good Automation.” Automation frameworks are a serious technical challenge and you need to put technically sophisticated people in charge of it. Also, don’t try to shove a tool down the throats of people who know better than you do what they need. If you’re looking to have your entire company on a single framework, you need to involve all the teams in requirements gathering. You also need to incorporate the best of their tooling into it. Also, if you don’t want to drive your automation staff into the arms of your competitors, don’t keep asking why they aren’t done every two days.

Share This:

Yet More Reasons Why You Don’t Have Good Automation

I got a bit sidetracked with the Brixen posts that I forgot to spend some more time explaining Why You Don’t Have Good Automation.

I am tired today. I spent the day dividing my time between way too many tasks, one of them boring as hell, but which should make it possible to dynamically put together a test run with Zephyr for Jira using keywords. The keywords correspond to group names on the automated tests for each test case. It’s a giant pain in the ass to add all these labels to the Zephyr test case and then add them as group names to the test methods, but it’s necessary to make what I want to do possible.

In addition to this mind-numbing data entry horror, I also spent an hour or two trying to figure out how to run these automated tests on the production system. I have never set up automation for the production system. My suite runs every night on whichever of our two test systems that isn’t currently crapping its pants and painting the walls with its own feces. I have some configuration files containing a list of test users and test accounts and I have a neat little set up routine that pulls this test data out of the config files and delivers it to tests running in parallel so that no single user login is used by more than one thread at a time. Behavior on our awful test environments gets even more awful and unpredictable if a user is logged in from multiple browser instances.

I still get random errors resulting from a user being logged in more than once because I am not the only person running tests on the test system. There are hundreds of people who use these environments and everyone has access to the same user accounts that I do. So, shit happens. Shit happens a lot less since I set up my tests to ensure that at least they don’t inadvertently use the same login credentials, but I don’t control the accounts that other engineers are using for their tests.

But I digress. The production system is another can of worms. I need a way to store the login credentials and deliver them to the tests in a secure way. Storing it all in a plain text configuration file and checking it into source control is a pile of bad idea topped with steaming fresh dog turds. It would violate PCI security standards six ways from Sunday. I’m getting some help from the engineer who wrote the original and now retired automation suite that used to run on production. He had a database running on a managed VM that stored the credentials and used a config file to specify the database access credentials. The config file was actually a stub because checking the DB credentials into source control in a plain text file is no better than checking in the login credentials themselves. A human being had to check out the repo and add the DB credentials to the config file. Then they could trigger the tests which used the DB credentials from the config file to log into the database to retrieve the user login credentials.

I found out that I can’t use the same strategy because a DB running on a managed VM isn’t kosher anymore due to PCI security standards. I have to find some other way to deliver the production login credentials for automated tests to use. Did I mention that I am not an expert in security standards? And there is no documented process for dealing with this problem? A few months ago, our entire organization was tortured with a giant PCI training effort in which we had to watch lots of videos and take lots of quizzes to prove we had been trained in PCI security awareness. I guess developing procedures and systems for dealing with practical, real-life problems just like this one and training us in that was too much to ask for.

Another part of my workday was devoted to speaking to an account executive at Sauce Labs in order to get a cost estimate for their services for the four teams in my particular mini-organization. I think he was excited up until he found out that I had no budget power and that I actually don’t know what the management class here is willing to spend. I was handed the task for getting an estimate for using them in an offhand way in the middle of a meeting that was supposed to be about something else two weeks ago.

This Sauce Labs POC task has been a hot potato that got passed around for the last year. A couple managers had this task sitting on their plates for weeks and every week during our status meetings, it just kept getting pushed off. Eventually, we stopped talking about it. Then some engineers on one of our teams actually did a trial with them and did a great presentation on the experience. I was excited because I thought something was actually going to happen at that point, but then I never heard another word about it until I said we needed Sauce Labs in a meeting two weeks ago because I was pissed at the lack of infrastructure for Selenium test suites. Suddenly the task of doing the same thing that has already been done by others was given to me. So, I am doing what others already did, which sadly, will lead to the the same likely result. Nothing will happen and a year from now, some other over-worked person will be told to do it again.

So, you’re wondering, “What does this have to do with why I don’t have some of that real good automation?!” One thing I have learned in all my years of working in quality assurance is that ‘quality’ is not a thing. It’s actually a living system made up of processes, people and tools. It’s also a ungodly complex system in a large organization. Automation is only one piece of it. And unfortunately for you, it’s a piece that depends largely on the smooth functioning of almost every other piece of the system. How many times have you acknowledged that there is a problem somewhere in your system that relates to quality that resulted in thousands of hours of lost productivity? Shitty test systems? Shitty or non-existent documentation? Shitty or non-existent approach to generating and managing test data? Shitty, fractured approach to tools and infrastructure procurement and maintenance?

If your approach to any or all of these deficits has been to click your heels and wish for your own personal Automation Savior to swoop in and drop a load of some real good automation on you like Rapunzel spinning gold out of straw, I would like to point out that you need to provide the straw first. Providing the straw involves the procurement of arable land, seeds, fertilizer and labor to grow and harvest it, along with a transportation network to get it you in good condition. Because Rapunzel ain’t gonna spin no gold out of moldy, wet, rotting straw with rat droppings all over it. ‘Kay?

Share This:

‘UXD Legos’ Has Been Reborn as ‘Brixen’…

‘Lego’ is a registered trademark after all. The source is here. It looks very different from what I presented at the Selenium conference. I did a ton of refactoring around how web controls are specified. I will do a write up on the differences soon. Today, I am exhausted, and I’m just happy to get it posted to GitHub.  There’s a half-assed example of usage in the org.brixen.example.priceline package.  This is also vastly untested at the moment. My next steps will be:

  •  Do a complete write-up on this new incarnation of the ‘UXD Legos’ concept and how it differs from its previous form
  • Start translating it into C#
  • In tandem with the C# translation, write tests for the Java classes and the C# classes as I do the translation
  • In tandem with the translation and testing, develop some full-assed usage examples

I am not sure how long this phase will take. Hopefully not too long because I still have the Python and Ruby incarnations to write. This should be fun. Way more fun than running boring manual regression tests on crappy test systems that seem to want to give me the middle finger every five seconds.

Share This:

Even More Reasons Why You Don’t Have Good Automation w/ Update on Generic UXD Legos Source Code Packages

Update on the example source code packages for UXD Legos:

I am writing Javadoc for the newly refactored source code. This is quite an undertaking, but one that I feel is important to do. I also have expanded the dynamically configurable options for hover controls. As Oren Rubin kindly pointed out to me after my presentation at the Selenium conference, the Javascript hover workaround doesn’t always work. Some hover controls use the CSS :hover pseudoclass which cannot be triggered with Javascript. So, in these situations, there is nothing that can be done via Selenium or the JavascriptExecutor to trigger a hover action. He also said that some security protections also prevent this workaround from working properly as well.

I found a test case in a public website where the Javascript workaround doesn’t work for hovering and came up with some other ideas for workarounds that would allow a tester to trigger the side-effects that occur when the element is hovered if they cannot rely on WebDriver or JavascriptExecutor. The benefit is that they can still automate tests that rely on the ability to trigger the side effects, but which do not test the result of the hover action itself. Obviously, for the environments in which the hover action cannot be triggered, the result of the hover would have to be tested manually, but downstream functional tests don’t have to be blocked by the inability to trigger the hover in automation.

I have a test case, which is probably an uncommon edge case:

  1. The control is invisible and must be hovered before it can be clicked
  2. Clicking the control expands a menu
  3. The ‘Expanded’ state of the menu is sticky
  4. The menu becomes invisible if the focus leaves the control that expands it
  5. If the focus subsequently returns to the control, the menu will appear without clicking the control because the ‘Expanded’ state is sticky
  6. If the control is then hovered and clicked a second time, the menu is collapsed
  7. If the focus leaves the control and then comes back to it, the menu is not visible

This is quite a complicated test case and would definitely require some testing and experimentation to determine what, if any, dynamically configurable kinds of options can be used to handle cases where the hover cannot be triggered via Selenium or the JavascriptExecutor. I have some ideas, but unfortunately, I can’t try any of them out at the moment. The only example of this kind of component that I am aware of exists on the product I test. And if you read my previous post, you know about the vast array of test environments which are available to me (and hundred of other engineers) in my workplace. There is a bug that prevents the page which has this control from loading at all. The estimated time-to-fix is like…. 5 days. And because I can’t just spin up an environment with an earlier, working version of the system, my ability to test and develop this part of the API has come to a screaming halt. Along with my effort to develop a comprehensive UI automation suite for the brand new front end for this application I test because NONE OF THE PAGES WILL LOAD. Anyway, this portion of my post isn’t supposed to talk about Why You Don’t Have Good Automation, but unfortunately, the two of them are bleeding into each other this time.

So, what I probably will do is finish the Javadoc for what I have and post the source code. It’s not terribly well-tested in its new, much refactored form, but I will start writing unit tests for it when I start trying to translate it to C#.

And now on to Why You Don’t Have Good Automation:

I am puzzled today. I am puzzled by a phenomenon I have encountered in every QA job I have ever had. EVERY. SINGLE. ONE. The company states without equivocation that they want Good Automation. They acknowledge that manual test execution is costly and slow. They acknowledge that not having Good Automation severely limits the scope and coverage of the testing they can have. They acknowledge that they would benefit enormously from the fast feedback that Good Automation would give them. They acknowledge that the development cycle for their products would become much shorter from this quick feedback. They acknowledge they could quickly catch regressions if they had Good Automation that delivered results within an hour of every new build. They acknowledge that catching regressions quickly in complicated systems would help shape design improvements by surfacing unnecessary coupling between seemingly unrelated parts of it because Good Automation would catch regressions triggered in one part of a system by changes in another part.

How often have you seen the following in a SQA job post:

Need strong automation developer with at least 8 years experience in the industry. Masters degree in Computer Science desired. 5 years experience in Java/C#/C++/Python/Ruby/Perl/COBOL/FORTRAN or some other object oriented programming language. Selenium experience in all of the above is highly desired. Responsibilities include mentoring junior engineers, building automation and release engineering infrastructure and developing comprehensive test plans for an array of products. Job requires fifty percent manual testing.

Whenever I encounter one of these job posts, what I actually see is the following:

Desperately in need of a strong, young and healthy unicorn that never needs to sleep or take a vacation. Must be willing to subsist on a steady diet of dirt and impossible expectations.  We have no fucking clue what it is we need or want, so we threw everything we could possibly think of into the list of requirements for this position, including the release engineering function which is totally a separate role from manual testing and automation development. We want someone who is both an amazing software developer with an expensive and lengthy advanced education that has a first-year drop out rate exceeding fifty percent as well as an amazing quality assurance expert with the associated superb communication, writing and analytical skills. We also want someone who is amazing at dealing with the intense and demoralizing office politics that come with working for a company that has the same unreasonable and impossible expectations we have.

We want you to have the same level of magical thinking we engage in because we wholeheartedly believe that we should get all of the skills, experience and talent of three strong professionals for just one salary. We want you to believe as much as we do in the fantasy that we really truly want good automation because the only way you will take this job is if you believe things that just aren’t true! And when you find that instead of the fifty percent manual testing we said you would do is actually more like seventy-five percent, we want you to cheerfully and politely accept endless questioning of your abilities and talents because we want to know where the all that good automation is and why it is taking you so long. You must be doing something wrong!

Let’s just consider the problems inherent in expecting that an individual employee should agree to perform two functions with two different skills sets — Testing strategy, test planning, designing good test cases and developing robust software systems to automate, execute and report results for these test cases. THEY ARE NOT THE SAME THING. These are actually two distinct skill sets. I am not saying they don’t often coexist in the same person. In fact, I think it is not uncommon for a really good test automation expert to have both skill sets because a lot of the time, they started out in manual testing. I do not believe, however, that the path to Good Automation will be found in thinking you can have that person do both jobs at the same time. Because they are both really hard jobs to do really well. You try to hire a single person to do both, one or both of the functions you want them to perform will suffer.

I need to break some really unpleasant news to the modern tech workplace. Multitasking? IT’S BULLSHIT. Computers can have multiple processors. Human beings only have one processor and the quality of what it can do suffers when it is forced to divide its focus between multiple and competing tasks. Please stop smoking the crack that made you believe humans can perform multiple tasks at the same level of quality and speed and within the same amount of time that they could do each task individually.

Now, let’s talk about the meaning of ‘Manual’ testing. I think manual testing has gotten a really bad name. I have noticed a trend lately in job candidates which are applying for positions that are billed as ‘Developer In Test’. They don’t want to do ‘manual’ testing. They look down in it and feel that it is a lesser function than automation is. They see it as less prestigious and less well-compensated. It is, in short, a deterrent to taking the job. It’s sort of like spraying automation developer repellent all over your position and seeing if you can find the developers with the right genetic makeup that makes them resistant to it. These job candidates are correct in many of their assumptions — salaries for traditional SQA employees are lower and the jobs are considered less desirable and it became common sometime in the last 15 years or so, for every traditional SQA employee to suddenly want to ‘get out of manual testing.’ Employers have contributed to the stigma of this role by requiring that most if not all of their SQA hires have some automation experience and often a computer science degree.

The problem with all this is that every software company really needs the skills that a talented traditional SQA engineer can bring to the table. The reason you need those employees is that they will provide you with a necessary precondition to Good Automation, which are good test plans with well-designed test cases. This is not an easy thing to do, and if you find an SQA engineer who is really good at it, you should compensate them highly and treated them like the treasured and important asset that they are. Don’t insult them by acting like their skills are out-of-date artifacts of a bygone era and demand that they transform themselves into a software engineer in order to be considered valuable and desirable as an employee. Let me list the skills a really good SQA engineer needs to have:

  1. They need to write well
  2. They need to communicate well
  3. They need to have really good reading comprehension
  4. They need to be able to synthesize a lot of information sources from design specs and requirements documents into test plans with well-written test cases which can be run by someone who is not an expert in the system. THIS IS HARD.
  5. They need to write test cases that can be automated. THIS IS HARD.

Still think this person shouldn’t be treated with the same level of respect as a good software developer? No? Fine, you don’t deserve that employee and I hope they leave you and your company in the rearview mirror as they get on the highway out of Asshole Town where you are the self-elected mayor and village idiot.

I work on a team of 8 people. There is one person on my team who I feel has had the most positive impact on product quality. She just always seems to master her domain no matter what it is she is doing. She builds all the right relationships and somehow manages to extract information out of this crazy and chaotic environment where very little is documented in a testable fashion before it is coded into the products. This person does the traditional SQA function. Her expertise was invaluable in onboarding several of us and I still find myself going to her with questions after working here for three years myself.

Now, lets get to the subject of what automation developers find so hateful about ‘Manual’ testing that they can’t run from it fast enough. Let’s get the simple reasons out of the way. Some of them just don’t have the chops to do the up front work that a good traditional SQA engineer needs to do to pave the way to Good Automation and they know it. But the more common reason is that ‘Manual’ testing frequently means that there is a large, often poorly written, manual regression suite that you want to automate, but you can’t seem to get around the fact that it’s just too big, tedious and time-consuming that the automation engineers you hired to give you Good Automation just don’t have the space and time to actually do any automation. Running the same test cases over and over again, release after release after release is just awful. It’s awful no matter who you are or what you are good at. It’s the kind of job you should give to a temporary contractor or an offshore company that specializes in providing these kinds of services. It doesn’t make the work any more pleasant, but at least you aren’t paying top-dollar to have it done and you aren’t bullshitting anyone about how rewarding it is. Because it’s not rewarding by any stretch of the imagination.

If you are having trouble getting good results with the temporary contractor or out-sourcing strategy, let’s circle back around to Step One which is the person you hired to write that regression suite. Did you perhaps have some poor judgment about the necessary talent and skills to write a good regression suite you can effectively outsource? Because that just might have something to do with why you can’t seem to get satisfactory results with outsourcing it. Don’t treat the traditional SQA function as a lessor function than development. Take care to hire the right people to do it and make sure you treat them like the valuable asset that they are and you will not be disappointed. Don’t make the mistake of thinking that their job is easy. Hire the right people to perform this function and don’t expect them to do a second full-time job and you will find yourself on the road to Good Automation.

Share This:

Some More Reasons Why You Don’t Have Good Automation w/ Update on Generic UXD Legos Source Code Packages

Update on the example source code packages for UXD Legos:

I am writing the sample page objects to demonstrate how to use the API and its components. I have refactored a lot of things since I presented the API at the conference. The configuration service implementation has been refactored to auto-initialize itself by reading in all the available configuration profiles. It has also been updated to be more robust in a multi-threaded environment.

I have significantly re-worked the model for controls on UI components. The new version is more flexible. Controls are treated as separate entities from the components they control. Since some interactions, like dismissing a UI component, are actually achievable by multiple controls, it makes sense to do it this way. My original design didn’t robustly handle this. For example, a chooser dialog can be submitted, or cancelled, or closed via three different controls. All three actions result in the dialog becoming hidden from view, but my abstract model for components which can be toggled visible and invisible didn’t handle the fact that there could be multiple controls that dismiss a component from view. The new model can handle all three of those scenarios in a generic way. I am working out a good interface for the builder of the UI component to handle specification of all of its controls. Each type control I modeled has a specific builder implementation for it, which is great if you are just focused on specifying and building a control, but not so great if you just want to specify and build the whole component, along with all of its possible controls. It’s clunky to have to interact with more than one builder. I think I have found the right design for this, and I am implementing it right now, along with the sample page objects.

I think, in addition to delivering the C#, Python, and Ruby versions of this API, I will also need to write an updated slide show about it.

And now on to Why You Don’t Have Good Automation:

I am grumpy today. I had a meeting this morning to discuss the automation strategy for a major product. We ended up talking about how the test suite is growing in size by 5-15 tests a day and I pointed out that I have a GIANT bottleneck in the form of inadequate infrastructure for executing Selenium test suites. It was revealed that this is a big problem for many other teams as well. There is no company-wide Selenium grid system available. We also do not have the green light for using Sauce Labs. I was under the impression that other teams had their own private Selenium grids which they jealously guarded because they are a limited resource and building a comprehensive Selenium grid is a big task. If you read my previous post on this subject, it’s not particularly easy to acquire the necessary resources and IT support to build one. I suspect that the reason we do not use Sauce Labs is that we would need to scrub all data in the test environments of personally identifying information. Which… will probably not happen anytime soon, so let’s backtrack to Square One: there is a giant bottleneck related to the lack of test infrastructure for running large suites of Selenium tests. It’s just crazy that there are tests that could be running with every build, that are not running because we only have the infrastructure for getting quick results for the highest priority tests.

If your company, like my company, spends a lot of time recruiting engineers with Teh Awesome Mad Skillz, but they do not budget resources to handle all those awesome automated tests, they are wasting a lot of money and a lot of talent. They would be better off hiring a cheap team of off-shore manual testers to run the tests manually. Otherwise, you have to force those amazing engineers to run tests manually in lieu of writing automation you can’t support in order to ensure that your products are tested. It’s expensive to hire development engineers to run manual tests all day. Plus, it pisses them off and then they leave for jobs where they get to write code.

The other issue that came up is that my tests take a long time to run because we have _TWO_ whole test environments available for an engineering organization of hundreds. TWO. WHOLE. ENVIRONMENTS. And they are unstable because there are too many people using them. Hence, all my tests have to do a lot of checking in order to capture failures related to environment craziness as opposed to real failures so that they can be re-tried until a verifiable true pass or fail can be determined. Other teams are running destructive tests all day long on these environments, so I can make absolutely no assumptions about the availability of test data on the system. Therefore, I have to make each test method search for test data that meets the necessary conditions for the test first.

The end result is that the bottleneck in Selenium grid infrastructure is doubly painful because the terrible test environment situation requires lots of retries and querying that would be unnecessary if each engineer had an isolated environment only they were using. Tests take more time to write because of all the extra overhead of dealing with the environment, so I’m not as productive as I could be. Not that it matters because…. Square One: Inadequate selenium grid infrastructure to run all the tests.

To summarize:

Automation is not magic dust that is gathered by tiny little fairy hands and deposited at your doorstep overnight for free. It takes a lot of planning and infrastructure. So, you like… need to invest in it and stuff. If you don’t, you won’t have good automation. You will have shitty automation that mostly functions as an Automated Smoke Test Teddy Bear that you can cuddle up to at release time for that warm feeling of false security that it gives you after forcing your resentful automation engineers to run manual tests until they are ready to grab the nearest fork and dig out their eyeballs from the sheer boredom of it all.

Share This: