Tag: automation

Treading Water Is Too Much Work

I honestly meant to post about the daily insanity of my workplace about a dozen times over the last two months. The only problem was that I was working all the time. Something amazing happens every time I step off the Endless Manual Testing Train To Perdition. I always have a giant pile of technical debt to address and I have to work overtime for weeks to catch up on the deferred maintenance. The current task on the pile is implementing an interface for a REST API to query the customer configuration management application for configurations meeting necessary criteria for automated tests. I didn’t want to use the UI to search for these configurations because the test systems are still a steaming pile of crap. Using the UI introduces a ton of new opportunities for their horrible performance to blow a test execution run to smithereens. Plus, these tests already take long enough to run that any means I have at my disposal to optimize the execution time should be put to work.

The API belongs to an ecosystem of APIs the company developed for programmatically interacting with our services and systems. There is a tool available in the The Chosen Framework v2.0 for executing requests against these APIs, but the design is not to my liking. Given the political shit storm that always surrounds any attempt to alter the design or function of any of the services in The Chosen Framework v2.0 and the total lack of staffing to support and maintain the framework, I decided to secretly write my own tool. There is an overly complex authentication system for using any of these APIs, so I had to build out my model for the access control group and user admin business domain to include the credentials needed for authentication. I have a penchant for wanting all my test data to live outside my test suite source code, so everything needed to be serializable/deserializable from both XML and JSON because who am I to dictate which storage format a person should use in order to employ my tools for their own purposes?

One thing lead to another, and suddenly, I had 6 different repositories for tools I have written to model parts of the business domain and as well as services I built on top of those models to automate interactions with the products I test. Not to mention that the number is set to grow because I have plans for additional services for even more automation goodness. In short, this collection of tools and services has become a total beast. It is really an automation framework in its own right. I finally managed to get a Stash project dedicated to them, so I moved everything there. As soon as I did this, I started to get nervous about getting caught for not using the services in The Chosen Framework v2.0. QA Director has been on the warpath against people who don’t want to use the services in the The Chosen Framework v2.0. I believe that the failed effort to shove it down the throats of other business units in the company is a sour and somewhat embarrassing memory for him, especially because several people on our team have turned their own noses up at various tools in framework. I can only imagine what the reaction will be if I get caught for building a rogue framework.

I confessed to my boss today that I was guilty of coding without a license. He asked me why I didn’t use the existing tool and I explained my reasons, and he said I should just add my tool to The Chosen Framework v2.0. This meant I had to explain that this would mean adding other tools to it, including the business objects and page objects that I had been asked to remove last year. He seemed confused and said he would address the issue in Some Very Important Discussions taking place next week among QA Director, the other managers and Principal SDET regarding the future of The Chosen Framework v2.0. I don’t think he realizes yet what a clusterfuck this whole thing has been. I would prefer to have nothing to do with The Chosen Framework v2.0 and continue building out my secret framework, but I know this is never going to happen.

The summer was a busy affair. It started out with a massive fail by the new organization that has taken over support for development tools and infrastructure. They officially took over maintenance of Stash in June and immediately buggered system the first time they did routine maintenance on it. For some reason, the most recent backup available was a month old. Thankfully, I had the most recent version of all my work saved locally, but some teams lost work. I am very happy to see that we are getting something that looks a lot like a real IT support staff, but these handoffs are a bitch. At the very least, before doing anything to the source control system, one should make a backup and then make a couple copies of that backup and store each of them in different locations. They handled the recovery process well, though, with good communication and organization of the effort to glue Humpty Dumpty together again.

As part of my job duties over the summer, I helped Junior SDET Who Should Be Senior mentor our summer intern from MIT. This was one of the most pleasant and enjoyable experiences I’ve had. It’s a lovely experience to work with one of the smart ones who learns quickly and is able to extrapolate abstract principles from a few basic examples and work independently. I highly recommend hiring MIT students as interns. I think a lot of them could easily drop out of school and start working as developers or even start their own companies and be wildly successful.

QA Director has been on everyone’s case about work from home policies and has lately taken up the cause of monitoring what we are doing on our computers at any given time. At one point, he asked Junior SDET Who Should Be Senior to talk to our intern about not playing games or watching any videos while he is in the office. QA Director claimed that ‘someone’ had complained to him about it, but I wouldn’t put it past him to lie about that so that he doesn’t come off as the ‘bad guy’ with a bad case of Hall Monitor Syndrome. I personally think that people who are productive and hit their milestones should be left alone, but I’m not the boss. Right after this, he claimed that ‘someone’ complained about the whole QA team being absent from the office on the first Friday after a giant release date. This was first and foremost, incorrect because I was in the office that day. I’m just too short for anyone to see my head over the top of my cube. Second, several people took vacation after that horrific release cycle, one of us couldn’t work that day because their new H1B visa wouldn’t be in effect until the following Monday, another of us was out sick and the rest were working from home on Friday because that was the day they had chosen for their one day a week allowance.

This of course, ignited a tense and difficult team meeting in which many of us expressed anger and resentment because developers seem to do whatever they want when they want and we are the only team that ever gets called to account for our whereabouts. QA Director became very defensive and sort of pissed at us and then he said he was going to tell the person who complained to mind their own business. Which is what he should have done in the first place. There’s nothing more demoralizing than a boss or director who won’t stand up for their team. Our team manager seems to be getting really tired of all this and has said he is not going to get into the habit of taking attendance because we are all grown adults and he would prefer to focus on the work and whether it is getting done or not. Although I am really pissed that my former boss was fired, I like the new manager a lot. He’s focused on the work and getting the developers to work together with us as a team, which is nice.

I found a senior developer role elsewhere in the company that is with one of the business units that is turning a profit. I have an informational interview with the hiring manager next week. Hopefully, this will be the right fit because it would be really great if I could continue working with the company in a part of it that is actually making some money. I really like the technology here and working with a core piece of it would be a great opportunity for me to gain some new experience and skills. I am still working on acquiring the knowledge and skills to get into data science or machine learning. I underestimated the amount of time that would take. There’s a lot of math. SO MUCH MATH. I built an Anki flashcard deck for the MIT discrete class I studied through MIT OCW. After finishing up the book and the lectures, I started using it study the material. For the first two weeks, it felt like it wasn’t doing anything for me, but now I have started to see cards which are easy for me to answer. The other benefit is that there are many proofs that I ‘know’, but understand more fully as a result of repeatedly reviewing them. Our summer intern is taking the class this fall, so I shared the deck publicly, and it appears to have taken off like wildfire because it has been downloaded 41 times so far. Hopefully, Anki will catch on at MIT and students will make decks for their other classes and post them publicly so I can use them myself.

I also took on the task of identifying the root cause of some TestNG defects related to parallel test execution. These defects have been making it impossible to implement the ability to log my tests to a separate log file by thread without advanced jujitsu configuration and hacking in Logback. If the tests would just freaking execute in the right thread in the expected fashion, this would be trivial, but all these random extra threads get spawned because a new thread worker is inappropriately spawned when you have test method dependencies. TestNG works as expected when test methods don’t depend on each other. Actually, group and test method dependencies are notoriously buggy when trying to use the various parallel execution modes offered by TestNG. It was quite a ride debugging that code base, but I found the bug in the source code and now I am trying to figure out how to fix it. It seems to be difficult to get a pull request accepted in TestNG (for good reason), but I want this bug fixed so bad, I am willing to jump through whatever hoops they want.

Basically, I have a lot of irons in the fire all at once. Hopefully, switching to a role in a profitable part of the company will put me in a less over-burdened role. I am sick of working on an absurdly understaffed team with people in positions of technical leadership who shouldn’t be there because they can’t code or understand basic software architecture design principles. Also, I really want to work for an organization that doesn’t treat me or my team like a bunch of middle schoolers who need a hall pass and a reason to be absent for a bathroom break.

Share This:

Wherein I Give Up Due To Fatigue and Not Giving a Shit Anymore

I am exhausted today. I have been consumed by a tide of work that a team twice our size would not be able to competently manage. It started with a monster new feature that can not be quickly tested via our functional UI regression suite. It requires real end-to-end verification. Normally, we only need to create configurations which use features with the web application for configuration management. For this feature, we have to verify that the configuration is applied correctly once it goes active. This involved a journey of discovery that highlighted the insane lack of coordination and communication between our particular organization and the rest of the company at large.

First, we did not know how to do this end-to-end testing because we have never done it before. Second, we were under the mistaken impression that there was no apparatus for doing this kind of testing in our two test environments. We believed this because we have been told repeatedly that there was no such thing since we were hired. The entire development team and their managers also believed this to be true. A co-worker and I began trying to master this unknown art of end-to-end testing by contacting the team which develops and tests that part of the network. During this first meeting in which the two of us probably sounded as intelligent as brain-damaged turkeys riding the short bus to school, we discovered that both of our test environments did in fact have the networks for doing this end-to-end testing. Not only that, they have had them for FIFTEEN YEARS. It’s impossible to understand that our entire organization could share the delusion that this part of the system was not present in either test environment for so long. I still don’t have an explanation for it. The only thing I can say is that inter-group and inter-department communication is abominable.

We also discovered that another business unit was nearly done with building a system which allows them to go to a website, fill out a simple data entry form and press a button which spins up test VMs on demand that function as a nice little sandbox testing environment for testing for this part of the end-to-end pipeline in isolation. It is well on its way to being part of a continuous deployment testing apparatus where there is little to no human intervention required to spin it up and do tests. I am astonished that my entire organization seemed to be under the delusion that this was an impossible pipe dream. The sheer embarrassment of talking to one team after another about their testing which is so much better than ours and their systems knowledge, which is so much more comprehensive than ours, and their test automation which is so much better than ours has broken something in me. It didn’t help I calculated that it was going to take a month to design, write and execute these test cases. A month in which I would not be writing any code.

Also weighing on me is the task of writing all the functional test cases for the entire UI. There is a hiring freeze and getting new requisitions is akin to an act of god. We have two principal-level employees who want nothing to do with this task. And who would? It’s tedious and unrewarding work to write functional UI test cases. Since I have been tagged as the person who couldn’t get any automation done for three years, I figured it was better to bite the bullet and be the SQA lead even if I am not getting paid to do that role. I’m not even getting paid enough to do my current role. This decision was the genesis for an unpleasant discovery regarding my company’s IT support infrastructure, which is that there really isn’t one. You’d think that a company with 6000 employees would have gotten around to hiring a 24/7 team to support and administer things like the build system, the source code control system and the defect tracking and project management systems. But, sadly, this is not true.

Our JIRA installation started out as a rebel application running off the director of engineering’s desktop computer because a cadre of new employees refused to use Bugzilla. Naturally, because there was no formal introduction of JIRA as a tool, there were no knowledgeable professionals who controlled the adoption. The result is an explosion of project templates with a ton of totally unnecessary custom fields and overly complicated workflows. An organization full of supposedly amazing engineers should be able to design a system that cleanly separates the work of product design, development, testing and customer support, but what we got was a system full of templates and workflows that was designed by a drunken squirrel on acid. Eventually, it got too big to be a rebel application running on the director of engineering’s desktop computer. It did not get big enough, however, to merit a 24/7 dedicated support team. It had gotten big enough to have a part-time support team who were all doing other jobs before they got stuck with the responsibility for managing and administering this monstrosity.

This is where I entered the picture with my quixotic notions of quickly configuring three projects — one for the APIs I wrote for our test automation framework, one for the domain models I have written using the APIs, and one for the QA team to use for defining and managing work related to testing the customer configuration application. I had dreams of using these three projects to define and manage the workload of writing all the formal test cases for the application and the catalog of work that must be done to automate them. This seemingly simple task has been going on since early February. I am in the final stages of getting the correct workflow defined for the Zephyr test tickets. This customized workflow has been most painful part of the process. For everything else, I was able to use the default, out-of-the-box Jira workflow and fields or the agile-style workflow and fields that the team building the new front end uses. The Zephyr test cases were another matter. I wanted a workflow that would accommodate test case design and development for manual tests and automated tests.

I wanted a design phase followed by a review phase. Then, if the test case is to be automated, there is a development phase followed by a code review phase, followed by a testing phase. If manual, the ticket is ‘complete’ after review. If an automated test case successfully gets through testing, it is ‘complete’. I also wanted a workflow that could accommodate sending the ticket back for re-work if review, code review or testing revealed issues as well as a workflow that could handle flagging a test case for needed updates because of changes in the behavior of the application. Granted, this is not a simple workflow design and it didn’t help that I did not initially understand the JIRA terminology or means of representing a workflow. The other problem was that the number of custom issue statuses and fields had completely gotten out of control during the wild west period for our JIRA installation. Apparently, this can cause serious performance degradation in JIRA, so they were no longer allowing any new statuses to be defined. I was thankfully able to choose good statuses from the list of existing ones. That fact that there are about four different statuses signifying that something is ready for development or testing just shows how important it is for a company to get out in front of the problem of building IT support infrastructure instead of chasing after the cowboys who just get tired of waiting for a modern tool chain.

In the midst of this mess, I also took on the task of implementing localization support for my page object library. Our California team requested this feature before Christmas. There are eleven possible languages that a user can select when browsing our customer portal. It took 3 weeks of late evenings and working weekends to pull off locale-agnostic page objects. I also chose to do a major refactor of the DSL API I had written because it badly needed to be simplified and scaled down. I think the system works well. Now all the page objects refer to enumerated constants which represent these localized identities. There is some under the hood tooling that they use for mapping locale-sensitive identifiers to the enumerated constants. It was the best I could do given the short time frame for building it. It was a huge task that burned me completely out.

So, I looking at this job and I have come to the conclusion that nothing is going to change. I have been complaining about the fact that the traditional SQA work and the automation work need to be separate roles since I joined this company nearly four years ago. Nothing has changed. I’m still stuck with work I am not interested in doing and having to burn the candle at both ends to do the work I actually want to do. So, in short, I am looking for a new job as we speak. I finally realized that this company which is supposedly full of amazing engineers is not actually interested in applying solid engineering principles to the test automation. They aren’t even interested in applying solid SQA practices because they severely under-resource the function to the point of absurdity. I keep thinking back to my first real SQA job with a medical device company. The size and scope of the application was smaller, yet we had two managers, four people on the backend and six people on the front end. That’s half again the size of my current team. When I think about this, I realized how totally delusional my company is about the work burden and the complexity of not only testing these applications, but also building robust test automation for them.

Share This:

Some More Reasons Why You Don’t Have Good Automation

It has been a while since our last edition of Why You Don’t Have Good Automation. I am hopeful today. On Monday, I did a presentation on the reasoning and purpose behind my detailed labeling of test cases. The gist of the presentation is that placing emphasis on end-to-end testing as if it were the only way to do automated testing was going to lead us to a limited return on the investment we put into writing all that sweet, sweet automation infrastructure. Once I replaced the older smoke test suite, I was happy to see the dev manager and my boss go all gooey-eyed over it, but I am bothered by the false sense of security that it gives. I am not saying that it is useless, but it has limited potential.

I am disillusioned where end-to-end testing is concerned. It is hideously difficult to build everything you need to implement the plumbing to even get to the point where you can do one end-to-end test. I have been through a couple iterations of this nightmare. The first time was during the Frameworks Wars when my co-worker and I toiled mightily to automate a suite of forty or so manual tests. There was just so much to do and we were slammed over and over again every few weeks with an avalanche of manual testing work for the application. We were also new to the company and not all that familiar with the surrounding, flaky systems that our application interacts with for each of these end-to-end scenarios. Combine this with our total lack of organized training with the some very unreasonable expectations, we were set up to fail and fail hard.

At the time, both my co-worker and I were operating under the same limited perspective as everyone else. End-to-end testing was where it was at and if you weren’t doing that, you were not doing Good Automation. So, we coded and coded, all the while being asked, “Where is all that sweet, sweet good automation?” We actually built a fairly extensive library of page objects which later became of the basis of the current page object library, but because we didn’t have the plumbing to get all the way from A to Z, it amounted to nothing in the eyes of the non-coding people around us. We made some mistakes such as thinking we could not throw our code out there unless it was ‘done’ and leaving the results reporting to the end. We should have had a parade up and down the hallways once a week, given tech talks and otherwise tooted our horns relentlessly. Our unpolished code was better than most of the test code I had seen up to that point. If we had a simple HTML report with green and red pass and fail indicators, I think the non-coding cadre would have believed we were actually doing something.

Our biggest mistake, however, was trying to do end-to-end testing first. If we had broken those long end-to-end test cases up into smaller, atomic test cases that could be strung together in end-to-end scenarios or run as standalone tests, we would have been able to report each week that we had automated twenty-five new test cases. It’s very easy to automate a test that verifies that a dialog opens when you click its accessor. It’s easy to automate a test that verifies that the title on the dialog is correct. It’s easy to automate a test that verifies that the dialog is dismissed if you close, cancel or submit it. Since our application had a gazillion dialogs, we could have written an entire test suite for verifying the basic behavior of dialogs. It’s not easy to verify that a given type of user can log in, access the appropriate accounts, create a configuration, fill out a complicated data entry form, save their configuration and then make it active. There are just so many interactions with the UI along the way, the chances that some flaky thing will happen to short-circuit the test are too high. Maybe one of the dialogs gets stuck, or a page gets stuck and doesn’t load. Given the sorry state of our test environments, this kind of problem arises every other test run, especially during periods of active and intense development. I won’t go into the details, but one of the systems that the application must interact with at the end of the long process for making a configuration go live is the bane of my existence. Imagine what it is like to kick off a test run that has a lower bound on it of ninety minutes to go from start to finish and it fails at the VERY. LAST. STEP. when that final sub-system in the whole pipeline chokes because of a common environmental issue.

The advantage of writing lots and lots of small tests that get into your app quick, verify a few small things and get out quick is that the chain of interactions with the UI is so much shorter. There are fewer opportunities for an unstable test environment to screw up your test results. And if a test does fail? WHO CARES. You just retry it a few times to see if it passes on a subsequent attempt. It’s easy to retry because this isolated little test lacks all those dependencies that a step in a fifty step end-to-end scenario would have. And why do you have to verify everything in one end-to-end chain? Why is it not sufficient to find a way to verify all the little steps in isolation? I’m not saying that you shouldn’t have some end-to-end test automation. There is a place for this kind of test automation. It’s just not terribly comprehensive. I know that sounds crazy. It’s an end-to-end test. By definition, that’s comprehensive, right? Nope. Nope. Nope. It’s anything but comprehensive. It represents one, very narrow and defined pathway through your system. If you want a comprehensive test suite and you want it to deliver results sometime in the next decade, you need a test suite that is composed of thousands of atomic little tests that are scattered across your system’s functionality.

This brings me back to the beginning — Why all these labels on the test cases? This is a way for categorizing all the tests so that it is easy to search for them. When you write a test suite of atomic tests, you will end up with hundreds, if not thousands of tests. Hence, you need to be able to easily find test cases. Second, these labels make it possible to dynamically specify at commit time which tests the developer would like to run. The end-to-end tests, given their limited coverage, long running time, and relative instability are not helpful for this scenario. The developer wants relevant, fast feedback. We also don’t have a giant Selenium grid at our fingertips, so we have to be conservative about how we use these resources. If there are eight commits in a day, the suite of end-to-end tests would overwhelm our resources very easily.

So, in short, if you are a regular worshipper at the idol of the end-to-end test automation god, you won’t have good automation. This god is a fickle liar who tucks you into bed at night with his minion, the Smoke Test Teddy Bear. Smoke Test Teddy Bear whispers sweet nothings into your ear, telling you if the end-to-end test suite passes, all is good. Don’t worry about the thousands of other possible user interactions with your application that it _didn’t_ cover. They are inconsequential seeds of evil doubt that the faithful believers must banish from their thoughts. Shut the closet door on those demons!

Share This:

In Which I Bitch and Moan a Bit More

First, I would like to bitch about C#. I finally translated the whole state Brixen Java bean package into C# and both the C# and Java versions are fully unit-tested. Yay! Second, C# needs something like Lombok _ASAP_. I did not fully understand The Giant Teh Suck So Much that it is to write all this freaking boilerplate for ToString(), Equals() and GetHashCode() over and over again until I enjoyed not doing it for Java classes for the last year. It’s glorious to open up a class file and Not. See. That. Shit. I am bummed there is no way to do the decorator pattern in C# as I have done it in Java. That little beauty of a design pattern _really_ cuts down on boilerplate. And Equals() for collections such as Dictionary, the equivalent of Map for Java, totally blows because it does reference equality, not ‘contains the same shit’ equality like Java Map implementations do. Seriously… what the fuck? Who thought that was a good idea? This took far far longer than I thought it would. I thought I would be done with the C# translation before Christmas and that I would be blazing through the Python implementation by now. Writing unit tests is tedious and time-consuming, but having caught some bugs with them, I am happy I have committed to doing them in tandem with the C# translation. But enough about that. Brixen marches onward and the source is available here. Looks like someone forked my repo a while back. I am flattered.

Today, I am _not_ gruntled. Far from it. You probably would have guessed that from the mere fact that there is a new post today. I had my annual performance review and the result was good, but not spectacular. I feel satisfied with this outcome and it is what I expected. This was a rocky year for me, and I did a few things which embarrassed the QA director. I publicly contradicted his goals and strategy on a fairly high-level initiative he was pushing, and I chewed out the Principal SDET over email for checking crappy page objects into the core of the automation framework without so much as a word with me over it. So, the chickens are coming home to roost for me. I have always had a mouth that got me into trouble before my rational brain could stop me. I have a habit of just expressing things in a way that others find too blunt and not diplomatic enough. I also have a temper.

Today was one of the days in which my rational brain was too slow out of the gate to catch up with my mouth. A junior developer is taking on the task of translating the regression suite I wrote to run against the new, totally rebuilt front end. I expected this task to be difficult for her. The regressions suite tests the application that has the biggest, most complicated UI in the entire customer portal. That shit is hard and complicated and the page object modeling is a non-trivial challenge. Today, she called a code review that included me, the Principal SDET and two other team members. In the midst of the code review, the question came up about labels I have attached to the test cases in JIRA which are reproduced on the test method as group names. I have a longer term plan for using those labels and the group name mappings as a way to dynamically select methods for specific functionality on the fly so that a developer can trigger a build that selects tests that cover the functionality they touched in their commit. Principal SDET claimed I had agreed to take the labels off the Jira test cases and the test methods. Problem is, I have near perfect memory for conversations going back months, and I knew that I had not agreed to do that. In fact, she and the QA director had backed down from it when it last came up. I disagreed with this assertion and before I knew it we were shouting at each other over the phone and she was refusing to explain why the presence of these labels was harmful to anyone.

After a chat with the QA director, I will be doing a presentation next week to explain why I did this and where it is going and then put it forth to the team to decide to chuck this or not. If they decide to chuck it, then I will copy all my work elsewhere and continue implementing a parallel test suite that uses this plumbing and present it to the development team when it is ready because this whole thing was meant to help them anyway. I have no idea where this came from and it is not the first time an issue which had been settled has been raised again from the dead where I am told I have to change how I did something without getting a clear explanation for why I need to. I am getting… weary of this. Principal SDET has a habit of honing in on trivial, non-architectural things which harm no one and making a giant fucking deal out of them. Test data storage formats are a giant bugaboo and apparently everything must and should be stored in a Java Properties file even though Java Properties files are flat, key/value pairs and there is no tooling I know of or that she can point me to which will allow me to suck in a Java Properties file containing serialized data that represents collections of objects and their state and instantiate collections of POJOs with it. At one point, the directory structure of the framework was a giant burning issue. If someone checked something into the wrong directory or tried to use Maven modules to manage multiple, related projects, they were not following ‘architectural’ guidelines. I have no idea what important architectural implications any of this has since they have NO EFFECT on how any particular subsystem interfaces with any other subsystem.

The code review barely touched on anything related to actual form and function of code. I don’t know if it is worth it in the long haul to stick around in this role. I like the challenge of the stuff I work on, but this perennial micromanagement and drama is getting really old.

Share This:

Long Time, No Post

My translation of the existing Java codebase for Brixen into C# continues to slowly move forward, hampered mostly by long hours for my job and the inevitable craziness of the holiday season. I have not yet encountered anything that I have been unable to duplicate in C#. I have learned how to implement extension methods and how to use generics in C# which is awesome.

I am struggling to replace an old smoke test suite for the product I support before I leave for Christmas vacation. I feel like every victory I have is rewarded with a new and difficult obstacle. Every time I successfully clear one hurdle and need to take the test suite to the next level, there is some new, difficult problem to solve that takes a week to deal with. Meanwhile, the QA Director keeps asking, “Where is that sweet sweet automation you promised us!” It’s hard to explain to people who don’t code the difficulties and challenges of re-tooling something to get around things like defects in the open source tools you’re using, or building out infrastructure to support things like configurable suite options and test data feeding. It always sounds like I’m making excuses for my pitiful lack of amazing progress in producing Teh Automation Suite of Amazingness and Wow-dom.

One thing that annoys the shit of me is that it seems that all prior progress I have made is forgotten as soon as a week goes by without new test cases getting automated. It doesn’t matter that I managed to build infrastructure that allows me to execute the exiting test cases across multiple scenarios as opposed to one. It doesn’t matter I had to stop and produce something for the test automation framework that we are trying to shove down the throats of the other business units. It doesn’t matter that I had to implement a work around for a TestNG defect. All that matters is that new test cases were not automated that week. The fact that I have automated almost 75 percent of the test cases covered by the old test suite I am replacing in addition to automating test cases that the old suite didn’t cover is quickly forgotten.

The QA director has taken to the habit of stopping by my desk and publicly ‘joking’ that I emailed him the night before to tell him that I was already done with the smoke test. Or that I told him I would be done early. Nagging that is disguised as good-natured teasing is more annoying that direct nagging could ever be. Seriously, it just makes me want to spend the next hour Christmas shopping on Amazon for my kids out of spite. The agreed upon deadline for finishing the smoke test replacement effort was still TWO WEEKS AWAY when the nagging began. It doesn’t help that we have all been told to expect lower bonuses this year and that my bonus is riding largely upon the successful completion of this project before I leave on vacation. It isn’t worth working the kind of hours I do to put up with that crap. Plus, all this overtime and stress is getting in the way of moving forward with Brixen which is far more interesting and fun to work on.

Another giant pain in my ass is that the principal SDET and Architect of the automation framework we are trying to shove down the throats of the other business units tried to quietly sneak in some page objects from her old framework into the new one because her junior engineer didn’t want to spend 10 minutes asking me some questions about how to follow the far superior approach I busted my ass to develop. Instead of making a real effort to convert to the new framework, he just wanted to be lazy and just do it the way he had gotten used to doing it. I tried to point out the irony of our effort to make the other business units switch over to our way of doing things while allowing an engineer on our team to avoid doing exactly that, but the principal SDET played dumb and pretended that my point was totally incomprehensible. This lead to a demand for a code review of the core classes in the page object API by the principal SDET.

I wrote a few Wiki instructions for how to use the core classes. For example usages of the API, I used the page objects I had built for the customer portal to support my automation. At one time in the recent past, I had advocated that the API as well as the page objects built with it should both be part of the framework, but the principal SDET insisted that only the API should be in it. So, at her insistence, I moved them out of the framework project. During the code review, the members of the West Coast team who were taking part interrupted me as I was in the process of explaining some of the examples on the Wiki page to ask, “This LoginPage class you show here…. where is it?” So, we had an unscheduled discussion about the new features they wanted in the the LoginPage class and the other page object classes for navigating through the customer portal. I would feel more smugly satisfied if I had not just signed myself up for even more work.

Share This:

Fumbling Around Like a Total N00b: Translating Brixen from Java to C#

I spent a week scratching my head over failed unit tests that were ultimately the result of jackass n00b mistakes, and sadly, they were not C# n00b mistakes, but Coding 101 n00b mistakes like not checking for null and failing to actually set the value of a property after validating the parameter for its setter method. Oh, and leaving off the parentheses for a method call. Thankfully, I didn’t turn to Stack Overflow for help. My reputation would have taken a severe nosedive for requesting help with such embarrassingly basic bugs. And this, my friends, is why you should write unit tests. It saves you from checking dumbass Coding 101 mistakes like this into source control where they can be found by your co-workers who might be tempted to slowly freeze you out from the important and rewarding work because they just can’t trust you to write a decent for-loop without a user’s manual at your side.

I still haven’t figured out how to write an integrated build script for Brixen that would compile, test and deploy both brixen-java and brixen-dotnet. I did however, take the time to learn some basic Nunit framework usage so that I could actually write some unit tests. I also tried to integrate Nlog logging into the unit tests, but for some reason, the logging statements don’t get output to the console in Xamarin Studio, so I took it out. I think I need to try this on a Windows system with Visual Studio.

When I started coding, I realized I had no idea how a standard .NET project should be organized. I went in search of a guide and found that finding such a thing was not all that easy. It’s as if there is this pervasive assumption in the .NET coding community that you should just know how to structure a .NET project by osmosis. With Maven, there is a well-documented project structure and there’s generally no question where something should live. Java enforces a directory structure convention for packages which does not exist in the C# world. YouTube came to my rescue in the form of a short, awesome video tutorial where the creator of the video explained that these things are in fact often hard to figure out because they are not explicitly documented by Microsoft.

After basically learning the basics of a totally different tool chain, I finally got something that I could check into the Brixen GitHub repo that I was not ashamed to call my own. So far, I’ve only translated a handful of the state beans into C#, but things should move faster now that I have surmounted some of the learning curve. The code is definitely going to be more verbose than the Java version due to the lack of a tool like Lombok. I have to write implementations of ToString(), GetHashChode() and Equals(Some Object) which is tedious. In Java, I can just throw a couple annotations on a class and these methods are auto-generated at compile time by Lombok. I’ve updated the corresponding Java classes too. In the source code I presented at the Selenium conference, I did not do any parameter validation for the setter methods because I wanted to cut down on the sheer volume of code I had to present. I am now adding back in the parameter validation that I cut out because it’s just a good idea to validate parameters. So, the Java code is more verbose than the code from my presentation.

The Decorator translation is a little harder to pull off in C# because I have to use extension methods to produce something like the default method implementations that Java 8 allows, but at least, thankfully, it is possible to do. I treated myself to the pleasure of reading explanations of why Java 8 was just totally wrong for allowing this atrocity into the language because it violates the very definition of an interface. This may be true, but default interface methods are just astoundingly convenient in ways that justifies the atrocity in my opinion.

Share This:

Learning a Foreign Language: Translating Brixen From Java to C#

I started translating the existing codebase for Brixen from Java into C#. I mistakenly thought this would be a quick and easy task because the syntax of C# appeared to be so similar to Java. Syntax Schmintax. I didn’t realize how fundamentally Java had arranged my thinking around How Development Works. My first jolt was discovering how bonkers I was for thinking that it would be a breeze to do this. I was thinking that all I would need to do is install a plugin for IntelliJ and I would be golden. If I couldn’t do that, surely I could just use Eclipse. Fast forward two weeks, skip over a lot of cursing and forehead slapping, and I am using Xamarin Studio. It took a while, but I came around to accepting that you just don’t mix C# and Java in the same IDE. It’s. Just. Not. Done. I also eventually accepted that C# isn’t platform independent, and developing in C# on a Mac is kind of whack.

My second more unpleasant jolt was that Maven doesn’t support C#. This one was a lot harder to accept. I thought it would just be a matter of adding some new dependencies in the POM for Brixen, adding a build plugin or two and I’d be good to go. I would have brixen-java and brixen-dotnet playing all nicely together and all I needed to do is type ‘mvn install’ and shit would just work. It was really hard, but I finally accepted that this isn’t how it’s going to be. I still have no idea how to do the build and deploy thing for the C# packages, but I decided to defer that task while I try to learn the C# language.

There’s no Lombok for C#, but there are these things called properties which sort of do the same thing. The naming conventions in C# are whack. Seriously?! PASCAL IS SO OVER! Interfaces can’t have fields?! The languages are similar enough on the surface that it’s not too difficult to translate from one to the other, but they are different enough that I am regularly irritated and pissed off in some fashion. It’s like a rash that isn’t bad enough to make you go see a doctor, but still annoying enough that you really really want a steroid shot JUST TO MAKE IT STOP ITCHING. I just have to say to all you people who love C# — WHY? Why do you love it?

Share This:

Yet More Reasons Why You Don’t Have Good Automation

It has been a terrible two weeks for me. I lost my last living grandparent and my workplace drama culminated in the latest episode of humiliating and ham-fisted attempts to micromanage my work because The People Who Matter can’t seem to grasp the reality that test automation is really really hard.

I am demoralized today. I have hesitated to write about the ‘people’ element involved in developing a robust automation framework because this tends to be volatile and subjective content, but my work week totally blew, so I am going to do what has become a time-honored tradition since The InterWubs became a mainstream pastime — I am going to bitch about it online and talk some shit. When I joined my current workplace about three years ago, the state of UI automation was less mature than it is now. There were a lot of teams who were just starting out and there were some teams who had more mature tooling.

At this time, a complete overhaul of a big system was also just beginning, which meant disparate teams would have to collaborate together in ways that had never occurred at this company before. There was a sense of urgency about getting this ‘Automation’ thing going because there was so much to test and manual regression testing is slow and expensive. The various teams of SQA engineers were trying to get a handle on the automation and some were succeeding, but many teams were not. I was newly hired and excited about finally getting a job where I could spend more time writing code than running the same old boring manual regression tests over and over again.

I was assigned to a complex web application that saddles many domains across the company. Both the UI and the data/business logic aspects of this application are complex. It’s the kind of meaty automation challenge that gets a weirdo nerd like me pumped up and ready to code. My boss had recruited a team of senior employees and we wrote a large suite of test cases for this application. Then we started to investigate how we could approach the automation. We decided to go with the Page Object Model design pattern that is the gold standard for Selenium UI test automation. We evaluated some other frameworks, but, surprisingly, none of them were actually using this design pattern, so we started writing our own framework. Although I am biased, I will state that the code was sophisticated and high-quality stuff that was up to the task of being a great API for writing automated tests against the rich and complex UI of the application we were responsible for.

Then along came a high-level ‘architect’ who decided that everyone had to use the same ‘framework’. I won’t go into the ridiculous lack of consensus about what the term ‘framework’ meant because it’s a distraction. Let’s just say, there was no consensus on what this term meant. He took a look at the some of the various tools that some teams had built which they called ‘frameworks’ and then decided to declare one of them the ‘winner’. Then he decreed that all teams must use the one he had declared the winner. On the surface, this isn’t necessarily a bad idea and if the person making the decision actually knew how to evaluate the candidate frameworks, this could have ended well. The losers would have felt slighted and upset, naturally, but technical people generally can be persuaded to accept standards and tools they didn’t originally choose or build for themselves if they are good.

The key ingredient in making the above scenario end well is the word ‘good’. In my opinion, the following things must be true for a test automation framework to be good:

  1. It should be built with industry standard tools, not someone’s pet bleeding edge thing that is new and sexy, but not mature and certainly not in widespread use
  2. It should be built with tools that you can easily recruit talent to use and extend, not tools which have few experts you have to struggle to find and hire
  3. It should be built with tools that your current employees are mostly familiar with
  4. It should be built with tools that are well-documented and well-supported
  5. It should be built with a well-designed and properly articulated architecture so that its components are maintainable, reusable, and scalable
  6. The separate components of the system should be cleanly separated and use APIs to communicate with each other
  7. There should be no hardened dependencies on specific test management or defect management systems

Now that we have this settled, let’s return to the high-level ‘architect’ who decided to require all the QA teams to use the same automation framework. A person without good programming or software design skills can handle the first four items on the list. The last three items need to be evaluated by someone who does. Guess which category homeboy belonged to.

The framework he declared the ‘winner’ was just Not. Very. Good. It was actually one of the frameworks my team evaluated when we were trying to figure out what we were going to do with our automation work. We quickly passed on this one because it was very poorly designed and had none of the characteristics we were looking for. It didn’t help that the person who wrote it clearly did not have much familiarity with the WebDriver API. When I investigated how other teams were using it after Sir Architect declared it the winner, I noticed that a lot of them included it as a dependency in their POMs, but didn’t actually use anything in it. I think this was mostly because the design was so bad that it was nearly impossible to understand unless you methodically walked through every line of code. I eventually did this myself, and I have to say, I am a stronger person for the experience.

My co-worker and I couldn’t bear to dump our system which worked great for us and adopt this other system which totally didn’t work and would require us to shoehorn some ugly-ass spaghetti code into our test code in order to ‘use’ a framework that did absolutely nothing for us. We went on writing our page objects and incrementally automating some steps in our suite of manual end-to-end tests. Meanwhile, my boss decided to become an individual contributor and a member of our team who never even wanted to manage was given responsibility for our whole team. She had recently been hired and had not even had time to settle in. After she took over, Sir Architect kept asking if we were doing our automation using The Chosen Framework.

I should stop here and explain that our new manager, unlike the one who decided to stop managing, was not someone who had strong programming and software design experience. So, she put a lot of pressure on us to deliver lots of real good automation real quick. The problem with people who don’t ‘do’ software is that they tend to think that test automation is easy and when they hear the word ‘framework’ they think, “Magic system which shits a pound of automation gold every hour with little to no effort and so little technical expertise required that even a preschooler could automate the whole internet with it.”

We tried to explain that in order to automate the regression suite for our application, we needed to build models for all the UI components so that we could interact with them programmatically. We also tried to explain that the data and business logic domain was also something that needed to be modeled so that we could define and feed test data into automated tests instead of hard-coding it right there in the test methods. We tried explaining that The Chosen Framework contained some badly designed mechanism for getting a reference to a WebDriver instance of some desired type and that was basically all there was. All the rest of the necessary tooling, namely the models, were not provided with The Chosen Framework. Not only that, the Chosen Framework required that we use a ‘base’ page object which seemed to be super confused about whether it was a generic page object or a misguided effort to wrap some basic WebDriver functionality in a way that was inferior to WebDriver’s own native interface.

It. Just. Did. Not. Sink. In. Part of the problem was that other teams had managed by some miracle to tease some automated tests out of The Chosen Framework. We couldn’t seem to make our manager or Sir Architect understand that these tests didn’t do very much and that they were for very simple applications with maybe a page or two with a very limited data domain. Our application is not that type of application. It has lots of dynamic UI elements, numerous dialogs, menus, and data entry forms along with some complicated multistep state changes that have to be tracked and verified along the way. Not only that are a lot of validation rules on the data that are in play as well as a complicated user and role access permission domain to make it even more fun. And then there is the collection of features and behavior related to migrating customer artifacts from the older application that our application is supposed to replace. I also cannot neglect to mention the oh so very special test environments which have been featured in earlier episodes of “Why You Don’t Have Good Automation”.

A subset of these complications should be enough to send any self-respecting automation engineer over the edge, but throw in the need to periodically explain your failure to deliver some sweet, sweet automation for this mess to people who just refuse to understand that a badly designed mechanism for getting a WebDriver instance is not going to result in Automation Jesus marching triumphantly down our cubical-lined aisles and you have yourself a heaping helping of Soul-Destroying Is This Really My Life Cognitive Dissonance. We had a test suite of some 40 manual tests with roughly 15-30 steps apiece that we were running on two versions of IE, the latest version of Firefox on Windows 7 and OSX, the latest version of Chrome on Windows 7 and OSX in addition to Safari. At one point, our manager just declared that if we could not complete the automation for this test suite in three weeks, we had to dump our work and switch over to The Chosen Framework. This was after saying time and again, that the Chosen Framework provided no interfaces for interacting with the UI, nor did it provide any efficiencies for building the interfaces.

Fast forward two years. My team falls back under my original boss who has decided to manage again. My team now includes the person who built The Chosen Framework who has been promoted to Principal SDET. It is announced that we will collaborate with some other QA engineers in our organization to built a new framework and the designer of The Chosen Framework, V1, will ‘architect’ it. We spend a year and a half building various components while our Principal SDET regularly becomes very concerned about the directory structure of the repositories and the very large amount of code which some people are writing because large amounts of code is ‘heavy’ and that is bad. Oh, and ‘the communication protocol will be a POM dependency’. Somehow we write something that some of can use and there is an effort under way by our upper management to shove our framework down the throats of every other business unit in the company. A framework which none of them helped build or design and does not include anything from their now mature and fully-featured frameworks that are mostly better than ours. You can imagine how fast they all are running from the directive to adopt The Chosen Framework, V2.

I have finally been given space and time to do the automation after two years of relentless and never-ending manual testing stints all the while hearing, “Where is all that good automation?!!!! We can’t afford to hire manual testers. Just do it in your spare time!” Now I have about 200 automated tests of which only a couple are flappers which need some tweaking to make stable. Nevertheless, the new QA director has decided that we need to ‘start making some progress’ and a close working relationship and some coding ‘help’ from the creator of The Chosen Framework, V1, is just the solution to this problem.

It almost made me want to quit and take up goat-herding as a profession after pouring acid over my laptop and taking a shit right on top of my workstation out of spite.

So, folks, this is the latest edition of “Why You Don’t Have Good Automation.” Automation frameworks are a serious technical challenge and you need to put technically sophisticated people in charge of it. Also, don’t try to shove a tool down the throats of people who know better than you do what they need. If you’re looking to have your entire company on a single framework, you need to involve all the teams in requirements gathering. You also need to incorporate the best of their tooling into it. Also, if you don’t want to drive your automation staff into the arms of your competitors, don’t keep asking why they aren’t done every two days.

Share This:

Brixen Decorators: What’s Changed

The basic concept of the decorators hasn’t changed. There are just more of them. I have added decorators for state beans, configuration beans and builders in addition to the decorators for the component implementations. The decorators help reduce boilerplate code by using a new feature introduced in Java 8: default interface methods. A class can implement numerous interfaces, but extend only one class. Prior to Java 8, this meant implementing the same methods over and over again in different classes extending a particular interface. After Java 8 however, a decorator interface can extend another interface and provide default method implementations for the interface it extends.

Here is an example of the PolleableBeanDecorator, which provides default method implementations for the methods required by PolleableBean:

The decorator defers all the PolleableBean method calls to the default PolleableBean implementation — PolleableBeanImpl. This is achieved by requiring a ‘provider’ for an internal reference to a PolleableBeanImpl instance. So any class which implements PolleableBeanDecorator needs to define an accessor method for this provider. Let’s have a look at the provider for a state bean:

I didn’t want to have a method in a decorator interface that would return a reference to the internal state bean itself because that would break encapsulation. So, I came up with the idea of a ‘provider’ which would give protected access to this internal state bean reference so that only a sub-class or a class within the same package as the provider would be able to access the internal state bean reference. The packaging structure I chose allows only the provider and its sub-classes, the decorator and other classes in their package to access the provider’s internal state bean reference.

Here is an example of a state bean which extends PolleableBeanDecorator:

DynamicControllableBean is a state bean for specifying a page object that contains one or more web controls which needs to be polled for state change of some kind, usually after an interaction with one of its controls. A drop down menu, for example, would need to be polled after interacting with the control that expands or collapses it to determine if the menu has expanded or collapsed as expected. The default implementation of DynamicControllableBean can only extend a single class, so its parent is ControllableBeanImpl.

It would be a drag to have to provide implementations for the methods required by PolleableBean which are exactly the same as the implementations provided in PolleableBeanImpl. By implementing the PolleableBeanDecorator interface, DynamicControllableBeanImpl can satisfy all the requirements of PolleableBean by providing only an accessor to a LoadableBeanProvider that wraps an instance of PolleableBeanImpl. Lombok helps reduce the amount of source code even more because the provider field only has to be annotated with the Getter annotation.

The decorators for the configuration beans operate in the same way. Here is the PolleableConfigDecorator and the provider for a configuration bean:

And here is DynamicControllableConfigImpl, a configuration bean which implements PolleableConfigDecorator:

Decorators for the builders was a bit trickier to pull off, but I managed to find a way. Here is the PolleableBuilderDecorator:

And AbstractDynamicControllableBuilder, which implements it:

This decorator implementation works by declaring a provider which wraps the builder implementing the decorator. The same state bean must encapsulate all the state for the component, so declaring a builder with a separate state bean instance wouldn’t work.

The decorators for the components haven’t changed, so I won’t post any examples here since it would just duplicate what I presented at the conference. The source code for Brixen is here.

Share This:

Brixen Configuration Beans: What’s Changed

The changes in the configuration bean package are mirrored largely by the changes in the state beans and the builders. The other change is that I added JSON type information to all of the configuration bean interfaces. When I originally conceived of the configuration bean idea, I foolishly didn’t consider the possibility that some configuration beans may contain other configuration beans as fields. So, I only added the type information to the LoadableConfig bean. Without this type information, Jackson cannot properly deserialize polymorphic types.

The other change is that LoadableConfig allows the definition of custom properties, through JsonAnyGetter and JsonAnySetter methods. It is entirely conceivable that one might want to define a configuration option for a page object which doesn’t have general significance to a class of page objects, but which is important for a specific context:

The ControllableConfig is the most significant new addition. This configuration bean is for defining the dynamically configurable options for a page object which contains web controls. This configuration also encapsulates the configurable options for each one of its controls:

There is a marker interface for a dynamic Controllable that needs to be polled on intervals for a state change via a FluentWait:

Here is an example of what the configuration source for a ControllableConfig would look like:

In the same fashion as the state bean and builders, there is a marker interface that is parent to all the control configuration beans:

The marker interface for a click control configuration bean:

It extends ClickableConfig which is a configuration bean for a wider class of clickable page objects besides controls:

The configuration bean for a hover control:

The configuration bean for a hover and click control:

And that’s a wrap for the changes in the configuration beans. The source for Brixen is available here.

Share This: