Category: Selenium

Some More Reasons Why You Don’t Have Good Automation

It has been a while since our last edition of Why You Don’t Have Good Automation. I am hopeful today. On Monday, I did a presentation on the reasoning and purpose behind my detailed labeling of test cases. The gist of the presentation is that placing emphasis on end-to-end testing as if it were the only way to do automated testing was going to lead us to a limited return on the investment we put into writing all that sweet, sweet automation infrastructure. Once I replaced the older smoke test suite, I was happy to see the dev manager and my boss go all gooey-eyed over it, but I am bothered by the false sense of security that it gives. I am not saying that it is useless, but it has limited potential.

I am disillusioned where end-to-end testing is concerned. It is hideously difficult to build everything you need to implement the plumbing to even get to the point where you can do one end-to-end test. I have been through a couple iterations of this nightmare. The first time was during the Frameworks Wars when my co-worker and I toiled mightily to automate a suite of forty or so manual tests. There was just so much to do and we were slammed over and over again every few weeks with an avalanche of manual testing work for the application. We were also new to the company and not all that familiar with the surrounding, flaky systems that our application interacts with for each of these end-to-end scenarios. Combine this with our total lack of organized training with the some very unreasonable expectations, we were set up to fail and fail hard.

At the time, both my co-worker and I were operating under the same limited perspective as everyone else. End-to-end testing was where it was at and if you weren’t doing that, you were not doing Good Automation. So, we coded and coded, all the while being asked, “Where is all that sweet, sweet good automation?” We actually built a fairly extensive library of page objects which later became of the basis of the current page object library, but because we didn’t have the plumbing to get all the way from A to Z, it amounted to nothing in the eyes of the non-coding people around us. We made some mistakes such as thinking we could not throw our code out there unless it was ‘done’ and leaving the results reporting to the end. We should have had a parade up and down the hallways once a week, given tech talks and otherwise tooted our horns relentlessly. Our unpolished code was better than most of the test code I had seen up to that point. If we had a simple HTML report with green and red pass and fail indicators, I think the non-coding cadre would have believed we were actually doing something.

Our biggest mistake, however, was trying to do end-to-end testing first. If we had broken those long end-to-end test cases up into smaller, atomic test cases that could be strung together in end-to-end scenarios or run as standalone tests, we would have been able to report each week that we had automated twenty-five new test cases. It’s very easy to automate a test that verifies that a dialog opens when you click its accessor. It’s easy to automate a test that verifies that the title on the dialog is correct. It’s easy to automate a test that verifies that the dialog is dismissed if you close, cancel or submit it. Since our application had a gazillion dialogs, we could have written an entire test suite for verifying the basic behavior of dialogs. It’s not easy to verify that a given type of user can log in, access the appropriate accounts, create a configuration, fill out a complicated data entry form, save their configuration and then make it active. There are just so many interactions with the UI along the way, the chances that some flaky thing will happen to short-circuit the test are too high. Maybe one of the dialogs gets stuck, or a page gets stuck and doesn’t load. Given the sorry state of our test environments, this kind of problem arises every other test run, especially during periods of active and intense development. I won’t go into the details, but one of the systems that the application must interact with at the end of the long process for making a configuration go live is the bane of my existence. Imagine what it is like to kick off a test run that has a lower bound on it of ninety minutes to go from start to finish and it fails at the VERY. LAST. STEP. when that final sub-system in the whole pipeline chokes because of a common environmental issue.

The advantage of writing lots and lots of small tests that get into your app quick, verify a few small things and get out quick is that the chain of interactions with the UI is so much shorter. There are fewer opportunities for an unstable test environment to screw up your test results. And if a test does fail? WHO CARES. You just retry it a few times to see if it passes on a subsequent attempt. It’s easy to retry because this isolated little test lacks all those dependencies that a step in a fifty step end-to-end scenario would have. And why do you have to verify everything in one end-to-end chain? Why is it not sufficient to find a way to verify all the little steps in isolation? I’m not saying that you shouldn’t have some end-to-end test automation. There is a place for this kind of test automation. It’s just not terribly comprehensive. I know that sounds crazy. It’s an end-to-end test. By definition, that’s comprehensive, right? Nope. Nope. Nope. It’s anything but comprehensive. It represents one, very narrow and defined pathway through your system. If you want a comprehensive test suite and you want it to deliver results sometime in the next decade, you need a test suite that is composed of thousands of atomic little tests that are scattered across your system’s functionality.

This brings me back to the beginning — Why all these labels on the test cases? This is a way for categorizing all the tests so that it is easy to search for them. When you write a test suite of atomic tests, you will end up with hundreds, if not thousands of tests. Hence, you need to be able to easily find test cases. Second, these labels make it possible to dynamically specify at commit time which tests the developer would like to run. The end-to-end tests, given their limited coverage, long running time, and relative instability are not helpful for this scenario. The developer wants relevant, fast feedback. We also don’t have a giant Selenium grid at our fingertips, so we have to be conservative about how we use these resources. If there are eight commits in a day, the suite of end-to-end tests would overwhelm our resources very easily.

So, in short, if you are a regular worshipper at the idol of the end-to-end test automation god, you won’t have good automation. This god is a fickle liar who tucks you into bed at night with his minion, the Smoke Test Teddy Bear. Smoke Test Teddy Bear whispers sweet nothings into your ear, telling you if the end-to-end test suite passes, all is good. Don’t worry about the thousands of other possible user interactions with your application that it _didn’t_ cover. They are inconsequential seeds of evil doubt that the faithful believers must banish from their thoughts. Shut the closet door on those demons!

Share This:

In Which I Bitch and Moan a Bit More

First, I would like to bitch about C#. I finally translated the whole state Brixen Java bean package into C# and both the C# and Java versions are fully unit-tested. Yay! Second, C# needs something like Lombok _ASAP_. I did not fully understand The Giant Teh Suck So Much that it is to write all this freaking boilerplate for ToString(), Equals() and GetHashCode() over and over again until I enjoyed not doing it for Java classes for the last year. It’s glorious to open up a class file and Not. See. That. Shit. I am bummed there is no way to do the decorator pattern in C# as I have done it in Java. That little beauty of a design pattern _really_ cuts down on boilerplate. And Equals() for collections such as Dictionary, the equivalent of Map for Java, totally blows because it does reference equality, not ‘contains the same shit’ equality like Java Map implementations do. Seriously… what the fuck? Who thought that was a good idea? This took far far longer than I thought it would. I thought I would be done with the C# translation before Christmas and that I would be blazing through the Python implementation by now. Writing unit tests is tedious and time-consuming, but having caught some bugs with them, I am happy I have committed to doing them in tandem with the C# translation. But enough about that. Brixen marches onward and the source is available here. Looks like someone forked my repo a while back. I am flattered.

Today, I am _not_ gruntled. Far from it. You probably would have guessed that from the mere fact that there is a new post today. I had my annual performance review and the result was good, but not spectacular. I feel satisfied with this outcome and it is what I expected. This was a rocky year for me, and I did a few things which embarrassed the QA director. I publicly contradicted his goals and strategy on a fairly high-level initiative he was pushing, and I chewed out the Principal SDET over email for checking crappy page objects into the core of the automation framework without so much as a word with me over it. So, the chickens are coming home to roost for me. I have always had a mouth that got me into trouble before my rational brain could stop me. I have a habit of just expressing things in a way that others find too blunt and not diplomatic enough. I also have a temper.

Today was one of the days in which my rational brain was too slow out of the gate to catch up with my mouth. A junior developer is taking on the task of translating the regression suite I wrote to run against the new, totally rebuilt front end. I expected this task to be difficult for her. The regressions suite tests the application that has the biggest, most complicated UI in the entire customer portal. That shit is hard and complicated and the page object modeling is a non-trivial challenge. Today, she called a code review that included me, the Principal SDET and two other team members. In the midst of the code review, the question came up about labels I have attached to the test cases in JIRA which are reproduced on the test method as group names. I have a longer term plan for using those labels and the group name mappings as a way to dynamically select methods for specific functionality on the fly so that a developer can trigger a build that selects tests that cover the functionality they touched in their commit. Principal SDET claimed I had agreed to take the labels off the Jira test cases and the test methods. Problem is, I have near perfect memory for conversations going back months, and I knew that I had not agreed to do that. In fact, she and the QA director had backed down from it when it last came up. I disagreed with this assertion and before I knew it we were shouting at each other over the phone and she was refusing to explain why the presence of these labels was harmful to anyone.

After a chat with the QA director, I will be doing a presentation next week to explain why I did this and where it is going and then put it forth to the team to decide to chuck this or not. If they decide to chuck it, then I will copy all my work elsewhere and continue implementing a parallel test suite that uses this plumbing and present it to the development team when it is ready because this whole thing was meant to help them anyway. I have no idea where this came from and it is not the first time an issue which had been settled has been raised again from the dead where I am told I have to change how I did something without getting a clear explanation for why I need to. I am getting… weary of this. Principal SDET has a habit of honing in on trivial, non-architectural things which harm no one and making a giant fucking deal out of them. Test data storage formats are a giant bugaboo and apparently everything must and should be stored in a Java Properties file even though Java Properties files are flat, key/value pairs and there is no tooling I know of or that she can point me to which will allow me to suck in a Java Properties file containing serialized data that represents collections of objects and their state and instantiate collections of POJOs with it. At one point, the directory structure of the framework was a giant burning issue. If someone checked something into the wrong directory or tried to use Maven modules to manage multiple, related projects, they were not following ‘architectural’ guidelines. I have no idea what important architectural implications any of this has since they have NO EFFECT on how any particular subsystem interfaces with any other subsystem.

The code review barely touched on anything related to actual form and function of code. I don’t know if it is worth it in the long haul to stick around in this role. I like the challenge of the stuff I work on, but this perennial micromanagement and drama is getting really old.

Share This:

Fumbling Around Like a Total N00b: Translating Brixen from Java to C#

I spent a week scratching my head over failed unit tests that were ultimately the result of jackass n00b mistakes, and sadly, they were not C# n00b mistakes, but Coding 101 n00b mistakes like not checking for null and failing to actually set the value of a property after validating the parameter for its setter method. Oh, and leaving off the parentheses for a method call. Thankfully, I didn’t turn to Stack Overflow for help. My reputation would have taken a severe nosedive for requesting help with such embarrassingly basic bugs. And this, my friends, is why you should write unit tests. It saves you from checking dumbass Coding 101 mistakes like this into source control where they can be found by your co-workers who might be tempted to slowly freeze you out from the important and rewarding work because they just can’t trust you to write a decent for-loop without a user’s manual at your side.

I still haven’t figured out how to write an integrated build script for Brixen that would compile, test and deploy both brixen-java and brixen-dotnet. I did however, take the time to learn some basic Nunit framework usage so that I could actually write some unit tests. I also tried to integrate Nlog logging into the unit tests, but for some reason, the logging statements don’t get output to the console in Xamarin Studio, so I took it out. I think I need to try this on a Windows system with Visual Studio.

When I started coding, I realized I had no idea how a standard .NET project should be organized. I went in search of a guide and found that finding such a thing was not all that easy. It’s as if there is this pervasive assumption in the .NET coding community that you should just know how to structure a .NET project by osmosis. With Maven, there is a well-documented project structure and there’s generally no question where something should live. Java enforces a directory structure convention for packages which does not exist in the C# world. YouTube came to my rescue in the form of a short, awesome video tutorial where the creator of the video explained that these things are in fact often hard to figure out because they are not explicitly documented by Microsoft.

After basically learning the basics of a totally different tool chain, I finally got something that I could check into the Brixen GitHub repo that I was not ashamed to call my own. So far, I’ve only translated a handful of the state beans into C#, but things should move faster now that I have surmounted some of the learning curve. The code is definitely going to be more verbose than the Java version due to the lack of a tool like Lombok. I have to write implementations of ToString(), GetHashChode() and Equals(Some Object) which is tedious. In Java, I can just throw a couple annotations on a class and these methods are auto-generated at compile time by Lombok. I’ve updated the corresponding Java classes too. In the source code I presented at the Selenium conference, I did not do any parameter validation for the setter methods because I wanted to cut down on the sheer volume of code I had to present. I am now adding back in the parameter validation that I cut out because it’s just a good idea to validate parameters. So, the Java code is more verbose than the code from my presentation.

The Decorator translation is a little harder to pull off in C# because I have to use extension methods to produce something like the default method implementations that Java 8 allows, but at least, thankfully, it is possible to do. I treated myself to the pleasure of reading explanations of why Java 8 was just totally wrong for allowing this atrocity into the language because it violates the very definition of an interface. This may be true, but default interface methods are just astoundingly convenient in ways that justifies the atrocity in my opinion.

Share This:

Learning a Foreign Language: Translating Brixen From Java to C#

I started translating the existing codebase for Brixen from Java into C#. I mistakenly thought this would be a quick and easy task because the syntax of C# appeared to be so similar to Java. Syntax Schmintax. I didn’t realize how fundamentally Java had arranged my thinking around How Development Works. My first jolt was discovering how bonkers I was for thinking that it would be a breeze to do this. I was thinking that all I would need to do is install a plugin for IntelliJ and I would be golden. If I couldn’t do that, surely I could just use Eclipse. Fast forward two weeks, skip over a lot of cursing and forehead slapping, and I am using Xamarin Studio. It took a while, but I came around to accepting that you just don’t mix C# and Java in the same IDE. It’s. Just. Not. Done. I also eventually accepted that C# isn’t platform independent, and developing in C# on a Mac is kind of whack.

My second more unpleasant jolt was that Maven doesn’t support C#. This one was a lot harder to accept. I thought it would just be a matter of adding some new dependencies in the POM for Brixen, adding a build plugin or two and I’d be good to go. I would have brixen-java and brixen-dotnet playing all nicely together and all I needed to do is type ‘mvn install’ and shit would just work. It was really hard, but I finally accepted that this isn’t how it’s going to be. I still have no idea how to do the build and deploy thing for the C# packages, but I decided to defer that task while I try to learn the C# language.

There’s no Lombok for C#, but there are these things called properties which sort of do the same thing. The naming conventions in C# are whack. Seriously?! PASCAL IS SO OVER! Interfaces can’t have fields?! The languages are similar enough on the surface that it’s not too difficult to translate from one to the other, but they are different enough that I am regularly irritated and pissed off in some fashion. It’s like a rash that isn’t bad enough to make you go see a doctor, but still annoying enough that you really really want a steroid shot JUST TO MAKE IT STOP ITCHING. I just have to say to all you people who love C# — WHY? Why do you love it?

Share This:

Yet More Reasons Why You Don’t Have Good Automation

It has been a terrible two weeks for me. I lost my last living grandparent and my workplace drama culminated in the latest episode of humiliating and ham-fisted attempts to micromanage my work because The People Who Matter can’t seem to grasp the reality that test automation is really really hard.

I am demoralized today. I have hesitated to write about the ‘people’ element involved in developing a robust automation framework because this tends to be volatile and subjective content, but my work week totally blew, so I am going to do what has become a time-honored tradition since The InterWubs became a mainstream pastime — I am going to bitch about it online and talk some shit. When I joined my current workplace about three years ago, the state of UI automation was less mature than it is now. There were a lot of teams who were just starting out and there were some teams who had more mature tooling.

At this time, a complete overhaul of a big system was also just beginning, which meant disparate teams would have to collaborate together in ways that had never occurred at this company before. There was a sense of urgency about getting this ‘Automation’ thing going because there was so much to test and manual regression testing is slow and expensive. The various teams of SQA engineers were trying to get a handle on the automation and some were succeeding, but many teams were not. I was newly hired and excited about finally getting a job where I could spend more time writing code than running the same old boring manual regression tests over and over again.

I was assigned to a complex web application that saddles many domains across the company. Both the UI and the data/business logic aspects of this application are complex. It’s the kind of meaty automation challenge that gets a weirdo nerd like me pumped up and ready to code. My boss had recruited a team of senior employees and we wrote a large suite of test cases for this application. Then we started to investigate how we could approach the automation. We decided to go with the Page Object Model design pattern that is the gold standard for Selenium UI test automation. We evaluated some other frameworks, but, surprisingly, none of them were actually using this design pattern, so we started writing our own framework. Although I am biased, I will state that the code was sophisticated and high-quality stuff that was up to the task of being a great API for writing automated tests against the rich and complex UI of the application we were responsible for.

Then along came a high-level ‘architect’ who decided that everyone had to use the same ‘framework’. I won’t go into the ridiculous lack of consensus about what the term ‘framework’ meant because it’s a distraction. Let’s just say, there was no consensus on what this term meant. He took a look at the some of the various tools that some teams had built which they called ‘frameworks’ and then decided to declare one of them the ‘winner’. Then he decreed that all teams must use the one he had declared the winner. On the surface, this isn’t necessarily a bad idea and if the person making the decision actually knew how to evaluate the candidate frameworks, this could have ended well. The losers would have felt slighted and upset, naturally, but technical people generally can be persuaded to accept standards and tools they didn’t originally choose or build for themselves if they are good.

The key ingredient in making the above scenario end well is the word ‘good’. In my opinion, the following things must be true for a test automation framework to be good:

  1. It should be built with industry standard tools, not someone’s pet bleeding edge thing that is new and sexy, but not mature and certainly not in widespread use
  2. It should be built with tools that you can easily recruit talent to use and extend, not tools which have few experts you have to struggle to find and hire
  3. It should be built with tools that your current employees are mostly familiar with
  4. It should be built with tools that are well-documented and well-supported
  5. It should be built with a well-designed and properly articulated architecture so that its components are maintainable, reusable, and scalable
  6. The separate components of the system should be cleanly separated and use APIs to communicate with each other
  7. There should be no hardened dependencies on specific test management or defect management systems

Now that we have this settled, let’s return to the high-level ‘architect’ who decided to require all the QA teams to use the same automation framework. A person without good programming or software design skills can handle the first four items on the list. The last three items need to be evaluated by someone who does. Guess which category homeboy belonged to.

The framework he declared the ‘winner’ was just Not. Very. Good. It was actually one of the frameworks my team evaluated when we were trying to figure out what we were going to do with our automation work. We quickly passed on this one because it was very poorly designed and had none of the characteristics we were looking for. It didn’t help that the person who wrote it clearly did not have much familiarity with the WebDriver API. When I investigated how other teams were using it after Sir Architect declared it the winner, I noticed that a lot of them included it as a dependency in their POMs, but didn’t actually use anything in it. I think this was mostly because the design was so bad that it was nearly impossible to understand unless you methodically walked through every line of code. I eventually did this myself, and I have to say, I am a stronger person for the experience.

My co-worker and I couldn’t bear to dump our system which worked great for us and adopt this other system which totally didn’t work and would require us to shoehorn some ugly-ass spaghetti code into our test code in order to ‘use’ a framework that did absolutely nothing for us. We went on writing our page objects and incrementally automating some steps in our suite of manual end-to-end tests. Meanwhile, my boss decided to become an individual contributor and a member of our team who never even wanted to manage was given responsibility for our whole team. She had recently been hired and had not even had time to settle in. After she took over, Sir Architect kept asking if we were doing our automation using The Chosen Framework.

I should stop here and explain that our new manager, unlike the one who decided to stop managing, was not someone who had strong programming and software design experience. So, she put a lot of pressure on us to deliver lots of real good automation real quick. The problem with people who don’t ‘do’ software is that they tend to think that test automation is easy and when they hear the word ‘framework’ they think, “Magic system which shits a pound of automation gold every hour with little to no effort and so little technical expertise required that even a preschooler could automate the whole internet with it.”

We tried to explain that in order to automate the regression suite for our application, we needed to build models for all the UI components so that we could interact with them programmatically. We also tried to explain that the data and business logic domain was also something that needed to be modeled so that we could define and feed test data into automated tests instead of hard-coding it right there in the test methods. We tried explaining that The Chosen Framework contained some badly designed mechanism for getting a reference to a WebDriver instance of some desired type and that was basically all there was. All the rest of the necessary tooling, namely the models, were not provided with The Chosen Framework. Not only that, the Chosen Framework required that we use a ‘base’ page object which seemed to be super confused about whether it was a generic page object or a misguided effort to wrap some basic WebDriver functionality in a way that was inferior to WebDriver’s own native interface.

It. Just. Did. Not. Sink. In. Part of the problem was that other teams had managed by some miracle to tease some automated tests out of The Chosen Framework. We couldn’t seem to make our manager or Sir Architect understand that these tests didn’t do very much and that they were for very simple applications with maybe a page or two with a very limited data domain. Our application is not that type of application. It has lots of dynamic UI elements, numerous dialogs, menus, and data entry forms along with some complicated multistep state changes that have to be tracked and verified along the way. Not only that are a lot of validation rules on the data that are in play as well as a complicated user and role access permission domain to make it even more fun. And then there is the collection of features and behavior related to migrating customer artifacts from the older application that our application is supposed to replace. I also cannot neglect to mention the oh so very special test environments which have been featured in earlier episodes of “Why You Don’t Have Good Automation”.

A subset of these complications should be enough to send any self-respecting automation engineer over the edge, but throw in the need to periodically explain your failure to deliver some sweet, sweet automation for this mess to people who just refuse to understand that a badly designed mechanism for getting a WebDriver instance is not going to result in Automation Jesus marching triumphantly down our cubical-lined aisles and you have yourself a heaping helping of Soul-Destroying Is This Really My Life Cognitive Dissonance. We had a test suite of some 40 manual tests with roughly 15-30 steps apiece that we were running on two versions of IE, the latest version of Firefox on Windows 7 and OSX, the latest version of Chrome on Windows 7 and OSX in addition to Safari. At one point, our manager just declared that if we could not complete the automation for this test suite in three weeks, we had to dump our work and switch over to The Chosen Framework. This was after saying time and again, that the Chosen Framework provided no interfaces for interacting with the UI, nor did it provide any efficiencies for building the interfaces.

Fast forward two years. My team falls back under my original boss who has decided to manage again. My team now includes the person who built The Chosen Framework who has been promoted to Principal SDET. It is announced that we will collaborate with some other QA engineers in our organization to built a new framework and the designer of The Chosen Framework, V1, will ‘architect’ it. We spend a year and a half building various components while our Principal SDET regularly becomes very concerned about the directory structure of the repositories and the very large amount of code which some people are writing because large amounts of code is ‘heavy’ and that is bad. Oh, and ‘the communication protocol will be a POM dependency’. Somehow we write something that some of can use and there is an effort under way by our upper management to shove our framework down the throats of every other business unit in the company. A framework which none of them helped build or design and does not include anything from their now mature and fully-featured frameworks that are mostly better than ours. You can imagine how fast they all are running from the directive to adopt The Chosen Framework, V2.

I have finally been given space and time to do the automation after two years of relentless and never-ending manual testing stints all the while hearing, “Where is all that good automation?!!!! We can’t afford to hire manual testers. Just do it in your spare time!” Now I have about 200 automated tests of which only a couple are flappers which need some tweaking to make stable. Nevertheless, the new QA director has decided that we need to ‘start making some progress’ and a close working relationship and some coding ‘help’ from the creator of The Chosen Framework, V1, is just the solution to this problem.

It almost made me want to quit and take up goat-herding as a profession after pouring acid over my laptop and taking a shit right on top of my workstation out of spite.

So, folks, this is the latest edition of “Why You Don’t Have Good Automation.” Automation frameworks are a serious technical challenge and you need to put technically sophisticated people in charge of it. Also, don’t try to shove a tool down the throats of people who know better than you do what they need. If you’re looking to have your entire company on a single framework, you need to involve all the teams in requirements gathering. You also need to incorporate the best of their tooling into it. Also, if you don’t want to drive your automation staff into the arms of your competitors, don’t keep asking why they aren’t done every two days.

Share This:

Brixen Decorators: What’s Changed

The basic concept of the decorators hasn’t changed. There are just more of them. I have added decorators for state beans, configuration beans and builders in addition to the decorators for the component implementations. The decorators help reduce boilerplate code by using a new feature introduced in Java 8: default interface methods. A class can implement numerous interfaces, but extend only one class. Prior to Java 8, this meant implementing the same methods over and over again in different classes extending a particular interface. After Java 8 however, a decorator interface can extend another interface and provide default method implementations for the interface it extends.

Here is an example of the PolleableBeanDecorator, which provides default method implementations for the methods required by PolleableBean:

The decorator defers all the PolleableBean method calls to the default PolleableBean implementation — PolleableBeanImpl. This is achieved by requiring a ‘provider’ for an internal reference to a PolleableBeanImpl instance. So any class which implements PolleableBeanDecorator needs to define an accessor method for this provider. Let’s have a look at the provider for a state bean:

I didn’t want to have a method in a decorator interface that would return a reference to the internal state bean itself because that would break encapsulation. So, I came up with the idea of a ‘provider’ which would give protected access to this internal state bean reference so that only a sub-class or a class within the same package as the provider would be able to access the internal state bean reference. The packaging structure I chose allows only the provider and its sub-classes, the decorator and other classes in their package to access the provider’s internal state bean reference.

Here is an example of a state bean which extends PolleableBeanDecorator:

DynamicControllableBean is a state bean for specifying a page object that contains one or more web controls which needs to be polled for state change of some kind, usually after an interaction with one of its controls. A drop down menu, for example, would need to be polled after interacting with the control that expands or collapses it to determine if the menu has expanded or collapsed as expected. The default implementation of DynamicControllableBean can only extend a single class, so its parent is ControllableBeanImpl.

It would be a drag to have to provide implementations for the methods required by PolleableBean which are exactly the same as the implementations provided in PolleableBeanImpl. By implementing the PolleableBeanDecorator interface, DynamicControllableBeanImpl can satisfy all the requirements of PolleableBean by providing only an accessor to a LoadableBeanProvider that wraps an instance of PolleableBeanImpl. Lombok helps reduce the amount of source code even more because the provider field only has to be annotated with the Getter annotation.

The decorators for the configuration beans operate in the same way. Here is the PolleableConfigDecorator and the provider for a configuration bean:

And here is DynamicControllableConfigImpl, a configuration bean which implements PolleableConfigDecorator:

Decorators for the builders was a bit trickier to pull off, but I managed to find a way. Here is the PolleableBuilderDecorator:

And AbstractDynamicControllableBuilder, which implements it:

This decorator implementation works by declaring a provider which wraps the builder implementing the decorator. The same state bean must encapsulate all the state for the component, so declaring a builder with a separate state bean instance wouldn’t work.

The decorators for the components haven’t changed, so I won’t post any examples here since it would just duplicate what I presented at the conference. The source code for Brixen is here.

Share This:

Brixen Configuration Beans: What’s Changed

The changes in the configuration bean package are mirrored largely by the changes in the state beans and the builders. The other change is that I added JSON type information to all of the configuration bean interfaces. When I originally conceived of the configuration bean idea, I foolishly didn’t consider the possibility that some configuration beans may contain other configuration beans as fields. So, I only added the type information to the LoadableConfig bean. Without this type information, Jackson cannot properly deserialize polymorphic types.

The other change is that LoadableConfig allows the definition of custom properties, through JsonAnyGetter and JsonAnySetter methods. It is entirely conceivable that one might want to define a configuration option for a page object which doesn’t have general significance to a class of page objects, but which is important for a specific context:

The ControllableConfig is the most significant new addition. This configuration bean is for defining the dynamically configurable options for a page object which contains web controls. This configuration also encapsulates the configurable options for each one of its controls:

There is a marker interface for a dynamic Controllable that needs to be polled on intervals for a state change via a FluentWait:

Here is an example of what the configuration source for a ControllableConfig would look like:

In the same fashion as the state bean and builders, there is a marker interface that is parent to all the control configuration beans:

The marker interface for a click control configuration bean:

It extends ClickableConfig which is a configuration bean for a wider class of clickable page objects besides controls:

The configuration bean for a hover control:

The configuration bean for a hover and click control:

And that’s a wrap for the changes in the configuration beans. The source for Brixen is available here.

Share This:

Brixen Builders: What’s Changed

One of the awkward things about the original version of the API is how data from a page object’s JSON configuration source is retrieved and handled. I wanted to do something that was more elegant and less cumbersome than the multistep process of the original:

  1. Querying the service for a configuration
  2. Determining if a particular dynamically configurable option is defined in the configuration
  3. Determining if the value assigned to the option is null
  4. If the value is not null, then retrieving its value from the Optional that wraps in the configuration bean and setting that field for the page object through its builder

All of the the builder interfaces now overload all the methods that specify a page object’s dynamically configurable options. One version of the method takes a value for the field, and the other takes a configuration bean. The builder implementations take care of all the steps listed above save the first step. They also handle the case where the configuration bean is null. All the client class has to do is query the configuration service for the page object’s configuration by String ID and pass the result to the builder. If there is no configuration defined for the current environment under test, then the service will return null. If the configuration bean which is then passed to the builder is null, the builder will do nothing with the bean and leave the default value for that field in the page object’s state bean unchanged.

Let’s look at a couple of examples. The LoadableBuilder, a builder for a basic page object, and PolleableBuilder, a builder for a dynamic page object which needs to be polled on intervals for a state change via a FluentWait, are basically unchanged since the conference except for the new methods which take a configuration bean.

The default implementations of the new setter methods do all the work of checking and retrieving the data from the configuration bean which was previously the responsibility of the class building the object:

The other big change is related to the total refactoring of how web controls and the page objects that contain them are specified and built. Each of the three types of controls described in this post has a builder, but there is also a builder for a page object containing controls which has methods for specifying the controls.

The marker interface for a control builder:

The marker interface for a click control builder:

It extends ClickableBuilder which is a builder for a wider class of clickable page objects besides controls:

The builder for a hover control:

And finally, the builder for a hover and click control:

There is a _lot_ of duplicated code for the hover control and hover and click control, just as is the case for their state beans, which I acknowledged in my post about the changes in the state bean package. For now, I don’t see a good reason for extracting the common behavior into a parent interface for both of them to extend because the parent interface wouldn’t be reusable in other contexts. It’s a wart on the butt of this API, but I think I can live with it.

The interface for the builder of a page object which contains one or more web controls has a lot of syntactic sugar that allows you to build the whole page object, complete with all its controls if you don’t want to use the individual builders for the controls themselves. Each control is associated with a String ID that must be unique (which should go without saying, but I couldn’t help myself):

Share This:

Yet More Reasons Why You Don’t Have Good Automation

I got a bit sidetracked with the Brixen posts that I forgot to spend some more time explaining Why You Don’t Have Good Automation.

I am tired today. I spent the day dividing my time between way too many tasks, one of them boring as hell, but which should make it possible to dynamically put together a test run with Zephyr for Jira using keywords. The keywords correspond to group names on the automated tests for each test case. It’s a giant pain in the ass to add all these labels to the Zephyr test case and then add them as group names to the test methods, but it’s necessary to make what I want to do possible.

In addition to this mind-numbing data entry horror, I also spent an hour or two trying to figure out how to run these automated tests on the production system. I have never set up automation for the production system. My suite runs every night on whichever of our two test systems that isn’t currently crapping its pants and painting the walls with its own feces. I have some configuration files containing a list of test users and test accounts and I have a neat little set up routine that pulls this test data out of the config files and delivers it to tests running in parallel so that no single user login is used by more than one thread at a time. Behavior on our awful test environments gets even more awful and unpredictable if a user is logged in from multiple browser instances.

I still get random errors resulting from a user being logged in more than once because I am not the only person running tests on the test system. There are hundreds of people who use these environments and everyone has access to the same user accounts that I do. So, shit happens. Shit happens a lot less since I set up my tests to ensure that at least they don’t inadvertently use the same login credentials, but I don’t control the accounts that other engineers are using for their tests.

But I digress. The production system is another can of worms. I need a way to store the login credentials and deliver them to the tests in a secure way. Storing it all in a plain text configuration file and checking it into source control is a pile of bad idea topped with steaming fresh dog turds. It would violate PCI security standards six ways from Sunday. I’m getting some help from the engineer who wrote the original and now retired automation suite that used to run on production. He had a database running on a managed VM that stored the credentials and used a config file to specify the database access credentials. The config file was actually a stub because checking the DB credentials into source control in a plain text file is no better than checking in the login credentials themselves. A human being had to check out the repo and add the DB credentials to the config file. Then they could trigger the tests which used the DB credentials from the config file to log into the database to retrieve the user login credentials.

I found out that I can’t use the same strategy because a DB running on a managed VM isn’t kosher anymore due to PCI security standards. I have to find some other way to deliver the production login credentials for automated tests to use. Did I mention that I am not an expert in security standards? And there is no documented process for dealing with this problem? A few months ago, our entire organization was tortured with a giant PCI training effort in which we had to watch lots of videos and take lots of quizzes to prove we had been trained in PCI security awareness. I guess developing procedures and systems for dealing with practical, real-life problems just like this one and training us in that was too much to ask for.

Another part of my workday was devoted to speaking to an account executive at Sauce Labs in order to get a cost estimate for their services for the four teams in my particular mini-organization. I think he was excited up until he found out that I had no budget power and that I actually don’t know what the management class here is willing to spend. I was handed the task for getting an estimate for using them in an offhand way in the middle of a meeting that was supposed to be about something else two weeks ago.

This Sauce Labs POC task has been a hot potato that got passed around for the last year. A couple managers had this task sitting on their plates for weeks and every week during our status meetings, it just kept getting pushed off. Eventually, we stopped talking about it. Then some engineers on one of our teams actually did a trial with them and did a great presentation on the experience. I was excited because I thought something was actually going to happen at that point, but then I never heard another word about it until I said we needed Sauce Labs in a meeting two weeks ago because I was pissed at the lack of infrastructure for Selenium test suites. Suddenly the task of doing the same thing that has already been done by others was given to me. So, I am doing what others already did, which sadly, will lead to the the same likely result. Nothing will happen and a year from now, some other over-worked person will be told to do it again.

So, you’re wondering, “What does this have to do with why I don’t have some of that real good automation?!” One thing I have learned in all my years of working in quality assurance is that ‘quality’ is not a thing. It’s actually a living system made up of processes, people and tools. It’s also a ungodly complex system in a large organization. Automation is only one piece of it. And unfortunately for you, it’s a piece that depends largely on the smooth functioning of almost every other piece of the system. How many times have you acknowledged that there is a problem somewhere in your system that relates to quality that resulted in thousands of hours of lost productivity? Shitty test systems? Shitty or non-existent documentation? Shitty or non-existent approach to generating and managing test data? Shitty, fractured approach to tools and infrastructure procurement and maintenance?

If your approach to any or all of these deficits has been to click your heels and wish for your own personal Automation Savior to swoop in and drop a load of some real good automation on you like Rapunzel spinning gold out of straw, I would like to point out that you need to provide the straw first. Providing the straw involves the procurement of arable land, seeds, fertilizer and labor to grow and harvest it, along with a transportation network to get it you in good condition. Because Rapunzel ain’t gonna spin no gold out of moldy, wet, rotting straw with rat droppings all over it. ‘Kay?

Share This:

Brixen State Beans: What’s Changed

At the conference, I presented source code for Accessible and Dismissable objects, that is page objects which can be rendered visible or invisible by user interaction. Because some objects are dismissable, but not accessible, such as announcement dialogs and popup ads, I modeled them as separate entities with their own state beans and used a marker interface, ToggleableVisibilityBean to specify the state for a component which is both. There are some limitations with this design approach. First is the built-in assumption that there is only one control which toggles the visibility of the component, which is not true in some cases. A chooser dialog can actually have three such controls: Submit, Cancel and Close. The other is that the controls themselves are also page objects, but they are not modeled as standalone entities.

So, I decided to re-work the concept entirely. I created a state bean interface for a component which contains controls. It allows any number of controls to be added to the component’s state specification. It also makes no assumptions about the side effects of interacting with the controls. Therefore, it is generic and applicable to any such component, whether it is a component with toggleable visibility, or which has filterable content or with pagination behavior. Big win for reusability!

Here is the source for ControllableBean, the new state bean for a component which contains web controls. This is a terrible name, and I am open to suggestions for something better. At least it’s shorter than ToggleableVisiblity:

Each control is associated with a name, and the ControllableBean interface has syntactic sugar setter methods for defining the state of each of its controls. There are three distinct flavors of controls:

  • Controls that are visible by default and have meaningful behavior when they are clicked
  • Controls that are are visible by default and have meaningful behavior when they are hovered, such as expanding a menu
  • Controls that are are invisible by default, must be hovered to make them visible and that have meaningful behavior when clicked and which also can have meaningful behavior when hovered

The first type is a vanilla, garden variety web control. The third type introduces some complicated test cases that I have to try out. There is a pretty complex text case I described at the beginning of this post that I have to test with this new version of the API. That same post also explains an additional shortcoming of my original design, having to do with the fact that in some environments you just can’t trigger the mouseover action through Selenium or through Javascript using the JavascriptExecutor. I spent some time thinking about possible dynamically configurable workarounds for interacting with the control in such a way that the side effects of the interaction can be triggered, allowing a tester to automate tests which rely on that workflow to test something else. Obviously, you’d have to manually test the hover action because you can’t automate it, but it would be great if that didn’t block the automation of other tests.

For the second type of control, you can often just click it to trigger the same side effects that the hover action does. So, I added two dynamically configurable options to click instead of hover. One for using native Selenium and one for using a Javascript click workaround through JavascriptExecutor in cases where the native Selenium click fails silently. For the third type of control, when you can’t hover the control either through native Selenium or the Javascript hover workaround, you just can’t make the element visible. So how do you click it? By using the Javascript click workaround through JavascriptExecutor, which will execute the click even if the element is not visible.

Yes, I know this is whack. I know that when you must resort to this hack for a clickable control, regardless of whether it is the first type or the third type in my list, you probably should add a test for the click action to your manual test suite. But, the intent of the workarounds is to enable automation of the workflow for other tests without ugly-ass boilerplate if-then-else clauses, or worse, hard-coding the workaround by default for all environments into your page object which makes tests for the click action suspect for all environments.

The hoverable controls have what I call an ‘unhover’ element (another awful name, I admit). This is a WebElement to use for removing focus from a hoverable control. It should be an element in a safe location which itself has no meaningful behavior when it is hovered. This element also has the same set of workarounds as the control — hover with Javascript when the native Selenium hover action fails silently and the focus is not removed from your control as well as the two click workarounds when you just can’t do a mouseover by any means. Just be sure that the control also has no meaningful behavior when it is clicked!

I have another state bean marker interface for components containing controls which have dynamic state changes when one interacts with their controls. This dynamic state change should polleable on intervals with a FluentWait to determine if the expected state change has been achieved:

Let’s take a look at the new state beans for the controls themselves. ControlBean is a marker interface that mostly serves for type determination and it’s the parent to all controls. You gotta love Java 8 for default interface methods:

ClickControlBean is a marker interface for the first type of control in the list.

It extends ClickableBean, which is for defining a wider class of clickable elements besides controls:

HoverControlBean specifies the second type of control on my list:

HoverAndClickControlBean specifies the third type of control on my list and it is a PolleableBean due to the fact that this type of control has dynamic visibility, which means it should be possible to poll it on intervals with a FluentWait to determine if it has been toggled visible or invisible after hovering it.

I am not happy with the amount of repeated code in the two hover control state bean interfaces. I don’t think it’s worth re-working the inheritance hierarchy here since a parent class that defines the shared methods probably isn’t reusable anywhere else.

So, there you have it. The source for Brixen is available here, and it has complete Javadoc comments for all but a small handful of classes.

Share This: