Some More Reasons Why You Don’t Have Good Automation w/ Update on Generic UXD Legos Source Code Packages

Update on the example source code packages for UXD Legos:

I am writing the sample page objects to demonstrate how to use the API and its components. I have refactored a lot of things since I presented the API at the conference. The configuration service implementation has been refactored to auto-initialize itself by reading in all the available configuration profiles. It has also been updated to be more robust in a multi-threaded environment.

I have significantly re-worked the model for controls on UI components. The new version is more flexible. Controls are treated as separate entities from the components they control. Since some interactions, like dismissing a UI component, are actually achievable by multiple controls, it makes sense to do it this way. My original design didn’t robustly handle this. For example, a chooser dialog can be submitted, or cancelled, or closed via three different controls. All three actions result in the dialog becoming hidden from view, but my abstract model for components which can be toggled visible and invisible didn’t handle the fact that there could be multiple controls that dismiss a component from view. The new model can handle all three of those scenarios in a generic way. I am working out a good interface for the builder of the UI component to handle specification of all of its controls. Each type control I modeled has a specific builder implementation for it, which is great if you are just focused on specifying and building a control, but not so great if you just want to specify and build the whole component, along with all of its possible controls. It’s clunky to have to interact with more than one builder. I think I have found the right design for this, and I am implementing it right now, along with the sample page objects.

I think, in addition to delivering the C#, Python, and Ruby versions of this API, I will also need to write an updated slide show about it.

And now on to Why You Don’t Have Good Automation:

I am grumpy today. I had a meeting this morning to discuss the automation strategy for a major product. We ended up talking about how the test suite is growing in size by 5-15 tests a day and I pointed out that I have a GIANT bottleneck in the form of inadequate infrastructure for executing Selenium test suites. It was revealed that this is a big problem for many other teams as well. There is no company-wide Selenium grid system available. We also do not have the green light for using Sauce Labs. I was under the impression that other teams had their own private Selenium grids which they jealously guarded because they are a limited resource and building a comprehensive Selenium grid is a big task. If you read my previous post on this subject, it’s not particularly easy to acquire the necessary resources and IT support to build one. I suspect that the reason we do not use Sauce Labs is that we would need to scrub all data in the test environments of personally identifying information. Which… will probably not happen anytime soon, so let’s backtrack to Square One: there is a giant bottleneck related to the lack of test infrastructure for running large suites of Selenium tests. It’s just crazy that there are tests that could be running with every build, that are not running because we only have the infrastructure for getting quick results for the highest priority tests.

If your company, like my company, spends a lot of time recruiting engineers with Teh Awesome Mad Skillz, but they do not budget resources to handle all those awesome automated tests, they are wasting a lot of money and a lot of talent. They would be better off hiring a cheap team of off-shore manual testers to run the tests manually. Otherwise, you have to force those amazing engineers to run tests manually in lieu of writing automation you can’t support in order to ensure that your products are tested. It’s expensive to hire development engineers to run manual tests all day. Plus, it pisses them off and then they leave for jobs where they get to write code.

The other issue that came up is that my tests take a long time to run because we have _TWO_ whole test environments available for an engineering organization of hundreds. TWO. WHOLE. ENVIRONMENTS. And they are unstable because there are too many people using them. Hence, all my tests have to do a lot of checking in order to capture failures related to environment craziness as opposed to real failures so that they can be re-tried until a verifiable true pass or fail can be determined. Other teams are running destructive tests all day long on these environments, so I can make absolutely no assumptions about the availability of test data on the system. Therefore, I have to make each test method search for test data that meets the necessary conditions for the test first.

The end result is that the bottleneck in Selenium grid infrastructure is doubly painful because the terrible test environment situation requires lots of retries and querying that would be unnecessary if each engineer had an isolated environment only they were using. Tests take more time to write because of all the extra overhead of dealing with the environment, so I’m not as productive as I could be. Not that it matters because…. Square One: Inadequate selenium grid infrastructure to run all the tests.

To summarize:

Automation is not magic dust that is gathered by tiny little fairy hands and deposited at your doorstep overnight for free. It takes a lot of planning and infrastructure. So, you like… need to invest in it and stuff. If you don’t, you won’t have good automation. You will have shitty automation that mostly functions as an Automated Smoke Test Teddy Bear that you can cuddle up to at release time for that warm feeling of false security that it gives you after forcing your resentful automation engineers to run manual tests until they are ready to grab the nearest fork and dig out their eyeballs from the sheer boredom of it all.

Share This:


Leave a Reply

Your email address will not be published.