The Reality of UI Test Automation

by Krishna on June 12, 2010

Michael Feathers explains how the reality of user interface test automation differs widely from the expectations:

The Dream

Janet comes into work in the morning and she sits down at her super-duper testing console. She presses a button and the testing system springs to life. The application comes up all at once across ten monitors. Cursors move, selections are made (silently) and tests run against the user interface magically, as if some eager set of ghost elves took control, mischievously burrowing through the nooks and crannies of the application, running scripts to completion, and making little notes whenever there is a failure. Janet sits back in her chair, waiting for the elves to report back to her. She stirs her coffee gently.

The Reality

Janet hasn’t gone home yet. It’s 2AM and she has to report completion of all her test cases at a meeting in the morning. She thinks she’s past the last configuration issue but she’s not sure. For the last hour, she’s been trying to make sure that a particular button is pressed at step 14 of her script, but quirky latency on the server is preventing it from happening consistently. Sadly, she has to run the script from the beginning each time. Oh, and five hours ago she discovered UI changes which invalidated 30% of the regression tests. Most of the changes were easy but she still has 12 cases to go and her 9AM meeting looms ahead of her.

It is a good article. For caveats, read the comments where there are some important counter-points.

I have seen many managers who sincerely, but wrongly, believe that UI test automation is going to save them effort and money. The problem is that you cannot automate everything and so you still need some level of manual testing in addition to the automated testing. So savings are less than you expect. If there is any change in the application, most of the time it will result in user interface changes. And so you have to change your tests repeatedly, whereas with manual testing, there is nothing to change except the testing itself.

Good automation testers are more expensive and harder to find than manual testers. An effective automation tester must know both development and testing. They should be able to go beyond simple record and playback and be able to write scripts and change generated test code. That requires understanding of logic and programming structure. At the same time, they should have the mindset of a tester so that they can create test cases that can discover bugs instead of mirroring the application code. This combination of skills is rare.

It is also not very clear whether UI test automation is worth doing if you have got your suite of unit tests and integration tests set up. Your unit tests should already cover scenarios with different kinds of data. UI test automation should therefore be for checking whether different input has an effect on other UI elements in the screen. And that is a function of how rich your UI is. So if you have very simple user interfaces, there is not much of a gain from automated testing.

As usual, it depends. If your application is in maintenance and is undergoing few changes, it may be worthwhile attempting to write some level of UI automation to test complex user interactions and workflows. It adds another layer of support for refactoring and bug fixes. It may be less useful during the early life cycle of the product when changes are coming fast.

    { 2 comments }

    Hannibal June 15, 2010 at 3:34 am

    I beg to differ. The reality of UI Automation is that it changes from reality to reality.

    We have a perfectly fine working framework which is executed every time before a developer commits his work.

    There are places where it does not work, but there are a few where it does work. 🙂 Although it did took a really long time to come where we are now…

    Krishna June 15, 2010 at 1:48 pm

    Your last sentence is the point. It does take a lot of effort to get the testing framework up and running effectively to make a difference. Many people assume that it is so easy and rush headlong into it without being prepared for the hard work needed.

    Comments on this entry are closed.

    Previous post:

    Next post: