PIE "GUI-TEST"

Implementation and Results of the ESSI PIE 24306

Tilo Linz, Matthias Daigl

Key Words: Software Process Improvement, Software Testing, Testing Tools, Test Case Library, Testing Metrics, Graphical User Interfaces.

This report describes how the Process Improvement concept is integrated in the imbus Company's Quality Management System and which action has been taken to improve the Testing of Graphical User Interfaces within the framework of the Process Improvement Experiment (PIE) currently in progress. As one main issue of this project was to select and evaluate a commercially available capture&playback GUI Testing Tool, the benefits that can currently be achieved by using this tool will be discussed.

Process Improvement as a Component of a QM System

The imbus Company's Quality Management System has been ISO 9001-certified since the beginning of 1995. In order to guarantee not only appropriate functioning but further enhancement of this system, a Process Improvement Procedure has been integrated into the company's philosophy.

Our Quality Management System thus specifies major yearly improvement steps. Goals are defined by the Quality Management Representative for a period of about 2-3 years and annually reviewed by the Executive Management. Doing so we can assure not only the continual achievement of minor step by step improvements, but make sure that each year we make one crucial improvement of our methods.

This years goal is to optimise and automate our methods, used for testing Graphical User Interfaces (GUI) throughout the company, within the framework of our existing Software Test-Process.

The Process Improvement Experiment

As a small to mediumsized software company (25 employees), the execution of this improvement step to the extent described below was only possible because we could execute it as a Process Improvement Experiment (PIE) within the framework of the ESSI Program of the European Union.

"A PIE provides the opportunity for an organisation to demonstrate the benefits of process improvement, through a controlled, limited experiment. The PIE is undertaken in conjunction with a real software project (the baseline project) which forms part of the normal business of the organisation. A PIE allows an organisation to try out new procedures, new technologies and organisational changes before taking the decision as to whether or not they should be replicated throughout the software developing unit." [EU1].

Picture 2: Expenditure data for Baseline Project

Our current strategy for the testing of software systems is a structured path from test specification to test implementation and testing, controlled by error tracking mechanisms. This process can take two different shapes. Either as a part of our internal software development process; or as a separate service, since besides the development of software we also offer testing services and act exclusively as a third-party tester for a considerable part of our customers.

The goal of the PIE as described here is to technically (and thus economically) optimise the step "run tests", by introducing a commercially available tool into the process of an automated execution of software tests under MS Windows.

From the very beginning of the PIE project we were aware of the fact that a GUI-test tool can only achieve its full rationalisation potential, if it can be perfectly integrated in the chain of further processes.

We therefore first looked for state-of-the-art methods for the formal specification of GUI tests and a test-notation which would permit the test specifications to be transformed into the syntax required by GUI test tools with a minimum amount of manpower.

Since we saw a possible solution in the use of the formal specification language Z (see [Z1], [Z2]) for derivations and definitions of tests, we familiarized ourselves with this use of the language. In an internal one-day workshop, examples and exercises were used to demonstrate how "Z" can be used to formally specify software and how, subsequent to a Z-specification, testcases can be deduced systematically and semi-automatically. Unfortunately the workshop also demonstrated that specifying software in Z is a very involved process and requires additional tool support (Z-editors, Z-syntax-checker).

For this reason we decided to take a more practical approach and abandon the use of a formal specification-language. The core of the new testing-process will instead be a new frame test-specification which will serve as a basis for all test-specifications executed in our company. It does not only guarantee the uniformity of specifications, but also contains guidelines for describing tests (even when using formal methods like deterministic finite automaton and regular expressions) and for executing tests. Moreover it contains standard testcases and describes standard test-procedures.

To complete the frame test-specification in the environment of the GUI-tool, a testcase-library will be build up which will keep at hand functions which are often used for devising tests as well as elements of standard tests and testcases. This library will be centrally maintained and continually extended. With the increasing extension of this library we expect to cover more tests and be able to generate testcases more quickly.

Tool Selection and Introduction

Parallel to work-phase1, we started to investigate commercially available GUI testing tools, and asked four tool vendors for a test installation on our network system. Each one of the tools was tested for suitability in a near to one-week test operation. In this test run the following criteria were investigated: operating system platforms supported, software development environments supported, functions for creating, executing and evaluating tests, operator-friendliness and stability. The complete test-tool checklist and the addresses of the manufacturer can be downloaded under [i1].

Regarding functionality, the tools examined vary only insignificantly. The biggest differences showed in the respective operator-philosophy and in the software development environments. The tool best suited for our purposes proved to be "WinRunner" by Mercury Interactive, so this tool was used in the baseline project and was put to a more detailed test. The results of this test are shown here in the example of a selected testcase

How to Test GUIs using a Capture&Playback Tool

Using the graphical editor developed in the baseline project, it is possible to equip and configure electrical switchgear cabinets of different types with various modules. To this end, switchgear components are displayed in the editor as bitmaps. Double clicking with the mouse on certain positions within the bitmaps, will either call a context menu, in which modules can be selected as equipment, or, if the position can be equipped with more than one module at a time, will call up another bitmap which can be used in the same way as the first one. Up to three bitmap levels can be supported.

The testcase shown here as an example is designed to test the correct navigation through these levels, the decision to either select a menu or a new window, and the contents and functions of the menus.

Because the great number of monotonous navigation steps would quickly wear out a human tester, this task is ideal for automation.

At a closer look, however, automating this test proved to be difficult, since the bitmaps were hard to address (the coordinates had to be introduced to the test programme beforehand in a teach-in-step). Another obstacle was the realisation of the context menus as pop-ups, which appeared to be a major problem for the test tool. Access was not possible via standard methods, and special routines had to be developed.

Have we really succeeded in making GUI Testing painless ?

To determine to which degree the automation of GUI-tests is really more economical than manual testing we have used both methods in the Baseline project.

The tests defined in the test-specification were executed by a tester within the baseline-project. Simultaneously the PIE team converted the specified tests into test scripts and reran them, using WinRunner.

The time costs of the example above are listed in the following table.

In this example, with an initial expenditure three times as high as when devised manually we found that it was possible to reduce the time required for executing a test by a factor of eight.

A corresponding evaluation for all testcases automated within the baseline-project was not available at the time of writing. If, however, the single result above can be confirmed, the use of the GUI-test-tool will pay for itself by the second regression test.

When evaluating these values it is important to take into account the number of faults detected within the programme when using the different methods. At this stage it appears that a trend can be recognized where more mistakes are detected by executing tests manually. The reason for this seems to be that in addition to the specified checks, the human tester will also coincidentally notice further mistakes.

Summary and Prospects

To sum up, one can say that automated testing can not entirely replace manual testing. The full advantage of automating tests can be achieved in regression testing. Considerable cost-advantages can already be achieved at the second regression test.

When devising automated tests, two opposing tendencies can be observed. On the one hand, pure capture&playback tests can be generated without or with a small effort in programming. These can be realised with a comparatively small expenditure, and thus pay for themselves more quickly. However they are inflexible with regards to changing the programme, are hard to maintain and can easily become unwieldy. Alternatively, the automation can be executed with a high effort in elaborate programming. The expenditure for devising the tests is then higher but the tests are more flexible, can be extended, and are easier to maintain and to reuse. The long term benefit is thus much higher.

The most basic problem with automating tests is not so much the execution of the tests, but the automated recognition of correct or faulty behaviour of the programme. It is thus essential to plan the tests very carefully in order to ensure a successful testcase construction. Ideally the testability of the application will be taken into account at the specification phase.

Besides manual and automated tests a new category of tests has been created: semi-automated tests. These combine the advantages of the human being and the machine.

The computer carries out the monotonous tasks, and guides the human tester systematically through all specified tests. The human decides whether the programme-behaviour, triggered by the test-tool is correct, or whether further side-effects occur which have not been included in the test specification, and can therefore not be registered by the tool.

 Literature

Authors

Dipl.-Inf. Tilo Linz, year of birth: 1964

Mr. Linz has been a partner in imbus GmbH since 1992, where he is in charge of internal Quality Management.

As an imbus consultant, he is involved in several software testing and QA projects and is responsible for the execution of imbus training seminars in the field of software quality assurance.

Tilo Linz, imbus GmbH, Kleinseebacher Strasse 9, D-91096 Möhrendorf, Germany
Phone: +49 (0)9131 75180, Fax: +49 (0)9131 751850,
Email: Tilo.Linz@imbus.de , WWW: http://www.imbus.de

Dipl.-Inf. Matthias Daigl, year of birth: 1968

Mr. Daigl studied informatics at the University of Erlangen-Nürnberg, Germany. As a software engineer in the medical field, he has attained comprehensive knowledge of Quality Management Systems.

He has been involved in the Quality Management activities of imbus GmbH since the beginning of 1997. As part of the "ESSI"-Initiative, Mr. Daigl was selected to be the Project Test Manager in the PIE-Project "Automated Testing of Graphical User Interfaces".

Matthias Daigl, imbus GmbH, Kleinseebacher Strasse 9, D-91096 Möhrendorf, Germany
Phone: +49 (0)9131 75180, Fax: +49 (0)9131 751850,
Email: Matthias.Daigl@imbus.de , WWW: http://www.imbus.de