Testing

Introduction

This document provides guidelines for all aspects of testing during AstroGrid development. It covers requirements for unit, package system and integration testing of code as well as checklists for documentation review. Wherever possible, testing should be automated and integrate with other tests. Some testing is inevitably manual (perceived performance, usability etc). Where this is true, manual scripts for actions performed against results expected should be produced. These documents should themselves be maintained in the CVS repositories.

Software test planning begins before the software design. The steps in test planning include, in this order:

• Establish acceptance criteria.
• Develop unit and package test strategies.
• Develop an overall plan for integrating software modules (system test).
• Develop an overall plan for testing the modules in an operational environment (integration test).

Acceptance Test Criteria

The foundation of a good Test Plan is the acceptance test criteria that are agreed upon with the project user community. These criteria will be used to judge whether AstroGrid meets their needs. They also form part of the “contract” between developers and users as to what is required and expected of the final product.

Acceptance criteria cover:

• Function and performance
• System failover and recovery from various types of failures
• Human factors relating to usability
• Error handling and messages
• Stress testing
• Security and integrity

Test Plan Checklist

Regardless of the level of test (unit to integration, Alpha, Beta etc), the plan must cover:

• What is being tested?
• What are the pass/fail criteria?
• When will each test occur (sequence, workflow etc)?
• What hardware and software environment is required?
• What features must be tested?
• What features will not be tested?
• What are the responsibilities of individuals and organizations involved in the project?
• Start-up and shut down sequences

Unit and Package Testing

All code must be tested before entry into the development branch of the CVS source trees. Where a class is sufficiently complex, individual unit tests should be prepared. Other classes may be tested in a group test or as part of a package test. All packages should be individually testable with a single automated script wherever possible. Test early, test often. It goes without saying that maintaining tests and verifying their results is of paramount importance. Whilst trivial at the start of a project, unless tests evolve and expand with the development effort, the project will quickly become untestable and thus unstable. This must be avoided at all costs.

Unit test practices are detailed in JUnitTesting.

System Testing

As a development iteration approaches completion (function or soft freeze – see [Release Management]), plans to test all the packages together need to be prepared. This will probably involve the production of sophisticated test harnesses, although it is to be hoped that these will grow out of existing package testing routines. With these tests complete, the packages will be ready to form part of the Alpha test release of the overall iteration.

Integration Testing

With the Alpha Test Release prepared, installation and testing can take place in a testbed environment. This testbed will provide the same environment in which AstroGrid will be deployed, although perhaps not to the same extent. The system test scripts will once again be used to evaluate the release, this time without the test harnesses. Bugs will be addressed as described in the [Release Management] and [Change Management] documents. With integration (or Alpha) testing complete, the release will be moved to Beta test status.

Beta Testing

The package given to beta testers needs to cover:

• Technical approach – what is expected of the tester and how they should go about the testing.
• Usability – some features may not be presented in their final form, but the substance may need verification
• User documentation

The evaluation materials available to each beta tester should include:

• List of functions to be tested
• Comments & check-off for each function tested. This is probably best implemented as a web form to make the process as easy and painless as possible. It will also provide an audit trail for feedback incorporation
• Sign off with date of completion (generally done via e-mail)

Materials for Beta Testers

When conducting beta tests, the following items must be sent to each Beta Tester:

• Test Instruction Sheet that describes items such as testing objectives, steps to follow, data to enter, functions to invoke
• Preliminary User's Manual documentation
• Installation instructions
• Solved example problems that cover the major features of the code, or a tutorial. These should be interactive, requiring user input and generating the documented output
• Executable software

Documentation

All documentation should pass the following checklist:

• Do all the procedures listed in the documentation work if followed to the letter? Procedures listed in the documentation must work if followed step-by-step. When necessary, include explanatory comments.
• Are all important selections described? The documentation should describe all commands or selections found in the software, such as menu items, buttons, and dropdown menus. Exercise reasonable judgment here. For example, describing what "Cancel" does for every instance in every window is probably not necessary, but describing the function of a button called "Update Temperature History" which only occurs in one menu of the program is necessary.
• Does the menu structure in the documentation agree with that in the software? The user documentation, which includes both printed and online versions, should describe the menu structure correctly. The commands listed in the menus, including their order and exact spelling, must agree with the descriptions provided in the documentation. The overall structure describing how menus are nested inside other menus also needs to agree with the documentation. The developer may choose to include in the user documentation a description showing the menu structure of the program either graphically (such as a tree) or in tabular form (a list or table).
• Where exporting files is an option, the User Documentation needs to describe the file format of the export choices, and the titles and versions of specific software applications that are intended destinations.
• Does the documentation let the users know what type of machine and software they need in order to use this application? The user documentation must describe the appropriate hardware and software platform(s) required. This includes processor type, processor speed, memory required (RAM), hard disk space needed (kB or MB or even GB), graphics resolution needed, and the presence of any necessary hardware components such as CD-ROM drives, modems, pointing devices (mouse, joystick, etc.), multimedia components (speakers, special cards or drivers). The operating system(s) and any special third-party software that is required to run this software must also be specified.
• Does the layout of windows, tables, and graphical output in the manual agree with what is seen on the screen? The user documentation must accurately describe the layout of windows, tables, and graphical output plots that are displayed by the software. For example, if the documentation states that the OK and Cancel buttons are found at the bottom right of a particular screen, then this is where they need to appear on the screen.
• Do special formats used agree on the screen and in the documentation? Fonts, font styles (bold, italic), font size and font colors must agree on the screen and in the documentation description. For example, text or instructions that tell users to click selections with certain colors or fonts must be able to be followed exactly.
• Do screen snapshots agree with what is seen when using the software? Screen snapshots are often included in the user documentation to improve user understanding of the software's use. All screen shots in the documentation need to match what is seen by the user on the screen. If the user's screen will differ from the illustration, explain this situation. For instance, if the user is instructed to open the file "Test1.myg", the screen shot in the user manual must appear as "Test1.myg" and not "Example2.myg". The idea is to provide complete consistency between the user documentation and the screen, and thus make it easy for new users to learn how to use the software.

-- KeithNoddle - 04 Dec 2002