Testing

The IT glossary [2007-2010] says [with my additions]:

(1) Testing an information system involves checking that procedures, equipment, and staff process data as expected. Each component may be tested [component testing] and then the whole system tested [integration testing].

(2) Test data should be developed to ensure the procedures and formulas of a solution operate and process data as expected. This test is normally performed during [informal testing] and immediately after the development of the solution [formal testing]. When using a programming language it is called debugging. Once debugged, someone not involved in the program development tests it.

Testing is an art, a science, and (to some) a career. All features of a solution and its output must be tested, including:

• comparing the expected output with the actual output
• checking the relevance and completeness of the output
• performing manual calculations to check results of formulae

The specifications of software and hardware and testing have already been laid down during the design phase.

You need to test for correct operation under normal and abnormal conditions. Of particular concern is testing for boundary conditions where the behaviour of the system is meant to change at some important stage (e.g. if a person is 17 it does one thing, but after midnight on their 18th birthday the system should do something else). Boundary conditions are often where most bugs occur.

Testing can include:

• Informal testing - is carried out as each new feature of a system is incorporated to check it is working properly.
• Formal testing - is done when the solution is finished, and needs to be tested as a whole rather than as a set of parts.
• Component testing: does each component in the system work properly by itself, in isolation?
• Integration testing: when components are integrated do they still work properly and interact properly, or are they incompatible (due to hardware, software or data inconsistencies)? For examoke if a stock control system is added to the transaction processing system, can it warn the TPS when stocks are low or depleted?)
• Reliability - does it handle expected and exceptional situations well? e.g. can it handle the expected 10,000 transactions per hour? Will it recover gracefully if a subsystem fails or the power suddenly gets shut off? Will it detect invalid data?
• Is the output accurate and clear? Is it formatted properly? Does it provide each user of the system with relevant information in the format they require?
• Black box testing does not care how the system works - it just looks at the input and checks to see that the expected output comes out the other end. It does not care what happens inbetween.
• White box testing checks the processes within a component to make sure each small subcomponent (hardware or software) is producing the correct output.
• Acceptance testing is where the customer who commissioned the system demands a demonstration that the system meets its original specifications before you get paid.
• User acceptance testing involve testing whether the needs of the end-user of the system are satisfied. This can be done by observing end-users use the system and surveying a sample of users to find their opinions of the system. Before launching products, manufacturers often conduct market research to see how people respond to the new product (e.g. 'taste tests', supervised trials). A supermarket testing its new aisle layouts might observe trial customers and record their movements, time their actions and record the quantity and types of purchases they make. They might find, for example, that the new aisle layout is confusing and customers are buying less because they give up and leave. Interviews and questionnaires can discover the users' opinions, which cannot always be discovered by observation.

A common testing tool is the Testing Table, which has columns headed:

Attributes to Test

Before you can test a solution, you need to know what has to be tested. This should be decided on during the analysis of a problem. The attributes to test are those that are most important in a solution. If you were buying a race horse, you would test its speed, not its ability to type. If you are testing a military tank, you would mainly want to test its strength and reliability, not its attractiveness.

Depending on the nature of a solution, the main solution attributes to be tested include:

functionality - does it have the features required to do the job it's meant to do? Can an email program actually compose and send email? Does it need to have an address book, spam detector, filters, spell checker etc? If it does need them, do they work properly?

presentation - the appearance of some solutions is all-important. Software interfaces (the part of the program a user sees and interacts with), for example, must be carefully designed to be neat, attractive, logical and easy to understand. Published documents (e.g. word processed pages, web pages, magazines, instruction manuals) must be laid out carefully to be attractive and easily readable.

usability - of course, software must be easy to use if it is to succeed. Are commands easy to carry out, and do they produce expected results? Are dangerous commands suitably guarded to protect users from unintended actions? Other publications' usability needs testing too. For example, how easy is it to find a particular topic in a big document?

accessibility - can the solution be used by people with disabilities or restricted abilities? Can sight-impaired people, for example, use a word processor (it might use a synthesized voice to read text to them.) Typing requires quite nimble dexterity which many people lack - a solution might need to be tested for its ability to cater for those people.

communication of message - obviously, this is relevant to publications. Has the main message of a publication been successfully conveyed to the reader or audience? Do they understand it? Are they confused? Do they think they understand it, but actually don't?

reliability - can the solution be relied on to consistently produce the right results, even under abnormal conditions (e.g. corrupted data). Will it keep working all day ever day, or will it fail regularly? Testing software (debugging) is a long and complex job, and is rarely 100% successful.