Evaluation

What is evaluation?

It's not testing! You are not finding faults to fix - they should have been found and fixed well before the system was implemented.

See more on the difference between evaluation and testing.

Evaluation measures how well the original ambitions of the new system (i.e. the logical design laid down during the analysis phase) have been achieved.

Evaluation doesn't really serve to improve the system that is being evaluated; it serves to improve the next system you will work on. You learn lessons for next time when you evaluate.

Different systems will have different criteria that demonstrate their effectiveness or efficiency. Just make sure that the criteria you evaluate are relevant to the goals the system was supposed to have when it was originally analysed. For example, if you buy a cow, you don't evaluate its speed, and if you buy a greyhound, you don't evaluate its milk output.

Typical evaluation criteria include:

• speed
• accuracy
• quality of output
• reliability
• cost of operation
• attractiveness
• ease of use
• robustness
• capacity
• compatibility with other systems
• functionality (does it have the features it needs?)
• security
• flexibility (can it be upgraded, modified, adapted, tweaked, configured?)
• size, portability

Criteria for evaluating efficiency

Do you remember the difference between efficiency and effectiveness?

Evaluation criteria describe what features will be measured. Evaluation methods describe how they will be measured. Don't get them confused!

Whenever the word "efficiency" pops up, you should automatically be thinking: "time, money, effort/labour". So let's use them as a structure for an answer.

To evaluate time efficiency, you could look at the time taken to:

• start the system
• enter data
• process data
• produce output
• communicate information
• maintain the hardware and software.

To evaluate economic efficiency, you could look at the costs of:

• initial purchase costs of hardware, software, installation, consultancy
• training and documentation
• wages
• consumables
• repairs, upgrades, maintenance, service contracts
• interest rates, leasing costs

To evaluate labour efficiency, you could look at:

• the number of staff required to operate the new system (e.g. the old system may have required 2 people, whereas the new system only requires 1 person)
• the number of man-hours spent operating the system (e.g. the new payroll system may take half the time to process weekly pay runs than did the original system

Criteria for evaluating effectiveness

Effectiveness is a measure of how well a job is done, regardless of the time, money and effort poured into it. It determines how well a systems intended goals have been satisfied. These goals are set down in a logical design during the analysis phase.

Evaluation methods

Once you know what criteria you need to evaluate in a system, you need to devise a way to evaluate it.

There are two main ways: objective and subjective.

Objective evaluation involves collecting facts, figures and measurements.

Subjective evaluation involves finding out opinions.

Where possible, get objective measurements. They are more reliable.

Typical evaluation methods include:

• timing how long an operation takes
• counting errors recorded in an error log
• visually inspecting output for quality, listening to sound reproduction
• counting and assessing incidents of failure or downtime
• observing users to see how quickly they learn to use the new system, how often they need help or make errors
• stress-testing the system or abusing it in ways that would be expected in real-life operation
• using the system to its stated capacity, or beyond (e.g. filling a database or hard disk drive)
• using the system in conjunction with another system to see if data and hardware are compatible
• carrying out functions to see if they exist and work as expected
• attempting to break into the system or get access to secured data
• attempt to configure the system to different users, or add modules, or change the way it works to suit new needs
• weigh it, measure it, carry it around to see if it's comfortable.

When to evaluate?

After implementation, but how long after?

o If you wait too long, users remember less about the development process and how it might be improved
o If too soon, users have insufficient time to assess system strengths and weaknesses

Six months of operation is desirable, but pressure to finish sooner often exists. Wait until the system is 'bedded down' and users are familiar with it, then evaluate whether it has solved the original problem. Has it achieved the goal specified in the Problem Analysis?

If not, fix the thing! Don't sit around with a new system that is as bad, maybe worse, than the old system.

Ideally, post-implementation evaluation should be performed by people who were not involved in the development process. External auditors often are involved, since they are impartial and don't have a stake in the success or failure of the system.

Example evaluation criteria and evaluation methods

• measure speed of operation
• meaure total output over a given time
• monitor staffing levels to see if fewer staff were needed

• survey customer opinion of output
• make subjective ("qualitative") judgements of product quality
• monitor complaints from customers
• monitor customer helpline enquiries about poor or good performance

• add up costs, income benefits
• compare current year's system costs to date with previous year

• monitor customer complaints
• monitor customer email related to errors or failures

• check fault log, service calls to get breakdowns fixed
• count hours of "downtime" when the system was unusable

• Monitor absentee rates (unhappy workers may be taking "sickies")
• Conduct staff surveys
• Time how long it takes to train new staff to use the system