They are not the same thing!
Testing is a method of finding out whether a solution is working as it should, e.g. giving correct output, working fast enough, handling expected loads, responding to user inputs properly.
Testing takes place during development as the solution is being built.
Testing is carried out by the solution’s developers.
Evaluation is a process of judging how well the system’s original intended goals have been achieved.
Evaluation happens after the solution has been developed, and its users have used it long enough to become familiar with it and can use it effectively.
Evaluation usually is carried out the by the solution’s owners and users.
CRITERIA
Testing and evaluation can use the same criteria, which are the qualities or attributes you want to test or evaluate.
Sample effectiveness criteria:
- accuracy
- robustness
- security
- enjoyability
- completeness
- readability
- attractiveness
- clarity
- accessibility (for those with disabilities)
- timeliness (being available when it’s needed)
- communication of message
- relevance
- usability
Efficiency criteria include:
- ease of use
- cost (purchase price
- speed
- the amount of labour required to make it work
METHODS
Evaluation and testing methods are the actions taken to measure the chosen criteria.
SOME TESTING METHODS
- Manual recalculation
- Penetration testing of networks
- Stress testing – deliberately working the solution hard to see if it fails under pressure
- Testing that validation rules work properly
- User acceptance testing by a typical user
- Using typical, unusual and invalid test data to test the solution’s behaviour
- Integration testing – to see if components of the solution interact properly
- performing all available user actions to check that the system responds appropriately
TYPICAL EVALUATION METHODS
- Interviewing users to get their opinions about the solution’s ease of use
- Questionnaires
- Surveys
- Analysis of records and data to find differences between the old system and the new one
- Observation of the system in actual use
Evaluation usually involves conducting interviews, surveys, looking at logs, observing the system in use, looking at the solution’s recent performance over time.
A comparison of testing and evaluation:
CRITERION |
TESTING |
EVALUATING |
Accuracy |
Manually recalculate a value and compare the result with the output of the solution’s formula |
Count the number of complaints about inaccurate payslips over the past 3 months |
Speed |
Use a stopwatch to time how long it takes to process and print a batch of invoices |
Use logs to count how many invoices have been issued in the past month |
Security |
Conduct a penetration test to find whether security can be bypassed |
Refer to security logs to find how many successful and thwarted intrusion attempts were recorded. |
Communication of message |
Give the solution to a typical user and test them on their understanding of it. |
Count the number of queries posted to the help desk or website forum relating to what the solution is trying to say. |
Reliability |
Pull out the power plug to see if there is major data loss |
Refer to system logs to count how many times the solution has failed over time |
With software - testing would mean to ensure that the program doesn't give any errors by looking at the internals. Evaluation occurs post-production and deals with customers to make sure that software you develop complies with their requirements.
Example – a company wants to increase their sales. They decide to create a fun website to attract young people who may buy their products.
They create the website and test all of its links, menus, readability, relevance, usability, accessibility.
They test every page in the major browsers and every test is 100% successful.
They put the website online and wait for the money to pour in.
They want to evaluate whether the site is successful in achieving their aims of popularity and profit.
What criteria should they evaluate? Popularity with young people. Increased profit because of the site.
So they look at the site statistics: they find large numbers of young visitors. They find zero profit.
They realise that none of these young visitors have credit cards or Paypal.
So testing showed that the site was fully functional and 100% accurate.
The evaluation showed it was a complete failure.
In conclusion: Evaluation is not meant to prove that the solution works. That’s the job of testing.
Evaluation is meant to measure how well the solution is meeting the user’s needs. |
