Objectives of vehicle certification

Vehicle certification is about ensuring that vehicles used on New Zealand’s roads meet the roadworthiness requirements defined in New Zealand law – when they enter the fleet and throughout their on-road lives.

Objectives of the Performance Review System (PRS)

The NZ Transport Agency’s goal is to improve transport for all New Zealanders by improving the integration, safety, responsiveness and sustainability of the transport system. To do this, everyone involved in vehicle certification must apply the requirements set out in law accurately and consistently. For certifiers, this means following the requirements specified in their Vehicle inspection requirements manual (VIRM) and other required documentation.

The PRS is a tool used by the Transport Agency for all certifiers and is intended to:

• focus on issues important to road safety
• be open and transparent in reviewing the performance of certifiers
• provide a level playing field, as it applies equally to all certifiers
• be easy to understand and use
• allow certifiers to measure their own performance in exactly the same way as the Transport Agency reviewer during a review
• allow and encourage certifiers to identify problems and opportunities to improve so they can take early action on their own initiative
• identify certifiers who perform well so they can be given incentives to maintain their performance and look for ways of continuously improving it
• identify certifiers who perform poorly so they can be encouraged through more frequent reviews to improve their performance.

How the PRS works

The PRS clearly sets out:

• what is expected of certifiers
• the way the Transport Agency will assess certifiers’ performance against these expectations
• a scoring system that enables certifiers’ performance to be measured.

The PRS is made up of two main parts:

The General part measures how well certifiers perform in complying with all legal requirements (including Transport Agency requirements) relating to vehicle certification. It does this by measuring performance in five categories:

Each category is made up of a number of elements that are relevant to the category. For example, the resources category is made up of the following elements:

• facilities
• technical equipment
• administration equipment
• technical information
• controlled certification documents
• certification staff.

In this manual each category has a scoring page associated with it which provides performance descriptions and corresponding scores for each element. Scores range from 0 to 3 and are allocated by matching a certifier’s performance to the corresponding descriptions set out in the scoring pages.

The Technical part also has scoring pages. The scores from these feed into the General part to produce a total score.

Figure 1-1-1. Sample scoring page (detail)

How the parts of the PRS fit together

How Transport Agency reviewers will use the PRS

During performance reviews, our reviewers will observe you at work. Reviewers will compare your performance with the descriptions on the PRS scoring sheets, and will score you accordingly. These scores will be combined into an overall PRS score.

We will tell you the result for each area reviewed, and you will be able to use your PRS manual to see how the score was calculated.

There are no hidden measurement or scoring systems. Reviewers are encouraged to help you understand how the system works and how it can be used to help continuous improvement. Success for the Transport Agency will be measured in higher performance scores – not in greater numbers of faults detected.

Use the PRS to improve your performance

We encourage certifiers to use the PRS regularly to review their own performance, and to identify problems and opportunities to improve. This section is called Measuring your performance, and the scoring guidelines section explains how this is done.

Benefits of good performance

The Transport Agency will reward good performance scores by reducing the frequency of planned review visits. This will save you money by reducing the cost of planned review fees. It will also reduce the disruption to your business caused by reviews.

Penalties for poor performance

Poor performance scores will reduce the Transport Agency’s confidence in you as a certifier. Such scores will mean more planned review visits so that your performance can be more closely monitored (see PRS Scores and actions table in the Measuring your performance section). Poor performance scores will increase the cost to you in planned review fees and the disruption to your business caused by reviews. We hope this will encourage you to use the results of the PRS to improve your performance.

Enforcement action

The PRS is a powerful tool to enable the Transport Agency to identify persistently poor performers. We are confident that most certifiers will respond positively to the PRS approach and use it to help improve their performance. When certifiers fail to respond positively and continue to perform poorly, this will be identified by PRS reviews. In these cases, the Transport Agency will focus its enforcement and investigation resources on obtaining evidence of poor performance.

The final sanction

Clear evidence of persistently unacceptable performance will lead to disciplinary action. If certifiers fail to respond to warnings and suspension action, evidence gained during reviews will be used to support the case for withdrawal of their authorisation.

Road safety and the maintenance of a fair vehicle certification system demand firm and decisive action by the Transport Agency. When certifiers fail to carry out their responsibilities the Transport Agency will act to remove their authorisation. This will be done in a fair and reasonable manner and decisions will be open to appeal.

Working together in partnership

The Transport Agency’s aim is to work together with certifiers openly and transparently to achieve our joint goals of high standards of certification, leading to an improvement in road safety.

Definitions and abbreviations

As part of the Performance Review System (PRS), you are expected to regularly assess your own performance (also known as self-assessment). You can do this using the electronic scoresheet available in the Master records section.

This section explains how to carry out a self-assessment and how you can measure your performance and calculate your total PRS score. Transport Agency Transport Officers will use the same measures when they review you to see how well you meet the Transport Agency’s expectations, as set out in this manual.

Purpose of self-assessments

When you carry out a self-assessment, your main purpose is to assess your performance against the descriptions in the General part and the Technical part of the PRS. This will enable you to identify and address any issues and improve the performance of your inspection work. It will also help you to achieve the highest score possible at your next review – and the higher your score, the less frequent the reviews.

When carrying out a self assessment, if you identify any areas where you need improvement (ie you score yourself less than 3), write it down on an Improvement record form along with what you will do to improve.

Overview of the self-assessment process

Before you do your first self-assessment, you should read these guidelines so you know how to score your performance. To help you understand the self-assessment process, the brief outline below shows you what is involved.

1. Print the scoring sheet PDF. It contains both the General Part and Technical Part scoring sheets. Alternatively, download you may use the electronic scoring sheets.

2. Read the descriptions for each element in the Technical and General parts of the PRS and identify the ones that best match your performance. We recommend you work from right (score 3) to left (score 0).

3. Enter the corresponding element scores into the Technical and General part scoring sheets as you go, and write any comments and items for improvement into the comments fields.

4. Calculate the average element scores for the Technical part (one each for Correct technical decisions, Technical competence and Technical equipment) and transfer these to the General part scoring sheet.

5. If you wish, you can calculate the category scores and the total and adjusted total scores. These will give you some idea of how well you perform overall.

6. Transfer any issues identified during your self-assessment onto an Improvement record form. Make sure you address these issues as soon as possible and record your actions on this form.

7. If you wish, you may like to recalculate the total and adjusted scores, taking into account the improvements you have made, so you can see how much you have improved your performance.

What are the scores and what do they mean?

The PRS calculates four types of scores:

• element scores
• category scores
• total score
• adjusted score.

The performance review system is based on four major score values:

0 – performance is unacceptable

1 – performance is marginal and requires substantial improvement

2 – performance is adequate but there is room for improvement

3 – performance is consistently good and meets the required standard.

In the PRS, you will find four descriptions (one for each score) for each element covered in the PRS. Assess your performance against these descriptions, and score yourself according to the one that most closely represents your performance.

If a description does not fully describe your performance, you can vary the score with:

• a plus (+) to indicate that your performance is slightly better than the description, or
• a minus (–) to show that your performance is close, but does not quite match the description.

This table sets out all the possible element scores and their numerical values. The numerical values are used to calculate your total score.

Using the scoring process described in the following pages, you will arrive at a total score for your performance.

How do I score myself?

The following guidelines explain how the total adjusted score is calculated and will allow you to understand the process. When you carry out your self-assessment, you may perform the calculations as shown or you may use the electronic scoresheets which are available on the Transport Agency website. These are easy to use and will automatically perform the calculations for you.

The performance review system (PRS) is divided into two parts. Your total score is determined by the General part. Scores from the Technical part feed into the General part.

For each part, you have a corresponding PRS Self-assessment record.

You can start with either the General part or the Technical part when carrying out your self-assessments.

If you start with the Technical part, proceed as follows:

Self-assessment – Technical part

• Decide how you want to carry out the technical assessment. For example, you may choose to observe a certifier carrying out an inspection, or carry out an inspection yourself, or arrange for another inspector to observe your inspection (peer review) or re-inspect a vehicle or component after a certifier has completed an inspection.
• Choose what vehicle components you want to assess. Your PRS technical Self-assessment record has space for seven items because this is the number of items for which the Transport Agency reviewer will check your performance in the course of a regular review. The Transport Agency recommends that you do self-assessments on all items over a period of time.
• Based on the descriptions in the Technical part of the PRS manual, score your or your staff’s performance for each element. Use the description that best fits your performance. Enter your scores in your PRS technical Self-assessment record.
• When you have completed the PRS technical Self-assessment record, calculate the average scores for each of these three elements:

– Correct technical decisions

– Technical competence

– Technical equipment.

Do this by:

• finding the equivalent numerical value for each score (use the element score/numerical values table as outlined on the following page)
• adding together all the numerical values for each element, then
• dividing the total by the number of scores you have assessed.

Alternatively, fill out the electronic scoresheets which are available on the Transport Agency website.

For example, for ‘Correct technical decision’ you may have scored the following vehicle components and assigned the corresponding numerical values:

Divide the total by the number of scores (7 in this case) to get your average Correct technical decision score:

14.7 ÷ 7 = 2.1

Follow the same process for Technical competence and Technical equipment.

Note that for some items no technical equipment is used, so no Technical equipment score can be allocated. In such cases, you would divide the total by the number of items you have assessed which can have a score.

When you come to fill in the PRS administration Self-assessment record, transfer these average scores into the appropriate sections:

Correct technical decisions: into section 1.2 of PRS administration Self-assessment record

Technical competence: into section 2.1 of PRS administration Self-assessment record

Technical equipment: into section 3.2 of PRS administration Self-assessment record

Self-assessment – General part

Based on the descriptions provided in the General part of the PRS, score your or your staff’s performance for each element using the description that best fits your performance. Enter your scores in your PRS general Self-assessment record.

• When you have completed the PRS general Self-assessment record, calculate the average score for each of the five categories of the administration part: certification outcomes, competence, resources, management and performance improvements.

Do this by:

• finding the equivalent numerical value for each score (use the element score/numerical values table as outlined on page 2)
• adding together all the numerical values for each element, then
• dividing the total by the number of scores in that category.

For example, in the certification outcomes category, add together the numerical values of your scores for the four elements in that category:

• Correct certification outcomes
• Correct technical decisions (from Technical part)
• Correct use of certification documents
• Correct entry of certification information

Then add up the four numerical values for this category and divide this figure by four to get the average.

How do I calculate my total score?

Work out your total score using the average scores of the five categories of the administration part. When you calculate the total score, each of the averages is ‘weighted’ to reflect its relative importance.

Use the table below to work out your total score. In the left column, fill in the average numerical values from your scores in the five categories of the administration part.

Then calculate the weighted scores by multiplying the averages by the corresponding weights. The total of the weighted scores is your total score.

Note that the total weighted value is not necessarily your total adjusted score as a further adjustment may be applied to take account of any serious areas of weakness.

For example:

Your total score is the total weighted value rounded to the nearest decimal place. In the example, your total score, based on a total weighted value of 2.22, is 2.2 – a good score.

The electronic scoresheet

All scores may be entered in an electronic scoresheet which is available in the Master records. This will calculate your total score automatically. It will also calculate an adjusted score which may be different from the total score.

How do I calculate my adjusted score?

Your total score can be adjusted downwards to reflect any major gaps in your knowledge or skills.

This is because the total score will be capped if there are any significant weaknesses in important areas. These are shown by scores of 1+ or less in any of the technical scores. The count of such scores can progressively limit the level of the total score you can achieve. Your total score adjusted by these limiters is your adjusted score.

The lowest score in the unweighted category scores may also affect the adjusted score.

These two factors, which may limit the total score to give an adjusted score, work as follows:

a) scores of 1+ or less scored for Technical decisions and Technical competence in the Technical part, and scores of 1+ or less in any of the administration part category scores (with the exception of Management). The higher the count of these scores, the lower the adjusted score will be. The following table illustrates their effect.

b) the lowest score in any category may downgrade the adjusted total further according to the following table.

The final consideration

The total adjusted scores will be used to determine how often you need to be reviewed. Consistently good total adjusted scores will mean fewer reviews (and lower costs) while consistently poor total adjusted scores will lead to more frequent reviews (and higher costs). Persistent low total adjusted scores will lead to some form of sanction.

Refer to the Scores and actions table  for further details.

Scores and actions table

Objectives of vehicle certification

Vehicle certification is about ensuring that vehicles used on New Zealand’s roads meet the roadworthiness requirements defined in New Zealand law – when they enter the fleet and throughout their on-road lives.

Objectives of the Performance Review System (PRS)

The NZ Transport Agency’s goal is to improve transport for all New Zealanders by improving the integration, safety, responsiveness and sustainability of the transport system. To do this, everyone involved in vehicle certification must apply the requirements set out in law accurately and consistently. For certifiers, this means following the requirements specified in their Vehicle inspection requirements manual (VIRM) and other required documentation.

The PRS is a tool used by the Transport Agency for all certifiers and is intended to:

• focus on issues important to road safety
• be open and transparent in reviewing the performance of certifiers
• provide a level playing field, as it applies equally to all certifiers
• be easy to understand and use
• allow certifiers to measure their own performance in exactly the same way as the Transport Agency reviewer during a review
• allow and encourage certifiers to identify problems and opportunities to improve so they can take early action on their own initiative
• identify certifiers who perform well so they can be given incentives to maintain their performance and look for ways of continuously improving it
• identify certifiers who perform poorly so they can be encouraged through more frequent reviews to improve their performance.

How the PRS works

The PRS clearly sets out:

• what is expected of certifiers
• the way the Transport Agency will assess certifiers’ performance against these expectations
• a scoring system that enables certifiers’ performance to be measured.

The PRS is made up of two main parts:

The General part measures how well certifiers perform in complying with all legal requirements (including Transport Agency requirements) relating to vehicle certification. It does this by measuring performance in five categories:

Each category is made up of a number of elements that are relevant to the category. For example, the resources category is made up of the following elements:

• facilities
• technical equipment
• administration equipment
• technical information
• controlled certification documents
• certification staff.

In this manual each category has a scoring page associated with it which provides performance descriptions and corresponding scores for each element. Scores range from 0 to 3 and are allocated by matching a certifier’s performance to the corresponding descriptions set out in the scoring pages.

The Technical part also has scoring pages. The scores from these feed into the General part to produce a total score.

Figure 1-1-1. Sample scoring page (detail)

How the parts of the PRS fit together

How Transport Agency reviewers will use the PRS

During performance reviews, our reviewers will observe you at work. Reviewers will compare your performance with the descriptions on the PRS scoring sheets, and will score you accordingly. These scores will be combined into an overall PRS score.

We will tell you the result for each area reviewed, and you will be able to use your PRS manual to see how the score was calculated.

There are no hidden measurement or scoring systems. Reviewers are encouraged to help you understand how the system works and how it can be used to help continuous improvement. Success for the Transport Agency will be measured in higher performance scores – not in greater numbers of faults detected.

Use the PRS to improve your performance

We encourage certifiers to use the PRS regularly to review their own performance, and to identify problems and opportunities to improve. This section is called Measuring your performance, and the scoring guidelines section explains how this is done.

Benefits of good performance

The Transport Agency will reward good performance scores by reducing the frequency of planned review visits. This will save you money by reducing the cost of planned review fees. It will also reduce the disruption to your business caused by reviews.

Penalties for poor performance

Poor performance scores will reduce the Transport Agency’s confidence in you as a certifier. Such scores will mean more planned review visits so that your performance can be more closely monitored (see PRS Scores and actions table in the Measuring your performance section). Poor performance scores will increase the cost to you in planned review fees and the disruption to your business caused by reviews. We hope this will encourage you to use the results of the PRS to improve your performance.

Enforcement action

The PRS is a powerful tool to enable the Transport Agency to identify persistently poor performers. We are confident that most certifiers will respond positively to the PRS approach and use it to help improve their performance. When certifiers fail to respond positively and continue to perform poorly, this will be identified by PRS reviews. In these cases, the Transport Agency will focus its enforcement and investigation resources on obtaining evidence of poor performance.

The final sanction

Clear evidence of persistently unacceptable performance will lead to disciplinary action. If certifiers fail to respond to warnings and suspension action, evidence gained during reviews will be used to support the case for withdrawal of their authorisation.

Road safety and the maintenance of a fair vehicle certification system demand firm and decisive action by the Transport Agency. When certifiers fail to carry out their responsibilities the Transport Agency will act to remove their authorisation. This will be done in a fair and reasonable manner and decisions will be open to appeal.

Working together in partnership

The Transport Agency’s aim is to work together with certifiers openly and transparently to achieve our joint goals of high standards of certification, leading to an improvement in road safety.

Definitions and abbreviations

This section explains how to carry out a self-assessment and how you can measure your performance and calculate your total PRS score. Transport Agency Transport Officers will use the same measures when they review you to see how well you meet the Transport Agency’s expectations, as set out in this manual.

Purpose of self-assessments

When you carry out a self-assessment, your main purpose is to assess your performance against the descriptions in the General part and the Technical part of the PRS. This will enable you to identify and address any issues and improve the performance of your inspection work. It will also help you to achieve the highest score possible at your next review – and the higher your score, the less frequent the reviews.

When carrying out a self assessment, if you identify any areas where you need improvement (ie you score yourself less than 3), write it down on an Improvement record form along with what you will do to improve.

Overview of the self-assessment process

Before you do your first self-assessment, you should read these guidelines so you know how to score your performance. To help you understand the self-assessment process, the brief outline below shows you what is involved.

1. Print the scoring sheet PDF. It contains both the General Part and Technical Part scoring sheets. Alternatively, download you may use the electronic scoring sheets.

2. Read the descriptions for each element in the Technical and General parts of the PRS and identify the ones that best match your performance. We recommend you work from right (score 3) to left (score 0).

3. Enter the corresponding element scores into the Technical and General part scoring sheets as you go, and write any comments and items for improvement into the comments fields.

4. Calculate the average element scores for the Technical part (one each for Correct technical decisions, Technical competence and Technical equipment) and transfer these to the General part scoring sheet.

5. If you wish, you can calculate the category scores and the total and adjusted total scores. These will give you some idea of how well you perform overall.

6. Transfer any issues identified during your self-assessment onto an Improvement record form. Make sure you address these issues as soon as possible and record your actions on this form.

7. If you wish, you may like to recalculate the total and adjusted scores, taking into account the improvements you have made, so you can see how much you have improved your performance.

What are the scores and what do they mean?

The PRS calculates four types of scores:

• element scores
• category scores
• total score
• adjusted score.

The performance review system is based on four major score values:

0 – performance is unacceptable

1 – performance is marginal and requires substantial improvement

2 – performance is adequate but there is room for improvement

3 – performance is consistently good and meets the required standard.

In the PRS, you will find four descriptions (one for each score) for each element covered in the PRS. Assess your performance against these descriptions, and score yourself according to the one that most closely represents your performance.

If a description does not fully describe your performance, you can vary the score with:

• a plus (+) to indicate that your performance is slightly better than the description, or
• a minus (–) to show that your performance is close, but does not quite match the description.

This table sets out all the possible element scores and their numerical values. The numerical values are used to calculate your total score.

Using the scoring process described in the following pages, you will arrive at a total score for your performance.

How do I score myself?

The following guidelines explain how the total adjusted score is calculated and will allow you to understand the process. When you carry out your self-assessment, you may perform the calculations as shown or you may use the electronic scoresheets which are available on the Transport Agency website. These are easy to use and will automatically perform the calculations for you.

The performance review system (PRS) is divided into two parts. Your total score is determined by the General part. Scores from the Technical part feed into the General part.

For each part, you have a corresponding PRS Self-assessment record.

You can start with either the General part or the Technical part when carrying out your self-assessments.

If you start with the Technical part, proceed as follows:

Self-assessment – Technical part

• Decide how you want to carry out the technical assessment. For example, you may choose to observe a certifier carrying out an inspection, or carry out an inspection yourself, or arrange for another inspector to observe your inspection (peer review) or re-inspect a vehicle or component after a certifier has completed an inspection.
• Choose what vehicle components you want to assess. Your PRS technical Self-assessment record has space for seven items because this is the number of items for which the Transport Agency reviewer will check your performance in the course of a regular review. The Transport Agency recommends that you do self-assessments on all items over a period of time.
• Based on the descriptions in the Technical part of the PRS manual, score your or your staff’s performance for each element. Use the description that best fits your performance. Enter your scores in your PRS technical Self-assessment record.
• When you have completed the PRS technical Self-assessment record, calculate the average scores for each of these three elements:

– Correct technical decisions

– Technical competence

– Technical equipment.

Do this by:

• finding the equivalent numerical value for each score (use the element score/numerical values table as outlined on the following page)
• adding together all the numerical values for each element, then
• dividing the total by the number of scores you have assessed.

Alternatively, fill out the electronic scoresheets which are available on the Transport Agency website.

For example, for ‘Correct technical decision’ you may have scored the following vehicle components and assigned the corresponding numerical values:

Divide the total by the number of scores (7 in this case) to get your average Correct technical decision score:

14.7 ÷ 7 = 2.1

Follow the same process for Technical competence and Technical equipment.

Note that for some items no technical equipment is used, so no Technical equipment score can be allocated. In such cases, you would divide the total by the number of items you have assessed which can have a score.

When you come to fill in the PRS administration Self-assessment record, transfer these average scores into the appropriate sections:

Correct technical decisions: into section 1.2 of PRS administration Self-assessment record

Technical competence: into section 2.1 of PRS administration Self-assessment record

Technical equipment: into section 3.2 of PRS administration Self-assessment record

Self-assessment – General part

Based on the descriptions provided in the General part of the PRS, score your or your staff’s performance for each element using the description that best fits your performance. Enter your scores in your PRS general Self-assessment record.

• When you have completed the PRS general Self-assessment record, calculate the average score for each of the five categories of the administration part: certification outcomes, competence, resources, management and performance improvements.

Do this by:

• finding the equivalent numerical value for each score (use the element score/numerical values table as outlined on page 2)
• adding together all the numerical values for each element, then
• dividing the total by the number of scores in that category.

For example, in the certification outcomes category, add together the numerical values of your scores for the four elements in that category:

• Correct certification outcomes
• Correct technical decisions (from Technical part)
• Correct use of certification documents
• Correct entry of certification information

Then add up the four numerical values for this category and divide this figure by four to get the average.

How do I calculate my total score?

Work out your total score using the average scores of the five categories of the administration part. When you calculate the total score, each of the averages is ‘weighted’ to reflect its relative importance.

Use the table below to work out your total score. In the left column, fill in the average numerical values from your scores in the five categories of the administration part.

Then calculate the weighted scores by multiplying the averages by the corresponding weights. The total of the weighted scores is your total score.

Note that the total weighted value is not necessarily your total adjusted score as a further adjustment may be applied to take account of any serious areas of weakness.

For example:

Your total score is the total weighted value rounded to the nearest decimal place. In the example, your total score, based on a total weighted value of 2.22, is 2.2 – a good score.

The electronic scoresheet

All scores may be entered in an electronic scoresheet which is available in the Master records. This will calculate your total score automatically. It will also calculate an adjusted score which may be different from the total score.

How do I calculate my adjusted score?

Your total score can be adjusted downwards to reflect any major gaps in your knowledge or skills.

This is because the total score will be capped if there are any significant weaknesses in important areas. These are shown by scores of 1+ or less in any of the technical scores. The count of such scores can progressively limit the level of the total score you can achieve. Your total score adjusted by these limiters is your adjusted score.

The lowest score in the unweighted category scores may also affect the adjusted score.

These two factors, which may limit the total score to give an adjusted score, work as follows:

a) scores of 1+ or less scored for Technical decisions and Technical competence in the Technical part, and scores of 1+ or less in any of the administration part category scores (with the exception of Management). The higher the count of these scores, the lower the adjusted score will be. The following table illustrates their effect.

b) the lowest score in any category may downgrade the adjusted total further according to the following table.

The final consideration

The total adjusted scores will be used to determine how often you need to be reviewed. Consistently good total adjusted scores will mean fewer reviews (and lower costs) while consistently poor total adjusted scores will lead to more frequent reviews (and higher costs). Persistent low total adjusted scores will lead to some form of sanction.

Refer to the Scores and actions table  for further details.

Scores and actions table