Proficiency testing is an important element of ISO/IEC 17025 accreditation. So important, that it has its own ISO standard (i.e. ISO/IEC 17043).
In addition to the international standard, proficiency testing has plenty of endorsement via ILAC and accreditation body policies. There is a ton of information on proficiency testing available to laboratories seeking accreditation.
So, why do so many laboratories struggle with proficiency testing?
Most of the labs that I have spoken with either have a hard time finding a proficiency testing provider or have a unique test or calibration where proficiency testing is not available.
To get accredited, you need to know;
• all of the requirements that you need to meet beyond the ISO/IEC 17025,
• how to find a proficiency testing provider,
• how to understand your proficiency testing results,
• how to submit your results to your accreditation body, and
• what to do if you fail a proficiency test.
In this guide, I am going to cover everything that you need to know about proficiency testing for ISO/IEC 17025 accreditation including the answers to your problems listed above.
So, grab something to drink and take a seat, I am about to give you a lot of information.
Personally, I have never had much of a problem meeting proficiency testing requirements.
Even with a 30+ page scope of accreditation, most of the measurement functions were able to be covered using a single proficiency testing provider.
The only measurement function that I had a hard time finding a PT provider was for chemical gas concentration.
Otherwise, I have only had to organize two interlaboratory comparisons in my career.
Looking at my experience, it seems that proficiency testing is a pretty easy requirement to meet. However, many labs seem to have a hard time.
So, what is the deal?
Is it a lack of knowledge, training, available information, or time? Whatever the reason, it appears that Proficiency Testing is a problem for many accredited labs.
On average, I probably encounter 3 clients a month who need help with proficiency testing and 20 or more subscribers who have questions.
Therefore, I decided to create a Proficiency Testing Guide to help you and others who struggle to meet ISO/IEC 17025 requirements.
In this guide, you will learn;
- What is Proficiency Testing
- Why is Proficiency Testing Important
- Proficiency Testing vs Inter-laboratory Comparisons
- Proficiency Testing Schemes
- How to Evaluate Proficiency Testing Results
- Proficiency Testing Standards
- ISO/IEC 17025 Requirements For Proficiency Testing
- ILAC Requirements For Proficiency Testing
- Proficiency Testing Requirements By Accreditation Body
- How To Find Proficiency Testing Service Providers
- How To Create A Proficiency Testing Program
If you would like to jump ahead to a particular section, just use the links provided above. Otherwise, take a seat. We have a lot of information to cover.
What is Proficiency Testing
According to ISO/IEC 17043:2010, proficiency testing (PT) is the evaluation of participant performance against pre-established criteria by means of interlaboratory comparisons.
In other words, a proficiency test is a method used to demonstrate competency and validate a laboratory’s measurement process by comparing your results to the results of a reference laboratory and other participant laboratories.
To get a better idea of what a proficiency test is, look at the image below.
A coordinating body sends a test item or artifact to a reference laboratory for testing. Then, the coordinating body sends the item to each participating laboratory for subsequent testing.
Each participant laboratory will independently test the item, submit their results to the coordinating body, and forward the item to the next participating laboratory.
After each participating laboratory has completed testing, the artifact is returned to the coordinating body.
The coordinating body will evaluate the all the test results and issue a performance report to each participating laboratory.
This is typically referred to as Round Robin Testing, and is one of the most common proficiency testing schemes used by PT providers.
Why is Proficiency Testing Important
Proficiency Testing is important for several reasons. Primarily, it enables your laboratory to demonstrate competency for a particular measurement discipline which can be used to validate;
• A measurement method;
• Technical training of personnel;
• Traceability of standards, and
• Estimates of measurement uncertainty.
Imagine that your laboratory is adding a new measurement or testing capability to your scope of accreditation. You have purchased equipment, had it calibrated, written methods, trained personnel, estimated uncertainty, and performed numerous internal verification studies.
Even with all of that hard work, time, and money invested into your new process, are you confident that your results are adequate and comparable to other laboratories?
By participating in proficiency testing, your laboratory can externally validate your new measurement or testing process.
Hence, the reason your accreditation body requires you to successfully complete a proficiency test before they will add it to your scope of accreditation.
We will cover more about that later in this guide.
Proficiency Testing vs Interlaboratory Comparisons
Proficiency testing and inter-laboratory comparisons are terms used synonymously in the Test and Measurement industry. However, they are not exactly the same.
They are similar, but slightly different.
According to ISO/IEC 17043:2010, inter-laboratory comparison (ILC) is the organization, performance, and evaluation of measurements or tests on the same or similar items by two or more laboratories or inspection bodies in accordance with predetermined conditions.
According to ISO Guide 43, Proficiency Testing is a formal exercise managed by a coordinating body which includes a standard or reference laboratory. The results are issued in a formal report that clearly provides the En and Z score.
Furthermore, ISO Guide 43 describes an inter-laboratory comparison as an exercise that is performed by agreement between two or more participating laboratories where the results are issued in a formal report.
So, the difference is a proficiency test is an inter-laboratory comparison that is organized and managed by an independent third party. Additionally, a proficiency test includes the participation of a reference laboratory and uses their results to determine participant performance.
An inter-laboratory comparison does not require the use of a reference laboratory or a coordinating body. Therefore, participant laboratories are only comparing performance amongst the group of participating members.
As you can see, they are similar, but different.
Proficiency Testing Schemes
When it comes to proficiency testing, there are a few different schemes used to conduct interlaboratory comparisons.
Each scheme is unique to maintain homogeneity and stability of the artifact throughout the testing process. Otherwise, the test results could contain errors and become invalid for use.
In this section, I will show you the two most common proficiency testing schemes recommended by the ISO/IEC 17043:2010;
• Sequential Participation Schemes
• Simultaneous Participation Schemes
If you conduct interlaboratory comparisons without the use of a proficiency testing provider, these test schemes may be of value to you.
Sequential Participation Schemes
In a sequential participation scheme, artifacts are successively circulated from one participant to the next, or occasionally returned to the proficiency testing provider or reference laboratory for retesting.
Sequential participation schemes are very common in proficiency testing. Two of the most common sequential designs are;
• Ring Test
• Petal Test
The ring test (i.e. round-robin test) is a proficiency testing scheme where a reference laboratory initially measures an artifact (to establish a reference) and then successively submits the artifact to each participant laboratory.
This ring test is typically used for artifacts known to have better long term stability.
The petal test is a proficiency testing scheme where a pivot laboratory is used to measure an artifact more than once during testing. In some cases, artifacts are returned to the pivot laboratory before and after each shipment to a participant laboratory.
The petal test is typically used for artifacts with short-term stability or when the participants are national metrology institutes (NMI’s).
Simultaneous Participation Schemes
In a simultaneous participation scheme, sub-samples are randomly selected from a material source and simultaneously distributed to participant laboratories for concurrent testing.
Simultaneous participation schemes are very common in proficiency testing and typically used for reference materials or single use samples that are destroyed or discarded after testing.
Three of the most common simultaneous testing designs are;
• Split-Level Test,
• Split-Sample Test, and
• Partial-Process Test.
In a split-level proficiency testing scheme, similar (but not identical) levels of a measurand are incorporated into two separate proficiency test items.
In a split-sample testing scheme, material or product samples are split into two or more parts, where each participant only test one part of the sample.
In a partial-process scheme, participants only perform specific parts of the overall testing or measurement process.
How to Evaluate Proficiency Testing Results
If you are going to participate in proficiency testing or inter-laboratory comparisons, it is beneficial for you to know how to evaluate your testing results.
This is especially true if you need to perform an interlaboratory comparison without the aid of a proficiency testing provider.
Furthermore, it is always a good idea to double-check your PT results even if you are using a proficiency testing provider.
I have found mistakes in proficiency testing reports before.
One time, a provider issued a report to my accreditation body indicating that I had an unsatisfactory result. However, when I double-checked the calculation of En, I discovered that I was well within.
So, I contacted the PT provider and notified them of my findings. When they double-checked the calculations, they discovered that there was a mistake. As a result, the proficiency testing report was updated and A2LA retracted their discrepancy letter.
Every once in a while it is nice to get a lucky break. However, you will never know if you do not verify your proficiency testing results. Therefore, let’s check out some of the methods used by proficiency testing providers.
Proficiency testing results are commonly evaluated using two methods described in ISO/IEC 17043;
- Normalized Error, and
Normalized error is a statistical evaluation used to compare proficiency testing results between the participant and the reference laboratory where the uncertainty in the measurement result is included.
Typically, it is the first evaluation used to determine conformance or nonconformance (i.e. Satisfactory/Unsatisfactory) in proficiency testing.
When determining whether a participant’s results are satisfactory or unsatisfactory, the following rules are used;
• When the value of |En| ≤ 1 (i.e. between -1 and +1), the results are considered satisfactory.
• When the value of |En| > 1 (i.e. greater than +1 or less than -1), the results are considered unsatisfactory.
To calculate normalized error, use the equation provided below;
If you are having a hard time understanding the equation above, use the step-by-step instructions below to calculate normalized error (i.e. En);
- Subtract the result from the participating laboratory by the result of the reference laboratory (i.e. laboratory bias).
- Calculate the root sum of squares for both laboratories’ reported estimates of measurement uncertainty.
- Divide the value calculated in step 1 and by the value calculated in step 2.
Z-score is a statistical measurement of a score’s relationship (i.e. how many standard deviations above or below the population mean) to the mean in a set of scores.
It is a statistical evaluation used to review the results of all participants and identify outliers and exclude their data from proficiency testing results.
When determining whether a participant’s results are satisfactory, unsatisfactory, or questionable, the following rules are used;
• When the value of Z <=2, the results are considered satisfactory.
• When the value of Z >=3, the results are considered unsatisfactory.
• When the value of Z >=2 and Z <=3, the results are considered questionable.
To calculate z-score, use the equation provided below;
If you are having a hard time understanding the equation above, use the step-by-step instructions below to calculate z-score;
- Subtract the participant laboratory’s result by the population mean (i.e. average).
- Calculate the standard deviation of all participant results.
- Divide the result of step 1 by the result of step 2.
Proficiency Testing Standards
The International Organization for Standardization (ISO) has a standard for just about everything, including proficiency testing.
If you are interested in becoming an accredited proficiency testing provider or want to learn more about proficiency testing standards, check out the list of active and obsolete ISO standards provided below.
Current and Active Standards
Obsolete and Withdrawn Standards
ISO/IEC 17025 Requirements For Proficiency Testing
The new ISO/IEC 17025:2017 standard requires laboratories to participate in proficiency testing. In the 2005 standard, proficiency testing was recommended in section 5.9.1b, but not required.
Now, it is required (where available and appropriate). Therefore, I recommend that you develop and implement a proficiency testing program if you do not have one in place.
To familiarize yourself with the ISO/IEC 17025:2017 standard, let’s take a look at all of the sections that mention proficiency testing so you can be better prepared for accreditation.
Proficiency Testing Definition
In section 3.5, the ISO/IEC 17025 standard defines the term “proficiency testing.” According to the standard, proficiency testing is the evaluation of participant performance against pre-established criteria by means of interlaboratory comparison.
The definition comes from the proficiency testing standard, ISO/IEC 17043:2010.
See the excerpt below:
“3 Terms and definitions
3.5 proficiency testing
evaluation of participant performance against pre-established criteria by means of interlaboratory comparisons (3.3)
[SOURCE: ISO/IEC 17043:2010, 3.7, modified — Notes to entry have been deleted.]”
Ensuring the Validity of Results
In section 7.7.2, the ISO/IEC 17025 standard states that laboratories shall monitor their performance by comparing their results with other laboratories.
The two methods that are recommended are:
a. Proficiency Testing or
b. Interlaboratory Comparisons
See the excerpt below:
“7.7 Ensuring the validity of results
7.7.2 The laboratory shall monitor its performance by comparison with results of other laboratories, where available and appropriate. This monitoring shall be planned and reviewed and shall include, but not be limited to, either or both of the following:
a) participation in proficiency testing;
NOTE – ISO/IEC 17043 contains additional information on proficiency tests and proficiency testing providers. Proficiency testing providers that meet the requirements of ISO/IEC 17043 are considered to be competent.
b) participation in interlaboratory comparisons other than proficiency testing.”
Externally Provided Products and Services
In section 6.6.1, the ISO/IEC 17025 standard states that laboratories must use only suitable externally provided services when it affect laboratory activities. If you read the note below the section, you will see that proficiency testing services should be included in externally provided services.
If you maintain an Approved Supplier List (like many other labs), then you may want to add your proficiency testing providers to it.
See the excerpt below:
“6.6 Externally provided products and services
6.6.1 The laboratory shall ensure that only suitable externally provided products and services that affect laboratory activities are used, when such products and services:
a) are intended for incorporation into the laboratory’s own activities;
b) are provided, in part or in full, directly to the customer by the laboratory, as received from the external provider;
c) are used to support the operation of the laboratory.
NOTE Products can include, for example, measurement standards and equipment, auxiliary equipment, consumable materials and reference materials. Services can include, for example, calibration services, sampling services, testing services, facility and equipment maintenance services, proficiency testing services and assessment and auditing services.”
In section 8.6.1, the ISO/IEC 17025 standard states that laboratories must identify and select opportunities for improvement and implement any necessary actions. If you read the note just below the section, you will see that the standard recommends using proficiency testing results to find opportunities for improvement.
See the excerpt below:
“8.6 Improvement (Option A)
8.6.1 The laboratory shall identify and select opportunities for improvement and implement any necessary actions.
NOTE Opportunities for improvement can be identified through the review of the operational procedures, the use of the policies, overall objectives, audit results, corrective actions, management review, suggestions from personnel, risk assessment, analysis of data, and proficiency testing results.”
As you can see, the ISO/IEC 17025:2017 standard requires you to participate in a proficiency testing or interlaboratory comparison program (where available and appropriate).
Additionally, the standard recommends that you;
• consider proficiency testing providers as service providers and
• use proficiency testing results to find opportunities for improvement.
Keep reading to find out what are ILAC and your accreditation body’s proficiency testing requirements for ISO/IEC 17025 accreditation.
ILAC Requirements For Proficiency Testing
In addition to the ISO/IEC 17025 standard, the International Laboratory Accreditation Cooperation (ILAC) has a policy on proficiency testing. It is known as the;
It is a policy for accreditation bodies that sets requirements and gives guidance on the use of proficiency testing in the accreditation process for laboratories (i.e. ISO/IEC 17025:2005) and inspection bodies (ISO/IEC 17020:2012).
To fully understand the requirements of ILAC P9 policy, I recommend that you read the policy.
However, I have summarized the core concepts of the policy for you below.
Demonstration of Competency of Accredited Laboratories
In section 4.1, the policy requires accreditation bodies to demonstrate the technical competency of their calibration and testing laboratories (i.e. your laboratory).
One of the recommended ways that your laboratory can demonstrate competency is by participating in proficiency testing activities and yielding satisfactory results.
Minimum Activity Required For Accreditation
Section 4.2 requires accreditation bodies to establish the minimum proficiency activity required for accreditation, including;
• satisfactorily completing a proficiency test before granting accreditation.
• establishing an ongoing proficiency testing plan to cover your entire scope of accreditation.
Proficiency Testing Activities In the Accreditation Process
Section 4.3 requires accreditation bodies to have a policy on how proficiency testing will be used in assessments and the accreditation process which must include;
• a reference to the importance of proficiency testing,
• requirements for the minimum level and frequency of participation,
• instructions on how proficiency testing participation and performance will be used during the assessment and accreditation process.
• actions laboratories must take in response to poor performance
• how you will notify your accreditation body
• any other proficiency testing requirements set by regulators, industry, or other parties.
Proficiency Testing Policies, Procedures, and Plans
Section 4.4 requires accreditation bodies to fully document their proficiency testing polices and procedures. Furthermore, it requires accreditation bodies to review your proficiency testing plan for suitability in relation to your scope of accreditation.
Provide Information To Assist Laboratories
Section 4.5 recommends that accreditation bodies provide information to help laboratories find proficiency testing providers and develop proficiency testing plans, including;
• listings of proficiency testing providers and considerations for selecting programs, and
• guidance for analyzing and formulating proficiency testing needs.
Alternatives To Proficiency Testing
Section 4.6 requires accreditation bodies to negotiate alternative methods, with testing and calibration laboratories, to assess performance where proficiency testing does not exist or is not practical.
Proficiency Testing Requirements By Accreditation Body
The ILAC P9 policy lists a set of requirements that accreditation bodies must meet in order to maintain their status as a signatory to the ILAC MRA.
The way each accreditation body meets these requirements is slightly different. So, it is best for you to review your accreditation body’s policies and requirements.
Below is a list of links for you to quickly access your accreditation body’s policy.
Proficiency Testing Service Providers
One of the most common problems that I encounter when working with laboratories is the ability to find a suitable proficiency testing provider to cover their scope of accreditation.
For whatever reason, many laboratories struggle to find a suitable proficiency testing provider and develop a proficiency testing plan.
To help you out, I have compiled a list of 2 methods you can use to find a proficiency testing provider; and, what to do if there isn’t any suitable options available.
Accreditation Body Recommended Proficiency Testing Providers
Many people do not know this, but your accreditation body has documents and resources that will help you find a proficiency testing provider.
Below is a list of helpful links for each accreditation body in North America.
An Alternative Method to Find a Proficiency Testing Provider
If the resources provided above are not useful, I recommend that you visit your accreditation body’s website and search their database for ISO/IEC 17043:2010 accredited providers.
For example, let’s search the A2LA database.
1. Visit the A2LA website;
2. Click on “Search Accredited Organizations;”
3. Click on the “Accreditation Field” dropdown menu;
4. Select “Proficiency Testing Provider;”
5. Search for a Proficiency Testing Provider from the results.
In this example, the search returned 34 proficiency testing providers.
Now, you can search the providers scope of accreditation and website to see if their services can help your laboratory. If not, try searching another accreditation body’s website database.
For example, let’s try the ANAB search database.
1. Visit the ANAB website;
2. Hover your mouse over “Accredited Organizations,” and click “Labs/Inspection/Forensics/PT/RMP,”
3. Click on the “Standard” drop down menu and select “ISO/IEC 17043,”
4. Now, click on the “Search Now” button at the bottom of the screen,
5. Search for a Proficiency Testing Provider from the results.
After searching each of the accreditation body databases, I was only able to find proficiency testing providers in the A2LA and ANAB databases. However, there may be more PT providers available from the other accreditation bodies. So, make sure that you search their directories.
To help you find a proficiency testing provider faster, I have provided links below to each accreditation body’s database.
What To Do If A Proficiency Testing Provider Is Not Available
Now, some of you may be thinking, “What if there isn’t a Proficiency Testing Provider?”
First of all, don’t panic!
If you have endlessly searched for a suitable proficiency testing provider only to find one does not currently exist, it’s okay. It happens more often than you think, and the solution is rather simple.
Just follow this three-step process;
- Pick up the phone,
- Call your accreditation body, and
- Negotiate an alternative solution for your laboratory.
If you are too busy to make a telephone call, don’t like talking to people on the phone, or too scared to call your accreditation officer, then try this alternate two-step process;
- Compose an email to your accreditation body,
- Click send.
Shortly after, your accreditation body will reply to your email and help you negotiate an alternative method to proficiency testing.
Now I know that this may seem rather simple, but all you need to do is contact your accreditation body. They are obligated to help you find an alternative solution.
According to section 4.6 of the ILAC P9, accreditation bodies and the laboratory shall discuss and agree on suitable alternative means by which performance can be assessed and monitored.
How to Create A Proficiency Testing Program
If you are similar to the majority of ISO/IEC 17025 accredited laboratories, you will participate in proficiency testing. Therefore, you will need to establish a proficiency testing program.
To setup a program, you primarily need to do five things;
- Find a Proficiency Testing Provider,
- Create a Proficiency Testing Plan,
- Participate in Proficiency Testing Schemes,
- Review and Evaluate Your Results, and
- Submit Your Results to Your Accreditation Body.
Find a Proficiency Testing Provider
Once you have decided to participate in proficiency testing, you need to find one of more proficiency testing providers to cover your scope of accreditation.
Using your scope of accreditation, identify the measurement disciplines, functions, or tests listed in your scope of accreditation.
Next, use the resources that I provided to you earlier in this guide to help you find proficiency testing providers. When you have found a proficiency testing provider, contact them to get test availability and pricing.
Additionally, it is a good idea to keep a record of their scope of accreditation and add them to your Approved Supplier’s List.
Create a Proficiency Testing Plan
If you are participating in proficiency testing, you must create a plan. Most accreditation bodies require you to maintain a proficiency testing plan for a specified numbers of years.
In some cases, your proficiency testing provider will create a plan for you. If they offer to create your plan, let them. It will save you time and the hassle of doing it yourself.
However, if your provider does create a plan for your laboratory, make sure to review it and verify that the plan will cover your entire scope of accreditation. If you wish to change your plan or schedule, for whatever reason, contact your proficiency testing provider and ask them to revise your plan.
In the rare case that you wish to manage your own plan, I recommend that you create an excel spreadsheet that lists your proficiency testing activities for the next four years.
As an example, it could look like the image provided below;
Participate in a Proficiency Testing Scheme
Now that you have found a proficiency testing provider and created a plan, it’s time to participate in a proficiency testing scheme.
When it is your turn to participate, your proficiency testing provider will typically notify you a few weeks in advance so you can prepare to receive the artifact.
The key to performing proficiency tests is to essentially follow the instructions and perform the test similar to the tests or calibrations you are currently performing in your laboratory.
So, receive the artifact and run it through your normal test or calibration process similar to the work you perform for your customers.
When you have completed the test, contact your proficiency testing provider. They will tell you where to ship the artifact.
Finally, submit your results to the proficiency testing provider. Within a few weeks, you should receive your results.
Review and Evaluate Your Proficiency Testing Results
After each participate has completed the proficiency test, your provider will issue a finalized report to you. This report will showcase your performance and how you compared to other laboratories.
Furthermore, it will notify you whether your performance was satisfactory or unsatisfactory.
If your performance was satisfactory, great! You successfully completed a proficiency test. Now, evaluate your results, maintain your proficiency testing report for your records, and submit copies to your accreditation body.
On the other hand, if your performance was unsatisfactory, you have some work to do. You will need to follow your accreditation body’s requirements, but you will mostly likely need to document the nonconformance, conduct a root-cause investigation, and formulate corrective actions.
Afterward, you will need to implement the corrective action and provide your accreditation body with objective evidence that the corrective actions were effective. This is typically accomplished by repeating the proficiency test and showcasing that your results were satisfactory.
Again, refer to your accreditation body’s requirements and you will be okay.
Submit Your Results to Your Accreditation Body
Finally, submit your proficiency testing results to your accreditation body. If your proficiency testing provider offers to submit your results to your accreditation body, I recommend that you let them.
Otherwise you will have to do it.
Now, some consultants and assessors recommend that you submit the results yourself. However, I prefer to let the proficiency testing provider do it. In my opinion, it will save you time by delegating the task to your proficiency testing provider so you can focus on completing other tasks.
Where possible, I prefer to automate or delegate tasks so I can spend more time focusing on other things.
So, determine how you want to submit your results to your accreditation body and make sure to refer to their policies and requirements.
Proficiency testing is an important aspect of ISO/IEC 17025 accreditation. Even if it is not strictly required by the international standard, proficiency testing is required and highly recommended by international committees and accreditation bodies.
If there is a proficiency testing scheme available to support your scope of accreditation, you will most likely be
encouraged required to participate.
In this guide, I have given you 90% of everything you need to know about proficiency testing. With this information, you should be able to;
• develop a proficiency testing program,
• find a PT provider,
• find your accreditation bodies requirements,
• create a proficiency testing plan, and
• evaluate your results
Now, all you have to do is put in the work and make it happen. So, if you have been struggling with proficiency testing, use this guide to help you overcome your challenges and take action.
Develop a program, find a PT provider, and participate in your first proficiency test.
If you have any success stories, failures, challenges, or any additional helpful information, be sure to leave a comment below for others.