Proficiency testing is an important element of ISO/IEC 17025 accreditation. So important, that it has its own ISO standard (i.e. ISO/IEC 17043).
In addition to the international standard, proficiency testing has plenty of endorsement via ILAC and accreditation body policies. There is a ton of information on proficiency testing available to laboratories seeking accreditation.
So, why do so many laboratories struggle with proficiency testing?
Most of the labs that I have spoken with either have a hard time finding a proficiency testing provider or have a unique test or calibration where proficiency testing is not available.
To get accredited, you need to know;
• all of the requirements that you need to meet beyond the ISO/IEC 17025,
• how to find a proficiency testing provider,
• how to understand your proficiency testing results,
• how to submit your results to your accreditation body, and
• what to do if you fail a proficiency test.
In this guide, I am going to cover everything that you need to know about proficiency testing for ISO/IEC 17025 accreditation including the answers to your problems listed above.
So, grab something to drink and take a seat, I am about give you a lot of information.
Personally, I have never had much of a problem meeting proficiency testing requirements.
Even with a 30+ page scope of accreditation, most of the measurement functions were able to be covered using a single proficiency testing provider.
The only measurement function that I had a hard time finding a PT provider was for chemical gas concentration.
Otherwise, I have only had to organize two interlaboratory comparisons in my career.
Looking at my experience, it seems that proficiency testing is a pretty easy requirement to meet. However, many labs seem to have a hard time.
So, what is the deal?
Is it a lack of knowledge, training, available information, or time? Whatever the reason, it appears that Proficiency Testing is a problem for many accredited labs.
On average, I probably encounter 3 clients a month who need help with proficiency testing and 20 or more subscribers who have questions.
Therefore, I decided to create a Proficiency Testing Guide to help you and others who struggle to meet ISO/IEC 17025 requirements.
In this guide, you will learn;
- What is Proficiency Testing
- Why is Proficiency Testing Important
- Proficiency Testing vs Inter-laboratory Comparisons
- Proficiency Testing Schemes
- How to Evaluate Proficiency Testing Results
- Proficiency Testing Standards
- ISO/IEC 17025 Requirements For Proficiency Testing
- ILAC Requirements For Proficiency Testing
- Proficiency Testing Requirements By Accreditation Body
- How To Find Proficiency Testing Service Providers
- How To Create A Proficiency Testing Program
If you would like to jump ahead to a particular section, just use the links provided above. Otherwise, take a seat. We have a lot of information to cover.
What is Proficiency Testing
According to ISO/IEC 17043:2010, proficiency testing (PT) is the evaluation of participant performance against pre-established criteria by means of interlaboratory comparisons.
In other words, a proficiency test is a method used to demonstrate competency and validate a laboratory’s measurement process by comparing your results to the results of a reference laboratory and other participant laboratories.
To get a better idea of what a proficiency test is, look at the image below.
A coordinating body sends a test item or artifact to a reference laboratory for testing. Then, the coordinating body sends the item to each participating laboratory for subsequent testing.
Each participant laboratory will independently test the item, submit their results to the coordinating body, and forward the item to the next participating laboratory.
After each participating laboratory has completed testing, the artifact is returned to the coordinating body.
The coordinating body will evaluate the all the test results and issue a performance report to each participating laboratory.
This is typically referred to as Round Robin Testing, and is one of the most common proficiency testing schemes used by PT providers.
Why is Proficiency Testing Important
Proficiency Testing is important for several reasons. Primarily, it enables your laboratory to demonstrate competency for a particular measurement discipline which can be used to validate;
• A measurement method;
• Technical training of personnel;
• Traceability of standards, and
• Estimates of measurement uncertainty.
Imagine that your laboratory is adding a new measurement or testing capability to your scope of accreditation. You have purchased equipment, had it calibrated, written methods, trained personnel, estimated uncertainty, and performed numerous internal verification studies.
Even with all of that hard work, time, and money invested into your new process, are you confident that your results are adequate and comparable to other laboratories?
By participating in proficiency testing, your laboratory can externally validate your new measurement or testing process.
Hence, the reason your accreditation body requires you to successfully complete a proficiency test before they will add it to your scope of accreditation.
We will cover more about that later in this guide.
Proficiency Testing vs Interlaboratory Comparisons
Proficiency testing and inter-laboratory comparisons are terms used synonymously in the Test and Measurement industry. However, they are not exactly the same.
They are similar, but slightly different.
According to ISO/IEC 17043:2010, inter-laboratory comparison (ILC) is the organization, performance, and evaluation of measurements or tests on the same or similar items by two or more laboratories or inspection bodies in accordance with predetermined conditions.
According to ISO Guide 43, Proficiency Testing is a formal exercise managed by a coordinating body which includes a standard or reference laboratory. The results are issued in a formal report that clearly provides the En and Z score.
Furthermore, ISO Guide 43 describes an inter-laboratory comparison as an exercise that is performed by agreement between two or more participating laboratories where the results are issued in a formal report.
So, the difference is a proficiency test is an inter-laboratory comparison that is organized and managed by an independent third party. Additionally, a proficiency test includes the participation of a reference laboratory and uses their results to determine participant performance.
An inter-laboratory comparison does not require the use of a reference laboratory or a coordinating body. Therefore, participant laboratories are only comparing performance amongst the group of participating members.
As you can see, they are similar, but different.
Proficiency Testing Schemes
When it comes to proficiency testing, there are a few different schemes used to conduct interlaboratory comparisons.
Each scheme is unique to maintain homogeneity and stability of the artifact throughout the testing process. Otherwise, the test results could contain errors and become invalid for use.
In this section, I will show you the two most common proficiency testing schemes recommended by the ISO/IEC 17043:2010;
• Sequential Participation Schemes
• Simultaneous Participation Schemes
If you conduct interlaboratory comparisons without the use of a proficiency testing provider, these test schemes may be of value to you.
Sequential Participation Schemes
In a sequential participation scheme, artifacts are successively circulated from one participant to the next, or occasionally returned to the proficiency testing provider or reference laboratory for retesting.
Sequential participation schemes are very common in proficiency testing. Two of the most common sequential designs are;
• Ring Test
• Petal Test
The ring test (i.e. round-robin test) is a proficiency testing scheme where a reference laboratory initially measures an artifact (to establish a reference) and then successively submits the artifact to each participant laboratory.
This ring test is typically used for artifacts known to have better long term stability.
The petal test is a proficiency testing scheme where a pivot laboratory is used to measure an artifact more than once during testing. In some cases, artifacts are returned to the pivot laboratory before and after each shipment to a participant laboratory.
The petal test is typically used for artifacts with short-term stability or when the participants are national metrology institutes (NMI’s).
Simultaneous Participation Schemes
In a simultaneous participation scheme, sub-samples are randomly selected from a material source and simultaneously distributed to participant laboratories for concurrent testing.
Simultaneous participation schemes are very common in proficiency testing and typically used for reference materials or single use samples that are destroyed or discarded after testing.
Three of the most common simultaneous testing designs are;
• Split-Level Test,
• Split-Sample Test, and
• Partial-Process Test.
In a split-level proficiency testing scheme, similar (but not identical) levels of a measurand are incorporated into two separate proficiency test items.
In a split-sample testing scheme, material or product samples are split into two or more parts, where each participant only test one part of the sample.
In a partial-process scheme, participants only perform specific parts of the overall testing or measurement process.
How to Evaluate Proficiency Testing Results
If you are going to participate in proficiency testing or inter-laboratory comparisons, it is beneficial for you to know how to evaluate your testing results.
This is especially true if you need to perform an interlaboratory comparison without the aid of a proficiency testing provider.
Furthermore, it is always a good idea to double-check your PT results even if you are using a proficiency testing provider.
I have found mistakes in proficiency testing reports before.
One time, a provider issued a report to my accreditation body indicating that I had an unsatisfactory result. However, when I double-checked the calculation of En, I discovered that I was well within.
So, I contacted the PT provider and notified them of my findings. When they double-checked the calculations, they discovered that there was a mistake. As a result, the proficiency testing report was updated and A2LA retracted their discrepancy letter.
Every once in a while it is nice to get a lucky break. However, you will never know if you do not verify your proficiency testing results. Therefore, let’s check out some of the methods used by proficiency testing providers.
Proficiency testing results are commonly evaluated using two methods described in ISO/IEC 17043;
- Normalized Error, and
Normalized error is a statistical evaluation used to compare proficiency testing results between the participant and the reference laboratory where the uncertainty in the measurement result is included.
Typically, it is the first evaluation used to determine conformance or nonconformance (i.e. Satisfactory/Unsatisfactory) in proficiency testing.
When determining whether a participant’s results are satisfactory or unsatisfactory, the following rules are used;
• When the value of |En| ≤ 1 (i.e. between -1 and +1), the results are considered satisfactory.
• When the value of |En| > 1 (i.e. greater than +1 or less than -1), the results are considered unsatisfactory.
To calculate normalized error, use the equation provided below;
If you are having a hard time understanding the equation above, use the step-by-step instructions below to calculate normalized error (i.e. En);
- Subtract the result from the participating laboratory by the result of the reference laboratory (i.e. laboratory bias).
- Calculate the root sum of squares for both laboratories’ reported estimates of measurement uncertainty.
- Divide the value calculated in step 1 and by the value calculated in step 2.
Z-score is a statistical measurement of a score’s relationship (i.e. how many standard deviations above or below the population mean) to the mean in a set of scores.
It is a statistical evaluation used to review the results of all participants and identify outliers and exclude their data from proficiency testing results.
When determining whether a participant’s results are satisfactory, unsatisfactory, or questionable, the following rules are used;
• When the value of Z <=2, the results are considered satisfactory.
• When the value of Z >=3, the results are considered unsatisfactory.
• When the value of Z >=2 and Z <=3, the results are considered questionable.
To calculate z-score, use the equation provided below;
If you are having a hard time understanding the equation above, use the step-by-step instructions below to calculate z-score;
- Subtract the participant laboratory’s result by the population mean (i.e. average).
- Calculate the standard deviation of all participant results.
- Divide the result of step 1 by the result of step 2.
Proficiency Testing Standards
The International Organization for Standardization (ISO) has a standard for just about everything, including proficiency testing.
If you are interested in becoming an accredited proficiency testing provider or want to learn more about proficiency testing standards, check out the list of active and obsolete ISO standards provided below.
Current and Active Standards
Obsolete and Withdrawn Standards
ISO/IEC 17025 Requirements For Proficiency Testing
The ISO/IEC 17025 standard does not specifically require laboratories to participate in proficiency testing. Instead, it strongly recommends it.
If you are surprised or confused, it is probably because your accreditation body has made it a requirement; not the ISO/IEC 17025 standard.
Proficiency testing is referred to several times in the international standard, but none of these references require you to participate.
If you do not believe me, let’s take a look at each section that mentions proficiency testing. Then, you can make the determination for yourself.
In section 5.9.1, the ISO/IEC 17025 states that interlaboratory comparison or proficiency testing programs may be used to monitor the validity of tests and calibrations undertaken by the laboratory.
See the excerpt below;
“5.9 Assuring the quality of test and calibration results
5.9.1 The laboratory shall have quality control procedures for monitoring the validity of tests and calibrations undertaken. The resulting data shall be recorded in such a way that trends are detectable and, where practicable, statistical techniques shall be applied to the reviewing of the results. This monitoring shall be planned and reviewed and may include, but not be limited to, the following:
b) participation in interlaboratory comparison or proficiency-testing programmes;
5.9.2 Quality control data shall be analysed and, where they are found to be outside pre-defined criteria, planned action shall be taken to correct the problem and to prevent incorrect results from being reported.”
Note 2 in section 4.4.1 states that proficiency testing results can be used to determine capability to perform a test or calibration as part of contract review.
“NOTE 2 The review of capability should establish that the laboratory possesses the necessary physical, personnel and information resources, and that the laboratory’s personnel have the skills and expertise necessary for the performance of the tests and/or calibrations in question. The review may also encompass results of earlier participation in interlaboratory comparisons or proficiency testing and/or the running of trial test or calibration programmes using samples or items of known value in order to determine uncertainties of measurement, limits of detection, confidence limits, etc.”
Note 2 in section 4.12.2 states that proficiency testing results can be used to analyze data for preventative actions.
“NOTE 2 Apart from the review of the operational procedures, the preventive action might involve analysis of data, including trend and risk analyses and proficiency-testing results.”
In section 4.15.1, the standard states that the results of interlaboratory comparisons or proficiency tests shall be part of the laboratory’s management review.
“4.15.1 In accordance with a predetermined schedule and procedure, the laboratory’s top management shall periodically conduct a review of the laboratory’s management system and testing and/or calibration activities to ensure their continuing suitability and effectiveness, and to introduce necessary changes or improvements. The review shall take account of:
– the results of interlaboratory comparisons or proficiency tests;”
As you can see, the ISO/IEC 17025 standard never states that proficiency testing is a requirement for compliance with the international standard. However, it appears to be highly recommended.
As you keep reading, you will find out that ILAC and your accreditation body may have a different opinion and requirements for ISO/IEC 17025 accreditation.
ILAC Requirements For Proficiency Testing
In addition to the ISO/IEC 17025 standard, the International Laboratory Accreditation Cooperation (ILAC) has a policy on proficiency testing. It is known as the;
It is a policy for accreditation bodies that sets requirements and gives guidance on the use of proficiency testing in the accreditation process for laboratories (i.e. ISO/IEC 17025:2005) and inspection bodies (ISO/IEC 17020:2012).
To fully understand the requirements of ILAC P9 policy, I recommend that you read the policy.
However, I have summarized the core concepts of the policy for you below.
Demonstration of Competency of Accredited Laboratories
In section 4.1, the policy requires accreditation bodies to demonstrate the technical competency of their calibration and testing laboratories (i.e. your laboratory).
One of the recommended ways that your laboratory can demonstrate competency is by participating in proficiency testing activities and yielding satisfactory results.
Minimum Activity Required For Accreditation
Section 4.2 requires accreditation bodies to establish the minimum proficiency activity required for accreditation, including;
• satisfactorily completing a proficiency test before granting accreditation.
• establishing an ongoing proficiency testing plan to cover your entire scope of accreditation.
Proficiency Testing Activities In the Accreditation Process
Section 4.3 requires accreditation bodies to have a policy on how proficiency testing will be used in assessments and the accreditation process which must include;
• a reference to the importance of proficiency testing,
• requirements for the minimum level and frequency of participation,
• instructions on how proficiency testing participation and performance will be used during the assessment and accreditation process.
• actions laboratories must take in response to poor performance
• how you will notify your accreditation body
• any other proficiency testing requirements set by regulators, industry, or other parties.
Proficiency Testing Policies, Procedures, and Plans
Section 4.4 requires accreditation bodies to fully document their proficiency testing polices and procedures. Furthermore, it requires accreditation bodies to review your proficiency testing plan for suitability in relation to your scope of accreditation.
Provide Information To Assist Laboratories
Section 4.5 recommends that accreditation bodies provide information to help laboratories find proficiency testing providers and develop proficiency testing plans, including;
• listings of proficiency testing providers and considerations for selecting programs, and
• guidance for analyzing and formulating proficiency testing needs.
Alternatives To Proficiency Testing
Section 4.6 requires accreditation bodies to negotiate alternative methods, with testing and calibration laboratories, to assess performance where proficiency testing does not exist or is not practical.
Proficiency Testing Requirements By Accreditation Body
The ILAC P9 policy lists a set of requirements that accreditation bodies must meet in order to maintain their status as a signatory to the ILAC MRA.
The way each accreditation body meets these requirements is slightly different. So, it is best for you to review your accreditation body’s policies and requirements.
Below is a list of links for you to quickly access your accreditation body’s policy.
Proficiency Testing Service Providers
One of the most common problems that I encounter when working with laboratories is the ability to find a suitable proficiency testing provider to cover their scope of accreditation.
For whatever reason, many laboratories struggle to find a suitable proficiency testing provider and develop a proficiency testing plan.
To help you out, I have compiled a list of 2 methods you can use to find a proficiency testing provider; and, what to do if there isn’t any suitable options available.
Accreditation Body Recommended Proficiency Testing Providers
Many people do not know this, but your accreditation body has documents and resources that will help you find a proficiency testing provider.
Below is a list of helpful links for each accreditation body in North America.
An Alternative Method to Find a Proficiency Testing Provider
If the resources provided above are not useful, I recommend that you visit your accreditation body’s website and search their database for ISO/IEC 17043:2010 accredited providers.
For example, let’s search the A2LA database.
1. Visit the A2LA website;
2. Click on “Search Accredited Organizations;”
3. Click on the “Accreditation Field” dropdown menu;
4. Select “Proficiency Testing Provider;”
5. Search for a Proficiency Testing Provider from the results.
In this example, the search returned 34 proficiency testing providers.
Now, you can search the providers scope of accreditation and website to see if their services can help your laboratory. If not, try searching another accreditation body’s website database.
For example, let’s try the ANAB search database.
1. Visit the ANAB website;
2. Hover your mouse over “Accredited Organizations,” and click “Labs/Inspection/Forensics/PT/RMP,”
3. Click on the “Standard” drop down menu and select “ISO/IEC 17043,”
4. Now, click on the “Search Now” button at the bottom of the screen,
5. Search for a Proficiency Testing Provider from the results.
After searching each of the accreditation body databases, I was only able to find proficiency testing providers in the A2LA and ANAB databases. However, there may be more PT providers available from the other accreditation bodies. So, make sure that you search their directories.
To help you find a proficiency testing provider faster, I have provided links below to each accreditation body’s database.
What To Do If A Proficiency Testing Provider Is Not Available
Now, some of you may be thinking, “What if there isn’t a Proficiency Testing Provider?”
First of all, don’t panic!
If you have endlessly searched for a suitable proficiency testing provider only to find one does not currently exist, it’s okay. It happens more often than you think, and the solution is rather simple.
Just follow this three-step process;
- Pick up the phone,
- Call your accreditation body, and
- Negotiate an alternative solution for your laboratory.
If you are too busy to make a telephone call, don’t like talking to people on the phone, or too scared to call your accreditation officer, then try this alternate two-step process;
- Compose an email to your accreditation body,
- Click send.
Shortly after, your accreditation body will reply to your email and help you negotiate an alternative method to proficiency testing.
Now I know that this may seem rather simple, but all you need to do is contact your accreditation body. They are obligated to help you find an alternative solution.
According to section 4.6 of the ILAC P9, accreditation bodies and the laboratory shall discuss and agree on suitable alternative means by which performance can be assessed and monitored.
How to Create A Proficiency Testing Program
If you are similar to the majority of ISO/IEC 17025 accredited laboratories, you will participate in proficiency testing. Therefore, you will need to establish a proficiency testing program.
To setup a program, you primarily need to do five things;
- Find a Proficiency Testing Provider,
- Create a Proficiency Testing Plan,
- Participate in Proficiency Testing Schemes,
- Review and Evaluate Your Results, and
- Submit Your Results to Your Accreditation Body.
Find a Proficiency Testing Provider
Once you have decided to participate in proficiency testing, you need to find one of more proficiency testing providers to cover your scope of accreditation.
Using your scope of accreditation, identify the measurement disciplines, functions, or tests listed in your scope of accreditation.
Next, use the resources that I provided to you earlier in this guide to help you find proficiency testing providers. When you have found a proficiency testing provider, contact them to get test availability and pricing.
Additionally, it is a good idea to keep a record of their scope of accreditation and add them to your Approved Supplier’s List.
Create a Proficiency Testing Plan
If you are participating in proficiency testing, you must create a plan. Most accreditation bodies require you to maintain a proficiency testing plan for a specified numbers of years.
In some cases, your proficiency testing provider will create a plan for you. If they offer to create your plan, let them. It will save you time and the hassle of doing it yourself.
However, if your provider does create a plan for your laboratory, make sure to review it and verify that the plan will cover your entire scope of accreditation. If you wish to change your plan or schedule, for whatever reason, contact your proficiency testing provider and ask them to revise your plan.
In the rare case that you wish to manage your own plan, I recommend that you create an excel spreadsheet that lists your proficiency testing activities for the next four years.
As an example, it could look like the image provided below;
Participate in a Proficiency Testing Scheme
Now that you have found a proficiency testing provider and created a plan, it’s time to participate in a proficiency testing scheme.
When it is your turn to participate, your proficiency testing provider will typically notify you a few weeks in advance so you can prepare to receive the artifact.
The key to performing proficiency tests is to essentially follow the instructions and perform the test similar to the tests or calibrations you are currently performing in your laboratory.
So, receive the artifact and run it through your normal test or calibration process similar to the work you perform for your customers.
When you have completed the test, contact your proficiency testing provider. They will tell you where to ship the artifact.
Finally, submit your results to the proficiency testing provider. Within a few weeks, you should receive your results.
Review and Evaluate Your Proficiency Testing Results
After each participate has completed the proficiency test, your provider will issue a finalized report to you. This report will showcase your performance and how you compared to other laboratories.
Furthermore, it will notify you whether your performance was satisfactory or unsatisfactory.
If your performance was satisfactory, great! You successfully completed a proficiency test. Now, evaluate your results, maintain your proficiency testing report for your records, and submit copies to your accreditation body.
On the other hand, if your performance was unsatisfactory, you have some work to do. You will need to follow your accreditation body’s requirements, but you will mostly likely need to document the nonconformance, conduct a root-cause investigation, and formulate corrective actions.
Afterward, you will need to implement the corrective action and provide your accreditation body with objective evidence that the corrective actions were effective. This is typically accomplished by repeating the proficiency test and showcasing that your results were satisfactory.
Again, refer to your accreditation body’s requirements and you will be okay.
Submit Your Results to Your Accreditation Body
Finally, submit your proficiency testing results to your accreditation body. If your proficiency testing provider offers to submit your results to your accreditation body, I recommend that you let them.
Otherwise you will have to do it.
Now, some consultants and assessors recommend that you submit the results yourself. However, I prefer to let the proficiency testing provider do it. In my opinion, it will save you time by delegating the task to your proficiency testing provider so you can focus on completing other tasks.
Where possible, I prefer to automate or delegate tasks so I can spend more time focusing on other things.
So, determine how you want to submit your results to your accreditation body and make sure to refer to their policies and requirements.
Proficiency testing is an important aspect of ISO/IEC 17025 accreditation. Even if it is not strictly required by the international standard, proficiency testing is required and highly recommended by international committees and accreditation bodies.
If there is a proficiency testing scheme available to support your scope of accreditation, you will most likely be
encouraged required to participate.
In this guide, I have given you 90% of everything you need to know about proficiency testing. With this information, you should be able to;
• develop a proficiency testing program,
• find a PT provider,
• find your accreditation bodies requirements,
• create a proficiency testing plan, and
• evaluate your results
Now, all you have to do is put in the work and make it happen. So, if you have been struggling with proficiency testing, use this guide to help you overcome your challenges and take action.
Develop a program, find a PT provider, and participate in your first proficiency test.
If you have any success stories, failures, challenges, or any additional helpful information, be sure to leave a comment below for others.