Introduction
Reproducibility testing is an important part of estimating uncertainty in measurement. It is a type A uncertainty component that should be included in every uncertainty budget.
However, a lot of laboratories neglect to test reproducibility and include it in their analyses.
When talking with clients and friends, I get a lot of reasons excuses for why they do not test reproducibility;
• “We don’t have time.”
• “We don’t know how.”
• “We are a one person laboratory.”
• “We only have one measurement standard.”
Well, I am here today to tell you that you can test reproducibility even if you are only a one person laboratory.
In this guide, you are going to learn all that you need to know, and more, to perform your own reproducibility tests.
If you are calculating uncertainty in measurement, you need to know how reproducible your measurement results are.
It is not only important for quality control. There are so many things that you can learn from testing reproducibility that it is worth the effort.
Background
There has been a lot of discussions and opinions about including type A uncertainty into your estimates of uncertainty in measurement.
Most laboratories include repeatability test data into their uncertainty budgets. However, many neglect to include reproducibility test data.
Why?
I am not sure, but I would like to find out.
Recently, I attended the A2LA Annual Meeting for the Measurement Advisory Committee; and, I was shocked to hear someone ask why ‘Reproducibility’ should be considered for uncertainty analysis.
After making the statement, this person continued speaking to tell everyone that it is not necessary or feasible because the laboratory only had one work station and one technician.
Honestly, I was shocked!
The person that made this statement is an intelligent person from a reputable company, and I was very confused why this person could not conceptualize the rationale for reproducibility testing.
Unfortunately, this person is not alone. Many people question the use reproducibility testing for estimating uncertainty. Over the years, I have had a number of readers, leads, and clients who have also had questions about reproducibility testing.
With this information, I can only assume;
1. They do not understand reproducibility testing,
2. They have not tried reproducibility testing, and(or)
3. They have not researched reproducibility testing
Therefore, I have decided to write a guide that will teach you everything that you will ever need to know about reproducibility testing for collecting type A uncertainty data.
Additionally, I am going to show you 5 ways that you can test measurement reproducibility for calculating uncertainty in measurement even if you are a single person laboratory with only one workbench.
So, if you are interested in collecting type A uncertainty data, keep reading because I am going to cover the following seven topics;
1. What is Reproducibility
2. Why is Reproducibility Important
3. Reproducibility Testing Requirements
4. Types of Reproducibility Tests
5. How to Perform Reproducibility Testing
6. How to Analyze Reproducibility Testing Results
7. How to Evaluate Reproducibility Test Results
What is Reproducibility
According to the Vocabulary in International Metrology (VIM), reproducibility is “measurement precision under reproducibility conditions of measurement.”
While reading the definition, I am sure that you noticed the terms “measurement precision” and “reproducibility condition of measurement.”
Don’t you hate it when terms are defined by their own words? Me too. It’s frustrating.
Therefore, let’s try to simplify the definition so you don’t need a PhD to understand the concept.
First, let’s define measurement precision. The VIM defines it as “the closeness of agreement between indications.”
Essentially, this is the spread or standard deviation calculated from a set of repeated measurements, most likely from a repeatability test.
Next, let’s define reproducibility condition of measurement. The VIM defines it as a “condition of measurement, out of a set of conditions that includes different locations, operators, measuring systems, and replicate measurements on the same or similar objects.”
Therefore, a reproducibility condition of measurement is another repeatability test where a condition of measurement has been changed.
Now that we have broken down the definition of reproducibility, it can be best explained as the standard deviation of multiple repeatability test results where the conditions of measurement have been changed.
The goal is to determine how closely the results of one repeatability test agree with another to determine how reproducible your results are when performed under various conditions.
Fundamentally, you need to recreate your measurement results after changing one variable at a time and evaluate the impact it has on your results.
Hopefully, you now have a better understanding of the term reproducibility.
In the next section, you will learn why it is important to estimating uncertainty in measurement.
Why is Reproducibility Important
Reproducibility is important because it demonstrates that your laboratory has the ability to replicate measurement results under various conditions.
Additionally, reproducibility testing offers you the ability to experiment with different factors that can influence your measurement results and estimated uncertainty.
When you know which factors significantly impact your measurement results, you can take action to control your measurement process and reduce uncertainty in measurement.
For example, if your laboratory were to conduct a reproducibility test evaluating every technician and one operator’s measurement results were significantly different from the sampled group, you could investigate the cause and take appropriate action (e.g. provide them training to improve their skills).
In another example, imagine that your laboratory is evaluating various test or calibration methods using reproducibility testing. If one method yields significantly different results than the other methods, you could take action to revise the method or eliminate its use for tests and calibrations.
As you can see, reproducibility testing can be a powerful tool for quality control and optimizing your laboratory processes. This is why it is important.
However, it is only beneficial if you actually analyze the data and use it to improve your measurement process. So, think beyond estimating uncertainty, reproducibility testing can be used for;
• Monitoring the quality of work,
• Validating methods, procedures, training, etc,
• Finding problems in measurement systems, and
• Increasing confidence in measurement results.
If your goal is to provide better quality measurement results with less uncertainty, start performing reproducibility tests and use the results to improve your process.
Requirements for Reproducibility Testing
Currently, reproducibility testing is not required by ISO/IEC 17025 or any other normative document; unless you are accredited via A2LA. However, there are strong recommendations for the use of reproducibility testing.
In this section, learn how reproducibility is referenced in various policies and requirements documents.
ISO/IEC 17025 International Standard
In section 5.4.5.3 of the ISO/IEC 17025 standard, reproducibility is listed as a value that may be obtained from validated methods.
Additionally, in note 3 under section 5.4.5.3, reproducibility is listed as an example or component where the uncertainty of values can be listed.
ILAC P14 Policy
In section 5.4 of the ILAC P14 policy recommends that reproducibility should be included in your estimation of CMC uncertainty.
“A reasonable amount of contribution to uncertainty from repeatability shall be included and contributions due to reproducibility should be included in the CMC uncertainty component, when available.”
JCGM 100:2008 Guide to the Estimation of Uncertainty in Measurement
In Appendix B, section 2.16 of the Guide to the Estimation of Uncertainty in Measurement (GUM), reproducibility is defined and provides a list of conditions or variables that can be changed for reproducibility testing.
ASTM E177 & ASTM E456
In table 1 of the ASTM E177, Practice for Use of the Terms Precision and Bias, reproducibility is listed as a condition for precision. Furthermore, the table below provides a list of conditions or variables that can be changed to determine reproducibility.
As you can see, reproducibility is not strictly required for estimating uncertainty; but, it sure is recognized as a significant contributor and recommended for consideration in your uncertainty budgets.
A2LA R205
Only one accreditation body in North America, A2LA, has made reproducibility a requirement for estimates of CMC uncertainty statements.
In section 6.7.1 of the A2LA R205 Specific Requirements: Calibration Laboratory Accreditation Program, reproducibility is listed as a key component that shall be considered in every CMC uncertainty calculation.
In my opinion, reproducibility is a significant contributor to uncertainty in measurement and should be included in your uncertainty budgets.
It is a type A uncertainty component that is just as important as repeatability.
Furthermore, it is a test that you can easily perform in your laboratory. It only takes a few additional minutes of your time to collect the data and analyze it.
Types of Reproducibility Tests
When it comes to reproducibility testing, there are a lot of different conditions that you can test. However, most metrologists (that I have spoken with or surveyed) only compare operators (i.e. one technician’s results compared to another).
While I agree that comparing operators is important, there are so many additional conditions that you can test. Why only test one condition?
You will never be sure that one condition is more significant than the other unless you test it yourself.
In this section, you will learn about five types of reproducibility tests that you can perform in your laboratory to collect type A uncertainty data. Afterwards, you should have enough information to help you decide which conditions are best for testing in your laboratory.
The five conditions that we will cover in this guide are;
1. Operator vs Operator
2. Equipment vs Equipment
3. Environment vs Environment
4. Method vs Method
5. Day vs Day
Operator vs Operator
The most common reproducibility test for collecting type A uncertainty data is looking for variances in measurement results via the operator.
By comparing two or more technicians, engineers, etc, you can learn a lot about inconsistencies in your measurement process.
All you need to do is have two technicians independently perform that same measurement process.
First, have technician A perform a simple repeatability test and record their results.
With this information, calculate the average and standard deviation.
Next, have technician B perform the same repeatability test and record the results.
Again, calculate the average and standard deviation.
Now, calculate the standard deviation of technician A’s average and technician B’s average.
The result will be the operator to operator reproducibility.
If the calculated result is larger than you prefer or larger than you expected, consider providing the technicians training and repeat the experiment.
Continue this process until you achieve a result that you are happy with.
To perform an operator vs operator reproducibility test, use the following instructions;
1. Perform a repeatability test with operator A.
2. Record your results,
3. Calculate the mean, standard deviation, and degrees of freedom,
4. Perform a repeatability test with operator B,
5. Record your results,
6. Calculate the mean, standard deviation, and degrees of freedom,
7. Calculate the standard deviation of the two means recorded in steps 3 and 6.
Day vs Day
Another common way to test reproducibility of a measurement process is to test the variance in measurement results from day to day.
This method is great for collecting type A uncertainty data for laboratories with only one technician, one workbench, or both.
The only thing that you need to change is the day or time that the test is performed.
For example, you can compare;
• Morning vs Afternoon,
• Monday vs Tuesday, or
• Monday vs Friday.
You can compare any scenario you want as long as the only variable that changes if day or time of day.
To get started, have a technician perform a repeatability test on day 1.
From their results, calculate the average and the standard deviation.
Next, have the technician perform the exact same repeatability test on day 2.
Again, calculate the average and standard deviation from their results.
Now, calculate the standard deviation of the averages calculated from day1 and day 2. The result will be your day to day reproducibility.
To perform a day vs day reproducibility test, use the following instructions;
1. Perform a repeatability test on day A.
2. Record your results,
3. Calculate the mean, standard deviation, and degrees of freedom,
4. Perform a repeatability test on day B,
5. Record your results,
6. Calculate the mean, standard deviation, and degrees of freedom,
7. Calculate the standard deviation of the two means recorded in steps 3 and 6.
Method vs Method
Testing methods for reproducibility is not a common test performed for type A uncertainty. However, it is a very beneficial reproducibility test if you are seeking to reduce your measurement uncertainty.
The method that you choose can significantly affect your measurement uncertainty, Therefore, method vs method reproducibility can be used to determine the variance between two measurement methods.
From the results, you can determine which measurement process yields less uncertainty in measurement results.
To perform method vs method reproducibility test, use the following instructions;
1. Perform a repeatability test using method A.
2. Record your results,
3. Calculate the mean, standard deviation, and degrees of freedom,
4. Perform a repeatability test using method B,
5. Record your results,
6. Calculate the mean, standard deviation, and degrees of freedom,
7. Calculate the standard deviation of the two means recorded in steps 3 and 6.
Equipment vs Equipment
If you work in a medium to large-size laboratory, you probably have multiple workbenches and duplicate measurement equipment.
Therefore, you are likely to have variance in your measurement results from using different workbenches or measurement equipment. Wouldn’t it be interesting to see how selecting different equipment can affect your measurement results?
If you said “Yes,” I recommend that you try this type of reproducibility test in your laboratory. You may be surprised by the results.
Every measuring instrument or device will perform differently from another, even though they may have the same manufacture and model number.
Even if your equipment was manufactured on the same day, on the same production line, it’s performance will slightly vary.
This is why it is important to consider equipment vs equipment reproducibility testing for your type A uncertainty data. The probability or likelihood that your measurement results will be affected by the equipment you choose is pretty high!
It makes a good substitute for operator vs operator reproducibility testing.
To perform equipment vs equipment reproducibility test, use the following instructions;
1. Perform a repeatability test using equipment A.
2. Record your results,
3. Calculate the mean, standard deviation, and degrees of freedom,
4. Perform a repeatability test using equipment B,
5. Record your results,
6. Calculate the mean, standard deviation, and degrees of freedom,
7. Calculate the standard deviation of the two means recorded in steps 3 and 6.
Environment vs Environment
Performing work outside of the laboratory is becoming more common these days. If your laboratory performs calibration or testing in the field or at customer sites, your measurement results will most likely be impacted by the environment where testing and calibration is performed.
Therefore, it may be a good idea to perform an environment vs environment reproducibility test.
To perform an environment vs environment reproducibility test , just use the following instructions;
1. Perform a repeatability test in environment A.
2. Record your results,
3. Calculate the mean, standard deviation, and degrees of freedom,
4. Perform a repeatability test in environment B,
5. Record your results,
6. Calculate the mean, standard deviation, and degrees of freedom,
7. Calculate the standard deviation of the two means recorded in steps 3 and 6.
Now, environment vs environment reproducibility testing does not have to be limited to in laboratory vs in the field.
Some laboratories have different rooms each with their own unique environment, even if they are performing similar tests and calibrations.
In these organizations, it may be a good idea to test the variance in measurement results between the two environmental conditions.
For example, I have piston gauges that require calibration. Although I send them to the same manufacture for calibration, the piston gauges may be calibrated in different environments.
In the past, I have received calibration reports that list different environmental conditions. One piston gauge will be calibrated at 20°C and the other one at 23°C.
It literally drives me nuts, because it creates extra work for me to correct the results for my laboratory environment or to compare the results from previous calibrations.
In another example, imagine calibrating an instrument in your laboratory at 20°C and a customer site at 28°C. Could this impact your measurement results?
Maybe? However, you will never know unless you tested it yourself and have data to support your claim.
How to Perform Reproducibility Testing
Performing a reproducibility test is pretty easy and straightforward. However, it never hurts to outline the process.
In this section, you will learn how to perform a reproducibility test step-by-step. All you need to do is following the instructions below.
1. Establish a Goal
2. Determine What You Will Test or Measure
3. Select a Variable or Condition to Change
4. Perform a Test With Variable A
5. Perform a Test With Variable B
6. Analyze the Results
1. Establish a Goal
When deciding to perform a reproducibility test, it is important to establish a goal.
Ask yourself, “Why am I performing a reproducibility test?”
Some common reasons people perform reproducibility tests are;
• Collect type A uncertainty data,
• Validate a new calibration or test method,
• Evaluate the quality and consistency of laboratory results,
• Investigate factors that cause measurement uncertainty or error.
There are plenty of reasons that may lead you to perform a reproducibility. However, you need to determine the purpose or goal.
Without determining the goal, you may incorrectly design the experiment. This could cause you to collect the wrong data and force you to repeat the experiments.
So, make sure you establish your goals first.
2. Determine What You Will Measure
Now that you know why you are performing a reproducibility test, it is time to determine what you will measure.
For example, maybe measuring DC voltage is a critical function in your laboratory. Therefore, you want to conduct a reproducibility test to verify that your measurement results are repeatable.
Choose the measurement function and parameter that you wish to test. Then, define how you will perform the measurement.
To complete this step, you will need to choose;
• When you will perform the test (Time),
• Where you will perform the test (Environment),
• How you will perform the test (Method),
• What you will use to perform the test (Equipment),
• Who you will perform the test (Operator).
3. Select a Variable or Condition to Change
Now that you have established the parameters of your reproducibility test, it is time to choose what condition you want to change in the experiment.
If you have multiple personnel who are qualified to perform a test or calibration, consider choosing operators as your condition.
This is the most common variable selected in reproducibility testing. It will offer you the ability to see how reproducible measurement results are when tests and calibrations are performed by different people.
If you perform test and calibrations in the laboratory and the field, consider changing the environment for your reproducibility test.
Therefore, perform one experiment in the laboratory and the other in the field.
It will also you to see how reproducible your measurement results are outside of the laboratory.
If your personnel can select different methods to perform a test or calibration, consider changing the method for your reproducibility test.
It will show you how much variance you may have selecting one method over another.
If you have duplication of standards or multiple workbenches for conducting similar tests or calibrations, consider selecting equipment as your variable.
This reproducibility test will offer you the ability to determine if your measurement results are repeatable choosing one piece of test equipment versus another.
If you do not have many options for testing other variables, try selecting time as your variable. Especially, if you are a laboratory with only one operator and one workbench.
This is easily accomplished by performing a day to day reproducibility test. It will allow you to see how much variance is in your measurement results over a period of time.
4. Perform a Repeatability Test with Condition A
Similar to an A/B Test, you will perform a reproducibility test to evaluate the results of condition A versus condition B.
Simply perform a repeatability test with condition A. Collect 20 or more repeated samples over a short period of time and record your results.
After performing the test, calculate the mean (i.e. average), standard deviation, and degrees of freedom for your sample data set.
I find it easy to log my results in a Microsoft Excel spreadsheet. Once the data is in the spreadsheet, I can easily analyze the data and store it on my computer.
Later on, I can use the spreadsheet to collect more data and compare the results with prior experiments.
Additionally, this is a great practice to add to your process if you plan to develop and use control charts.
5. Perform a Repeatability Test with Condition B
After completing the experiment with condition A, perform a new repeatability test. However, this time change your condition to option B.
That’s it.
It is really simple. You do not want to change anything else about the measurement process in the experiment except the condition.
Remember, the goal is to see how each condition effects the measurement results.
With condition B, perform a new repeatability test where you collect 20 or more repeated samples over a short period of time and record your results.
Afterward, calculate the mean (i.e. average), standard deviation, and degrees of freedom for your sample data set.
Then, proceed to the next step where you will analyze the results and determine the uncertainty introduced by the test variable.
6. Analyze the Results to Calculate Reproducibility
Using the data collected from the reproducibility test, you can analyze the results to calculate the uncertainty caused by reproducibility.
First, calculate the mean (i.e. average) result from testing with condition A.
Next, calculate the mean (i.e. average) result from testing with condition B.
If you tested more than two conditions in your experiment include the additional mean (i.e. average) measurement results in your analysis.
Now, calculate the standard deviation of the two or more mean values from the previous steps. This will be your value for reproducibility uncertainty.
To recap the process;
1. Calculate the Mean or Average of Results With Condition A,
2. Calculate the Mean or Average of Results With Condition B,
3. Calculate the Standard Deviation of Results for A and B.
Although this may seem like a lot of work, it really isn’t. In fact, this is typically a simple process to perform that does not require a lot of time.
When I perform repeatability and reproducibility testing in the laboratory, I usually block off 30 minutes of my day (at the beginning or end of the work day) to conduct these experiments.
Most of the time, it usually requires less than 15 minutes to complete the process including the data analysis.
So to successfully incorporate this process in your laboratory, do the following;
1. Determine what needs to be tested,
2. Determine how often it should be tested,
3. Set a schedule,
4. Implement, and
5. Repeat
Once you commit to adding this task to this to your calendar and stick to a routine, it becomes much easier to accomplish repeatability and reproducibility testing. Before you know it, you will have acquired a lot of type A uncertainty data.
How to Analyze Test Results
Although we briefly covered analyzing test results in the previous section. Let’s elaborate further on the process and add a few images to help you learn how to analyze your data.
First, you will want to record your results and enter them into a spreadsheet program. I prefer to use Microsoft Excel or Google Sheets.
It is much easier and faster than using statistical software such as SPSS, Minitab, JMP, etc.
1. Calculate the Mean or Average of Results With Condition A
After entering all of your test results into Microsoft Excel, you will want to begin analyzing the test results of condition A.
Using the ‘Average’ function in Excel, calculate the mean or average value. Simply, enter the following equation into an empty cell and hit enter;
=AVERAGE(Cell1:Cell#)
If done correctly, it should look like this;
2. Calculate the Mean or Average of Results With Condition B
Now that you have calculated the average of results for condition A, repeat the process for condition B and any additional conditions that you may have tested (e.g. C, D, E, etc.).
Again, using the ‘Average’ function in Excel, calculate the mean or average value. Simply, enter the following equation into an empty cell and hit enter;
=AVERAGE(Cell1:Cell#)
If done correctly, it should look like this;
3. Calculate the Standard Deviation of Results for A and B.
Finally, you will want to calculate the standard deviation of the average values you calculated for each condition. The result will give you a value to assign to your type A uncertainty for reproducibility.
Using the ‘STDEV’ function in Excel, calculate the standard deviation of the average values you calculated for each condition. Simply, enter the following equation into an empty cell and hit enter;
=STDEV(Cell1:Cell#)
If done correctly, it should look like this;
As you can see, calculating reproducibility is not difficult. It is just a process that you need to follow to adequately analyze your test results. The best part is that you can complete this task in a matter of minutes.
So, there is no excuse. Add reproducibility to your estimation of uncertainty in measurement.
How to Evaluate Test Results
Reproducibility test data can be used for a lot more than uncertainty analysis. It can show you how to improve the quality of your measurement processes.
You just have evaluate the test results to find problems and opportunities.
Without making this process overly complex, you should evaluate your results for;
1. Errors in Data,
2. Significant Differences in Mean (i.e. average) Value, and(or)
3. Significant Differences in Standard Deviation
By evaluating your test results for these three occurrences, you can quickly take action to improve your measurement process.
Errors in Data
Searching for errors in your test results is pretty straight forward. However, a lot of people don’t do it.
If you just set aside five or ten minutes of your time to review your data for errors, you can avoid problems later on.
However, you have to be careful. Sometimes errors are easy to find. Other times, they are not. You will have to use objective evidence and your judgement to find errors in your data.
If you are not sure, sometimes it is best to repeat an experiment for verification.
Significant Differences in Mean or Average Value
If you ever notice a significant difference in the reported mean or average value while evaluating data, you may want to conduct an investigation.
Most of the time, a large difference in the mean value can indicate a problem;
• An operator may lack the skill to perform the measurement process,
• A method may not be appropriate for the test performed,
• Measurement equipment may not be performing as intended,
• Environmental conditions may be unstable, or
• A test location may have introduced an additional error.
However, the difference in results could indicate a potential opportunity to improve your measurement process. For example;
• An operator may be more skilled than his coworkers,
• A method may minimize measurement uncertainty,
• Measurement equipment may be more accurate or precise,
• Environmental conditions may be more appropriate for the test performed,
• A location may be more favorable for the test performed.
Make sure look at the results from multiple perspectives. Be both optimistic and pessimistic.
You do not know what really happened or why it happened until you evaluate your results and conduct an investigation.
Even though it is difficult, try not assume that all significant differences are bad. You may miss opportunities for improvement.
Significant Differences in Standard Deviation
If you ever notice a significant difference in the reported standard deviations while evaluating data, you may want to conduct an investigation.
Many times, it is easy to assume that there is a problem or that it is a random error. However, you should really take the time to evaluate your results further to discover why it happened.
When you see larger standard deviations, there may be a problem. For example;
• An operator may lack the skill to perform the measurement process,
• A method may not be appropriate for the test performed,
• Measurement equipment may not be performing as intended,
• Environmental conditions may be unstable, or
• A test location may have introduced an additional error.
On the other hand, a smaller standard deviation may indicate an opportunity for improvement. For example;
• An operator may be more skilled than his coworkers,
• A method may minimize more sources of uncertainty,
• Measurement equipment may be more accurate or precise,
• Environmental conditions may be more appropriate for the test performed,
• A location more be more favorable for the test performed.
Again, you will not know what really happened or why it happened until you evaluate your results and conduct an investigation.
Most of the time, you will discover that significant difference indicate that there is a problem. However, every once in a while, you will find an opportunity to improve your measurement process.
In either case, it is always an opportunity for improvement. So, take action to improve your measurement process and the reproducibility of your measurement results.
All you need to do is put in the effort and take the time to evaluate your test results.
Conclusion
Reproducibility testing is an important component that should be added to your uncertainty budgets. It is a type A uncertainty source that is typically accompanied with repeatability data and shows you how good you are at reproducing your measurement results.
This is why it should be included in practically every uncertainty budget.
In this guide, you should have learned all about reproducibility testing, including five ways to conduct reproducibility experiments even if your laboratory has limited resources (e.g. personnel, equipment, etc.).
Most importantly, you should have learned;
• What is reproducibility,
• Why it is important, and
• How to calculate it
Now, I want you to give it a try.
Perform a reproducibility test in your laboratory. Then, post a comment and tell me what type of experiment you conducted and why.
It is just a matter to making the time and doing the work. So, make sure to perform your own reproducibility test and leave a comment below letting me know what type of experiment you conducted and why.
2 Comments