Calculating Uncertainty: 10 Terrible Mistakes You Need to Avoid

calculating uncertainty 10 mistakes to avoid

 

Introduction

Calculating uncertainty is not easy. It is a skill that requires education, training, and hours of practice to master. This is why it is one the top 10 deficiencies in ISO/IEC 17025 audits.

Whether you are a beginner or an expert at calculating uncertainty, I am sure that you have made mistakes. In fact, I have even made a number of mistakes over the years.

However, the goal is to learn from mistakes. The best part about learning from mistakes is they do not have to be your mistakes. You can learn from others mistakes. The benefit is you can improve your processes at the expensive of others pains and frustrations.

Sounds pretty good, right?

Well, in this article I am going to help you improve your ability to calculate uncertainty. I am going to show you the 10 most common mistakes that laboratories make when calculating uncertainty in measurement. Plus, I am going to teach you how to avoid these mistakes.

As a result, you should learn how to eliminate these mishaps from your uncertainty calculations and avoid deficiencies during your ISO/IEC 17025 audit.

If you are ready to learn, let’s get started.

 

Background

I have been in the Measurement Science industry for 14 years. However, I did not learn how to calculate uncertainty until 2008.

When I first started, my uncertainty calculations were not great. My uncertainty budgets looked awful and they did not include enough influences to measurement uncertainty. It definitely took me a couple of years to master the skill. During that time, I made some very silly mistakes.

So, I decided to start documenting the mistakes that I was making and track them. As a result, I was able to determine which mistakes I was routinely repeating. Therefore, I created checklists to help me perform quality assurance checks to avoid these mistakes.

Additionally, I monitored my staff and tracked which mistakes they were commonly making when estimating uncertainty. With this information, I could focus their training efforts to target these errors and improve their ability to calculate uncertainty.

However, I did not stop there. I started requesting statistical data (of audit results) from ISO/IEC 17025 accreditation bodies to see what makes other laboratories were making.

I have even monitored and tracked all of the mistakes that my clients and customers have made over the years.

From all of the data that I have collected, I was able to compile a list of the 25 most common mistakes people make when estimating uncertainty in measurement. Today, I am going to share the top 10 mistakes with you, so you can avoid making these mistakes and improve your uncertainty estimates.

 

Most Common Mistakes When Calculating Uncertainty

Over the years, I have compiled a list of mistakes people make when calculating uncertainty. Below is my list of top ten mistakes.

1. Not Calculating Uncertainty or Missing Uncertainty Budgets
2. Not Following the GUM Method to Calculate Uncertainty
3. No Measurement Traceability
4. Not Including Type A Uncertainty Data
5. Not Considering All Influences to Measurement Uncertainty
6. Combining Incompatible Units of Measure
7. Not Updating Your Uncertainty Budgets
8. Using Manufacture’s Uncertainty Specifications
9. Not Reporting Uncertainty to Two Significant Figures
10. Reporting Uncertainty Smaller Than Your CMC Uncertainty

Bonus: Learn how to avoid and fix the 10 biggest mistakes when calculating uncertainty. Get a FREE copy of my exclusive cheat sheet. Download the Cheat Sheet Now.

 

1. Not Calculating Uncertainty or Missing Uncertainty Budgets

The Problem

One of the biggest mistakes laboratories make is they are not calculating uncertainty in measurement. This means that they do not have;
Uncertainty budgets,
• Uncertainty calculations,
• Uncertainty estimates in their test or calibration reports.

Many laboratories spend months preparing for their ISO/IEC 17025 accreditation audit, but never calculate uncertainty or create a single uncertainty budget.

uncertainty budget example

As a result, they spend the next several months after their assessment calculating uncertainty for their scope of accreditation. In some cases, the laboratory takes so long to complete the process that they have conduct an entire new audit.

Not calculating uncertainty is an expensive mistake. The time and money spent fixing this problem could have been better spent elsewhere.

How to Fix It

To fix this, create a plan to prepare for your ISO/IEC 17025 audit. Make sure to read ALL of your application requirements and all of your accreditation body’s policies and requirements.

Next, make sure to develop a proposed scope of accreditation. Once the scope is complete, you now know what measurement functions need uncertainty budgets.

Now, calculate uncertainty for every measurement parameter that is in your proposed scope of accreditation. If your scope is too large to complete in time, focus on the most critical measurement functions that you must get accredited.

If you are unsure which measurement functions are most important, let your customers’ needs be your guide.

The Summary

To recap what you have learned;
1. Create a plan for you ISO/IEC 17025 audit,
2. Read your accreditation body’s polices and requirements,
3. Develop a proposed scope of accreditation,
4. Create uncertainty budgets for each function in your proposed scope.

 

2. Not Following the GUM Method to Calculate Uncertainty

The Problem

Another common mistake laboratories make when estimating uncertainty is they are not using the GUM method.

The GUM stands for the Guide to the Expression of Uncertainty in Measurement (GUM). It is the most common method used by calibration and testing laboratories to estimate uncertainty for ISO/IEC 17025. In fact, most of my articles and guides are based on the GUM method.

If you are not using this method, prepare to get a deficiency during your assessment; unless, you can adequately justify why you are not using it.

guide uncertainty in measurement GUM cover

How to Fix It

To fix this, learn how to use the GUM method by reading the Guide to the Expression of Uncertainty in Measurement. Next, put what you have learned into practice. Try using an uncertainty calculator or software to make the process easier.

If you find the GUM too complex, try downloading my uncertainty guide: 7 Steps to Calculate Measurement Uncertainty. It is easy to read and understand. Plus, it is written like a procedure to help you learn and apply the right process to calculate uncertainty.

The Summary

To recap what you have learned;
1. Read the Guide to the Expression of Uncertainty in Measurement,
2. Calculate uncertainty using the GUM method,
3. Use an uncertainty calculator or software to make the process easier.

 

3. No Measurement Traceability

The Problem

The single biggest mistake that laboratories make when calculating uncertainty is they do not have measurement traceability.

Measurement traceability is critical to the uncertainty analysis process. You must have it to properly estimate uncertainty. Without it, your results are completely unreliable.

Starting with the realization of the SI Unit, each calibration performed thereafter will have an increase in the magnitude of uncertainty associated with the measurement results. The further down the traceability chain you are, the larger your uncertainty will most likely be.

Therefore, you need to know the measurement uncertainty associated with the calibration of your equipment. Furthermore, you need to know that the measurement results are traceable. Without this information, it is difficult to calculate uncertainty with confidence.

How to Fix It

To fix this, obtain measurement traceability. First, get your equipment calibrated by an ISO/IEC 17025 accredited laboratory. Next, make sure your calibration reports contain As Found and As Left data. Finally, verify your calibration reports have estimates of measurement uncertainty.

You need to know the uncertainty associated with the calibration of your equipment. This will be known as your “Reference Standard Uncertainty” Or “Calibration Uncertainty.” The uncertainty that you calculate afterward cannot be less than this amount.

Next, you need to know your measurement results (i.e. As Found and As Left data) to determine how much bias is in your measurement results. So, you can either correct for bias or include it in your uncertainty analysis if you do not.

The Summary

To recap what you have learned;
1. Get your equipment calibrated by an ISO/IEC 17025 accredited laboratory,
2. Verify your calibration reports contain As Found and As Left data,
3. Verify your reports have estimates of measurement uncertainty.

 

4. Not Including Type A Uncertainty Data

The Problem

Type A data is important in uncertainty analysis because it allows you to quantify the actual variability associated with your measurement processes. It is real data collected by you from performing experiments. Plus, most accreditation bodies now require that you include it in your uncertainty analysis.

The most common forms of Type A data included in uncertainty analysis;
• Repeatability
• Reproducibility
• Stability

type a uncertainty data

Many laboratories do not include Type A data in their uncertainty analysis. As a result, they are rewarded with a deficiency during their ISO/IEC 17025 audit.

Therefore, assessors typically mark sections of their scope of accreditation with ‘TBD,’ and request that laboratories update their uncertainty budgets with Type A data.

How to Fix It

To fix this, create a plan and start testing. First, you should perform a repeatability test. Then, you should conduct a reproducibility experiment.

Afterwards, you should statistically analyze the results and add them to your uncertainty budgets. If you need help with analysis, I have created a great Statistics guide for uncertainty analysis.

The Summary

To recap what you have learned;
1. Create a plan,
2. Perform a repeatability test,
3. Perform a reproducibility test,
4. Analyze the results,
5. Add the results to your uncertainty budgets.

 

5. Not Considering All Influences to Measurement Uncertainty

The Problem

Uncertainty in measurement results can be influenced by many different factors. If you are not considering them in your uncertainty analysis, you may have a problem.

Some common influences to measurement uncertainty;
• Operator
• Environment
• Equipment
• Method
• Calibration

uncertainty-influence

Calculating uncertainty without considering all influences typically leads to inadequate or understated uncertainty estimates. Most likely, your auditor is going to find that you improperly estimated uncertainty and cite you a deficiency.

As a result, you will spend the next few months researching, testing, and analyzing new influences to include in your uncertainty analysis.

Although many auditors will make you feel like you did something wrong, most of the time, that is not the case.

Typically, this mistake occurs because people (like yourself) were unaware that some these influences existed. Don’t worry, it happens to all of us.

How to Fix It

To fix this, do some research and learn something new about your measurement processes.

First, look for reference material that may be able to help you identify new sources of uncertainty. Some great places to look are;
• Textbooks
• Industry guides
• Journal articles
• Conference Papers
• Results from other labs

Next, identify new sources of uncertainty and record your findings. It always helps to document what new sources of uncertainty you have found and how they influence your measurement results. Taking good notes is very important; especially, when an assessor wants to know more a particular influence.

Then, test your new sources to determine the how much they influence your measurement results. If you are unable to test your new influences, I recommend that you at least include the influence in your uncertainty budget and add a note that states you were unable to quantify the magnitude of influence.

However, it is a good idea to review the results from other laboratories performing similar measurements to see what data they found. You may be able to use data from their analysis.

The Summary

To recap what you have learned;
1. Do some research,
2. Identify new sources of uncertainty,
3. Test, Quantify, and Analyze,
4. Include the results in your uncertainty budgets.

 

6. Combining Incompatible Units of Measure

The Problem

Combining incompatible units of measure is like mixing oil and water; it does not work. You will incorrectly calculate uncertainty for your measurement process and end up with results that are significantly understated or overstated.

Think of something relatable, like money. You cannot add 20 US dollars and 20 Euros and claim that you have 40 US dollars or 40 Euros. You must first convert one to a compatible unit so you can determine how much money you actually have in total.

Units of measurement work the same way. You cannot add volts and amps together; and you cannot add feet and meters together (without converting first). It does not work.

You need to convert your uncertainty sources to similar units of measure to properly calculate combined uncertainty.

How to Fix It

To fix this, convert your uncertainty sources to similar units of measurement. To accomplish this, you try to use one of these methods to combine uncertainty with different units of measure;
• Sensitivity coefficients
• Fractional uncertainty
• Mathematical Simulation

By using any one of these three methods, you can convert your uncertainty influences to similar units of measure that are ready to calculate combined uncertainty.

The Summary

To recap what you have learned;
1. Identify the desired unit of measurement for your uncertainty analysis.
2. Convert your uncertainty sources to similar units.
3. Calculate uncertainty for your measurement process.

 

7. Not Updating Your Uncertainty Budgets

The Problem

Calculating uncertainty can be frustrating. If you are uncomfortable doing it, you are probably very happy after you have created all of the uncertainty budgets needed to cover your scope of accreditation.

Similar to many laboratories, you may think that you will never have to calculate uncertainty for you scope of accreditation again.

Well, you are wrong!

Consider the fact that things change over time, including uncertainty in measurement.
• your equipment performs differently,
• you have different technicians,
• your technicians have improved their skills,
• you are using different methods, or
• you are using different laboratories to calibrate your equipment.

The list above could go on and on forever.

Uncertainty in measurement results is influenced by so many factors that you need to routinely update your uncertainty budgets. As a minimum, I recommend that you at least update your uncertainty budgets once before each ISO/IEC 17025 audit.

How to Fix It

To fix this, create a plan and implement it.

Try updating your uncertainty budgets each time you have your reference standards calibrated. The last thing you want to do when preparing for an audit is calculate uncertainty for your entire scope of accreditation.

Instead, try breaking up this task into smaller tasks that you can perform throughout the year. When you get your standards back from calibration, update your uncertainty budgets.

It is simply much easier to manage.

BONUS: Plus, you will improve your ability to estimate uncertainty because you are performing it more frequently.

The Summary

To recap what you have learned;
1. Create a plan,
2. Implement it, and
3. Update your uncertainty budgets

 

8. Using Manufacture’s Uncertainty Specifications

The Problem

It is commonly known that many manufacturers publish uncertainty specifications in their manuals and datasheets. However, this does not mean that your measurement results will yield the same measurement uncertainty.

Fluke 5720 DC Voltage Specifcations

Depending on your measurement process, your estimated uncertainty will yield results that are larger or smaller than the manufacturer’s published uncertainty. In most cases, your measurement uncertainty will be larger than the values that the manufacturer claims.

The problem that most laboratories encounter during an ISO/IEC 17025 audit is their uncertainty budget only includes the manufacturer’s published uncertainty. Typically, this will not fly with assessors unless you can justify the use of manufacture specifications.

Now, there are times when using manufacture uncertainty specifications is acceptable, but most likely that is not your case.

How to Fix It

To fix this, perform your own uncertainty analysis. Collect some Type A data and review your calibration reports. Next, analyze your data and use an uncertainty budget or calculator to estimate your uncertainty.

The Summary

To recap what you have learned;
1. Perform your own uncertainty analysis,
2. Collect some Type A data,
3. Review your calibration reports,
4. Analyze your results,
5. Estimate uncertainty using a budget or calculator.

 

9. Not Reporting Uncertainty to Two Significant Figures

The Problem

Rounding and reporting calculated uncertainty to two significant figures is fairly easy to do. However, many laboratories cannot seem to get it right. So, they get cited a deficiency during their ISO/IEC 17025 audit.

According to section 7.2.6 of the Guide to the Expression of Uncertainty in Measurement Results, “the numerical values of the… expanded uncertainty U should not be given with an excessive number of digits. It usually suffices to quote… U to at most two significant digits.”

According to section 6.3 of the ILAC P14:01/2013: ILAC Policy for Uncertainty in Calibration, “the numerical value of the expanded uncertainty shall be given to, at most, two significant figures.”

As you can see, you must report your calculated uncertainty to two significant figures.

Some common reasons why this mistake occurs…
1. Personnel did not know what a significant figure is,
2. Oversight during quality assurance inspection,
3. Reporting software errors/limitations.

While the causes listed above may apply to you, it is not an excuse to omit reporting uncertainty to two significant figures. Your auditor will still write you up for nonconformance.

How to Fix It

To fix this, learn more about significant figures and how to round to two significant figures. Try this process.
1. Learn what a significant figure is,
2. Learn how to round to two significant figures,
3. Round calculated uncertainty to two significant figures,
4. Double check your results.

First things first. What is a significant figure?

According to the Meriam Webster Dictionary, a significant figure is “any of the digits of a number beginning with the digit farthest to the left that is not zero and ending with the last digit farthest to the right that is either not zero or that is a zero but is considered to be exact.”

Take a look at the image below. Here are two examples of reporting uncertainty to two significant figures.

round uncertainty two significant figures

Now, you want to find the first two significant figures. Next, you want to round the second significant figure to the nearest value (use conventional rounding). Finally, report your expanded uncertainty to two significant figures.

The Summary

To recap what you have learned;
1. Find the first two significant figures,
2. Round the third significant figure to the nearest zero,
3. Report your uncertainty to two significant figures,
4. Double check your results.

PRO TIP: Try using scientific notation to make the process easier.

 

10. Reporting Uncertainty Smaller Than Your CMC Uncertainty

The Problem

The CMC Uncertainty listed in your scope of accreditation is typically the BEST uncertainty in measurement results that your laboratory can provide under normal conditions.

When you report uncertainty that is smaller than what is listed in your scope of accreditation, you have a problem. You are breaking the rules!

According to the ILAC P14, section 3.2, CMC stands for Calibration and Measurement Capability available to your customers under normal conditions.

Additionally, in section 6.5, the policy states, “…accredited calibration laboratories shall not report a smaller uncertainty of measurement that the uncertainty of the CMC for which the laboratory is accredited.”

How to Fix It

To avoid this, implement this process for estimating uncertainty;
1. Read the ILAC P14,
2. Calculate uncertainty for your calibration results using;

a. your CMC Uncertainty,
b. UUT resolution, and
c. UUT repeatability.

3. Verify your calculated uncertainty is larger than your CMC Uncertainty.

If you implement this process, your estimated uncertainty should always be larger than the CMC Uncertainty published in your scope of accreditation. To be sure, always check your results.

PRO TIP: I recommend making it part of your quality assurance process when reviewing calibration reports before issuing certificates to your customer.

If you can provide measurement results with less uncertainty, try contacting your accreditation officer to find out how to update your scope of accreditation. It is far less paperwork than responding to an audit finding.

 

More Common Mistakes When Calculating Uncertainty

Below is a list of 15 more common mistakes people make when calculating uncertainty. Some of these mistakes are conditional depending on how your laboratory functions. I will be elaborating more on these mistakes in the future.

11. Not including UUT Resolution and Repeatability in Your Analysis
12. Not Including Measurement Bias in Your Analysis
13. Not Calculating Uncertainty for Each Test Point
14. Reporting CMC Uncertainty Directly From Your Scope of Accreditation
15. Not Expanding Uncertainty to 95%
16. Improperly Reporting Measurement Uncertainty
17. Incorrectly Rounding Uncertainty
18. Not Using An Uncertainty Budget
19. Incorrectly Calculating Measurement Uncertainty
20. Including UUT Resolution Twice in Your Uncertainty Analysis
21. Not Calculating Significance of Your Uncertainty Influences
22. Not Using Equations to Estimate Uncertainty
23. Not Using the Student’s T Table to Find Coverage Factors
24. Not Receiving Adequate Uncertainty Training
25. Not Contacting Professional Help When You Need It

 

Conclusion

Calculating uncertainty is not easy. So, it is quite common to see people make mistakes. Use the information in this article to help you prevent from making the similar mistakes.

Any one of the mistakes listed in this article can cause you to get a deficiency during your ISO/IEC 17025 audit. So, take the time to correct these mistakes now to avoid headaches later. No one likes spending hours of their precious time to correct audit deficiencies.

Now, tell me your story. If you have made one these mistakes and been cited a deficiency, leave a comment below and tell me what mistake you made.

Posted in:
About the Author

Richard Hogan

Richard Hogan is the CEO of ISO Budgets, L.L.C., a U.S.-based consulting and data analysis firm. Services include measurement consulting, data analysis, uncertainty budgets, and control charts. Richard is a systems engineer who has laboratory management and quality control experience in the Metrology industry. He specializes in uncertainty analysis, industrial statistics, and process optimization. Richard holds a Masters degree in Engineering from Old Dominion University in Norfolk, VA. Connect with Richard on LinkedIn.

12 Comments

  1. Rick’s writing on these topics is consistently as good as I ever see anywhere on the Internet. It is always thought provoking, very clear and never leaves me feeling not as smart as the author. He always is encouraging and very positive. I look forward to reading whatever he has to say each month. It also gives me insight into the general health of the practice of measurement uncertainty.

    Keep it up!

  2. One more mistake is omitting to include the “best measuring device” as per stated in ilac p14 when calculatinh CMC.
    I think also rounding relative values (%) is more difficult than absolute values. This may be given as example.
    Great article!!

    • Hi Ali,

      You can omit the “best existing device” from your CMC Uncertainty as long as you identify it in your Scope of Accreditation.

      In section 5.4 of the ILAC P14:01/2013, the 3rd paragraph states, “It is recognized that for some calibrations a “best existing device” does not exist and/or contributions to the uncertainty attributed to the device significantly affect the uncertainty. If such contributions to uncertainty from the device can be separated from other contributions, then the contributions from the device may be excluded from the CMC statement. For such a case, however, the scope of accreditation shall clearly identify that the contributions to the uncertainty from the device are not included.”

      Afterward, you can add the UUT resolution and UUT repeatability when calculate your calibration uncertainty. I have calculated uncertainty for years using this method. However, it does not work for everyone; so, choose the method that works best for you.

    • Hi Stephen,

      Thank you for you feedback. It is always appreciated.

      Feel free to shoot me an email if you would like to tell me some of the biggest mistakes that you have observed as a consultant. I would love to hear them!

      Rick

Leave a Reply

Your email address will not be published. Required fields are marked *