How Often to Update Uncertainty Budgets

 
 

Introduction

A common question people ask is “When should I update my uncertainty budgets?” or “How often should I update my uncertainty budgets?” Since the question so common, I decided to create an article to answer the question and offer advice.

In this article, you will learn more about the following:

 
Scroll to keep reading or click one of the links above to go to that section.

 
 

When to Update Uncertainty Budget

According to the consensus of assessors and accreditation bodies, you should update your uncertainty budget anytime there is a significant change.

This consensus has come from conversations that I have had with assessors and accreditation body personnel in North America and Europe. Additionally, I agree with the consensus. It makes sense, and I recommend this advice to anyone who asks about updating uncertainty budgets.

 
 

What is a Significant Change?

Now, you may be wondering, “What is a Significant Change?” so you know when to update your budgets.

First, A2LA is the only accreditation body that defines significant. According to the A2LA R205 requirements document, “Significant” is a contributor that increases the CMC uncertainty by five percent (5%) or more.

Look at the image below to see the definition of significant from A2LA R205.

 
Based on the definition of “Significant,” you could infer that any change to your uncertainty budget that results in a 5% or greater increase or decrease in your CMC uncertainty is a significant change. Therefore, you need to update your uncertainty budget.

Although this definition is unique to A2LA, I have used it in uncertainty budgets with other accreditation bodies in North America, Europe, Africa, and Oceania. To date, I have not received any objective evidence or opinions from assessors or technical experts that suggest a different value. If you are aware of any alternative criteria, please share it with me.

 
 

Events to Update Uncertainty Budgets

Next, there are several events that can cause significant changes to your measurement uncertainty. Knowing these events can help you determine when to update your uncertainty budgets.

Therefore, consider updating your uncertainty budgets anytime there is a significant change to the following categories:

1. Equipment,
2. Methods,
3. Personnel,
4. Environment,
5. Suppliers, or
6. Unit Under Test (i.e. UUT).

In the sections below, you will learn more about event that typically cause significant changes related to the categories listed above. Knowing these events can prompt you to update your budgets after they occur.

 
 

A) Replacing or Upgrading Equipment and Reference Standards

When you upgrade or replace your laboratory equipment or reference standards, you need to update your uncertainty budgets. New equipment may perform significantly different than your previous equipment, especially if you upgraded to a newer or different model. Newer versions of equipment may have enhanced performance or additional capabilities (i.e. significant change) compared to previous models.

Therefore, you need to review your current uncertainty budgets and determine if they need to updated or not. Make sure to review equipment specifications and calibration reports to find significant changes. If you find significant changes, then update your uncertainty budgets and scope of accreditation.

Most of the time, replacing equipment requires you to create entirely new uncertainty budgets rather than simply updating older budgets. If this is true, then you will need to perform a new uncertainty analysis.

 
 

B) Replacing of Changing Reference Standards

When you change or replace your reference standards, you receive a new item with a new certificate of analysis or calibration report. The new reference standard may have a significantly different certified value and(or) the associated measurement uncertainty may significantly change.

In either case, this can affect your uncertainty analysis prompting you to update your uncertainty budgets. Make sure to review your new certificates and update your budgets.

If the new reference standard’s (i.e. reference material) certified value is significantly different than your previous reference standard, you need to create an entirely new uncertainty budget.

If the new reference standard’s certified value is similar to your previous reference standard, you should be able to simply update your uncertainty budget.

In the image below, you will see an uncertainty budget where the reference standard uncertainty, from the certificate of analysis, is updated after a reference standard material is replaced.

 
 

C) Calibration of Equipment

When your laboratory equipment and standards are calibrated, you may need to update uncertainty budgets. New calibration certificates may give you evidence of new performance trends and measurement uncertainties.

After your equipment is calibrated, review your new and previous calibration reports to see if there are any significant changes to the following sources of uncertainty:

1. Stability,
2. Drift,
3. Bias,
4. Reference Standard Uncertainty,
5. Specifications or Tolerances,
6. Linearity,
7. Hysteresis,
8. Interpolation Error, and
9. Other sources of uncertainty

If you find that any of these sources of uncertainty have significantly increased or decreased, then update your uncertainty budgets to make sure they are consistent your laboratory’s capabilities and records.

Additionally, if your CMC Uncertainty significantly changes, make sure to update your scope of accreditation.

Updating budgets based on calibration results is critical. During ISO/IEC 17025 assessments, auditors typically review your calibration reports (usually for metrological traceability) and may check to verify your uncertainty budgets match the performance trends observed in your calibration results.

If your uncertainty budgets are not supported by your calibration records, you are likely to get a nonconformance and the affected parameters in your scope of accreditation will be marked “TBD” until you update uncertainty budgets.

So, make sure to take appropriate action when this event occurs.

In the image below, you will see the typical sources of uncertainty that we update when laboratory equipment is calibrated.

1. Stability,
2. Bias,
3. Drift, and
4. Reference Standard Uncertainty.

Other factors such as linearity, hysteresis, interpolation error, etc are only updated if they are applicable to the evaluated measurement system.

 
 

D) Introduction of New or Updated Methods

You may need to update uncertainty budgets when:

1. New methods and procedures are introduced to the laboratory, or
2. Current methods and procedures are updated.

New methods may be significantly different than the methods currently used by the laboratory. When new methods are introduced, they should be reviewed and examined to determine if it will impact the uncertainty in measurement results. If yes, then update your uncertainty budgets.

When current methods are updated, the changes may be significantly different than previous versions. Additionally, updated standard methods may include new information on estimating the uncertainty in measurement results. These changes should be reviewed and examined to determine if it will impact the uncertainty in measurement results. If yes, then update your uncertainty budgets.

 
 

E) Changes in Personnel

Changes in personnel can significantly affect your measurement uncertainty. Therefore, consider updating uncertainty budgets when:

1. Key personnel leave the laboratory,
2. New personnel are added to the laboratory, or
3. Personnel qualifications and skills change.

If significant changes to personnel occur, then collect new repeatability and reproducibility data and update your uncertainty budgets.

In the image below, you will see the typical sources of uncertainty that we update when there significant changes to qualified laboratory personnel.

 
 

F) Changes in Environmental Conditions

Changes to environmental conditions can significantly affect your measurement uncertainty. Therefore, consider updating uncertainty budgets when:

1. The laboratory moves to a new location,
2. The laboratory begins performing testing or calibration in the field,
3. HVAC systems are upgraded or replaced,
4. Environmental monitoring equipment is upgraded or replaced,
5. Laboratory environmental limits change, or
6. Supplier calibrates your equipment in a different environment.

If significant changes to environmental conditions occur, then re-evaluate the effects of temperature, barometric pressure, relative humidity, local gravity, etc., and update your uncertainty budgets.

In the image below, you will see the typical sources of uncertainty that we update when there significant changes to the laboratory’s location and(or) environment.

Most of the time, temperature is the only uncertainty contributor updated in the uncertainty budget. However, it depends on the measurement system. Some measurements are affected by barometric pressure and(or) humidity. Therefore, you may need to evaluate them too.

Other measurements are affected by local gravity. If the testing or calibration activities are performed at a different location, you may need to update uncertainties related to local gravity.

 
 

G) Changing Suppliers

Switching suppliers and service providers can significantly affect your measurement uncertainty. Therefore, consider updating uncertainty budgets when:

1. Equipment is calibrated by different laboratories,
2. Reference materials and standards are purchased from a new supplier, or
3. Purchasing critical consumables from different manufacturers.

If new suppliers or products are used, then re-evaluate the effects on measurement uncertainty and update your uncertainty budgets.

For example, using new suppliers could impact the reference standard uncertainty found in calibration certificates and certificates of analysis.

Additionally, purchasing critical consumables from different suppliers or manufacturers can significantly impact your measurement performance.

Therefore, it is worth investigating and evaluating the effects changing suppliers has your measurement uncertainty.

In the image below, you will see the typical sources of uncertainty that we update when laboratories’ change suppliers.

 
When laboratories change suppliers for their reference materials and calibration services, the most commonly affected source of uncertainty is the reference standard uncertainty.

However, you should check your new supplier’s location and environmental conditions. If you measurements are effected by local gravity, a new supplier could have a significant impact on your uncertainty.

Additionally, your new supplier’s environmental conditions could be significantly different compared to your previous supplier. Furthermore, the environmental conditions could be significantly different than conditions recommended by the OEM or any standard methods used by your laboratory.

Any of these scenarios would require you to update your uncertainty budgets.

 
 

H) Changes to Unit Under Test

Significant changes to the items that you typically test or calibrate can significantly affect your measurement uncertainty. Therefore, consider updating uncertainty budgets when:

1. New item types are tested (test lab),
2. Best existing devices significantly change (calibration labs),
3. UUT resolution significantly changes, or
4. UUT repeatability significantly changes.

 
 

Calibration Laboratory

According to ILAC P14, calibration labs are required to include contributions from the unit under test (UUT) in their uncertainty budgets (unless they option to include them at the time of calibration). Therefore, when the best existing device calibrated by the laboratory changes, the lab should update the following items in their uncertainty budgets:

1. UUT resolution,
2. UUT repeatability, and
3. Any other significant contributor related to the UUT.

In the image below, you will see the sources of uncertainty that most calibration laboratories should update when there is a significant change to the unit under test (UUT).

 
 

Test Laboratory

For test laboratories, update uncertainty budgets when:

1. new types of test items are introduced to the laboratory, or
2. items tested by the laboratory significantly changes.

Determining what factors test labs update depends on the type of laboratory. Here are some common recommendations for what sources of uncertainty to update. However, review your testing activities and determine which uncertainty contributors need to be updated.

1. Test Laboratory (typical), update:
1. Repeatability,
2. Reproducibility;
2. Chemistry Test Laboratory¹, update:
1. Repeatability,
2. Reproducibility;
3. Reference Standard Uncertainty,
4. Sample inhomogeneity,
5. Sample Preparation uncertainties,
6. Standard preparation uncertainties, and
7. Equipment calibration uncertainties;
3. Microbiology Test Laboratory, update:
1. Technical Uncertainty (e.g. Reproducibility Replicates),
2. Inhomogeneity, and
3. Matrix uncertainty (if applicable).

 
In the image below, you will see the sources of uncertainty that most chemistry laboratories should update when a new analyte is added to the laboratory’s testing capabilities.

For more information, you can checkout the:

• EURACHEM CG4 Guide or
• ISOBudgets guide on estimating uncertainty in Chemistry.

 
In the image below, you will see the sources of uncertainty (per ISO 19036) that most microbiology laboratories should update when a new microbes and(or) matrices are introduced to the laboratory’s testing capabilities.

For more information, check out the:

• ISO 19036,
• A2LA G108 guide, or
• ISOBudgets guide to estimating uncertainty for Microbiology labs.

 
 

How Often to Update Uncertainty Budgets

People often ask me, “How often should I update my uncertainty budgets?”

The simple answer is: “Update uncertainty budgets anytime there is a significant change.”

However, you will not know if you have a significant change unless you have a routine, process, or a set of rules for finding significant changes.

Therefore, I recommend creating a plan for your laboratory. When creating a plan, you have two options. You can create a:

1. Time-Based Plan, or
2. Event-Based Plan

In the sections below, you will learn more about each type of plan.

 
 

Time-Based Plans

Updating uncertainty budget using time-based plan is easy. Create a schedule and update your budgets based on the specified schedule.

For example, if you want to update your budgets annually, then create a plan or schedule to review and update them once every 12 months.

If you prefer to update your budgets before each audit, then create a plan to review them six months before each assessment.

The plans given above are examples of time-based routines. Personally, I prefer using a time-based routine to update uncertainty budgets because they are easy to implement and manage. Simply create a schedule and add a recurring event to your calendar. When it is time to review and update budgets, the calendar will notify you.

This is a very simple way to update your budgets based on time.

 
 

Event-Based Plans

Alternatively, you can use an event-based plan to update your uncertainty budgets. Using an event-based plan can be more challenging because it requires more work to setup and implement. Typically, it is a better option for full-time Quality Management personnel compared to personnel that perform quality management as a collateral duty.

To setup an event-based plan, you first need to create a list of events that will prompt you to update your budgets. Using the significant events listed in the previous section would be a good start. When any of these events occur, update your uncertainty budgets.

“If [event] occurs, then review and update uncertainty budgets.”

You may be able to automate notifications for some of these events. For example, you can create a custom notification in your calibration management software (CMS) or laboratory information management software (LIMS) to trigger when:

1. New laboratory equipment calibration records are created or quality checked,
2. New equipment, standards, or materials are added to the system, or
3. New methods are added to the system.

Additionally, you may want to create a table or chart that list events that will prompt you to update your budgets, such as:

1. When management reviews are completed,
2. When nonconformities occur,
3. When PT/ILC schemes have unsatisfactory results, or
4. Any other event (you consider important) occurs.

If you prefer to use a chart or table, it is a good idea to also list what type of updates should be made based on the event that occurred. This can help you make updates faster because it eliminates the need to think about or lookup what action needs to be taken.

The solutions listed above are examples of event-based plans. They can be very effective but may be more difficult (for most) to implement and maintain.

 
 

Recommendations Based on Experience

Reviewing uncertainty budgets annually or after equipment is calibrated are the most common recommendations.

As a minimum, I recommend reviewing your uncertainty budgets once every two years. This is consistent with ISO/IEC 17025 audit schedules. Review your budgets 6 months before each assessment, so you have time to make any necessary updates.

Using a process similar to this will help you be more productive (with your main laboratory activities) and minimize your chances of a deficiency.

 
 

Recommendations Based on Audit Results

Most of the time, accredited laboratories do not update their uncertainty budgets until an assessor finds and documents a nonconformity. Based on our internal statistics, the likelihood of getting a deficiency for outdated uncertainty budgets significantly increases after 4 years.

Therefore, you are more likely to get a deficiency for outdated uncertainty budgets every two or three audits.

However, I have worked with laboratories that have not updated their uncertainty budgets in 10, 12, and 15 years (Yes, these numbers are correct); and, they have not received a deficiency for outdated budgets. Most of these labs have not had any significant changes to their personnel, equipment, methods, environment, etc. Therefore, it may be reasonable to presume that their uncertainty budgets are still valid for their laboratory activities.

Still, I would recommend that they routinely update their Type A uncertainties; specifically repeatability and reproducibility. However, it is up to assessors to find and cite these as deficiencies. Since no one has, these accredited laboratories haven’t performed an R&R study in 10 to 15 years which seems like a long time in my opinion.

Furthermore, based on our internal statistics, accredited calibration laboratories are more likely to get a deficiency for out-of-date uncertainty budgets compared to accredited testing labs. If you are a calibration laboratory, you should consider updating your budgets more frequently to avoid deficiencies.

 
 

Conclusion

Make sure to update your laboratory’s uncertainty budgets.

Outdated or obsolete uncertainty budgets can result in a nonconformity during an ISO/IEC 17025 assessment. According to the consensus of assessors and accreditation bodies, you should update your uncertainty budgets anytime there is a significant change.

In summary, the following is a list of common events can cause significant changes that prompt updating uncertainty budgets:

1. Replacing or upgrading laboratory equipment,
2. Replacing or changing reference materials and standards,
3. Calibration of laboratory equipment,
4. Recertifying reference materials and standards,
5. Changes in personnel,
6. Introduction or updating of test/calibration/sampling methods,
7. Changes in environmental conditions,
8. Changes in suppliers, and
9. Changes in test/calibration items.

Therefore, make sure to create a plan or routine for updating your uncertainty budgets. This will ensure that your budgets are up to date and audit ready.

If I left anything out, let me know in the comments below.