Study guide

Sensitivity, Variability, and Precision

ASQ Certified Calibration Technician CCT

Track your progress as you read

Finished sections get a check mark.

1 SI Definitions and Realizations

The Seven SI Base Units and the 2019 Revision Realizing SI Units in the Laboratory

2 Derived Units, Prefixes, and Constants

Derived Units and Physical Calculations SI Prefixes and Compound-Unit Conversions Physical Constants and Their Applications

3 Measurement Standards and Selection

Types and Roles of Measurement Standards Selecting Standards for an Operational Hierarchy

4 Traceability and Standard Substitution

Traceability Evidence and Impact Review Evaluating and Authorizing Standard Substitution

5 Dimensional and Mechanical Metrology

Linear Displacement and Dimensional Instruments Geometry, Thermal Expansion, and Mass Force, Torque, Pressure, Density, and Viscosity

6 Temperature, Humidity, Volume, and Flow

Temperature Sensors and Thermometer Corrections Humidity, Volume, and Flow

7 Electrical, Optical, and Time Measurements

Voltage, Current, Resistance, and AC Measurement Time, Frequency, Light, Power, and Energy

8 Measurement Methods

Direct, Indirect, and Ratio Methods Transfer, Differential, and UUT Substitution Methods

9 Instrument Characteristics and Measurement Data

Sensitivity, Variability, and Precision Bias, Linearity, Stability, and Hysteresis Data Formats, Readability, and Suitability Resolution, Confidentiality, and Data Integrity

10 Specifications and Error Sources

Interpreting IM&TE Specification Terms Specification Calculations and Operating Qualifiers Instrument, Operator, and Circuit Errors Environmental, Geometric, and Interference Errors

11 Measurement Assurance

Check Standards and Statistical Process Control Proficiency Comparisons and Gage R&R

12 Procedures and Governing Requirements

Procedure Structure and Published Practice Sources Method Development, Deviations, and Handling Regulations, Contracts, and Requirement Precedence

13 Instrument Adjustment and Standardization

Calibration, Zero, and Span Adjustment Linearization, Nulling, and Digital Adjustment

14 Calibration Environment

Temperature and Humidity Control ESD, Dust, Vibration, and Pressure

15 Operations, Roles, and Scheduling

Receiving, As-Found Testing, and Calibration Flow Asset Identification, Tracking, and Release Laboratory Roles and Initial Scheduling Interval Review, Overdue Control, and Workloads

16 Method and Software Validation

Method Verification and Validation Studies Software, Firmware, and Spreadsheet Validation

17 Records and Formal Reporting

Controlled Records and Calibration Certificates Status Labels and Out-of-Tolerance Reporting

18 Uncertainty Concepts and Decision Risk

Error, Uncertainty, and Measurand Definition TAR, TUR, Tolerance Consumption, and Capability Conformity Decisions and False-Accept Risk Guard Bands and Conformity Reporting

19 Uncertainty Models and Budgets

Measurement Models and Uncertainty Contributions Distribution Selection and Budget Assembly

20 Type A and Type B Evaluation

Type A Statistics and Repeatability Outlier Review and Worked Type A Analysis Type B Distribution Models

21 Uncertainty Reporting and Applied Mathematics

Propagation, Covariance, and Effective Degrees of Freedom Coverage Factors and Expanded-Uncertainty Reporting Notation, Unit Conversions, Ratios, and Decibels Rounding, Algebra, Graphs, and Angular Conversions

22 Quality Systems and Tools

Quality-System Components, Planning, and Improvement The Seven Basic Quality Tools

23 Audits and Corrective Action

Audit Types, Planning, and Nonconformance Control Impact Assessment, Root Cause, and Effectiveness

24 Safety, Standards, and Professional Conduct

Electrical, Pressure, Chemical, and Laboratory Hazards PPE, Safety Data Sheets, 6S, and Safe Restoration Quality Standards, GUM, and VIM Accreditation Scope and Professional Ethics

Sensitivity, Variability, and Precision

Key Takeaways Explain This Topic AI

Sensitivity is the change in indication divided by the corresponding change in the measured quantity.

Repeatability concerns repeated measurements under specified unchanged conditions.

Traditional gage R&R terminology uses between-appraiser reproducibility, while VIM distinguishes intermediate precision from reproducibility conditions.

Last updated: October 2026

Quantifying instrument operating characteristics is the core technical activity of calibration metrology. Every measuring device exhibits static and dynamic behaviors that dictate its fitness for purpose, including sensitivity, repeatability, reproducibility, bias, linearity, drift, hysteresis, and deadband. Calibration technicians must distinguish between random variability and systematic error, model instrument transfer functions, and recognize the physical mechanisms—such as mechanical backlash, viscoelastic creep, and thermal aging—that cause an instrument's indications to deviate from true reference values.

  • The Measurand and Influence Quantities
  • The Measurand (VIM 2.3)

In the International Vocabulary of Metrology (VIM), the measurand is defined as the "quantity intended to be measured" . While deceptively simple, an incomplete definition of the measurand introduces definitional uncertainty ( u def u_{\text{def}} u def ​ ), which acts as an irreducible uncertainty floor in the calibration budget.

A rigorous definition of a measurand requires stating:

The exact physical entity and state (e.g., "diameter of cylinder master SN-1048" ).

The spatial location and orientation of measurement (e.g., "at mid-height, along the orthogonal axis A-B" ).

The standard environmental reference state (e.g., "at 20.00 ∘ C 20.00^\circ\text{C} 20.0 0 ∘ C and 101.325 kPa 101.325\text{ kPa} 101.325 kPa barometric pressure" ).

The physical contact conditions (e.g., "under an applied measuring force of 0.50 N ± 0.05 N 0.50\text{ N} \pm 0.05\text{ N} 0.50 N ± 0.05 N using spherical carbide anvils of 5 mm 5\text{ mm} 5 mm radius" ).

Influence Quantities

An influence quantity is a quantity that, in a direct measurement, does not affect the quantity that is actually measured, but affects the relationship between the indication and the measurement result (VIM 2.52).

Environmental Influence Quantities: Ambient temperature ( T T T ), relative humidity (RH), barometric pressure ( P atm P_{\text{atm}} P atm ​ ), gravitational acceleration ( g g g ), and ambient magnetic or electrostatic fields.

Operational Influence Quantities: Mains line voltage stability, RF radiated electromagnetic interference (EMI), mechanical foundation vibrations, and operator body heat dissipation.

Technicians must either actively control influence quantities within tightly specified tolerances (e.g., 20 ∘ C ± 0.2 ∘ C 20^\circ\text{C} \pm 0.2^\circ\text{C} 2 0 ∘ C ± 0. 2 ∘ C in a dimensional laboratory) or continuously log them to apply deterministic mathematical corrections.

  • Variability and Random Dispersion
  • Nature of Measurement Variability

No measurement process yields identical results upon indefinite replication. Variability represents the random dispersion of measurement results obtained under repeated trials. It arises from an accumulation of microscopic, uncorrelated, and unpredictable physical variations (e.g., electronic thermal Johnson noise, air turbulence in optical paths, mechanical stick-slip friction in indicator pivots).

Statistical Metrics and Type A Evaluation

Under the GUM framework, variability is quantified through Type A evaluation of measurement uncertainty , calculated via statistical analysis of series of observations:

Sample Mean ( x ˉ \bar{x} x ˉ ): The best estimate of the expectation of the measurand: x ˉ = 1 n ∑ i = 1 n x i \bar{x} = \frac{1}{n} \sum_{i=1}^n x_i x ˉ = n 1 ​ i = 1 ∑ n ​ x i ​

Sample Standard Deviation ( s s s ): Quantifies the dispersion of the individual observations around the mean: s = 1 n − 1 ∑ i = 1 n ( x i − x ˉ ) 2 s = \sqrt{\frac{1}{n - 1} \sum_{i=1}^n (x_i - \bar{x})^2} s = n − 1 1 ​ i = 1 ∑ n ​ ( x i ​ − x ˉ ) 2 ​

Standard Error of the Mean ( s x ˉ s_{\bar{x}} s x ˉ ​ ): Represents the standard uncertainty of the calculated mean value: u A = s x ˉ = s n u_A = s_{\bar{x}} = \frac{s}{\sqrt{n}} u A ​ = s x ˉ ​ = n ​ s ​

For independent observations from a stable process with finite variance, the standard uncertainty of the mean is s / n s/\sqrt{n} s / n ​ . Statistical control or identical setup alone does not establish independence: check drift and autocorrelation. Pool compatible within-run estimates only when they represent the intended repeatability population; between-day shifts may need a separate component. s p = ∑ j = 1 k ( n j − 1 ) s j 2 ∑ j = 1 k ( n j − 1 ) s_p = \sqrt{\frac{\sum_{j=1}^k (n_j - 1) s_j^2}{\sum_{j=1}^k (n_j - 1)}} s p ​ = ∑ j = 1 k ​ ( n j ​ − 1 ) ∑ j = 1 k ​ ( n j ​ − 1 ) s j 2 ​ ​ ​

  • Sensitivity and Sensitivity Drift
  • Defining Sensitivity

Sensitivity ( S S S ) of a measuring system is the quotient of the change in an indication ( Δ y \Delta y Δ y ) and the corresponding change in the value of the quantity being measured ( Δ x \Delta x Δ x ): S = lim ⁡ Δ x → 0 Δ y Δ x = d y d x S = \lim_{\Delta x \to 0} \frac{\Delta y}{\Delta x} = \frac{dy}{dx} S = Δ x → 0 lim ​ Δ x Δ y ​ = d x d y ​

Sensitivity represents the derivative slope of the instrument's calibration curve (transfer function):

In a linear instrument ( y = m x + b y = m x + b y = m x + b ), sensitivity is constant across the entire range ( S = m S = m S = m ).

In non-linear sensors (such as thermocouples or thermistors), sensitivity varies dynamically with operating point. For example, a Type K thermocouple exhibits a sensitivity of ≈ 39 μ V / ∘ C \approx 39\ \mu\text{V}/^\circ\text{C} ≈ 39 μ V / ∘ C at 0 ∘ C 0^\circ\text{C} 0 ∘ C , rising to ≈ 42 μ V / ∘ C \approx 42\ \mu\text{V}/^\circ\text{C} ≈ 42 μ V / ∘ C at 500 ∘ C 500^\circ\text{C} 50 0 ∘ C .

Distinguishing Sensitivity from Resolution

A critical conceptual trap in metrology is confusing sensitivity with resolution :

Sensitivity is a transfer ratio: output change divided by input change (e.g., 10.0 mV/kPa 10.0\text{ mV/kPa} 10.0 mV/kPa or 20 mm/g 20\text{ mm/g} 20 mm/g ).

Resolution is the smallest input increment that produces a discernible output change (e.g., 0.01 kPa 0.01\text{ kPa} 0.01 kPa or 0.001 g 0.001\text{ g} 0.001 g ).

Example: An instrument can have exceptionally high sensitivity (e.g., a gain of 1 , 000 , 000 V/V 1,000,000\text{ V/V} 1 , 000 , 000 V/V ), but if internal amplifier noise is 50 mV 50\text{ mV} 50 mV , its effective measurement resolution will be terribly degraded. Conversely, a low-sensitivity transducer paired with an ultra-low-noise 24-bit ADC can resolve minute signal changes.

Sensitivity Drift (Gain Error)

Sensitivity drift (also termed scale factor drift or span error) occurs when the slope of the calibration curve changes over time or across temperature, while the zero intercept remains fixed: y ( x , T ) = S 0 [ 1 + T C S ⋅ ( T − 20 ∘ C ) ] ⋅ x + y 0 y(x, T) = S_0[1 + TC_S \cdot (T - 20^\circ\text{C})] \cdot x + y_0 y ( x , T ) = S 0 ​ [ 1 + T C S ​ ⋅ ( T − 2 0 ∘ C )] ⋅ x + y 0 ​

Here T C S TC_S T C S ​ is a fractional change per degree in this equation. Convert ppm/°C to reciprocal degrees by multiplying by 10 − 6 10^{-6} 1 0 − 6 , or percent/°C by dividing by 100, before substitution. Coefficients expressed relative to full scale require the model specified for that instrument. Temperature can affect the elastic element and strain-gage response.

Repeatability vs. Reproducibility

The distinction between repeatability and reproducibility is central to the ASQ CCT Body of Knowledge, Measurement Systems Analysis (MSA), and the GUM.

Repeatability (Measurement Repeatability, VIM 2.21)

Repeatability is measurement precision under a set of repeatability conditions of measurement :

The same measurement procedure.

The same operators.

The same measuring system.

The same operating conditions and physical location.

Replicate measurements on the same or similar objects over a short period of time.

Repeatability isolates the intrinsic short-term noise and mechanical stability of the equipment (often labeled Equipment Variation , E V EV E V ). It is evaluated by having one technician measure a single artifact 10 10 10 to 20 20 20 consecutive times without disturbing the physical setup.

Reproducibility and intermediate precision

VIM reproducibility conditions include different locations, operators, and measuring systems, with the changed conditions specified. Changes of operator or day within one laboratory can instead be intermediate precision conditions. Traditional gage R&R uses “reproducibility” for between-appraiser variation; identify that study convention rather than silently treating it as the full VIM definition.

In Gage R&R studies, the total measurement system variance ( σ gage 2 \sigma_{\text{gage}}^2 σ gage 2 ​ ) is the orthogonal sum of repeatability and reproducibility: σ gage 2 = σ repeatability 2 + σ reproducibility 2 \sigma_{\text{gage}}^2 = \sigma_{\text{repeatability}}^2 + \sigma_{\text{reproducibility}}^2 σ gage 2 ​ = σ repeatability 2 ​ + σ reproducibility 2 ​ Gage R&R = E V 2 + A V 2 \text{Gage R\&R} = \sqrt{EV^2 + AV^2} Gage R&R = E V 2 + A V 2 ​

Where E V EV E V is Equipment Variation (repeatability) and A V AV A V is Appraiser Variation (reproducibility across operators).