Study guide
Sensitivity, Variability, and Precision
ASQ Certified Calibration Technician CCT
Track your progress as you read
Finished sections get a check mark.
1 SI Definitions and Realizations
The Seven SI Base Units and the 2019 Revision Realizing SI Units in the Laboratory
2 Derived Units, Prefixes, and Constants
Derived Units and Physical Calculations SI Prefixes and Compound-Unit Conversions Physical Constants and Their Applications
3 Measurement Standards and Selection
Types and Roles of Measurement Standards Selecting Standards for an Operational Hierarchy
4 Traceability and Standard Substitution
Traceability Evidence and Impact Review Evaluating and Authorizing Standard Substitution
5 Dimensional and Mechanical Metrology
Linear Displacement and Dimensional Instruments Geometry, Thermal Expansion, and Mass Force, Torque, Pressure, Density, and Viscosity
6 Temperature, Humidity, Volume, and Flow
Temperature Sensors and Thermometer Corrections Humidity, Volume, and Flow
7 Electrical, Optical, and Time Measurements
Voltage, Current, Resistance, and AC Measurement Time, Frequency, Light, Power, and Energy
8 Measurement Methods
Direct, Indirect, and Ratio Methods Transfer, Differential, and UUT Substitution Methods
9 Instrument Characteristics and Measurement Data
Sensitivity, Variability, and Precision Bias, Linearity, Stability, and Hysteresis Data Formats, Readability, and Suitability Resolution, Confidentiality, and Data Integrity
10 Specifications and Error Sources
Interpreting IM&TE Specification Terms Specification Calculations and Operating Qualifiers Instrument, Operator, and Circuit Errors Environmental, Geometric, and Interference Errors
11 Measurement Assurance
Check Standards and Statistical Process Control Proficiency Comparisons and Gage R&R
12 Procedures and Governing Requirements
Procedure Structure and Published Practice Sources Method Development, Deviations, and Handling Regulations, Contracts, and Requirement Precedence
13 Instrument Adjustment and Standardization
Calibration, Zero, and Span Adjustment Linearization, Nulling, and Digital Adjustment
14 Calibration Environment
Temperature and Humidity Control ESD, Dust, Vibration, and Pressure
15 Operations, Roles, and Scheduling
Receiving, As-Found Testing, and Calibration Flow Asset Identification, Tracking, and Release Laboratory Roles and Initial Scheduling Interval Review, Overdue Control, and Workloads
16 Method and Software Validation
Method Verification and Validation Studies Software, Firmware, and Spreadsheet Validation
17 Records and Formal Reporting
Controlled Records and Calibration Certificates Status Labels and Out-of-Tolerance Reporting
18 Uncertainty Concepts and Decision Risk
Error, Uncertainty, and Measurand Definition TAR, TUR, Tolerance Consumption, and Capability Conformity Decisions and False-Accept Risk Guard Bands and Conformity Reporting
19 Uncertainty Models and Budgets
Measurement Models and Uncertainty Contributions Distribution Selection and Budget Assembly
20 Type A and Type B Evaluation
Type A Statistics and Repeatability Outlier Review and Worked Type A Analysis Type B Distribution Models
21 Uncertainty Reporting and Applied Mathematics
Propagation, Covariance, and Effective Degrees of Freedom Coverage Factors and Expanded-Uncertainty Reporting Notation, Unit Conversions, Ratios, and Decibels Rounding, Algebra, Graphs, and Angular Conversions
22 Quality Systems and Tools
Quality-System Components, Planning, and Improvement The Seven Basic Quality Tools
23 Audits and Corrective Action
Audit Types, Planning, and Nonconformance Control Impact Assessment, Root Cause, and Effectiveness
24 Safety, Standards, and Professional Conduct
Electrical, Pressure, Chemical, and Laboratory Hazards PPE, Safety Data Sheets, 6S, and Safe Restoration Quality Standards, GUM, and VIM Accreditation Scope and Professional Ethics
Sensitivity, Variability, and Precision
Key Takeaways Explain This Topic AI
Sensitivity is the change in indication divided by the corresponding change in the measured quantity.
Repeatability concerns repeated measurements under specified unchanged conditions.
Traditional gage R&R terminology uses between-appraiser reproducibility, while VIM distinguishes intermediate precision from reproducibility conditions.
Last updated: October 2026
Quantifying instrument operating characteristics is the core technical activity of calibration metrology. Every measuring device exhibits static and dynamic behaviors that dictate its fitness for purpose, including sensitivity, repeatability, reproducibility, bias, linearity, drift, hysteresis, and deadband. Calibration technicians must distinguish between random variability and systematic error, model instrument transfer functions, and recognize the physical mechanisms—such as mechanical backlash, viscoelastic creep, and thermal aging—that cause an instrument's indications to deviate from true reference values.
- The Measurand and Influence Quantities
- The Measurand (VIM 2.3)
In the International Vocabulary of Metrology (VIM), the measurand is defined as the "quantity intended to be measured" . While deceptively simple, an incomplete definition of the measurand introduces definitional uncertainty ( u def u_{\text{def}} u def ), which acts as an irreducible uncertainty floor in the calibration budget.
A rigorous definition of a measurand requires stating:
The exact physical entity and state (e.g., "diameter of cylinder master SN-1048" ).
The spatial location and orientation of measurement (e.g., "at mid-height, along the orthogonal axis A-B" ).
The standard environmental reference state (e.g., "at 20.00 ∘ C 20.00^\circ\text{C} 20.0 0 ∘ C and 101.325 kPa 101.325\text{ kPa} 101.325 kPa barometric pressure" ).
The physical contact conditions (e.g., "under an applied measuring force of 0.50 N ± 0.05 N 0.50\text{ N} \pm 0.05\text{ N} 0.50 N ± 0.05 N using spherical carbide anvils of 5 mm 5\text{ mm} 5 mm radius" ).
Influence Quantities
An influence quantity is a quantity that, in a direct measurement, does not affect the quantity that is actually measured, but affects the relationship between the indication and the measurement result (VIM 2.52).
Environmental Influence Quantities: Ambient temperature ( T T T ), relative humidity (RH), barometric pressure ( P atm P_{\text{atm}} P atm ), gravitational acceleration ( g g g ), and ambient magnetic or electrostatic fields.
Operational Influence Quantities: Mains line voltage stability, RF radiated electromagnetic interference (EMI), mechanical foundation vibrations, and operator body heat dissipation.
Technicians must either actively control influence quantities within tightly specified tolerances (e.g., 20 ∘ C ± 0.2 ∘ C 20^\circ\text{C} \pm 0.2^\circ\text{C} 2 0 ∘ C ± 0. 2 ∘ C in a dimensional laboratory) or continuously log them to apply deterministic mathematical corrections.
- Variability and Random Dispersion
- Nature of Measurement Variability
No measurement process yields identical results upon indefinite replication. Variability represents the random dispersion of measurement results obtained under repeated trials. It arises from an accumulation of microscopic, uncorrelated, and unpredictable physical variations (e.g., electronic thermal Johnson noise, air turbulence in optical paths, mechanical stick-slip friction in indicator pivots).
Statistical Metrics and Type A Evaluation
Under the GUM framework, variability is quantified through Type A evaluation of measurement uncertainty , calculated via statistical analysis of series of observations:
Sample Mean ( x ˉ \bar{x} x ˉ ): The best estimate of the expectation of the measurand: x ˉ = 1 n ∑ i = 1 n x i \bar{x} = \frac{1}{n} \sum_{i=1}^n x_i x ˉ = n 1 i = 1 ∑ n x i
Sample Standard Deviation ( s s s ): Quantifies the dispersion of the individual observations around the mean: s = 1 n − 1 ∑ i = 1 n ( x i − x ˉ ) 2 s = \sqrt{\frac{1}{n - 1} \sum_{i=1}^n (x_i - \bar{x})^2} s = n − 1 1 i = 1 ∑ n ( x i − x ˉ ) 2
Standard Error of the Mean ( s x ˉ s_{\bar{x}} s x ˉ ): Represents the standard uncertainty of the calculated mean value: u A = s x ˉ = s n u_A = s_{\bar{x}} = \frac{s}{\sqrt{n}} u A = s x ˉ = n s
For independent observations from a stable process with finite variance, the standard uncertainty of the mean is s / n s/\sqrt{n} s / n . Statistical control or identical setup alone does not establish independence: check drift and autocorrelation. Pool compatible within-run estimates only when they represent the intended repeatability population; between-day shifts may need a separate component. s p = ∑ j = 1 k ( n j − 1 ) s j 2 ∑ j = 1 k ( n j − 1 ) s_p = \sqrt{\frac{\sum_{j=1}^k (n_j - 1) s_j^2}{\sum_{j=1}^k (n_j - 1)}} s p = ∑ j = 1 k ( n j − 1 ) ∑ j = 1 k ( n j − 1 ) s j 2
- Sensitivity and Sensitivity Drift
- Defining Sensitivity
Sensitivity ( S S S ) of a measuring system is the quotient of the change in an indication ( Δ y \Delta y Δ y ) and the corresponding change in the value of the quantity being measured ( Δ x \Delta x Δ x ): S = lim Δ x → 0 Δ y Δ x = d y d x S = \lim_{\Delta x \to 0} \frac{\Delta y}{\Delta x} = \frac{dy}{dx} S = Δ x → 0 lim Δ x Δ y = d x d y
Sensitivity represents the derivative slope of the instrument's calibration curve (transfer function):
In a linear instrument ( y = m x + b y = m x + b y = m x + b ), sensitivity is constant across the entire range ( S = m S = m S = m ).
In non-linear sensors (such as thermocouples or thermistors), sensitivity varies dynamically with operating point. For example, a Type K thermocouple exhibits a sensitivity of ≈ 39 μ V / ∘ C \approx 39\ \mu\text{V}/^\circ\text{C} ≈ 39 μ V / ∘ C at 0 ∘ C 0^\circ\text{C} 0 ∘ C , rising to ≈ 42 μ V / ∘ C \approx 42\ \mu\text{V}/^\circ\text{C} ≈ 42 μ V / ∘ C at 500 ∘ C 500^\circ\text{C} 50 0 ∘ C .
Distinguishing Sensitivity from Resolution
A critical conceptual trap in metrology is confusing sensitivity with resolution :
Sensitivity is a transfer ratio: output change divided by input change (e.g., 10.0 mV/kPa 10.0\text{ mV/kPa} 10.0 mV/kPa or 20 mm/g 20\text{ mm/g} 20 mm/g ).
Resolution is the smallest input increment that produces a discernible output change (e.g., 0.01 kPa 0.01\text{ kPa} 0.01 kPa or 0.001 g 0.001\text{ g} 0.001 g ).
Example: An instrument can have exceptionally high sensitivity (e.g., a gain of 1 , 000 , 000 V/V 1,000,000\text{ V/V} 1 , 000 , 000 V/V ), but if internal amplifier noise is 50 mV 50\text{ mV} 50 mV , its effective measurement resolution will be terribly degraded. Conversely, a low-sensitivity transducer paired with an ultra-low-noise 24-bit ADC can resolve minute signal changes.
Sensitivity Drift (Gain Error)
Sensitivity drift (also termed scale factor drift or span error) occurs when the slope of the calibration curve changes over time or across temperature, while the zero intercept remains fixed: y ( x , T ) = S 0 [ 1 + T C S ⋅ ( T − 20 ∘ C ) ] ⋅ x + y 0 y(x, T) = S_0[1 + TC_S \cdot (T - 20^\circ\text{C})] \cdot x + y_0 y ( x , T ) = S 0 [ 1 + T C S ⋅ ( T − 2 0 ∘ C )] ⋅ x + y 0
Here T C S TC_S T C S is a fractional change per degree in this equation. Convert ppm/°C to reciprocal degrees by multiplying by 10 − 6 10^{-6} 1 0 − 6 , or percent/°C by dividing by 100, before substitution. Coefficients expressed relative to full scale require the model specified for that instrument. Temperature can affect the elastic element and strain-gage response.
Repeatability vs. Reproducibility
The distinction between repeatability and reproducibility is central to the ASQ CCT Body of Knowledge, Measurement Systems Analysis (MSA), and the GUM.
Repeatability (Measurement Repeatability, VIM 2.21)
Repeatability is measurement precision under a set of repeatability conditions of measurement :
The same measurement procedure.
The same operators.
The same measuring system.
The same operating conditions and physical location.
Replicate measurements on the same or similar objects over a short period of time.
Repeatability isolates the intrinsic short-term noise and mechanical stability of the equipment (often labeled Equipment Variation , E V EV E V ). It is evaluated by having one technician measure a single artifact 10 10 10 to 20 20 20 consecutive times without disturbing the physical setup.
Reproducibility and intermediate precision
VIM reproducibility conditions include different locations, operators, and measuring systems, with the changed conditions specified. Changes of operator or day within one laboratory can instead be intermediate precision conditions. Traditional gage R&R uses “reproducibility” for between-appraiser variation; identify that study convention rather than silently treating it as the full VIM definition.
In Gage R&R studies, the total measurement system variance ( σ gage 2 \sigma_{\text{gage}}^2 σ gage 2 ) is the orthogonal sum of repeatability and reproducibility: σ gage 2 = σ repeatability 2 + σ reproducibility 2 \sigma_{\text{gage}}^2 = \sigma_{\text{repeatability}}^2 + \sigma_{\text{reproducibility}}^2 σ gage 2 = σ repeatability 2 + σ reproducibility 2 Gage R&R = E V 2 + A V 2 \text{Gage R\&R} = \sqrt{EV^2 + AV^2} Gage R&R = E V 2 + A V 2
Where E V EV E V is Equipment Variation (repeatability) and A V AV A V is Appraiser Variation (reproducibility across operators).