Study guide
Rounding, Algebra, Graphs, and Angular Conversions
ASQ Certified Calibration Technician CCT
Track your progress as you read
Finished sections get a check mark.
1 SI Definitions and Realizations
The Seven SI Base Units and the 2019 Revision Realizing SI Units in the Laboratory
2 Derived Units, Prefixes, and Constants
Derived Units and Physical Calculations SI Prefixes and Compound-Unit Conversions Physical Constants and Their Applications
3 Measurement Standards and Selection
Types and Roles of Measurement Standards Selecting Standards for an Operational Hierarchy
4 Traceability and Standard Substitution
Traceability Evidence and Impact Review Evaluating and Authorizing Standard Substitution
5 Dimensional and Mechanical Metrology
Linear Displacement and Dimensional Instruments Geometry, Thermal Expansion, and Mass Force, Torque, Pressure, Density, and Viscosity
6 Temperature, Humidity, Volume, and Flow
Temperature Sensors and Thermometer Corrections Humidity, Volume, and Flow
7 Electrical, Optical, and Time Measurements
Voltage, Current, Resistance, and AC Measurement Time, Frequency, Light, Power, and Energy
8 Measurement Methods
Direct, Indirect, and Ratio Methods Transfer, Differential, and UUT Substitution Methods
9 Instrument Characteristics and Measurement Data
Sensitivity, Variability, and Precision Bias, Linearity, Stability, and Hysteresis Data Formats, Readability, and Suitability Resolution, Confidentiality, and Data Integrity
10 Specifications and Error Sources
Interpreting IM&TE Specification Terms Specification Calculations and Operating Qualifiers Instrument, Operator, and Circuit Errors Environmental, Geometric, and Interference Errors
11 Measurement Assurance
Check Standards and Statistical Process Control Proficiency Comparisons and Gage R&R
12 Procedures and Governing Requirements
Procedure Structure and Published Practice Sources Method Development, Deviations, and Handling Regulations, Contracts, and Requirement Precedence
13 Instrument Adjustment and Standardization
Calibration, Zero, and Span Adjustment Linearization, Nulling, and Digital Adjustment
14 Calibration Environment
Temperature and Humidity Control ESD, Dust, Vibration, and Pressure
15 Operations, Roles, and Scheduling
Receiving, As-Found Testing, and Calibration Flow Asset Identification, Tracking, and Release Laboratory Roles and Initial Scheduling Interval Review, Overdue Control, and Workloads
16 Method and Software Validation
Method Verification and Validation Studies Software, Firmware, and Spreadsheet Validation
17 Records and Formal Reporting
Controlled Records and Calibration Certificates Status Labels and Out-of-Tolerance Reporting
18 Uncertainty Concepts and Decision Risk
Error, Uncertainty, and Measurand Definition TAR, TUR, Tolerance Consumption, and Capability Conformity Decisions and False-Accept Risk Guard Bands and Conformity Reporting
19 Uncertainty Models and Budgets
Measurement Models and Uncertainty Contributions Distribution Selection and Budget Assembly
20 Type A and Type B Evaluation
Type A Statistics and Repeatability Outlier Review and Worked Type A Analysis Type B Distribution Models
21 Uncertainty Reporting and Applied Mathematics
Propagation, Covariance, and Effective Degrees of Freedom Coverage Factors and Expanded-Uncertainty Reporting Notation, Unit Conversions, Ratios, and Decibels Rounding, Algebra, Graphs, and Angular Conversions
22 Quality Systems and Tools
Quality-System Components, Planning, and Improvement The Seven Basic Quality Tools
23 Audits and Corrective Action
Audit Types, Planning, and Nonconformance Control Impact Assessment, Root Cause, and Effectiveness
24 Safety, Standards, and Professional Conduct
Electrical, Pressure, Chemical, and Laboratory Hazards PPE, Safety Data Sheets, 6S, and Safe Restoration Quality Standards, GUM, and VIM Accreditation Scope and Professional Ethics
Rounding, Algebra, Graphs, and Angular Conversions
Key Takeaways Explain This Topic AI
Apply the required rounding convention after calculations with adequate intermediate precision.
Interpolation estimates between known points, while extrapolation extends the model beyond them and requires justification.
For sufficiently small angles in radians, sine-bar support-height difference is approximately length times angle.
Last updated: October 2026
- Significant Figures and ASTM E29 Rounding Rules
- Rules for Significant Figures
Non-zero digits are always significant ( 14.8 14.8 14.8 has 3 sig figs).
Any zeros between non-zero digits are significant ( 100.05 100.05 100.05 has 5 sig figs).
Leading zeros are never significant; they serve only as decimal place-holders ( 0.0042 0.0042 0.0042 has 2 sig figs).
Trailing zeros to the right of the decimal point are significant ( 12.000 12.000 12.000 has 5 sig figs).
Arithmetic Operation Rules
Multiplication/division classroom convention: Match the least precise factor’s significant figures when no uncertainty information is provided. In calibration reporting, use the actual evaluated uncertainty and prescribed reporting rules rather than replacing them with this shortcut. A = ( 12.45 cm ) × ( 2.3 cm ) = 28.635 cm 2 ⟹ 29 cm 2 (2 sig figs) A = (12.45\text{ cm}) \times (2.3\text{ cm}) = 28.635\text{ cm}^2 \implies \mathbf{29\text{ cm}^2} \quad \text{(2 sig figs)} A = ( 12.45 cm ) × ( 2.3 cm ) = 28.635 cm 2 ⟹ 29 cm 2 (2 sig figs)
Addition / Subtraction: The final result is limited by the least precise decimal position (fewest decimal places), regardless of the total number of significant figures. m = 102.450 g + 0.12 g + 1.2345 g = 103.8045 g ⟹ 103.80 g (2 decimal places) m = 102.450\text{ g} + 0.12\text{ g} + 1.2345\text{ g} = 103.8045\text{ g} \implies \mathbf{103.80\text{ g}} \quad \text{(2 decimal places)} m = 102.450 g + 0.12 g + 1.2345 g = 103.8045 g ⟹ 103.80 g (2 decimal places)
Banker's Rounding: The ASTM E29 "Round-to-Even" Standard
Rounding ties upward can create an upward tendency for some data distributions. Round-to-even is one way to avoid always favoring the upper value at exact ties; it does not prove zero bias for every dataset. Use the convention required by the method and validate its implementation.
ASTM E29 describes a round-to-even convention for exact ties. IEEE 754 supports several rounding modes, with ties-to-even commonly the default; it does not require every application to use that mode. Follow the convention specified for the result being reported:
If the digit following the last retained digit is strictly less than 5, round down ( 14.24 → 14.2 14.24 \to 14.2 14.24 → 14.2 ).
If the digit following the last retained digit is greater than 5, or is 5 followed by any nonzero digits, round up ( 14.26 → 14.3 14.26 \to 14.3 14.26 → 14.3 ).
If the digit following the last retained digit is exactly 5 (or 5 followed only by zeros):
If the last retained digit is EVEN , leave it unchanged (round down): 12.450 ⟹ 12.4 12.450 \implies \mathbf{12.4} 12.450 ⟹ 12.4
If the last retained digit is ODD , round it up to make it even: 12.550 ⟹ 12.6 12.550 \implies \mathbf{12.6} 12.550 ⟹ 12.6
Notice that both 12.450 12.450 12.450 and 12.550 12.550 12.550 round to an even final digit ( 12.4 12.4 12.4 and 12.6 12.6 12.6 ). Over millions of measurements, approximately half round up and half round down, reducing systematic tie-rounding bias when ties are reasonably balanced.
Truncation vs. Rounding
Truncation: Drop digits beyond the specified position without rounding. For a positive 14.999 truncated to one decimal, the result is 14.9. Truncation toward zero moves negative values in the opposite numerical direction; it differs from floor and can introduce bias. Use it only when the task calls for it, such as conservative degrees-of-freedom lookup.
Intermediate precision and order: Follow parentheses, powers, multiplication/division, then addition/subtraction. Retain adequate intermediate precision for the error budget and decision; two extra digits are a rule of thumb, not a universal guarantee. Round the final reported result by the required convention.
Algebraic Formula Manipulation
Technicians must be proficient in isolating unknown calibration variables:
Solving for voltage from power and resistance: P = V 2 R ⟹ V 2 = P ⋅ R ⟹ V = P ⋅ R P = \frac{V^2}{R} \implies V^2 = P \cdot R \implies V = \sqrt{P \cdot R} P = R V 2 ⟹ V 2 = P ⋅ R ⟹ V = P ⋅ R
Solving for temperature change from RTD resistance: R T = R 0 ( 1 + α Δ T ) ⟹ R T R 0 = 1 + α Δ T ⟹ Δ T = R T − R 0 α R 0 R_T = R_0 (1 + \alpha \Delta T) \implies \frac{R_T}{R_0} = 1 + \alpha \Delta T \implies \Delta T = \frac{R_T - R_0}{\alpha R_0} R T = R 0 ( 1 + α Δ T ) ⟹ R 0 R T = 1 + α Δ T ⟹ Δ T = α R 0 R T − R 0
Graphical Analysis: Slope, Intercept, and Linear Interpolation
Linearity calibration curves relate instrument indication ( Y Y Y ) to assigned reference input value ( X X X ): y = m x + b y = m x + b y = m x + b
Slope ( m m m ): Represents sensitivity or scale factor: m = y 2 − y 1 x 2 − x 1 = Δ y Δ x m = \frac{y_2 - y_1}{x_2 - x_1} = \frac{\Delta y}{\Delta x} m = x 2 − x 1 y 2 − y 1 = Δ x Δ y
Y-intercept ( b b b ): Represents the zero offset (indication when input is zero): b = y 1 − m x 1 b = y_1 - m x_1 b = y 1 − m x 1
Linear Interpolation
When a sensor calibration certificate provides discrete calibration points, intermediate values are determined by linear interpolation: y = y 1 + ( x − x 1 x 2 − x 1 ) ( y 2 − y 1 ) y = y_1 + \left( \frac{x - x_1}{x_2 - x_1} \right) (y_2 - y_1) y = y 1 + ( x 2 − x 1 x − x 1 ) ( y 2 − y 1 )
Worked Example
A thermocouple calibration table records output voltages for Type K thermocouples:
At T 1 = 100.0 ∘ C T_1 = 100.0^\circ\text{C} T 1 = 100. 0 ∘ C , V 1 = 4.096 mV V_1 = 4.096\text{ mV} V 1 = 4.096 mV
At T 2 = 110.0 ∘ C T_2 = 110.0^\circ\text{C} T 2 = 110. 0 ∘ C , V 2 = 4.509 mV V_2 = 4.509\text{ mV} V 2 = 4.509 mV
If the technician measures an output voltage of V = 4.300 mV V = 4.300\text{ mV} V = 4.300 mV , find the interpolated temperature T T T : T = T 1 + ( V − V 1 V 2 − V 1 ) ( T 2 − T 1 ) = 100.0 ∘ C + ( 4.300 − 4.096 4.509 − 4.096 ) ( 110.0 − 100.0 ) T = T_1 + \left( \frac{V - V_1}{V_2 - V_1} \right) (T_2 - T_1) = 100.0^\circ\text{C} + \left( \frac{4.300 - 4.096}{4.509 - 4.096} \right) (110.0 - 100.0) T = T 1 + ( V 2 − V 1 V − V 1 ) ( T 2 − T 1 ) = 100. 0 ∘ C + ( 4.509 − 4.096 4.300 − 4.096 ) ( 110.0 − 100.0 ) T = 100.0 + ( 0.204 0.413 ) ( 10.0 ) = 100.0 + ( 0.49395 × 10.0 ) = 100.0 + 4.94 = 104.94 ∘ C T = 100.0 + \left( \frac{0.204}{0.413} \right) (10.0) = 100.0 + (0.49395 \times 10.0) = 100.0 + 4.94 = \mathbf{104.94^\circ\text{C}} T = 100.0 + ( 0.413 0.204 ) ( 10.0 ) = 100.0 + ( 0.49395 × 10.0 ) = 100.0 + 4.94 = 104.9 4 ∘ C
Caution
Hazard of Extrapolation: Linear extrapolation beyond calibrated end points ( x < x min x < x_{\min} x < x m i n or x > x max x > x_{\max} x > x m a x ) is dangerous in calibration because sensor transducers (load cells, pressure diaphragms, thermistors) suffer severe non-linear distortion, saturation, or hysteresis outside their validated operating range.
Angular Conversions and the Radian Small-Angle Approximation
Angular measurement is central to optical alignment, autocollimators, theodolites, and sine bar height calculations.
Angular Systems and Conversion Factors
Degrees, Minutes, Seconds (DMS): 1 ∘ = 60 ′ = 3 , 600 ′ ′ 1^\circ = 60' = 3,600'' 1 ∘ = 6 0 ′ = 3 , 60 0 ′′ 1 ′ = ( 1 60 ) ∘ ≈ 0.016667 ∘ 1' = \left(\frac{1}{60}\right)^\circ \approx 0.016667^\circ 1 ′ = ( 60 1 ) ∘ ≈ 0.01666 7 ∘ 1 ′ ′ = ( 1 3 , 600 ) ∘ ≈ 0.0002778 ∘ 1'' = \left(\frac{1}{3,600}\right)^\circ \approx 0.0002778^\circ 1 ′′ = ( 3 , 600 1 ) ∘ ≈ 0.000277 8 ∘
Gradians (Gons): A full circle equals 400 grads 400\text{ grads} 400 grads ( 100 grads = 90 ∘ 100\text{ grads} = 90^\circ 100 grads = 9 0 ∘ ). 1 grad = 0.9 ∘ = 54 ′ = 3 , 240 ′ ′ 1\text{ grad} = 0.9^\circ = 54' = 3,240'' 1 grad = 0. 9 ∘ = 5 4 ′ = 3 , 24 0 ′′
Radians (rad): The SI coherent derived unit of plane angle. 2 π rad = 360 ∘ ⟹ 1 rad = 180 ∘ π ≈ 57.2957795 ∘ ≈ 206 , 264.8 ′ ′ 2\pi\text{ rad} = 360^\circ \implies 1\text{ rad} = \frac{180^\circ}{\pi} \approx 57.2957795^\circ \approx 206,264.8'' 2 π rad = 36 0 ∘ ⟹ 1 rad = π 18 0 ∘ ≈ 57.295779 5 ∘ ≈ 206 , 264. 8 ′′ 1 arcsecond (1”) = 1 206 , 264.8 rad ≈ 4.848137 × 10 − 6 rad 1\text{ arcsecond (1'')} = \frac{1}{206,264.8}\text{ rad} \approx 4.848137 \times 10^{-6}\text{ rad} 1 arcsecond (1”) = 206 , 264.8 1 rad ≈ 4.848137 × 1 0 − 6 rad
The Radian Small-Angle Approximation in Metrology
In dimensional metrology, the Taylor series expansion for trigonometric functions around θ = 0 \theta = 0 θ = 0 is: sin θ = θ − θ 3 3 ! + θ 5 5 ! − … \sin\theta = \theta - \frac{\theta^3}{3!} + \frac{\theta^5}{5!} - \dots sin θ = θ − 3 ! θ 3 + 5 ! θ 5 − … tan θ = θ + θ 3 3 + 2 θ 5 15 + … \tan\theta = \theta + \frac{\theta^3}{3} + \frac{2\theta^5}{15} + \dots tan θ = θ + 3 θ 3 + 15 2 θ 5 + …
For small angles ( θ < 1 ∘ ≈ 0.0175 rad \theta < 1^\circ \approx 0.0175\text{ rad} θ < 1 ∘ ≈ 0.0175 rad ), the higher-order terms may be small, but acceptability depends on the error budget (the relative tangent approximation error at 1 degree is about 0.0102%). Therefore, when θ \theta θ is expressed in radians : sin θ ≈ tan θ ≈ θ ( in radians ) \sin\theta \approx \tan\theta \approx \theta \quad (\text{in radians}) sin θ ≈ tan θ ≈ θ ( in radians )
Calibration Application: Sine Bar Elevation Calculation
A sine bar of length L L L resting on a surface plate is elevated at one end by a stack of gage blocks of height H H H to set an angle θ \theta θ : sin θ = H L ⟹ H = L ⋅ sin θ \sin\theta = \frac{H}{L} \implies H = L \cdot \sin\theta sin θ = L H ⟹ H = L ⋅ sin θ
Using the small-angle approximation for small angular adjustments: H ≈ L ⋅ θ ( with θ in radians ) H \approx L \cdot \theta \quad (\text{with } \theta \text{ in radians}) H ≈ L ⋅ θ ( with θ in radians )
Worked Example
A technician sets up a 5.0000 inch 5.0000\text{ inch} 5.0000 inch sine bar to generate an angle of nominally θ = 10.00 arcseconds \theta = 10.00\text{ arcseconds} θ = 10.00 arcseconds ( 10 ′ ′ 10'' 1 0 ′′ ). What end-height difference H H H is required? In a practical setup use suitable equal-height supports and a fine height difference; this tiny difference is not a conventional standalone gage-block stack.
Convert angle from arcseconds to radians: θ = 10.00 ′ ′ × ( 4.848137 × 10 − 6 rad/arcsec ) = 4.848137 × 10 − 5 rad \theta = 10.00'' \times \left(4.848137 \times 10^{-6}\text{ rad/arcsec}\right) = 4.848137 \times 10^{-5}\text{ rad} θ = 10.0 0 ′′ × ( 4.848137 × 1 0 − 6 rad/arcsec ) = 4.848137 × 1 0 − 5 rad
Calculate the required support-height difference H H H : H = L ⋅ θ = ( 5.0000 in ) × ( 4.848137 × 10 − 5 rad ) = 2.42407 × 10 − 4 in = 0.0002424 in = 242.4 μ in H = L \cdot \theta = (5.0000\text{ in}) \times (4.848137 \times 10^{-5}\text{ rad}) = 2.42407 \times 10^{-4}\text{ in} = 0.0002424\text{ in} = 242.4\ \mu\text{in} H = L ⋅ θ = ( 5.0000 in ) × ( 4.848137 × 1 0 − 5 rad ) = 2.42407 × 1 0 − 4 in = 0.0002424 in = 242.4 μ in
If the technician attempted to use a standard calculator without radians, rounding errors would easily destroy the sub-microinch precision required for precision angle standards!
Reading slope, intercept, and extrapolated values
Suppose a hypothetical linear sensor gives 1 V at an input of zero units and 5 V at an input of 100 units. Its slope is 0.04 V per input unit and its intercept is 1 V. An observed 3 V corresponds to 50 units by interpolation. A 6 V observation would correspond to 125 units only if the same linear model remains valid beyond the measured interval. That last calculation is extrapolation, not evidence that the sensor is calibrated there.
Check the axes, units, scale, and whether the plot shows indication or error. A slope in an error plot represents change in error per input unit; it is not necessarily the sensor’s overall sensitivity. A nonzero error-plot intercept indicates an offset at the plotted origin. Residuals around a fitted line help reveal nonlinearity, but a high correlation or R-squared alone does not establish negligible residuals or fitness for the required tolerance.