Study guide
1.1 Basic Shop Math & Precision Measurement
ASQ Certified Quality Inspector CQI
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Shop Mathematics & Algebra
1.1 Basic Shop Math & Precision Measurement 1.2 Basic Algebra & Technical Formulas 1.3 Measurement Systems & Unit Conversions
2 Chapter 2: Geometry & Applied Shop Trigonometry
2.1 Applied Shop Geometry 2.2 Right-Triangle Trigonometry 2.3 Shop Trigonometry Applications & Sine Bars
3 Chapter 3: Handheld Variable & Attribute Gages
3.1 Variable Hand Measuring Instruments 3.2 Fixed Attribute & Limit Gages 3.3 Transfer Gages & Precision Accessories 3.4 Special Gages: Electronic, Pneumatic, Force & Environmental Instrumentation
4 Chapter 4: Surface Plate Inspection, Optical Systems & CMMs
4.1 Gage Selection, Care & Environmental Control 4.2 Surface Plate Tools & Layout Inspection 4.3 Optical Comparators & Vision Systems 4.4 Coordinate Measuring Machines (CMM) & 3-2-1 Alignment 4.5 Gage Correlation & Cross-Instrument Agreement
5 Chapter 5: Surface Metrology, Calibration Systems & MSA
5.1 Surface Texture & Finish Measurement 5.2 Calibration Systems & NIST Traceability 5.3 Measurement System Analysis (MSA) & Gage R&R 5.4 Measuring Mass, Form & Profile, and Inclination
6 Chapter 6: Technical Drawing & Blueprint Reading
6.1 Engineering Drawing Structure & Conventions 6.2 Views, Sectioning & Feature Representation 6.3 Dimensions, Tolerancing & Fit Classifications
7 Chapter 7: Geometric Dimensioning & Tolerancing (GD&T)
7.1 GD&T Fundamentals & Datum Reference Frames 7.2 Material Condition Modifiers & Bonus Tolerance 7.3 Form & Orientation Tolerances 7.4 Location, Profile & Runout Tolerances
8 Chapter 8: Sampling Inspection & Acceptance Sampling Plans
8.1 Sampling Fundamentals & Operating Characteristic (OC) Curves 8.2 Attribute Sampling with ANSI/ASQ Z1.4 8.3 Variables Sampling with ANSI/ASQ Z1.9
9 Chapter 9: Inspection Planning, Processes & Material Disposition
9.1 Inspection Planning, Receiving & In-Process Inspection 9.2 First Article Inspection (FAI) & Documentation 9.3 Nonconforming Material Control & MRB 9.4 Inspection Errors & Human Factors 9.5 Product Traceability, Age Control & Shelf Life
10 Chapter 10: Materials Testing & Nondestructive Examination
10.1 Mechanical & Destructive Testing 10.2 Surface & Subsurface NDE Methods 10.3 Volumetric & Electromagnetic NDE Methods 10.4 Functionality, Force & Elastomer Hardness Testing 10.5 Software for Test Equipment & Software Validation
11 Chapter 11: Statistical Process Control & Process Capability
11.1 Statistics Fundamentals & Normal Distribution 11.2 Control Charts for Variables 11.3 Control Charts for Attributes & Run Rules 11.4 Process Capability & Performance Analysis
12 Chapter 12: Quality Systems, Continuous Improvement & Safety
12.1 The 7 Classic Quality Control Tools 12.2 Continuous Improvement & Problem-Solving Methodologies 12.3 Quality Management Systems, Audits & CAPA 12.4 Workplace Safety, Environmental Controls & PPE 12.5 Failure Mode and Effects Analysis (FMEA) & Fault Tree Analysis (FTA)
13 Chapter 13: Quality Fundamentals, Teams & Documentation Systems
13.1 Quality Improvement Concepts, Products & Processes 13.2 Audit Tools, Tracing Techniques & Audit Communication 13.3 Employees as Resources: Empowerment, Team Roles & Team Development 13.4 Quality Documentation, Reference Standards & Document Control
1.1 Basic Shop Math & Precision Measurement
Key Takeaways Explain This Topic AI
Standard shop binary fractions (e.g., 1/64 through 63/64) convert to terminating decimals; 1/64 inch equals exactly 0.015625 inches.
ASTM E29 and metrology standards mandate 'round-to-even' (banker's rounding) when the discarded digit is exactly 5 to eliminate positive cumulative bias across inspection datasets.
In significant figure arithmetic, addition and subtraction are governed by the fewest decimal places, whereas multiplication and division are governed by the fewest significant digits.
Intermediate inspection calculations must retain at least one or two guard digits to prevent premature rounding error from failing an in-tolerance part.
Percentage error compares measurement deviation against an accepted reference standard: Percent Error = (|Measured - Standard| / Standard) * 100%.
Last updated: September 2026
- 1.1 Basic Shop Math & Precision Measurement
- Introduction to Precision Shop Mathematics
In precision manufacturing, quality inspectors serve as the critical technical gatekeepers between engineering intent and delivered product. Modern inspection requires seamless conversion between blueprint fractions and electronic gage decimals, rigorous calculation of tolerance stackups, and defensible data rounding. Inaccurate mathematical operations or arbitrary rounding can lead to the acceptance of nonconforming product (consumer's risk, or Type II error / beta) or the unwarranted rejection of conforming product (producer's risk, or Type I error / alpha). Mastery of basic shop math is the baseline competency tested in Domain I of the ASQ Certified Quality Inspector (CQI) examination.
Fraction and Decimal Operations in Manufacturing
Mechanical drawings—particularly legacy prints and aerospace specifications under ASME Y14.5 or ANSI standards—frequently specify stock sizes, clearance holes, tap drills, and raw material thicknesses using fractional inches. However, precision measuring instruments (digital calipers, micrometers, height stands, and coordinate measuring machines) output dimensions in decimal inches (typically to three or four decimal places) or metric millimeters.
Operating with Common Shop Fractions
Working with shop fractions requires standard arithmetic operations:
Finding Common Denominators for Addition and Subtraction : When determining overall part dimensions from stepped features, fractions must share a common denominator. For standard fractional rule divisions (1/64, 1/32, 1/16, 1/8, 1/4, 1/2), the common denominator is almost always a power of 2 (most commonly 64).
Worked Example : A machinist mills three successive steps: 5/16 in, 27/64 in, and 3/8 in.
Convert to 64ths: 5/16 = 20/64, 27/64 = 27/64, 3/8 = 24/64.
Total length: (20 + 27 + 24) / 64 = 71/64 = 1 7/64 in (1.109375 in).
Fraction Multiplication and Division in Raw Stock Calculation :
When calculating raw stock usage or cut-off allowances with kerf loss: Total Length = (Part Length + Cutoff Width) * Quantity.
For division (e.g., dividing a standard 72-inch bar into pins measuring 2 3/8 inches including parting blade width): 72 / (19/8) = 72 * (8/19) = 576 / 19 = 30.31 pins (yielding exactly 30 finished pins).
Reference Table: Standard Binary Fractions to Decimal Equivalents
Inspectors must know the decimal equivalents of common binary shop fractions to immediately recognize drawing callouts and identify gross measurement errors:
- Fraction (in) Decimal Equivalent (in) Exact / Approximate Typical Quality Inspection Application
- 1/64 0.015625 Exact Fine shim stock, feeler gage step
- 1/32 0.03125 Exact Small corner radius, sheet metal gage
- 3/64 0.046875 Exact Pin punch diameter, retaining pin hole
- 1/16 0.0625 Exact Wall thickness, snap ring slot width
- 3/32 0.09375 Exact Cotter pin hole diameter, small drill guide
- 1/8 0.1250 Exact Standard plate, keyway depth, o-ring groove
- 5/32 0.15625 Exact Dowel hole, thread relief width
- 3/16 0.1875 Exact Dowel pin nominal size, bracket tab
- 1/4 0.2500 Exact Common fastener shank, standard plate step
- 5/16 0.3125 Exact Tapped hole nominal diameter (5/16-18 UNC)
- 3/8 0.3750 Exact Flange thickness, manifold port
- 7/16 0.4375 Exact Heavy bolt diameter, counterbore depth
- 1/2 0.5000 Exact Standard shaft step, baseplate thickness
- 5/8 0.6250 Exact Shaft journal diameter, coupling bore
- 3/4 0.7500 Exact Heavy structural plate, spindle diameter
- 7/8 0.8750 Exact Bushing OD, hydraulic cylinder wall
- 1 1.0000 Exact Standard gage block base, master setting ring
- Powers and Square Roots in Metrology
Powers and square roots appear repeatedly across dimensional inspection and statistical quality control:
Pythagorean Theorem for Triangular Inspection :
When calculating oblique depths, hole centers, or sine plate heights: c = a 2 + b 2 c = \sqrt{a^2 + b^2} c = a 2 + b 2 a = c 2 − b 2 a = \sqrt{c^2 - b^2} a = c 2 − b 2 Shop Scenario : A quality inspector inspects a 90-degree corner chamfer by measuring the horizontal leg (a = 0.300 in) and vertical leg (b = 0.400 in). The theoretical hypotenuse is: c = 0.300 2 + 0.400 2 = 0.0900 + 0.1600 = 0.2500 = 0.500 in c = \sqrt{0.300^2 + 0.400^2} = \sqrt{0.0900 + 0.1600} = \sqrt{0.2500} = 0.500\text{ in} c = 0.30 0 2 + 0.40 0 2 = 0.0900 + 0.1600 = 0.2500 = 0.500 in
Root-Mean-Square (RMS) and Standard Deviation :
Surface finish analysis historically utilized RMS roughness (Rq), calculating the square root of the arithmetic mean of the squared profile deviations: R q = 1 n ∑ i = 1 n y i 2 R_q = \sqrt{\frac{1}{n} \sum_{i=1}^n y_i^2} R q = n 1 i = 1 ∑ n y i 2
Sample standard deviation (s), the bedrock of Statistical Process Control (SPC) and capability indices (Cp, Cpk), calculates the square root of variance: s = ∑ ( x i − x ˉ ) 2 n − 1 s = \sqrt{\frac{\sum (x_i - \bar{x})^2}{n - 1}} s = n − 1 ∑ ( x i − x ˉ ) 2
Rounding, Truncating, and Metrology Conventions
Rounding inspection data incorrectly is a primary cause of audit findings, customer return disputes, and calibration discrepancies.
Standard Rounding ("Round Half Up")
In elementary arithmetic, if the digit immediately following the desired place value is 5, 6, 7, 8, or 9, the preceding digit rounds up by 1. If it is 0, 1, 2, 3, or 4, the digit remains unchanged.
Example: 0.1256 rounded to three decimal places becomes 0.126.
Example: 0.1254 rounded to three decimal places becomes 0.125.
"Round to Even" (Banker's Rounding / ASTM E29)
The traditional "round half up" rule has an inherent statistical flaw: the digits 1, 2, 3, 4 round down (4 numbers), while 5, 6, 7, 8, 9 round up (5 numbers). Over large production datasets (e.g., 5,000 inspection measurements), this creates a systematic positive upward bias.
To eliminate this bias, ASTM E29 ( Standard Practice for Using Significant Digits in Test Data to Determine Conformance with Specifications ) and ISO standards mandate the Round to Even (or Banker's Rounding) convention:
If the digit following the last retained place is strictly greater than 5 (or a 5 followed by non-zero digits), round up .
If the digit following the last retained place is strictly less than 5 , round down (truncate).
If the digit following the last retained place is exactly 5 (or 5 followed only by zeros):
Round to the nearest even number.
If the retained digit is odd , round up to make it even.
If the retained digit is even , keep it even (round down).
Measured Value Target Precision Digit Following Target Retained Digit Result (ASTM E29 Round to Even) Contrast: Round Half Up
- 2.4550 2 decimal places 5 followed by 0 5 (odd) 2.46 2.46
- 2.4450 2 decimal places 5 followed by 0 4 (even) 2.44 2.45 (Biased!)
- 12.37500 2 decimal places 5 followed by 0s 7 (odd) 12.38 12.38
- 12.36500 2 decimal places 5 followed by 0s 6 (even) 12.36 12.37 (Biased!)
- 0.082501 3 decimal places 5 followed by 01 (>5) 2 (even) 0.083 0.083
- Truncation Risks in Quality Assurance
Truncation refers to dropping all digits past a designated decimal place without rounding. For example, truncating 0.3758 to three decimal places yields 0.375. Inspector Caution : Many digital readouts (DROs) or automated data logging scripts inadvertently truncate rather than round. This creates an artificial negative shift in inspection records. Truncation should never be substituted for rounding unless explicitly mandated by customer drawing notes.
Significant Digits (Significant Figures) in Inspection
A reported dimension communicates both the magnitude of the feature and the precision of the measuring system.
Rules for Identifying Significant Digits
Non-zero digits are always significant: 48.7 has 3 significant figures.
Captive zeros (zeros between non-zero digits) are always significant: 5.009 has 4 significant figures.
Leading zeros (zeros preceding the first non-zero digit) are never significant; they merely locate the decimal point: 0.0028 has 2 significant figures.
Trailing zeros to the right of the decimal point are always significant: 6.300 has 4 significant figures (indicating measurement resolution to 0.001 in).
Trailing zeros in an integer without a decimal point are ambiguous: 1500 may have 2, 3, or 4 significant digits. In quality reporting, use scientific notation (1.50 x 10^3) or write an explicit decimal point (1500. indicates 4 significant figures).
Mathematical Rules for Operations
Addition and Subtraction (Least Decimal Places) :
The result can have no more decimal places than the measurement with the fewest decimal places.
Example : Adding pin heights: 1.205 in + 0.03 in + 0.4562 in = 1.6912 in.
The measurement 0.03 in has only two decimal places.
Correct reported result: 1.69 in .
Multiplication and Division (Least Significant Figures) :
The final calculated value must contain no more significant figures than the factor with the fewest significant figures.
Example : Calculating cross-sectional area of a test coupon: Width = 1.25 in (3 sig figs), Thickness = 0.1250 in (4 sig figs).
Area = 1.25 * 0.1250 = 0.15625 sq in.
Reporting limit: 3 significant figures -> 0.156 sq in .
Guard Digits in Multi-Step Metrology Calculations :
In complex dimensional calculations (such as calculating true position, coordinate rotations, or volume), never round intermediate numbers. Carry all available calculator digits (or at least two guard digits beyond the required resolution) and apply significant figure rounding only to the final reported result.
Percent Change and Percentage Error
Quality inspectors frequently evaluate material shrinkage, tool wear rates, calibration deviations, and scrap rate variations using percentage formulas.
Percent Change Formula
Percent change measures the relative increase or decrease of a parameter over time or processing: Percent Change = New Value − Original Value Original Value × 100 % \text{Percent Change} = \frac{\text{New Value} - \text{Original Value}}{\text{Original Value}} \times 100\% Percent Change = Original Value New Value − Original Value × 100%
If positive, it reflects a percent increase (e.g., thermal expansion or plating build-up).
If negative, it reflects a percent decrease (e.g., sintering shrinkage or grinding stock removal).
Worked Example : A sintered metal bushing measures 1.028 in outside diameter green (compacted) and shrinks to 1.006 in after furnace firing. Percent Change = 1.006 − 1.028 1.028 × 100 % = − 0.022 1.028 × 100 % = − 2.14 % \text{Percent Change} = \frac{1.006 - 1.028}{1.028} \times 100\% = \frac{-0.022}{1.028} \times 100\% = -2.14\% Percent Change = 1.028 1.006 − 1.028 × 100% = 1.028 − 0.022 × 100% = − 2.14% The process exhibits a 2.14% shrinkage rate.
Percentage Error Formula
In calibration and gage verification, percentage error compares an instrument's measured reading against a certified, traceable reference standard: Percentage Error = ∣ Measured Reading − Certified Standard Value ∣ Certified Standard Value × 100 % \text{Percentage Error} = \frac{|\text{Measured Reading} - \text{Certified Standard Value}|}{\text{Certified Standard Value}} \times 100\% Percentage Error = Certified Standard Value ∣ Measured Reading − Certified Standard Value ∣ × 100%
Worked Example : A 2.0000-inch Class X master gage block is measured using an electronic bench comparator. The inspector records five readings averaging 2.0004 in. Percentage Error = ∣ 2.0004 − 2.0000 ∣ 2.0000 × 100 % = 0.0004 2.0000 × 100 % = 0.02 % \text{Percentage Error} = \frac{|2.0004 - 2.0000|}{2.0000} \times 100\% = \frac{0.0004}{2.0000} \times 100\% = 0.02\% Percentage Error = 2.0000 ∣2.0004 − 2.0000∣ × 100% = 2.0000 0.0004 × 100% = 0.02%
Real Shop Inspection Scenarios & Common Exam Traps
Exam Trap: Cumulative Rounding Error ("Rounding Twice") :
Consider a feature specification of 0.50 +/- 0.02 in (tolerance band: 0.48 to 0.52 in). An unrounded calculation yields 0.5248 in. If an inspector prematurely rounds to three decimal places (0.525 in) and subsequently rounds to two decimal places, they round 0.525 up to 0.53 in—falsely declaring a conforming part nonconforming! Rounding directly from 0.5248 to two decimal places yields 0.52 in, which is strictly conforming.
Exam Trap: Confusing Measurement Error with Percent of Tolerance Consumed :
A drawing specifies a feature at 1.000 +/- 0.005 in (total tolerance 0.010 in). A part measures 1.003 in.
The measurement deviation (error from nominal) is +0.003 in.
The percent error relative to nominal is (0.003 / 1.000) * 100% = 0.3%.
The percent of tolerance band consumed is (0.003 / 0.005) * 100% = 60% of unilateral tolerance (or 30% of total bilateral tolerance band). Exam questions frequently test whether candidates distinguish between error relative to nominal size versus consumption of allowable tolerance.