Study guide
Notation, Unit Conversions, Ratios, and Decibels
ASQ Certified Calibration Technician CCT
Track your progress as you read
Finished sections get a check mark.
1 SI Definitions and Realizations
The Seven SI Base Units and the 2019 Revision Realizing SI Units in the Laboratory
2 Derived Units, Prefixes, and Constants
Derived Units and Physical Calculations SI Prefixes and Compound-Unit Conversions Physical Constants and Their Applications
3 Measurement Standards and Selection
Types and Roles of Measurement Standards Selecting Standards for an Operational Hierarchy
4 Traceability and Standard Substitution
Traceability Evidence and Impact Review Evaluating and Authorizing Standard Substitution
5 Dimensional and Mechanical Metrology
Linear Displacement and Dimensional Instruments Geometry, Thermal Expansion, and Mass Force, Torque, Pressure, Density, and Viscosity
6 Temperature, Humidity, Volume, and Flow
Temperature Sensors and Thermometer Corrections Humidity, Volume, and Flow
7 Electrical, Optical, and Time Measurements
Voltage, Current, Resistance, and AC Measurement Time, Frequency, Light, Power, and Energy
8 Measurement Methods
Direct, Indirect, and Ratio Methods Transfer, Differential, and UUT Substitution Methods
9 Instrument Characteristics and Measurement Data
Sensitivity, Variability, and Precision Bias, Linearity, Stability, and Hysteresis Data Formats, Readability, and Suitability Resolution, Confidentiality, and Data Integrity
10 Specifications and Error Sources
Interpreting IM&TE Specification Terms Specification Calculations and Operating Qualifiers Instrument, Operator, and Circuit Errors Environmental, Geometric, and Interference Errors
11 Measurement Assurance
Check Standards and Statistical Process Control Proficiency Comparisons and Gage R&R
12 Procedures and Governing Requirements
Procedure Structure and Published Practice Sources Method Development, Deviations, and Handling Regulations, Contracts, and Requirement Precedence
13 Instrument Adjustment and Standardization
Calibration, Zero, and Span Adjustment Linearization, Nulling, and Digital Adjustment
14 Calibration Environment
Temperature and Humidity Control ESD, Dust, Vibration, and Pressure
15 Operations, Roles, and Scheduling
Receiving, As-Found Testing, and Calibration Flow Asset Identification, Tracking, and Release Laboratory Roles and Initial Scheduling Interval Review, Overdue Control, and Workloads
16 Method and Software Validation
Method Verification and Validation Studies Software, Firmware, and Spreadsheet Validation
17 Records and Formal Reporting
Controlled Records and Calibration Certificates Status Labels and Out-of-Tolerance Reporting
18 Uncertainty Concepts and Decision Risk
Error, Uncertainty, and Measurand Definition TAR, TUR, Tolerance Consumption, and Capability Conformity Decisions and False-Accept Risk Guard Bands and Conformity Reporting
19 Uncertainty Models and Budgets
Measurement Models and Uncertainty Contributions Distribution Selection and Budget Assembly
20 Type A and Type B Evaluation
Type A Statistics and Repeatability Outlier Review and Worked Type A Analysis Type B Distribution Models
21 Uncertainty Reporting and Applied Mathematics
Propagation, Covariance, and Effective Degrees of Freedom Coverage Factors and Expanded-Uncertainty Reporting Notation, Unit Conversions, Ratios, and Decibels Rounding, Algebra, Graphs, and Angular Conversions
22 Quality Systems and Tools
Quality-System Components, Planning, and Improvement The Seven Basic Quality Tools
23 Audits and Corrective Action
Audit Types, Planning, and Nonconformance Control Impact Assessment, Root Cause, and Effectiveness
24 Safety, Standards, and Professional Conduct
Electrical, Pressure, Chemical, and Laboratory Hazards PPE, Safety Data Sheets, 6S, and Safe Restoration Quality Standards, GUM, and VIM Accreditation Scope and Professional Ethics
Notation, Unit Conversions, Ratios, and Decibels
Key Takeaways Explain This Topic AI
Engineering notation uses exponents that are multiples of three.
Area and volume conversions require squared and cubed linear conversion factors.
Power-ratio decibels use 10 log₁₀; voltage-ratio 20 log₁₀ requires the appropriate impedance relationship.
Last updated: October 2026
Daily operations in calibration laboratories require rapid, error-free execution of applied technical mathematics. From converting non-SI process transmitter readings into base SI units to resolving dynamic decibel attenuation on RF spectrum analyzers, the calibration technician must possess a rigorous command of mathematical principles.
Scientific vs. Engineering Notation and SI Metric Prefixes
In metrology, physical quantities span dozens of orders of magnitude—from sub-atomic displacements in laser interferometers ( 10 − 12 m 10^{-12}\text{ m} 1 0 − 12 m ) to electrical insulation resistances ( 10 12 Ω 10^{12}\ \Omega 1 0 12 Ω ).
Scientific Notation
A number expressed in scientific notation has the form: m × 10 n , where 1 ≤ ∣ m ∣ < 10 and n ∈ Z m \times 10^n, \quad \text{where } 1 \le |m| < 10 \text{ and } n \in \mathbb{Z} m × 1 0 n , where 1 ≤ ∣ m ∣ < 10 and n ∈ Z
Example: 1 , 475 , 000 Ω = 1.475 × 10 6 Ω 1,475,000\ \Omega = 1.475 \times 10^6\ \Omega 1 , 475 , 000 Ω = 1.475 × 1 0 6 Ω
Example: 0.000000035 s = 3.5 × 10 − 8 s 0.000000035\text{ s} = 3.5 \times 10^{-8}\text{ s} 0.000000035 s = 3.5 × 1 0 − 8 s
Engineering Notation
Engineering notation imposes an additional operational constraint: the exponent n n n must be an integer multiple of 3 ( n ∈ { … , − 9 , − 6 , − 3 , 0 , 3 , 6 , 9 , 12 , … } n \in \{\dots, -9, -6, -3, 0, 3, 6, 9, 12, \dots\} n ∈ { … , − 9 , − 6 , − 3 , 0 , 3 , 6 , 9 , 12 , … } ), and the mantissa m m m satisfies 1 ≤ ∣ m ∣ < 1000 1 \le |m| < 1000 1 ≤ ∣ m ∣ < 1000 .
Engineering notation is preferred in calibration because every exponent of 3 corresponds directly to an established SI prefix , facilitating direct instrument range selection:
SI Prefix Symbol Multiplication Factor Engineering Exponent Calibration Laboratory Example
Tera T \text{T} T 1 , 000 , 000 , 000 , 000 1,000,000,000,000 1 , 000 , 000 , 000 , 000 10 12 10^{12} 1 0 12 Insulation resistance: 2.4 × 10 12 Ω = 2.4 T Ω 2.4 \times 10^{12}\ \Omega = 2.4\text{ T}\Omega 2.4 × 1 0 12 Ω = 2.4 T Ω
Giga G \text{G} G 1 , 000 , 000 , 000 1,000,000,000 1 , 000 , 000 , 000 10 9 10^9 1 0 9 RF frequency standard: 10.0 × 10 9 Hz = 10.0 GHz 10.0 \times 10^9\text{ Hz} = 10.0\text{ GHz} 10.0 × 1 0 9 Hz = 10.0 GHz
Mega M \text{M} M 1 , 000 , 000 1,000,000 1 , 000 , 000 10 6 10^6 1 0 6 Megohmmeter test: 1.5 × 10 6 Ω = 1.5 M Ω 1.5 \times 10^6\ \Omega = 1.5\text{ M}\Omega 1.5 × 1 0 6 Ω = 1.5 M Ω
Kilo k \text{k} k 1 , 000 1,000 1 , 000 10 3 10^3 1 0 3 High-voltage divider: 25.0 × 10 3 V = 25.0 kV 25.0 \times 10^3\text{ V} = 25.0\text{ kV} 25.0 × 1 0 3 V = 25.0 kV
(none) (none) 1 1 1 10 0 10^0 1 0 0 Base SI unit: 1.00000 V 1.00000\text{ V} 1.00000 V
Milli m \text{m} m 0.001 0.001 0.001 10 − 3 10^{-3} 1 0 − 3 Shunt current: 4.5 × 10 − 3 A = 4.5 mA 4.5 \times 10^{-3}\text{ A} = 4.5\text{ mA} 4.5 × 1 0 − 3 A = 4.5 mA
Micro μ \mu μ 0.000001 0.000001 0.000001 10 − 6 10^{-6} 1 0 − 6 Strain gage output: 125.0 × 10 − 6 V = 125.0 μ V 125.0 \times 10^{-6}\text{ V} = 125.0\ \mu\text{V} 125.0 × 1 0 − 6 V = 125.0 μ V
Nano n \text{n} n 0.000000001 0.000000001 0.000000001 10 − 9 10^{-9} 1 0 − 9 Time interval counter: 25.0 × 10 − 9 s = 25.0 ns 25.0 \times 10^{-9}\text{ s} = 25.0\text{ ns} 25.0 × 1 0 − 9 s = 25.0 ns
Pico p \text{p} p 0.000000000001 0.000000000001 0.000000000001 10 − 12 10^{-12} 1 0 − 12 Capacitance standard: 100.0 × 10 − 12 F = 100.0 pF 100.0 \times 10^{-12}\text{ F} = 100.0\text{ pF} 100.0 × 1 0 − 12 F = 100.0 pF
Femto f \text{f} f 0.000000000000001 0.000000000000001 0.000000000000001 10 − 15 10^{-15} 1 0 − 15 Ultra-low electrometer: 10.0 × 10 − 15 A = 10.0 fA 10.0 \times 10^{-15}\text{ A} = 10.0\text{ fA} 10.0 × 1 0 − 15 A = 10.0 fA
English to Metric Unit Conversions
Under international treaty agreements dating back to the 1959 International Yard and Pound Agreement, English (US Customary) units are legally defined in terms of exact SI metric equivalents.
Length, Area, and Volume Conversions
Length: 1 inch (in) = 25.4 mm = 0.0254 m (EXACT by definition) 1\text{ inch (in)} = 25.4\text{ mm} = 0.0254\text{ m} \quad \text{(EXACT by definition)} 1 inch (in) = 25.4 mm = 0.0254 m (EXACT by definition) 1 foot (ft) = 12 in = 304.8 mm = 0.3048 m 1\text{ foot (ft)} = 12\text{ in} = 304.8\text{ mm} = 0.3048\text{ m} 1 foot (ft) = 12 in = 304.8 mm = 0.3048 m 1 microinch ( μ in) = 10 − 6 in = 0.0254 μ m = 25.4 nm 1\text{ microinch (}\mu\text{in)} = 10^{-6}\text{ in} = 0.0254\ \mu\text{m} = 25.4\text{ nm} 1 microinch ( μ in) = 1 0 − 6 in = 0.0254 μ m = 25.4 nm
Area: 1 in 2 = ( 25.4 mm ) 2 = 645.16 mm 2 = 6.4516 × 10 − 4 m 2 1\text{ in}^2 = (25.4\text{ mm})^2 = 645.16\text{ mm}^2 = 6.4516 \times 10^{-4}\text{ m}^2 1 in 2 = ( 25.4 mm ) 2 = 645.16 mm 2 = 6.4516 × 1 0 − 4 m 2
Volume: 1 in 3 = ( 25.4 mm ) 3 = 16 , 387.064 mm 3 = 16.387064 cm 3 = 16.387064 mL 1\text{ in}^3 = (25.4\text{ mm})^3 = 16,387.064\text{ mm}^3 = 16.387064\text{ cm}^3 = 16.387064\text{ mL} 1 in 3 = ( 25.4 mm ) 3 = 16 , 387.064 mm 3 = 16.387064 cm 3 = 16.387064 mL 1 US Gallon = 231 in 3 = 3.785411784 Liters (L) 1\text{ US Gallon} = 231\text{ in}^3 = 3.785411784\text{ Liters (L)} 1 US Gallon = 231 in 3 = 3.785411784 Liters (L)
Mass and Force Conversions
Mass: 1 pound-mass (lb) = 0.45359237 kg (EXACT by definition) 1\text{ pound-mass (lb)} = 0.45359237\text{ kg} \quad \text{(EXACT by definition)} 1 pound-mass (lb) = 0.45359237 kg (EXACT by definition) 1 ounce (oz) = 1 16 lb = 28.349523125 g 1\text{ ounce (oz)} = \frac{1}{16}\text{ lb} = 28.349523125\text{ g} 1 ounce (oz) = 16 1 lb = 28.349523125 g
Force: 1 pound-force (lbf) = 1 lb × 9.80665 m/s 2 = 4.4482216152605 N ≈ 4.44822 N 1\text{ pound-force (lbf)} = 1\text{ lb} \times 9.80665\text{ m/s}^2 = 4.4482216152605\text{ N} \approx 4.44822\text{ N} 1 pound-force (lbf) = 1 lb × 9.80665 m/s 2 = 4.4482216152605 N ≈ 4.44822 N
Pressure Conversions
Pressure is force per unit area ( P = F / A P = F/A P = F / A ). Converting between English and metric pressure units is a core CCT skill: 1 psi = 1 lbf 1 in 2 = 4.4482216 N 6.4516 × 10 − 4 m 2 = 6 , 894.757 Pa = 6.894757 kPa 1\text{ psi} = \frac{1\text{ lbf}}{1\text{ in}^2} = \frac{4.4482216\text{ N}}{6.4516 \times 10^{-4}\text{ m}^2} = 6,894.757\text{ Pa} = 6.894757\text{ kPa} 1 psi = 1 in 2 1 lbf = 6.4516 × 1 0 − 4 m 2 4.4482216 N = 6 , 894.757 Pa = 6.894757 kPa
From Unit To Unit Exact or Standard Multiplier Direct Conversion Formula
psi \text{psi} psi kPa \text{kPa} kPa 6.894757 6.894757 6.894757 P kPa = P psi × 6.894757 P_{\text{kPa}} = P_{\text{psi}} \times 6.894757 P kPa = P psi × 6.894757
psi \text{psi} psi bar \text{bar} bar 0.06894757 0.06894757 0.06894757 P bar = P psi × 0.06894757 P_{\text{bar}} = P_{\text{psi}} \times 0.06894757 P bar = P psi × 0.06894757
bar \text{bar} bar kPa \text{kPa} kPa 100.000 100.000 100.000 P kPa = P bar × 100 P_{\text{kPa}} = P_{\text{bar}} \times 100 P kPa = P bar × 100
bar \text{bar} bar psi \text{psi} psi 14.50377 14.50377 14.50377 P psi = P bar × 14.50377 P_{\text{psi}} = P_{\text{bar}} \times 14.50377 P psi = P bar × 14.50377
atm \text{atm} atm Pa \text{Pa} Pa 101 , 325 101,325 101 , 325 P Pa = P atm × 101 , 325 P_{\text{Pa}} = P_{\text{atm}} \times 101,325 P Pa = P atm × 101 , 325
atm \text{atm} atm psi \text{psi} psi 14.69595 14.69595 14.69595 P psi = P atm × 14.69595 P_{\text{psi}} = P_{\text{atm}} \times 14.69595 P psi = P atm × 14.69595
Torr \text{Torr} Torr Pa \text{Pa} Pa 101325 / 760 ≈ 133.322368 101325/760\approx133.322368 101325/760 ≈ 133.322368 Multiply Torr by 101325 / 760 101325/760 101325/760 ; distinguish physical mercury-column conventions.
inHg, conventional \text{inHg, conventional} inHg, conventional psi \text{psi} psi 0.491154 0.491154 0.491154 P psi = P inHg × 0.491154 P_{\text{psi}} = P_{\text{inHg}} \times 0.491154 P psi = P inHg × 0.491154
inHg, conventional \text{inHg, conventional} inHg, conventional kPa \text{kPa} kPa 3.386389 3.386389 3.386389 P kPa = P inHg × 3.386389 P_{\text{kPa}} = P_{\text{inHg}} \times 3.386389 P kPa = P inHg × 3.386389
inH 2 O (at 60 ∘ F) \text{inH}_2\text{O (at } 60^\circ\text{F)} inH 2 O (at 6 0 ∘ F) Pa \text{Pa} Pa 248.84 248.84 248.84 P Pa = P inH2O × 248.84 P_{\text{Pa}} = P_{\text{inH2O}} \times 248.84 P Pa = P inH2O × 248.84
inH 2 O (at 60 ∘ F) \text{inH}_2\text{O (at } 60^\circ\text{F)} inH 2 O (at 6 0 ∘ F) psi \text{psi} psi 0.03606 0.03606 0.03606 P psi = P inH2O × 0.03606 P_{\text{psi}} = P_{\text{inH2O}} \times 0.03606 P psi = P inH2O × 0.03606
- Ratios, Percentages, Parts Per Million (ppm), and Decibels (dB)
- Percentages vs. Parts Per Million (ppm)
In high-accuracy electrical and dimensional metrology, fractional tolerances are too minute to express comfortably as percentages. The metrology community uses parts per million (ppm) : Fraction = Error Nominal \text{Fraction} = \frac{\text{Error}}{\text{Nominal}} Fraction = Nominal Error Percentage (%) = Fraction × 10 2 = Error Nominal × 100 \text{Percentage (\%)} = \text{Fraction} \times 10^2 = \frac{\text{Error}}{\text{Nominal}} \times 100 Percentage (%) = Fraction × 1 0 2 = Nominal Error × 100 Parts Per Million (ppm) = Fraction × 10 6 = Error Nominal × 1 , 000 , 000 \text{Parts Per Million (ppm)} = \text{Fraction} \times 10^6 = \frac{\text{Error}}{\text{Nominal}} \times 1,000,000 Parts Per Million (ppm) = Fraction × 1 0 6 = Nominal Error × 1 , 000 , 000
- Direct Conversion Rules
- 1 % = 10 , 000 ppm 1\% = 10,000\text{ ppm} 1% = 10 , 000 ppm
- 0.1 % = 1 , 000 ppm 0.1\% = 1,000\text{ ppm} 0.1% = 1 , 000 ppm
- 0.01 % = 100 ppm 0.01\% = 100\text{ ppm} 0.01% = 100 ppm
- 0.001 % = 10 ppm 0.001\% = 10\text{ ppm} 0.001% = 10 ppm
- 0.0001 % = 1 ppm 0.0001\% = 1\text{ ppm} 0.0001% = 1 ppm
- Worked Example: Converting ppm Tolerance to Absolute Voltage
A multifunction calibrator is specified on its 10.00000 V 10.00000\text{ V} 10.00000 V range with a 1-year specification of: Tolerance = ± ( 15 ppm of Reading + 3 ppm of Range ) \text{Tolerance} = \pm (15\text{ ppm of Reading} + 3\text{ ppm of Range}) Tolerance = ± ( 15 ppm of Reading + 3 ppm of Range )
If the technician sets the calibrator output to 5.00000 V 5.00000\text{ V} 5.00000 V : Reading error = 5.00000 V × ( 15 × 10 − 6 ) = 0.000075 V = 75 μ V \text{Reading error} = 5.00000\text{ V} \times (15 \times 10^{-6}) = 0.000075\text{ V} = 75\ \mu\text{V} Reading error = 5.00000 V × ( 15 × 1 0 − 6 ) = 0.000075 V = 75 μ V Range error = 10.00000 V × ( 3 × 10 − 6 ) = 0.000030 V = 30 μ V \text{Range error} = 10.00000\text{ V} \times (3 \times 10^{-6}) = 0.000030\text{ V} = 30\ \mu\text{V} Range error = 10.00000 V × ( 3 × 1 0 − 6 ) = 0.000030 V = 30 μ V Total Absolute Tolerance = 75 μ V + 30 μ V = 105 μ V = 0.000105 V \text{Total Absolute Tolerance} = 75\ \mu\text{V} + 30\ \mu\text{V} = 105\ \mu\text{V} = 0.000105\text{ V} Total Absolute Tolerance = 75 μ V + 30 μ V = 105 μ V = 0.000105 V
Decibels (dB) in Power and Voltage Metrology
The decibel (dB) is a logarithmic ratio used in RF, acoustics, and signal processing.
Power Ratios ( 10 log 10 10 \log_{10} 10 lo g 10 )
Because power is directly proportional to energy, the decibel power ratio is defined as: dB = 10 log 10 ( P 2 P 1 ) \text{dB} = 10 \log_{10}\left(\frac{P_2}{P_1}\right) dB = 10 lo g 10 ( P 1 P 2 )
+ 3 dB ≈ 2 × power +3\text{ dB} \approx 2\times\text{ power} + 3 dB ≈ 2 × power (e.g., 10 mW → 20 mW 10\text{ mW} \to 20\text{ mW} 10 mW → 20 mW )
+ 10 dB = 10 × power +10\text{ dB} = 10\times\text{ power} + 10 dB = 10 × power (e.g., 1 mW → 10 mW 1\text{ mW} \to 10\text{ mW} 1 mW → 10 mW )
+ 20 dB = 100 × power +20\text{ dB} = 100\times\text{ power} + 20 dB = 100 × power (e.g., 1 mW → 100 mW 1\text{ mW} \to 100\text{ mW} 1 mW → 100 mW )
− 3 dB ≈ 0.5 × power -3\text{ dB} \approx 0.5\times\text{ power} − 3 dB ≈ 0.5 × power (half-power cut-off frequency)
− 10 dB = 0.1 × power -10\text{ dB} = 0.1\times\text{ power} − 10 dB = 0.1 × power ( 90 % 90\% 90% attenuation)
Voltage and Field Ratios ( 20 log 10 20 \log_{10} 20 lo g 10 )
By Joule's law, power is proportional to voltage squared ( P = V 2 / R P = V^2 / R P = V 2 / R ). When impedances are equal ( R 1 = R 2 R_1 = R_2 R 1 = R 2 ): dB = 10 log 10 ( V 2 2 / R V 1 2 / R ) = 10 log 10 [ ( V 2 V 1 ) 2 ] = 20 log 10 ( V 2 V 1 ) \text{dB} = 10 \log_{10}\left(\frac{V_2^2 / R}{V_1^2 / R}\right) = 10 \log_{10}\left[\left(\frac{V_2}{V_1}\right)^2\right] = 20 \log_{10}\left(\frac{V_2}{V_1}\right) dB = 10 lo g 10 ( V 1 2 / R V 2 2 / R ) = 10 lo g 10 [ ( V 1 V 2 ) 2 ] = 20 lo g 10 ( V 1 V 2 )
+ 6 dB ≈ 2 × voltage +6\text{ dB} \approx 2\times\text{ voltage} + 6 dB ≈ 2 × voltage (doubling voltage)
+ 20 dB = 10 × voltage +20\text{ dB} = 10\times\text{ voltage} + 20 dB = 10 × voltage (decade voltage increase)
- + 40 dB = 100 × voltage +40\text{ dB} = 100\times\text{ voltage} + 40 dB = 100 × voltage
- − 6 dB ≈ 0.5 × voltage -6\text{ dB} \approx 0.5\times\text{ voltage} − 6 dB ≈ 0.5 × voltage (half-voltage)
− 20 dB = 0.1 × voltage -20\text{ dB} = 0.1\times\text{ voltage} − 20 dB = 0.1 × voltage ( 90 % 90\% 90% voltage drop)
Warning
Classic Exam Trap: Using 10 log 10 10 \log_{10} 10 lo g 10 for voltage ratios or 20 log 10 20 \log_{10} 20 lo g 10 for power ratios. Always verify whether the physical quantity is power ( 10 log 10 10 \log_{10} 10 lo g 10 ) or field/voltage/current ( 20 log 10 20 \log_{10} 20 lo g 10 ).
Worked conversion checks with units
Consider a flow indication of 2 US gallons per minute. The volume factor is approximately 3.785412 litres per US gallon, giving about 7.570824 litres per minute. Dividing by sixty gives about 0.1261804 litre per second. The gallon definition matters: the imperial gallon has a different volume. A number labeled only “gallons” is incomplete information for a defensible conversion. Record the specified convention before calculating.
Area and volume provide another check. A rectangular opening measuring 2 inches by 3 inches has area 6 square inches. Multiplying by 645.16 square millimetres per square inch gives 3870.96 square millimetres. Multiplying only by the linear factor 25.4 would leave incorrect dimensions. A cubed-length conversion likewise needs the cubed linear factor, not the area factor. Cancel units on paper before entering numbers into a calculator.
Now suppose an amplifier’s output power is ten times its input power. The gain is 10 dB. If input and output resistances are equal, the associated voltage ratio is the square root of ten, approximately 3.162. A tenfold voltage ratio would instead correspond to a hundredfold power ratio and 20 dB under those equal-resistance conditions. With different resistances, calculate the actual power ratio from voltage squared divided by resistance. Do not use twenty times the voltage logarithm as a substitute for that impedance information.
To reverse the calculation, a power ratio equals ten raised to the gain in decibels divided by ten. Thus a negative 10 dB attenuation leaves one tenth of the initial power. Check direction: output divided by input gives negative gain for attenuation, while input divided by output gives a positive loss. Both conventions can appear in instrument documentation, so identify the stated ratio and sign before comparing results.
A conversion factor can be exact while a measurement remains uncertain. The defined inch-to-metre factor does not improve the uncertainty of the measured length. Convert the result and its uncertainty consistently, retaining intermediate precision until reporting.
Pressure-column conventions matter. The 248.84 Pa water-column factor above is for 60°F, not 20°C. Conventional inch-of-water uses a different factor, approximately 249.0889 Pa. Check the instrument convention before conversion. See NIST SP 811 conversion factors .