Study guide

5.3 Least-Squares Adjustment of Level Nets, Triangulation, and Plane Traverses

PRC Geodetic Engineering Licensure Examination Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Laws, Professional Practice, and Code of Ethics

1.1 Republic Act 8560 & RA 9200 (Philippine Geodetic Engineering Act) and PRC Board Rules 1.2 Code of Ethics for Geodetic Engineers, Professional Responsibilities & Practice Limits 1.3 PRC Licensure Examination Rules, Continuing Professional Development (CPD), and Legal Penalties

2 Chapter 2: Property, Land Registration, and Natural Resources Laws

2.1 Presidential Decree 1529 (Property Registration Decree) & The Torrens System 2.2 Commonwealth Act 141 (Public Land Act), Land Classification, and Public Land Patents 2.3 Comprehensive Agrarian Reform Law (RA 6657) & Environmental/Forestry/Mining Laws (RA 7942, PD 705) 2.4 Civil Code Provisions on Ownership, Property Boundaries, Easements, Obligations, and Contracts 2.5 DENR Administrative Order DAO 2007-29 (Revised Regulations for Land Surveys) & Technical Standards

3 Chapter 3: Mathematics, Geometry, and Trigonometry

3.1 Advanced Algebra, Analytic Geometry, and Solid Geometry in Geodetic Calculations 3.2 Plane and Spherical Trigonometry for Surveying and Astronomical Applications 3.3 Differential and Integral Calculus Applied to Curves, Surfaces, and Surveying Computations

4 Chapter 4: Engineering Sciences, Mechanics, and Economics

4.1 Engineering Mechanics (Statics, Kinematics, Dynamics) for Engineering Applications 4.2 Engineering Economics, Time Value of Money, Cash Flows, Depreciation, and Capital Costing

5 Chapter 5: Theory of Errors and Adjustment of Observations

5.1 Theory of Errors, Precision vs. Accuracy, Random/Systematic Errors, and Error Propagation 5.2 Principles of Least Squares, Most Probable Value, and Weighted Observations 5.3 Least-Squares Adjustment of Level Nets, Triangulation, and Plane Traverses

6 Chapter 6: Plane & Distance/Angle Surveying Fundamentals

6.1 Distance Measurement Methods, Taping Corrections (Temperature, Tension, Sag), and EDM Operations 6.2 Angular Measurements, Transit/Theodolite/Total Station Operations, Bearings, Azimuths, and Magnetic Declination 6.3 Differential, Trigonometric, Barometric, and Profile Leveling Methods

7 Chapter 7: Cadastral, Property, and Special Surveys

7.1 Traverse Computations, Latitudes, Departures, Error of Closure, and Transit/Compass Rule Adjustments 7.2 Area Computations (DMD, DPD, Coordinates) and Subdivision/Consolidation Survey Standards 7.3 Mineral and Mining Surveys, Hydrographic Surveying, Route Alignment, and Construction Layout

8 Chapter 8: Geodetic Surveying and Reference Datums

8.1 Ellipsoids, Geoid Undulation, Geodetic Datums, and the Philippine Reference System of 1992 (PRS92) 8.2 Geodetic Position Computations: Direct and Inverse Problems on the Reference Ellipsoid 8.3 Triangulation, Trilateration, Baseline Measurements, and National Geodetic Control Networks

9 Chapter 9: Geodetic Astronomy and Satellite Positioning (GNSS)

9.1 Celestial Sphere, Equatorial/Horizon Coordinate Systems, and Astronomical Time Systems (UT, ST, LMT) 9.2 Determination of Latitude, Longitude, and Solar/Polaris Azimuth by Astronomical Observations 9.3 Global Navigation Satellite Systems (GNSS/GPS), Carrier-Phase Positioning, RTK, and Static Baselines

10 Chapter 10: Cartography, Map Projections, and Coordinate Systems

10.1 Cartographic Principles, Map Scales, Map Symbolization, and Cartographic Generalization 10.2 Map Projections: Universal Transverse Mercator (UTM) and Philippine Transverse Mercator (PTM) Grid Systems 10.3 Survey Plan Preparation, Technical Description Formatting, and DENR Approval Workflows

11 Chapter 11: Photogrammetry, Remote Sensing, and Geographic Information Systems (GIS)

11.1 Aerial Photogrammetry, Photographic Scale, Relief Displacement, Flight Planning, and Stereoscopy 11.2 Digital Photogrammetry, UAV/Drone Surveying, Orthophoto Generation, and LiDAR Data Processing 11.3 Remote Sensing Spectral Signatures, Multispectral Image Classification, and GIS Spatial Analysis

5.3 Least-Squares Adjustment of Level Nets, Triangulation, and Plane Traverses

Key Takeaways Explain This Topic AI

Parametric least squares expresses observations as direct functions of unknown parameters (V = AX - L), solving for unknown coordinate or elevation updates via X = (A^T P A)^(-1) A^T P L.

1D leveling net adjustments construct design matrix A with entries +1 and -1 based on level line directional topology, weighting each line inversely to its length.

Non-linear observation equations in 2D triangulation/trilateration must be linearized using first-order Taylor series expansion around initial approximate station coordinates.

Least-squares plane traverse adjustment is superior to the empirical Compass (Bowditch) Rule because it incorporates rigorous instrument standard errors and computes coordinate error ellipses.

Standardized residuals (|v_i*| > 3.0) flag potential blunders in large surveying networks for rejection or re-observation.

Last updated: July 2026

Least-Squares Adjustment of Level Nets, Triangulation, and Plane Traverses

Modern geodetic data processing relies on rigorous matrix formulation of least squares to adjust complex 1D elevation networks, 2D triangulation/trilateration control figures, and plane traverses. This section details parametric matrix techniques ( A T P A X = A T P L A^T P A X = A^T P L A T P A X = A T P L ), non-linear model linearization, error ellipse computation, and residual quality control tests required for GELE mastery.

1. Parametric (Observation Equations) vs. Correlative (Condition Equations) Methods

Least-squares adjustments are executed through two primary formulation approaches:

Feature Parametric Method (Observation Equations) Correlative Method (Condition Equations)

Primary Unknowns Parameter corrections to point coordinates or elevations ( X X X ) Lagrange multipliers (correlatives K K K )

Formulation Expresses each observation directly as a function of parameters ( V = A X − L V = AX - L V = A X − L ) Expresses exact geometric condition closure equations ( B V = W B V = W B V = W )

Number of Equations Equals number of parameters u u u Equals redundancy / degrees of freedom r = n − u r = n - u r = n − u

Matrix Normal System ( A T P A ) u × u X = A T P L (A^T P A)_{u \times u} X = A^T P L ( A T P A ) u × u ​ X = A T P L ( B P − 1 B T ) r × r K = W (B P^{-1} B^T)_{r \times r} K = W ( B P − 1 B T ) r × r ​ K = W

Best Application Complex 2D/3D networks, GNSS vectors, hybrid observations Simple closed leveling loops, geometric polygon triangles

2. Rigorous Matrix Formulation of Parametric Least Squares

For a system of n n n observations and u u u unknown parameters ( n > u n > u n > u ), the linear observation model in matrix notation is: V = A X − L V = A X - L V = A X − L

where:

V n × 1 V_{n \times 1} V n × 1 ​ : Vector of observation residuals ( [ v 1 , v 2 , … , v n ] T [v_1, v_2, \dots, v_n]^T [ v 1 ​ , v 2 ​ , … , v n ​ ] T ).

A n × u A_{n \times u} A n × u ​ : Design Matrix containing partial derivatives of observation equations with respect to unknown parameters ( ∂ f i / ∂ x j \partial f_i / \partial x_j ∂ f i ​ / ∂ x j ​ ).

X u × 1 X_{u \times 1} X u × 1 ​ : Vector of unknown parameter corrections ( [ δ x 1 , δ x 2 , … , δ x u ] T [\delta x_1, \delta x_2, \dots, \delta x_u]^T [ δ x 1 ​ , δ x 2 ​ , … , δ x u ​ ] T ).

L n × 1 L_{n \times 1} L n × 1 ​ : Vector of misclosures ( L = L o b s e r v e d − L c o m p u t e d 0 L = L_{observed} - L_{computed}^0 L = L o b ser v e d ​ − L co m p u t e d 0 ​ ).

P n × n P_{n \times n} P n × n ​ : Weight Matrix (diagonal matrix P = diag ( p 1 , p 2 , … , p n ) P = \text{diag}(p_1, p_2, \dots, p_n) P = diag ( p 1 ​ , p 2 ​ , … , p n ​ ) for independent observations).

2.1 Matrix Derivation of Normal Equations

Minimizing the weighted residual sum V T P V = ( A X − L ) T P ( A X − L ) V^T P V = (A X - L)^T P (A X - L) V T P V = ( A X − L ) T P ( A X − L ) : V T P V = X T A T P A X − 2 X T A T P L + L T P L V^T P V = X^T A^T P A X - 2 X^T A^T P L + L^T P L V T P V = X T A T P A X − 2 X T A T P L + L T P L

Taking partial derivatives with respect to X X X and setting to zero: ∂ ( V T P V ) ∂ X = 2 A T P A X − 2 A T P L = 0 ⟹ ( A T P A ) X = A T P L \frac{\partial(V^T P V)}{\partial X} = 2 A^T P A X - 2 A^T P L = 0 \implies (A^T P A) X = A^T P L ∂ X ∂ ( V T P V ) ​ = 2 A T P A X − 2 A T P L = 0 ⟹ ( A T P A ) X = A T P L

Solving for parameter corrections X X X : X = ( A T P A ) − 1 A T P L X = (A^T P A)^{-1} A^T P L X = ( A T P A ) − 1 A T P L

  • 2.2 Residuals and Variance-Covariance Matrices
  • Adjusted Residuals Vector: V = A X − L V = A X - L V = A X − L

A Posteriori Reference Variance ( σ ^ 0 2 \hat{\sigma}_0^2 σ ^ 0 2 ​ ): σ ^ 0 2 = V T P V n − u = V T P V r \hat{\sigma}_0^2 = \frac{V^T P V}{n - u} = \frac{V^T P V}{r} σ ^ 0 2 ​ = n − u V T P V ​ = r V T P V ​

Parameter Variance-Covariance Matrix ( Σ X \Sigma_X Σ X ​ ): Σ X = σ ^ 0 2 ( A T P A ) − 1 = [ σ x 1 2 σ x 1 x 2 … σ x 2 x 1 σ x 2 2 … ⋮ ⋮ ⋱ ] \Sigma_X = \hat{\sigma}_0^2 (A^T P A)^{-1} = \begin{bmatrix} \sigma_{x_1}^2 & \sigma_{x_1 x_2} & \dots \\ \sigma_{x_2 x_1} & \sigma_{x_2}^2 & \dots \\ \vdots & \vdots & \ddots \end{bmatrix} Σ X ​ = σ ^ 0 2 ​ ( A T P A ) − 1 = ​ σ x 1 ​ 2 ​ σ x 2 ​ x 1 ​ ​ ⋮ ​ σ x 1 ​ x 2 ​ ​ σ x 2 ​ 2 ​ ⋮ ​ … … ⋱ ​ ​

Standard errors of adjusted parameters: σ x j = Σ X , j j \sigma_{x_j} = \sqrt{\Sigma_{X, jj}} σ x j ​ ​ = Σ X , jj ​ ​ .

  • 3. 1D Level Network Adjustment: Step-by-Step Worked Matrix Example
  • Network Geometry

Consider a level net with two fixed control benchmarks BM-1 ( E l e v = 10.000 m Elev = 10.000\text{ m} El e v = 10.000 m ) and BM-2 ( E l e v = 25.000 m Elev = 25.000\text{ m} El e v = 25.000 m ) and two unknown junction points A ( H A H_A H A ​ ) and B ( H B H_B H B ​ ).

Four elevation differences are measured:

  • Obs Line From To Obs Δ h \Delta h Δ h (m) Distance D D D (km) Weight p i = 1 / D p_i = 1/D p i ​ = 1/ D
  • Line 1 BM-1 A + 5.100 +5.100 + 5.100 1.0 1.0 1.0 1.0 1.0 1.0
  • Line 2 A B + 9.800 +9.800 + 9.800 2.0 2.0 2.0 0.5 0.5 0.5
  • Line 3 B BM-2 + 0.200 +0.200 + 0.200 1.0 1.0 1.0 1.0 1.0 1.0
  • Line 4 BM-1 B + 15.000 +15.000 + 15.000 2.0 2.0 2.0 0.5 0.5 0.5

Initial approximate elevations: H A 0 = 15.100 m H_A^0 = 15.100\text{ m} H A 0 ​ = 15.100 m , H B 0 = 24.900 m H_B^0 = 24.900\text{ m} H B 0 ​ = 24.900 m .

Step 1: Form Observation Equations ( v i = a i 1 δ H A + a i 2 δ H B − l i v_i = a_{i1} \delta H_A + a_{i2} \delta H_B - l_i v i ​ = a i 1 ​ δ H A ​ + a i 2 ​ δ H B ​ − l i ​ )

Line 1: H A − 10.000 = 5.100 + v 1 ⟹ δ H A − ( 5.100 − 5.100 ) = δ H A − 0.000 H_A - 10.000 = 5.100 + v_1 \implies \delta H_A - (5.100 - 5.100) = \delta H_A - 0.000 H A ​ − 10.000 = 5.100 + v 1 ​ ⟹ δ H A ​ − ( 5.100 − 5.100 ) = δ H A ​ − 0.000

Line 2: H B − H A = 9.800 + v 2 ⟹ δ H B − δ H A − ( 9.800 − ( 24.900 − 15.100 ) ) = − δ H A + δ H B − 0.000 H_B - H_A = 9.800 + v_2 \implies \delta H_B - \delta H_A - (9.800 - (24.900 - 15.100)) = -\delta H_A + \delta H_B - 0.000 H B ​ − H A ​ = 9.800 + v 2 ​ ⟹ δ H B ​ − δ H A ​ − ( 9.800 − ( 24.900 − 15.100 )) = − δ H A ​ + δ H B ​ − 0.000

Line 3: 25.000 − H B = 0.200 + v 3 ⟹ − δ H B − ( 0.200 − ( 25.000 − 24.900 ) ) = − δ H B − 0.100 25.000 - H_B = 0.200 + v_3 \implies -\delta H_B - (0.200 - (25.000 - 24.900)) = -\delta H_B - 0.100 25.000 − H B ​ = 0.200 + v 3 ​ ⟹ − δ H B ​ − ( 0.200 − ( 25.000 − 24.900 )) = − δ H B ​ − 0.100

Line 4: H B − 10.000 = 15.000 + v 4 ⟹ δ H B − ( 15.000 − 14.900 ) = δ H B − 0.100 H_B - 10.000 = 15.000 + v_4 \implies \delta H_B - (15.000 - 14.900) = \delta H_B - 0.100 H B ​ − 10.000 = 15.000 + v 4 ​ ⟹ δ H B ​ − ( 15.000 − 14.900 ) = δ H B ​ − 0.100

Step 2: Assemble Matrices A = [ 1 0 − 1 1 0 − 1 0 1 ] , P = [ 1.0 0 0 0 0 0.5 0 0 0 0 1.0 0 0 0 0 0.5 ] , L = [ 0.000 0.000 0.100 0.100 ] A = \begin{bmatrix} 1 & 0 \\ -1 & 1 \\ 0 & -1 \\ 0 & 1 \end{bmatrix}, \quad P = \begin{bmatrix} 1.0 & 0 & 0 & 0 \\ 0 & 0.5 & 0 & 0 \\ 0 & 0 & 1.0 & 0 \\ 0 & 0 & 0 & 0.5 \end{bmatrix}, \quad L = \begin{bmatrix} 0.000 \\ 0.000 \\ 0.100 \\ 0.100 \end{bmatrix} A = ​ 1 − 1 0 0 ​ 0 1 − 1 1 ​ ​ , P = ​ 1.0 0 0 0 ​ 0 0.5 0 0 ​ 0 0 1.0 0 ​ 0 0 0 0.5 ​ ​ , L = ​ 0.000 0.000 0.100 0.100 ​ ​

Step 3: Compute Normal Matrix N = A T P A N = A^T P A N = A T P A A T P = [ 1 − 0.5 0 0 0 0.5 − 1.0 0.5 ] A^T P = \begin{bmatrix} 1 & -0.5 & 0 & 0 \\ 0 & 0.5 & -1.0 & 0.5 \end{bmatrix} A T P = [ 1 0 ​ − 0.5 0.5 ​ 0 − 1.0 ​ 0 0.5 ​ ] N = A T P A = [ ( 1 + 0.5 ) ( − 0.5 ) ( − 0.5 ) ( 0.5 + 1.0 + 0.5 ) ] = [ 1.5 − 0.5 − 0.5 2.0 ] N = A^T P A = \begin{bmatrix} (1 + 0.5) & (-0.5) \\ (-0.5) & (0.5 + 1.0 + 0.5) \end{bmatrix} = \begin{bmatrix} 1.5 & -0.5 \\ -0.5 & 2.0 \end{bmatrix} N = A T P A = [ ( 1 + 0.5 ) ( − 0.5 ) ​ ( − 0.5 ) ( 0.5 + 1.0 + 0.5 ) ​ ] = [ 1.5 − 0.5 ​ − 0.5 2.0 ​ ]

Step 4: Compute Right-Hand Side Vector A T P L A^T P L A T P L A T P L = [ 1 ( 0 ) − 0.5 ( 0 ) + 0 ( 0.1 ) + 0 ( 0.1 ) 0 ( 0 ) + 0.5 ( 0 ) − 1.0 ( 0.1 ) + 0.5 ( 0.1 ) ] = [ 0.000 − 0.050 ] A^T P L = \begin{bmatrix} 1(0) - 0.5(0) + 0(0.1) + 0(0.1) \\ 0(0) + 0.5(0) - 1.0(0.1) + 0.5(0.1) \end{bmatrix} = \begin{bmatrix} 0.000 \\ -0.050 \end{bmatrix} A T P L = [ 1 ( 0 ) − 0.5 ( 0 ) + 0 ( 0.1 ) + 0 ( 0.1 ) 0 ( 0 ) + 0.5 ( 0 ) − 1.0 ( 0.1 ) + 0.5 ( 0.1 ) ​ ] = [ 0.000 − 0.050 ​ ]

Step 5: Invert Normal Matrix N − 1 N^{-1} N − 1 det ⁡ ( N ) = ( 1.5 ) ( 2.0 ) − ( − 0.5 ) ( − 0.5 ) = 3.0 − 0.25 = 2.75 \det(N) = (1.5)(2.0) - (-0.5)(-0.5) = 3.0 - 0.25 = 2.75 det ( N ) = ( 1.5 ) ( 2.0 ) − ( − 0.5 ) ( − 0.5 ) = 3.0 − 0.25 = 2.75 N − 1 = 1 2.75 [ 2.0 0.5 0.5 1.5 ] = [ 0.72727 0.18182 0.18182 0.54545 ] N^{-1} = \frac{1}{2.75} \begin{bmatrix} 2.0 & 0.5 \\ 0.5 & 1.5 \end{bmatrix} = \begin{bmatrix} 0.72727 & 0.18182 \\ 0.18182 & 0.54545 \end{bmatrix} N − 1 = 2.75 1 ​ [ 2.0 0.5 ​ 0.5 1.5 ​ ] = [ 0.72727 0.18182 ​ 0.18182 0.54545 ​ ]

Step 6: Solve for Parameter Corrections X = N − 1 A T P L X = N^{-1} A^T P L X = N − 1 A T P L X = [ 0.72727 0.18182 0.18182 0.54545 ] [ 0.000 − 0.050 ] = [ − 0.00909 m − 0.02727 m ] X = \begin{bmatrix} 0.72727 & 0.18182 \\ 0.18182 & 0.54545 \end{bmatrix} \begin{bmatrix} 0.000 \\ -0.050 \end{bmatrix} = \begin{bmatrix} -0.00909\text{ m} \\ -0.02727\text{ m} \end{bmatrix} X = [ 0.72727 0.18182 ​ 0.18182 0.54545 ​ ] [ 0.000 − 0.050 ​ ] = [ − 0.00909 m − 0.02727 m ​ ]

Step 7: Calculate Final Adjusted Elevations

H ^ A = H A 0 + δ H A = 15.100 − 0.0091 = 15.0909 m \hat{H}_A = H_A^0 + \delta H_A = 15.100 - 0.0091 = 15.0909\text{ m} H ^ A ​ = H A 0 ​ + δ H A ​ = 15.100 − 0.0091 = 15.0909 m

H ^ B = H B 0 + δ H B = 24.900 − 0.0273 = 24.8727 m \hat{H}_B = H_B^0 + \delta H_B = 24.900 - 0.0273 = 24.8727\text{ m} H ^ B ​ = H B 0 ​ + δ H B ​ = 24.900 − 0.0273 = 24.8727 m

4. Linearization of 2D Triangulation and Trilateration Equations

In 2D horizontal networks, distance and angle observation equations are non-linear functions of station coordinates ( X A , Y A , X B , Y B ) (X_A, Y_A, X_B, Y_B) ( X A ​ , Y A ​ , X B ​ , Y B ​ ) . They must be linearized using first-order Taylor series expansion around initial approximate coordinates ( X 0 , Y 0 ) (X^0, Y^0) ( X 0 , Y 0 ) .

Distance Observation Equation Linearization S A B = ( X B − X A ) 2 + ( Y B − Y A ) 2 S_{AB} = \sqrt{(X_B - X_A)^2 + (Y_B - Y_A)^2} S A B ​ = ( X B ​ − X A ​ ) 2 + ( Y B ​ − Y A ​ ) 2 ​ δ S A B = − cos ⁡ ( α A B ) δ X A − sin ⁡ ( α A B ) δ Y A + cos ⁡ ( α A B ) δ X B + sin ⁡ ( α A B ) δ Y B \delta S_{AB} = -\cos(\alpha_{AB}) \, \delta X_A - \sin(\alpha_{AB}) \, \delta Y_A + \cos(\alpha_{AB}) \, \delta X_B + \sin(\alpha_{AB}) \, \delta Y_B δ S A B ​ = − cos ( α A B ​ ) δ X A ​ − sin ( α A B ​ ) δ Y A ​ + cos ( α A B ​ ) δ X B ​ + sin ( α A B ​ ) δ Y B ​

where α A B \alpha_{AB} α A B ​ is the grid azimuth from station A to B.

Azimuth Observation Equation Linearization α A B = arctan ⁡ ( X B − X A Y B − Y A ) \alpha_{AB} = \arctan\left(\frac{X_B - X_A}{Y_B - Y_A}\right) α A B ​ = arctan ( Y B ​ − Y A ​ X B ​ − X A ​ ​ ) d α A B = cos ⁡ ( α A B ) S A B δ X A − sin ⁡ ( α A B ) S A B δ Y A − cos ⁡ ( α A B ) S A B δ X B + sin ⁡ ( α A B ) S A B δ Y B d\alpha_{AB} = \frac{\cos(\alpha_{AB})}{S_{AB}} \, \delta X_A - \frac{\sin(\alpha_{AB})}{S_{AB}} \, \delta Y_A - \frac{\cos(\alpha_{AB})}{S_{AB}} \, \delta X_B + \frac{\sin(\alpha_{AB})}{S_{AB}} \, \delta Y_B d α A B ​ = S A B ​ cos ( α A B ​ ) ​ δ X A ​ − S A B ​ sin ( α A B ​ ) ​ δ Y A ​ − S A B ​ cos ( α A B ​ ) ​ δ X B ​ + S A B ​ sin ( α A B ​ ) ​ δ Y B ​

The adjustment is iterated until parameter update vector ∥ X ∥ < ϵ \|X\| < \epsilon ∥ X ∥ < ϵ (e.g. < 0.1 mm < 0.1\text{ mm} < 0.1 mm ).

5. Plane Traverse Adjustment: Least Squares vs. Compass Rule

For decades, boundary surveys in the Philippines were adjusted using the empirical Compass (Bowditch) Rule . However, modern regulations under DENR Administrative Orders specify Least Squares for control networks.

Comparison Parameter Compass (Bowditch) Rule Least-Squares Adjustment

Theoretical Rigor Empirical rule; assumes distance & angle errors equal Mathematically rigorous; minimizes V T P V V^T P V V T P V

Observation Weighting Ignores individual instrument accuracies Incorporates individual distance/angle standard errors

Geometry Handling Limited to simple closed loop traverses Handles complex interconnected networks, cross-ties, and GNSS

Error Estimation Provides no statistical variance for coordinates Computes full covariance matrix Σ X Y \Sigma_{XY} Σ X Y ​ and confidence ellipses

  • 6. Point Error Ellipses and Residual Analysis
  • 6.1 Point Error Ellipses (Confidence Ellipses)

For a 2D station i i i , the coordinate covariance matrix is extracted from Σ X \Sigma_X Σ X ​ : Σ X Y = [ σ x 2 σ x y σ x y σ y 2 ] \Sigma_{XY} = \begin{bmatrix} \sigma_x^2 & \sigma_{xy} \\ \sigma_{xy} & \sigma_y^2 \end{bmatrix} Σ X Y ​ = [ σ x 2 ​ σ x y ​ ​ σ x y ​ σ y 2 ​ ​ ]

The geometric parameters of the standard ( 39.4 % 39.4\% 39.4% 2D probability) error ellipse are:

Orientation angle of semi-major axis ( θ \theta θ ): tan ⁡ ( 2 θ ) = 2 σ x y σ x 2 − σ y 2 \tan(2\theta) = \frac{2 \sigma_{xy}}{\sigma_x^2 - \sigma_y^2} tan ( 2 θ ) = σ x 2 ​ − σ y 2 ​ 2 σ x y ​ ​

Semi-major axis ( a a a ) and Semi-minor axis ( b b b ): a = 1 2 ( σ x 2 + σ y 2 + ( σ x 2 − σ y 2 ) 2 + 4 σ x y 2 ) a = \sqrt{ \frac{1}{2} \left( \sigma_x^2 + \sigma_y^2 + \sqrt{(\sigma_x^2 - \sigma_y^2)^2 + 4 \sigma_{xy}^2} \right) } a = 2 1 ​ ( σ x 2 ​ + σ y 2 ​ + ( σ x 2 ​ − σ y 2 ​ ) 2 + 4 σ x y 2 ​ ​ ) ​ b = 1 2 ( σ x 2 + σ y 2 − ( σ x 2 − σ y 2 ) 2 + 4 σ x y 2 ) b = \sqrt{ \frac{1}{2} \left( \sigma_x^2 + \sigma_y^2 - \sqrt{(\sigma_x^2 - \sigma_y^2)^2 + 4 \sigma_{xy}^2} \right) } b = 2 1 ​ ( σ x 2 ​ + σ y 2 ​ − ( σ x 2 ​ − σ y 2 ​ ) 2 + 4 σ x y 2 ​ ​ ) ​

To scale to a 95 % 95\% 95% confidence ellipse, multiply a a a and b b b by K = χ 2 , 0.05 2 = 5.991 ≈ 2.4477 K = \sqrt{\chi_{2, 0.05}^2} = \sqrt{5.991} \approx 2.4477 K = χ 2 , 0.05 2 ​ ​ = 5.991 ​ ≈ 2.4477 . Y (North)

  • ^ / Semi-Major Axis (a)
  • | /

| /---.

| .' | `.

  • | / | \
  • | | +-----|---> X (East)
  • | \\ / /
  • | `. \\ .'
  • | `---' \\ Semi-Minor Axis (b)
  • | \\ Angle theta
  • +----------------------->
  • 6.2 Residual Analysis and Outlier Detection

To detect blunders in large observations sets, Standardized Residuals ( v i ∗ v_i^* v i ∗ ​ ) are evaluated: v i ∗ = v i σ v i = v i σ ^ 0 ( P − 1 − A ( A T P A ) − 1 A T ) i i v_i^* = \frac{v_i}{\sigma_{v_i}} = \frac{v_i}{\hat{\sigma}_0 \sqrt{ (P^{-1} - A (A^T P A)^{-1} A^T)_{ii} }} v i ∗ ​ = σ v i ​ ​ v i ​ ​ = σ ^ 0 ​ ( P − 1 − A ( A T P A ) − 1 A T ) ii ​ ​ v i ​ ​

If ∣ v i ∗ ∣ > 3.0 |v_i^*| > 3.0 ∣ v i ∗ ​ ∣ > 3.0 , the observation is flagged as a statistical outlier (blunder) at the 99.7 % 99.7\% 99.7% confidence level and subjected to rejection or re-measurement.