Study guide

10.3 Network Troubleshooting Scenarios & End-to-End Connectivity

Designing and Implementing Microsoft Azure Networking Solutions AZ-700

Track your progress as you read

Finished sections get a check mark.

1 Introduction & AZ-700 Exam Overview

0.1 About the AZ-700 Exam: Format, Scoring & Skills Measured

2 Chapter 1: Azure Virtual Networks, Addressing & Subnet Architecture

1.1 VNet Planning, IP Addressing & Subnet Design 1.2 Virtual Network Peering & Global Peering 1.3 Azure NAT Gateway & Outbound Connectivity 1.4 Azure Virtual Network Manager (AVNM) & Network Groups 1.5 Public IP Addresses, Public IP Prefixes & Custom IP Prefixes (BYOIP)

3 Chapter 2: Azure Name Resolution & Core Routing

2.1 Azure DNS & Private DNS Zones 2.2 Azure Private DNS Resolver & Hybrid Name Resolution 2.3 System Routes & User-Defined Routes (UDR) 2.4 BGP & Custom Route Propagation 2.5 Azure Route Server & Dynamic NVA Route Exchange

4 Chapter 3: Hybrid Connectivity with Azure VPN Gateway

3.1 Azure VPN Gateway Architecture & SKUs 3.2 Site-to-Site (S2S) VPN & High Availability 3.3 Point-to-Site (P2S) VPN & Client Authentication 3.4 Azure Extended Network, Always On VPN & Azure Network Adapter

5 Chapter 4: Azure ExpressRoute & Virtual WAN

4.1 Azure ExpressRoute Architecture & Peering Models 4.2 ExpressRoute Routing, Resiliency & Global Reach 4.3 Azure Virtual WAN Architecture & Hub Routing 4.4 ExpressRoute BFD, Monitoring & Connection Troubleshooting

6 Chapter 5: Regional Load Balancing & Application Gateway

5.1 Azure Load Balancer Architecture & Rules 5.2 Azure Application Gateway Architecture & Routing 5.3 Web Application Firewall (WAF) on Application Gateway 5.4 Inbound NAT Rules, Port Forwarding & Health Probe Design

7 Chapter 6: Global Traffic Management & Azure Front Door

6.1 Azure Front Door Architecture & Routing 6.2 Azure Traffic Manager & DNS Load Balancing 6.3 Load Balancing Decision Matrix & Multi-Tier Design 6.4 Web Application Firewall on Azure Front Door

8 Chapter 7: Azure Private Link & Service Endpoints

7.1 Virtual Network Service Endpoints & Policies 7.2 Azure Private Endpoints & Private Link Architecture 7.3 Private Endpoint DNS Integration & Hybrid Resolution 7.4 Azure Private Link Service for Multi-Tenant Solutions

9 Chapter 8: Network Security Groups & Azure Bastion

8.1 Network Security Groups (NSGs) & Application Security Groups (ASGs) 8.2 Azure Bastion Architecture & Secure Remote Access 8.3 Azure Virtual Network Manager Security Admin Rules

10 Chapter 9: Azure Firewall & Security Policies

9.1 Azure Firewall Architecture & Rule Processing 9.2 Azure Firewall Manager & DDoS Protection

11 Chapter 10: Azure Network Observability & Troubleshooting

10.1 Network Watcher Diagnostic Tools 10.2 NSG Flow Logs, VNet Flow Logs & Traffic Analytics 10.3 Network Troubleshooting Scenarios & End-to-End Connectivity 10.4 Azure Monitor Network Insights & Defender for Cloud Network Posture

10.3 Network Troubleshooting Scenarios & End-to-End Connectivity

Key Takeaways Explain This Topic AI

Asymmetric routing occurs when outbound and return packets traverse different paths, causing stateful inspection firewalls to drop return packets; remediation requires Source NAT (SNAT) on the NVA or symmetric UDR routing.

Private Endpoint split-brain DNS failures occur when on-premises clients query public DNS resolvers or non-routable link-local IPs (168.63.129.16) instead of forwarding queries to an Azure Private DNS Resolver Inbound Endpoint.

Standard Load Balancer SNAT port exhaustion occurs when concurrent outbound connections exceed the pre-allocated port pool (default 1,024 ports per VM); deploying Azure NAT Gateway resolves this via fully dynamic port allocation of up to 64,000 ports per public IP.

BGP routing loops and AS-path prepending issues are diagnosed by inspecting effective routes and route limits; Azure requires unique ASNs outside the reserved range (65515-65520) and uses AS-path length as a primary path selection attribute.

VPN tunnel negotiation failures are diagnosed via IKE diagnostic logs; Phase 1 (Main Mode) failures typically stem from Pre-Shared Key (PSK) mismatches or crypto proposals, while Phase 2 (Quick Mode) failures stem from traffic selector mismatches.

Last updated: August 2026

Network Troubleshooting Scenarios & End-to-End Connectivity

Enterprise Azure networking environments combine hybrid connectivity (ExpressRoute, S2S VPN), hub-and-spoke topologies, stateful security appliances (Azure Firewall, NVAs), and private PaaS integrations (Private Endpoints). When connectivity fails in these multi-layered topologies, network engineers must execute systematic, structured root-cause analysis.

This section details five critical real-world troubleshooting scenarios frequently tested on the AZ-700 exam: asymmetric routing, Private Endpoint DNS resolution failure, SNAT port exhaustion, BGP route manipulation, and IPsec VPN negotiation failures.

1. Scenario 1: Asymmetric Routing Across Stateful Firewalls & Dual NVAs

Asymmetric routing occurs when an outbound network packet traverses one path (e.g., through a stateful firewall or NVA), but the corresponding return packet takes a different path (e.g., bypassing the firewall or hitting a different NVA instance in a scale set). +-----------------------------------------------------------------------------+

  • | ASYMMETRIC ROUTING BREAKDOWN & REMEDIATION |
  • | |
  • | FAILURE TOPOLOGY (ASYMMETRIC DROP): |
  • | 1. Spoke VM (10.1.0.4) sends TCP SYN to On-Premises (192.168.1.50). |
  • | 2. UDR on Spoke routes packet -> Azure Firewall / NVA (10.100.0.4). |
  • | 3. Firewall creates state table entry: [10.1.0.4:51234 -> 192.168.1.50:443]|
  • | 4. On-Premises server receives SYN and replies with TCP SYN-ACK. |
  • | 5. On-Premises gateway has a direct route or ExpressRoute path to Spoke |
  • | VNet (10.1.0.0/16) that BYPASSES the Azure Firewall. |
  • | 6. Spoke VM receives SYN-ACK from an unexpected path, or if return hits a|
  • | different NVA instance, the NVA drops the SYN-ACK (no state entry). |
  • | |
  • | REMEDIATION 1: Source NAT (SNAT) on NVA / Firewall |
  • | - NVA replaces Source IP (10.1.0.4) with NVA's own IP (10.100.0.4). |
  • | - On-Premises server MUST reply directly to NVA (10.100.0.4). |
  • | - Symmetric path is guaranteed across all connection states. |
  • | |
  • | REMEDIATION 2: Symmetric Route Tables (UDRs) & GatewaySubnet Routing |
  • | - Associate a UDR to 'GatewaySubnet' routing Spoke CIDRs -> NVA IP. |
  • | - Enforces bidirectional symmetry without requiring SNAT. |
  • +-----------------------------------------------------------------------------+

Diagnostic Workflow:

Inspect Effective Routes on both the source VM NIC and the GatewaySubnet .

Verify whether GatewaySubnet has a UDR directing return traffic back through the NVA.

Check NVA / Azure Firewall diagnostic logs ( AzureDiagnostics or AZFWNetworkRule ) for dropped packets marked with invalid TCP state or out-of-order ACK flags.

Loading diagram... Asymmetric Routing Drop vs GatewaySubnet UDR Symmetric Remediation

2. Scenario 2: Split-Brain DNS Failures for Private Endpoints

When organizations deploy Azure Private Endpoints for PaaS resources (e.g., Azure SQL Database, Azure Blob Storage), Azure automatically assigns a private IP within the virtual network. To ensure seamless access, Azure registers the resource in an Azure Private DNS Zone (e.g., privatelink.blob.core.windows.net ).

The Problem: Hybrid DNS Resolution Failure

On-premises clients connecting to an Azure Storage account ( mystorage.blob.core.windows.net ) resolve the public IP address instead of the Private Endpoint private IP ( 10.1.5.4 ), receiving an HTTP 403 Forbidden or connection timeout. +-----------------------------------------------------------------------------+

  • | HYBRID PRIVATE ENDPOINT DNS RESOLUTION |
  • | |
  • | [ On-Premises Client ] |
  • | | (1) Queries: mystorage.blob.core.windows.net |
  • | v |
  • | [ On-Premises DNS Server (Active Directory / BIND) ] |
  • | | |
  • | | (2) Conditional Forwarder: 'blob.core.windows.net' -> |
  • | | Targets Inbound Endpoint IP (10.100.0.132) |
  • | v |
  • | [ Azure Private DNS Resolver: Inbound Endpoint (10.100.0.132) ] |
  • | | |
  • | | (3) Queries linked Private DNS Zone: |
  • | | privatelink.blob.core.windows.net |
  • | v |
  • | [ Azure Private DNS Zone ] |
  • | - Record: mystorage.privatelink.blob.core.windows.net -> 10.1.5.4 |
  • | | |
  • | | (4) Returns Private IP: 10.1.5.4 |
  • | v |
  • | [ Client Connects Directly to 10.1.5.4 via ExpressRoute / S2S VPN ] |
  • +-----------------------------------------------------------------------------+

Diagnostic Checklist for Private Endpoint DNS:

Do NOT Target 168.63.129.16 from On-Premises: The Azure recursive resolver IP ( 168.63.129.16 ) is a virtual non-routable link-local IP accessible only from within Azure virtual networks . On-premises DNS servers cannot forward queries directly to 168.63.129.16 .

Deploy Azure Private DNS Resolver: Provision an Inbound Endpoint in a dedicated subnet inside the hub VNet. Configure on-premises DNS conditional forwarders to forward queries for blob.core.windows.net (or privatelink.blob.core.windows.net ) to the Inbound Endpoint private IP.

Virtual Network Links: Verify that the Azure Private DNS Zone has an active Virtual Network Link associated with the Hub VNet and all relevant Spoke VNets.

3. Scenario 3: SNAT Port Exhaustion on Load Balancers & NAT Gateways

When virtual machines inside Azure initiate outbound connections to external public endpoints (APIs, third-party services, package repositories), Azure performs Source Network Address Translation (SNAT) to map the VM's private IP to a public IP address.

The Failure Mechanism: Pre-Allocated SNAT Starvation

Standard Load Balancer Default Allocation: When outbound rules are configured on a Standard Load Balancer without custom allocation, Azure allocates a fixed, static number of SNAT ports (typically 1,024 ports per VM instance ) from each assigned public IP.

Exhaustion Condition: If an application opens hundreds of concurrent outbound connections in rapid succession, it exhausts its 1,024-port quota. Additional outbound connection attempts are dropped, resulting in intermittent connection timeouts and high latency. +-----------------------------------------------------------------------------+

  • | SNAT PORT CAPACITY & ALLOCATION MATRIX |
  • | |
  • | Architectural Feature Standard Load Balancer Azure NAT Gateway |
  • | ------------------------ ---------------------- ----------------- |
  • | SNAT Allocation Model Static (Pre-allocated) Dynamic (On-Demand) |
  • | Ports Per Public IP 64,000 64,000 |
  • | Concurrency per Subnet VM Fixed (e.g., 1,024 max) Dynamic (Up to 64,000)|
  • | Max Public IPs Attached Configurable Up to 16 IPs (1M+ pts)|
  • | Availability Zone Resilient Yes Yes (Zone-Redundant) |
  • | Subnet Association Model NIC Backend Pool Direct Subnet Binding |
  • +-----------------------------------------------------------------------------+

Diagnostic Metrics & Remediation:

Monitor Azure Metrics:

Used SNAT Ports and Allocated SNAT Ports (Load Balancer / NAT Gateway).

Failed SNAT Connections (Load Balancer) / Dropped Packets (NAT Gateway).

A spike in Failed SNAT Connections while Used SNAT Ports approaches Allocated SNAT Ports confirms SNAT port exhaustion.

Best Practice Remediation: Deploy an Azure NAT Gateway on the subnet. NAT Gateway provides fully dynamic, on-demand SNAT port allocation across up to 16 Public IPs (yielding up to 1,024,000 concurrent flows), eliminating fixed per-VM port starvation.

4. Scenario 4: BGP Routing Loops & AS-Path Prepending Issues

In hybrid architectures utilizing BGP over ExpressRoute and Site-to-Site VPN, incorrect BGP attribute propagation can lead to routing loops, suboptimal asymmetric routing, or complete route withdrawal. +-----------------------------------------------------------------------------+

  • | BGP ROUTE SELECTION & AS-PATH PREPENDING |
  • | |
  • | AZURE BGP BEST PATH SELECTION CRITERIA (IN ORDER): |
  • | 1. Longest Prefix Match (LPM) — e.g., /24 preferred over /16 |
  • | 2. User-Defined Routes (UDR) — Overrides BGP routes |
  • | 3. BGP Local Preference (if configured) |
  • | 4. Shortest AS-Path Length (Evaluates number of ASNs in AS_PATH) |
  • | 5. Origin Type (IGP preferred over EGP, EGP preferred over Incomplete) |
  • | 6. Lowest Multi-Exit Discriminator (MED) |
  • | |
  • | ACTIVE / PASSIVE FAILOVER VIA AS-PATH PREPENDING: |
  • | - Primary Link (ExpressRoute): Advertises ASN 65001 (AS-Path Length = 1) |
  • | - Backup Link (S2S VPN): Advertises ASN 65001 65001 65001 (Length = 3) |
  • | - Azure Result: Selects ExpressRoute because AS-Path is shorter. |
  • | - Failover: If ExpressRoute drops, Azure automatically routes via VPN. |
  • +-----------------------------------------------------------------------------+

Common BGP Pitfalls for AZ-700:

Reserved ASNs: Azure reserves ASNs 65515 through 65520 for internal gateway infrastructure (VPN Gateway, ExpressRoute Gateway, Azure Route Server). Attempting to configure an on-premises BGP peer with ASN 65515 causes peering session establishment failure.

Route Leaking: If an on-premises edge router or Azure NVA re-advertises routes learned from Azure back into Azure over a second circuit, a routing loop occurs. Enforce strict BGP route filtering on customer premises equipment (CPE).

ExpressRoute Route Limits: Standard SKU ExpressRoute circuits support a maximum of 4,000 routes ; Premium SKU supports up to 10,000 routes . If on-premises edge routers advertise prefixes exceeding this quota, the BGP session is instantly dropped by Microsoft Enterprise Edge (MSEE) routers.

5. Scenario 5: VPN Gateway Tunnel Negotiation Failures

Site-to-Site (S2S) VPN connectivity issues between an Azure Virtual Network Gateway and an on-premises firewall/router typically occur during cryptographic handshake negotiation. +-----------------------------------------------------------------------------+

  • | IPSEC / IKE NEGOTIATION PHASES |
  • | |
  • | PHASE 1: IKE SA (Main Mode) |
  • | - Purpose: Authenticates peers and creates secure management channel. |
  • | - Negotiated Parameters: |
  • | * Pre-Shared Key (PSK) — MUST MATCH EXACTLY |
  • | * IKE Version (IKEv1 vs IKEv2) |
  • | * Encryption Algorithm: AES256 / AES128 |
  • | * Integrity / Hash: SHA256 / SHA1 |
  • | * Diffie-Hellman (DH) Group: Group 14 (2048-bit) / Group 24 |
  • | * IKE SA Lifetime: Default 28,800 seconds |
  • | - #1 Failure Cause: Pre-Shared Key (PSK) typo or crypto proposal mismatch|
  • | |
  • | PHASE 2: Child IPsec SA (Quick Mode) |
  • | - Purpose: Establishes encrypted data-plane tunnel for customer payload. |
  • | - Negotiated Parameters: |
  • | * IPsec Encryption: AES256 / AES-GCM |
  • | * IPsec Integrity: SHA256 |
  • | * Perfect Forward Secrecy (PFS) Group: PFS2048 / None |
  • | * IPsec SA Lifetime: Default 27,000 seconds / 7,200 seconds |
  • | * Traffic Selectors (Local & Remote Address Prefixes) |
  • | - #1 Failure Cause: Traffic Selector / Policy-Based mismatch |
  • +-----------------------------------------------------------------------------+

Diagnostic Steps & Azure Logs:

Check Connection Status:

Run Get-AzVirtualNetworkGatewayConnection or az network vpn-connection show .

Status Connecting indicates Phase 1 or Phase 2 failure; NotConnected indicates unreachable peer IP or platform shutdown.

Enable VPN Diagnostic Logs in Log Analytics:

Ingest IKEDiagnosticLog , RouteDiagnosticLog , and TunnelDiagnosticLog .

Query IKEDiagnosticLog for INVALID_PAYLOAD_TYPE (incorrect PSK), NO_PROPOSAL_CHOSEN (crypto suite mismatch), or TS_UNACCEPTABLE (traffic selector mismatch).

Policy-Based vs. Route-Based Traffic Selectors:

Azure Route-Based gateways negotiate wildcards ( 0.0.0.0/0 <-> 0.0.0.0/0 ).

If connecting to a legacy Policy-Based device expecting specific subnet pairs (e.g., 10.1.0.0/24 <-> 192.168.1.0/24 ), enable UsePolicyBasedTrafficSelectors: $true on the Azure Connection resource.