Study guide
10.3 Network Troubleshooting Scenarios & End-to-End Connectivity
Designing and Implementing Microsoft Azure Networking Solutions AZ-700
Track your progress as you read
Finished sections get a check mark.
1 Introduction & AZ-700 Exam Overview
0.1 About the AZ-700 Exam: Format, Scoring & Skills Measured
2 Chapter 1: Azure Virtual Networks, Addressing & Subnet Architecture
1.1 VNet Planning, IP Addressing & Subnet Design 1.2 Virtual Network Peering & Global Peering 1.3 Azure NAT Gateway & Outbound Connectivity 1.4 Azure Virtual Network Manager (AVNM) & Network Groups 1.5 Public IP Addresses, Public IP Prefixes & Custom IP Prefixes (BYOIP)
3 Chapter 2: Azure Name Resolution & Core Routing
2.1 Azure DNS & Private DNS Zones 2.2 Azure Private DNS Resolver & Hybrid Name Resolution 2.3 System Routes & User-Defined Routes (UDR) 2.4 BGP & Custom Route Propagation 2.5 Azure Route Server & Dynamic NVA Route Exchange
4 Chapter 3: Hybrid Connectivity with Azure VPN Gateway
3.1 Azure VPN Gateway Architecture & SKUs 3.2 Site-to-Site (S2S) VPN & High Availability 3.3 Point-to-Site (P2S) VPN & Client Authentication 3.4 Azure Extended Network, Always On VPN & Azure Network Adapter
5 Chapter 4: Azure ExpressRoute & Virtual WAN
4.1 Azure ExpressRoute Architecture & Peering Models 4.2 ExpressRoute Routing, Resiliency & Global Reach 4.3 Azure Virtual WAN Architecture & Hub Routing 4.4 ExpressRoute BFD, Monitoring & Connection Troubleshooting
6 Chapter 5: Regional Load Balancing & Application Gateway
5.1 Azure Load Balancer Architecture & Rules 5.2 Azure Application Gateway Architecture & Routing 5.3 Web Application Firewall (WAF) on Application Gateway 5.4 Inbound NAT Rules, Port Forwarding & Health Probe Design
7 Chapter 6: Global Traffic Management & Azure Front Door
6.1 Azure Front Door Architecture & Routing 6.2 Azure Traffic Manager & DNS Load Balancing 6.3 Load Balancing Decision Matrix & Multi-Tier Design 6.4 Web Application Firewall on Azure Front Door
8 Chapter 7: Azure Private Link & Service Endpoints
7.1 Virtual Network Service Endpoints & Policies 7.2 Azure Private Endpoints & Private Link Architecture 7.3 Private Endpoint DNS Integration & Hybrid Resolution 7.4 Azure Private Link Service for Multi-Tenant Solutions
9 Chapter 8: Network Security Groups & Azure Bastion
8.1 Network Security Groups (NSGs) & Application Security Groups (ASGs) 8.2 Azure Bastion Architecture & Secure Remote Access 8.3 Azure Virtual Network Manager Security Admin Rules
10 Chapter 9: Azure Firewall & Security Policies
9.1 Azure Firewall Architecture & Rule Processing 9.2 Azure Firewall Manager & DDoS Protection
11 Chapter 10: Azure Network Observability & Troubleshooting
10.1 Network Watcher Diagnostic Tools 10.2 NSG Flow Logs, VNet Flow Logs & Traffic Analytics 10.3 Network Troubleshooting Scenarios & End-to-End Connectivity 10.4 Azure Monitor Network Insights & Defender for Cloud Network Posture
10.3 Network Troubleshooting Scenarios & End-to-End Connectivity
Key Takeaways Explain This Topic AI
Asymmetric routing occurs when outbound and return packets traverse different paths, causing stateful inspection firewalls to drop return packets; remediation requires Source NAT (SNAT) on the NVA or symmetric UDR routing.
Private Endpoint split-brain DNS failures occur when on-premises clients query public DNS resolvers or non-routable link-local IPs (168.63.129.16) instead of forwarding queries to an Azure Private DNS Resolver Inbound Endpoint.
Standard Load Balancer SNAT port exhaustion occurs when concurrent outbound connections exceed the pre-allocated port pool (default 1,024 ports per VM); deploying Azure NAT Gateway resolves this via fully dynamic port allocation of up to 64,000 ports per public IP.
BGP routing loops and AS-path prepending issues are diagnosed by inspecting effective routes and route limits; Azure requires unique ASNs outside the reserved range (65515-65520) and uses AS-path length as a primary path selection attribute.
VPN tunnel negotiation failures are diagnosed via IKE diagnostic logs; Phase 1 (Main Mode) failures typically stem from Pre-Shared Key (PSK) mismatches or crypto proposals, while Phase 2 (Quick Mode) failures stem from traffic selector mismatches.
Last updated: August 2026
Network Troubleshooting Scenarios & End-to-End Connectivity
Enterprise Azure networking environments combine hybrid connectivity (ExpressRoute, S2S VPN), hub-and-spoke topologies, stateful security appliances (Azure Firewall, NVAs), and private PaaS integrations (Private Endpoints). When connectivity fails in these multi-layered topologies, network engineers must execute systematic, structured root-cause analysis.
This section details five critical real-world troubleshooting scenarios frequently tested on the AZ-700 exam: asymmetric routing, Private Endpoint DNS resolution failure, SNAT port exhaustion, BGP route manipulation, and IPsec VPN negotiation failures.
1. Scenario 1: Asymmetric Routing Across Stateful Firewalls & Dual NVAs
Asymmetric routing occurs when an outbound network packet traverses one path (e.g., through a stateful firewall or NVA), but the corresponding return packet takes a different path (e.g., bypassing the firewall or hitting a different NVA instance in a scale set). +-----------------------------------------------------------------------------+
- | ASYMMETRIC ROUTING BREAKDOWN & REMEDIATION |
- | |
- | FAILURE TOPOLOGY (ASYMMETRIC DROP): |
- | 1. Spoke VM (10.1.0.4) sends TCP SYN to On-Premises (192.168.1.50). |
- | 2. UDR on Spoke routes packet -> Azure Firewall / NVA (10.100.0.4). |
- | 3. Firewall creates state table entry: [10.1.0.4:51234 -> 192.168.1.50:443]|
- | 4. On-Premises server receives SYN and replies with TCP SYN-ACK. |
- | 5. On-Premises gateway has a direct route or ExpressRoute path to Spoke |
- | VNet (10.1.0.0/16) that BYPASSES the Azure Firewall. |
- | 6. Spoke VM receives SYN-ACK from an unexpected path, or if return hits a|
- | different NVA instance, the NVA drops the SYN-ACK (no state entry). |
- | |
- | REMEDIATION 1: Source NAT (SNAT) on NVA / Firewall |
- | - NVA replaces Source IP (10.1.0.4) with NVA's own IP (10.100.0.4). |
- | - On-Premises server MUST reply directly to NVA (10.100.0.4). |
- | - Symmetric path is guaranteed across all connection states. |
- | |
- | REMEDIATION 2: Symmetric Route Tables (UDRs) & GatewaySubnet Routing |
- | - Associate a UDR to 'GatewaySubnet' routing Spoke CIDRs -> NVA IP. |
- | - Enforces bidirectional symmetry without requiring SNAT. |
- +-----------------------------------------------------------------------------+
Diagnostic Workflow:
Inspect Effective Routes on both the source VM NIC and the GatewaySubnet .
Verify whether GatewaySubnet has a UDR directing return traffic back through the NVA.
Check NVA / Azure Firewall diagnostic logs ( AzureDiagnostics or AZFWNetworkRule ) for dropped packets marked with invalid TCP state or out-of-order ACK flags.
Loading diagram... Asymmetric Routing Drop vs GatewaySubnet UDR Symmetric Remediation
2. Scenario 2: Split-Brain DNS Failures for Private Endpoints
When organizations deploy Azure Private Endpoints for PaaS resources (e.g., Azure SQL Database, Azure Blob Storage), Azure automatically assigns a private IP within the virtual network. To ensure seamless access, Azure registers the resource in an Azure Private DNS Zone (e.g., privatelink.blob.core.windows.net ).
The Problem: Hybrid DNS Resolution Failure
On-premises clients connecting to an Azure Storage account ( mystorage.blob.core.windows.net ) resolve the public IP address instead of the Private Endpoint private IP ( 10.1.5.4 ), receiving an HTTP 403 Forbidden or connection timeout. +-----------------------------------------------------------------------------+
- | HYBRID PRIVATE ENDPOINT DNS RESOLUTION |
- | |
- | [ On-Premises Client ] |
- | | (1) Queries: mystorage.blob.core.windows.net |
- | v |
- | [ On-Premises DNS Server (Active Directory / BIND) ] |
- | | |
- | | (2) Conditional Forwarder: 'blob.core.windows.net' -> |
- | | Targets Inbound Endpoint IP (10.100.0.132) |
- | v |
- | [ Azure Private DNS Resolver: Inbound Endpoint (10.100.0.132) ] |
- | | |
- | | (3) Queries linked Private DNS Zone: |
- | | privatelink.blob.core.windows.net |
- | v |
- | [ Azure Private DNS Zone ] |
- | - Record: mystorage.privatelink.blob.core.windows.net -> 10.1.5.4 |
- | | |
- | | (4) Returns Private IP: 10.1.5.4 |
- | v |
- | [ Client Connects Directly to 10.1.5.4 via ExpressRoute / S2S VPN ] |
- +-----------------------------------------------------------------------------+
Diagnostic Checklist for Private Endpoint DNS:
Do NOT Target 168.63.129.16 from On-Premises: The Azure recursive resolver IP ( 168.63.129.16 ) is a virtual non-routable link-local IP accessible only from within Azure virtual networks . On-premises DNS servers cannot forward queries directly to 168.63.129.16 .
Deploy Azure Private DNS Resolver: Provision an Inbound Endpoint in a dedicated subnet inside the hub VNet. Configure on-premises DNS conditional forwarders to forward queries for blob.core.windows.net (or privatelink.blob.core.windows.net ) to the Inbound Endpoint private IP.
Virtual Network Links: Verify that the Azure Private DNS Zone has an active Virtual Network Link associated with the Hub VNet and all relevant Spoke VNets.
3. Scenario 3: SNAT Port Exhaustion on Load Balancers & NAT Gateways
When virtual machines inside Azure initiate outbound connections to external public endpoints (APIs, third-party services, package repositories), Azure performs Source Network Address Translation (SNAT) to map the VM's private IP to a public IP address.
The Failure Mechanism: Pre-Allocated SNAT Starvation
Standard Load Balancer Default Allocation: When outbound rules are configured on a Standard Load Balancer without custom allocation, Azure allocates a fixed, static number of SNAT ports (typically 1,024 ports per VM instance ) from each assigned public IP.
Exhaustion Condition: If an application opens hundreds of concurrent outbound connections in rapid succession, it exhausts its 1,024-port quota. Additional outbound connection attempts are dropped, resulting in intermittent connection timeouts and high latency. +-----------------------------------------------------------------------------+
- | SNAT PORT CAPACITY & ALLOCATION MATRIX |
- | |
- | Architectural Feature Standard Load Balancer Azure NAT Gateway |
- | ------------------------ ---------------------- ----------------- |
- | SNAT Allocation Model Static (Pre-allocated) Dynamic (On-Demand) |
- | Ports Per Public IP 64,000 64,000 |
- | Concurrency per Subnet VM Fixed (e.g., 1,024 max) Dynamic (Up to 64,000)|
- | Max Public IPs Attached Configurable Up to 16 IPs (1M+ pts)|
- | Availability Zone Resilient Yes Yes (Zone-Redundant) |
- | Subnet Association Model NIC Backend Pool Direct Subnet Binding |
- +-----------------------------------------------------------------------------+
Diagnostic Metrics & Remediation:
Monitor Azure Metrics:
Used SNAT Ports and Allocated SNAT Ports (Load Balancer / NAT Gateway).
Failed SNAT Connections (Load Balancer) / Dropped Packets (NAT Gateway).
A spike in Failed SNAT Connections while Used SNAT Ports approaches Allocated SNAT Ports confirms SNAT port exhaustion.
Best Practice Remediation: Deploy an Azure NAT Gateway on the subnet. NAT Gateway provides fully dynamic, on-demand SNAT port allocation across up to 16 Public IPs (yielding up to 1,024,000 concurrent flows), eliminating fixed per-VM port starvation.
4. Scenario 4: BGP Routing Loops & AS-Path Prepending Issues
In hybrid architectures utilizing BGP over ExpressRoute and Site-to-Site VPN, incorrect BGP attribute propagation can lead to routing loops, suboptimal asymmetric routing, or complete route withdrawal. +-----------------------------------------------------------------------------+
- | BGP ROUTE SELECTION & AS-PATH PREPENDING |
- | |
- | AZURE BGP BEST PATH SELECTION CRITERIA (IN ORDER): |
- | 1. Longest Prefix Match (LPM) — e.g., /24 preferred over /16 |
- | 2. User-Defined Routes (UDR) — Overrides BGP routes |
- | 3. BGP Local Preference (if configured) |
- | 4. Shortest AS-Path Length (Evaluates number of ASNs in AS_PATH) |
- | 5. Origin Type (IGP preferred over EGP, EGP preferred over Incomplete) |
- | 6. Lowest Multi-Exit Discriminator (MED) |
- | |
- | ACTIVE / PASSIVE FAILOVER VIA AS-PATH PREPENDING: |
- | - Primary Link (ExpressRoute): Advertises ASN 65001 (AS-Path Length = 1) |
- | - Backup Link (S2S VPN): Advertises ASN 65001 65001 65001 (Length = 3) |
- | - Azure Result: Selects ExpressRoute because AS-Path is shorter. |
- | - Failover: If ExpressRoute drops, Azure automatically routes via VPN. |
- +-----------------------------------------------------------------------------+
Common BGP Pitfalls for AZ-700:
Reserved ASNs: Azure reserves ASNs 65515 through 65520 for internal gateway infrastructure (VPN Gateway, ExpressRoute Gateway, Azure Route Server). Attempting to configure an on-premises BGP peer with ASN 65515 causes peering session establishment failure.
Route Leaking: If an on-premises edge router or Azure NVA re-advertises routes learned from Azure back into Azure over a second circuit, a routing loop occurs. Enforce strict BGP route filtering on customer premises equipment (CPE).
ExpressRoute Route Limits: Standard SKU ExpressRoute circuits support a maximum of 4,000 routes ; Premium SKU supports up to 10,000 routes . If on-premises edge routers advertise prefixes exceeding this quota, the BGP session is instantly dropped by Microsoft Enterprise Edge (MSEE) routers.
5. Scenario 5: VPN Gateway Tunnel Negotiation Failures
Site-to-Site (S2S) VPN connectivity issues between an Azure Virtual Network Gateway and an on-premises firewall/router typically occur during cryptographic handshake negotiation. +-----------------------------------------------------------------------------+
- | IPSEC / IKE NEGOTIATION PHASES |
- | |
- | PHASE 1: IKE SA (Main Mode) |
- | - Purpose: Authenticates peers and creates secure management channel. |
- | - Negotiated Parameters: |
- | * Pre-Shared Key (PSK) — MUST MATCH EXACTLY |
- | * IKE Version (IKEv1 vs IKEv2) |
- | * Encryption Algorithm: AES256 / AES128 |
- | * Integrity / Hash: SHA256 / SHA1 |
- | * Diffie-Hellman (DH) Group: Group 14 (2048-bit) / Group 24 |
- | * IKE SA Lifetime: Default 28,800 seconds |
- | - #1 Failure Cause: Pre-Shared Key (PSK) typo or crypto proposal mismatch|
- | |
- | PHASE 2: Child IPsec SA (Quick Mode) |
- | - Purpose: Establishes encrypted data-plane tunnel for customer payload. |
- | - Negotiated Parameters: |
- | * IPsec Encryption: AES256 / AES-GCM |
- | * IPsec Integrity: SHA256 |
- | * Perfect Forward Secrecy (PFS) Group: PFS2048 / None |
- | * IPsec SA Lifetime: Default 27,000 seconds / 7,200 seconds |
- | * Traffic Selectors (Local & Remote Address Prefixes) |
- | - #1 Failure Cause: Traffic Selector / Policy-Based mismatch |
- +-----------------------------------------------------------------------------+
Diagnostic Steps & Azure Logs:
Check Connection Status:
Run Get-AzVirtualNetworkGatewayConnection or az network vpn-connection show .
Status Connecting indicates Phase 1 or Phase 2 failure; NotConnected indicates unreachable peer IP or platform shutdown.
Enable VPN Diagnostic Logs in Log Analytics:
Ingest IKEDiagnosticLog , RouteDiagnosticLog , and TunnelDiagnosticLog .
Query IKEDiagnosticLog for INVALID_PAYLOAD_TYPE (incorrect PSK), NO_PROPOSAL_CHOSEN (crypto suite mismatch), or TS_UNACCEPTABLE (traffic selector mismatch).
Policy-Based vs. Route-Based Traffic Selectors:
Azure Route-Based gateways negotiate wildcards ( 0.0.0.0/0 <-> 0.0.0.0/0 ).
If connecting to a legacy Policy-Based device expecting specific subnet pairs (e.g., 10.1.0.0/24 <-> 192.168.1.0/24 ), enable UsePolicyBasedTrafficSelectors: $true on the Azure Connection resource.