Study guide
10.3 High Availability Troubleshooting, System Maintenance & Log Forwarding
Palo Alto Networks Certified Network Security Professional Exam
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: PAN-OS Architecture, Interfaces & Routing
1.1 Security Zones, Virtual Routers & Interface Types 1.2 Advanced Routing Engine, Logical Routers & Dynamic Routing 1.3 High Availability Architecture, Links & Failover Mechanics
2 Chapter 2: PAN-OS Core Policy, App-ID & NAT
2.1 App-ID Classification Engine & Application Management 2.2 User-ID Architecture, Identity Mapping & Group Enforcement 2.3 Security Policy Enforcement & NAT Architecture
3 Chapter 3: SSL/TLS Decryption Architecture & Policy
3.1 SSL Forward Proxy Architecture & Enterprise PKI 3.2 SSL Inbound Inspection, Server Certificates & TLS 1.3 3.3 Decryption Policy, Decryption Profiles & Decryption Broker 3.4 SSH Proxy, No-Decrypt Policy & Decryption Exclusions
4 Chapter 4: Cloud-Delivered Security Services (CDSS) & Threat Prevention
4.1 Advanced Threat Prevention & Security Profiles 4.2 Advanced WildFire & Inline Machine Learning Models 4.3 Advanced URL Filtering & DNS Security 4.4 Device Security (IoT), Device-ID & IoMT Visibility 4.5 Enterprise SaaS Security: Discovered Apps, Data Security & SSPM
5 Chapter 5: Prisma Access Infrastructure & Network Onboarding
5.1 Prisma Access Cloud Architecture & Multi-Tenant Fabric 5.2 Remote Networks (RN) Onboarding & Branch Routing 5.3 Mobile Users (MU) Architecture, Locations & Dedicated IPs
6 Chapter 6: ZTNA 2.0, Cloud Identity & Digital Experience
6.1 ZTNA Connector Architecture & Private Application Publishing 6.2 Cloud Identity Engine (CIE) & Modern Authentication 6.3 Autonomous Digital Experience Management (ADEM) & Enterprise DLP
7 Chapter 7: Strata Cloud Manager (SCM) & AI-Driven Management
7.1 Strata Cloud Manager (SCM) Architecture & Cloud Control Plane 7.2 Unified Policy Hierarchy, Folders & Snippets 7.3 Configuration Push Preview, Validation & Migration Pathways 7.4 Strata Copilot & AIOps for NGFW
8 Chapter 8: Prisma SD-WAN & Branch Edge Architecture
8.1 Prisma SD-WAN ION Devices & Branch Topologies 8.2 Application-Defined Fabric (AppFabric) & Path Policies 8.3 Prisma SD-WAN & Prisma Access CloudBlade Integration 8.4 PAN-OS SD-WAN Subscription, Link Tags & Path Quality
9 Chapter 9: GlobalProtect, Cloud NGFW & AI Runtime Security
9.1 GlobalProtect Remote Access Architecture & Gateways 9.2 Host Information Profile (HIP) & Dynamic Split Tunneling 9.3 Cloud NGFW & Virtual Firewall Architecture (AWS, Azure & Containers) 9.4 AI Runtime Security & Generative AI Protection 9.5 Explicit Proxy, Prisma Access Browser & Remote Browser Isolation
10 Chapter 10: Packet Processing Architecture & System Troubleshooting
10.1 PAN-OS Single-Pass Parallel Processing (SP3) Architecture & Packet Flow 10.2 Operational Troubleshooting CLI Commands & Packet Diagnostics 10.3 High Availability Troubleshooting, System Maintenance & Log Forwarding
11 Chapter 11: Panorama, Subscriptions, Digital Trust & Quantum Readiness
11.1 Panorama Architecture, Device Groups & Template Stacks 11.2 CDSS Subscriptions, Premium GlobalProtect & License Lifecycle 11.3 Next-Generation Trust Security (NGTS) & Certificate Lifecycle 11.4 Quantum Security Risks & Post-Quantum Readiness
10.3 High Availability Troubleshooting, System Maintenance & Log Forwarding
Key Takeaways Explain This Topic AI
PAN-OS High Availability operational diagnostics rely on show high-availability state and show high-availability all to verify control link (HA1) heartbeat synchronization, configuration parity, and dataplane session mirroring (HA2).
A split-brain condition occurs when both HA1 and HA1-backup control links fail concurrently while data links remain up, causing both peers to transition to Active; recovery requires isolating secondary data interfaces immediately before restoring control link communication.
Zero-downtime PAN-OS software upgrades in Active/Passive clusters strictly require upgrading the passive peer first, verifying synchronization, initiating a controlled failover via request high-availability state suspend , and upgrading the remaining unit.
PAN-OS major and minor version upgrades enforce a strict base image requirement: the target family's base .0 release must reside on local disk before installing any maintenance release.
Dynamic content update management balances security posture against operational stability using threshold hold times and the 'Disable new apps in content update' safeguard to review App-ID modifications prior to enforcement.
Last updated: September 2026
- 10.3 High Availability Troubleshooting, System Maintenance & Log Forwarding
- High Availability Operational Troubleshooting
Palo Alto Networks High Availability (HA) ensures continuous enterprise network connectivity by pairing two firewalls in either an Active/Passive or Active/Active topology. Operational stability depends on the uninterrupted synchronization of control state, configuration hierarchies, and real-time session tables across dedicated physical clustering links.
High Availability Link Infrastructure
HA1 (Control Link): Exchanges heartbeats, hello packets, and configuration synchronization. Operates over TCP port 28769 and TCP port 28260 (for cleartext/SSL-secured communications). HA1 maintains cluster state and coordinates failover elections.
HA1-Backup: Provides an alternate, redundant path for HA1 control traffic. Typically mapped to the dedicated out-of-band Management ( MGT ) interface or an auxiliary physical port to prevent single-point-of-failure disconnects.
HA2 (Data Link): Synchronizes real-time stateful connection tables, including active session IDs, TCP sequence numbers, NAT translations, IPSec security associations, and ARP tables. Operates using UDP port 28770 or raw Ethernet (Ethertype 0x7261 ).
HA2-Backup: Redundant link for session table replication.
HA3 (Packet Forwarding Link): Exclusively utilized in Active/Active deployments. Uses a dedicated raw Layer 2 connection (Ethertype 0x7262 ) to forward asymmetric transit traffic between active peers without re-evaluating slow-path policy. +---------------------------+ +---------------------------+
- | Firewall-A (Peer 1) | | Firewall-B (Peer 2) |
- | |=== HA1 (Control Link) ==| |
- | - State: Active |--- HA1-Backup (MGT) ----| - State: Passive |
- | - Priority: 100 (Wins) |=== HA2 (Session Sync) ==| - Priority: 200 |
- | - Preemption: Enabled |--- HA2-Backup (Data) ---| - Preemption: Enabled |
- +---------------------------+ +---------------------------+
- | |
- +----------------== Data Paths ==---------------------+
- (Transit Traffic)
- Essential HA Diagnostic Commands # Display overall HA operational state, peer status, and link health
- admin@PA-3410> show high-availability state
- # Display exhaustive HA configuration, timer values, and failover metrics
- admin@PA-3410> show high-availability all
- # Verify active session synchronization between cluster peers
- admin@PA-3410> show high-availability state-synchronization
- # Inspect HA link interface statistics and packet error counters
- admin@PA-3410> show high-availability link-monitoring
- Testing Controlled Failover
Administrators validate cluster resilience using non-disruptive, software-triggered failover commands: # Suspend the local firewall, forcing all active traffic to failover to the peer
admin@PA-3410> request high-availability state suspend
- # Verify the peer transitions to Active and assumes forwarding duties
- admin@PA-3410> show high-availability state
- # Return the suspended firewall back to functional status
- admin@PA-3410> request high-availability state functional
- Split-Brain Failure Scenarios & Recovery Procedures
A split-brain condition occurs when both the primary HA1 link and secondary HA1-backup link fail simultaneously while the data plane interfaces remain physically UP. Because neither peer receives heartbeat keepalives, both firewalls assume the partner has suffered a fatal hardware failure. Consequently, both appliances transition to the Active state .
Operational Impact of Split-Brain
Both firewalls broadcast identical gratuitous ARPs, claiming ownership of identical virtual IP and virtual MAC addresses.
Upstream and downstream switches experience severe MAC address table flapping, causing asymmetric traffic distribution.
Packets belonging to the same TCP session arrive on different firewalls; because session state is un-synchronized, both firewalls drop packets due to out-of-order sequence violations.
Step-by-Step Split-Brain Recovery Procedure
Isolate Secondary Data Paths Immediately: Connect to the upstream and downstream access switches and administratively shut down the physical switch ports connecting to the secondary firewall. This stops duplicate MAC advertisements and restores single-path routing.
Out-of-Band Administrative Access: Establish a dedicated serial console or isolated out-of-band management connection to the secondary firewall.
Suspend Secondary HA: Execute request high-availability state suspend on the secondary unit to force its interfaces into a suspended, non-forwarding state.
Restore HA1 Control Links: Diagnose and repair the physical cabling, SFP transceivers, intermediate VLANs, or MTU mismatches on the HA1 and HA1-backup links.
Verify Peer Communication: Run show high-availability state on both peers to confirm heartbeat communication and configuration synchronization have recovered.
Restore Functional Mode: Re-enable the switch ports and execute request high-availability state functional on the secondary unit.
PAN-OS Software Upgrade Procedures
Upgrading production PAN-OS clusters requires strict adherence to dependency rules and structured failover sequences to ensure zero downtime.
Core Upgrade Prerequisites: The Base Image Requirement
PAN-OS software releases follow a hierarchical structure: Major/Minor feature releases (e.g., 11.0.0 , 11.1.0 ) and Maintenance releases (e.g., 11.0.4 , 11.1.2-h3 ).
Exam Trap Alert: Upgrading across feature release versions (such as moving from 10.2.x to 11.0.x or from 11.0.x to 11.1.x ) strictly requires downloading the base .0 image (e.g., 11.0.0 or 11.1.0 ) to the firewall's local disk before the target maintenance release can be installed. The base .0 image does not need to be installed or rebooted, but its file system package must reside locally on disk. Attempting to install a maintenance release without the base image present will fail with an explicit dependency error!
Zero-Downtime HA Upgrade Sequence (Active/Passive) Step 1: Verify HA Sync & Disable Preemption temporarily on both peers
- Step 2: Download Base (.0) and Target Maintenance Images on both peers
- Step 3: Install Target Image on PASSIVE Peer & Reboot Passive Peer
- Step 4: Verify Passive Peer Health (Operates in Version Mismatch Mode)
- Step 5: Suspend Active Peer (`request high-availability state suspend`)
- > Upgraded Peer becomes ACTIVE (Carries Production Traffic)
- Step 6: Verify Production Health on Newly Active Peer
- Step 7: Install Target Image on Suspended Peer & Reboot
- Step 8: Return Peer to Functional (`request high-availability state functional`)
- Step 9: Re-enable Preemption & Verify Synchronized HA Cluster State
Pre-Upgrade Validation: Verify configuration sync ( show high-availability state ). Temporarily disable preemption on both firewalls to prevent unintended failbacks during intermediate reboots.
Image Staging: Download the required base image (e.g., PAN-OS 11.1.0 ) and the target maintenance release (e.g., PAN-OS 11.1.2-h3 ) onto both Firewall-A and Firewall-B.
Upgrade the Passive Peer First: Install the target maintenance release on the Passive firewall and initiate a reboot: admin@PA-Passive> request system software install version 11.1.2-h3
admin@PA-Passive> request restart system
Verify Health of Upgraded Passive Peer: After the firewall boots, confirm operational stability ( show system info ). The HA cluster will display a status of Version Mismatch , which is fully supported temporarily during upgrade procedures to maintain session synchronization.
Execute Controlled Failover: Suspend the currently Active firewall to shift all production traffic to the newly upgraded firewall: admin@PA-Active> request high-availability state suspend
Validate Production Traffic: Confirm that the newly active firewall processes user sessions, GlobalProtect connections, and App-ID inspection normally.
Upgrade the Remaining Peer: Install the target maintenance release on the suspended firewall and reboot it: admin@PA-Suspended> request system software install version 11.1.2-h3
admin@PA-Suspended> request restart system
Restore Functional Cluster State: After reboot, restore the unit: request high-availability state functional . Re-enable preemption if dictated by network policy, and confirm cluster synchronization.
Dynamic Content Updates Lifecycle & Threshold Hold Times
Palo Alto Networks delivers regular security updates via dynamic content packages:
Dynamic Content Type Release Frequency Primary Contents Recommended Enterprise Schedule
Applications and Threats Daily / Weekly New App-ID definitions, IPS vulnerability and spyware signatures Scheduled download and install with threshold delay
Antivirus Hourly Stream-based malware signatures, trojan definitions Scheduled download and install every 1-2 hours
WildFire Every 1 minute (Real-Time) Zero-day malware signatures generated by cloud analysis Real-time streaming updates or scheduled every 15 mins
Threshold Hold Times for Operational Stability
While security best practices dictate rapid signature deployment, unvalidated application definitions in mission-critical networks can introduce false positives or alter security policy matching. PAN-OS provides Threshold Hold Times :
Allows administrators to delay the automatic installation of downloaded content by a configurable window (e.g., 24 to 48 hours).
Provides an observation buffer during which the global threat research community and peer enterprises validate signature stability.
Managing New App-IDs: The 'Disable New Apps' Safeguard
When Palo Alto Networks introduces new App-IDs in an Applications and Threats update, traffic that previously matched generic protocols (such as web-browsing or ssl ) may now match a newly introduced specific App-ID (such as an obscure SaaS tool).
If the security policy only permits web-browsing , traffic transitioning to the new App-ID will be dropped by the inter-zone-default rule!
Protection Setting: In Device > Dynamic Updates , enable Disable new apps in content update .
Newly introduced App-IDs are installed in a disabled state, preserving existing policy matches. Administrators review the Review New Apps report, modify security rules to accommodate approved applications, and manually enable the new App-IDs.
Enterprise Log Forwarding & Strata Logging Service
PAN-OS firewalls generate rich telemetry across multiple log types: Traffic, Threat, URL Filtering, WildFire Submissions, Data Filtering, Authentication, Tunnel Inspection, and System logs.
Log Forwarding Profiles
Security policies do not forward logs directly. Instead, an administrator attaches a Log Forwarding Profile to each security rule, dictating where matching event logs are streamed in real time. [ Security Policy Rule Match ]
- |
- v
- [ Log Forwarding Profile ]
- |
- +---> [ Syslog Servers (SIEM) ] (RFC 5424 over TLS / TCP)
- +---> [ SNMP Trap Receivers ] (SNMP v2c / v3)
- +---> [ HTTP Webhook Receivers] (JSON POST to ServiceNow, Slack, SOAR)
- +---> [ Cortex Data Lake / Strata Logging Service ] (Cloud-Native Fabric)
- External Forwarding Destinations
Syslog: Transmits standard RFC 5424 (TLS/TCP) or RFC 3164 (UDP) formatted records to external SIEM platforms (such as Splunk, QRadar, or Microsoft Sentinel). Supports custom format strings.
SNMP Traps: Sends alerts for system alarms, environmental thresholds, and high-severity threat detections using SNMP v2c or SNMP v3 with AES/SHA encryption.
HTTP Webhooks: Formats log data into structured JSON payloads and executes HTTP POST requests targeting external REST APIs (such as triggering automated incident tickets in ServiceNow or alerts in PagerDuty).
Cortex Data Lake (CDL) / Strata Logging Service:
Cloud-native logging infrastructure providing petabyte-scale elastic storage, sub-second query execution, and centralized visibility.
Feeds telemetry into Cortex XDR, Strata Cloud Manager (SCM), and Cloud-Delivered Security Services without consuming local firewall disk I/O.
Verification Command: request logging-service-forwarding status verifies mutual TLS certificate authentication, cloud gateway connectivity, and queue backlogs.
- System Diagnostics & Disaster Recovery
- Generating Tech Support Dumps
A Tech Support Dump ( tech-support-dump ) is an exhaustive diagnostic archive required by Palo Alto Networks Technical Support (TAC) for deep root-cause failure analysis. It captures system logs, core crash dumps, running configurations, routing tables, and hardware register telemetry: # Generate a tech support bundle on the firewall
admin@PA-3410> request tech-support-dump
- # Export the generated archive to a remote SCP server
- admin@PA-3410> scp export tech-support to user@10.10.10.50:/var/support/PA-3410-TS.tgz
- Maintenance Mode for Disaster Recovery
If an unrecoverable system corruption occurs (e.g., sudden power loss during a flash write, lost superuser credentials, or corrupted root partitions), the firewall cannot boot into PAN-OS. Administrators access Maintenance Mode via the physical serial console port (9600 baud, 8 data bits, no parity, 1 stop bit / 8-N-1):
Reboot the firewall while observing serial console output.
When prompted during the initial bootloader countdown (GRUB/U-Boot), type maint and press Enter.
The system boots into an interactive, menu-driven recovery environment:
Factory Reset: Wipes local storage media and restores the factory-default PAN-OS image.
Image Rollback: Switches the active boot partition back to the previously operational PAN-OS version.
File System Check (FSCK): Scans and repairs corrupted disk partition structures.
Install Image / Maintenance via TFTP: Installs a fresh PAN-OS software image across the management network.