Study guide
9.4 Event Grid Retries & Event-Driven Workflows
Exam AI-200: Developing AI Cloud Solutions on Azure Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Audience Profile & Study Path
2 Chapter 1: Container Registry & App Service Hosting
1.1 ACR: Build, Store, Version & Manage Images 1.2 ACR Tasks: Build & Run Images 1.3 Deploy Containers to Azure App Service
3 Chapter 2: Azure Container Apps & KEDA Scaling
2.1 Azure Container Apps: Environments, Config & Revisions 2.2 KEDA Event-Driven Scaling in Container Apps
4 Chapter 3: AKS Manifests & Container Troubleshooting
3.1 Deploy & Manage Apps on AKS with Manifests 3.2 Monitor & Troubleshoot AKS and Container Apps
5 Chapter 4: Cosmos DB for NoSQL Fundamentals
4.1 Cosmos DB for NoSQL: SDK Connect & Queries 4.2 Optimize RUs: Indexing Policies & Consistency Levels
6 Chapter 5: Cosmos DB Vectors & Change Feed
5.1 Embeddings & Vector Similarity Search in Cosmos DB 5.2 Change Feed Processor for New/Updated Items
7 Chapter 6: Azure PostgreSQL Foundations for AI
6.1 Azure Database for PostgreSQL: Connect & Query with SDKs 6.2 Schema Design, Data Types & Indexing Strategies 6.3 pgvector Indexing: Latency & Compute Overhead
8 Chapter 7: PostgreSQL Vector Workloads & RAG
7.1 Configure Compute, Memory & Storage for Vector Workloads 7.2 Vector Search, Semantic Retrieval & RAG with Metadata Filters 7.3 Connection Optimization for Throughput & Latency
9 Chapter 8: Azure Managed Redis for AI Solutions
8.1 Azure Managed Redis: Caching, Expiration & Invalidation 8.2 Redis Vector Indexing for Similarity Search
10 Chapter 9: Service Bus & Event Grid for AI Pipelines
9.1 Service Bus Queues & Message Processing 9.2 Topics, Subscriptions & Dead-Letter Queues 9.3 Event Grid: Custom Events & Filters 9.4 Event Grid Retries & Event-Driven Workflows
11 Chapter 10: Azure Functions for Serverless AI Back Ends
10.1 Azure Functions: Triggers & Bindings 10.2 Build Serverless APIs with Functions 10.3 Configure & Deploy Function Apps
12 Chapter 11: Key Vault & App Configuration
11.1 Azure Key Vault: Secret Storage & Retrieval 11.2 Key Vault Secret Rotation Patterns 11.3 Azure App Configuration for App Settings
13 Chapter 12: OpenTelemetry & KQL Observability
12.1 Distributed Tracing with OpenTelemetry SDKs 12.2 Correlating Traces into Azure Monitoring 12.3 Write KQL Queries to Analyze Logs 12.4 KQL for Metrics & Troubleshooting Scenarios
9.4 Event Grid Retries & Event-Driven Workflows
Key Takeaways Explain This Topic AI
Event Grid retries failed deliveries with exponential backoff for a configurable duration; persistent failures can be dead-lettered to a Storage Blob container
Handlers should return proper HTTP status codes: 200–299 success, and retriable vs non-retriable failures influence whether Event Grid retries
Dead-lettered Event Grid events preserve delivery context so you can replay or diagnose AI workflow failures
Event-driven AI workflows often chain system topics → Functions → Service Bus work → custom events → multiple filtered subscriptions
- Design handlers to be idempotent because at-least-once delivery and retries can surface duplicate events
- Last updated: July 2026
Reliable Delivery: Retries, Backoff, and Dead-Lettering
Event Grid’s push model only works if handlers are reachable and healthy. When a destination returns a failure or times out, Event Grid retries delivery using an exponential backoff policy for up to a maximum event time-to-live / retry period (configurable on the subscription; defaults are generous—often on the order of 24 hours unless you tighten them).
Understanding retries is essential for AI workflows: a Function that enqueues embedding work might fail because Service Bus is briefly throttling, or a webhook might time out while a cold start occurs. Retries bridge those transient gaps—but they also mean your handler can see the same event more than once .
What Counts as Success or Failure?
For HTTP-style destinations:
Success: destination returns 200–299 within the timeout.
Retryable failure: timeouts, 503 , and many 5xx responses → Event Grid schedules another attempt.
Non-retryable / terminal scenarios: certain 400 -class failures may stop retries depending on policy and error type; misconfigured endpoints that fail validation never become healthy subscriptions.
Azure Functions with Event Grid triggers integrate with this model: throw/fail the invocation for transient issues you want retried; succeed only after side effects are safely committed or durably queued.
Dead-Letter Destinations
If Event Grid exhausts retries (or the event expires), it can write the event to a dead-letter destination —commonly a Storage account blob container configured on the event subscription, with a managed identity or SAS granting Event Grid permission to write.
Dead-letter blobs typically include the original event and metadata about why delivery failed. For AI operations teams:
Alert on dead-letter container growth.
Inspect failing payloads ( documentId , model name, tenant).
Fix the handler (bug, auth, wrong queue name).
Replay by re-publishing to the custom topic or by a reclaim script that re-invokes the pipeline.
Compare with Service Bus DLQ: both isolate poison/failed work, but Event Grid dead-letters delivery failures to subscribers , whereas Service Bus DLQ isolates consumer processing failures of queued messages. Mature AI platforms use both at their respective layers.
- Concern Service Bus DLQ Event Grid dead-letter
- Trigger Max delivery, TTL, explicit dead-letter Retry budget exhausted / expiration
- Typical store Service Bus subqueue Blob container
- Failure meaning Worker could not process message Event Grid could not deliver to handler
- Reclaim Receive from DLQ, resubmit Re-publish event or reprocess blob
- Idempotent Handlers Are Mandatory
Because retries and competing architecture layers create duplicates:
Use event id + handler name as an idempotency key in Cosmos DB/Redis before starting expensive GPU work.
Make “enqueue Service Bus message” idempotent with Service Bus duplicate detection (MessageId = event id) where appropriate.
Ensure “update document status to Embedded” is a conditional write, not a blind append of duplicate rows.
Composing Event-Driven AI Workflows
The official skill emphasizes implementing event-driven workflows with Event Grid—including filters, custom events, and retries. A reference workflow that often appears in solution designs:
Scenario: Secure Document Q&A Ingestion
User uploads PDF to inbox container.
System topic on the storage account emits BlobCreated .
Event Grid subscription filters subject ends with .pdf and data.contentType for application/pdf .
Function A validates virus scan status and tenant path; on success, sends a Service Bus queue message ( ingest job) and returns 200. On transient Service Bus errors, Function fails → Event Grid retries.
Service Bus workers run Document Intelligence, chunking, embeddings (Peek-Lock, sessions per document).
On success, worker publishes custom CloudEvent Document.Embedded to a custom topic.
Subscription B (filter type = embedded) updates Azure AI Search and returns 200.
Subscription C (filter tenant prefix) pushes a SignalR/user notification.
Subscription D (advanced filter data.chunkCount NumberGreaterThan 500) triggers an extra quality-evaluation job.
If Subscription B’s search endpoint is down beyond the retry window, the event lands in Event Grid’s dead-letter container for replay after search recovers.
This design separates reaction (Event Grid) from heavy work (Service Bus) and uses filters so each subscriber stays focused.
Configuring Retry Policy Thoughtfully for AI
Short-lived glue Functions: default retries are fine; ensure cold starts still finish within Event Grid’s per-attempt timeout or use premium plans.
Handlers that call slow model APIs directly inside the Event Grid request path are a design smell—offload to Service Bus/queue-triggered Functions so Event Grid only needs a quick enqueue success.
Tighten retry TTL for events that must be ignored if stale (for example, “cancel generation” may not be useful 24 hours later); lengthen for “billing meter” style events that must eventually land.
Always configure dead-lettering in production subscriptions—silent drop after retries hides AI pipeline outages.
Failure Modes to Recognize on the Exam
Handler returns 200 before enqueuing work → Event Grid will not retry, but work may be lost if the process crashes after the 200. Commit then acknowledge (or outbox pattern).
Handler takes minutes calling a model during Event Grid delivery → timeouts → repeated retries → duplicate partial work. Enqueue and finish quickly .
No dead-letter destination configured → exhausted retries vanish from easy recovery.
Filters too broad → storm of irrelevant invocations increases failure noise and cost.
Filters too narrow → missed events look like “AI pipeline stuck” when Event Grid never delivered.
End-to-End Reliability Checklist
Event Grid subscription: event types + subject/advanced filters aligned to AI domain properties.
Retry policy + dead-letter container with monitoring.
Handler idempotency using event ids.
Bridge to Service Bus for durable back-end operations (queues/topics/DLQ from earlier sections).
Custom events announce stage completion for fan-out without tight coupling.
Mastering this chapter means you can explain—not just name—when to queue with Service Bus, when to react with Event Grid, how filters shape delivery, and how retries/dead-lettering keep asynchronous AI pipelines trustworthy under partial failure.