Study guide
14.2 Latency, Throughput, Token Use, and Cost Metrics
Microsoft AI-300: Operationalizing Machine Learning and Generative AI Solutions Exam
Track your progress as you read
Finished sections get a check mark.
1 Introduction & Exam Overview
1.1 Current Exam Facts 1.2 Skills Measured, Format, and How This Guide Is Organized 1.3 Candidate Profile, Azure ML v2, Foundry, and DevOps Tooling
2 Chapter 2: Azure Machine Learning Workspaces, Datastores, Compute, and IAM
2.1 Create and Manage Azure Machine Learning Workspaces 2.2 Create and Manage Datastores 2.3 Create and Manage Compute Targets 2.4 Configure Identity and Access Management for Workspaces
3 Chapter 3: Azure Machine Learning Data Assets, Environments, Components, and Registries
3.1 Create and Manage Data Assets 3.2 Create and Manage Environments 3.3 Create and Manage Components 3.4 Share Assets Across Workspaces with Registries
4 Chapter 4: Infrastructure as Code, GitHub, and Network Isolation for Azure ML
4.1 GitHub Integration and Git Source Control for Machine Learning Projects 4.2 Deploy Workspaces and Resources with Bicep and Azure CLI 4.3 Automate Resource Provisioning with GitHub Actions 4.4 Restrict Network Access to Machine Learning Workspaces
5 Chapter 5: MLflow Tracking, AutoML, Training Jobs, and Hyperparameter Sweeps
5.1 Configure Experiment Tracking with MLflow 5.2 Use Automated Machine Learning to Explore Optimal Models 5.3 Notebooks, Command Jobs, and Training Scripts 5.4 Automate Hyperparameter Tuning
6 Chapter 6: Distributed Training, Training Pipelines, and Job Comparison
6.1 Manage Distributed Training for Large and Deep Learning Models 6.2 Implement Training Pipelines 6.3 Compare Model Performance Across Jobs
7 Chapter 7: Model Registration, Responsible AI, and Lifecycle Management
7.1 Register MLflow Models and Package Feature Retrieval Specs 7.2 Evaluate Models with Responsible AI Principles 7.3 Manage Model Lifecycle and Archiving
8 Chapter 8: Online Endpoints, Batch Endpoints, Rollout, and Rollback
8.1 Deploy Models to Managed Online Endpoints 8.2 Deploy Models to Batch Endpoints 8.3 Test, Troubleshoot, Progressive Rollout, and Safe Rollback
9 Chapter 9: Data Drift, Production Metrics, Retraining, and Alerts
9.1 Detect and Analyze Data Drift 9.2 Monitor Production Metrics and Trigger Retraining or Alerts
10 Chapter 10: Microsoft Foundry Resources, IAM, Networking, and IaC
10.1 Create and Configure Foundry Resources and Project Environments 10.2 Managed Identities and RBAC for Foundry 10.3 Foundry Network Security and Private Networking 10.4 Deploy Foundry Infrastructure with Bicep and Azure CLI
11 Chapter 11: Deploying and Managing Foundation Models
11.1 Serverless API Endpoints and Managed Compute for Foundation Models 11.2 Select Foundation Models for Specific Use Cases 11.3 Model Versioning and Production Deployment Strategies 11.4 Provisioned Throughput Units for High-Volume Workloads
12 Chapter 12: Prompt Design, Variants, and Git Version Control
12.1 Design and Develop Prompts 12.2 Create Prompt Variants and Compare Performance 12.3 Version Control for Prompts with Git
13 Chapter 13: Evaluation Datasets, Quality Metrics, Safety, and Automated Workflows
13.1 Test Datasets and Data Mapping for Evaluation 13.2 Groundedness, Relevance, Coherence, and Fluency 13.3 Risk and Safety Evaluations 13.4 Built-in and Custom Automated Evaluation Workflows
14 Chapter 14: Continuous Monitoring, Performance, Cost, Tracing, and Logging
14.1 Continuous Monitoring in Microsoft Foundry 14.2 Latency, Throughput, Token Use, and Cost Metrics 14.3 Logging, Distributed Tracing, and Production Debugging
15 Chapter 15: Retrieval Tuning, Embeddings, Hybrid Search, and RAG Evaluation
15.1 Tune Similarity Thresholds, Chunk Sizes, and Retrieval Strategies 15.2 Select and Fine-Tune Embedding Models 15.3 Hybrid Search: Semantic and Keyword Retrieval 15.4 RAG Relevance Metrics and A/B Testing
16 Chapter 16: Fine-Tuning Methods, Synthetic Data, and Promotion to Production
16.1 Advanced Fine-Tuning Methods (LoRA, QLoRA, Full) 16.2 Create and Manage Synthetic Data for Fine-Tuning 16.3 Monitor, Optimize, and Promote Fine-Tuned Models
14.2 Latency, Throughput, Token Use, and Cost Metrics
Key Takeaways Explain This Topic AI
End-to-end latency follows TTLT = TTFT + (TBT × tokens generated). Always pair a latency metric with a token-count metric before calling a regression.
For streaming, use Time to Response (AzureOpenAITimeToResponse) as TTFT. For overall duration, use Time to Last Byte (AzureOpenAITTLTInMS). Do not use the legacy Cognitive Services Latency metric for Azure OpenAI.
Throughput is observed RPM and TPM. Standard quota is admission control, not guaranteed capacity. Agent dashboard hints: latency above about 10 seconds, run success rate below about 95 percent.
Pay-as-you-go bills input and output tokens; output tokens usually cost more and dominate TTLT. Cache hits appear as cached_tokens under prompt_tokens_details. Cached tokens do not consume PTU capacity.
PTU billing is hourly for deployed PTUs, idle or busy (Utilization V2 at 100 percent returns 429). Optimize with a smaller model, shorter prompts, prompt caching, then PTU for steady load and spillover for bursts.
Last updated: August 2026
Latency, Throughput, Token Use, and Cost Metrics
Quick Answer: Measure time to first token (TTFT) separately from end-to-end latency. TTLT = TTFT + (TBT × tokens generated) . Throughput is requests per minute (RPM) and tokens per minute (TPM) , not "the quota number." Cost splits input , output , and cached tokens. Provisioned throughput units (PTUs) are billed hourly for deployed capacity ; pay-as-you-go bills tokens . Optimize with smaller models , shorter prompts , prompt caching , and PTU for steady load .
Domain 4 pairs performance metrics (latency, throughput, response times) with cost metrics (tokens, resource usage) . Chapter 11 taught you to buy PTUs. This section teaches you to read whether those PTUs, or a Standard deployment, are healthy and cheap enough .
Latency: TTFT is not end-to-end
Azure OpenAI in Microsoft Foundry publishes a formula you should treat as exam-ready:
TTLT = TTFT + (TBT × Tokens Generated)
TTFT (Time to First Token): prompt submitted until the first token. Dominated by prefill (prompt size), queueing , and cache misses .
TBT (Time Between Tokens): average milliseconds between generated tokens (decode speed).
TTLT (Time to Last Token / Time to Last Byte): end-to-end generation time.
Because TTLT scales with output length , a doubling of wall-clock time with a doubling of Generated Completion Tokens is often expected , not a platform regression. Always pair a latency metric with a token metric .
Question you have Metric (display / REST) Notes
How long until the user sees the first word (streaming)? Time to Response / AzureOpenAITimeToResponse Maps to TTFT at the API gateway. Grows with prompt size and cache misses. Does not include your client.
How long until the full answer exists? Time to Last Byte / AzureOpenAITTLTInMS Maps to TTLT . Use for non-streaming SLAs.
Is decode slow? Time Between Tokens / AzureOpenAINormalizedTBTInMS Generation rate.
Are long prompts making TTFT look worse than it is? Normalized Time to First Byte / AzureOpenAINormalizedTTFTInMS For comparing prompt sizes, not absolute customer latency.
How many input / output tokens? Processed Prompt Tokens / Generated Completion Tokens Pair with latency.
Do not use the Cognitive Services - HTTP Requests Latency metric for Azure OpenAI. Microsoft documents that legacy metric as misleading for these workloads.
On the agent dashboard, latency above about 10 seconds is a clue for throttling , heavy tool chains , or network , not a hard SLA Microsoft publishes for every model. Timeouts are a client-side twin of TTLT: a 60-second non-streaming call is cancelled by many gateways and browsers. Streaming ( stream: true ) does not shorten TTLT, but it delivers tokens as they exist so the client is less likely to time out and the user perceives a faster start (TTFT).
Gateway metrics still omit browser , VPN , and your orchestration delay. If the portal looks healthy and users do not, log a correlation ID on the client (Section 14.3).
Throughput: RPM, TPM, and admission versus capacity
System throughput is what the deployment can process: RPM and TPM . Per-call latency is a different question. Quota on a Standard (pay-as-you-go) deployment is admission control , not a guaranteed TPM. Under load you see HTTP 429 before you "use the quota number."
Estimate TPM from Azure Monitor Processed Prompt Tokens + Generated Completion Tokens , or from each response usage object times observed RPM. That estimate ignores cache hits , so it is conservative for PTU sizing.
Agent run success rate below about 95 percent is Microsoft's dashboard investigation hint: look at failed runs , not only average latency. Split AzureOpenAIRequests by StatusCode and ModelDeploymentName .
Separate workloads onto separate deployments . A bulk summarizer batched with an interactive copilot both lengthens short calls and destroys cache hit rate .
Tokens and cost: input, output, cached
Pay-as-you-go (Standard) bills tokens . Output tokens cost more than input on current Azure OpenAI price lists, and they dominate TTLT because decode is sequential. Reasoning models add reasoning_tokens inside completion details — they count toward cost and latency even when the user never sees them.
Prompt caching (Microsoft Learn, updated 2026-08-11):
Requires about 1,024 tokens minimum, and the first 1,024 tokens must be identical .
A single character change in that prefix is a cache miss ( cached_tokens = 0).
Hits appear as cached_tokens under prompt_tokens_details .
Place stable system instructions, tools, and files first ; put the user question last .
On Standard , cache reads are discounted versus ordinary input. On provisioned deployments, cached tokens get up to a 100 percent discount on input for capacity: cached tokens do not consume PTU capacity .
Monitor Prompt Token Cache Match Rate ( AzureOpenAIContextTokensCacheMatchRate ) and Active Tokens (total minus cached) on PTU.
Do not invent a cache discount percentage for the exam unless the item prints the price list. Know where the signal lives ( cached_tokens , cache match rate) and that cache rate is a PTU sizing input .
PTU utilization versus PAYG spend
Provisioned-managed Utilization V2 ( AzureOpenAIProvisionedManagedUtilizationV2 ) is (PTUs consumed / PTUs deployed) × 100 . At 100 percent or more , the service throttles with 429 . The older Utilization metric is deprecated .
PTU billing is hourly for the deployed count , idle or busy. You cannot pause a provisioned deployment; billing stops when you delete (or resize). PAYG bills tokens and has no latency SLA . Spillover can send overflow to Standard when PTUs return non-200. Azure Reservations discount the PTU meter ; they do not reserve capacity (Chapter 11).
- Pattern Cost lever Performance lever
- Variable / dev Standard PAYG Accept variable latency
Steady high volume, latency SLA PTU , then a reservation once capacity is deployed Watch Utilization V2; size with RPM, prompt, completion, cache rate
- Bursts on top of PTU Spillover to Standard Avoid 429s without buying peak PTUs
- Long static prefixes Prompt cache Lower TTFT and PTU burn
Optimize in this order (cheap to expensive): pick a smaller model that still passes Chapter 13 evals → shorten prompts and lower max_tokens / add stop sequences → cache the prefix → stream for perceived latency → PTU for steady load → reservations for the PTU meter. Mixing eval judges onto the production PTU deployment shows up as utilization and cost that is not user traffic.
Content filtering adds safety latency . That tradeoff is real; do not "fix TTFT" by silently disabling filters on a production agent without a policy decision.
Scenario
A support copilot's p95 TTLT jumps from 4 seconds to 9 seconds after a prompt change. TTFT is flat. Generated Completion Tokens doubled because the new prompt asked for "detailed citations and a full policy restatement." Utilization V2 is 40 percent. Resizing PTUs will not fix this. Cut the completion, add a stop sequence, or summarize citations. If instead TTFT rose while prompt tokens were flat and cache match rate collapsed, look at a prefix change in the first 1,024 tokens.
Common trap
Treating quota TPM as observed throughput , or treating TTLT as a platform outage without checking output tokens . Second trap: using the legacy Cognitive Services Latency metric. Third: assuming PTU hourly cost drops at night — it does not, unless you delete , and you may not get capacity back in the morning.
Operator checklist
Chart TTLT or TTFT with Generated Completion Tokens and Processed Prompt Tokens.
Chart 429 on AzureOpenAIRequests split by StatusCode .
Chart Utilization V2 on PTU; token spend on PAYG.
Chart cache match rate after every prompt-template change.
Keep judge traffic off the user-facing deployment when you can.