Study guide
8.2 AI Computer Vision for Unattended & Remote Interfaces
UiPath Automation Developer Professional UiPath-ADPv1
Track your progress as you read
Finished sections get a check mark.
1 Chapter 1: Exam Overview & Blueprint Map
1.1 About the UiPath-ADPv1 Exam: Format, Fees & Policies 1.2 The Exam Topics Mapped to This Guide & a Study Plan
2 Chapter 2: REFramework Architecture & State Dynamics
2.1 State Machine Architecture & The Four Core States 2.2 Configuration Management: Settings, Constants, and Assets 2.3 Transaction Types & The Dispatcher vs. Performer Architecture 2.4 Adapting REFramework: Tabular Data, Collections & Linear Processes
3 Chapter 3: REFramework Exception Handling & Recovery
3.1 BusinessRuleException vs. System.Exception Handling Patterns 3.2 Retry Mechanisms: Queue Retries vs. REFramework MaxRetryNumber 3.3 Consecutive System Exceptions & Graceful Process Termination
4 Chapter 4: State Machines, Custom Log Fields & REFramework Testing
4.1 Building State Machine Projects in Studio 4.2 Logging Levels, Robot Logs & Custom Log Fields in REFramework 4.3 Testing a REFramework Project With and Without Orchestrator Queues
5 Chapter 5: LINQ Queries & DataTable Operations
5.1 LINQ Essentials for Collections & Filtering 5.2 Advanced DataTable Querying with LINQ 5.3 In-Memory DataTable Transformations & Column Operations
6 Chapter 6: Regex, DateTime, JSON & Invoke Activities
6.1 Regular Expressions & Text Parsing for Automation 6.2 DateTime Formatting, Timezone Conversions & TimeSpan Calculations 6.3 JSON Parsing, JSONPath & Serialization with Newtonsoft.Json 6.4 Invoke Method & Invoke Code Activities 6.5 Data Referencing in Invokes: DataTable, List & Dictionary Arguments
7 Chapter 7: Modern UI Automation & Unified Targeting
7.1 Modern Design Experience & The Unified Target Concept 7.2 Anchors, Relative Elements & UI Automation Activities 7.3 Dynamic Selectors, Variables & Wildcards in Selectors
8 Chapter 8: Object Repository, Computer Vision & Remote Runtime
8.1 Object Repository & UI Descriptor Libraries 8.2 AI Computer Vision for Unattended & Remote Interfaces 8.3 Automation in Virtual Environments & UiPath Remote Runtime
9 Chapter 9: Advanced UI Automation: Browser Modes, Agentic UI & macOS
9.1 Browser Automation with the WebDriver Protocol 9.2 Agentic UI Automation: ScreenPlay, Semantic Activities & Healing Agent 9.3 macOS Automation: Native Apps, Office Apps & AppleScript
10 Chapter 10: Advanced Control Flow & Exception Architecture
10.1 Flowcharts, Sequences & Parallel Control Flow Patterns 10.2 Global Exception Handler Architecture & ErrorAction Directives 10.3 TryCatch, Retry Scope & Custom Exception Architectures
11 Chapter 11: Orchestrator Queues & Transaction Management
11.1 Queue Architecture, SpecificContent & Transaction Lifecycles 11.2 Queue Items SLA, Postpone, Deadlines & Priority Scheduling 11.3 Bulk Operations, Transaction Status Updates & Queue Triggers
12 Chapter 12: Orchestrator Tenant & Folder Entities
12.1 Asset Types, Per-Robot Assets & External Credential Stores 12.2 Modern Folder Hierarchy, User/Robot Assignment & Permissions 12.3 Storage Buckets, Webhooks & Environment Promotion 12.4 Orchestrator Tags: Labels, Properties & Governance
13 Chapter 13: Unattended Execution, API Triggers & Cloud Robots
13.1 Unattended Job Execution: Folders, Priorities, Background vs. Foreground & Runtimes 13.2 API Triggers: Starting Jobs from External Systems over HTTP 13.3 Serverless and Cloud VM Robots: Cloud-Hosted Execution Options
14 Chapter 14: Monitoring, Recording & Solutions Management
14.1 Orchestrator Monitoring, Trigger Alerts & Job Health 14.2 Job and Transaction Recording: Troubleshooting Faulted Unattended Jobs 14.3 Solutions Management: Packaging and Deploying Automations with Their Resources
15 Chapter 15: Debugging, Remote Debugging & Profiling
15.1 Studio Debugging Tools: Breakpoints, Panels & Execution Control 15.2 Remote Debugging: Remote Machine and Unattended Robot Connections 15.3 Profile Execution: Finding Performance Bottlenecks in Studio
16 Chapter 16: RPA Testing & Test Suite
16.1 UiPath Test Suite Overview, Test Cases & Given-When-Then Structure 16.2 Data-Driven Testing, Test Verification Activities & Mock Activities 16.3 CI/CD Pipelines with uipcli & Test Manager Traceability
17 Chapter 17: Long-Running Workflows, Action Center & Apps
17.1 Long-Running Workflows: Persistence, Suspend and Resume 17.2 Action Center & Human-in-the-Loop Orchestration 17.3 UiPath Apps: Action Apps, Automations & Integration Service
18 Chapter 18: Trigger-Based Attended Automation & Callouts
18.1 Trigger-Based Attended Automation: Forms, Triggers & Global Variables 18.2 Running Local Triggers & the Event Inspection Tool 18.3 Callouts & Forms for Attended Guidance
19 Chapter 19: UiPath Assistant & Robot Sessions
19.1 UiPath Assistant: Automations, Reminders, Widgets & Configuration 19.2 Robot Session (Picture in Picture): Child Session vs. Virtual Desktop
20 Chapter 20: Autopilot for Developers & Autopilot for Everyone
20.1 Autopilot for Developers: AI Assistance in Studio and Studio Web 20.2 Autopilot for Everyone: Chat, Automations & Clipboard AI
21 Chapter 21: API Automation & Integration Service
21.1 Integration Service: Connectors, Connections & Connector Activities 21.2 API Workflows in Studio Web 21.3 HTTP Requests: Web APIs, the Orchestrator API & Connector Requests 21.4 Connector Builder: Custom Integration Service Connectors 21.5 Integration Service Event Triggers: Connected and Disconnected 21.6 GenAI Activities: Generative AI Inside Workflows
22 Chapter 22: Data Service, Context Grounding & Coded Workflows
22.1 Data Fabric (Data Service) Entities: Modeling Business Data 22.2 Data Fabric Activities and the Platform Connector 22.3 Context Grounding: Retrieval-Augmented Generation with Company Data 22.4 Coded Automations: Structure, Arguments & Invoking Workflows 22.5 Coded Automation Services: Orchestrator, UI Automation, Microsoft 365 & Custom Services
23 Chapter 23: Integrated Scenario Practice
23.1 Integrated Scenario Practice: Designing an End-to-End Solution
8.2 AI Computer Vision for Unattended & Remote Interfaces
Key Takeaways Explain This Topic AI
UiPath AI Computer Vision (CV) uses deep-learning convolutional neural networks and OCR to recognize UI elements semantically rather than relying on brittle pixel matching or native accessibility trees.
Computer Vision is the fallback for Citrix, VDI, RDP, and HTML5 canvas applications where native accessibility APIs or UiPath Remote Runtime are unavailable; in Modern activities it is a Unified Target method ranked after selectors.
The CV Screen Scope captures a screenshot when it starts, sends it to the Computer Vision service (UiPath cloud, AI Center, or a local server), and caches the recognized elements for its child CV activities.
Core CV activities (CV Click, CV Type Into, CV Get Text, CV Check) execute against the cached visual tree and support multi-anchor targeting to link ambiguous inputs to visual text or icon anchors.
When an interaction modifies the user interface (such as opening a modal window, switching tabs, or expanding a dropdown), developers must execute CV Refresh to re-analyze the screen and update the cached visual model.
Last updated: September 2026
8.2 AI Computer Vision for Unattended & Remote Interfaces
Core Concept: UiPath AI Computer Vision (CV) is an artificial-intelligence-driven automation technology engineered to recognize, parse, and interact with user interfaces purely through visual appearance. Utilizing deep-learning convolutional neural networks (CNNs) paired with advanced Optical Character Recognition (OCR), Computer Vision interprets screens human-like: identifying UI elements (text boxes, buttons, checkboxes, tables) semantically based on their visual morphology rather than relying on underlying operating system accessibility trees or Document Object Models (DOM).
In an ideal enterprise automation environment, every application exposes a rich accessibility API—such as Microsoft UI Automation (UIA), Microsoft Active Accessibility (MSAA), or an accessible browser DOM—allowing software robots to extract precise, deterministic XML selectors. However, enterprise automations frequently operate in restrictive environments where accessibility trees are completely inaccessible. When automating over Virtual Desktop Infrastructure (VDI), secure remote terminal sessions, or legacy graphical runtimes, robots receive only a flat stream of graphical pixels. AI Computer Vision bridges this divide, delivering high-speed, resilient automation without requiring native selectors.
1. The Computer Vision Paradigm vs. Classic Image Automation
Prior to AI Computer Vision, automating remote desktop environments relied on classic image automation and basic OCR. These legacy techniques are notoriously brittle in enterprise production:
The Failure Modes of Classic Image Automation
Pixel-by-Pixel Template Matching: Classic image activities (such as Click Image or Find Image ) utilize normalized cross-correlation algorithms. If a single pixel shifts, or if display resolution, font anti-aliasing, color palettes, or rendering themes differ between the developer's laptop and the unattended runner VDI, template matching fails.
DPI and Scaling Breakdowns: When an unattended robot logs into a remote virtual machine with a display scaling factor of 125% or 150% while the automation was recorded at 100%, classic image matching suffers catastrophic selector faults.
Lack of Semantic Understanding: A classic image search has no concept of what a "text input field" or "checkbox" represents; it merely evaluates whether an array of RGB pixel values matches a reference bitmap.
How Deep-Learning AI Computer Vision Operates
UiPath AI Computer Vision replaces template matching with deep-learning neural networks trained on millions of enterprise UI screenshots across Windows, macOS, Linux, web frameworks, and terminal emulators. The neural network understands UI semantics:
Visual Element Classification: The CV engine analyzes visual contours, shadows, borders, and margins to classify components into functional types: Button , InputField , Checkbox , RadioButton , Dropdown , Table , or Icon .
Visual OCR Fusion: Simultaneous OCR processing (using UiPath Screen OCR, Google Cloud OCR, or OmniPage) extracts text labels, numbers, and symbols across the entire viewport.
Relational Context Mapping: The engine automatically correlates extracted text labels with adjacent input fields (e.g., recognizing that the text "First Name" positioned directly above or to the left of an empty rectangular box serves as the semantic anchor for that input).
Display Invariance: Because the neural network identifies high-level features and morphological patterns, it operates with complete invariance across font anti-aliasing shifts, minor color theme updates, and resolution variations.
2. Target Environments & Use Cases
Computer Vision is the definitive automation strategy for scenarios where native UI trees cannot be extracted:
Automation Environment Technical Barrier Why Computer Vision Is Required
Citrix / VDI / VMware Horizon / RDP Remote Runtime cannot be deployed due to strict vendor restrictions, compliance rules, or multi-tenant hosting boundaries. The robot only receives an H.264/AVC compressed graphical video stream; CV visually reconstructs the interactive element tree.
HTML5 <canvas> & WebGL Applications Modern browser apps (e.g., charting engines, online design tools, browser games) render the entire interface into a flat <canvas> element without individual DOM nodes. Browser DOM selectors only see the outer canvas tag; CV parses individual buttons and inputs inside the canvas.
Legacy Flash & Silverlight Apps Deprecated runtime plugins embedded in enterprise browsers lack exposed accessibility APIs. CV treats the rendered plugin viewport as a standard graphical screen, identifying controls visually.
Java Applets Without JAB Legacy Java desktop clients running in environments where the Java Access Bridge (JAB) is corrupted, disabled, or blocked by security policies. Bypasses the need for Java accessibility hooks entirely by analyzing the visual interface.
Kiosk Terminals & Medical Software Specialized medical imaging workstations or ATM software running on locked-down operating systems accessible only via video capture hardware. Enables reliable robotic interaction without installing any agents or software on the host machine.
3. The CV Screen Scope Architecture
Computer Vision reaches workflows in two ways. In the Modern experience, Computer Vision is one of the Unified Target methods of ordinary activities such as Click and Type Into, ranked after selectors and switched on per project or per element. The dedicated CV Screen Scope container and its CV activities are the older, CV-only way of working, and they are still useful for screens where no selector works at all. +-----------------------------------------------------------------------------------+
- | CV SCREEN SCOPE |
- | Target: Citrix Client - SAP GUI Window |
- | Server Endpoint: UiPath cloud service, AI Center, or a local CV server |
- | API Key: ******************** |
- | |
- | +-----------------------------------------------------------------------------+ |
- | | [Step 1: Capture Viewport Screenshot] | |
- | | │ | |
- | | ▼ | |
- | | [Step 2: Transmit to CV Server & Run Neural Network + OCR] | |
- | | │ | |
- | | ▼ | |
- | | [Step 3: Generate & Store Local Visual Tree Cache in RAM] | |
- | | │ | |
- | | ▼ | |
- | | [Child CV Activities Execute Instantly Against Local Cache] | |
- | | - CV Type Into 'Username' (uses Anchor 'User ID:') | |
- | | - CV Type Into 'Password' (uses Anchor 'Password:') | |
- | | - CV Click 'Log On' Button | |
- | +-----------------------------------------------------------------------------+ |
- +-----------------------------------------------------------------------------------+
- Server Connection Options
The CV models run on a server that the robot calls:
UiPath cloud service. Robots connected to Automation Cloud can use UiPath's hosted Computer Vision service. Outbound HTTPS access from the robot machine is required.
AI Center or on-premises deployments. Organizations with data-residency requirements can host the Computer Vision model themselves and point the activities at that endpoint, supplying an API key where the setup requires one.
Computer Vision Local Server. UiPath documents a separately installed local server for running Computer Vision inside the corporate network.
Whichever option you use, screenshots of the target window are sent to that endpoint, so confirm the choice with your security team before automating sensitive screens.
Image Caching Mechanics: Why CV Is Fast
A common misconception is that Computer Vision makes expensive cloud or neural network queries for every individual click or keystroke. The CV Screen Scope eliminates this overhead through Visual Tree Caching :
Upon entering the CV Screen Scope , the activity captures a single, full screenshot of the specified target window or bounding area.
This screenshot is sent once to the CV engine along with the OCR request.
The neural network detects and classifies every element across the entire screen, returning a structured JSON visual tree containing the bounding boxes, element types, and OCR text coordinates.
This visual tree is stored in local volatile memory (RAM) as a cache.
All subsequent child activities inside the scope—such as ten consecutive CV Type Into and CV Click activities—execute in milliseconds because they query the local in-memory cache directly without taking new screenshots or initiating network calls.
4. Core Computer Vision Activities
UiPath Studio provides a dedicated suite of activities tailored specifically for the Computer Vision engine:
Activity Name Primary Function Key Properties & Operational Behavior
CV Click Clicks on buttons, icons, links, or text labels within the cached scope. Supports ClickType (Single, Double), MouseButton (Left, Right, Middle), and keyboard modifiers ( Shift , Ctrl , Alt ). Can target elements directly or by anchor association.
CV Type Into Types alphanumeric text or keystroke commands into recognized input fields. Supports Activate , ClickBeforeTyping , EmptyField (clears existing contents), and DelayBetweenKeys . Does not require native window handles.
CV Get Text Extracts text strings from a recognized field, label, or user-defined region. Leverages the scope's OCR engine to extract on-screen alphanumeric strings into a String variable.
CV Check Sets or toggles the state of a checkbox or radio button. Uses neural visual inspection to determine whether the checkbox is currently checked or empty, and applies the desired target action ( Check , Uncheck , Toggle ).
CV Highlight Draws a colored border around a recognized element on screen. Primarily used for diagnostic tracing and visual validation during attended automation or workflow debugging.
CV Extract Table Automatically parses multi-row, multi-column tabular data from remote interfaces. Analyzes visual table borders, headers, and cell spacing to construct a structured System.Data.DataTable variable directly from remote screens.
5. Anchor-Based Targeting in Computer Vision
In real-world enterprise applications, interfaces frequently contain multiple identical or unlabeled input fields. For instance, a customer creation form might feature three empty text boxes arranged vertically for Home Phone, Mobile Phone, and Work Phone. If a robot searches purely for an InputField , it cannot determine which box corresponds to which phone number.
Computer Vision resolves this ambiguity using Visual Anchors :
Visual Pairing: During element indication, the developer selects the target control (e.g., the empty text box) and links it to a distinct visual anchor (e.g., the static text label "Mobile Phone:" ).
Spatial Relationship Modeling: The CV engine calculates relative spatial vectors (e.g., Target is positioned 15 pixels to the right of Anchor or Target is positioned 8 pixels below Anchor ).
Multi-Anchor Fallbacks: Developers can configure multiple anchors for a single target (e.g., anchoring an input box to both a field label on the left and a section header above). If one anchor shifts or is partially occluded, the secondary anchor guarantees deterministic element resolution.
Fuzzy Label Matching: If an application's text label undergoes minor capitalization or typographical changes (e.g., "Mobile:" vs. "Mobile Phone:" ), CV's OCR fuzzy matching maintains target identification without throwing runtime exceptions. +---------------------------------------------------------------------+
- | ANCHOR-BASED TARGETING LOGIC |
- | |
- | [Anchor: Text Label] [Target: InputField] |
- | +--------------------+ +-----------------------------+ |
- | | "Invoice Number:" | ---------> | [ ] | |
- | +--------------------+ Relative +-----------------------------+ |
- | Spatial |
- | Vector |
- | |
- | The CV Engine identifies "Invoice Number:" via OCR, locates the |
- | closest InputField along the horizontal axis, and routes the click/ |
- | typing action directly into the target bounding box. |
- +---------------------------------------------------------------------+
- 6. Refreshing and Tuning CV Scopes
While the visual caching mechanism delivers exceptional execution speed, it introduces a critical design requirement: managing stale screen states .
The Stale Cache Dilemma and CV Refresh
Because the CV Screen Scope analyzes the interface once upon container entry, any user interaction that alters the visual structure of the screen renders the cached visual model obsolete. Examples include:
Clicking a dropdown menu that displays an overlay list of selectable options.
Clicking a "Next Page" button that loads a new batch of records into a table.
Clicking an expandable accordion panel or tab control that renders new input fields.
Submitting a form that prompts a modal confirmation dialog.
If an automation attempts to execute a child CV activity against an element that appeared after the scope was initialized, the robot will query the stale cache, fail to find the element, and throw an element-not-found error.
To resolve this, developers use the CV Refresh activity:
When CV Refresh executes, it instructs the parent CV Screen Scope to capture a fresh screenshot of the target window.
The new screenshot is transmitted to the CV engine and re-analyzed.
The in-memory visual tree cache is immediately updated with the new controls and text labels, allowing subsequent CV activities to interact with the updated screen seamlessly.
Scope Tuning Best Practices
Optimize OCR Engine Selection:
UiPath Screen OCR: Specifically optimized for digital screen fonts, anti-aliased text, and low-contrast UI displays. Default and recommended for virtually all VDI and remote desktop automations.
OmniPage OCR / Google Cloud OCR: Ideal when remote applications display scanned document previews, PDF attachments, or non-standard character sets.
Configure Scope Informer & Accuracy:
Adjust accuracy settings (a scale of 0.0 to 1.0 ) based on environmental stability. Lowering accuracy helps with heavily compressed video streams, while raising it prevents false-positive matches on crowded interfaces.
Isolate Scope Boundaries:
Avoid setting the CV Screen Scope to monitor an entire 4K multi-monitor desktop. Constrain the scope's target selector to the specific application window or sub-panel being automated. This minimizes image payload sizes, reduces network transmission latency, and accelerates neural inference times.