ExamFinch.com My Study

Study guide

1.2 Data Transfer and Migration Tools

Practice this exam

Google Cloud Associate Data Practitioner Exam

Track your progress as you read

Finished sections get a check mark.

1 Introduction & Exam Overview

About the Associate Data Practitioner Exam

2 Chapter 1: Data Processing Paradigms & Ingestion Strategies

1.1 ETL, ELT, and ETLT Transformation Paradigms 1.2 Data Transfer and Migration Tools 1.3 Data Quality, Cleaning, and Validation

3 Chapter 2: Data Formats & Storage Solution Selection

2.1 File Formats & Serialization Characteristics 2.2 Cloud Storage Buckets, Location Types & Storage Classes 2.3 Selecting Google Cloud Databases and Storage Services

4 Chapter 3: Loading Data & Modern Ingestion Architectures

3.1 Batch Loading Data into BigQuery 3.2 Streaming Ingestion & Deduplication Patterns 3.3 BigQuery External Tables, BigLake & Federated Queries 3.4 Schema Definition & Evolution for Loaded Data

5 Chapter 4: BigQuery SQL Analytics & Query Foundations

4.1 BigQuery SQL Syntax & Analytical Functions 4.2 Complex Data Types: Arrays, Structs, and JSON 4.3 Answering Business Questions with BigQuery: Worked Analyses

6 Chapter 5: BigQuery Performance Optimization & Cost Governance

5.1 Table Partitioning Strategies 5.2 Table Clustering & Data Skipping 5.3 Query Optimization, Execution Plans, and Cost Controls

7 Chapter 6: Interactive Analysis, Notebooks & Visual Presentation

6.1 Exploratory Analysis with Jupyter & Colab Enterprise 6.2 Looker Studio (Data Studio) Reporting & Dashboard Design 6.3 Enterprise BI with Looker & LookML Foundations 6.4 Building, Modifying & Sharing Looker Dashboards

8 Chapter 7: Machine Learning with BigQuery ML & Vertex AI

7.1 Planning an ML Project: BigQuery ML, AutoML, or Pretrained Models? 7.2 Creating, Training & Evaluating Models with BigQuery ML 7.3 Running Inference and Organizing Models in Model Registry 7.4 Using Pretrained Gemini Models from BigQuery with Remote Connections

9 Chapter 8: Pipeline Engines & Transformation Services

8.1 Transformation Engine Selection Matrix 8.2 Apache Beam & Cloud Dataflow Core Concepts 8.3 Stream Processing: Windowing, Watermarks, and Triggers

10 Chapter 9: Workflow Orchestration, Automation & Monitoring

9.1 SQL Pipelines with Dataform & Scheduled Queries 9.2 Workflow Orchestration with Cloud Composer & Workflows 9.3 Event-Driven Ingestion: Pub/Sub to BigQuery & Eventarc Triggers 9.4 Monitoring Pipelines: Dataflow Job UI, Cloud Logging & Cloud Monitoring

11 Chapter 10: Access Control, Governance & Data Sharing

10.1 IAM Principles & Role Hierarchies for Data Services 10.2 Cloud Storage Security & Access Controls 10.3 Fine-Grained Security: Column-Level & Row-Level Controls 10.4 Secure Data Sharing with Analytics Hub (BigQuery Sharing)

12 Chapter 11: Lifecycle Management, Reliability & Compliance

11.1 Object Lifecycle Management & Storage Tiering 11.2 High Availability & Disaster Recovery Architectures 11.3 Data Retention, Time Travel, and Point-in-Time Recovery 11.4 Encryption Key Management & Privacy Compliance

1.2 Data Transfer and Migration Tools

Key Takeaways Explain This Topic AI

Storage Transfer Service (STS) is a managed, serverless service for migrating object data from cloud providers (AWS S3, Azure Blob) and on-premises POSIX file systems to Cloud Storage.

Transfer Appliance (TA40 or TA300, about 40 TB or 300 TB each) is Google's offline option, recommended when uploading over the network would take more than about one week.

The transfer feasibility formula [Transfer Time = Data Volume / Effective Network Bandwidth] dictates whether online transfer (STS, gcloud storage) or offline transfer (Transfer Appliance) is viable.

Database Migration Service (DMS) moves MySQL, PostgreSQL, SQL Server, and Oracle databases into Cloud SQL or AlloyDB with minimal downtime by pairing an initial load with continuous change data capture (CDC).

Last updated: October 2026

Data Transfer and Migration Tools

Core Focus: Choosing the optimal data transfer mechanism requires balancing data volume, available network bandwidth, migration timelines, and data change velocity. Google Cloud provides specialized tools ranging from serverless cloud-to-cloud transfer services to ruggedized offline shipping appliances and continuous database replication engines.

Migrating enterprise data to Google Cloud is rarely a one-size-fits-all endeavor. Organizations must transfer diverse workloads: petabytes of unstructured archives stored on-premises, active object buckets in third-party clouds like AWS and Azure, and mission-critical relational databases that cannot tolerate extended downtime. Google Cloud provides four primary transfer vectors: Storage Transfer Service (STS) , Transfer Appliance , Database Migration Service (DMS) , and command-line utilities ( gcloud storage ).

Google Cloud Data Transfer Portfolio: An Overview

Before diving into service architectures, we can categorize transfer tools across two critical operational axes: online vs. offline and object/file vs. structured database .

Service Migration Type Primary Source Locations Primary Target Destinations Typical Data Scale

Storage Transfer Service (STS) Online (Managed Network) Amazon S3 and S3-compatible storage, Azure Blob Storage, Cloud Storage, HTTP/S URL lists, on-premises POSIX file systems and HDFS (agent-based) Cloud Storage (and POSIX file systems through agents) Tens of Gigabytes to Multiple Petabytes

Transfer Appliance Offline (Physical Shipping) On-premises data centers, edge facilities, NAS/SAN storage Cloud Storage Tens of terabytes to multiple petabytes

Database Migration Service (DMS) Online (Continuous CDC) On-premises, other clouds (for example Amazon RDS), Compute Engine (MySQL, PostgreSQL, SQL Server, Oracle) Cloud SQL, AlloyDB for PostgreSQL Relational databases of all sizes

CLI ( gcloud storage ) Online (Ad-hoc / Scripted) Local file systems, workstations, on-premises servers Cloud Storage Under a few Terabytes (ad-hoc / dev)

Storage Transfer Service (STS)

Storage Transfer Service (STS) is Google Cloud's fully managed, serverless solution for migrating large-scale unstructured object and file data into Cloud Storage. STS eliminates the operational burden of provisioning virtual machines, writing transfer scripts, monitoring failures, and managing network retries.

1. Cloud-to-Cloud Transfers

STS natively integrates with external cloud object stores, allowing direct data movement without routing bytes through the customer's on-premises network or intermediate Compute Engine instances:

Supported Sources: Amazon Simple Storage Service (AWS S3), Microsoft Azure Blob Storage, public or authenticated HTTP/HTTPS endpoints, and inter-bucket Cloud Storage copies (e.g., cross-region or cross-project).

Serverless Architecture: Google manages the underlying compute fleet. Data moves directly across high-speed cloud interconnects into Google Cloud Storage.

Security & Integrity: Data is encrypted in transit using TLS. STS performs automated checksum validation (comparing MD5 or CRC32c hashes) to verify that target objects match the source byte-for-byte before marking a transfer successful.

2. On-Premises File Transfers (Agent-Based)

To migrate data from on-premises Network Attached Storage (NAS), Storage Area Networks (SAN), or local POSIX file systems, STS utilizes an agent-based architecture :

Agent Deployment: Lightweight Docker container agents are installed on on-premises host machines that have read access to the local file storage (NFS/CIFS/POSIX).

Agent Pools: Multiple agents can be grouped into an Agent Pool . The STS control plane coordinates work across the pool, automatically balancing file chunks across agents to maximize throughput. If an agent crashes, other agents in the pool automatically absorb the workload.

Bandwidth Throttling: Network administrators can configure bandwidth caps on agent pools (for example, cap the pool at 2 Gbps while business traffic is heavy and raise the limit later). This prevents migration traffic from overwhelming business-critical WAN traffic.

Incremental & Scheduled Sync: STS can run on automated cron schedules, copying only new or modified files since the last execution. It can also be configured to delete source files upon confirmed ingestion to free up local storage.

Transfer Appliance: High-Capacity Offline Migration

When data volumes reach dozens of terabytes or multiple petabytes, physical network constraints often render online transfers impractical. In these scenarios, Google Cloud provides the Transfer Appliance .

What is Transfer Appliance?

Transfer Appliance is a ruggedized, high-capacity, rack-mountable storage device that Google ships directly to the customer's data center. Customers connect the appliance to their local high-speed network (10 Gbps RJ45, or 40 Gbps QSFP+ or 100 Gbps QSFP28 ports depending on the model version), capture data locally at high throughput, and ship the physical appliance back to a Google facility where data is ingested directly into Cloud Storage.

Models and Timeline

TA40: holds roughly 40 TB of encrypted data (rackable or freestanding).

TA300: holds roughly 300 TB of encrypted data, for data center migrations.

Multiple appliances: for multi-petabyte migrations, order several appliances to parallelize the copy.

Cycle time: Google lists an end-to-end cycle (delivery, copy, return shipping, upload) of roughly three weeks .

Security and Custody Architecture

Because physical appliances traverse commercial logistics networks, Google enforces rigorous cryptographic and physical security protocols:

Customer-managed encryption key (CMEK): Data written to the appliance is encrypted with AES-256 using a Cloud KMS key that you control, so the data is unreadable without your key.

Device protections: Remote attestation with Secure Boot and a unique PIN protect the appliance while it is in transit.

NIST 800-88 erasure: After Google uploads and verifies your data in Cloud Storage, the appliance is securely wiped to NIST 800-88 guidelines, and you can request a wipe certificate.

The Network Bandwidth vs. Migration Timeline Calculation

A fundamental competency for a Google Cloud Data Practitioner is determining whether a dataset should be transferred online via the network or offline via Transfer Appliance.

The Transfer Duration Formula

The theoretical time required to transfer data over a network is calculated as: Theoretical Transfer Time (seconds) = Data Size (in bits) / Available Bandwidth (in bits per second)

Because enterprise networks experience packet overhead, protocol latency (TCP slow start and window sizing), and shared concurrent traffic, realistic planning must apply a Network Efficiency Factor (typically 0.70 to 0.80, representing 70% to 80% network utilization).

Therefore, the operational formula is: Practical Transfer Time (seconds) = (Total Bytes * 8) / (Bandwidth (bps) * Efficiency Factor)

Calculating Real-World Migration Times

Let us analyze three typical enterprise scenarios moving 50 Terabytes (TB) of data (50 TB = 50 * 10^12 bytes * 8 = 400 * 10^12 bits):

Over a 100 Mbps Uplink (80% efficiency = 80 Mbps):

  • Practical Time = (400 * 10^12 bits) / (80 * 10^6 bps) = 5,000,000 seconds
  • In days: 5,000,000 / 86,400 ≈ 57.9 days

Evaluation: Transferring 50 TB over a 100 Mbps link takes nearly two continuous months, completely saturating the uplink. Transfer Appliance is mandatory.

Over a 1 Gbps Uplink (80% efficiency = 800 Mbps):

  • Practical Time = (400 * 10^12 bits) / (800 * 10^6 bps) = 500,000 seconds
  • In days: 500,000 / 86,400 ≈ 5.8 days

Evaluation: If the organization can dedicate the entire 1 Gbps connection exclusively to migration, an online transfer completes in under a week. However, if only 200 Mbps can be allocated during business hours, the timeline stretches to nearly a month.

Over a 10 Gbps Dedicated Interconnect (80% efficiency = 8 Gbps):

  • Practical Time = (400 * 10^12 bits) / (8 * 10^9 bps) = 50,000 seconds
  • In hours: 50,000 / 3,600 ≈ 13.9 hours

Evaluation: With a 10 Gbps pipe, 50 TB transfers in less than 14 hours. Storage Transfer Service is the ideal solution; ordering a physical appliance would be slower due to shipping logistics.

Network Transfer Time Reference Guide

The following table outlines realistic transfer durations across data scales and network speeds (assuming ~75% effective throughput):

Data Volume 100 Mbps Link (~75 Mbps net) 1 Gbps Link (~750 Mbps net) 10 Gbps Interconnect (~7.5 Gbps net) Recommended Transfer Strategy

  • 1 TB ~30 hours ~3 hours ~18 minutes gcloud storage or Storage Transfer Service
  • 10 TB ~12.5 days ~30 hours ~3 hours Storage Transfer Service
  • 50 TB ~62 days ~6.2 days ~15 hours STS (if link >= 1 Gbps) or Transfer Appliance (if link <= 100 Mbps)
  • 200 TB ~248 days ~25 days ~2.5 days Transfer Appliance (unless 10 Gbps dedicated pipe exists)

1 PB (1,000 TB) ~3.4 years ~125 days ~12.5 days Transfer Appliance Fleet (or dedicated 10+ Gbps Cloud Interconnect)

Decision Threshold: Google's guidance is that Transfer Appliance is a good fit when uploading the data over the network would take more than one week . Because an appliance round trip takes about three weeks, compare that cycle with the online estimate before you choose.

Command-Line Tools: gcloud storage vs. Legacy gsutil

For ad-hoc, developer-driven, or scriptable file movements, Google Cloud provides command-line interfaces.

The Superiority of gcloud storage

Historically, developers used the standalone Python-based CLI tool gsutil . Google has replaced gsutil with the modern gcloud storage CLI component integrated directly into the Google Cloud SDK:

Performance: gcloud storage parallelizes uploads and downloads by default; Google reported transfers up to 94% faster than legacy gsutil for large datasets.

Unified Syntax: Standardizes commands across the gcloud ecosystem ( gcloud storage cp , gcloud storage rsync ).

Parallel composite uploads: When enabled, files above a configurable size threshold are split into parts, uploaded in parallel, and composed into a single object in Cloud Storage. # High-speed parallel recursive directory synchronization using gcloud storage

  • gcloud storage rsync -r ./local_dataset gs://analytics-landing-bucket/datasets/
  • When NOT to Use CLI Tools

While gcloud storage is fast, it relies on the compute, memory, and network connection of the machine executing the command. It should not be used for large enterprise data center migrations (> 10 TB) because it lacks centralized scheduling, distributed agent coordination, automatic fault recovery across servers, and managed cloud-to-cloud offloading. For large migrations, always choose Storage Transfer Service .

Database Migration Service (DMS)

When migrating relational databases, file transfer tools (like STS or gcloud storage ) are inadequate because operational databases cannot undergo days of offline downtime while full backups are copied and restored.

Database Migration Service (DMS) is Google Cloud's serverless solution for migrating relational databases to Cloud SQL and AlloyDB with minimal application downtime.

Key Capabilities of DMS

Homogeneous and heterogeneous migrations:

Homogeneous (same engine): MySQL -> Cloud SQL for MySQL; PostgreSQL -> Cloud SQL for PostgreSQL or AlloyDB for PostgreSQL; SQL Server -> Cloud SQL for SQL Server.

Heterogeneous (engine change, with schema and code conversion): Oracle -> Cloud SQL for PostgreSQL or AlloyDB for PostgreSQL; SQL Server -> Cloud SQL for PostgreSQL or AlloyDB for PostgreSQL.

Serverless Execution: DMS requires no virtual machines or replication instances to provision or manage. The replication infrastructure scales dynamically behind the scenes.

Change Data Capture (CDC):

DMS initiates an initial full data dump (baseline snapshot) without locking database write operations.

Once the baseline snapshot is restored in the target Cloud SQL or AlloyDB instance, DMS establishes continuous replication using native database replication logs:

MySQL: Binary logging ( binlog ).

PostgreSQL: logical replication through the pglogical extension.

The target database stays in near real-time sync with the source database as transactions continue to occur.

Minimal-Downtime Cutover: When the target database replication lag approaches zero, administrators schedule a brief cutover window (typically minutes):

Stop writes on the source application.

Allow DMS to replicate the final delta transactions.

Promote the Cloud SQL or AlloyDB replica to a standalone primary database.

Update the application connection string to the new Cloud SQL endpoint and resume operations.

Choosing the Extraction Tool

The exam guide names four extraction tools. Match each one to the source and the amount of transformation needed:

Tool Best source What it does Typical exam cue

BigQuery Data Transfer Service (DTS) SaaS and Google sources (Google Ads, Campaign Manager 360, YouTube, Google Play), Amazon S3, Azure Blob Storage, Cloud Storage, Teradata and Amazon Redshift Scheduled, managed loads straight into BigQuery with no code "Load our Google Ads data into BigQuery every day without writing a pipeline"

Database Migration Service (DMS) MySQL, PostgreSQL, SQL Server, Oracle databases Initial load plus CDC into Cloud SQL or AlloyDB, then a short cutover "Move the production database to Cloud SQL with minimal downtime"

Dataflow Any source with an Apache Beam connector (Pub/Sub, Kafka, JDBC databases, files) Code-based batch or streaming extraction with transformation in flight "Parse, validate, and enrich records while they stream into BigQuery"

Cloud Data Fusion Hundreds of prebuilt connectors (databases, SaaS such as Salesforce, files) Visual, low-code pipelines that run on Dataproc "Analysts with no coding skills must build the pipeline"

Storage Transfer Service and Transfer Appliance move files and objects ; DMS moves databases ; DTS lands data in BigQuery on a schedule; Dataflow and Data Fusion extract and transform in one pipeline.

Common Exam Traps and Scenarios

Exam Tip: Look carefully at three variables in migration questions: (1) Data volume , (2) Available network bandwidth , and (3) Tolerance for downtime . A relational database requires DMS, an S3 bucket requires STS, and 100 TB over a slow 100 Mbps uplink requires Transfer Appliance.

Trap 1: Attempting to Use Transfer Appliance for Cloud-to-Cloud Migration

The Trap: Recommending Transfer Appliance to move 200 TB of data from an Amazon Web Services S3 bucket to Google Cloud Storage.

The Reality: Transfer Appliance is a physical hardware box that must be physically cabled to a local on-premises network rack. You cannot ship physical hardware to an AWS or Azure data center. Cloud-to-cloud migrations must use Storage Transfer Service , which leverages high-speed fiber backbones between cloud providers.

Trap 2: Recommending pg_dump and Cloud Storage for Production Database Migrations

The Trap: Using pg_dump to export a 2 TB active transactional database to Cloud Storage, then importing it into Cloud SQL.

The Reality: An active transactional database will accumulate thousands of changes while the export, transfer, and import take place. Unless the business can tolerate hours or days of complete downtime, this approach results in severe data loss or stale records. Database Migration Service (DMS) with continuous CDC is required.

  • Trap 3: Recommending Transfer Appliance for Small Datasets
  • The Trap: Choosing Transfer Appliance for 4 TB of data because the team wants "maximum physical security."

The Reality: Requesting, receiving, copying, returning, and ingesting an appliance takes about three weeks end to end. A 4 TB dataset moves over a 1 Gbps link at 80% efficiency in about 11 hours (32 × 10^12 bits ÷ 800 Mbps ≈ 40,000 seconds) with Storage Transfer Service, which also encrypts data in transit.