7.3 Schema Inspection, Row Iteration, and List Conversion

Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Spark Architecture and Execution

1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage

2 Chapter 2: Shuffles, Memory and Persistence

2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels

3 Chapter 3: Spark SQL Sessions, Tables and Views

3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views

4 Chapter 4: SQL Sources and Stored Layout

4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout

5 Chapter 5: SQL Queries and Analytics

5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics

6 Chapter 6: DataFrame Construction and Columns

6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions

7 Chapter 7: DataFrame Validation and Inspection

7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion

8 Chapter 8: Aggregations and Dates

8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions

9 Chapter 9: DataFrame Joins and File IO

9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning

10 Chapter 10: Shared Variables and Stateful Processing

10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores

11 Chapter 11: Troubleshooting and Tuning

11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins

12 Chapter 12: Structured Streaming

12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks

13 Chapter 13: Spark Connect and Deployment

13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices

14 Chapter 14: Pandas API on Spark

14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization

7.3 Schema Inspection, Row Iteration, and List Conversion

Key Takeaways Explain This Topic AI

printSchema prints and returns None; schema is a StructType property and columns/dtypes are local metadata.

collect returns a local list of Row objects; toLocalIterator consumes rows incrementally but can still need memory for a large partition.

Use explicit sorting and tie breakers when a bounded sample must be deterministic.

foreach runs task-side code and is unsuitable for populating a driver-side Python list.

Last updated: October 2026

Inspect structure before values

Exam questions often mix a schema operation with an action. A DataFrame has named SQL columns and a plan; it is not a local list of dictionaries. Read the method's return type and purpose before deciding whether it prints, returns metadata, requests distributed results or builds another DataFrame. from pyspark.sql import functions as F

• people = spark.createDataFrame(
• [(2, "Bo", None), (1, "Ari", 20), (3, "Cy", 30)],
• "id INT, name STRING, score INT"
• )
• people.printSchema()
• schema_object = people.schema
• column_names = people.columns
• name_type_pairs = people.dtypes

printSchema() prints a schema tree and returns None. schema is a StructType property, not a method called schema(). columns is a list of names; dtypes is a list of name/type-string pairs. Neither columns nor dtypes provides the actual column values. A local dictionary can map field names to types when needed, but it does not validate every record's business meaning. A reader may have already performed schema discovery, so "metadata inspection" does not imply the source was never contacted.

Display versus return

• API Result in Python Main caution
• show(n) None; prints a table Preview may truncate text unless configured
• take(n) List of at most n Rows Bound result width as well as row count
• first() One Row, or None for empty data No stable first row without ordering
• head(n) List when n supplied head() without n has a different scalar shape
• collect() List of all Rows Entire result must fit local memory/limits
• limit(n) New DataFrame A transformation, not a returned row list

Calling print(people.show()) prints the table and then None. Calling rows = people.limit(2) stores a DataFrame, not two Python rows. A question asking for a list of rows requires an action such as take or collect on a bounded result. A question asking for another distributed DataFrame instead needs a transformation.

• Deterministic small conversion ordered = people.orderBy(F.col("score").desc_nulls_last(), F.col("id").asc())
• rows = ordered.limit(2).collect()
• row_dicts = [row.asDict(recursive=True) for row in rows]
• name_scores = [(row["name"], row["score"]) for row in rows]
• # name_scores -> [("Cy", 30), ("Ari", 20)]

The explicit tie breaker id makes the selected order stable if it is unique. A sort by score alone leaves equal-score ordering unspecified. dropDuplicates followed by a sort cannot recover a particular duplicate row already discarded; use row_number over a fully specified window when survivor identity matters. Null ordering is explicit here rather than inferred from an API default.

Row supports field access by name and positional access, but duplicate column names after a join can make conversion ambiguous. Resolve the schema first by joining on named shared keys or selecting aliased columns. asDict(recursive=True) converts nested Row objects to dictionaries, but does not remove array/map values or turn arbitrary complex data into a flat CSV representation. Choose a deliberate serialization format for external consumers.

Iteration and memory

For a result too large for one Python list, toLocalIterator() returns an iterator consumed in the driver/client process. In classic Spark, its memory usage can include the largest partition; prefetching may include another partition. It is not a distributed Python loop with unlimited memory safety. One enormous partition can still cause trouble, and issuing a separate external request for every iterated row may be very slow. for row in people.select("id", "name").toLocalIterator():

print(row["id"], row["name"])

This is suitable only for an intentionally manageable local consumer. Use distributed writes for large outputs. toPandas() also collects the full distributed result locally, even with Arrow enabled. Arrow can reduce conversion overhead but does not change the single-machine memory requirement. Under Connect, the receiving process is the client rather than necessarily the remote driver, so assess client memory as well.

Task-side iteration is a different tool

foreach and foreachPartition apply functions in task execution. They do not return a list of processed rows to the driver, and appending to a driver list in those functions does not create shared mutation. foreachPartition can initialize a connection once per task partition instead of once per row, but retries can repeat side effects. Use idempotent writes and consider whether an existing DataFrameWriter connector already handles the target more clearly.

Sorting does not determine concurrency: a dataset with many partitions can still run only as many simultaneous tasks as allocated task slots permit. Conversely, collecting just ten rows does not mean every upstream operation is cheap; an upstream global aggregation or join may still process the entire input. Limit the data logically and inspect the physical plan when cost matters.

Round trip and validation

A small local sequence can become a DataFrame with spark.createDataFrame(sequence, explicit_schema). Local input must fit memory before it is distributed, and the schema should match Python value representations. Empty or all-null collections often require explicit types. Compare expected row count, nullable fields and ordering after a round trip; constructing a new DataFrame from a list does not preserve the original partitioning or sorted presentation contract automatically.

See DataFrame API . For an exam code fragment, name the exact returned object at every step and locate where data crosses from distributed execution into a local collection. That habit prevents confusing schema metadata, lazy transformations and eager actions.