ExamFinch.com My Study

Study guide

8.2 Dates, Unix Epoch, and Time-Zone Conversions

Practice this exam

Databricks Certified Associate Developer for Apache Spark — Free Study Guide 2026 Exam

Track your progress as you read

Finished sections get a check mark.

1 Chapter 1: Spark Architecture and Execution

1.1 Spark Benefits, Modules, and DataFrame/Dataset Concepts 1.2 Driver, Executors, and Cluster Managers 1.3 Execution Hierarchy: Applications, Jobs, Stages, and Tasks 1.4 Lazy Evaluation, Transformations vs Actions, and Resilient Lineage

2 Chapter 2: Shuffles, Memory and Persistence

2.1 Narrow vs Wide Dependencies and the Shuffle Mechanism 2.2 Memory Management, Fault Tolerance, and Garbage Collection 2.3 Caching, Persistence, and StorageLevels

3 Chapter 3: Spark SQL Sessions, Tables and Views

3.1 SparkSession, Catalogs, Databases, and Namespaces 3.2 Managed vs External Tables and CTAS 3.3 Views: Temporary vs Global Temporary Views

4 Chapter 4: SQL Sources and Stored Layout

4.1 JDBC Reads, Writes, and Parallelism 4.2 Direct File SQL, Save Modes, and Persistent Layout

5 Chapter 5: SQL Queries and Analytics

5.1 SQL Queries, CTEs, Subqueries, and Built-in Expressions 5.2 Built-in SQL Expressions and an Integrated Query 5.3 Window Functions and Complex Joins in Spark SQL 5.4 SQL Join Semantics and Integrated Analytics

6 Chapter 6: DataFrame Construction and Columns

6.1 DataFrame Construction, Schemas, and StructTypes 6.2 Column Operations: Selecting, Renaming, Deriving, and Casting 6.3 Complex Nested Types and User-Defined Functions

7 Chapter 7: DataFrame Validation and Inspection

7.1 Row Operations: Filtering, Sorting, and Deduplication 7.2 Handling Missing and Null Data 7.3 Schema Inspection, Row Iteration, and List Conversion

8 Chapter 8: Aggregations and Dates

8.1 Grouping, Multi-Dimensional Aggregations, and Pivoting 8.2 Dates, Unix Epoch, and Time-Zone Conversions

9 Chapter 9: DataFrame Joins and File IO

9.1 DataFrame Joins and Combining Datasets 9.2 Data Sources, SaveModes, and Partitioning

10 Chapter 10: Shared Variables and Stateful Processing

10.1 Broadcast Variables, Accumulators, and Closures 10.2 Stateless UDFs, Stateful Operators, and StateStores

11 Chapter 11: Troubleshooting and Tuning

11.1 Diagnosing Performance with the Spark UI & Execution Plans 11.2 Data Skew and Partition Tuning: repartition vs coalesce 11.3 Join Optimization and Broadcast Hash Joins

12 Chapter 12: Structured Streaming

12.1 Streaming Architecture, Sources, and Sinks 12.2 Output Modes, Triggers, and Checkpointing 12.3 Event-Time Processing, Watermarking, and Windowing 12.4 Streaming Deduplication With and Without Watermarks

13 Chapter 13: Spark Connect and Deployment

13.1 Spark Connect Architecture & Client-Server Decoupling 13.2 Working with Spark Connect: Connections and Best Practices

14 Chapter 14: Pandas API on Spark

14.1 Pandas API on Apache Spark: Core Architecture and Syntax 14.2 Pandas-on-Spark Operations, Indexing, and Optimization

8.2 Dates, Unix Epoch, and Time-Zone Conversions

Key Takeaways Explain This Topic AI

from_unixtime accepts epoch seconds and returns a formatted string in the session time zone.

Epoch milliseconds must be converted deliberately; Spark 3.5 provides timestamp_millis for millisecond inputs.

DATE has no time-of-day/time-zone, while TIMESTAMP rendering and parsing can depend on the session time zone.

datediff(end, start) returns day difference; extract year/month/day with date functions rather than arbitrary substring rules.

Last updated: October 2026

Start with the input type and unit

A DateType value represents a calendar date without a time-of-day or time-zone. A timestamp represents a time value whose interpretation depends on the timestamp type and session configuration; TimestampType normally describes an instant rendered in the session time zone. TimestampNTZType represents a local date-time without a time zone. A string such as "2026-10-16" is text until parsed or cast. Formatting a timestamp produces text, not a date-typed column.

Unix epoch is a numeric count from the UTC epoch. APIs commonly send seconds or milliseconds, and confusing them produces dramatically wrong dates. A ten-digit-looking example is not enough proof of units; read the data contract. Do not divide an already second-valued timestamp by 1000, or pass milliseconds unchanged to a seconds function. Precision and overflow also depend on the chosen numeric type.

  • Seconds to a date string from pyspark.sql import functions as F
  • spark.conf.set("spark.sql.session.timeZone", "UTC")
  • raw = spark.createDataFrame([(0,), (86400,)], "epoch_seconds LONG")
  • converted = raw.select(
  • "epoch_seconds",
  • F.from_unixtime("epoch_seconds", "yyyy-MM-dd HH:mm:ss").alias("utc_text"),
  • F.to_date(F.from_unixtime("epoch_seconds")).alias("event_date")
  • )
  • converted.orderBy("epoch_seconds").show()
  • # 0 -> 1970-01-01 00:00:00 and date 1970-01-01
  • # 86400 -> 1970-01-02 00:00:00 and date 1970-01-02

There are 24 × 60 × 60 = 86,400 seconds in the illustrated UTC day. from_unixtime returns a string using the session zone, so the explicit UTC setting is part of the example. A different zone may show a prior calendar date for epoch zero. The epoch still describes the same instant; the displayed calendar fields differ. This is why a quiz must state the zone when asking for a precise rendered date.

Milliseconds and timestamps

Spark 3.5 offers timestamp_seconds, timestamp_millis and timestamp_micros to create timestamps from specified epoch units. Choose the one matching the input contract. raw_ms = spark.createDataFrame([(1000,)], "epoch_ms LONG")

  • typed = raw_ms.select(
  • F.timestamp_millis("epoch_ms").alias("event_ts")
  • )
  • # In UTC, 1000 milliseconds -> 1970-01-01 00:00:01

Dividing integer milliseconds by 1000 before from_unixtime can lose fractional precision if the calculation or cast truncates. For second-resolution output that may be intentional; for subsecond event ordering it may be wrong. Avoid converting repeatedly between timestamp and strings when the subsequent calculation needs timestamp semantics. Keep typed values for joins, filters and date/time calculations.

  • Parse with an explicit pattern text_dates = spark.createDataFrame(
  • [("16/10/2026", "2026-10-16 09:30:00")],
  • "calendar_text STRING, time_text STRING"
  • )
  • parsed = text_dates.select(
  • F.to_date("calendar_text", "dd/MM/yyyy").alias("day"),
  • F.to_timestamp("time_text", "yyyy-MM-dd HH:mm:ss").alias("ts")
  • )

Pattern letters are case-sensitive: MM is month while mm is minute. The displayed pattern must match the data order. Do not guess whether 03/04/2026 means March 4 or April 3 from locale assumptions. State the expected pattern, then measure failed parses. A syntactically valid date can still violate a business range, such as a delivery date before an order date.

Invalid-input behavior varies with ANSI mode and parser policy. In configurations that raise on malformed input, an action can fail instead of yielding null. Spark 3.5 has try_to_timestamp for tolerant timestamp parsing; inspect the installed API when choosing a tolerant function. Explicitly distinguish "we reject invalid input" from "we keep it with a null parsed value" and preserve raw text when diagnostics require it.

Calendar components and arithmetic

  • Function Example intent Output meaning
  • year, month, dayofmonth Extract date components Integer calendar fields
  • date_add(day, 7) Seven calendar days later DATE
  • date_sub(day, 7) Seven calendar days earlier DATE
  • datediff(end, start) Elapsed date difference Integer days, with argument order
  • add_months(day, 1) Calendar-month shift Month-end behavior matters
  • date_format(ts, pattern) Display/report formatting STRING
  • date_trunc("month", ts) Timestamp month boundary TIMESTAMP

For example, datediff(to_date(lit("2026-10-16")), to_date(lit("2026-10-13"))) returns 3. Reversing the arguments returns -3. Adding a calendar month is not the same as adding thirty days: month lengths vary, and end-of-month rules must be considered. A midnight boundary computed in a local zone is not necessarily a fixed UTC-hour offset through daylight-saving changes.

A robust event pipeline

Suppose an API supplies event_ms and a user-facing local display date. Parse event_ms into a timestamp using its documented unit, retain the original numeric value for traceability, and derive a reporting date in the required zone. Validate missing values, unit ranges and impossible future/past values before aggregation. Two users can see different local dates for the same instant; decide whether the report groups by UTC day, business-location day or user-local day.

Use date functions rather than hard-coded substrings when the source is already typed. If a requirement asks for a date string, format deliberately; if it asks for DateType, use to_date and inspect the schema. See date/time function APIs and datetime patterns .