cloud

What is DynamicFrame?

An AWS Glue-specific distributed data abstraction built on top of Apache Spark DataFrames that can handle semi-structured data with inconsistent schemas across rows.

Detailed Explanation

Unlike a Spark DataFrame which requires a uniform schema, a DynamicFrame stores a schema per row (called a DynamicRecord). This makes it ideal for reading messy JSON or CSV where some rows have extra or missing fields. Use resolveChoice() to handle columns with multiple data types, and relationalize() to flatten nested structures. Convert to a DataFrame using toDF() for full Spark SQL access.

Code Example

Examplepython

dynamic_frame = glueContext.create_dynamic_frame.from_catalog(database="db", table_name="tbl")
df = dynamic_frame.toDF()  # Convert to Spark DataFrame

AWS GlueSparksemi-structuredschema