An open-source storage layer that adds ACID transactions, schema enforcement, and time travel on top of Parquet files in a data lake (like ADLS Gen2).

In the _delta_log/ folder — a series of JSON commit files + Parquet checkpoint files.

A folder (_delta_log/) that records every change as a numbered JSON file. It's the single source of truth for the table's current state.

Atomicity (all or nothing), Consistency (schema rules enforced), Isolation (readers don't see partial writes), Durability (committed data survives crashes).

Parquet format. Delta Lake = Parquet files + transaction log. The log is what makes it "Delta."

A Parquet summary of the table state, created every 10 commits. Instead of reading 10,000 JSON files, read 1 checkpoint + recent JSONs.

A small file that tells Delta which checkpoint is the latest — so it doesn't have to scan the entire _delta_log/ folder.

When you start reading a table, you see it as it was at that exact moment — even if someone writes new data while you're reading. You always get a consistent view.

When two writers modify the same files. Example: Writer A and B both UPDATE rows in the same partition → one gets a ConcurrentModificationException and must retry.

Delta stores min/max statistics for each data file. When you query WHERE booking_date = '2026-03-15', Delta skips files where min > March 15 or max < March 15. Reads only relevant files.

First 32 columns. Configurable via delta.dataSkippingNumIndexedCols. ⚠️ Put your most-filtered columns FIRST in schema!

1. Read _last_checkpoint → find latest checkpoint
2. Read that checkpoint file (Parquet) → get base state
3. Read all JSON commits AFTER the checkpoint → apply recent changes
4. Result: current list of valid Parquet data files
5. Read only those Parquet files → return query results

1. Spark writes new Parquet file(s) to the table folder
2. Delta creates a new JSON commit in _delta_log/ with "add" action pointing to the new file(s)
3. The commit is atomic — either the JSON file is fully written, or it's not

1. Delta identifies which Parquet files contain the rows to delete
2. Reads those files, removes matching rows, writes NEW Parquet files with remaining rows
3. Creates a commit with "remove" (old files) + "add" (new files)
4. Old files are NOT physically deleted — they stay until VACUUM cleans them

NO! DELETE only marks files as "removed" in the log. The physical Parquet files remain on disk. You need VACUUM to physically delete them. This is why time travel works — old files are still there.

• Two writers start at same snapshot (version 5)
• Writer A commits first → version 6 (success)
• Writer B tries to commit → Delta checks: "Did A modify the same files I'm modifying?"
  • If NO conflict (different files/partitions) → auto-resolve, Writer B becomes version 7
  • If YES conflict (same files) → ConcurrentModificationException → retry from version 6

A single SQL command that does INSERT + UPDATE + DELETE in one atomic operation. Also called "upsert" (update + insert).

Update existing rows if they match, Insert new rows if they don't. MERGE does this in a single pass.

Yes! Add WHEN MATCHED AND source.deleted = true THEN DELETE — three operations in one command.

Use ROW_NUMBER() window function to keep only the latest record per key:

WITH deduped AS (
  SELECT *, ROW_NUMBER() OVER (
    PARTITION BY booking_id ORDER BY updated_at DESC
  ) AS rn FROM source_data
)
SELECT * FROM deduped WHERE rn = 1

When source has NEW columns that target doesn't have, MERGE can auto-add them. Enable with: SET spark.databricks.delta.schema.autoMerge.enabled = true Or: .option("mergeSchema", "true")

1. Join: Inner join target with source on the match condition
2. Classify: Each row is classified as "matched" or "not matched"
3. Write: Rewrite affected data files with updates + append new files for inserts

MERGE must scan the ENTIRE target to find matches. Optimizations:

• Liquid Clustering on merge key → scans only relevant files (biggest win)
• Filter source early → MERGE INTO target USING (SELECT * FROM source WHERE date = today)
• OPTIMIZE before MERGE → fewer small files to scan
• Low shuffle merge → reduces data movement between nodes
• Partition by merge key → prunes irrelevant partitions (old approach)

Problem: In SCD Type 2, when a record changes, you need to:

1. Close the old row (set end_date, is_current = false)
2. Insert a new row (with new values, is_current = true)

• For rows to UPDATE: merge_key = booking_id (matches target)
• For rows to INSERT: merge_key = NULL (never matches → goes to NOT MATCHED → INSERT)

Compacts many small Parquet files into fewer large files (target: ~1 GB each). Faster reads because fewer files to open.

Physically deletes old Parquet files that are no longer referenced by the transaction log. Frees up storage.

7 days (168 hours). Files older than this are deleted. ⚠️ Setting it lower breaks time travel!

Yes, but DANGEROUS — breaks time travel and concurrent reads. Only use for GDPR "right to be forgotten":

SET spark.databricks.delta.retentionDurationCheck.enabled = false;
VACUUM bookings RETAIN 0 HOURS;

Sorts data within files by specific columns so that similar values are stored together. When you filter by those columns, Delta reads far fewer files.

The new replacement for partitioning + Z-ORDER (Databricks 2024+). Automatically maintains data layout — no manual OPTIMIZE ZORDER needed. You just define clustering keys at table creation.

NO! They are mutually exclusive. Liquid Clustering replaces partitioning. You must choose one or the other.

Instead of rewriting an entire Parquet file to delete a few rows, Delta marks rows as "deleted" in a small side file. The data file stays untouched. Much faster deletes/updates.

Yes, on Databricks (since 2024). They're a table property: delta.enableDeletionVectors = true.

• When partition column has HIGH cardinality (>10,000 values) → too many tiny folders
• When table is small (<1 TB) → partitioning adds overhead
• When using Liquid Clustering → they're mutually exclusive
• Rule of thumb: Each partition should be >1 GB. If not, don't partition.

• Each file needs a separate read operation → high I/O overhead
• More files = more metadata in the transaction log → slower planning
• Cloud storage (ADLS) charges per API call → more files = higher cost
• Fix: OPTIMIZE (compaction) or Auto Loader with trigger-based batching

Ability to query previous versions of a table using version numbers or timestamps. Works because old Parquet files are kept until VACUUM removes them.

SELECT * FROM bookings VERSION AS OF 5; or TIMESTAMP AS OF '2026-03-20';

DESCRIBE HISTORY bookings; — shows every version, timestamp, operation, and user.

RESTORE TABLE bookings TO VERSION AS OF 5; — rolls back to version 5. This creates a NEW version (safe, doesn't rewrite history).

After VACUUM runs — it physically deletes old files. Default retention is 7 days. If you VACUUM with 0 hours, ALL old versions are gone immediately.

A single platform combining the cheap storage of a data lake with the ACID transactions, schema enforcement, and governance of a data warehouse. No need for separate lake + warehouse.

1. Delta Lake → ACID transactions on data lake files
2. Photon Engine → Fast SQL queries (warehouse-speed on lake data)
3. Unity Catalog → Governance, security, lineage
4. Serverless SQL Warehouses → On-demand compute, no cluster management

Databricks automatically runs OPTIMIZE and VACUUM based on table usage patterns. You don't schedule these manually anymore. Unity Catalog required.

A new feature (GA March 2026) that allows Delta tables to handle OLTP workloads (point lookups, single-row updates) — like a traditional database (PostgreSQL) but built on Delta Lake.

Allows external tools (that only speak Iceberg/Hive) to read your Delta tables without conversion. Unity Catalog rewrites metadata on-the-fly.

A new data type in Delta Lake 4.x that stores semi-structured JSON data natively — no need to stringify JSON. Faster queries on nested JSON fields.