🧱
Databricks
Delta Lake & Lakehouse Architecture
🧱
🧱
Databricks · Section 10 of 18

Delta Lake & Lakehouse Architecture

🔒

This section is locked

Unlock every deep-dive, lab, mock interview, and memory map across all 10 topics.

View Plans — from ₹299/month

Already have a plan? Sign in

Delta Lake & Lakehouse Architecture

Focus: Transaction log, MERGE scenarios, optimization, architecture decisions Complete Delta Guide: For the full standalone Delta Lake page (32 questions), see /learn/delta

Memory Map

🧠 T → Transaction Log (the brain — _delta_log/)
DELTA & LAKEHOUSE→TACOVS-L
─────────────────────────────
TTransaction Log (the brain — _delta_log/)
AACID (Atomicity, Consistency, Isolation, Durability)
CCommands (MERGE, OPTIMIZE, VACUUM, Z-ORDER)
OOptimization (data skipping, file statistics, compaction)
VVersioning (Time Travel, RESTORE, DESCRIBE HISTORY)
SSchema (enforcement, evolution, column mapping)
LLakehouse (Bronze → Silver → Gold, open format, unified)

SECTION 1: DELTA LAKE INTERNALS

Q1: What is the Delta Lake transaction log (_delta_log)? Explain how it ensures ACID transactions.

Simple Explanation: Think of a bank ledger. Every deposit, withdrawal, and transfer is recorded in order. If someone asks "what was the balance yesterday?", you replay the ledger up to yesterday. The Delta transaction log works the same way — it records every change to your table in numbered JSON files.

Answer: The _delta_log/ directory is an ordered record of every transaction performed on a Delta table.

Structure:

🗂️my_table/
_delta_log/
00000000000000000000.json ← Commit 0
00000000000000000001.json ← Commit 1
...
00000000000000000010.checkpoint.parquet ← Checkpoint at commit 10
_last_checkpoint ← Points to latest checkpoint
part-00000-abc123.snappy.parquet
part-00001-def456.snappy.parquet
...

Each JSON commit file contains:

  • add actions: New Parquet files added
  • remove actions: Files logically deleted (still physically present until VACUUM)
  • metaData: Schema changes, table properties
  • protocol: Reader/writer version
  • commitInfo: Timestamp, operation, user, metrics

How ACID is ensured:

  • **Atomicit
← Previous Next →