🧱 Databricks

Interviewers want the product, not a Spark lecture. Say compute, data, and governance as three parts. Spark still does the work. Databricks adds Unity Catalog, Photon, serverless, Auto Loader, and Jobs. You still pay cloud VMs or serverless usage plus Databricks DBUs. Production teams treat it as a platform with identity, cost tags, and audit, not as a single cluster.

A data lake stores files with weak quality and weak transactions. A warehouse is fast and governed but often copies data out of the lake. A lakehouse tries to do both: open files in the lake, plus table contracts. On Databricks that contract is Delta, governed by Unity Catalog. Cost stays lower when you do not copy the same data into a second warehouse. Governance still matters, because a lake of ungoverned files is not a lakehouse.

Think of the metastore as the region-level registry, usually one per cloud region. Catalogs often map to environments or domains, such as prod or sales. Schemas group tables for a team. You name a table as catalog.schema.table. Privileges inherit down this tree. Lineage and audit sit on the same objects. Production access should use groups and service principals, not one shared admin user.

Hive Metastore used two levels: database.table. That made prod and dev easy to mix. Three levels let you separate prod.sales.orders from dev.sales.orders. SQL, Python, and BI tools should use the full name. Cost and governance both improve when catalogs are split by environment. Do not rely on a default catalog in production jobs.

A volume lives in a schema, just like a table. The path looks like /Volumes/catalog/schema/volume/folder/file. Auto Loader, COPY INTO, and ML artifacts often read volumes. Do not create a Delta table if the data is unstructured files. Do not dump files into random DBFS folders if Unity Catalog is on. Volumes give GRANT, audit, and lineage-style governance for files. Tables give ACID, CDF, and time travel for rows.

Managed tables are the default for new lakehouse work. Predictive optimization and some layout features target Unity Catalog managed tables. External tables are for landing zones you already own, or for sharing a path with another engine. Storage credentials and external locations govern those paths. Current Unity Catalog managed drops keep data for a retention window, often about 8 days, before permanent delete. Hive and older runtimes can differ, so check the workspace docs. Cost: managed is simpler, but you still pay for the cloud files until they are gone.

The chain is credential, then location, then table or volume. The credential is often a managed identity or IAM role. Users get CREATE EXTERNAL TABLE or READ FILES on the location. They should not get the raw cloud secret. This is a governance control and a cost control, because one credential can be scoped to one prefix. If you skip this chain and mount storage with keys, Unity Catalog cannot enforce least privilege.

Hive tables can live in a workspace metastore that other workspaces do not share. Privileges are weaker and easy to bypass with paths. Unity Catalog centralizes identity and securables across workspaces attached to the same metastore. You also get volumes, row filters, column masks, and Delta Sharing. Migration is a project: names change, mounts go away, and jobs must run as a service principal. Cost does not drop by itself, but one catalog reduces duplicate copies and shadow tables.

SELECT on a table is not enough if the user cannot enter the catalog and schema. Teams often grant USE CATALOG and USE SCHEMA to a group, then SELECT only on the gold tables they need. MODIFY is write. ALL PRIVILEGES is rarely needed. Service principals used by jobs should get only what the job writes. Audit who has MANAGE or ownership. Over-granting on the catalog is a common production incident.

Row filters implement tenant isolation, such as one airline seeing only its bookings. Masks protect PII, such as email. Policies use SQL functions and group membership. They apply to notebooks, jobs, JDBC, and BI if those tools use Unity Catalog. Shared or standard access mode is required for fine-grained security on a cluster. Test with a non-admin user. There is a cost: filters can block some optimizations, so do not treat them as a substitute for splitting huge tenants into separate tables when scale demands it.

Open Catalog Explorer and look at the lineage graph. If gold revenue is wrong, walk upstream to silver and bronze. If you rename a silver column, see which gold tables break. Lineage is strongest for Spark SQL and DataFrame reads of UC tables. Some paths, such as raw JDBC in a UDF, may not show up. Governance teams use it with audit logs. It does not by itself stop a bad write.

If a job runs as alice@company.com, the job dies or stays privileged when Alice leaves. A service principal has its own grants, secrets access, and audit trail. Git deploy, REST API, and Terraform should use that identity. Grant it least privilege on the catalogs it writes. Cost tags still apply to the job, not to the person. Do not share one "admin SP" across every pipeline.

Scopes can be Databricks-backed or backed by Azure Key Vault, AWS Secrets Manager, or similar. Jobs and notebooks fetch values at runtime. Cluster environment variables can reference secrets without putting the value in source. Init scripts should not echo secrets. Rotate keys and remove user-level copies. Governance: who can READ the scope is as important as the table GRANT. Cost is not the issue here; leakage is.

Leaving an all-purpose cluster on overnight is a classic bill shock. Job clusters isolate libraries and Spark conf per pipeline. They also make retries cleaner because the machine is fresh. Interactive work still needs all-purpose or serverless compute. Use auto-terminate. Tag both kinds for chargeback. Do not "save money" by pinning every team to one giant always-on cluster unless you have measured it.

Warehouses size with T-shirt sizes and auto-stop. Serverless warehouses start fast and scale with query load. Photon is commonly on. You still pay for warehouse uptime, so auto-stop matters. Do not point every Python pipeline at a warehouse. Do not point Power BI at a random all-purpose cluster. Governance is the same Unity Catalog, which is the point: one table, many compute engines.

Serverless exists for SQL warehouses, jobs, and notebooks depending on the workspace. You lose some low-level Spark knobs. You gain less ops work and fewer idle VMs. Cost can go up if queries are sloppy, because scale-out is easy. Cost can go down if you used to leave clusters idle. Tag workloads. Set budgets and query limits. Serverless is not a free tier.

Photon sits under the same Spark SQL plan. You still write DataFrames. Look at the query profile for Photon operators. There is a DBU multiplier, so faster runtime must beat extra DBU price. Python row-at-a-time UDFs often fall off Photon. Prefer built-in functions. Production advice: turn Photon on for SQL and ETL, measure wall time and cost, and keep UDFs out of the hot path.

If min equals max, you are not autoscaling. If min is high, you pay for idle capacity. Scale-up helps a sudden shuffle. Scale-down can lag because Spark still holds shuffle files. SQL warehouses autoscale differently, by cluster count. For jobs, a well-sized fixed job cluster can be cheaper than a wide min-max range. Always combine autoscaling with auto-terminate and tags.

People use init scripts for native packages, drivers, or security agents. Workspace-global init scripts are hard to debug and easy to break every cluster. Unity Catalog volumes are the current place to store the script file. Do not curl random installers from the internet in production. Do not put secrets in the script body. Cluster policies should control who can attach scripts. Libraries and compute images are cleaner when they are enough.

Policies can force cost tags, max workers, Photon, Unity Catalog access mode, and auto-terminate. This is both a cost control and a governance control. Without policies, one person can start a huge GPU cluster with no tags. Job compute should use a production policy that requires a service principal and logging. Do not treat policy as optional documentation.

A job can run a notebook, Python wheel, SQL, dbt, JAR, or pipeline. Tasks can depend on other tasks. You set schedules, timeouts, retries, and email or webhook alerts. Production should use job compute, not an always-on all-purpose cluster. Lakeflow Jobs is newer naming in some docs. The API still says jobs. Version your job definition with Asset Bundles, not clicks only.

You write @dlt.table or CREATE STREAMING TABLE plus expectations. Bronze, silver, and gold become a DAG. Streaming and batch can mix. The product manages orchestration that you would otherwise build with jobs plus custom checkpoint code. You still pay for pipeline compute. Expectations can drop, fail, or only warn. Production teams still need tests, alerts, and a service principal. It is not "no-ops Spark."

Bronze should be replayable and cheap to rebuild from files. Silver applies types, dedup, and keys. Gold is what BI and data products query. Do not skip bronze "to save storage" if you cannot replay. Do not let analysts query raw bronze as the source of truth. Unity Catalog catalogs or schemas often map to layers. Cost: extra copies have a price, but debug and GDPR replay are cheaper when bronze exists.

Directory listing mode lists the prefix. File notification mode uses queue events and scales better on huge landings. Schema lives in a schema location. Bad records can go to rescued data. Combine it with volumes, not random mounts. Exactly-once file processing depends on the checkpoint. Cost: listing every minute on millions of files is expensive, so use notifications at scale. Auto Loader is not a replacement for MERGE into silver.

Delta sinks use the checkpoint plus the transaction log for exactly-once table writes. Auto Loader file positions live there too. Put checkpoints on a volume or a dedicated storage prefix with the same governance as the table. Backfill with a new checkpoint, not by deleting prod. If you change the query in incompatible ways, you may need a new checkpoint. Cost is small compared with duplicate gold loads after a lost checkpoint.

ALWAYS blocks user inserts into that column. BY DEFAULT lets you supply a value, which is useful in backfills. Identity is per table, not global across a catalog. Clones and CTAS need care because generated values may not copy the way you expect. Production MERGE should still match on customer_id or another business key. Identity is for warehouse-style surrogate keys in gold dimensions.

Enable delta.enableChangeDataFeed. Readers see change type plus preimage and postimage for updates. This is cleaner than scanning the whole silver table every hour. Gold jobs apply those changes with MERGE. CDF is not the same as cloud object notifications. It is also not a full audit log of who ran the query. Keep enough history for your SLA. Cost: extra change files, less full-table rewrite downstream.

DESCRIBE HISTORY shows versions. Time travel is for debug, audit, and fix-forward. It is not an infinite backup. delta.logRetentionDuration and delta.deletedFileRetentionDuration matter. After VACUUM, old versions may fail. Production restores should be rare; prefer writing a new correct batch. Cost: longer retention means more storage. Governance: who can RESTORE production tables should be limited.

Small files create thousands of tiny tasks and slow listings. Compaction targets larger files, often around 1 GB, workload dependent. ZORDER is not a B-tree index. It rewrites files so min/max stats are tighter. Pick a few filter columns, not twenty. ZORDER rewrite is heavier than liquid clustering on large tables. Run it on a schedule or let predictive optimization do it for eligible managed tables. Cost is extra write I/O for faster reads.

Partition folders are rigid. ZORDER is a full rewrite when you run it. Liquid clustering is a table property plus incremental OPTIMIZE. Writes do not fully recluster by themselves. Predictive optimization can schedule OPTIMIZE on Unity Catalog managed tables. Do not mix old Hive-style partitioning with liquid clustering on the same keys. Choose columns used in filters and joins. This is a production layout choice, not a SQL hint.

Enable it at account, catalog, schema, or table level depending on the workspace. External tables and some features are not eligible. If it is on, do not also run aggressive hourly OPTIMIZE unless you measured a gap. You still choose clustering keys. You still set retention so VACUUM does not kill time travel too soon. Cost: Databricks spends compute to save later read cost. Watch the billing tables.

OPTIMIZE, MERGE, and DELETE leave old files around so readers and time travel still work. VACUUM reclaims storage. Databricks blocks very short retention unless you disable a safety check. Retention of 0 hours can break concurrent readers. Production should keep at least the time you need to recover a bad load. Cost falls after VACUUM. Recovery risk rises. Pair VACUUM with a tested restore plan.

Separate INSERT plus UPDATE is not one transaction. MERGE is. Deduplicate the source first or MERGE fails. Put the most specific WHEN MATCHED clauses first. Bound the target with a partition or filter when you can, so you do not rewrite the whole lake. MERGE can still rewrite many files, so deletion vectors and clustering matter. Production jobs should log insert/update counts.

mergeSchema can add new columns on write. overwriteSchema replaces the schema and can drop columns. Auto Loader has schema evolution modes and rescued data. Gold tables should evolve slower than bronze. Unity Catalog and downstream BI break when names change. Production: add columns in a reviewed change, backfill, then notify consumers. Cost of a surprise string-to-int change is a failed job plus bad dashboards.

You share tables, views, or volumes depending on the setup. Recipients get current data, not a weekly dump, unless you design it that way. This beats emailing parquet extracts. You still need least privilege, row filters if required, and a legal agreement. There is network and compute cost on the provider when recipients query. It is not the same as Lakehouse Federation, which pulls from foreign databases.

Some companies have both Databricks and Iceberg readers. UniForm reduces extra copies. It is not a full Iceberg feature clone, and some reader features lag. Interviews only need this brief idea. Production still needs one owner for writes, or you will corrupt the table. Cost benefit is fewer duplicated lakes.

Databricks has custom tags. Cloud resources also have tags. They do not always map 1:1, so design both. Jobs, warehouses, and pipelines all need tags. Unity Catalog has separate classification tags for PII, which is governance, not billing. Look at system billing tables to verify. Production bar: no tag, no cluster. Tags do not reduce usage by themselves; they make owners visible.

Click-ops jobs drift. Bundles keep cluster size, schedules, and grants in Git. CI impersonates a service principal and deploys. Variables change catalog names per environment. This is how you stop "it worked in my workspace." Bundles do not replace tests. They also do not freeze DBU cost if the YAML asks for huge clusters.

Users and service principals live at the account. Workspaces contain notebooks, jobs, and compute. Two workspaces can share one metastore so prod.sales.orders is the same table. They can also use different metastores, which splits governance. Do not create a metastore per workspace "just in case." Production identity should be account-level groups, not workspace-only users when UC is on.

Without them, deleting one row rewrites the entire file. That is write amplification and cost. With them, the write is a small sidecar. Tables on current Databricks often have this on. UniForm and some external readers may have limits, so check if you share files outside Databricks. VACUUM and OPTIMIZE still reclaim space. This is a production write/read trade-off, not a user-facing SQL feature.

Each file can become a task. Ten thousand 2 MB files are worse than twenty 1 GB files for many scans. Partitioning by a high-cardinality column is a common cause. Auto Loader maxFilesPerTrigger and target file size settings help bronze. Gold needs OPTIMIZE or predictive optimization. Cost shows up as longer jobs and more LIST calls on cloud storage. Do not repartition(1) huge facts as the fix.

COPY INTO tracks files it already loaded inside the target table. It is easy SQL. It can struggle when the directory has enormous listing costs. Auto Loader can use file notifications and schema evolution. Both should read from volumes. Neither one is MERGE. Bronze first, then MERGE to silver. Cost: running COPY INTO every minute on a giant bucket is the wrong tool.

Older names are single-user and shared. Newer names are dedicated and standard. Interactive teams that need table-level GRANT enforcement should use standard access mode. A pipeline that must use RDDs or special libraries may need dedicated compute as the job identity. No-isolation clusters do not fit Unity Catalog. This is a governance question first, then a compatibility question. Pick access mode in the cluster policy so users cannot bypass it.

expect logs violations. expect_or_drop filters. expect_or_fail fails the pipeline. Bronze often warns, because you want the raw row. Silver often drops or fails on keys and amounts. Gold usually fails on contract breaks. Metrics should go to alerts. Expectations are not a full test suite and not a unique constraint like a warehouse PK. Production still needs quarantine tables for dropped rows if the business must review them.

You define watermark on an event-time column, such as 2 hours. Events older than the watermark can be ignored. That is a business choice: late orders after two hours might go to a catch-up batch. Checkpoints store the state. Auto Loader watermarks are not the same thing as event-time watermarks. Production: monitor state size in the streaming UI. Cost of "infinite late data" is memory and shuffle, not just extra rows.

Static spark.sql.shuffle.partitions = 200 is a blunt default. AQE can reduce tiny tasks after a filter. It cannot fix a Python UDF or a huge cartesian join. Photon and AQE work together on SQL plans. Look at the query profile after the run, not only EXPLAIN before the run. Production: leave AQE on unless you have a measured reason. Cost savings come from fewer empty tasks and fewer wrong join strategies.

Databricks can fall back to on-demand if spot is missing. Shuffle loss from a killed worker causes retries. That can erase the savings. Streaming jobs with large state are poor spot candidates. Batch bronze loads with checkpoints are better. Tag the job and watch retry time in the run UI. Cost optimization is real only if the job still finishes inside the SLA.

dbutils.fs.mount was the old pattern. Anyone who can see the mount path can often see the files. Volumes use storage credentials and external locations, or managed storage. The same groups that own the schema own the files. Init scripts, Auto Loader, and ML artifacts can all use volumes. Migration means changing paths and removing mounts. Governance and production both get simpler after that.

A DBU is a Databricks usage unit, not a VM. All-purpose is priced higher than jobs compute. SQL warehouses bill while they are running, so auto-stop matters. Serverless removes VM idle time but not query waste. Storage is a separate cloud bill, reduced by VACUUM and by not copying every layer twice without a reason. Read system.billing.usage if it is enabled. Production FinOps: policy-enforced tags, budgets, and no unowned clusters.

Interviewers want the catalog.schema.table name, Delta, and a real schema. Location clauses make the table external. Production tables also need ownership, grants, and clustering or partitioning choices. Do not create tables in the default Hive database.

1. It creates a managed Delta table in the main catalog and silver schema.
2. It declares types and NOT NULL on keys.
3. It stores the table as Delta, so you get ACID, time travel, and MERGE.

Gold dimensions often want a warehouse key plus a source key. ALWAYS blocks users from inserting their own ids. BY DEFAULT is for backfills. Clones and reloads can confuse people who thought identity was globally unique.

1. It makes customer_sk an auto number.
2. It still stores customer_id so CDC can match the same person later.
3. The table is Delta, so SCD2 MERGE can add versions.

Generated columns help clustering and skipping when people filter on event_date but writes only have event_time. The expression must be deterministic. Do not use a Python UDF here.

1. It stores event_time as the raw timestamp.
2. It always computes event_date from that timestamp.
3. It clusters by event_date so day filters skip files.

CLUSTER BY is a layout declaration. Writes do not fully recluster by themselves. Predictive optimization can schedule OPTIMIZE on eligible Unity Catalog managed tables. Changing keys later is ALTER TABLE ... CLUSTER BY.

1. It creates a Delta table with liquid clustering on customer and date.
2. OPTIMIZE rewrites files so rows with the same keys sit together.
3. Later reads that filter on those keys can skip more files.

This is the most common Databricks coding question. Clause order matters. Without a newer-row check you can apply stale CDC. Without dedup, MERGE throws. Bound the target if the table is huge.

1. It ranks landing rows so each order_id appears once.
2. It updates silver only when the source timestamp is newer.
3. It inserts keys that silver does not have yet.
4. Both actions commit together.

CDC often sends DELETE as an operation flag. A full snapshot can use WHEN NOT MATCHED BY SOURCE. A partial incremental batch must not. Production MERGE should log operation counts.

1. It matches on order_id.
2. It deletes silver rows whose CDC event is a delete.
3. It updates remaining matches when the event is newer.
4. It inserts new keys unless the event itself is a delete.

This is the coding follow-up after a MERGE error. dropDuplicates(["order_id"]) is not enough because it is not ordered. Use updated_at or an event sequence. Then call DeltaTable.merge.

1. It ranks landing rows inside each order_id.
2. It keeps rank 1, the newest event.
3. It updates silver when the event is newer.
4. It inserts brand-new orders.

Streaming bronze and MERGE create small files. OPTIMIZE reduces task count on the next read. On liquid clustered tables, OPTIMIZE also moves data toward clustering keys. Predictive optimization may already do this on managed tables.

1. It reads the current small files.
2. It writes fewer larger files.
3. The table version moves forward. Old files remain until VACUUM.

ZORDER co-locates column values so min/max stats skip more files. Do not ZORDER 10 columns. Do not ZORDER the partition column only. Prefer liquid clustering on new Unity Catalog tables.

1. It compacts files.
2. It sorts data along customer_id and order_date.
3. Later filters on those columns can skip more files.

After OPTIMIZE and MERGE, old parquet files linger on purpose. VACUUM is the storage reclaim step. Databricks blocks retention under 7 days unless you disable a safety check. Do not disable that in prod without a reason.

1. Dry run lists candidate files and deletes nothing.
2. The real VACUUM deletes files older than 168 hours that the current table no longer needs.
3. Time travel older than that retention can fail.

Interviewers ask this to hear the risk, not the syntax. Streaming jobs, long BI queries, and VERSION AS OF need old files. Cost savings are real. Data loss is also real. Prefer 7 days plus predictive optimization.

1. It disables the safety check that blocks short retention.
2. It deletes unreferenced files with zero hours of grace.
3. It turns the check back on so later jobs are protected.

This is how you debug a bad MERGE. Compare current gold to last night. Do not build nightly BI on time travel if VACUUM is 7 days. For a lasting fix, RESTORE or write a new table.

1. History shows version numbers and timestamps.
2. The first SELECT reads version 12 without changing the table.
3. The second SELECT reads the snapshot at that time.

You will see MERGE, WRITE, OPTIMIZE, RESTORE. The operationMetrics column has files added and removed. Governance of who ran the write is here. Who queried the table is in audit logs, not this command.

1. It prints recent commits.
2. It filters to MERGE so you can find the bad load.
3. Metrics tell you how many rows and files that MERGE touched.

Restore is for a bad overwrite. After restore, tell downstream jobs. A safer pattern is clone the old version to a side table, validate, then swap. Production should restrict who can restore gold.

1. It checks history so you pick the right version.
2. It restores data files from version 12 as the current table.
3. History still shows the bad commit and the restore commit.

CDF is cheaper than scanning all of silver. You get insert, update, delete, plus preimage and postimage. Enable it before you need the first incremental window. Starting CDF in the middle does not invent old changes.

1. It enables CDF on future commits.
2. It reads the change rows between two versions.
3. _change_type tells you how to MERGE into gold.

Idempotent re-runs are the point. force reloads everything and can duplicate rows. mergeSchema adds columns. For continuous arrival or huge prefixes, Auto Loader is better.

1. It reads JSON files from a governed volume.
2. It inserts only files not already tracked by this table.
3. It allows new columns to be added on the bronze table.

This is the standard bronze pattern. Schema location is not the same as checkpoint. File notification mode is for large prefixes. Rescued data holds extra columns that do not match the schema.

1. It incrementally discovers new JSON files.
2. It infers and evolves schema from a durable schema location.
3. It writes new rows to a Delta table with exactly-once checkpoints.
4. availableNow processes pending files and stops.

Bronze usually keeps rows. Silver drops or fails on keys. Expectations are metrics plus optional filters, not a unique index. The pipeline still needs a service principal, alerts, and a catalog target.

1. It lands files into a bronze streaming table.
2. It builds silver from bronze.
3. Null order_id rows are dropped.
4. Negative amounts are counted but kept.

This is the Unity Catalog coding question. Least privilege: analysts get SELECT on gold, engineers get MODIFY on silver. Show grants after you change them. Production should not grant ALL PRIVILEGES on the catalog to a person.

1. It lets analysts open the main catalog.
2. It lets them open the gold schema.
3. It lets them read one gold table.
4. It lets the job service principal update silver orders.

Shuffle partitions, AQE, and Delta safety checks are common. Serverless may ignore some classic cluster knobs. Do not hide secrets in Spark conf. Put durable settings in the job cluster YAML.

1. It turns AQE on for this session.
2. It avoids a hard-coded 200 shuffle partitions.
3. It keeps short VACUUM blocked.
4. It prints the AQE flag so you can show the interviewer.

Use deep clone for a durable sandbox or migration. Use shallow clone for a quick dev table. Clones are new tables with their own history from that point. Grants do not automatically copy the same way people expect, so grant again.

1. Deep clone writes a standalone copy.
2. Shallow clone creates a new table name that points at existing files.
3. Version clone freezes last night's snapshot into a new table.

This is for legacy lakes. After convert, you can MERGE and time travel. Concurrent writers during convert are unsafe. New tables should be created as Delta, not converted later.

1. It writes Delta transaction log files beside the parquet files.
2. It records order_date partitions in that log.
3. It exposes the converted path as a Unity Catalog table.

Volumes govern files the same way tables govern rows. Managed volumes use Unity Catalog storage. External volumes use an external location. This is the current landing-zone pattern. Analysts should not need the cloud key.

1. It creates a governed volume named landing.
2. It grants the job service principal file read access.
3. It copies new JSON files from /Volumes/... into a Delta bronze table.

Delta rejects unknown columns by default. Auto Loader and bronze often need additive evolution. Downstream MERGE must then select the new columns or ignore them. Production: add the column in a review, then turn this on for one job, then notify consumers.

1. It reads the latest JSON files.
2. It appends rows to bronze.
3. If coupon_code is new, Delta adds that column instead of failing.

This is how people accidentally delete a gold column and break BI. Prefer additive mergeSchema, or CTAS into a new table and swap. Unity Catalog permissions stay on the table name, but downstream queries still break.

1. It selects a smaller set of columns.
2. It overwrites a sandbox table.
3. The table schema becomes exactly those three columns.

This is the classic warehouse coding question on Databricks. Hash tracked columns so you do not close a row when nothing changed. Identity columns can fill the surrogate key. Do not overwrite the old address. Audit reports need the old row.

1. It hashes email and address so unchanged customers are skipped.
2. MERGE closes the current row when the hash changes.
3. MERGE inserts customers who never existed.
4. INSERT adds a new current version for the customers who just closed.

The checkpoint stores offsets. The Delta log stores table commits. Together they give exactly-once table writes. availableNow is the usual batch-style trigger on Databricks. Production jobs should fail if the checkpoint path is missing, not create a second one silently.

1. It reads new bronze rows as a stream.
2. It writes them into a silver Delta table.
3. It stores offsets under a volume checkpoint so reruns do not duplicate.
4. availableNow processes pending data and stops.

Jobs should use a service principal that can read the scope. Source code should contain the scope name and key name only. Cloud-backed scopes are easier to rotate. If you print or display the secret, it can land in logs.

1. It fetches the JDBC URL and password from a secret scope.
2. It reads the source database without putting the password in Git.
3. It writes the result to a bronze Delta table.

Masks are governance, not a view people can skip. Account groups drive the CASE. Shared or standard access mode is required for this to apply on a cluster. There is some query cost. Do not mask in Python and leave SQL open.

1. It creates a masking function in the gold schema.
2. It attaches that function to email.
3. Later SELECTs rewrite the column based on group membership.

This is why liquid clustering beat Hive partitions for many tables. Predictive optimization can OPTIMIZE eligible managed tables for you. Choose keys from real filters. Do not change keys every week.

1. It declares new liquid clustering columns.
2. OPTIMIZE rewrites data files toward those columns.
3. Reads that filter on customer and date can skip more files after the rewrite.

CTAS is good for gold rebuilds. CREATE OR REPLACE TABLE is destructive for readers during the swap, so use it with a schedule and a contract. Managed UC tables keep the data lifecycle in the catalog.

1. It aggregates completed silver orders by date.
2. It writes a managed Delta gold table.
3. It clusters by order_date for dashboard filters.

This is safer than overwriting the whole silver table. It is still a data rewrite for that slice. Combine with a volume landing of the fixed files. Time travel can undo a bad replace if you have not vacuumed.

1. It deletes the current 2026-09-15 slice from silver as part of the write.
2. It inserts the reprocessed rows for that date.
3. Other dates are not rewritten.

Append mode cannot update existing keys. MERGE can. foreachBatch is not exactly-once unless the MERGE is idempotent and the checkpoint is safe. Use availableNow for scheduled catch-up. Watch small files.

1. Each microbatch is a DataFrame of new bronze rows.
2. The function keeps the newest event per order_id.
3. MERGE updates silver or inserts new keys.
4. The checkpoint records which bronze data was already applied.

This is cheaper than rebuilding gold every hour. You must enable CDF on silver first. Skip update_preimage if you only need the latest row. Deletes need a delete clause. If VACUUM removed the versions, you must full-refresh.

1. It finds the latest silver version.
2. It reads CDF rows after the last applied version.
3. It keeps inserts, update postimages, and deletes.
4. It MERGEs those changes into a gold current table.

Photon and AQE still benefit from file stats and table stats. Data skipping uses min/max in the Delta log. ANALYZE helps cost-based choices such as broadcast. Run it after a large load if predictive optimization is off.

1. It computes stats on every column.
2. The optimizer can pick better joins later.
3. DESCRIBE DETAIL shows size, files, and location.

Delta does not give you a warehouse primary key with indexes. Constraints still catch bad ETL early. Put heavy quality rules in DLT expectations or tests too. Informational constraints exist in some engines; on Delta, prefer enforced ones you have verified.

1. It rejects rows with a null order_id.
2. It rejects rows with amount <= 0.
3. Future MERGE and INSERT have to pass both rules.

Cost tags only help if you query them. Join identity metadata to see the service principal. Finance wants daily rollups, not a screenshot of one cluster. Storage is a separate cloud bill.

1. It reads the last 7 days of Databricks usage.
2. It groups by sku, identity, and cost tags.
3. It sorts so the most expensive lines show first.

This cuts write amplification and DBU on MERGE-heavy silver tables. Readers apply the marks, so too many leftover vectors can slow scans. OPTIMIZE and REORG clean up. Check UniForm and external Iceberg readers before you rely on this in a shared lake.

1. It turns on row-level delete marks for future writes.
2. MERGE and DELETE can avoid rewriting whole Parquet files.
3. REORG rewrites clean files and purges those marks.

Environment layout is a governance decision. dev and prod catalogs are easier than mixing in one schema. The creating user becomes owner unless you transfer ownership to a group. Production ownership should be a group or service principal.

1. It creates the main catalog and silver schema.
2. It lets engineers use that schema and create tables.
3. It lets the job service principal create tables there too.

This is the governance-friendly alternative to exporting parquet. You can share tables or views. Row filters still matter if the recipient should not see every tenant. Audit who has the recipient credential.

1. It creates a share object named sales_share.
2. It adds one gold table to that share.
3. It lets the acme_partner recipient SELECT those live rows.

Federation is for exploration and light joins. The source database still enforces its own passwords. UC adds another GRANT layer. If Spark pulls the whole remote table, you will pay twice and wait.

1. It stores a Postgres connection that reads secrets at runtime.
2. It mounts that database as a foreign catalog.
3. It queries one day of remote orders through Unity Catalog.

Finance reads compute tags from usage tables. Governance reads UC tags in Catalog Explorer. You need both. A job with no team tag is how bills go unexplained. Do not put secrets in tags.

1. It labels the table for catalog search and PII policy.
2. It adds a human comment for the next engineer.
3. It stamps team and env on compute usage for chargeback.