Apache Airflow Overview and Interview Atlas

Answer First: Airflow is an orchestrator: it schedules DAG runs, tracks task instances, dispatches work through an executor, and records state in a metadata database.

Memory Map: DAG -> task -> run -> scheduler -> executor -> metadata DB.

Airflow in one picture

Answer First: Remember DAG-TASK-RUN: the DAG is the workflow definition, the task is one node, and the run/task instance is execution at a specific logical date.

Memory Map: DAG-TASK-RUN -> recipe -> node -> execution -> state -> retry.

Airflow is best for dependency-heavy batch and ELT workflows where visibility, retries, backfills, and operator ecosystem matter more than sub-second latency.

Airflow vs Prefect, Dagster, and Luigi

Answer First: Airflow remains the enterprise default for scheduled DAG orchestration; Prefect optimizes developer ergonomics, Dagster emphasizes assets, and Luigi is a simpler older task framework.

Memory Map: orchestrator choice -> enterprise ecosystem -> dynamic flow ergonomics -> asset lineage -> legacy simplicity -> team fit.

In interviews, choose Airflow when you need a proven scheduler with a large operator ecosystem and operational UI.

Airflow DAGs, Scheduling, and Time Semantics

Answer First: Airflow scheduling is interval-based: a run represents a logical data window and usually executes after that interval completes.

Memory Map: start_date -> schedule -> logical date -> data interval -> actual run -> catchup.

execution_date, logical_date, data interval, and start_date

Answer First: logical_date is the timestamp the run represents, not when it physically executes; data_interval_start and data_interval_end define the data window.

Memory Map: start_date gate -> logical date label -> interval start -> interval end -> run after window -> UTC awareness.

Scheduler, executor, and worker

Answer First: The scheduler parses DAGs and queues ready tasks, the executor delivers queued work, and workers run the task code.

Memory Map: parse DAGs -> create runs -> queue tasks -> executor dispatch -> worker execution -> state update.

Debug queued-but-not-running tasks by checking executor capacity, pools, worker health, and scheduler logs.

Catchup and backfill

Answer First: catchup=True creates scheduled historical runs from start_date; backfill is an explicit manual replay over a chosen date range.

Memory Map: start date history -> catchup flood -> manual backfill -> max active runs -> latest-only need -> scheduler load.

Default production posture: set catchup=False unless historical reprocessing is intentional.

@daily vs cron

Answer First: @daily is equivalent to midnight cron, while explicit cron expressions make non-standard business windows easier to read.

Memory Map: preset schedule -> cron precision -> business calendar -> timetable option -> dataset trigger -> data window.

Use datasets/timetables for event-driven or calendar-specific scheduling in modern Airflow.

Airflow Operators, Sensors, XCom, and Task Patterns

Answer First: Airflow task code is built from operators, task instances, trigger rules, sensors, XComs, and TaskFlow mapping.

Memory Map: operator class -> task node -> task instance -> dependency rule -> small XCom -> mapped fan-out.

Operator vs task vs task instance

Answer First: An operator is the class/template, a task is one operator instance in the DAG, and a task instance is one execution of that task for one DAG run.

Memory Map: operator template -> task node -> DAG run -> task instance -> state machine -> retry.

When debugging, always ask whether the problem is definition-time (task), run-time (task instance), or scheduler-time (queued state).

XCom vs Variable vs Connection vs Params

Answer First: XCom passes small per-run values between tasks, Variables store global configuration, Connections store credentials, and Params hold run-time overrides.

Memory Map: small task output -> global config -> secret connection -> run parameter -> metadata DB limit -> path not payload.

Never push DataFrames through XCom. Push object-store paths or small counters.

Trigger rules and branching

Answer First: Trigger rules decide whether a task runs after upstream success, failure, skip, or completion; branching usually needs a tolerant join like none_failed_min_one_success.

Memory Map: upstream states -> all_success default -> all_done cleanup -> branch skip -> tolerant join -> clear intent.

Use all_done for cleanup and none_failed_min_one_success for joins after a branch.

Sensors: poke, reschedule, and deferrable operators

Answer First: poke sensors hold a worker slot, reschedule sensors release the slot between checks, and deferrable operators move long waits out of workers.

Memory Map: wait condition -> poke slot cost -> reschedule release -> deferrable triggerer -> timeout -> capacity guard.

Long waits should not occupy scarce worker slots.

Airflow Reliability, Scaling, Debugging, and Production Guardrails

Answer First: Reliable Airflow depends on idempotent tasks, bounded retries, fast DAG parsing, controlled concurrency, small metadata payloads, and clear alerting.

Memory Map: idempotent task -> retry policy -> parser health -> pool capacity -> metadata hygiene -> alert.

Top-level code runs on every parse

Answer First: Heavy imports, API calls, or file reads at module import time slow the scheduler because DAG files are parsed repeatedly.

Memory Map: DAG parse loop -> module top level -> scheduler freeze -> import errors -> move I/O into task -> parse metric.

Keep DAG files declarative. Put expensive work inside task functions.

Retries for flaky APIs

Answer First: Use bounded retries with exponential backoff only when the operation is idempotent; non-idempotent APIs need a unique request ID or compensating design.

Memory Map: flaky dependency -> retry count -> exponential backoff -> max delay -> idempotency key -> alert after failure.

Retries are not a correctness mechanism by themselves; they amplify side effects if the task is not safe to repeat.

Debugging a slow DAG

Answer First: Start from the UI timeline/Gantt, then check logs, XCom size, import errors, parser stats, executor capacity, pools, and worker isolation.

Memory Map: Gantt bottleneck -> logs -> metadata size -> import errors -> queued capacity -> executor fit -> isolation option.

If tasks are queued but not running, performance tuning the task code will not help; fix capacity or scheduling limits first.

DAG runs piling up

Answer First: Pause the DAG, decide which historical runs matter, clear or mark old runs intentionally, and control future pile-up with catchup, max active runs, and latest-only patterns.

Memory Map: outage backlog -> pause DAG -> choose replay window -> mark/clear runs -> max_active_runs -> latest-only protection.

Backlog handling is a business decision plus an Airflow operation; do not blindly rerun two years of work.

Airflow Scenarios, Labs, Gotchas, and Mock Interview

Answer First: Airflow scenarios are answered by naming the data interval, the dependency graph, the retry/idempotency policy, and the operational signal you will monitor.

Memory Map: scenario -> data window -> task graph -> dependency rule -> retry guard -> monitor.

Minimal DAG with XCom

Answer First: A three-task extract-transform-load DAG proves dependency order, task instance states, XCom visibility, and UI-driven inspection.

Memory Map: extract task -> XCom row -> transform task -> load task -> UI inspection -> small payload rule.

e >> t >> l

Dynamic task mapping lab

Answer First: Dynamic task mapping expands one task definition into one task instance per input item at run time.

Memory Map: list files -> expand inputs -> parallel task instances -> mapped state -> fan-out without boilerplate.

Use mapping when cardinality is known only when the DAG run executes.

Task state machine

Answer First: Task instances move from scheduled to queued to running to success, failure, retry, skip, or reschedule; the state tells you which subsystem to debug.

Memory Map: scheduled -> queued -> running -> success/failure -> retry/reschedule -> subsystem clue.

State literacy is an interview superpower: "queued" points to capacity; "up_for_reschedule" points to sensor behavior.

Production gotchas

Answer First: Common Airflow failures come from giant XComs, heavy parse-time code, deprecated schedule style, accidental catchup floods, slot-blocking sensors, and timezone assumptions.

Memory Map: metadata bloat -> scheduler parse freeze -> schedule API drift -> catchup flood -> worker slot starvation -> timezone mismatch.

Daily ETL scenario

Answer First: A strong daily ETL design names the schedule, catchup stance, API connection, validation, transformation, warehouse load, quality check, retry policy, and alert path.

Memory Map: daily schedule -> API connection -> validate -> transform -> load -> quality check -> alert.

Keep credentials in Connections, use TaskFlow for readable Python, and put alerting on the failure callback.

Final readiness checklist

Answer First: The Airflow interview checklist is time semantics, scheduler pipeline, executors, XCom limits, trigger rules, sensors, catchup, mapping, retries, and debugging.

Memory Map: time semantics -> scheduler path -> executor choice -> XCom limit -> trigger rules -> sensor mode -> retry debug.

If you can explain every item without looking, you can survive most Airflow rounds.

Airflow Interview Questions

These questions are link-only so there is one canonical answer owner.

Q-AIR-001: Design a daily ETL that pulls from API, transforms, loads to warehouse

Answer owner: Daily ETL scenario

Alternate source wording: content/airflow/Airflow_01_Confusions_Labs_MockInterview.md#L440.

Q-AIR-002: Difference between @daily and 0 0 * * *?

Answer owner: @daily vs cron

Alternate source wording: content/airflow/Airflow_01_Confusions_Labs_MockInterview.md#L453.

Q-AIR-003: A DAG is slow, how do you debug?

Answer owner: Debugging a slow DAG

Alternate source wording: content/airflow/Airflow_01_Confusions_Labs_MockInterview.md#L463.

Q-AIR-004: How do you handle task retries for flaky API calls?

Answer owner: Retries for flaky APIs

Alternate source wording: content/airflow/Airflow_01_Confusions_Labs_MockInterview.md#L475.

Q-AIR-005: DAG runs are piling up after a weekend outage

Answer owner: DAG runs piling up

Alternate source wording: content/airflow/Airflow_01_Confusions_Labs_MockInterview.md#L491.