🐍 Python

Think of a list as a column you keep editing. Think of a tuple as a fixed row shape, like (user_id, event_time). A set is for membership tests: "is this id already seen?" A dict is for lookup by name: row["country"]. In a pipeline, lists grow as you collect records, sets drop duplicates, and dicts hold config or a join map. Sets and dicts need hashable items or keys.

x = [1] then x.append(2) edits the same list. s = "ab" then s += "c" makes a new string. That difference matters when two names point at one object. If a function appends to a list you passed in, the caller sees the change. If a function does n = n + 1, the caller's number does not change. Production bug: putting a list inside a set, or using a list as a dict key, fails because a list is not hashable.

[1, 2] == [1, 2] is True because the values match. [1, 2] is [1, 2] is False because they are two lists. x is None is the normal None check. CPython caches small integers, so 256 is 256 may be True while larger ints may not. Do not use that cache in production. In data jobs, compare file names and ids with ==, and compare to missing with is None.

[x * 2 for x in nums] keeps every answer in memory. (x * 2 for x in nums) does not. You can loop the generator, but you cannot index it, and you cannot walk it twice. For a 10 GB log, a generator keeps memory flat. For a small lookup table you will reuse, a list is simpler. Pandas and Spark are different tools. This choice is about plain Python loops.

b = a.copy() on a list of lists still shares the inner lists. Change b[0][0] and a changes too. copy.deepcopy(a) breaks that link. Deep copy is slower and can fail on open files or Spark objects. In ETL, copy a nested JSON config before a task mutates it. For a flat list of ints, shallow copy is enough.

def load(path, *args, **kwargs) can take extra flags without listing every name. Helpers like pandas.read_csv(path, **opts) pass a config dict through. Order in a signature is: normal args, *args, keyword-only args, **kwargs. In production, prefer real names for required settings. Use kwargs for optional passthrough, then log the keys you accepted.

@timer above def run(): means run = timer(run). The wrapper can run code before and after the real function. functools.wraps keeps the original name and docstring. In a data job, a retry decorator can catch a flaky API. Keep wrappers thin. Heavy logic inside a decorator is hard to test and hard to see in a stack trace.

CPython uses the GIL to keep memory management simple. During time.sleep or a network read, the lock can be released, so other threads run. A pure-Python JSON parse on a huge string will not use four cores with threading. Use multiprocessing, a native library, Spark, or a warehouse SQL job for CPU work. Pandas groupby is often still one process. Spark executors are processes, not Python threads.

def rows(): yield 1; yield 2 does not build [1, 2] first. next() or a for-loop pulls values. That is how you stream a large file line by line. Generators are iterators, so they run once. If you need the data twice, materialize a list or open the file again. In ETL, a generator pipeline can read, clean, and write without holding the full dataset.

You can loop a list many times. You can loop a file or a generator only once. zip, map, and enumerate return iterators in Python 3. for x in data calls iter(data) for you. In production, do not pass a spent iterator into a second stage and wonder why output is empty. If a function should be reusable, take a path or a list, not a half-consumed file handle.

Open a file in try, parse in the same block, and handle ValueError in except. Put "parse succeeded" logging in else, so you do not log success after a caught error. Close resources in finally, or better, use with. Catch specific errors. except Exception can hide bugs. Never use bare except:. In jobs, fail loudly on schema errors, and retry only on timeout or connection errors.

with open(path) as f: opens the file and closes it even if the body raises. You can write your own with __enter__ and __exit__, or with contextlib.contextmanager. In production, always close sockets, temp files, and Spark sessions this way. __exit__ can see the exception. Return True only if you really handled it. Swallowing errors in __exit__ hides failures.

A thread pool is good for many HTTP calls or many small file reads. A process pool is good for heavy JSON, regex, or pandas on chunks. Process data must be pickled to the child, so huge objects are expensive to send. Shared memory and queues exist, but they add complexity. Data engineers usually scale with Spark, warehouse compute, or many container workers, not with 32 Python threads in one task.

async def functions are coroutines. await pauses until a network or timer finishes. Thousands of API calls can share one event loop. CPU work still blocks the loop, so do not parse a giant file inside a coroutine without an executor. Most batch ETL does not need asyncio. Use it for many slow HTTP lookups. Mixing blocking requests with asyncio is a common production stall.

A normal instance stores fields in a dict. That is flexible and a bit heavy. __slots__ = ("id", "ts") stores those fields more tightly. dataclasses.dataclass(slots=True) does the same with less boilerplate. Use this for huge in-memory event objects. Do not use it for everyday config classes. Multiple inheritance with slots is tricky. Pandas and Spark already store columns densely, so slots rarely beat a DataFrame.

@dataclass class Job: name: str; retries: int = 3 gives you Job("load", 5). Frozen dataclasses can be hashable if every field is hashable, so they can be dict keys. Prefer dataclasses over a raw dict when the shape is known. Prefer them over namedtuple when you want defaults and methods. In pipelines, keep them for config and small rows, not for million-row tables.

def get_country(row: dict[str, str]) -> str | None documents a lookup that can miss. Tools like mypy or pyright catch mistakes before a job runs. Runtime is still dynamic, so a bad CSV can still pass a hint. In production, pair hints with real validation for external data. Keep hints honest. list[Any] hides the problem you were trying to show.

def add(item, bucket=[]): appends into the same list forever. add(1) then add(2) returns [1, 2] on the second call. The fix is bucket=None and if bucket is None: bucket = []. This bites helper functions that collect errors or rows. In production it looks like "state leaked between DAG runs" even in one process.

f.read() loads the whole file. That is fine for small JSON. For logs, for line in f: streams. Always set encoding. Default encoding is not the same on every OS. Binary files need "rb". In jobs, prefer a context manager so a parse error still closes the handle. For cloud storage, use the vendor client, then the same line-by-line idea.

A dict becomes a JSON object. A list becomes an array. None becomes null. Tuples dump as arrays, so they come back as lists. Dates need default=str or an encoder, and you must parse them back. dumps is not the same as dump. In pipelines, pretty indent=2 is for humans. Compact JSON is smaller for storage. Watch trailing commas. Standard JSON does not allow them.

CSV looks simple and is not. Encoding, delimiter, header names, and quoting all matter. DictReader maps the header row to values. Empty fields come through as empty strings, not None. In production, check the delimiter. Europe often uses ;. For huge files, stream rows. For analytics, pandas or Spark is easier, but you should still know the stdlib for small jobs and unit tests.

from collections import Counter then Counter(["a", "a", "b"]) is {"a": 2, "b": 1}. Missing keys return 0, not KeyError. You can add counters. This is the fast answer for word count, duplicate ids, and status-code histograms. For a 20 GB file, count in a loop or in Spark. Do not load every token into a giant list first if you can stream.

heappush and heappop keep the smallest at index 0. For the k largest, Python has nlargest. For the k smallest, nsmallest. You can store tuples like (priority, item). For top 10 amounts in 10 million rows, a heap of size 10 beats sorting the whole list. Pandas nlargest does the same idea on a column. heapq is the stdlib version.

chain(a, b) walks two iterables as one. islice(rows, 100) takes a head without slicing a giant list. groupby groups consecutive keys only, so sort first if you need all matching keys together. product can explode into a huge cartesian set, so be careful. In production, groupby on already-sorted log partitions is cheap. A dict of lists is simpler if the data is small and unsorted.

while (line := f.readline()) != "": reads and checks in one place. if (m := pat.search(text)): saves the match without a second search. That avoids double work. Keep it readable. Nested walrus in a huge comprehension is hard to debug. In interviews, show one clean use. In production, prefer a plain assignment if the line is already busy.

match status: case 200: ... case 404: ... case _: ... is a clean switch. It can also do case {"level": "ERROR", "job": job}:. That is useful for event payloads. Patterns can bind names. A subject that matches a pattern does not need a chain of if/elif. In older runtimes, this syntax is a SyntaxError, so check the platform version before you ship it.

name = "sales"; f"table={name}" becomes "table=sales". {x=} is a debug form that prints the name and value. F-strings are usually faster and clearer than % or .format(). Do not build SQL or shell commands by sticking raw user text into an f-string. That is injection. Use parameters for SQL and shlex or argument lists for commands.

f(*args) is like listing each item. f(**{"a": 1}) is like f(a=1). merged = {**a, **b} copies keys, with b winning on clashes. In Python 3.9+, a | b also merges dicts. Star unpacking in a call must match the function signature. In data code, head, *tail = parts is a clean way to split a log line after you already split it.

sorted(rows, key=lambda r: r["amount"], reverse=True) sorts high amount first. For more than one column, return a tuple: key=lambda r: (r["country"], -r["amount"]). Stability means you can sort by amount, then by country, in two passes if you want. sort only works on lists. Use sorted for tuples, dict keys, or any iterable. In production, say how ties are broken.

To sum amount by country, do totals[country] += amount. That is the pandas groupby idea without pandas. Keep the grain clear: one output row per group key. If you need all member rows, store lists. If you only need a sum, store a number. For huge data, this dict must fit in memory, or you move the group to Spark or SQL.

if rows: is true when there is at least one row. if rows is None: is true only for missing. Those are different. any([]) is False and all([]) is True, because there is no counterexample. In ETL, a missing column and an empty string are not the same. Do not write if not value when 0 is a valid amount.

{"a": 1} is fine. {[1, 2]: "x"} raises TypeError. A tuple is hashable only if every item inside is hashable. (1, 2) can be a key. ([1], 2) cannot. This is why join maps use ids and names, not whole nested records, unless you freeze them. In production, pick a natural key that will not change, like an id, not a mutable row dict.

If you are collecting rows, rows.append(row) is correct. If you are concatenating two lists, rows.extend(more) or rows += more is correct. append of a list is a common accidental nest. That later breaks a flatten or a DataFrame constructor. + on lists makes a new list. extend edits in place.

sorted(rows, key=lambda r: r["ts"]) sorts by timestamp. lambda x: x * 2 is the same idea as def double(x): return x * 2. If the body needs if/else blocks or a name in a stack trace, write a real def. In production, named functions are easier to test. Interviewers still expect you to read a lambda on a key=.

python job.py sets __name__ to "__main__" and runs the loader. import job from a test does not start the load. That split lets you reuse functions. In production, keep side effects like "read a file and write output" under this guard, or under a real CLI. Spark and Airflow should import functions, not import a module that starts a job on contact.

"data engineer interview" becomes "interview engineer data". split() with no argument collapses extra spaces, which is what you want for messy logs. If you reverse the whole string with [::-1], letters flip too. In production, decide if punctuation stays attached to a word.

1. First we split the text into a list of words.
2. Then we reverse that list.
3. Then we join the words with a single space and print the result.

"listen" and "silent" match. "python" and "typhon" match. "python" and "typhoon" do not. Sorting both strings also works, but counting is linear. In data quality, this is the same idea as "same multiset of tokens."

1. First we lower-case both strings and remove spaces.
2. Then we count characters in each string.
3. Then we compare the two count maps.

This is O(n) time and O(n) extra space. Nested loops are O(n^2) and fail interviews on large lists. Ask if values can repeat and if you may use the same index twice. The usual rule is two different indexes.

1. First we make an empty dict of seen values.
2. Then we compute the partner each number would need.
3. Then we return the two indexes when that partner is already in the dict.

[1, [2, [3, 4], 5]] becomes [1, 2, 3, 4, 5]. Ask what "nested" means. Tuples? Dicts? Only lists? In ETL this shows up when JSON arrays sit inside arrays. Recursion depth in Python is limited, so a stack is safer for wild payloads.

1. First we put the top-level items on a stack.
2. Then we pop one item at a time.
3. Then we either expand a nested list or append a real value.

This is the classic MapReduce example in one process. For a file, stream lines and update one Counter. Do not build one giant string if the file is huge. Decide what a word is: split on whitespace, or strip punctuation.

1. First we lower-case the text.
2. Then we split it into words.
3. Then we count the words and print totals plus the top word.

A status-code counter or a country-amount map often needs "top 3." Say how ties break. sorted is stable, but equal counts keep original order only if you sort the pairs that way. For a million keys and N=5, use a heap.

1. First we walk the dict as (key, count) pairs.
2. Then we ask the heap for the three largest counts.
3. Then we print those pairs.

Config merge is a daily data-eng task: defaults, then job overrides. Ask if nested dicts should merge recursively. The | operator does one level only. Nested JSON usually needs a recursive merge or a library.

1. First we copy the base dict.
2. Then we write override keys on top.
3. Then we print the merged config.

Interviews love a line like 2024-01-02 10:11:12 ERROR job=load msg=timeout. Named groups are clearer than index numbers. In production, log formats drift. Count parse failures. Do not crash the whole job on one bad line unless the contract says the file is strict.

1. First we compile a regex with timestamp, level, and message groups.
2. Then we match each line.
3. Then we keep the parsed dicts and skip lines that do not match.

Windows show up in moving averages, "errors in last 5 minutes," and time-series features. If the window is time-based, not count-based, you need a deque of timestamps. Here the window is a fixed length.

1. First we sum the first k numbers.
2. Then we slide one step: add the new right value and subtract the old left value.
3. Then we collect each window sum.

Say the grain: one output row per country, or per (country, day). defaultdict(int) avoids KeyError. For more than one stat, store a small dict or a dataclass per key. If the data cannot fit in memory, do this in SQL or Spark instead.

1. First we make a dict that defaults missing keys to 0.
2. Then we add each row's amount into its country.
3. Then we print one total per country.

This is a favorite live-coding prompt. time.time() can jump if the clock changes. perf_counter is for durations. In production, send the metric to logs or Prometheus, and do not print in tight loops.

1. First we wrap the function with @timed.
2. Then we start a clock, call the real function, and stop the clock.
3. Then we print the duration and return the original result.

Read lines, drop blanks, parse, then filter errors. Each function is small and testable. If you need to walk the data twice, this pattern is the wrong tool. Materialize or re-read.

1. First we yield lines from the text.
2. Then we drop blanks, split out the level, and keep ERROR rows.
3. Then we materialize the last generator with list so we can print it.

read() is fine for small files. Line iteration is the production habit for logs and CSV. Always set encoding. Close happens when the with block ends, even on error.

1. First we loop each line from the file-like object.
2. Then we strip whitespace and skip empty lines.
3. Then we return the cleaned names.

set(items) drops duplicates but does not promise the original order as a data API you should rely on. The seen set is O(1) membership. Items must be hashable. For dict rows, dedup on an id field, not on the whole dict.

1. First we make an empty set of values we have already kept.
2. Then we append an item only when it is new.
3. Then we print the order-preserving unique list.

Sorting and picking index -2 is easy and O(n log n). One pass is O(n). If every value is the same, there is no second largest. Say that case out loud.

1. First we walk each number while holding the top and second-top unique values.
2. Then we promote the old maximum when we see a bigger number.
3. Then we return the second value, or None if it does not exist.

"abba" is a palindrome. "abca" is not. Two pointers from each end also work and use no extra copy. For interview speed, s == s[::-1] is enough after you clean the string.

1. First we drop spaces and punctuation and lower-case the rest.
2. Then we reverse that cleaned string.
3. Then we check equality.

A generator does not build the whole sequence first. That is the point. Do not use recursion with no cache in an interview unless they ask. It is exponential and hits the call stack.

1. First we start with 0 and 1.
2. Then we yield the left number n times.
3. Then we shift the pair so the next left number is ready.

Empty input should return no chunks. n <= 0 should be rejected. In production, a generator of chunks is nicer than a list of lists if the source is huge.

1. First we reject a non-positive chunk size.
2. Then we walk start indexes 0, 2, 4, ...
3. Then we slice each block, including a short last block.

Nested payloads from APIs do not load cleanly into a table. Flattening makes one row with dotted columns. Lists need an index in the path. In production, exploding lists into child tables can be cleaner than one ultra-wide row.

1. First we look at the current object type.
2. Then we recurse into dict keys and list indexes while building a path.
3. Then we store each leaf under that path.

This is the most famous Python output question. The fix is target=None and a new list inside the function. Interviewers want the reason, not only the printout.

1. First we append into a default list that lives on the function object.
2. Then we call it three times and see the list grow.
3. Then we show the None sentinel fix, which makes a new list per call.

[0] * 3 is fine because ints are immutable. [[]] * 3 is not fine because lists are mutable. This shows up when people build a matrix.

1. First we build a matrix with [[]] * 3, which repeats one inner list.
2. Then we append to the first row and see all rows change.
3. Then we rebuild with a comprehension so each row is a new list.

This tests if/elif order, not math skill. In data terms it is a simple case mapping. A later match/case version is fine on Python 3.10+.

1. First we walk numbers from 1 through n.
2. Then we test 15, then 3, then 5.
3. Then we collect the labels.

Ask whether the list is unsorted and whether n is len(nums) + 1. Do not sort unless you need to. A set difference is clear but uses extra memory.

1. First we figure out n from the list length plus one.
2. Then we compute the sum that 1..n should have.
3. Then we subtract the real sum to get the missing value.

{v: k for k, v in d.items()} is the unique-value case. For grouping "value -> [keys]", use defaultdict(list). Values must be hashable to become keys.

1. First we swap keys and values with a comprehension.
2. Then we show a grouping version that keeps every original key.
3. Then we print both results so the clash behavior is visible.

This is the pandas sort_values idea. Missing keys raise KeyError, so use .get if the schema is messy. Mixing None and numbers can TypeError on some Python versions.

1. First we start with unsorted product rows.
2. Then we sort by amount descending.
3. Then we print the ranked list.

Event logs often replay the same id. "Keep last" means the latest payload wins. If you also need last-seen order in the output, rebuild from a list at the end, or use a dict and accept first-seen key order with last values. Interviewers care that you say which order you return.

1. First we key a dict by id.
2. Then we overwrite with each later row for the same id.
3. Then we return the dict values.

StringIO stands in for a file. Production CSV needs encoding and a delimiter check. Empty fields are "", not None.

1. First we wrap CSV text in a file-like object.
2. Then DictReader maps the header to each row.
3. Then we keep US rows and convert amount to int.

This is the classic two-pointer string question. O(n) time. If you restart from zero on every repeat, you miss a longer window in the middle.

1. First we grow a right pointer through the string.
2. Then we jump the left pointer past a repeated character.
3. Then we store the longest window we saw.

Good for "top error codes" too. most_common() with no n returns all, sorted by count. Ties follow first-seen order in CPython's Counter.

1. First we keep letters and ignore other characters.
2. Then we count them.
3. Then we ask Counter for the top n.

You do not need to store the whole stream sorted. This is how you take "top 10 amounts today" from a file. nlargest does this for you when the data is already a list.

1. First we push each number onto a min-heap.
2. Then we pop when the heap is larger than k, so only large values remain.
3. Then we sort those k values high to low.

This matches "runs in a log": many ERROR lines in a row. If the file is not sorted by the key, use a dict group instead. The group iterator is lazy and dies when you move to the next key.

1. First we sort rows by job so equal keys are adjacent.
2. Then groupby yields each consecutive job run.
3. Then we count the rows in that run.

Event routers and log severity maps are the real use. If the runtime is older than 3.10, this is a SyntaxError, so mention the version.

1. First we match the event dict against patterns.
2. Then we bind job when the level is ERROR.
3. Then we return a label, including a default.

Log lines and CSV rows often have a head and a tail of extra fields. Catching extras is safer than hard indexes.

1. First we unpack a row into job, level, and leftover payload.
2. Then we merge two dicts with **.
3. Then we print both results.

This is a pandas rolling mean without pandas. Say if you want only full windows, or also partial tail windows. Full windows are simpler.

1. First we sum the first k values and divide by k.
2. Then we slide the window with add and subtract.
3. Then we append each average.

"aaabbc" becomes "a3b2c1". Ask if a count of 1 should be omitted. Compression is only shorter when runs are long. This is not a general compressor.

1. First we start a run with the first character.
2. Then we grow the count while the character stays the same.
3. Then we flush each run, including the last one.

Same idea as a Spark broadcast hash join, just in memory. If keys repeat, store lists in the map. If the map will not fit, use a real warehouse join.

1. First we index the user rows by user_id.
2. Then we walk events and look up matching users.
3. Then we merge each matching pair into one dict.

Useful for "balance after each transaction" and for simple CDF-style charts. Watch the first row. Empty input should return an empty list.

1. First we start the total at 0.
2. Then we add each number in order.
3. Then we store the total after every step.

This is why functions.append(lambda: i) in a for-loop is a trap. Default args bind at definition. In production, use functools.partial or a real def with a parameter.

1. First we store lambdas that read the loop variable i.
2. Then we call them after the loop, so they all see i == 2.
3. Then we bind n=i as a default so each function keeps its own number.