Parsing configuration files

The idea

Nearly every settings format is the same three shapes repeated: a comment, a section header, and a key with a value. So the parser is a loop over lines that asks which of those three a line is, and then records it.

The one rule that carries most of the correctness is this: split on the first separator, not on every one. A connection string, a URL, a base64 blob, a sentence: values contain separators all the time, and a parser that splits on all of them quietly truncates the value.

Then two things turn the loop into a real parser. Quoting, because a comment marker inside quotes is data. And references, because a value that mentions another key has to be resolved after the whole file is read, which is also where a file can accidentally refer to itself forever.

How it works

One pass to read the file, a second pass to resolve references. Keeping them separate is what lets a value refer to a key defined further down, and it is why the last assignment wins even for a reference written above it.

def parse(lines):
    settings, section = {}, ""
    for raw in lines:
        line = strip_comment(raw).strip()
        if not line:
            continue
        if line.startswith("[") and line.endswith("]"):
            section = line[1:-1].strip()
            continue
        if "=" not in line:
            continue
        key, _, value = line.partition("=")       # FIRST "=" only
        settings[(section, key.strip())] = value.strip()
    return settings

def strip_comment(line):
    out, quoted = [], False
    for ch in line:
        if ch == '"':
            quoted = not quoted                    # a "#" inside quotes is data
        elif ch == "#" and not quoted:
            break
        out.append(ch)
    return "".join(out)

def resolve(settings, key, seen=None):
    seen = seen or set()
    if key in seen:
        raise ValueError(f"reference cycle at {key[1]}")
    seen = seen | {key}
    def sub(m):
        target = (key[0], m.group(1))              # same section
        return resolve(settings, target, seen) if target in settings else m.group(0)
    return re.sub(r"\$\{(\w+)\}", sub, settings[key])

Note what partition buys over split("="): it always returns three parts and never loses the tail. split("=", 1) is the same idea; split("=") is the bug.

Cost

Watch out for

• Splitting on every separator. "url=https://x/a=b".split("=") gives three parts and the value silently becomes https://x/a. Partition, or split with a count of one. This is the single most common bug in this whole family.
• Stripping comments before knowing about quotes. Removing everything after the first # destroys motd = "welcome # to prod". The comment scan has to track quoted regions, which means it is a small state machine and not a call to find.
• Trimming the key but not the value, or the reverse. port = 8080 and port=8080 must produce the same setting. Decide once, at the split, and apply it to both sides. Then decide separately whether an intentionally quoted trailing space survives.
• Resolving references during the read. If a value is expanded the moment it is parsed, a reference to a key defined later resolves to nothing, and a key reassigned later keeps its stale expansion. Read everything first, then resolve, and both problems disappear.
• Following references without a visited set. A file where a points to b, b to c, and c back to a is a perfectly valid file to write and an infinite loop to read. The recursion needs a set of keys currently being resolved, not just a depth cap.
• Section state that leaks. A section header sets state that persists to the next header. Forgetting to reset it, or treating a header line as also being an assignment, silently files settings under the wrong section, and every value still looks right in isolation.

Worked example

Take the line motd = "welcome # to prod" in a file where # starts a comment.

A naive parser cuts at the first # and stores "welcome, an unterminated quote and a lost sentence. A quote-aware scan walks the line character by character, flips a quoted flag on each ", and only treats a # as a comment when that flag is false. The whole value survives.

Now add url = https://${host}:${port}/v1 above a line that reassigns port to 9090. Because resolution runs after the read, ${port} picks up 9090, not the value that was in scope when the parser walked past that line. That is the behaviour you want, and it is only true if the two passes are actually separate.