Network & protocol parsing

The idea

When you read from a TCP socket, you might expect one recv() to equal one complete message. In reality, bytes arrive in arbitrary chunks. One chunk might contain half a message, or 2.5 messages!

To fix this, we use Protocol Framing. A common technique is Length-Prefixed Framing: before sending the message, send a fixed-size header (e.g., 4 bytes) that declares the length of the upcoming message. The receiver reads the 4-byte length, then waits until exactly that many bytes are in its buffer before processing the message.

How it works (Length-Prefixed Framing)

def read_messages(sock):
    buffer = b""
    while True:
        chunk = sock.recv(1024)
        if not chunk: break
        buffer += chunk
        
        # While we have at least 4 bytes (the header length)
        while len(buffer) >= 4:
            # Read the 4-byte header to get the message length
            msg_len = int.from_bytes(buffer[:4], "big")
            
            # Do we have the full message yet?
            if len(buffer) < 4 + msg_len:
                break # Wait for more chunks
                
            # Extract the complete message
            msg = buffer[4 : 4+msg_len]
            process(msg)
            
            # Remove the processed bytes from the buffer
            buffer = buffer[4+msg_len:]