Catching AI mistakes

Hallucinated APIs and plausible-but-wrong logic.

The idea

When reviewing code generated by an LLM, you are not looking for syntax errors (it rarely makes them). Instead, you are hunting for AI Failure Modes.

These include Hallucinations (using functions that don't exist in the library), Off-by-one errors (failing at boundary conditions), and Silent Failures (swallowing exceptions to make the code look cleaner). You must read the code with a magnifying glass.

import boto3
def delete_old_logs(bucket_name, days=30):
    s3 = boto3.client('s3')
    objects = s3.list_all_objects(Bucket=bucket_name)
    for obj in objects['Contents']:
        if obj['Age'] > days:
            try:
                s3.delete_object(Bucket=bucket_name, Key=obj['Key'])
            except Exception:
                pass # Ignore errors

This AI generated Python snippet looks perfectly valid at first glance.

How it works (Common AI Bugs)

# 1. Hallucinated APIs
# AI models predict the "most likely" next token. If a library 
# SHOULD have a 'list_all' method, the AI will invent it.
# You must verify against the actual docs (it's actually list_objects_v2).

# 2. Plausible but wrong schema
# The AI assumed S3 objects have an 'Age' property. They don't.
# They have a 'LastModified' datetime. This code crashes immediately.

# 3. Swallow-all Exceptions
# AI loves to write "except Exception: pass". This silently hides
# permission errors, network timeouts, and actual bugs. Never allow it.