English word tokenizer

Write codeMedium25 minOpenAI style

Your task

Tokenize a line of plain English into word tokens. A word is a maximal run of letters (A-Z, a-z) and apostrophes ('), where an apostrophe only counts as part of a word if it sits BETWEEN two letters (so contractions like don't and possessives like cat's stay whole, but leading/trailing quotes are dropped). All other characters (digits, spaces, punctuation) are separators. Lowercase every token. Return the list of word tokens in order.

Examples

tokenize_words(text: str) -> list[str]
Example 1
Inputtokenize_words("Don't stop, it's fine.")
Output["don't", "stop", "it's", "fine"]