Split calculator formulas into typed tokens

You are building a calculator that reads formulas people type. The first step is a lexer: it splits the source text into tokens, the smallest pieces that mean something.

Write tokenize(expr) that returns the tokens of expr in source order.

Input and output:

• expr: an ASCII string of up to 10,000 characters.
• Returns a list of [type, lexeme] pairs. type is one of the five names below, and lexeme is the exact text of the token.

Rules:

• Read left to right. At each position, skip whitespace or read the longest token that fits.
• Whitespace is a space, tab, newline or carriage return. It separates tokens and is never emitted.
• NUM: digits 0 to 9 with at most one .. It starts with a digit, or with a . that has a digit right after it. A number that starts with . has already used its one dot.
• Inside a number, the first . always joins it, even when no digit follows, so a number can end with a dot. A second . ends the number.
• IDENT: a letter or _, then any mix of letters, digits and _. Letters are a to z and A to Z.
• OP: one of +, -, * and /, one character per token. A - is never part of a number.
• PAREN: ( or ), one character per token.
• UNKNOWN: any other single character, one per token. This includes a . that does not start or continue a number.
• An empty or all-whitespace string returns an empty list.