Anthropic's extravagant tokenizer complicates AI pricing

Claude looks substantially more token-hungry than OpenAI's GPT-5.x, thanks to the new tokenizer that Anthropic shipped with recent releases. Large language models (LLMs) use tokenizers to handle the mapping of text into tokens. There's no set definition of a token, but they're typically a set of three or four characters that are mapped to the integers LLMs actually process. Tokens have become the basic economic unit for billing use of AI models. Because the slicing of words into tokens and the t