What are tokens in AI and LLMs?

Tokens are the basic units that large language models (LLMs) use to process text. One token roughly corresponds to 4 English characters or about ¾ of a word. Both input and output are billed per token.

How does token counting work?

Different models use different tokenizers. GPT-4o uses the o200k_base encoding while GPT-4 uses cl100k_base. The same text may produce different token counts under different encodings.

Why does token count matter for API costs?

LLM APIs charge per token, typically priced per million tokens. Input and output have different rates, with output usually costing more. Understanding token usage helps optimize prompts and control costs.

Token Counter

Estimate tokens & cost

Token Counter

Encoding:

INPUT TEXT

STATS & COST

Enter text to see token stats & visualization

About AI Tokens

Tokens are the fundamental units LLMs use to process text. A token roughly maps to 4 English characters — the word "hamburger" becomes 3 tokens. Different models use different tokenizers: GPT-4o uses o200k_base while GPT-4 uses cl100k_base, producing different counts for the same input.

API costs are calculated per token — typically priced per million tokens, with output tokens costing more than input. Counting tokens before sending requests helps optimize prompts and predict costs across models like ChatGPT, Claude, and Gemini.

Ready

o200k_base

Related:String Inspector