https://www.fernandoux.com/tags/costs/Recent content in Costs on Fernando RuizHugoen-ushttps://www.fernandoux.com/en/wiki/ai/tokens/Mon, 01 Jan 0001 00:00:00 +0000https://www.fernandoux.com/en/wiki/ai/tokens/<div class="info-panel"> <div class="info-header"> <span class="material-symbols-outlined info-panel-icon">info</span> <span class="info-panel-label">Quick Definition</span> </div> <div class="info-content"> A <strong>token</strong> is the smallest unit of text a language model reads and writes. It can be a whole word, a fragment of a word or a punctuation mark. AI providers measure the size of each request, and its cost, in tokens. </div> </div> <h2 id="what-is-a-token">What is a token?</h2> <p>A model does not see letters or words the way we do. Before processing a text, a component called the <strong>tokenizer</strong> splits it into pieces from a fixed vocabulary. A frequent word is usually a single token; a rare, long or misspelled word is split into several. Numbers, spaces and emojis also take up tokens.</p>