All projects

CompleteUpdated Added

LLM Token Explorer

A browser-based workbench that runs small language models entirely on your device to show tokenization, next-token probabilities, the full token vocabulary, and token-by-token sampling.

Note: Desktop Recommended

Visit LLM Token Explorer (opens in a new tab)Visit GitHub Repo (opens in a new tab)

About this project

I made a little webapp to run small LLM AI models directly in a browser. No API calls are needed and installing additional software is not necessary. It is primarily meant to be a learning tool to help gain a better appreciation for the inference engine that runs to generate new tokens in a response when using an AI agent.

The app lets you enter any text into the context window, shows you the tokenization of the text into tokens, displays the probability distribution of potential next tokens, enables browsing through the entire token lexicon, and lets you iteratively sample tokens from the model you are running.

Screenshots

The context panel showing the text split into color-coded token chips with IDs and the inspector for one token (view full size)
The context split into its exact token IDs, with one token open in the inspector.
A bar chart and table comparing the original and adjusted probabilities of the most likely next tokens (view full size)
Next-token probabilities before and after temperature and top-k adjustments.
The full vocabulary table filtered to tokens containing "Par", with IDs, logits and probabilities (view full size)
Searching the model's full 49,152-token vocabulary.
A short story prompt extended with sampled tokens, and the history panel listing each sampled token with its probability (view full size)
Sampling a story one token at a time, with every step recorded in an undoable history.

Resources