ACCELERANDO-8192 — PERMANENT 8192-TOKEN GLYPH LANGUAGE

VOCABULARY VS DIMENSION
This is a tokenizer with exactly 8,192 vocabulary entries, not an 8,192-dimensional embedding model. An embedding dimension describes each token's numerical vector inside a neural network; it does not determine how many tokens appear in a song. No embedding model is needed here.

TOKENIZER
Custom byte-level BPE tokenizer, derived from the first 8,192 mergeable ranks of o200k_base and its original pretokenization pattern. All 256 single-byte values are included. It is not the full o200k_base tokenizer and is not claimed to be a frontier model tokenizer. There are no special tokens. The vocabulary covers arbitrary UTF-8 text losslessly by combining byte pieces. Some entries are partial UTF-8 sequences and need neighboring tokens to form a readable character.

MAPPING
Token ID = glyph ID, permanently, for every ID 0 through 8191. Unicode glyph character = U+F0000 + token ID. Every vocabulary entry has its own glyph whether or not it appears in the song. The full token byte definitions, tokenizer settings, and mapping are in tokenizer-8192.json. The same configuration must be used to keep encoded text compatible.

SONG RESULTS
34 lyric lines, 398 token occurrences, 132 distinct token types. The song uses a subset of the full 8,192-token vocabulary. Lyric lines were tokenized independently; headings and newlines remain document structure. Markdown links/emphasis were stripped from the supplied text. No additional lyrics were retrieved. The transcript decodes exactly to lyrics-provided.txt.

FILES
complete-8192-token-glyph-vocabulary.svg: all 8,192 token/glyph pairs, in a scalable 64-column x 128-row sheet. Labels are abbreviated when needed; each SVG title and the JSON preserve complete data. Spaces/newlines/tabs have visible markers; incomplete Unicode byte fragments are shown as hex.
tokenizer-8192.json: complete tokenizer and all 8,192 mappings.
accelerando-token-glyph-atlas-8192.png / .svg: all 398 song token occurrences plus the 132-entry song key.
accelerando-glyph-transcript.png / .svg: glyph-only song transcript.
transcript.json: source lines, token IDs, and glyph IDs.
token-key.json / .csv: subset used in this song.
glyph-transcript.txt: private-use Unicode transcript with ordinary-text headings.
Accelerando8192-Regular.ttf: glyph font, supporting U+F0000 through U+F1FFF.
decode.py: decode the formatted song transcript and check it against the supplied source; standard library only.
encode_decode.py: encode/decode arbitrary UTF-8 files with the permanent full mapping. The encode command requires tiktoken; decode is standard-library only. This stream encoder treats all text, including newlines, as tokens, so its output is a continuous glyph stream rather than the formatted song document.

VALIDATION
Exactly 8,192 distinct vocabulary byte strings; all 256 byte values available; token and glyph IDs identical; lossless multilingual round trips; song line and full-text round trips. Smaller vocabulary means more word fragments, explaining why the new song count is 398 occurrences / 132 unique tokens versus the earlier 284 / 96.
