A "token" in brief.max_tokens is dira's own conservative estimate, not a model's tokenizer

The decision

`brief.max_tokens` counts **dira tokens**, defined in `internal/brief/tokens.go` and computed in the binary with no table, no network and no model:

> One per line, plus, for each line, the greater of its **word pieces** — a run > of letters counts one per four characters, a run of digits or punctuation one > per character, and a non-ASCII rune one per UTF-8 byte — and its **byte > floor**, one per three bytes of that line with its whitespace collapsed.

Two independent estimates, and the larger wins. Prose that packs into few words is caught by the byte floor; a dense line of ids, dates and punctuation is caught by the word pieces. Neither alone is safe: a word count is blind to how long the words are, and a byte count prices `dec-0002 accepted 2026-07-29` like a sentence.

Why the error direction is the whole point

An estimator that is right on average is wrong low half the time, and cst-0001 is not an average — it is a ceiling the binary asserts it did not cross. So this one is built to be wrong high:

| text | dira tokens | what a cl100k-family BPE typically spends | |---|---|---| | ` constitutional` | 4 | 1 | | `dec-0002` | 6 | 4 | | `1,500` | 5 | 3 |

Measured over this repository's own brief the whole document counts 1,271 dira tokens for 3,637 bytes, which is **1.40x** the four-characters-per-token rule of thumb. That margin is what is paid for not shipping a vocabulary, and it is stated rather than hidden: a reader who sets `max_tokens = 1500` gets a brief that a real tokenizer would put nearer 900, and the direction of that error is deliberate.

What this buys the cap

The counter is **deterministic** — the same bytes always count the same, so a golden test over a brief is stable and two machines agree — and it is **subadditive over blocks that end in a newline**, which is what lets the renderer fill the ceiling one measured block at a time and still be under it when the blocks are concatenated. `TestTokensIsSubadditiveOverBlocks` pins that property, because the fill silently depends on it.

What it does not claim

It is not a tokenizer and it is not accurate. dira has nothing to measure against — that is dec-0003's consequence, accepted here rather than worked around — so the honest claim is bounded: the count is never below what any tokenizer would need for the same text (one per whitespace-delimited word, one per line), and it is typically well above it. `dira brief` reports the number it counted on stderr whenever it drops anything, so the estimate is observable rather than internal.

Alternatives on record — 3

✗Vendor a BPE table (tiktoken cl100k or similar) and count exactlyrefused It buys exactness against one vendor's tokenizer and pays for it three times.

The vocabulary is megabytes of data shipped in a binary whose whole pitch is that it starts in milliseconds and downloads in one file (int-0002, dec-0001); the exactness is a fiction the moment the reader uses a different model, because Claude, GPT and Llama tokenize the same brief to three different numbers, and dira cannot know which one is reading; and cst-0004 forbids fetching the table at runtime, so it must ship rather than be downloaded once. The thing being defended is a guardrail on attention and context, not a billing meter — nobody is invoiced for the difference between 1,380 and 1,500.

revisit ifdira ever has to price a real API call, where the number is money rather than a limit — at which point the counter belongs to whatever is making the call and not to the brief

✗The four-characters-per-token rule of thumb, applied to the whole documentrefused It is an average, and it is an average over ordinary English prose, which a brief is not: it is ids, dates, punctuation and box characters at a density no prose corpus has.

Worse, an estimator centred on the truth is wrong low half the time, and every time it is wrong low the constitutional cap has been exceeded by a binary reporting that it was not. A cap that is decorative half the time is worse than no cap, because it is believed.

revisit ifa measurement over real briefs shows the structural estimator is over-counting by more than about 1.6x, at which point the margin is costing more content than the safety is worth

✗Count bytes and call the setting max_bytes, admitting there is no tokenizer hererefused Honest and unteachable. cst-0001 is written in tokens, README.md and docs/design.md both say 1,500 tokens, and the number exists because context windows and attention are measured in tokens — so a byte budget would force every reader to translate a unit dira invented back into the one they think in.

It also prices a line of box-drawing the same as a line of prose, when the two cost very different numbers of real tokens. The byte floor survives as one half of the estimator below, which is where it is genuinely useful.

revisit ifthe estimator's two halves ever disagree so rarely that one of them is dead weight