Language models · Reproducible research · Case study
Turning a tokenizer hunch into a capacity-controlled sealed experiment.
smaLLM is an inspectable PyTorch laboratory covering corpus provenance, tokenization, causal Transformer modeling, controlled training, sealed evaluation, and candid experiment reports.
The problem
What had to become visible.
A boundary-aware byte tokenizer appeared to outperform a character model, but its larger vocabulary also increased the embedding and output parameter count.
The useful question was not whether one run won; it was whether the effect survived corrected evaluation, multiple seeds and corpora, sealed tests, and a near-parameter-matched control.
Constraints
Rules before implementation.
- Corpus identities, checksums, normalization, chronological splits, configurations, seeds, and checkpoint hashes must be preserved.
- Validation selects checkpoints; terminal test regions remain sealed until the final one-shot evaluation.
- Results must be normalized as bits per character so tokenizer sequence lengths remain comparable.
- Negative results, reversals, and threats to validity stay in the permanent record.
Engineering decisions
The choices that shaped the result.
Correct the measurement contract before scaling
The project fixed tokenizer leakage and incomplete validation coverage, then reran controlled comparisons rather than treating earlier numbers as final evidence.
Add a meaningful capacity control
The final char136 arm matched ByteBPE512 within 0.40% of its parameter count, alongside the historical char128 arm.
Freeze the protocol before source access
The final three-corpus × three-arm × three-seed panel was preregistered before the new public-domain sources were accessed.
Make sealed evaluation refuse ambiguous state
The evaluator verifies corpus and checkpoint identity, requires full coverage, writes once, and rejects incomplete or inconsistent panel artifacts.
What changed
Turning points, not a clean-room success story.
- Simple BPE128 shortened sequences and improved the appearance of generated text but underperformed the corrected character control on BPC.
- Boundary-aware ByteBPE512 produced a repeatable advantage, but the capacity confound required a new matched control rather than a stronger headline.
- The final Douglass seed-4242 comparison reversed by 0.0018 BPC. Keeping it visible narrowed the conclusion from universal superiority to a corpus-dependent result.
Measured result
What the evidence supports.
- The final study contains 27 sealed observations across three new corpora, three model arms, and three fixed seeds.
- ByteBPE512 won eight of nine same-seed comparisons against near-parameter-matched char136 and all nine against historical char128.
- Mean matched-control improvements were 0.1447 BPC on Frankenstein, 0.0187 on Douglass, and 0.1717 on Origin of Species.
- The repository closes with 182 tests, 90.49% branch coverage, strict mypy over 47 source files, package builds, dependency auditing, and a five-minute CPU demo.
Limitations
What remains unproven.
- The models are intentionally small and do not establish behavior at production language-model scale.
- Each corpus is one English public-domain work; the fixed panel is not a population sample.
- Chronological validation and terminal test regions can differ materially in difficulty.
- The observed advantage is tokenizer- and corpus-dependent, not a universal BPE claim.
Production perspective
What I would carry forward.
- Treat the project as a completed research artifact rather than extending the modeling roadmap indefinitely.
- Future replication should broaden languages, genres, corpus sizes, and model scales while preserving the sealed protocol.
- Use the documented controls and artifact contracts as the baseline for any independent extension.