Researchers retrofit language models to read raw bytes instead of tokens

Researchers at the Allen Institute for AI (Ai2), the University of Cambridge, the University of Washington, Imperial College London and LMU Munich have shown that an existing large language model can be converted to read raw bytes instead of subword tokens, for a small fraction of the cost of training a new one. The peer-reviewed paper, “Retrofitting language models to operate over bytes”, was published in Nature on October 7, 2026.

Most language models do not read text letter by letter. A tokenizer first splits the input into chunks drawn from a fixed vocabulary, which Ai2 says can make models less flexible with spelling, rare words and unusual strings. The authors write that subword tokenization “obscures fine-grained information,” which they say hurts work with scientific data such as code and biological sequences. They also list weak character-level understanding, responses that shift depending on how a prompt is tokenized, and an English-centric bias that comes from restricted vocabularies.

Byte-level models avoid the fixed vocabulary by processing the raw bytes that computers use to encode characters. The drawback, according to Ai2, is that such models have traditionally had to be trained from scratch, which is expensive.

How “byteification” works

The team’s method, which it calls byteification, starts from a trained subword model and adds new byte-level components: a local encoder, a boundary predictor and a local decoder. In the first stage, the original model is frozen while only the new parts are trained, on 9.8 billion tokens. In the second stage, the whole model is unfrozen and trained on a further 39.3 billion tokens.

That total of 49.1 billion tokens is less than 1% of a typical pretraining budget, the paper says. The training mix also included 75 million tokens of character-level data, about 0.04% of the total.

The authors applied the recipe to four open models. Ai2’s Olmo 3 7B became Bolmo 7B and OLMo 2 1B became Bolmo 1B. Two models from other developers were also converted: Qwen3 8B Base became Bwen 8B, and Llama 3 8B became Blama 8B. The conversion changes model size only slightly. Bolmo 7B has about 330 million more parameters than its source model, an increase of 4.5%, while Bolmo 1B has about 10 million fewer.

What the results show

According to the paper, Bolmo 7B “greatly outperformed the source Olmo 3 on character understanding and was better in certain coding settings.” When Ai2 first released Bolmo in December 2025, it reported an improvement of “nearly twenty points” over Olmo 3 on its character-understanding aggregate. Against BLT 7B, an earlier byte-level model trained from random initialization, Bolmo 7B scored 16.5 percentage points higher on STEM (science, technology, engineering and math) tasks, the authors report.

The paper also records a cost: the new architecture caused a “substantial degradation in Code (pass@1)” relative to a control version of Olmo 3 7B.

On speed, the authors say byteified models “come close to matching subword-level LLMs in inference speed,” and that they can be made faster by training them to pack more bytes into each patch. In measurements on H100 graphics processing units (GPUs) at a batch size of one, Bolmo 7B overtook Olmo 3 7B on time to first byte at a compression of about 6.6 bytes per patch. Ai2’s December 2025 post put Bolmo’s generation speed at about 125 bytes per second, against about 150 for the corresponding subword model at the same compression.

Existing fine-tuning also carries over. The December post reported that merging in the weights of the post-trained Olmo 3 lifted Bolmo’s score on the IFEval instruction-following benchmark from 31.1% to 67.4%, close to the post-trained Olmo 3’s 66.9%. The Nature paper says this kind of transfer works “generally, but not always.”

In its October post, Ai2 says Bwen 8B is its strongest byteified model so far and outperforms Bolmo 7B across the institute’s aggregate evaluation suite. The post gives no individual benchmark scores for Bwen 8B.

Why it matters

A tokenizer is fixed when a model is built, so its quirks are carried into everything the model does. A retrofit that costs less than 1% of pretraining gives researchers a way to test byte-level designs on strong existing models rather than paying to train new ones from scratch, and the paper applies the recipe to models from outside Ai2 as well as to its own.

The release is fully open. Ai2 has published the models and the first-stage checkpoints on Hugging Face, along with the training data, and the code is on GitHub. The institute says outside researchers are already building on Bolmo, including a developer who fine-tuned Bolmo 7B for Russian-to-English poetry and lyric translation.

The caveats

The evaluations were run by the model developers, largely with evaluation suites that Ai2 maintains, and the gains are uneven: character-level tasks improved markedly while some coding measures fell. The authors declared no competing interests.

Funding listed in the paper includes grants from the UK Engineering and Physical Sciences Research Council, the US National Science Foundation and the European Research Council. The authors also acknowledge computing support from the Oak Ridge Leadership Computing Facility, the National Artificial Intelligence Research Resource pilot, Microsoft Azure and Google’s TPU Research Cloud.

Corrections and tips: email the editor using the address on the Contact page.