Bart: A Vintage LLM

#Introduction

"The farther backward you can look, the farther forward you are likely to see." — Winston Churchill

We can't fully grasp modern questions like the rise of artificial intelligence, economic disruption, or automation while standing in the middle of them. Sometimes our best path forward is the one that takes us backward. In the past year, many have begun to explore this idea by applying it to large language models (LLM's). What can we learn about today's modern technology when we fuse it with information hundreds of years old? Recent models such as Talkie,Radford, Levine and Duvenaud. talkie — a 13B model trained on 260B tokens of pre-1931 English. talkie-lm/talkie-1930-13b-base and GPT-1900,Hla, M. GPT-1900 — a 3.3B model trained from scratch on pre-1900 text, to see whether it could arrive at quantum mechanics and relativity. michaelhla/gpt1900 have demonstrated that stripping away modern hindsight transforms an artificial mind from a simple archive into a powerful diagnostic lens. But while these models offer a proof of concept, true historical reasoning—like devising the transistor 20 years ahead of schedule—is still unreached. Material results remain the holy grail of historical AI. The next step, then, is to push historical LLM's into the future: state-of-the-art architectures, cleaner, more refined corpora, period-appropriate evaluations, and vintage-adapted post-training techniques. In this article, we introduce Bartholomew III, or Bart for short, our attempt to move the needle of historical language modeling.

#Results

Bart is a depth-32, 2.82B-parameter decoder-only transformer trained on 20.1B tokens of pre-1931 English. Every score below is Vintage CORE — CORE with its post-1930 knowledge removed, described under Evaluation — on the twenty tasks common to all three bundles.

#Leakage

The dataset is an independent reconstruction of the HISTORY-EVENT benchmark described in Pretraining Language Models on Historical Text.arXiv:2606.02991. This is not the authors' official dataset; our public reconstruction is at jbduran/history-event-reconstruction. We can see an increase after the knowledge cutoff starting around the 1900s for vintage models. In contrast, SmolLM3 remains broadly flat across decades, consistent with modern-event exposure.

#Pretraining

Arguably, the most important part of a model is its dataset. Curating a high quality dataset means obtaining raw pre-1930s text that's clean, correctly dated, and large enough to pretrain on. Combining these constraints leads to significant challenges.

Of the three, we felt that text-cleanliness was the limiting factor in dataset curation and therefore was a natural place to begin. Optical character recognition (OCR) quality is a score representing how accurately text is captured from real-world images and scans. This is important because pretty much all text used in a historical model will have been sourced via OCR. Here is an example of poor OCR:

For us, OCR quality is measured as a per-document confidence score. Low-scoring documents carry systematic transcription errors and stray page furniture that, if left in, corrupt the corpus at the subword level: the tokenizer allocates vocabulary to garbage merges, and the model spends capacity modeling scanner noise instead of language. One possible way of mitigating OCR issues would be to re-OCR pre-1930's texts via a modern LLM-based OCR, but after considering this approach we found that it would cost too much. Assuming roughly 16–20 million pages for an 8–10B token dataset at around 500 tokens per page, re-OCRing even a small dataset would cost about $16,000 at Mistral OCR batch pricing.Mistral AI. Introducing Mistral OCR 3 — a vision-language OCR model with a batch API discount. mistral.ai/news/mistral-ocr-3

Alternatively, we found it best to source pre-existing datasets and transform the existing corpora. A few options stood out. Project Gutenberg is a high quality dataset but is also very small.gutenberg.org — 75,000+ hand-proofread public-domain books. The American Stories dataset had great size, but we were concerned with the OCR quality due to the difficulty of OCR-ing newspapers.Dell et al. American Stories: A Large-Scale Structured Text Dataset of Historical U.S. Newspapers, NeurIPS 2023. arXiv:2308.12477 · dell-research-harvard/AmericanStories We kept searching, and came across Institutional Books 1.0. This dataset is a result of the digitization and post-processing work of the Harvard Library, boasting 242B tokens and almost a million documents.Cargnelutti et al. Institutional Books 1.0: A 242B Token Dataset from Harvard Library's Collections, 2025. arXiv:2506.08300 · institutional/institutional-books-1.0

Working from the Institutional Books (IB) dataset, we applied an aggressive OCR score threshold of 0.90, drawing on both OCRoscope and the OCR score from Google Books metadata, and required that the two metrics agree within 0.10 on any given document. Furthermore, we filtered by date (any undated books are rejected), we filtered for English (this resulted in the most rejects), and we filtered text with tokenizability scores less than 95 — a score indicating how efficiently a tokenizer encodes the text. This trimmed the 242B tokens down to 27B. We then deduplicated by book barcodes and shuffled the dataset. The result was our first cleaned dataset — bart-dataset-v1.

Another phase of filtering targeted boilerplate text and OCR corruption that snuck through IB's filters. This consisted of removing useless headers, footers and library stamps, alongside gibberish symbols and antiquated characters through heuristic regex filters. We also followed Michael Hla's log-prior filter, which estimates a text sequence's unconditional log-prior probability by summing token-level log probabilities and then trims improbable or out-of-distribution tokens.Documents whose log prior falls outside the 2.5th–97.5th percentile band are dropped, trimming both ends of the distribution. See michaelhla/gpt1900. This phase resulted in bart-dataset-v2. While we expected major improvements in our validation bpb (see Evaluation), our models actually saw a decline in performance because this phase removed most of the easy items for a model to replicate — that is, the boilerplate.

For our final phase, we decided to focus on the time period. Although we did filter previous stages by date, leaks and bad labeling would inevitably bring modern documents to contaminate our vintage dataset. We once again filtered through the boilerplate of documents, focusing on removing missed footers. However, the biggest part of this phase was the regex anachronism filter. We used a tiered banned words list to flag documents and remove them on too many "hits."

Rather than trimming, we removed entire documents when conditions were met. A single Tier 1 hit was enough to discard a document, while Tier 2 words required two occurrences. Tier 3 words never triggered removal on their own — they only counted when accompanied by at least one Tier 2 hit. We used this strategy later in midtraining to keep all of our data as vintage as possible. With all of these strategies, we created bart-dataset-v3 — our best vintage dataset yet.

#Architecture

Every successful model builds on a proven foundation and modifies it for its needs. When Qwen trained their first model family (Bai et al.),Bai et al. Qwen Technical Report, 2023. arXiv:2309.16609 they started from Llama's architecture. Kimi K2 started from DeepSeek-V3's MoE architecture.DeepSeek-AI. DeepSeek-V3 Technical Report, 2024. arXiv:2412.19437 We used Karpathy's Nanochat as a base since it's highly optimized and easily modifiable.karpathy/nanochat — an end-to-end LLM pipeline in a single readable codebase. Nanochat is an end-to-end LLM pipeline (extending from tokenization all the way to inference) built around a single complexity dial — "depth". Depth drives model scaling and is what we use to research the model at different sizes. We abbreviated it to "d#", where the number corresponds to the chosen depth (eg. d12, or d24). Within the codebase is the actual GPT module: a decoder-only transformer with rotary embeddings, QK-norm, untied embedding/lm_head, ReLU² MLP, Group-Query Attention, and Flash Attention 3.Shah et al. FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision, 2024. arXiv:2407.08608

While we started with almost all of nanochat's hyperparameters, we wanted to verify that they were optimal for our vintage data. Through later experimentation, we tested countless different hyperparameters, most notably context size and the Muon optimizer's momentum.

#Evaluation

We knew where our data was coming from, and how it would be processed, but before we could train anything we needed to know how to gauge progress. Think about it: how exactly do you quantify a good vintage model?

Validation BPB is a way of seeing how good a model is at autocompleting a set-aside, unseen piece of the dataset. Validation BPB is a great metric for pretraining, but it is dataset-specific. Consequently, we're not able to compare against other vintage or modern LLMs.

So we reached for CORE, the benchmark suite nanochat reports for its GPT-2 speedruns.Li et al. DataComp-LM: In search of the next generation of training sets for language models, 2024. arXiv:2406.11794. CORE averages a centered score over 22 tasks: (accuracy − random) / (1 − random), so 0 is chance and 1 is perfect. CORE is dataset-independent, cheap, and comes with a known GPT-2 reference point. One problem: CORE assumes a modern world model. Its tasks ask about websites, post-1930 organizations, brands, later scientific discoveries, and contemporary sports. Is our model dumb, or is it just being asked modern-laced questions?

We remove two benchmarks that cannot be salvaged, filter post-1930s content, restore small tasks to size with reviewed replacements, and build a second bundle where the surviving prose is rewritten into the style of 1800–1930. The result is Vintage CORE, released as a versioned dataset alongside its evaluator. That gives three measurements:

• Original — CORE as published.
• Filtered — post-1930 knowledge removed. Original prose.
• Restyled — same as filtered except vintage prose.

The same broad strategy appears in TypewriterLM's HellaSwag-1800: filter the temporally unfair items, then rewrite the survivors.Pretraining Language Models on Historical Text, 2026. arXiv:2606.02991 · typewriter.chat We apply it across a full 20-task suite rather than a single benchmark.

#Ablations

In our hopes to push the historical modeling space forward, we're including a short space here to document our exploration of the many mechanics and quirks of vintage language models. We explored model behavior through ablations (aka experiments). The goal of ablations was to run experiments at a small scale and get results we could confidently extrapolate to our Bart model. We tracked most of our ablations on GitHub via Issues.Every ablation has an issue with its config, its W&B run, and its result: zachnorton14/bart/issues. Below we list some of the more interesting ablations that guided our model's trajectory. Every run behind them — 39 in all, with their tokenizers, checkpoints and evaluations — is archived in bart-experiments. We hope this all helps you in your own development.

Briefly, we want to also emphasize the importance of two key development tools we used: Weights and Biases, and model configs. Weights and Biases is a diagnostic service that tracks all things regarding model training.wandb.ai — learning rate, FLOPs, even GPU wattage. Model configurations are simply JSON files that define model hyperparameters for each run. While often unspoken of in ML engineering, these two things are essential for scalable and easy experimentation.

#EXP007 · Variance testing

When testing on a small stand-in model, we found that CORE scores varied significantly, even for identical models, which, unfortunately, threw off our judgments of multiple previous experiments. A valuable reminder to not take anything for granted. While there is significant variation for CORE scores at the d12 scale, the amount of variation shown in EXP005 was unprecedented.

#EXP005 · Modern and vintage mixture

A simple test to explore just how much our performance depends on dataset quality. Even a small inclusion of the heavily-optimized ClimbMix dataset late into the training schema — the last 20% of train shards — saw an enormous CORE boost: 0.0673 → 0.1003. This confirmed a suspicion we had carried for some time: our model's dominant bottleneck was the absence of modern data, not the quality of the vintage data we had curated.

#EXP010 · Longer context length

Since the average pre-1930's document length is hundreds of times larger than that of modern pretrain corpora, increasing context length early in the training pipeline could benefit from these larger text sequences.This experiment had a flaw worth noting: it used full attention rather than sliding-window attention. Surprisingly, performance gains were merely cosmetic, and early tokens in the attention pattern showed minimal improvement in learned representation. The benefits still outweighed the costs, and we kept a 4096 context length.

#EXP012 · Tokenizer

Our wildly unique pretrain corpus lent us to test nanochat's built-in tokenizer variables — specifically document_cap, max_characters, and where documents start. A few findings emerged: random sampling improved compression ~1%, pushing max_characters above 80M barely improved compression, and high byte-fallback rates pointed to remaining OCR dataset junk. Despite the diminishing returns, it was cheap to train our tokenizer on our full corpus.

#EXP014 · Autoresearch

Prior to running our final model we ran 100 auto research experiments with the goal of improving our val/bpb relative to our compute budget. We adapted Karpathy's autoresearch to our model.karpathy/autoresearch — an agent edits one training file, runs a five-minute experiment, reads validation BPB, and iterates. Each experiment runs on a single H100, and the harness we ran them through is open source as bart-autoresearch. The training script runs for a fixed time budget of 5 minutes. GPT 5.6 Sol extra high was the model used for autoresearch and the loop ran for 10 hours. There were 26 improvements that reduced validation BPB from 1.244632 to 1.223533, or 1.70%. We only kept improvements to the Muon optimizer's momentum since they were the most impactful and we didn't have the time to scale up more potential improvements. Constant 0.90 was the best d12 result and was adopted for the final d32 run.

Nanochat's original architecture is already very optimized, and a lot of the low hanging fruit have already been eaten. Many of the improvements from autoresearch were not novel. The models clearly understand the objects they manipulate at a deep level, and yet very few genuinely new ideas emerge, which makes it hard to tell if this is an artifact of the speedrun setup or a real capability of the agent models.

#Midtraining

Midtraining sits between pretraining and downstream use: instead of continuing on the same broad corpus, we shift the data mixture toward a smaller, higher-quality set of documents while decaying the learning rate. The goal is to spend the model's final optimization steps on the text we most want it to internalize, which for us meant pre-1930s math, science, technology, and medicine. This section covers both how we built the midtraining dataset, and how we scheduled the run itself.

For our midtraining dataset, we focused on finding high quality math, science, technology and medicine documents from before the 1930s. For clarity, we broke each major step into stages. We started out by extracting these documents using their subject tags from bart-dataset-v1. Then, we extracted vintage documents from the Internet Archive by filtering by topic for Math, Science, Medicine, and Technology documents using their published date. We then de-duped these documents and used an OCR filter of 0.85. To get a second opinion, we filtered out 10% of documents using additional OCR filtering with OCRoscope (0.85) and an alpha numerical ratio of 0.65. Due to math documents having many different symbols, we wanted to keep the alpha numerical ratio relatively relaxed. Then, we filtered for OCR artifacts and removed portions of the boilerplate off of documents (headers, library stamps, etc). We mimicked our filtering in our main corpus using a log prior filter (2.5, 97.5).

Finally, we made sure our midtrain dataset was as vintage as possible, running it through the footer filter and the banned words list. As we cleaned our original corpus, we learned valuable tools and methodology for dataset cleaning. We applied this knowledge to midtraining extensively. After all the cleaning in midtraining, we removed 24% of the original midtraining data leaving us with 608M tokens of high quality midtraining data — bart-midtrain.

For the midtraining itself, we mimicked HuggingFace's SmolLM with midtraining in stages of decay.SmolLM's recipe decays the learning rate in stages while progressively raising the share of high-quality data. huggingface.co/blog/smollm3 Training ran in three stages. Because midtrain documents are much shorter than bart-dataset-v3 documents, we used tokens rather than documents to measure the mixtures of midtrain to original corpus. We wanted our midtrain data to have less than 3 epochs, so we used that as a guide for the ratios at each stage.

For a learning rate scheduler, we used nanochat's Warmup-Stable-Decay (WSD) starting decay at around ~35%. On the other hand, SmolLM started decaying at around ~75%.

One of the biggest pitfalls on this journey was during our final model run. We assumed that the size of midtrain documents would be roughly smaller than the original corpus. We were right, but we didn't realize how drastically smaller these midtrain documents were. Since we used document count rather than tokens to create the mixtures used for midtraining, we had to recreate our datasets with ratios with tokens rather than document counts. Since our final model was already running when we discovered this bug, we had 36 hours to create these new mixtures before our model would need them. Ultimately, we were able to get the mixtures ready for the model in time along with learning a valuable lesson — verify your intuition.

#Midtraining results

The goal of midtraining was to use the model's final optimization steps to internalize a smaller, higher-quality body of text. While we hoped this targeted phase would produce clear improvements, our evaluations do not provide conclusive evidence that midtraining was beneficial.

On the other hand, there is no conclusive evidence that our midtraining dataset is harmful. Due to the lack of resources and time, we were not able to run ablations on different data mixtures for midtraining. Because of this, we had to rely on a meager number of experiments before scaling up our model.

#Scaling

Scaling up to the d32, we knew we needed a hardware upgrade. Previously, we were using A100s with Flash Attention 2 for all experiments. We set our eyes on H100s for their 2–3 times speed up with Flash Attention 3. After researching multiple different options for leasing, vast.ai was by far the most price efficient.vast.ai — a marketplace for spot GPU rental, priced per host rather than per region. We considered running a 4x or 8x cluster to speed up the runs. Preparing for the 4x and 8x clusters, we created a docker image that has all of our dependencies and saves us 30 minutes on setup. We ran our first d24 on a 4x cluster to test the new docker image. One difficulty in using a multi-GPU cluster is that resuming requires similar hardware unless you are willing to drop the momentum of the optimizer, losing around 20 steps of learning. While there are possible solutions to circumvent this problem, we decided to resume our runs on the same hardware to spend our time in more valuable areas.

Compared to other options for leasing, vast.ai's GPUs shined brightest when using single devices. Since there are far more options to select from for single devices, it was far easier to find a bargain — especially past 11pm EST. While typical H100s cost ~$2–3 an hour, we were able to snag one for 99 cents an hour. Starting the d32 run Friday morning and with our budget approaching its maximum, we decided to opt for the single H100 as speed was not a huge priority. The run took 5 days on a single H100. Throughout our run we held a 60 MFU on our H100, our loss had minimal spikes and held a consistent 50k tokens per second. We had to pause the run at step 5500 (57%) to fix the midtraining shards, this resulted in 1 hour and 30 min of down time. The run remained healthy throughout, and the full d32 run is public on Weights & Biases, as are the two d24 runs that preceded it.The d24 ran in two parts on the 4x cluster: e69c1e59 · 2fa1b744

#Post-training

Making our model "feel" and act like an individual from the 1930's falls under the category of supervised fine tuning. SFT datasets are abundant for normal models, but for models trained exclusively on pre-1930's data the vast range of options vanish.A few do exist — vintage-ft-v2 and Violet-sft among them — but none at the scale or specificity we needed. Because authenticity was a priority for us it seemed smart to start by sourcing authentic question and answer (q/a) content that's already available online from the time period. This led to catechisms — texts specifically written in a question and answer format. About thirty catechisms total were sourced with topics such as engineering, logic, music, world history, and philosophy. A thorough extraction process included filtering, cleaning, enriching, and joining the q/a pairs. About 12k rows (with ~25% being multiturn conversations) were collected (authentic-pre1930-sft). For comparison, Hla's GPT-1900 model was trained on roughly 50k rows.Vintage-exam-qa. See michaelhla/gpt1900.

Research suggests that more important than size is quality and diversity.Zhou et al. LIMA: Less Is More for Alignment, 2023. arXiv:2305.11206 Since authentic sources would never be able to cover the many types of behaviors we wanted our model to exhibit, we pivoted to generating q/a pairs instead. Inspired by TypeWriterLM's History-SelfInstruct pipeline, we opted for grounded-answer based generation. In essence, pull excerpts from our corpus, trim and shape them to make good answers, and then generate a feasible question based on the given answer. This means generation and anachronism is halved, only the questions can carry an anachronistic risk. Further, because models typically mask the questions when fine tuning, in theory, the model never updates its weights directly on anachronistic text (though context still picks it up). To encourage diversity, about ten categories of q/a pairs were drafted consisting of narrative generation, knowledge recall, opinions, STEM reasoning, refusal calibration, etc.

Excerpts were algorithmically cherry-picked from the corpus, and then prompts were refined for each category to avoid context-bare, irrelevant questions, and phrase-repetition.

Examples of bare questions:

• "What error does the writer discover in the note from the Insurance Society concerning the premium?" — references the passage's author and an unnamed note; unanswerable to anyone who didn't read it.
• "What was the nationality and age of the captain of the corsair?" — which corsair?

Questions after tuning the prompt:

• "What was the method of cure, as given in 1857, for the disease in cattle evidenced by a frequent slight kicking of the hind foot?"
• "…the steamer built by Workman & Clark, of Belfast, which possessed a greater carrying capacity than…"

The result is synthetic-pre1930-sft, a monstrous 416k row (~70M tokens) dataset consisting of about 33% multiturn conversations, and roughly half of the whole dataset purely knowledge-based q/a pairs. The remaining half consists of the less common question types meant to encourage diverse behavior and reduce hallucinations. Two highlights: STEM reasoning, a modestly sized intellectual reasoning set, and refusal calibration, a vital non-answer set fostering honesty. Pairs are categorized, dated, and graded making this highly adaptable to different cutoff dates, SFT curriculums, and model personalities.

Grading is designed to answer the question, "How well does the q/a pair teach the model the intended goal of its category?" As shown in Figure 21, grading dramatically changes the landscape of the dataset. Around 196k q/a pairs are graded ≥ 95, and this increases to roughly 297k (+100.6k) for pairs ≥ 90. Grading is subtractive: if the model finds defects in the q/a pair then it loses points. Differences in scores above 95 can, therefore, reasonably be reduced to noise. As such, we found that questions graded below 80 were not worth including in our final fine tuning spread.

Speaking of fine tuning spreads: Chen et al. and Kaur et al. demonstrate that a small, rich dataset consistently outperforms a much larger, noisy one.Chen et al. AlpaGasus: Training a Better Alpaca with Fewer Data, 2023. arXiv:2307.08701 · Kaur et al. Instruct-SkillMix, 2024. arXiv:2408.14774 Applying these insights to our dataset means that we shouldn't naively dump all of the q/a pairs into our model. A fine tuning "curriculum" — as in a specific spread of learning material — needed to be concocted that would suit our model best. Six curricula were created to explore different ways of distributing the data. We wanted to find what drove the best model performance: data quantity, data quality, token distribution, domain scheduling, or domain difficulty. Surprisingly, we found that perhaps less isn't more in our case: our model chatted significantly better on the largest quantity of data. A cumulative 362k rows (~75M tokens) across 3 stages, progressing from easier to harder questions, thresholded at grades above 80, led to the best chatting experience by far. The 362k training rows were even later pushed to ~650k in a further iteration.

#Dataset limitations

Right at the end of the project, significant limitations surfaced regarding our fine tuning coverage. Experimentation on the smaller d24 model displayed weird quirks. For example, the model could handle questions just fine, but ask the very same question without a question mark and the model would collapse down to its base model behavior, spewing hundreds of incoherent tokens and often degrading into repetition. Similarly, the model couldn't handle unparseable text, typos, or simple conversational openers such as "Hello", or "How are you?". These types of inputs weren't included in the fine tuning dataset, so the model had no register for them.

The final d32 (2.8B) model appeared to generalize much better and was able to handle out-of-distribution inputs well. Still, the issues persisted and it led to a few last minute techniques being applied. One solution was to introduce artificial noise into the existing SFT corpus. Periodically and deterministically removing words, transposing letters, and adding or removing capital letters, punctuation, and spacing from questions made the model more sturdy at inference.

Larger-scale input corruption and diversity still needed to be addressed, though. So, a second solution was to create a complementary dataset aimed at teaching the model to respond as intended to outlying data. We made Robust-SFT for this purpose. User input that was uninterpretable, such as heavy typos, gibberish, or completely random character strings were paired with corpus-verbatim non-answers, such as "I beg your pardon" or "That is beyond me". 2,000 rows were synthesized across 11 classes of input corruption. Around 200 period and era grounding q/a pairs were synthesized to give the model some stiffness in this area. This comes with the highest anachronism risk, however, as they aren't corpus-grounded. Additionally, two types of conversational data were added: one which handled confusing but grammatically correct statements, and another prioritizing multiturn conversations and exchanges, which was very rare in the original dataset. 7,338 rows total, distributed across each stage of the original SFT dataset.

Despite our efforts, the issues weren’t completely fixed. Normal, structurally correct input without punctuation was still very volatile (pointing to an undertrained and underrepresentative fine tune). With time running low, we couldn’t spend more time curating data for the model to train on. We had to include a hotfix. The simplest change was to artificially append punctuation to the end of the user’s input. When experimenting, this cut the deterioration rates in half. Even better, though, was to prime the chat by prepending one full conversation turn. It’s like system prompting but simpler. The model essentially has the context of an earlier pre-specified introduction that the user doesn’t see. Our model is sufficiently trained on multiturn conversations, so it “recognizes” every interaction and responds properly even to those without proper grammar. The model that came out of all this — the C3 curriculum plus the robustness rows, branched from the base d32 at step 9,600 — is bart-sft, and it is what the demo serves.

#Evaluations

ChatCOREnanochat's chat evaluation: scripts/chat_eval.py · tasks/ is another benchmark suite reserved by Karpathy for models that come through the nanochat pipeline. The synthetic-pre1930-sft dataset was created with ChatCORE in mind. Unfortunately, evaluations on this benchmark weren’t run during development. It turns out that ChatCore is quite a biased benchmark and is mostly in place just to see how well a model learns Karpathy’s built in fine tuning tasks. In fact, ChatCORE is the reserved, set-aside validation set of nanochat’s fine tuning tasks. So when we don’t fine tune our model on his examples, the model does… poorly:

Because of the ChatCore oversight, Bart's post training remains largely unevaluated.

#Hosting

With our scaling strategy in place, the next challenge was deciding where and how to actually run our models. Hosting GPU-backed inference presented its own set of constraints, chief among them cost. From the outset, we sought a serverless GPU solution, evaluating Modal, RunPod, and Beam among several other providers.modal.com · runpod.io · beam.cloud Since our budget could not accommodate keeping a GPU running around the clock, we prioritized fast cold starts and automatic scaling. After weighing our options, we settled on Beam for a number of reasons. Foremost among them was the generosity of Beam's free tier, which offers up to five concurrent GPUs, free credits, automatic scaling, and access to affordable consumer hardware such as the RTX 4090 at just $0.69 per hour. The performance exceeded our expectations as inference was effectively instantaneous and cold starts were averaging only ten seconds.

#Finances

Training a language model is constrained as much by compute costs as by technical considerations. For this project, we worked with the limitations of being students without external funding or institutional support. Despite these constraints, we were able to train and evaluate a model that we are proud of, while keeping our overall expenses relatively modest.

For our ablation experiments, our primary expense was Google Colab Pro, followed by compute rented through Vast.ai. We used Hugging Face Pro for hosting our datasets and model checkpoints. For model development and experimentation, we used Claude Code alongside Codex, while DeepSeek and OpenCode were used primarily for SFT development and generation. Finally, for our larger d24 and d32 runs, we relied on Vast.ai due to the availability of comparatively inexpensive GPUs.

In total, our recorded expenses were approximately $800. This was not a fixed budget established at the beginning of the project; rather, it represents what we ultimately spent across development, training, and experimentation. The breakdown of these costs is shown below:

#Conclusions

Our goal throughout this project was to determine how far careful data selection, training decisions, and limited compute could take a vintage language model. Exploring this niche field gave us insight into countless problems that the frontier labs do not experience, such as the lack of large amounts of high quality SFT data. The results are not uniformly conclusive, but they give us a useful account of what worked, what did not, and — perhaps most importantly — what we would change in a future training run.

Looking back, there are several aspects of the training process that we would approach differently. During pretraining, we would have spent more time refining and validating the dataset before moving on to ablations. We also would have conducted a more thorough set of learning-rate experiments rather than relying primarily on established recommendations. For midtraining, we would have applied more aggressive filtering to the corpus, potentially using an LLM-based quality filter similar to the approach used by FineWeb-Edu. Additionally, budget and time constraints prevented us from systematically testing different mixtures of midtraining data, leaving an important part of the training recipe unexplored. Finally, delays in producing our final model compressed the timeline available for post-training, limiting the amount of time we could devote to refining the SFT dataset.

Despite these shortcomings, the process was an invaluable learning experience. Each limitation exposed a different challenge in training language models, from dataset construction and hyperparameter selection to experiment design and project management. While there are undoubtedly many things we would change in hindsight, these lessons provide a much stronger foundation for approaching our next training run.

Despite these challenges, we are proud of how much we accomplished over the summer while balancing the project with other commitments, including summer coursework. As researchers, we learned an enormous amount about training and evaluating language models, and we are especially proud of the resources we produced. bart-dataset-v3 is one of the first open-source filtering efforts focused on Institutional Books textbooks, while Vintage CORE provides a valuable evaluation benchmark for future vintage models. Although our SFT dataset is not perfect, the work required to create it demonstrated just how challenging building specialized SFT datasets can be. Most importantly, we are proud of how we worked together through deadlines, problems, setbacks, and successes. We are proud of the model we produced and what we learned along the way. As the first project for our lab, we consider Bart an enormous success and a valuable foundation for our future work.

#Acknowledgements

This project is built on work others did in the open.

• Andrej Karpathy — for nanochat, which this codebase forks, and autoresearch. His work has made from-scratch LLM training something a small team can actually carry end to end.
• Hugging Face Smol Models Research — for The Smol Training Playbook, the most useful practical guide to training a model from scratch that we found, and the closest thing the field has to an honest operations log.
• Michael Hla — for Machina Mirabilis and gpt1900, which showed that a vintage LLM was worth building.
• zakarth and croqaz — from the VintageLLM Discord, for advice and support throughout the creation of this model.

#Citation

Please cite this work as:

@article{unboundedlabs2026bart,
title   = {Bart: A Vintage LLM},
author  = {Unbounded Labs},
year    = {2026},
journal = {Unbounded Labs Research},
note    = {https://unboundedlabs.ai/blog/bart}
}