Vintage SLM 1: Introduction - Data and Pretraining
Why This Project This is the first in a series about my attempt to train a small ‘vintage’ language model. I wanted to tackle this because in my day job, I was mostly working on agentic systems built on proprietary model backbones, so most of the work was harness construction, tool-use development, and agent evaluation. I did experiment with basic fine-tuning for style transfer using OpenAI’s now-defunct fine-tuning API for GPT 3.5-turbo, and also conducted limited open-weight model LoRA fine-tuning using HuggingFace Deep Learning Containers on GCP, but I haven’t had the opportunity to train a model from scratch. ...