[{"data":1,"prerenderedAt":77},["ShallowReactive",2],{"term-p\u002Fpre-training":3,"related-p\u002Fpre-training":59},{"id":4,"title":5,"acronym":6,"body":7,"category":40,"description":41,"difficulty":42,"extension":43,"letter":16,"meta":44,"navigation":45,"path":46,"related":47,"seo":53,"sitemap":54,"stem":57,"subcategory":6,"__hash__":58},"terms\u002Fterms\u002Fp\u002Fpre-training.md","Pre-training",null,{"type":8,"value":9,"toc":33},"minimark",[10,15,19,23,26,30],[11,12,14],"h2",{"id":13},"eli5-the-vibe-check","ELI5 — The Vibe Check",[16,17,18],"p",{},"Pre-training is the first massive phase where an AI reads basically the entire internet and learns to predict the next word billions of times. This costs millions of dollars and takes months. The result is a smart base model that understands the world but hasn't been specialized for anything yet. Think of it as getting your degree before getting a job.",[11,20,22],{"id":21},"real-talk","Real Talk",[16,24,25],{},"Pre-training is the initial large-scale training phase where a model learns general representations from massive datasets using self-supervised objectives (e.g., next-token prediction, masked language modeling). It produces a foundation model that is subsequently fine-tuned for specific tasks or aligned via RLHF.",[11,27,29],{"id":28},"when-youll-hear-this","When You'll Hear This",[16,31,32],{},"\"Pre-training GPT-4 cost tens of millions.\" \u002F \"The pre-trained model is the starting point for fine-tuning.\"",{"title":34,"searchDepth":35,"depth":35,"links":36},"",2,[37,38,39],{"id":13,"depth":35,"text":14},{"id":21,"depth":35,"text":22},{"id":28,"depth":35,"text":29},"ai","Pre-training is the first massive phase where an AI reads basically the entire internet and learns to predict the next word billions of times.","intermediate","md",{},true,"\u002Fterms\u002Fp\u002Fpre-training",[48,49,50,51,52],"Fine-tuning","Transfer Learning","Training","LLM","Weights",{"title":5,"description":41},{"changefreq":55,"priority":56},"weekly",0.7,"terms\u002Fp\u002Fpre-training","YYse_Rt8-pdKwzteGvNISF8JI8V4kXvYFtdSYm8xtMw",[60,63,68,71,74],{"title":48,"path":61,"acronym":6,"category":40,"difficulty":42,"description":62},"\u002Fterms\u002Ff\u002Ffine-tuning","Fine-tuning is like taking a smart graduate student who knows everything and then sending them to a specialist bootcamp.",{"title":51,"path":64,"acronym":65,"category":40,"difficulty":66,"description":67},"\u002Fterms\u002Fl\u002Fllm","Large Language Model","beginner","An LLM is a humongous AI that read basically the entire internet and learned to predict what words come next, really really well.",{"title":50,"path":69,"acronym":6,"category":40,"difficulty":42,"description":70},"\u002Fterms\u002Ft\u002Ftraining","Training is the long, expensive process where an AI learns from data.",{"title":49,"path":72,"acronym":6,"category":40,"difficulty":42,"description":73},"\u002Fterms\u002Ft\u002Ftransfer-learning","Transfer Learning is using knowledge a model already has from one task to help it with a different task.",{"title":52,"path":75,"acronym":6,"category":40,"difficulty":42,"description":76},"\u002Fterms\u002Fw\u002Fweights","Weights are the numbers inside a neural network that determine what it knows and how it behaves — they're the AI's 'brain cells.",1776518302862]