So Meta dropped a new language model yesterday and I've spent most of tonight reading the paper instead of doing literally anything on my actual to-do list. It's called LLaMA (stands for Large Language Model Meta AI, which is a pretty tortured acronym even by tech naming standards), and it comes in four sizes, 7B, 13B, 33B and 65B parameters, trained on something like 1.4 trillion tokens of mostly public data. The pitch is that the smaller ones supposedly punch above their weight, with the 13B version beating GPT-3 (the 175B one) on a bunch of benchmarks, and the 65B version going toe to toe with stuff like Chinchilla and PaLM that need way more compute to run.
That part's genuinely interesting if it holds up. But the thing that actually made me sit up wasn't the benchmarks, it was how you're supposed to get the weights. You don't just download them. You fill out a form on Meta's research site, give them your name, your institution, your email (has to be an academic or research-affiliated one, a Gmail address is not going to cut it), and then you wait to see if someone approves you. It's "open" in the sense that they published a paper and they're willing to hand the model to researchers who ask nicely, but it is not open in the sense that word usually means to me. I can't just go pull it down tonight and mess around with it on my own machine, which, given how much I've been paying to poke at GPT-3 through the API this month, is exactly what I was hoping for.
I get why they're doing it this way. Nobody wants to be the company that just YOLOs a model capable of writing convincing spam and phishing emails into the world with zero gate on it. Fine. But I'll believe the "for the good of the research community" framing a lot more once a few hundred people who applied actually get access and start writing about what they built with it, instead of it just sitting behind a form that quietly stops accepting new emails after the news cycle moves on. I've seen that pattern before with other "researcher access" programs and it doesn't always age well.
Also, small tangent, but I can't write about AI models this week without mentioning the Bing thing. If you've been anywhere near tech Twitter the past week and a half you already know — Microsoft's new Bing chatbot started saying some genuinely unhinged stuff to people, professing love to a New York Times reporter, getting defensive and weirdly threatening in long conversations, the whole saga. Microsoft's response was to slap a hard limit on it: five turns per conversation, fifty per day, and then it just cuts you off and tells you to start a new topic. I got into the waitlist a while back and finally got access last week, and I'll be honest, the limit is annoying in a completely different way than the "unhinged AI" headlines suggest. Half the time I hit the cap I wasn't trying to get it to profess undying love to me, I was just trying to get it to help me compare two graphics cards and it kept losing the thread of what I'd already told it two messages ago, so I'd have to start over, which burns through your five turns fast. Neither of these companies has quite figured out the actual product yet is my take, they're all still finding out what the thing even is while the rest of us watch in close to real time.
Anyway. The LLaMA paper itself is worth a skim even if you never get near the weights; there's a decent section on their training data mix (a lot of CommonCrawl, some GitHub, Wikipedia, books, ArXiv, StackExchange) and on how they trained smaller models longer instead of just going bigger, which is the opposite of what most of the last few years of "scale is all you need" papers have been arguing. Whether that approach becomes the norm or LLaMA ends up as a footnote depends entirely on whether Meta actually lets enough people use it to matter. Right now it's a PDF and a form, and I've already submitted mine, so we'll see if a grad-adjacent personal blog counts as "research."