I do AI Alignment research. Currently at METR, but previously at: Redwood Research, UC Berkeley, Good Judgment Project.
I’m also a part-time fund manager for the LTFF.
Obligatory research billboard website: https://chanlawrence.me/
LawrenceC
Karma: 5,657
Super fair. I did not read that section in detail, and missed your interpretation on my skim. I interpreted it to mean “in the planning literature, we typically see k=3 for n<6” (which we do!), noted that they did not say the alternative value, and then went with the k=3 value they did say in the main body.
You’re right that your interpretation is more natural. If they actually used k=4, then the problem is solvable and the paper was (a bit) better than I portrayed it to be here.
Worth noting that, even assuming your interpretation is correct, it’s possible is that the person who wrote the problem spec did know about the impossibility result, but the person running the experiments did not (and thus the experiments were ran with k=3). But it does make it more likely that k=4 was used for n>5.
I think my statement above that “they did not say” “a different value of k (such that the problem is possible)” still seems true. And if they used k=4 because they knew that k=3 is impossible, it seems quite sloppy to say that they used k=3 in the main body. But a poorly written paper is a different and lesser problem to the experiment being wrong.
Fair, but in my head I did plan to get it done on the 10th. The tweet is not in itself the prediction, it’s just evidence that I made the prediction in my head.
And indeed I did finish the draft on June 10th, but at 11 PM and I decided to wait for feedback before posting. So I wasn’t that off in the end, but I still consider it off.
Oh, fair. Thanks for the correction, I didn’t realize how much artists were affected.
This isn’t my area of expertise, so take what I have to say with a grain of salt.
I wouldn’t say that anyone is particularly on the ball here, but there are certainly efforts to bring about more “mainstream” awareness about AI. See e.g. the recent post by the ControlAI folk about their efforts to brief British politicians on AI, in which they mention some of the challenges they ran into. Still within the greater LW/80k circle, we have Rational Animations, which has a bunch of AI explainers via animated Youtube videos. Then slightly outside of it, we have Rob Miles’s videos, both on his own channel and on Computerphile. Also, I’d take a look at the PauseAI folk to see if they’ve been working on anything lately.
My impression is that the primary problem is “Not feeling the AGI”, where I think the emphasis should placed on “AGI”. People see AI everywhere, but it’s kinda poopy. There’s AI slop and AI being annoying and lots of startups hyping AI, but all of those suggest AI to be mediocre and not revolutionary. Outside of entry-level SWE jobs I don’t think people have really felt much disruption from an employment perspective. It’s also just hard to distinguish between more or less powerful models (“o1” vs “o3“ vs “gpt-4o” vs “gpt-4.5” vs “ChatGPT”, empirically I notice that people just give up and use “ChatGPT” to refer to all OAI LMs).
A clean demonstration of capabilities tends to shock people into “feeling the AGI”. As a result, I expect this to be fixed in part over time, as we get more events like the release of GPT-3 (a clean demo for academics/AIS folk), the ChatGPT release in late 2022, or the more recent Ghiblification wave. I also think that, unfortunately, “giving informed and strongly evidence backed claims about AI progress” is just not easy to do for as broad an audience as National TV, so maybe demos are just the way to go.
Secondarily, I think the second problem is fatalism. People switch from “we won’t get AGI, those people are just being silly” to “we’ll get AGI and OAI/Anthropic/GDM/etc will take over/destroy the world, there’s nothing we can do”. In some cases this even goes into “I’ll choose not to believe it, because if I do, I’ll give up on everything and just cry.” To be honest, I’m not sure how to fix this one (antidepressants, maybe?).
Then there’s the “but China” views, but I think that’s much more prevalent on Twitter or the words of corporate lobbyists than a response I’ve heard from people in real life.
The authors specify that they used k=3 on page 6 of their paper:
If they used a different k (such that the problem is possible), they did not say so in their paper.
You’re right that I could’ve been more clear here, I’ve edited in a footnote to clarify.
There are indeed many, many silly claims out there, on either side of any debate. And yes, the people pretending that the AIs of 2025 have the limitations of those from 2020 are being silly, journalist or no.
I do want to clarify that I don’t think this is a (tech) journalist problem. Presumably when you mention Nightshade dismissively, it’s a combination of two reasons: 1) Nightshade artefacts are removable via small amounts of Gaussian blur and 2) Nightshade can’t be deployed at scale on enough archetypal images to have a real effect? If you look at the Nightshade website, you’ll see that the authors lie about 1):
As with Glaze, Nightshade effects are robust to normal changes one might apply to an image. You can crop it, resample it, compress it, smooth out pixels, or add noise, and the effects of the poison will remain.
So (assuming my recollection that Nightshade is defeatable by Gaussian noise is correct) this isn’t an issue of journalists making stuff up or misunderstanding what the authors said, it’s the authors putting things in their press release that, at the very least, are not at all backed up by their paper.
(Also, either way, Gary Marcus is not a tech journalist!)
Beware General Claims about “Generalizable Reasoning Capabilities” (of Modern AI Systems)
Anthropic releases Claude 3.7 Sonnet with extended thinking mode
It would be really great with human baselines, but it’s very hard to do in practice. For a human to do one of these tasks it would take several hours.
My guess is it’s <1 hour per task assuming just copilot access, and much less if you’re allowed to use e.g. o1 + Cursor in agent mode. That being said, I think you’d want to limit humans to comparable amounts of compute for comparable number, which seems a bit trickier to make happen.
I don’t really have any funding for this project, but I might find someone that wants to do one task for fun, or do my best effort myself on a fresh task when I make one.
Is the reason you can’t do one of the existing tasks, just to get a sense of the difficulty?
Makes sense, thanks!
For compute I’m using hardware we have locally with my employer, so I have not tracked what the equivalent cost of renting it would be, but I guess it would be of the same order of magnitude or as the API costs or a factor of a few larger.
It’s hard to say because I’m not even sure you can rent Titan Vs at this point,[1] and I don’t know what your GPU utilization looks like, but I suspect API costs will dominate.
An H100 box is approximately $2/hour/GPU and A100 boxes are a fair bit under $1/hour (see e.g. pricing on Vast AI or Shadeform). And even A100s are ridiculously better than a Titan V, in that it has 40 or 80 GB of memory and (pulling number out of thin air) 4-5x faster.
So if o1 costs $2 per task and it’s 15 minutes per task, compute will be an order of magnitude cheaper. (Though as for all similar evals, the main cost will be engineering effort from humans.)
- ^
I failed to find an option to rent them online, and I suspect the best way I can acquire them is by going to UC Berkeley and digging around in old compute hardware.
- ^
This is really impressive—could I ask how long this project took, how long does each eval take to run on average, and what you spent on compute/API credits?
(Also, I found the preliminary BoK vs 5-iteration results especially interesting, especially the speculation on reasoning models.)
(Disclaimer: have not read the piece in full)
If “reasoning models” count as a breakthrough of the relevant size, then I argue that there’s been quite a few of these in the last 10 years: skip connections/residual stream (2015-ish), transformers instead of RNNs (2017), RLHF/modern policy gradient methods (2017ish), scaling hypothesis (2016-20 depending on the person and which paper), Chain of Thought (2022), massive MLP MoEs (2023-4), and now Reasoning RL training (2024).
I think the title greatly undersells the importance of these statements/beliefs. (I would’ve preferred either part of your quote or a call to action.)
I’m glad that Sam is putting in writing what many people talk about. People should read it and take them seriously.
Nit:
> OpenAI presented o3 on the Friday before Thanksgiving, at the tail end of the 12 Days of Shipmas.Should this say Christmas?
I think writing this post was helpful to me in thinking through my career options. I’ve also been told by others that the post was quite valuable to them as an example of someone thinking through their career options.
Interestingly, I left METR (then ARC Evals) about a month and a half after this post was published. (I continued to be involved with the LTFF.) I then rejoined METR in August 2024. In between, I worked on ambitious mech interp and did some late stage project management and paper writing (including some for METR). I also organized a mech interp workshop at ICML 2024, which, if you squint, counts as “onboarding senior academics”.
I think leaving METR was a mistake ex post, even if it made sense ex ante. I think my ideas around mech interp when I wrote this post weren’t that great, even if I thought the projects I ended up working on were interesting (see e.g. Compact Proofs and Computation in Superposition). While the mech interp workshop was very well attended (e.g. the room was so crowded that people couldn’t get in due to fire code) and pretty well received, I’m not sure how much value it ended up producing for AIS. Also, I think I was undervaluing the resources available to METR as well as how much I could do at METR.
If I were to make a list for myself in 2023 using what I know now, I’d probably have replaced “onboarding senior academics” with “get involved in AI policy via the AISIs”, and instead of “writing blog posts or takes in general”, I’d have the option of “build common knowledge in AIS via pedagogical posts”. Though realistically, knowing what I know now, I’d have told my past self to try to better leverage my position at METR (and provided him with a list of projects to do at METR) instead of leaving.
Also, I regret both that I called it “ambitious mech interp”, and that this post became the primary reference for what this term meant. I should’ve used a more value-neutral name such as “rigorous model internals” and wrote up a separate post describing it.
I think this post made an important point that’s still relevant to this day.
If anything, this post is more relevant in late 2024 than in early 2023, as the pace of AI makes ever more people want to be involved, while more and more mentors have moved towards doing object level work. Due to the relative reduction of capacity in evaluating new AIS researchers, there’s more reliance on systems or heuristics to evaluate people now than in early 2023.
Also, I find it amusing that without the parenthetical, the title of the post makes another important point: “evals are noisy”.
I think this post was useful in the context it was written in and has held up relatively well. However, I wouldn’t active recommend it to anyone as of Dec 2024 -- both because the ethos of the AIS community has shifted, making posts like this less necessary, and because many other “how to do research” posts were written that contain the same advice.
Background
This post was inspired by conversations I had in mid-late 2022 with MATS mentees, REMIX participants, and various bright young people who were coming to the Bay to work on AIS (collectively, “kiddos”). The median kiddo I spoke with had read a small number of ML papers and a medium amount of LW/AF content, and was trying to string together an ambitious research project from several research ideas they recently learned about. (Or, sometimes they were assigned such a project by their mentors in MATS or REMIX.)
Unfortunately, I don’t think modern machine learning is the kind of field where you can take several where research consistently works out of the box. Many high level claims even in published research papers are just… wrong, it can be challenging to reproduce results even when they are right, and even techniques that work reliably may not work for the reasons people think they do.
Hence, this post.
What do I think of the content of the post?
I think the core idea of this post held up pretty well with time. I continue to think that making contact with reality is very important, and I think the concrete suggestions for how to make contact with reality are still pretty good.
If I were to write it today, I’d probably add a fifth major reason for why it’s important to make quick contact with reality: mental health/motivation. That is, producing concrete research outputs, even small ones, feels pretty essential to maintaining motivation for the vast majority of researchers. My guess is I missed this factor because I focused on the content of research projects, as opposed to the people doing the research.
Where do I feel the post stands now?
Over the past two years, the ethos of the AIS community has changed substantially toward empirical work, over the past two years, and especially in 2024.
The biggest part of this is because of the pace of AI. When this post was written, ChatGPT was a month old, and GPT-4 was still more than 2 months away. People both had longer timelines and thought of AIS in more conceptual terms. Many research conceptual research projects of 2022 have fallen into the realm of the empirical as of late 2024.
Part of this is due to the rise of (dangerous capability) evals as a major AIS focus in 2023, which is both substantially more empirical compared to the median 2022 AIS research topic, and an area where making contact with reality can be as simple as “pasting a prompt into claude.ai”.
Part of this is due to Anthropic’s rise to being the central place for AIS researchers. “Being able to quickly produce ML results” is a major part of what it takes to get hired there as a junior researcher, and people know this.
Finally, there’s been a decent amount of posts or write-ups giving the same advice, e.g. Neel’s written advice for his MATS scholars and a recent Alignment Forum post by Ethan Perez.
As a result, this post feels much less necessary or relevant in late December 2024 than in December 2022.
Evan joined Anthropic in late 2022 no? (Eg his post announcing it was Jan 2023 https://www.alignmentforum.org/posts/7jn5aDadcMH6sFeJe/why-i-m-joining-anthropic)
I think you’re correct on the timeline, I remember Jade/Jan proposing DC Evals in April 2022, (which was novel to me at the time), and Beth started METR in June 2022, and I don’t remember there being such teams actually doing work (at least not publically known) when she pitched me on joining in August 2022.
It seems plausible that anthropic’s scaring laws project was already under work before then (and this is what they’re referring to, but proliferating QA datasets feels qualitatively than DC Evals). Also, they were definitely doing other red teaming, just none that seem to be DC Evals
Otherwise, we could easily in the future release a model that is actually (without loss of generality) High in Cybersecurity or Model Autonomy, or much stronger at assisting with AI R&D, with only modest adjustments, without realizing that we are doing this. That could be a large or even fatal mistake, especially if circumstances would not allow the mistake to be taken back. We need to fix this.
[..]
This is a lower bound, not an upper bound. But what you need, when determining whether a model is safe, is an upper bound! So what do we do?Part of the problem is the classic problem with model evaluations: elicitation efforts, by default, only ever provide existence proofs and rarely if ever provide completeness proofs. A prompt that causes the model to achieve a task provides strong evidence of model capability, but the space of reasonable prompts is far too vast to search exhaustively to truly demonstrate mode incapability. Model incapability arguments generally rely on an implicit “we’ve tried as hard at elicitation as would be feasible post deployment”, but this is almost certainly not going to be the case, given the scale of pre-deployment evaluations vs post-deployment use cases.
The way you get a reasonable upper bound pre-deployment is by providing pre-deployment evaluators with some advantage over end-users, for example by using a model that’s not refusal trained or by allowing for small amounts of finetuning. OpenAI did do this in their original preparedness team bio evals; specifically, they provided experts with non—refusal fine-tuned models. But it’s quite rare to see substantial advantages given to pre-deployment evaluators for a variety of practical and economic reasons, and in-house usage likely predates pre-deployment capability/safety evaluations anyways.
Thanks for writing this!
I agree that current frontier AI risk management frameworks (henceforth RMFs) are insufficient to ensure acceptably low risks from AI development. I also think that you’ve pointed out the key reasons for why they’re inadequate: not addressing risks from development or internal deployment, a lack of planning for what happens assuming dangerous capabilities are detected, and (most importantly) a reliance on only checking for a few known risks (as operationalized by current evals). I think that the main value proposition of current RMFs is transparency—that is, it’s good to lay out what the evals and plans, and it’s good for people to do very basic checks for model capabilities. (and I think that this is a common view at METR).
I also agree that this is one way to characterize an underlying assumption behind most RMFs (though not necessarily all, and the people writing them would not phrase it this way).
Another perspective on this underlying assumption is that people don’t think of AI development, but instead think of it as software development and correctly note that “real” RMFs wouldn’t allow people to develop software as it’s normally developed. So actual RMFs end up a compromise between what they are in the traditional sense and what doesn’t over encumber the normal software development cycle.