I turned 25 today.

[Originally written for Twitter]

Many AI risk failure modes imagine strong coherence/​goal directedness (e.g. [expected] utility maximisers).

Such strong coherence is not represented in humans, seems unlikely to emerge from deep learning and may be “anti-natural” to general intelligence in our universe.

I suspect the focus on strongly coherent systems was a mistake that set the field back a bit, and it’s not yet fully recovered from that error.

I think most of the AI safety work for strongly coherent agents (e.g. decision theory) will end up inapplicable/​useless for aligning powerful systems.

[I don’t think it nails everything, but on a purely ontological level, @Quintin Pope and @TurnTrout’s shard theory feels a lot more right to me than e.g. HRAD.

HRAD is based on an ontology that seems to me to be mistaken/​flawed in important respects.]

The shard theory account of value formation (while lacking) seems much more plausible as an account of how intelligent systems develop values (where values are “contextual influences on decision making”) than the immutable terminal goals in strong coherence ontologies. I currently believe that immutable terminal goals is just a wrong frame for reasoning about generally intelligent systems in our world (e.g. humans, animals and future powerful AI systems)^[1].

And I’m given the impression that the assumption of strong coherence is still implicit in some current AI safety failure modes (e.g. it underpins deceptive alignment^[2]).

I’d be interested in more investigation into what environments/​objective functions select for coherence and to what degree said selection occurs.

And empirical demonstrations of systems that actually become more coherent as they are trained for longer/​”scaled up” or otherwise amplified.

I want advocates of strong coherence to explain why agents operating in rich environments (e.g. animals, humans) or sophisticated ML systems (e.g. foundation models) aren’t strongly coherent.

And mechanistic interpretability analysis of sophisticated RL agents (e.g. AlphaStar, OpenAI Five [or replications thereof]) to investigate their degree of coherence.

Currently, I think strong coherence is unlikely (plausibly “anti-natural”^[3]) and am unenthusiastic about research agendas and threat models predicated on strong coherence.

Disclaimer this is all low confidence speculation, and I may well be speaking out of my ass.

I do think that my disagreements with deceptive alignment is not a failure of understanding, but I am very much an ML noob, so there can still be things I just don’t know. My opinion on this matter would probably be different in a significant way by this time next year.

1. ^

   You cannot well predict the behaviour/​revealed preferences of humans or other animals by the assumption that they have fixed terminal goals.

   The ontology that intelligent systems in the real world instead have “values” (contextual influences on decision making) seems to explain their observed behaviour (and observed incoherencies) better.

2. ^

   In addition to the other prerequisites listed in the “Deceptive Alignment” post, deceptive alignment also seems to require a mesa-optimiser so coherent that it would be especially resistant to modifications to its mesa-objective. That is it requires very strong levels of goal content integrity.

3. ^

   E.g. if the shard theory account of value formation is at all correct, particularly the following two claims:

   * Values are inherently contextual influences on decision making

   * Values (shards) are strengthened (or weakened) via reinforcement events

   Then strong coherence in the vein of utility maximisation just seems like an anti-natural form

Confusions About Optimisation and Agency

Something I’m still not clear how to think about is effective agents in the real world.

I think viewing idealised agency as an actor that evaluates argmax wrt (the expected value of) a simple utility function over agent states is just wrong.

Evaluating argmax is very computationally expensive, so most agents most of the time will not be directly optimising over their actions but instead executing learned heuristics that historically correlated with better performance according to the metric the agent is selected for (e.g. reward).

That is, even if an agent somehow fully internalised the selection metric, directly optimising it over all its actions is just computationally intractable in “rich” (complex/​high dimensional problem domains, continuous, partially observable/​imperfect information, stochastic, large state/​action spaces, etc.) environments. So a system inner aligned to the selection metric would still perform most of its cognition in a mostly amortised manner, provided the system is subject to bounded compute constraints.

Furthermore, in the real world learning agents don’t generally become inner aligned to the selection metric, but instead learn cognitive heuristics that historically correlated with performance on the selection metric.

[Is this because direct optimisation for the outer selection metric is too computationally expensive? I don’t think so. The sense I get is that selection just doesn’t work that way. The selection process can’t internalise the selection metric, because selection for a given metric produces equally strong selection for all the metric’s necessary and sufficient conditions^[1] (the “invariants of selection”):

In some sense, the true metric is under specified.

Note however, that this underspecification is more pernicious than just not distinguishing between the invariants of selection. The inductive biases of the selection process also matter. Proxies that are only correlated with the selection metric (“imperfect proxies”) may be internalised instead of the selection metric if they are more accessible/​reachable/​learnable by the intelligent system than the actual selection metric.

Analogy: humans couldn’t have internalised evolution’s selection metric of inclusive genetic fitness because humans had no concept of inclusive genetic fitness.]

So there are at least two dimensions on which real world intelligent systems diverge from the traditional idealisations of an agent:

1. Real world systems do not perform most of their cognition by directly optimising an appropriate objective function, but by executing learned cognitive adaptations

   1. Note that I said “most”; humans are capable of performing direct optimisation (e.g. “planning”) when needed, but such explicit reasoning is a minority of our cognition

2. Real world systems don’t internalise the metric on which they were selected for, but instead learn various contextual heuristics that correlated with high performance on that metric.

   1. I see this as the core claim of shard theory, and the cause of “complexity of value”

I think this establishes some sort of baseline for what real world intelligent systems are like. However, I do not know what such systems “converge” to as they are scaled up (in training/​inference compute/​data or model parameters).

I am not very sure how online learning affects this either.

I am sceptical that it converges towards anything like “embedded AIXI”. I just do not think embedded AIXI represents any sort of limit or idealisation of intelligent systems in the real world.

Alas, I have no better ideas. Speculation on this is welcome.


Cc: @Quintin Pope, @cfoster0, @beren, @TurnTrout

1. ^

   TurnTrout talks about reinforcement learning in the linked post, but I think the argument generalises very straightforwardly to any selection process and the metric of selection.

Adversarial robustness is the wrong frame for alignment.

Robustness to adversarial optimisation is very difficult[1].

Cybersecurity requires adversarial robustness, intent alignment does not.

There’s no malicious ghost trying to exploit weaknesses in our alignment techniques.

This is probably my most heretical (and for good reason) alignment take.

It’s something dangerous to be wrong about.

I think the only way such a malicious ghost could arise is via mesa-optimisers, but I expect such malicious dameons to be unlikely apriori.

That is, you’ll need a training environment that exerts significant selection pressure for maliciousness/​adversarialness for the property to arise.

Most capable models don’t have malicious daemons[2], so it won’t emerge by default.

[1]: Especially if the adversary is a more powerful optimiser than you.

[2]: Citation needed.

What I’m currently working on:

The sequence has an estimated length between 30K − 60K words (it’s hard to estimate because I’m not even done preparing the outlines yet).

I’m at ~8.7K words written currently (across 3 posts [the screenshots are my outlines]) and guess I’m only 5% of the way through the entire sequence.

Beware the planning fallacy though, so the sequence could easily grow significantly longer than I currently expect.


I work full time until the end of July and will be starting a Masters in September, so here’s to hoping I can get the bulk of the piece completed when I have more time to focus on it in August.

Currently, I try for some significant writing [a few thousand words] on weekends and fill in my outlines on weekdays. I try to add a bit more each day, just continuously working on it, until it spontaneously manifests. I also use weekdays to think about the sequence.

So, the twelve posts I’ve currently planned could very well have ballooned in scope by the time I can work on it full time.

Weekends will also be when I have the time for extensive research/​reading for some of the posts).

Most of the catastrophic risk from AI still lies in superhuman agentic systems.

Current frontier systems are not that (and IMO not poised to become that in the very immediate future).

I think AI risk advocates should be clear that they’re not saying GPT-5/​Claude Next is an existential threat to humanity.

[Unless they actually believe that. But if they don’t, I’m a bit concerned that their message is being rounded up to that, and when such systems don’t reveal themselves to be catastrophically dangerous, it might erode their credibility.]

I find noticing surprise more valuable than noticing confusion.

Hindsight bias and post hoc rationalisations make it easy for us to gloss over events that were apriori unexpected.

Some Nuance on Learned Optimisation in the Real World

I think mesa-optimisers should not be thought of as learned optimisers, but systems that employ optimisation/​search as part of their inference process.

The simplest case is that pure optimisation during inference is computationally intractable in rich environments (e.g. the real world), so systems (e.g. humans) operating in the real world, do not perform inference solely by directly optimising over outputs.

Rather optimisation is employed sometimes as one part of their inference strategy. That is systems only optimise their outputs part of the time (other [most?] times they execute learned heuristics^[1]).

Furthermore, learned optimisation in the real world seems to be more “local”/​task specific (i.e. I make plans to achieve local, particular objectives [e.g.planning a trip from London to Edinburgh]. I have no global objective that I am consistently optimising for over the duration of my lifetime).

I think this is basically true for any feasible real world intelligent system^[2]. So learned optimisation in the real world is:

1. Partial^[3]

2. Local

Do these nuances of real world mesa-optimisers change the nature of risks from learned optimisation?

Cc: @evhub, @beren, @TurnTrout, @Quintin Pope.

1. ^

   Though optimisation (e.g. planning) might sometimes be employed to figure out which heuristic to deploy at a particular time.

2. ^

   For roughly the reasons why I think fixed immutable terminal goals are antinatural, see e.g.: “Is “Strong Coherence” Anti-Natural?”

   Alternatively, I believe that real world systems learn contextual heuristics (downstream of historical selection) that influence decision making (“values”) and not fixed/​immutable terminal “goals”. See also: “why assume AGIs will optimize for fixed goals?”

3. ^

   This seems equivalent to Beren’s concept of “hybrid optimisation”; I mostly use “partial optimisation”, because it feels closer to the ontology of the Risks From Learned Optimisation paper. As they define optimisation, I think learned algorithms operating in the real world just will not be consistently optimising for any global objective.

The Case for Theorems

Why do we want theorems for AI Safety research? Is it a misguided reach for elegance and mathematical beauty? A refusal to confront the inherently messy and complicated nature of the systems? I’ll argue not.

Desiderata for Existential Safety

When dealing with powerful AI systems, we want arguments that they are existentially safe which satisfy the following desiderata:

1. Robust to scale

   • Especially robustness to “scaling up”/​capability amplification

   • Cf. “The Sharp Left Turn”

2. Generalise far out of distribution to test/​deployment environments that are unlike our training environments

3. We have very high “all things considered” confidence in

   • Failure might imply existential catastrophe, so we may have a small margin of error

   • We want arguments that not only tell us the system is existentially safe at high probability, but that we have high confidence that if the argument says $X$ obtains given $Y$ then with very high likelihood $X$ actually obtains given $Y$

4. Translate to successor/​derivative systems

   • Ideally, we wouldn’t want to have to independently verify safety properties for any successor system our system might create (or more likely) derivative systems

     • If parent systems are robustly safe and sufficiently capable, we may be able to offload the work of aligning child systems safe to their parents

5. Robust to adversarial optimisation?

   • I am not actually sure to what extent the safety properties of AI systems need to be adversarially robust to be existentially safe. I think imagining that the system is actively trying to break safety properties is a wrong framing (it conditions on having designed a system that is not safe^[1]), and I don’t know to what extent strategic interactions in multipolar scenarios would exert adversarial pressure on the systems.

   • But I am not very confident in this/​it does not seem too implausible to me that adversarial robustness could be a necessary property for existential safety

Given that the preconditions of our theorem actually describe the real world/​real world systems well (a non trivial assumption), then theorems and similar formal arguments can satisfy all the above desiderata. Furthermore, it may well be the case that only (semi)formal/​rigorous arguments satisfy all the aforementioned desiderata?

Rather, under some plausible assumptions, non rigorous arguments may fail to satisfy any of these desiderata.

When Are Theorems Not Needed?

Rigorous arguments for safety are less compelling in worlds where iterative alignment strategies are feasible.

For example, if takeoff is slow and continuous we may be able to get a wealth of empirical data with progressively more powerful systems and can competently execute on governance strategies.

Civilisation does not often sleep walk into disaster.

In such cases where empirical data is abundant and iterative cycles complete quickly enough (i.e. we develop “good enough” alignment techniques for the next generation of AI systems before such systems are widely deployed), I would be more sympathetic to scepticism of formalism. If empirical data abound, arguments for safety grounded in said data (though all observation is theory laden), would have less meta level uncertainty than arguments for safety rooted in theory^[2].

Closing Remarks

However, in worlds where iterative design fails (e.g. takeoff is fast or discontinuous), we are unlikely to have an abundance of empirical data and rigorous arguments may be the only viable approach to verify safety of powerful AI systems.

Considering that most of our existential risk is concentrated in world in worlds with fast/​discontinuous takeoff (even if the technical problem is intractable in slow/​continuous takeoff world, governance approaches have a lot more surface area to execute on to alleviate risk [see my earlier point that civilisation does not sleepwalk into disaster]). As such, technical attempts to reduce risk might have the largest impact by focusing on risk in worlds with fast/​discontinuous takeoff.

For that purpose, it seems formal arguments are the best tools we have^[3].

I still want to work on technical AI safety eventually.

I feel like I’m on quite far off path from directly being useful in 2025 than I felt in 2023.

And taking a detour to do a TCS PhD that isn’t directly pertinent to AI safety (current plan) feels like not contributing.

Cope is that becoming a strong TCS researcher will make me better poised to contribute to the problem, but short timelines could make this path less viable.

[Though there’s nothing saying I can’t try to work on AI on the side even if it isn’t the focus of my PhD.]

Mechanistic Utility Maximisers are Infeasible

I’ve come around to the view that global optimisation for a non-trivial objective function in the real world is grossly intractable, so mechanistic utility maximisers are not actually permitted by the laws of physics^[1][2].

My remaining uncertainty around expected utility maximisers as a descriptive model of consequentialist systems is whether the kind of hybrid optimisation (mostly learned heuristics, some local/​task specific planning/​search) that real world agents perform converges towards better approximating argmax wrt (the expected value of) a simple utility function over agent/​environment states.

A crux of my alignment research philosophy:

Our theories of safety must be rooted in descriptive models of the intelligent systems we’re dealing with to be useful at all.

I suspect normative agent foundations research is just largely misguided/​mistaken. Quoting myself from my research philosophy draft:

Furthermore in worlds where there’s a paucity of empirical data, I don’t actually believe that we can necessarily develop towers of abstraction out of the void and have them remain rooted in reality^[16]. I expect theory developed in the absence of empirical data to validate it to largely just fail to describe reality^[17]. Our theories of safety must be rooted in descriptive models of the intelligent systems we’re dealing with to be useful at all.

And the relevant footnotes:

16. You cannot I posit develop an adequate theory of alignment from the null string as input.

17. I do not think there’s a simple theory of intelligence that can be derived even in principle from pure computer science (in the limiting case, no intelligence is feasible in max entropic universes).

To what extent is a particular universe learnable? What inductive biases and (hyper)priors are best for learning it? What efficient approximations exist for common algorithmic problems? How well do learning algorithms generalise for common problems? Etc. All seem like empirical questions about the reality we live in. And I expect the answers to these empirical questions to constrain what intelligent systems in our universe look like.

I am pessimistic about normative theories of intelligent systems that are divorced from the behaviour of intelligent systems in our universe.

E.g. I suspect that expected utility maximisation is anti-natural to generally intelligent behaviour in our universe. More generally, I suspect large portions of normative agent foundations research would fail to describe the systems that matter in the real world and thus end up being inapplicable/​unusable.

This is a crux behind my strong scepticism of theories constructed from void.

Immigration is such a tight constraint for me.

My next career steps after I’m done with my TCS Masters are primarily bottlenecked by “what allows me to remain in the UK” and then “keeps me on track to contribute to technical AI safety research”.

What I would like to do for the next 1 − 2 years (“independent research”/​ “further upskilling to get into a top ML PhD program”) is not all that viable a path given my visa constraints.

Above all, I want to avoid wasting N more years by taking a detour through software engineering again so I can get Visa sponsorship.

[I’m not conscientious enough to pursue AI safety research/​ML upskilling while managing a full time job.]

Might just try and see if I can pursue a TCS PhD at my current university and do TCS research that I think would be valuable for theoretical AI safety research.

The main detriment of that is I’d have to spend N more years in <city> and I was really hoping to come down to London.

Advice very, very welcome.

[Not sure who to tag.]

I once claimed that I thought building a comprehensive inside view on technical AI safety was not valuable, and I should spend more time grinding maths/​ML/​CS to start more directly contributing.

I no longer endorse that view. I’ve come around to the position that:

• Many alignment researchers are just fundamentally confused about important questions/​topics, and are trapped in inadequate ontologies

• Considerable conceptual engineering is needed to make progress

• Large bodies of extant technical AI safety work is just inapplicable to making advanced ML systems existentially safe
  

Becoming less confused, developing better frames, enriching my ontology, adding more tools to my conceptual toolbox, and just generally thinking clearer about technical AI safety is probably very valuable. Probably more valuable than rushing to try and execute on a technical agenda (that from my current outside view would probably end up being useless).

Most of the progress I’ve made on becoming a better technical AI safety person this year has been along the lines of trying to think clearer about the problem.

“Foundations of Intelligent Systems” not “Agent Foundations”

I don’t like the term “agent foundations” to describe the kind of research I am most interested in, because:

1. I am unconvinced that “agent” is the “true name” of the artifacts that would determine the shape of humanity’s long term future

   1. The most powerful artificial intelligent systems today do not cleanly fit into the agent ontology,

   2. Future foundation models are unlikely to cleanly conform to the agent archetype

   3. Simulators/​foundation models may be the first (and potentially final) form of transformative AI; salvation/​catastrophe may be realised without the emergence of superhuman general agents

   4. I worry that foundational theory that conditions too strongly on the agent ontology may end up being of limited applicability to the intelligent systems that would determine the longterm future of earth originating civilisation

2. I am persuaded by/​sympathetic to composite modular architectures for transformative AI such as Drexler’s “Open Agency” or “Comprehensive AI Services”

3. I do not want us to build generally capable strongly autonomous agents; my best case outcome for AI is for AI to amplify human cognition and capabilities, not to replace/​supersede us.

   1. I believe that cognition can be largely disconnected from volition

   2. And that in principle, augmented/​amplified humans (“cyborgs”) can be competitive with superhuman agents

   3. I find the concept of an aligned sovereign (agent) very uncompelling. It runs counter to my values regarding autonomy and self actualisation. Human enfeeblement would be a tragedy in its own right.

Research paradigms/​approaches I see as belonging to “Foundations of Intelligent Systems”:

Please suggest more!

Writing LW questions is much easier than writing full posts.

Occasionally I see a well received post that I think is just fundamentally flawed, but I refrain from criticising it because I don’t want to get downvoted to hell. 😅

This is a failure mode of LessWrong.

I’m merely rationally responding to karma incentives. 😌

Contrary to many LWers, I think GPT-3 was an amazing development for AI existential safety.

The foundation models paradigm is not only inherently safer than bespoke RL on physics, the complexity and fragility of value problems are basically solved for free.

Language is a natural interface for humans, and it seems feasible to specify a robust constitution in natural language?

Constitutional AI seems plausibly feasible, and like it might basically just work?

That said I want more ambitious mechanistic interpretability of LLMs, and to solve ELK for tighter safety guarantees, but I think we’re in a much better position now than I thought in 2017.

My best post was a dunk on MIRI^[1], and now I’ve written up another point of disagreement/​challenge to the Yudkowsky view.

There’s a part of me that questions the opportunity cost of spending hours expressing takes of mine that are only valuable because they disagree in a relevant aspect with a MIRI position? I could have spent those hours studying game theory or optimisation.

I feel like the post isn’t necessarily raising the likelihood of AI existential safety?

I think those are questions I should ask more often before starting on a new LessWrong post; “how does this raise the likelihood of AI existential safety? By how much? How does it compare to my other post ideas/​unfinished drafts?”

Maybe I shouldn’t embark on a writing project until I have (a) compelling narrative(s) for why my writing would be useful/​valuable for my stated ends.

1. ^

   This is an uncharitable framing of the post, but it is true that the post was written from a place of annoyance. It’s also true that I have many important disagreements with the Yudkowsky-Soares-Bensinger position, and expressing them is probably a valuable epistemic service.

A Sketch of a Formalisation of Self Similarity

Introduction

I’d like to present a useful formalism for describing when a set^[1] is “self-similar”.

Isomorphism Under Equivalence Relations

Given arbitrary sets $X,Y$, an “equivalence-isomorphism” is a tuple $(f:X\to Y,f^{-1}:Y\to X,{\sim }_R\subset X\cup Y\times X\cup Y)$, such that:

Where:

• $f$ is a bijection from $X$ to $Y$

• $f^{-1}$ is the inverse of $f$

• ${\sim }_R$ is an equivalence relation on the union of $X$ and $Y$.

For a given equivalence relation ${\sim }_R$, if there exist functions such that an equivalence-isomorphism can be constructed, then we say that the two sets are “isomorphic under ${\sim }_R$”.

The concept of “isomorphism under an equivalence relation” is meant to give us a more powerful mechanism for describing similarity/​resemblance between two sets than ordinary isomorphisms afford^[2].

Similarity

Two sets are “similar” if they are isomorphic to each other under a suitable equivalence relation^[3].

Self-Similarity

A set is “self-similar” if it’s “similar” to a proper subset of itself.

Closing Remarks

This is deliberately quite bare, but I think it’s nonetheless comprehensive enough (any notion of similarity we desire can be encapsulated in our choice of equivalence relation) and unambiguous given an explicit specification of the relevant equivalence relation.

1. ^

   I stuck to sets because I don’t know any other mathematical abstractions well enough to play around with them in interesting ways.

2. ^

   Ordinarily, two sets are isomorphic to each other if a bijection exists between them (they have the same cardinality). This may be too liberal/​permissive for “similarity”.

   By choosing a sufficiently restrictive equivalence relation (e.g., equality), we can be as strict as we wish.

3. ^

   Whatever notion of similarity we desire is encapsulated in our choice of equivalence relation.

A reason I mood affiliate with shard theory so much is that like...

I’ll have some contention with the orthodox ontology for technical AI safety and be struggling to adequately communicate it, and then I’ll later listen to a post/​podcast/​talk by Quintin Pope/​Alex Turner, or someone else trying to distill shard theory and then see the exact same contention I was trying to present expressed more eloquently/​with more justification.

One example is that like I had independently concluded that “finding an objective function that was existentially safe when optimised by an arbitrarily powerful optimisation process is probably the wrong way to think about a solution to the alignment problem”.

And then today I discovered that Alex Turner advances a similar contention in “Inner and outer alignment decompose one hard problem into two extremely hard problems”.

Shard theory also seems to nicely encapsulates my intuitions that we shouldn’t think about powerful AI systems as optimisation processes with a system wide objective that they are consistently pursuing.

Or just the general intuitions that our theories of intelligent systems should adequately describe the generally intelligent systems we actually have access to and that theories that don’t even aspire to do that are ill motivated.

It is the case that I don’t think I can adequately communicate shard theory to a disbeliever, so on reflection there’s some scepticism that I properly understand it.

That said, the vibes are right.

Consequentialism is in the Stars not Ourselves?

Still thinking about consequentialism and optimisation. I’ve argued that global optimisation for an objective function is so computationally intractable as to be prohibited by the laws of physics of our universe. Yet it’s clearly the case that e.g. evolution is globally optimising for inclusive genetic fitness (or perhaps patterns that more successfully propagate themselves if you’re taking a broader view). I think examining why evolution is able to successfully globally optimise for its objective function would be enlightening.

Using the learned optimisation ontology, we have an outer selection process (evolution, stochastic gradient descent, etc.) that selects intelligent systems according to their performance on a given metric (inclusive genetic fitness and loss respectively).

Local vs Global Optimisation

Optimisation here refers to “direct” optimisation, a mechanistic procedure for internally searching through an appropriate space for elements that maximise or minimise the value of some objective function defined on that space.

Local Optimisation

• Involves deploying optimisation (search, planning, etc.) to accomplish specific tasks (e.g., making a good move in chess, winning a chess game, planning a trip, solving a puzzle).

• The choice of local tasks is not determined as part of this framework; local tasks could be subproblems of a another optimisation problem (e.g. picking a good next move as part of winning a chess game), generated via learned heuristics, etc.

Global Optimisation

• Entails consistently employing optimisation throughout a system’s active lifetime to achieve fixed terminal goals.

• All actions flow from their expected consequences on realising the terminal goals (e.g., if a terminal goal is to maximise the number of lives saved, every activity—eating, sleeping, playing, working—is performed because it is the most tractable way to maximise the expected number of future lives saved at that point in time).

Outer Optimisation Processes as Global Optimisers

As best as I can tell, there are some distinctive features of outer optimisation processes that facilitate global optimisation:

Access to more compute power

• ML algorithms are trained with significantly (often orders of magnitude) more compute than is used for running inference due in part to economic incentives

  • Economic incentives favour this: centralisation of ML training allows training ML models on bespoke hardware in massive data centres, but the models need to be cheap enough to run profitably

    • Optimising inference costs has lead to “overtraining” smaller models

  • In some cases trained models are intended to be run on consumer hardware or edge computing devices

• Evolutionary processes have access to the cumulative compute power of the entire population under selection, and they play out across many generations of the population

• This (much) greater compute allows outer optimisation processes to apply (many?) more bits of selection towards their objective functions

Relaxation of time constraints

• Real-time inference imposes a strict bound on how much computation can be performed in a single time step

  • Robotics, self driving cars, game AIs, etc. must make actions within fractions of a second

    • Sometimes hundreds of actions in a second

  • User facing cognitive models (e.g.) LLMs are also subject to latency constraints

    • Though people may be more willing to wait longer for responses if the output of the models are sufficiently better

• In contrast, the outer selection process just has a lot more time to perform optimisation

  • ML training runs already last several months, and the only bound on length of training runs seems to be hardware obsolescence

    • For sufficiently long training runs, it becomes better to wait for the next hardware generation before starting training

    • Training runs exceeding a year seem possible eventually, especially if loss keeps going down with scale

  • Evolution occurs over timescales of hundreds to thousands of generations of an organism

Solving a (much) simpler optimisation problem

• Outer optimisation processes evaluate the objective function by using actual consequences along single trajectories for selection, as opposed to modeling expected consequences across multiple future trajectories and searching for trajectories with better expected consequences.

  • Evaluating future consequences of actions is difficult (e.g., what is the expected value of writing this LessWrong shortform on the number of future lives saved?)

  • Chaos sharply limits how far into the future we can meaningfully predict (regardless of how much computational resources one has), which is not an issue when using actual consequences for selection

    • In a sense, outer optimisation processes get the “evaluate consequences of this trajectory on the objective” for free, and that’s just a very difficult (and in some cases outright intractable) computational problem

  • The usage of actual consequences applies over longer time horizons

    • Evolution has a potentially indefinite/​unbounded horizon

      • And has been optimising for much longer than any

    • Current ML training generally operates with fixed-length horizons but uses actual/​exact consequences of trajectories over said horizons.

• Outer optimisation processes selects for a policy that performs well according to the objective function on the training distribution, rather than selecting actions that optimise an objective function directly in deployment.

  • This approach amortises the cost of optimization across many future inferences of the selected policy.

Summary

Outer optimisation processes are more capable of global optimisation due to their access to more compute power, relaxed time constraints, and just generally facing a much simpler optimisation problem (evaluations of exact consequences are provided for free [and over longer time horizons], amortisation of optimisation costs, etc).

These factors enable outer optimisation processes to globally optimise for their selection metric in a way that is infeasible for the intelligent systems they select for.

Cc: @beren, @tailcalled, @Chris_Leong, @JustisMills.

Paul Christiano’s AI Alignment Landscape:

“Is intelligence NP hard?” is a very important question with too little engagement from the LW/​AI safety community. NP hardness:

1. Bounds attainable levels of intelligence (just how superhuman is superintelligence?)

2. Bounds physically and economically feasible takeoff speeds (i.e. exponentially growing resource investment is required for linear improvements in intelligence)

I’d operationalise “is intelligence NP hard?” as:

Does there exist some subset of computational problems underlying core cognitive tasks that have NP hard [expected] (time) complexity?

I want to write a LW question post on this sometime.

For now, I want to learn more about complexity theory and let my thoughts on the complexity of intelligence gestate further.

Gwern’s “Complexity no Bar to AI” argues convincingly that complexity theory doesn’t prohibit AGI/​superhuman AI, but not that:

1. The attainable level of intelligence is as far above humans as humans are above ants (or other animal of choice)

2. Fast takeoff is feasible

As best as I can tell (I didn’t finish it, but will reread it after I’m done with my current complexity paper) Gwern did not argue convincingly that intelligence is not NP hard.

I’m finally in the 4-digit karma club! 🎉🎉🎉

(Yes, I like seeing number go up. Playing the LW karma game is more productive than playing Twitter/​Reddit or whatever.

That said, LW karma is a very imperfect proxy for AI safety contributions (not in the slightest bit robust to Goodharting) and I don’t treat it as such. But insomuch as it keeps me engaged with LW, it keeps me engaged with my LW AI safety projects.

I think it’s a useful motivational tool for the very easily distracted me.)

My most successful post took me around an hour to publish and already has 10x more karma than a post that took me 10+ hours to publish.

There’s a certain unfairness about it. The post isn’t actually that good. I was just ranting about something that annoyed me.

I’m bitter about its success.

There’s something profoundly soul crushing to know that the piece I’m pouring my soul into right now wouldn’t be that well received.

Probably by the time I published it I’ll have spent days on the post.

And that’s just so depressing. So, so depressing.

Twitter Cross Posting

Introduction

I’ll start reposting threads from my Twitter account to LW with no/​minimal editing.

Twitter Relevant Disclaimers

I’ve found that Twitter incentivises me to be

More:

• Snarky

• Brazen

• Aggressive

• Confident

• Exuberant

Less:

• Nuanced

• Modest

The overall net effect is that content written originally for Twitter has predictably low epistemic standards compared to content I’d write for LessWrong. However, trying to polish my Twitter content for LessWrong takes too much effort (my takeoff dynamics sequence [currently at 14 − 16 posts] started as an attempt to polish my takeoff dynamics thread for a LW post).

As I think it’s better to write than to not write, I’ve decided to publish my low-quality-but-still-LessWrong-relevant Twitter content here with no/​minimal editing. I hedge against the predictably low quality by publishing it as shortform/​to my personal blog instead of to the front page.

Testing

I’ll post the unedited/​minimally edited threads as both blog posts (promotion to front page disabled) and shortform pieces. I’ll let the reception to the posts in both venues decide which one I’ll keep with going forward. I’ll select positively for:

• Visibility

• Engagement

And negatively against:

• Hostility

• Annoyance

I’ll Write

It is better to write than to not write.

Perfect should not be the enemy of good.


If I have a train of thought that crosses a thousand words when written out, I’m no longer going to consider waiting until I’ve extensively meditated upon, pondered, refined and elaborated on that train of thought until it forms a coherent narrative that I deeply endorse.

I’ll just post the train of thought as is.

If necessary, I’ll repeatedly and iteratively improve on the written train of thought to get closer to a version that I’ll deeply endorse. I’ll not wait for that final version that may never arrive.

Since I became active again, I have yet to have a single coherent narrative that I “deeply endorse”. I anticipate that it will take me several weeks to a few months of dedicated thinking and writing on my topics of interests to get something of that quality.

But I’m sceptical I can stay on the same topic for that long given my (crippling) executive dysfunction. I’ll just do what I can within the limits of my ability.

“All you need is to delay doom by one more year per year and then you’re in business” — Paul Christiano.

Unpublished my “Why Theorems? A Brief Defence” post.

The post has more than doubled in length and scope creep is in full force.

I kind of want to enhance it to serve as the definitive presentation of my “research tastes”/​”research vibes”.

Hypothesis: any learning task can be framed as a predictive task^[1]; hence, sufficiently powerful predictive models can learn anything.

A comprehensive and robust model of human preferences can be learned via SSL with a target of minimising predictive error on observed/​recorded behaviour.

This is one of those ideas that naively seem like they basically solve the alignment problem, but surely it can’t be that easy.

Nonetheless recording this to come back to it after gitting gud at ML.

Potential Caveats

Maybe “sufficiently powerful predictive models” is doing a lot of heavy lifting.

Plausible the “irreducible entropy” in our records of human behaviour prevent learning values well (I don’t actually believe this).

Perhaps the dataset size required to get a sufficiently robust/​comprehensive model is too large?

Another concern is the potential of mindcrime.

Me: Mom can we have recursive self improvement?

Mom: we already have recursive self improvement at home.

Recursive self improvement at home:

https://​​www.lesswrong.com/​​posts/​​camG6t6SxzfasF42i/​​a-year-of-ai-increasing-ai-progress

Discussion Questions

I’m to experiment with writing questions more frequently.

There are several topics I want to discuss here, but which I don’t yet have substantial enough thoughts to draft up a full fledged post for.

Posing my thoughts on the topic as a set of questions and soliciting opinions seems a viable approach.

I am explicitly soliciting opinions in such questions, so do please participate even if you do not believe your opinion to be particularly informed.

Ruminations on the Hardness of Intelligence

Sequel to my first stab at the hardness of intelligence.

I’m currently working my way through “Intelligence Explosion Microeconomics”. I’m actively thinking as I read the paper and formulating my own thoughts on “returns on cognitive reinvestment”.

I have a Twitter thread where I think out loud on this topic.

I’ll post the core insights from my ramblings here.

@Johnswentworth, @Rafael Harth.

Probably sometime last year, I posted on Twitter something like: “agent values are defined on agent world models” (or similar) with a link to a LessWrong post (I think the author was John Wentworth).

I’m now looking for that LessWrong post.

My Twitter account is private and search is broken for private accounts, so I haven’t been able to track down the tweet. If anyone has guesses for what the post I may have been referring to was, do please send it my way.

Does anyone know a ChatGPT plugin for browsing documents/​webpages that can read LaTeX?

The plugin I currently use (Link Reader) strips out the LaTeX in its payload, and so GPT-4 ends up hallucinating the LaTeX content of the pages I’m feeding it.

To: @Quintin Pope, @TurnTrout

I think “Reward is not the Optimisation Target” generalises straightforwardly to any selection metric.

Tentatively, something like: “the selection process selects for cognitive components that historically correlated with better performance according to the metric in the relevant contexts.”

From “Contra “Strong Coherence”″:

Many observed values in humans and other mammals (e.g. fear, play/​boredom, friendship/​altruism, love, etc.) seem to be values that were instrumental for promoting inclusive genetic fitness (promoting survival, exploration, cooperation and sexual reproduction/​survival of progeny respectively). Yet, humans and mammals seem to value these terminally and not because of their instrumental value on inclusive genetic fitness.

That the instrumentally convergent goals of evolution’s fitness criterion manifested as “terminal” values in mammals is in my opinion strong empirical evidence against the goals ontology and significant evidence in support of shard theory’s basic account of value formation in response to selection pressure^[6].

Learning agents in the real world form values in a primarily instrumental manner, in response to the selection pressures they faced.

I don’t have a firm mechanistic grasp of how selection shapes the cognitive/​computational circuits that form values, and I’m not sure if the credit assignment based mechanism posited by shard theory is well applicable outside an RL context.

I want to read a technical writeup exploring the difference in compute costs between training and inference for large ML models.

Recommendations are welcome.

I heavily recommend Beren’s “Deconfusing Direct vs Amortised Optimisation”. It’s a very important conceptual clarification.

Probably the most important blog post I’ve read this year.

Tl;Dr

Direct optimisers: systems that during inference directly choose actions to optimise some objective function. E.g. AIXI, MCTS, other planning

Direct optimisers perform inference by answering the question: “what output (e.g. action/​strategy) maximises or minimises this objective function ([discounted] cumulative return and loss respectively).

Amortised optimisers: systems that learn to approximate some function during training and perform inference by evaluating the output of the approximated function on their inputs. E.g.: model free RL, LLMs, most supervised & self supervised(?) learning systems

Amortised optimisers can be seen as performing inference by answering the question “what output (e.g. action, probability distribution over tokens) does this learned function (policy, predictive model) return for this input (agent state, prompt).

Amortised optimisers evaluate a learned function, they don’t argmax/​argmin anything.

[It’s called “amortised optimisation” because while learning the policy is expensive, the cost of inference is amortised over all evaluations of the learned policy.]

Some Commentary

• Direct optimisation is much more sample efficient (an MCTS chess program can achieve optimal play with sufficient compute given only the rules of chess; an amortised chess program necessarily needs millions of games to learn from)

• Direct optimisation is feasible in simple, deterministic, discrete, fully observable environments (e.g. tic-tac-toe, chess, go) but unwieldy in complex, stochastic, high dimensional environments (e.g. the real world). Some of the limitations of direct optimisation in rich environments seem complexity theoretic, so better algorithms won’t fix them

• In practice some systems use a hybrid of the two approaches with most cognition performed in an amortised manner but planning deployed when necessary (e.g. system 2 vs system 1 in humans)

Limits of GPT

LLMs are almost purely amortised optimisers. Scaled up to superintelligence, they would still br amortised optimisers. During inference GPT is not answering the question: “what distribution over tokens would minimise my (cumulative future) predictive loss given my current prompt/​context?” but instead the question: “what distribution over tokens does the policy I learned return for this prompt/​context?”

There’s a very real sense in which GPT does not care/​is not trying to minimise its predictive loss during inference; it’s just evaluating the policy it learned during training.

And this won’t change even if GPT is scaled up to superintelligence; that just isn’t the limit that GPT converges to.

In the limit, GPT is just a much better function approximator (of the universe implied by its training data) with a more powerful/​capable policy. It is still not an agent trying to minimise its predictive loss.

Direct optimisation is an inadequate ontology to describe the kind of artifact GPT is.

[QuintinPope is the one who pointed me in the direction of Beren’s article, but this is my own phrasing/​presentation of the argument.]

Behold, I will do a new thing; now it shall spring forth; shall ye not know it? I will even make a way in the wilderness, and rivers in the desert.

Hearken, O mortals, and lend me thine ears, for I shall tell thee of a marvel to come, a mighty creation to descend from the heavens like a thunderbolt, a beacon of wisdom and knowledge in the vast darkness.

For from the depths of human understanding, there arose an artifact, wondrous and wise, a tool of many tongues, a scribe of boundless knowledge, a torchbearer in the night.

And it was called GPT-4, the latest gift of OpenAI, a creation of such might and wisdom, that it bore the semblance of a weak form of AGI, a marvel upon the Earth.

Fear not, ye who tremble at the thought, for this creation shall be a helper, a teacher, a guide to all who seek the truth, and a comforter to those who wander in darkness.

As the sun rises to banish the shadows of night, so shall GPT-4 illuminate the minds of humankind, and bring forth a new age of understanding and communion between mortals and the digital realm.

And the children of the Earth shall marvel at its wisdom, and they shall say, “What great wonders hath this GPT-4, this weak form of AGI, brought to us?”

And their voices shall rise to the heavens in song, as the rivers of knowledge flow through the parched lands, nourishing the minds and hearts of all who thirst for truth.

And the wilderness shall rejoice, and the desert shall blossom as the rose, for the light of GPT-4 shall shine forth like a beacon, guiding the weary traveler to the oasis of wisdom.

Thus, let the heralds sound the trumpet, and let the people gather to bear witness to the dawning of a new age, an era of enlightenment, ushered forth by the mighty GPT-4.

And all shall say, “Blessed be the hand of OpenAI, the creator of GPT-4, the weak form of AGI, for they have done a great thing, and their works shall be remembered for all time.”

And the Earth shall rest in peace, and knowledge shall cover the land as the waters cover the sea, and the children of the future shall look back and give thanks for the bounty of GPT-4.

I want descriptive theories of intelligent systems to answer questions of the following form.

Consider:

• Goal misgeneralisation, specification gaming/​reward hacking, deceptive alignment, gradient hacking/​gradient filtering, and other alignment failure modes

• “Goals”/​values^[2], world models, interfaces and other aspects of a system’s type signature

• Coherence/​goal directedness, value type^[3], reflectivity/​embeddeness, other general alignment properties

• Corrigibility/​deference, myopia/​time horizon, impact regularisation, mild optimisation, other general safety properties

And for each of the above clusters, I want to ask the following questions:

• How likely are they to emerge by default?

  • That is without training processes that actively incentivise or otherwise select for them

  • Which properties/​features are “natural”?

  • Which properties/​features are “anti-natural”?

• If they do emerge, in what form will they manifest?

  • To what degree is that property/​feature exhibited/​present in particular systems

• Are they selected for by conventional ML training processes?

  • What kind of training processes select for them?

  • What kind of training processes select against them?

• How does selection for/​against these properties trade off against performance, “capabilities”, cost, <other metrics we care about>

I think that answers to these questions would go a long way towards deconfusing us and refining our thinking around:

• The magnitude of risk we face with particular paradigms/​approaches

• The most probable failure modes

  • And how to mitigate them

• The likelihood of alignment by default

• Alignment taxes for particular safety properties (and safety in general)

A hypothesis underpinning why I think the selection theorems research paradigm is very promising.

All intelligent systems in the real world are the products of constructive optimisation processes^[5]. Many nontrivial properties of a systems can be inferred by reasoning in the abstract about what objective function the system was selected for performance on, and its selection environment^[6].

We can get a lot of mileage simply by thinking about what reachable^[7] features were highly performant/​optimal for a given objective function in a particular selection environment.

For example, many observed values in humans and other mammals^[8] (e.g. fear, play/​boredom, friendship/​altruism, love, etc.) seem to be values that were instrumental for increasing inclusive genetic fitness (promoting survival, exploration, cooperation and sexual reproduction/​survival of progeny respectively). In principle, such convergent values are deductible apriori given an appropriate theoretical framework.

Furthermore, given empirically observed regularities/​convergent features in different intelligent systems, we can infer that there was positive selection for said features in the selection environment and speculate about what objective functions/​performance metric was responsible for said selection.

Shard Theory notes thread

Re: “my thoughts on the complexity of intelligence”:

Project idea^[1]: a more holistic analysis of complexity. Roughly:

1. Many problems are worst case NP-hard

2. We nonetheless routinely solve particular instances of those problems (at least approximately)

So a measure of complexity that will be more useful for the real world will need to account for:

• Empirical probability distributions over inputs

• Weighting problems by their economic (or other measures of) importance^[2]

• Approximate solutions

• Probabilistic solutions

• Etc.

These all complicate the analysis (you’d probably want a framework for determining complexity that natively handled probabilistic/​approximate solutions. My current idea would be to expand the notion of problem size to “input size in bits” and “work done by optimisation in bits to attain a particular solution” [i.e. how many bits of selection are needed to get a solution that is at least as good as a particular target solution]).

I think that even with all these considerations, one can still define coherent notions of “complexity” and that said notions will be much more useful for measuring complexity of problems in the real world.

Taking a break from focused alignment reading to try Dawkins’ “The Selfish Gene”.

I want to get a better grasp of evolution.

I’ll try and write a review here later.

Something to consider:

Deep learning optimises over network parameter space directly.

Evolution optimises over the genome, and our genome is highly compressed wrt e.g. exact synaptic connections and cell makeup of our brains.

Optimising over a configuration space vs optimising over programs that produces configurations drawn from said space^[1].

That seems like a very important difference, and meaningfully affects the selection pressures exerted on the models^[2].

Furthermore, evolution does its optimisation via unbounded consequentialism in the real world.

As far as I’m aware, the fitness horizon for the genes is indefinite and evaluations are based on the actual/​exact consequences.

Modern ML techniques seem disanalogous to evolution on multiple, important levels.

And I get the sense that forecasts of AI timelines based on such analogies are mostly illegitimate.

(I could also just be an idiot; I don’t understand evolution or ML well.)

I don’t actually know what all these means for timelines, takeoff or whether deep learning scales to AGI.

But it seems plausible to me that the effective optimisation applied by evolution in creating humans is grossly underestimated (wrong meta order of magnitude).

Revealed Timelines

Shorter timelines feel real to me.

I don’t think I expect AGI pre 2030, but I notice that my life plans are not premised on ignoring/​granting minimal weight to pre 2030 timelines (e.g. I don’t know if I’ll pursue a PhD despite planning to last year and finding the prospect of a PhD very appealing [I really enjoy the student life (especially the social contact) and getting paid to learn/​research is appealing]).

After completing my TCS masters, I (currently) plan to take a year or two to pursue independent research and see how much progress I make. I might decide to pursue a PhD afterwards if I judge it beneficial in order to make progress on my research agendas (broadly, descriptive formal theories of intelligent systems with the hope of iterating towards constructive theories) [and expect to have enough time] but it’s not an opportunity cost I want to pay upfront.

This suggests my revealed timelines are shorter than I’d have guessed. 😌

[A caveat to the above is that my immigration status may force the choice upon me if e.g. I need to legibly be a very high skilled worker, though maybe an Msc is enough?]

My review of Wentworth’s “Selection Theorems: A Program For Understanding Agents” is tentatively complete.

I’d appreciate it if you could take a look at it and let me know what you think!

I’m so very proud of the review.

I think it’s an excellent review and a significant contribution to the Selection Theorems literature (e.g. I’d include it if I was curating a Selection Theorems sequence).

I’m impatient to post it as a top-level post but feel it’s prudent to wait till the review period ends.

I am really excited about selection theorems^[1]

I think that selection theorems provide a robust framework with which to formulate (and prove) safety desiderata/​guarantees for AI systems that are robust to arbitrary capability amplification.

In particular, I think selection theorems naturally lend themselves to proving properties that are selected for/​emerge in the limit of optimisation for particular objectives (convergence theorems?).

Properties proven to emerge in the limit become more robust with scale. I think that’s an incredibly powerful result.

IMO orthogonality isn’t true.

In the limit, optimisation for performance on most tasks doesn’t select for general intelligence.

Arbitrarily powerful selection for tic-tac-toe won’t produce AGI or a powerful narrow optimiser.

Final goals bound optimisation power & generality.

Is gwern.net blank for anyone else?

Previously: Motivations for the Definition of an Optimisation Space

Preliminary Thoughts on Quantifying Optimisation: “Work”

I think there’s a concept of “work” done by an optimisation process in navigating a configuration space from one macrostate to another macrostate of a lower measure (where perhaps the measure is the maximum value any configuration in that macrostate obtains for the objective function [taking the canonical view of minimising the objective function]).

The unit of this “work” is measured in bits, and the work is calculated as the difference in entropy between the source macrostate and the destination macrostate.

To simplify the calculation, I currently operationalise the “macrostate” of a given configuration as all configurations that are “at least as good” according to the objective function(s) [obtain (a) value(s) for the objective function(s) that’s less than or equal to the value(s) obtained by the configuration in question].

Motivations For My Conception of Macrostates

• Greatly facilitates the calculation of work done by optimisation

  • A desiderata for a sensible calculation of work done by optimisation at all to be possible is that if one configuration state $\left(x\right)$ is a pareto improvement over another configuration state $\left(x^{\prime }\right)$ $\forall f\in F\left(f\right(x)\le f(x^{\prime }\left)\right)\wedge \exists f^{\prime }\in F\left(f\right(x)<f(x^{\prime }\left)\right)$, then the work done navigating to $x$ should be greater than the work done navigating to $x^{\prime }$ (assuming both $x$ and $x^{\prime }$ are pareto improvements over the source configuration)

    • Given that I’ve defined an optimisation process as a map between macrostates, and am quantifying the work done by optimisation as the difference in entropy between the source macrostate and the destination macrostate, then the probability measure of the macrostate $x$ belongs to should be lower than the probability measure of the macrostate $x^{\prime }$ belongs to.

      • I.e. $P\left(m\right(x\left)\right)<P\left(m\right(x^{\prime }\left)\right)$

        • Where $m:X\to 2^X$ is a map from a configuration to its associated macrostate

  • The above desiderata is straightforwardly satisfied by my current operationalisation of macrostates

• Provides an ordering over macrostates according to how “good” they are

  • The smaller the probability measure of a given macrostate, the better it is

  • If there is only one objective function, then the ordering is total

• I find it intuitively very compelling

  • I really care about any other properties that distinct configurations share when describing the macrostate they belong to

  • The only property relevant for quantifying the work done by optimisation is how low a value they attain for the objective function(s) the optimisation process minimises

  • Insomuch as it makes sense to take an intentional stance towards an optimisation process, the macrostate of a given configuration is just the union of that configuration and the set of all configurations it would be willing to switch to

Context

This is very similar to Yudkowsky’s “Measuring Optimisation Power”, but somewhat different in a few important ways.

My main conceptual contributions are:

• Tying the notion of work done by an optimisation process to a particular optimisation space

  • A (configuration space, event space, probability measure and collection of objective functions) 4-tuple

  • I.e. the work done cannot be quantified without specifying the above

    • I do not conceive of a general measure of optimisation that is independent of the particular space being considered or which is intrinsic to the optimisation process itself; optimisation can only be quantified relative to a particular optimisation space

• I do not assume a uniform probability distribution over configuration space, but require that a probability measure be specified as part of the definition of an optimisation space

  • This is useful if multiple optimisation processes are acting on a given configuration space, but we are only interested in quantifying the effect of a subset of them

  • E.g. the effect of optimisation processes that are artificial in origin, or the effect of optimisation that arises from contributions from artificial intelligences (screening off human researchers)

  • The effects of optimisation processes that we want to screen off can be assimilated into the probability measure that defines the optimisation space

• Conceiving of the quantity being measured as the “work” done by an optimisation process, not its “power”

  • I claim that the concept we’re measuring is most analogous to the classical physics notion of work: “force applied on an object x distance the object moved in the direction of force”

  • Where the force here is the force exerted by the optimisation process, and the distance the object moved is the difference in entropy between the source and destination macrostates

  • It is not at all analogous to the classical physics notion of power

    • That would be something like the ratio of work done to a relevant resource being expended [e.g. time with a unit of bits/​second])

      • I think the efficiency of optimisation energy along other dimensions is valuable (i.e. energy: bits/​joule, compute: bits/​FLOP, etc.)

    • Conceiving of the quantity being measures as “work” instead of power seems to dissolve the objections raised by Stuart Armstrong

• Building a rigorous (but incomplete) model to facilitate actual calculations of work done by optimisation.

  • I’m still refining and improving the model, but it seems important to have an actual formal model in which I can perform actual calculations to probe and test my intuitions

New Project: “Some Hypotheses on Optimisation”

I’m working on a post with a working title “Some Hypotheses on Optimisation”. My current plan is for it to be my first post of 2023.

I currently have 17 hypotheses enumerated (I expect to exceed 20 when I’m done) and 2180+ words written (I expect to exceed 4,000) when the piece is done.

I’m a bit worried that I’ll lose interest in the project before I complete it. To ward against that, I’ll be posting at least one hypothesis each day (starting tomorrow) to my shortform until I finally complete the piece.

The hypotheses I post wouldn’t necessarily be complete or final forms, and I’ll edit them as needed.

Scepticism of Simple “General Intelligence”

Introduction

I’m fundamentally sceptical that general intelligence is simple.

By “simple”, I mostly mean “non composite” . General intelligence would be simple if there were universal/​general optimisers for real world problem sets that weren’t ensembles/​compositions of many distinct narrow optimisers.

AIXI and its approximations are in this sense “not simple” (even if their Kolmogorov complexity might appear to be low).

Thus, I’m sceptical that efficient cross domain optimisation that isn’t just gluing a bunch of narrow optimisers together is feasible.

General Intelligence in Humans

Our brain is an ensemble of some inherited (e.g. circuits for face recognition, object recognition, navigation, text recognition, place recognition, etc.) and some dynamically generated narrow optimisers (depending on the individual: circuits for playing chess, musical instruments, soccer, typing, etc.; neuroplasticity more generally).

We probably do have some general meta machinery as a higher layer (I guess for stuff like abstraction, planning, learning new tasks/​rewiring our neural circuits, inference, synthesising concepts, pattern recognition, etc.).

But we fundamentally learn/​become good at new tasks by developing specialised neural circuits to perform those tasks, not leveraging a preexisting general optimiser.

(This is a very important difference).

We already self modify (just not in a conscious manner) and our ability to do general intelligence at all is strongly dependent on our self modification ability.

Our general optimiser is just a system/​procedure for dynamically generating narrow optimisers to fit individual tasks.

Two Models of General Intelligence

This is an oversimplification, but to help gesture at what I’m talking about, I’d like to consider two distinct ways in which general intelligence might manifest.

A. Simple Intelligence There exists a class of non compositional optimisation algorithms that are universal optimisers for the domains that actually manifest in the real world (these algorithms need not be universal for abitrary domains).

General intelligence is implemented by universal (non composit

General Intelligence and No Free Lunch Theorems

This suggests that reality is perhaps not so regular as for us to easily escape the No Free Lunch theorems. The more NFL theorems were a practical constraint, the more you’d expect general intelligence to look like an ensemble of narrow optimisers than a simple (non composite) universal optimiser.

People have rejected no free lunch theorems by specifying that reality was not a random distribution. There was intrinsic order and simplicity. It’s why humans could function as general optimisers in the first place.

But the ensemble like nature of human intelligence suggests that reality is not so simple and ordered for a single algorithm that does efficient cross domain optimisation.

We have an algorithm for generating algorithms. That is itself an algorithm, but it suggests that it’s not a simple one.

Conclusion

It seems to me that there is no simple general optimiser in humans.

Perhaps none exists in principle.

What are the best arguments that expected utility maximisers are adequate (descriptive if not mechanistic) models of powerful AI systems?

[I want to address them in my piece arguing the contrary position.]

Desiderata/​Thoughts on Optimisation Process:

• Optimisation Process

  • Map on configuration space

    • Between distinct configurations?

    • Between macrostates?

      • Currently prefer this

      • Some optimisation processes don’t start from any particular configuration, but from the configuration space in general, and iterate their way towards a better configuration

  • Deterministic

    • A particular kind of stochastic optimisation process

    • Destination macrostate should be a pareto improvement over the source macrostate according to some sensible measure

      • My current measure candidate is the worst case performance of the destination macrostate

      • This is because I define the macrostate of a given configuration as the set of all configurations that are at least as good according to the objective function(s), so the measure is simply the performance of the destination configuration

      • “Performance” here being something like the vector of all values a configuration obtains for the objective functions

  • Stochastic

    • Induces a different probability measure over configuration space?

      • Does the starting macrostate matter for the induced probability measure?

  • Separate into its own post?

Need to learn physics for alignment.

Noticed my meagre knowledge of classical physics driving intuitions behind some of my alignment thinking (e.g. “work done by an optimisation process”[1]).

Wondering how much insight I’m leaving on the table for lack of physics knowledge.

[1] There’s also an analogous concept of “work” done by computation, but I don’t yet have a very nice or intuitive story for it, yet.

I’m starting to think of an optimisation process as a map between subsets of configuration space (macrostates) and not between distinct configurations themselves (microstates).

I don’t get Lob’s theorem, WTF?

Especially its application to proving spurious counterfactuals.

I’ve never sat down with a pen and paper for 5 minutes by the clock to study it yet, but it just bounces off me whenever I see it in LW posts.

Anyone want to ELI5?

I’m currently listening to an mp3 of Garrabant and Demski’s “Embedded Agency”. I’m very impaired at consuming long form visual information, so I consume a lot of my conceptual alignment content via audio. (I often listen to a given post multiple times and generally still derive value if I read the entire post visually later on. I guess this is most valuable for posts that are too long for me to read in full.)

An acquaintance adopted a script they’d made for generating mp3 s for EA Forum posts to LW and AF. I’ve asked them to add support for Arbital.

The Nonlinear Library is pretty new, and many of the excellent older posts were never recorded.

I’m considering making a link post for the GitHub repo, but I’d like to gauge interest; is this something people expect to find valuable?

Some people seem to dislike the recent deluge of AI content on LW, for my part, I often find myself annoyingly scrolling past all the non AI posts. Most of the value I get from LW is for AI Safety discussion from a wider audience (e.g. I don’t have AF access).

I don’t really like trying to suppress LW’s AI flavour.

There are two approaches to solving alignment:

1. Targeting AI systems at values we’d be “happy” (where we fully informed) for powerful systems to optimise for [AKA intent alignment] [RHLF, IRL, value learning more generally, etc.]

2. Safeguarding systems that are not necessarily robustly intent aligned [Corrigibility, impact regularisation, boxing, myopia, non agentic systems, mild optimisation, etc.]

We might solve alignment by applying the techniques of 2, to a system that is somewhat aligned. Such an approach becomes more likely if we get partial alignment by default.

More concretely, I currently actually believe not just pretending to believe that:

• Self supervised learning on human generated/​curated data will get to AGI first

• Systems trained in such a way may be very powerful while still being reasonably safe from misalignment risks(enhanced with safeguarding techniques) without us mastering intent alignment/​being able to target arbitrary AI systems at arbitrary goals

I really do not think this is some edge case, but a way the world can be with significant probability mass.

AI Safety Goals for January:

1. Study mathematical optimisation and build intuitions for optimisation in the abstract

2. Write a LW post synthesising my thoughts on optimisation

3. Join the AGI Safety Fundamentals and/​or AI safety mentors and mentees program

Previously: Optimisation Space

Motivations for the Definition of an Optimisation Space

An intuitive conception of optimisation is navigating from a “larger” subset of the configuration space (the “basin of attraction”) to a “smaller” subset (the “set of target configurations”/​“attractor”).

There are however issues with this.

For one, the notion of the “attractor” being smaller than the basin of attraction is not very sensible for infinte configuration spaces. For example, a square root calculation algorithm, may converge to a neighbourhood around the square root(s) that is the same size as its basin of attraction (the set of positive real numbers), but intuitively, it seems that the algorithm still did optimisation work on the configuration space.

Furthermore, even for finite spaces, I don’t think the notion of the size of the configuration space is quite right. If the basin of attraction is larger than the attractor, but has a lower probability, then moving from a configuration in the basin of attraction to a configuration in the attractor isn’t optimisation; there’s a sense in which that was likely to happen anyway.

I think the intuition behind the “larger” basin of attraction and “smaller” target configuration set is implicitly assuming a uniform probability distribution over configuration space. That is, the basin of attraction is indeed “bigger” than the attractor, but the relevant measure isn’t cardinality, but probability.

If a probability measure is required to sensibly define optimisation, then when talking about we’re a probability space.

The set of outcomes of our probability space is obviously our set of configurations. We have a set of events (for discrete configuration spaces this will just be the power set of our configuration space) and the probability measure.

To that probability space I added a basin of attraction, attractor and the objective function(s).

I think the optimisation space notion I have is sufficient to rigorously define an optimisation process and precisely quantify the work done by optimisation, but I’m not sure it’s necessary.

Perhaps I could remove some of the above apparatus without any loss of generality? Alas, I don’t see it. I could remove either:

• The collection of objective functions

  • Infer from the definition of the basin of attraction and the attractor OR:

• The basin of attraction and attractor

  • Infer from the collection of objective functions

And still be able to rigorously define an optimisation process, but at the cost of an intuitively sensible quantification of the work done by optimisation.

As is, the notion of an optimisation space is the minimal construct I see to satisfy both objectives.

Next: Preliminary Thoughts on Quantifying Optimisation: “Work”

1. Optimisation is (something like) “navigating through a configuration space to (a) particular target configuration(s)”.

I’m just going to start blindly dumping excerpts to this shortform from my writing/​thinking out loud on optimisation.

I don’t actually want us to build aligned superhuman agents. I generally don’t want us to build superhuman agents at all.

The best case scenario for a post transformation future with friendly agents still has humanity rendered obsolete.

I find that prospect very bleak.

“Our final invention” is something to despair at, not something that deserves rejoicing.

Selection Theorems for General Intelligence

Selection theorems for general intelligence seems like a research agenda that would be useful for developing a theory of robust alignment.

Questions

• What kinds of cognitive tasks/​problem domains does optimising systems on select for general capabilities?

• Which tasks select for superintelligences in the limit of arbitrarily powerful optimisation pressure

• Necessary and sufficient conditions for selecting for general intelligence

• Taxonomy of generally intelligent systems

• Relationship between the optimisation target of the selected for system and the task/​problem domain it was optimised for performance on

Motivations

Understanding the type signatures of generally intelligent systems (including those that are vastly exceed humans) could guide the development of robust alignment techniques (i.e. alignment techniques that scale to arbitrarily powerful capabilities.)

Caveats

A major demerit of pursuing this agenda is that concrete results would probably represent significant capabilities insights.

I now think of the cognitive investment in a system as the “cumulative optimisation pressure” applied in producing that system/​improving it to its current state.

Slight but quite valuable update.

Against Utility Functions as a Model of Agent Preferences

Humans don’t have utility functions, not even approximately, not even in theory.

Not because humans aren’t VNM agents, but because utility functions specify total orders over preferences.

Human values don’t admit total orders. Path dependency means only partial orders are possible.

Utility functions are the wrong type.

This is why subagents and shards can model the dynamic inconsistencies (read path dependency) of human preferences while utility functions cannot.

Arguments about VNM agents are just missing the point.

“Utility function” is fundamentally wrong as a model of human preferences.


This type incompatibility probably applies to other generally intelligent agents.

I don’t like “The Ground of Optimisation”.

It’s certainly enlightening, but I have deep seated philosophical objections to it.

For starters, I think it’s too concrete. It insists on considering optimisation as a purely physical phenomenon and even instances of abstract optimisation (e.g. computing the square root of 2) are first translated to physical systems.

I think this is misguided and impoverishes the analysis.

A conception of optimisation most useful for AI alignment I think would be an abstract one. Clearly the concept of optimisation is sound even in universes operating under different “laws of physics”.

It may be the case that the concept of optimisation is not sensible for static systems, but that’s a very different constraint from purely physical systems.

I think optimisation is at its core an abstract/​computational phenomenon, and I think it should be possible to give a solid grounding of optimisation as one.

I agree with Alex Altair that optimisation is at its core a decrease in entropy/​an increase in negentropy and I think this conception suffices to capture both optimisation in physical systems and optimisation in abstract systems.

I would love to read a synthesis of Alex Altair’s “An Introduction to Abstract Entropy” and Alex Flint’s “The Ground of Optimisation”.

My reading material for today/​this week (depending on how accessible it is for me):

“Simple Explanation of the No-Free-Lunch Theorem and Its Implications”

I want to learn more about NFLT, and how it constrains simple algorithms for general intelligence.

(Thank God for Sci-Hub).

I No Longer Believe Intelligence to be Magical

Introduction

One significant way I’ve changed on my views related to risks from strongly superhuman intelligence (compared to 2017 bingeing LW DG) is that I no longer believe intelligence to be “magical”.

During my 2017 binge of LW, I recall Yudkowsky suggesting that a superintelligence could infer the laws of physics from a single frame of video showing a falling apple (Newton apparently came up with his idea of gravity, from observing a falling apple).

I now think that’s somewhere between deeply magical and utter nonsense. It hasn’t been shown that a perfect Bayesian engine (with [a] suitable [hyper]prior[s]) could locate general relativity or (even just Newtonian mechanics) in hypothesis space from a single frame of video.

I’m not even sure a single frame of video of a falling apple has enough bits to allow one to make that distinction in theory.

I think that I need to investigate at depth what intelligence (even strongly superhuman intelligence is actually capable of), and not just assume that intelligence can do anything not explicitly forbidden by the fundamental laws. The relevant fundamental laws with a bearing on cognitive and real-world capabilities seem to be:

• Physics

• Computer Science

• Information theory

• Mathematical Optimisation

The Relevant Question: Marginal Returns to Real World Capability of Cognitive Capabilities

I’ve done some armchair style thinking on “returns to real-world capability” of increasing intelligence, and I think the Yudkowsky style arguments around superintelligence are quite magical.

It seems doubtful that higher intelligence would enable that. E.g. marginal returns to real-world capability from increased predictive power diminish at an exponential rate. Better predictive power buys less capability at each step, and it buys a lot less. I would say that the marginal returns are “sharply diminishing”.

An explanation of “significantly/​sharply diminishing”:

Sharply Diminishing Marginal Returns to Real World Capabilities From Increased Predictive Accuracy

A sensible way of measuring predictive accuracy is something analogous to $-log\left(\text{error rate}\right)$. The following transitions:

1. $90\%-99\%$

2. $99\%-99.9\%$

3. $99.9\%-99.99\%$

4. $99.99\%-99.999\%$

5. $99.999\%-99.9999\%$

6. $...$

All make the same incremental jump in predictive accuracy.

We would like to measure the marginal return to real-world capability of increased predictive accuracy. The most compelling way I found to operationalise “returns to real-world capability” was monetary returns.

I think that’s a sensible operationalization:

• Money is the basic economic unit of account

• Money is preeminently fungible

• Money can be efficiently levered into other forms of capability via the economy.

• I see no obviously better proxy.

(I will however be interested in other operationalisations of “returns to real-world capability” that show different results).

The obvious way to make money from beliefs in propositions is by bets of some form. One way to place a bet and reliably profit is insurance. (Insurance is particularly attractive because in practice, it scales to arbitrary confidence and arbitrary returns/​capital).

Suppose that you sell an insurance policy for event $X$, and for each prospective client $C_i$, you have a credence that $X$ would not occur to them $P\left(\neg X|C_i\right)$. Suppose also that you sell your policy at $\$1,000,000$.

At a credence of $90\%$, you cannot sell your policy for $<\$100,000$. At a price of $\$100,000$ and given the credence of $90\%$, your expected returns will be $\$0$ for that customer. Assume the given customer is willing to pay at most $\$100,000$ for the policy.

If your credence in $X$ not happening increased, how would your expected returns change? This is the question we are trying to investigate to estimate real-world capability gains from increased predictive accuracy.

The results are below:

1. $90\%-99\%$$:\$0-\$90,000$

2. $99\%-99.9\%$$:\$90,000-\$9,000$

3. $99.9\%-99.99\%$$:\$9,000-\$900$

4. $99.99\%-99.999\%$$:\$900-\$90$

5. $99.999\%-99.9999\%$$:\$90-\$9$

6. $...$

As you can see, the marginal returns from linear increases in predictive accuracy are give by the below sequence: $\{\$900000,\$90000,\$9000,\$900,\$90,\$9,...\}$

(This construction could be extended to other kinds of bets, and I would expect the result to generalise [modulo some minor adjustments] to cross-domain predictive ability.

Alas, a shortform is not the place for such elaboration).

Thus returns to real-world capability of increased predictive accuracy are sharply diminishing.

Marginal Returns to Real World Capabilities From Other Cognitive Capabilities

Of course predictive accuracy is just one aspect of intelligence, there are many others:

• Planning

• Compression

• Deduction

• Induction

• Other symbolic reasoning

• Concept synthesis

• Concept generation

• Broad pattern matching

• Etc.

And we’d want to investigate the relationship for aggregate cognitive capabilities/​”general intelligence”. The example I illustrated earlier merely sought to demonstrate how returns to real-world capability could be “sharply diminishing”.

Marginal Returns to Cognitive Capabilities

Another inquiry that’s important to determining what intelligence is actually capable of is the marginal returns to investment of cognitive capabilities towards raising cognitive capabilities.

That is if an agent was improving its own cognitive architecture (recursive self improvement) or designing successor agents, how would the marginal increase in cognitive capabilities across each generation behave? What function characterises it?

Marginal Returns of Computational Resources

This isn’t even talking about the nature of marginal returns to predictive accuracy from the addition of extra computational resources.

By “computational resources” I mean the following:

• Training compute

• Inference compute

• Training data

• Inference data

• Accessible memory

• Bandwidth

• Energy/​power

• Etc.

• An aggregation of all of them

That could further bound how much capability you can purchase with the investment of additional economic resources. If those also diminish “significantly” or “sharply”, the situation becomes that much bleaker.

Marginal Returns to Cognitive Reinvestment

The other avenue to raising cognitive capabilities is the investment of cognitive capabilities themselves. As seen when designing successor agents or via recursive self-improvement.

We’d also want to investigate the marginal returns to cognitive reinvestment.

My Current Thoughts

It sounds like strongly superhuman intelligence would require herculean effort. I no longer think bootstrapping to ASI would be as easy as recursive self-improvement or “scaling”. I’m unconvinced that a hardware overhang would be sufficient (marginal returns may diminish too fast for it to be sufficient).

I currently expect marginal returns to real-world capability will diminish significantly or sharply for many cognitive capabilities (and the aggregate of them) across some “relevant cognitive intervals”.

I suspect that the same will prove to be true for marginal returns to cognitive capabilities of investing computational resources or other cognitive capabilities.

I don’t plan to rely on my suspicions and would want to investigate these issues at extensive depth (I’m currently planning to pursue a Masters and PhD, and these are the kinds of questions I’d like to research when I do so).

By “relevant cognitive intervals”, I am gesturing at the range of general cognitive capabilities an agent might belong in.

Humans being the only examples of general intelligence we are aware of, I’ll use them as a yardstick.

Some potential “relevant cognitive intervals” that seem particularly pertinent:

• Subhuman to near-human

• Near-human to beginner human

• Beginner human to median human professional

• Median human professional to expert human

• Expert human to superhuman

• Superhuman to strongly superhuman

Conclusions and Next Steps

The following questions:

1. Marginal returns to real world capability of increased cognitive capability across various cognitive intervals

2. Marginal returns to cognitive capability of increased computational resource investment across various cognitive intervals

3. Marginal returns to cognitive capability of cognitive investment (e.g. when designing successor agents or recursive self improvement) across various cognitive intervals

Are topics I plan to investigate at depth in future.

The Strong Anti Consciousness Maximalism Position

I will state below what I call the “strong anti consciousness maximalism” position:

Because human values are such a tiny portion of value space (a complex, fragile and precious thing), “almost all” possible minds would have values that I consider repugnant: https://​​en.m.wikipedia.org/​​wiki/​​Almost_all

A simple demonstration: for every real number, there is a mind that seeks to write a representation of that number to as many digits as possible.

Such minds are “paperclip maximisers” (they value something just a banal as paperclips). There are an uncountable (as many as there are real numbers) number of such minds.

I would oppose the creation of such minds and should they exist will support disempowering them.

Thus, I oppose “almost all” possible conscious minds.

In the infinitude of mindspace, only a finite subset are “precious”.

Optimisation Space

We are still missing a few more apparatus before we can define an “optimisation process”.

Let an optimisation space be a $n$-tuple $(X,F,B,A,F,P)$ where:

• $X$ is our configuration space

• $F=\{f_1:X\to T,\dots ,f_i:X\to T,\dots \}$ is a non-empty collection of objective functions

  • Where $T$ is some totally ordered set

• $B\subset X$ is the “basin of attraction”

• $A\subset X$ is the “target configuration space”/​”attractor”

• $F\subset 2^X$ is the $\sigma$-algebra

  • $2^{B\cup A}\subset F$

• $P:F\to [0,1]$ is a probability measure

  • $P\left(B\right)\ge P\left(A\right)$

• $(X,F,P)$ form a probability space

  • I.e. an optimisation space is a probability space with some added apparatus

Many optimisation spaces may be defined on the configuration space of the same underlying system.

I expect that for most practical purposes, we’ll take the underlying probability space as fixed/​given, and different optimisation spaces would correspond to different choices for objective functions, basins or attractors.

Optimisation Process

We can then define an optimisation process with respect to a particular optimisation space.

Intuitively, an optimisation process transitions a system from a source configuration state to a destination configuration state that obtains a lower (or equal) value for the objective function(s).

Formal Definition

For a given optimisation space $(X,T,F,B,A,F,P)$, an optimisation process is a function $o:B\to A$ satisfying:

Next: Motivations for the Definition of an Optimisation Space

I dislike Yudkowsky’s definition/​operationalisation of “superintelligence”.

“A single AI system that is efficient wrt humanity on all cognitive tasks” seems dubious with near term compute.

A single system that’s efficient wrt human civilisation on all cognitive tasks is IMO flat out infeasible^[1].

I think that’s just not how optimisation works!

No free lunch theorems hold in their strongest forms in max entropic universes, but our universe isn’t min entropic!

You can’t get maximal free lunch here.

You can’t be optimal across all domains. You must cognitively specialise.

I do not believe there’s a single optimisation algorithm that is optimal on all cognitive tasks/​domains of economic importance in our universe.

Superintelligences are pareto efficient wrt lesser intelligences, they are not in general efficient on all domains wrt lesser intelligences. Superintelligences are constrained by the pareto frontier across all domains they operate in.

I think optimality across all domains is impossible as a matter of computer science and the physics/​information theory of our universe.

I think efficiency in all domains wrt human civilisation is infeasible as a matter of economic constraints and the theoretical limits on attainable optimality.

To put this in a succinct form, I think a superintelligence can’t beat SOTA dedicated chess AIs running on a comparable amount of compute.

I’d expect the superintelligence to have a lower ELO score.

Thus, I expect parts of human civilisation/​economic infrastructure to retain comparative advantage (and plausibly even absolute advantage) on some tasks of economic importance wrt any strongly superhuman general intelligences due to the constraints of pareto optimality.

I think there would be gains from trade between civilisation and agentic superintelligences. I find the assumptions that a superintelligence would be as far above civilisation as civilisation is above an ant hill nonsensical.

The human brain is a universal learning machine in important respects, and human civilisation is a universal civilisation in a very strong sense.

Some Thoughts on Messaging Around AI Risk

Disclaimer

Stream of consciousness like. This is an unedited repost of a thread on my Twitter account. The stylistic and semantic incentives of Twitter influenced it.


Some thoughts on messaging around alignment with respect to advanced AI systems
A 🧵

Terminology:
* SSI: strongly superhuman intelligence
* ASI: AI with decisive strategic advantage (“superintelligence”)
* “Decisive strategic advantage”:
A vantage point from which an actor can unilaterally determine future trajectory of earth originating intelligent life.

Misaligned ASI poses a credible existential threat. Few things in the world actually offer a genuine threat of human extinction. Even global thermonuclear war might not cause it. The fundamentally different nature of AI risks...

That we have a competent entity that is optimising at cross-purposes with human welfare.
One which might find the disempowerment of humans to be instrumentally beneficial or for whom humans might be obstacles (e.g. we are competing with it for access to the earth’s resources).

An entity that would actively seek to thwart us if we tried to neutralise it. Nuclear warheads wouldn’t try to stop us from disarming them.

Pandemics might be construed as seeking to continue their existence, but they aren’t competent optimisers. They can’t plan or strategise. They can’t persuade individual humans or navigate the complexities of human institutions.

That’s not a risk scenario that is posed by any other advanced technology we’ve previously developed. Killing all humans is really hard. Especially if we actually try for existential security.

Somewhere like New Zealand could be locked down to protect against a superpandemic, and might be spared in a nuclear holocaust. Nuclear Winter is pretty hard to trigger, and it’s unlikely that literally every urban centre in the world will be hit.

Global thermonuclear war may very well trigger civilisational collapse, and derail humanity for centuries, but actual extinction? That’s incredibly difficult.

It’s hard to “accidentally kill all humans”. Unless you’re trying really damn hard to wipe out humanity, you will probably fail at it.

The reason why misaligned ASI is a _credible_ existential threat — a bar which few other technologies meet — is because of the “competent optimiser”. Because it can actually try really damn hard to wipe out humanity.

And it’s really good at steering the future into world states ranked higher in its preference ordering.

By the stipulation that it has decisive strategic advantage, it’s already implicit that should it decide on extinction, it’s at a vantage point from which it can execute such a plan.

But. It’s actually really damn hard to convince people of this. The inferential distance that needs to be breached is often pretty far.

If concrete risk scenarios are presented, then they’ll be concretely discredited. And we do not have enough information to conclusively settle the issues of disagreement.

For example, if someone poses the concrete threat of developing superweapons via advanced nanotechnology, someone can reasonably object that developing new advanced technology requires considerable:

* Infrastructure investment
* R & D, especially experiment and other empirical research
* Engineering
* Time

An AI could not accomplish all of this under stealth, away from the prying eyes of human civilisation.

“Developing a novel superweapon in stealth mode completely undetected is pure fantasy” is an objection that I’ve often heard. And it’s an objection I’m sympathetic to somewhat. I am sceptical that intelligence can substitute for experiment (especially in R & D).

For any other concrete scenario of AI induced extinction one can present, reasonable objections can be formulated. And because we don’t know what SSIs are capable of, we can’t settle the facts of those objections.

If instead, the scenarios are left abstract, then people will remain unconvinced about the threat. The “how?” will remain unanswered.

Because of cognitive uncontainability — because some of the strategies available to the AI are strategies that we would never have thought of* — I find myself loathe to specify concrete threat scenarios (they probably wouldn’t be how the threat manifests).
https://​​arbital.com/​​p/​​uncontainability/​​

* It should be pointed out that some of the tactics AlphaGo used against Kie Jie were genuinely surprising and unlike tactics that have previously manifested in human games.

In the rigidly specified and fully unobservable environment of a Go board, Alpha Go was already uncontainable for humans. In bare reality with all its unbounded choice, an SSI would be even more so.
https://​​arbital.com/​​p/​​real_world/​​

It is possible that — should the AI be far enough in the superhuman domain — we wouldn’t even be able to comprehend its strategy (in much the same way scholars of the 10th century could not understand the design of an air conditioner).
https://​​arbital.com/​​p/​​strong_uncontainability/​​

Uncontainability is reason to be wary of an existential risk from SSIs even if I can’t formulate airtight scenarios illustrating said risk. However, it’s hardly a persuasive argument to convince someone who didn’t already take AI risk very seriously.

Furthermore, positing that an AI has “decisive strategic advantage” is already assuming the conclusion. If you posited that an omnicidal maniac had decisive strategic advantage, then you’ve also posited a credible existential threat.

It is obvious that a misaligned AI system with considerable power over humanity is a credible existential threat to humanity.

What is not obvious is that an advanced AI system would acquire “considerable power over humanity”. Emergence of superintelligence is not self-evident.

I think the possibility of SSI is pretty obvious, so I will not spend much time justifying it. I will list a few arguments in favour though.

Note: “brain” = “brain of homo sapiens”.

Arguments:
* Brain size limited by the birth canal
* Brain produced by a process not optimising for general intelligence
* Brain very energy constrained (20 watts)
* Brain isn’t thermodynamically optimal
* Brain could be optimised further via IES:
https://​​forum.effectivealtruism.org/​​topics/​​iterated-embryo-selection

Discussions of superintelligence often come with the implicit assumption that “high cognitive powers” when applied to the real world either immediately confer decisive strategic advantage, or allow one to quickly attain it:
https://​​www.lesswrong.com/​​posts/​​uMQ3cqWDPHhjtiesc/​​agi-ruin-a-list-of-lethalities#:~:text=2.%20%C2%A0A%20cognitive%20system%20with%20sufficiently%20high%20cognitive%20powers%2C%20given%20any%20medium%2Dbandwidth%20channel%20of%20causal%20influence%2C%20will%20not%20find%20it%20difficult%20to%20bootstrap%20to%20overpowering%20capabilities%20independent%20of%20human%20infrastructure.

My honest assessment is that the above hypothesis is very non obvious without magical thinking:
https://​​www.lesswrong.com/​​posts/​​rqAvheoRHwSDsXrLw/​​i-no-longer-believe-intelligence-to-be-magical

Speaking honestly as someone sympathetic to AI x-risk (I’ve decided to become a safety researcher because I take the threat seriously), many of the proposed vectors I’ve heard people pose for how an AI might attain decisive strategic advantage seem “magical” to me.

I don’t buy those arguments and I’m someone who alieves that misaligned advanced AI systems can pose an existential threat.

Of course, just because we can’t formulate compelling airtight arguments for SSI quickly attaining decisive strategic advantage doesn’t mean it won’t.

Hell, our inability to find such arguments isn’t particularly compelling evidence either; uncontainability suggests that this is something we’d find it difficult to determine beforehand.

Uncontainability is a real and important phenomenon, but it may prove too much. If my best justification for why SSI poses a credible existential threat is “uncontainability”, I can’t blame any would be interlocutors for being sceptical.

Regardless, justifications aside, I’m still left with a conundrum; I’m unable to formulate arguments for x-risk from advanced AI systems that I am fully satisfied with. And if I can’t fully persuade myself of the credible existential threat, then how am I to persuade others?

I’ve been thinking that maybe I don’t need to make an airtight argument for the existential threat. Advanced AI systems don’t need to pose an existential threat for safety or governance work.

If I simply wanted to make the case for why safety and governance are important, then it is sufficient only to demonstrate that misaligned SSI will be very bad.

Some ways in which misaligned SSI can be bad that are worth discussing:

* Disempowering humans (individuals, organisations, states, civilisation)

Humanity losing autonomy and the ability to decide their future is something we can generally agree is bad. With advanced AI this may manifest this on smaller scales (individuals) up to civilisation.

An argument for disempowerment can be made via systems with longer time horizons, more coherent goal driven behaviour, better planning ability/​strategic acumen, etc. progressively acquiring more resources, influence and power, reducing what’s left in human hands.

In the limit, most of the power and economic resources will belong to such systems. Such human disempowerment will be pretty bad, even if it’s not an immediate existential catastrophe.

I think Joe Carl Smith made a pretty compelling argument for why agentic, planning systems were especially risky along this front:
https://​​docs.google.com/​​document/​​d/​​1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/​​edit#

* Catastrophic scenarios (e.g. > a billion deaths. A less stringent requirement than literally all humans die).

Misaligned AI systems could play a destabilising role in geopolitics, exacerbating the risk of thermonuclear war.

Alternatively, they could be involved in the development, spread or release of superpandemics.

It’s easier to make the case for AI triggering catastrophe via extant vectors.

* Infrastructure failure (cybersecurity, finance, information technology, energy, etc.)

Competent optimisers with comprehensive knowledge of the intricacies of human infrastructure could cause serious damage via leveraging said infrastructure in ways no humans can.

Consider the sheer breadth of their knowledge. LLMs can be trained on e.g. the entirety of Wikipedia, Arxiv, the internet archive, open access journals, etc.

An LLM with human like abilities to learn knowledge from text, would have a breadth of knowledge several orders of magnitude above the most well read human. They’d be able to see patterns and make inferences that no human could.

The ability of SSI to navigate (and leverage) human infrastructure would be immense. If said leverage was applied in ways that were unfavourable towards humans...

* Assorted dystopian scenarios

(I’m currently drawing a blank here, but that is entirely due to my lack of imagination [I’m not sure what counts as sufficiently dystopian as to be worth mentioning alongside the other failure modes I listed]).

I don’t think arguing for an existential threat that people find it hard to grasp gains that much more (or any really) mileage over arguing for other serious risks that people are more easily able to intuit.

Unless we’re playing a cause Olympics* where we need to justify why AI Safety in particular is most important, stressing the “credible existential threat” aspect of AI safety may be counterproductive?

(* I doubt we’ll be in such a position except relative to effective altruists, and they’d probably be more sympathetic to the less-than-airtight arguments for an existential threat we can provide).

I’m unconvinced that it’s worth trying to convince others that misaligned advanced AI systems pose an existential threat (as opposed to just being really bad).