Today, AI surpasses human intelligence across a wide range of well-defined, isolated tasks, but not as a whole. To reach AGI, Artificial General Intelligence, two key elements are still missing: the ability to gather data that AI cannot access through data centres, and an increased capability to integrate and combine all data. These are necessary to develop the key dimensions of causal understanding, robustness and intentionality. This is part 3 in the series on the development of AGI and some of the risks it entails for the economy, governance, democracy, politics and international cooperation. AGI changes who, in practice, controls how decisions are made.
The blog series is structured as follows:
- What is missing to reach AGI? (first post)
- Impact on the economy and governance (previous post)
- The democratic and political risk from AGI (this post)
AGI shifts the balance of power between state, companies and citizens
The two previous posts outlined how AGI is more than a technological development; it will lead to a structural shift across the core processes of society. Those who control AI will, in effect, also be able to control how society functions.
This reinforces the classical tension between the state, civil society, and the individual, as analysed by Georg Wilhelm Friedrich Hegel. However, when intelligence (AGI) is no longer a scarce resource for growth, power can be scaled without a natural counterweight.
This is already visible in the conflict between the state and AI developers
The current conflict between the US Department of Defense and Anthropic illustrates the dynamic. Anthropic has refused to make its models available for, among other things, mass surveillance and autonomous weapons, arguing that the technology is still too immature. In response, the Pentagon has classified the company as a “supply chain risk”. Consequently, Anthropic’s products can no longer be used by, among others, subcontractors to the Pentagon.
At the core of the conflict is the state’s unwillingness to depend on actors that can refuse. This leads to a structural power struggle in which the state holds an inherent advantage through regulation, contracts, and control over infrastructure. As a result, companies’ fundamental ethical boundaries risk being eroded.
The conflict is an early example of a structural tension that is likely to recur across sectors and countries.
AI makes surveillance scalable, …
Historically, state surveillance has been constrained by capacity. That constraint disappears, as AI enables the analysis of vast amounts of data in real time, the identification of patterns and anomalies, and the prediction of behaviour with high precision and low uncertainty.
This creates an unprecedented scalability of mass surveillance at very low marginal cost. At the same time, AI’s ability to hyper-personalise communication provides the ingredients for a demagogue’s path to despotism.
… shifting power from institutional to technological control
Democracies rely on visible institutional control through legislation, separation of powers, and accountability. By design, this control is relatively slow and contestable. AGI, however, introduces technological control, where power is based on opaque access to data, compute, and models. This makes it less visible, more scalable, and harder to regulate and challenge.
At the same time, there is a significant asymmetry of information between developers and society, which limits external oversight
Two development paths are particularly likely, …
The trajectory may vary depending on speed and adaptation. In simplified terms, this can be described as two paths:
- Gradual adaptation, where institutions are able to adjust, labour markets and policy evolve incrementally, and democratic control is preserved
- Transformative acceleration, where AI is rapidly deployed across all sectors, power concentrates among a few actors, and institutions fall behind and are forced into reactive adjustments
In the latter scenario, the risk of economic and political instability increases significantly, as the pace of development exceeds institutional capacity to adapt.
… both of which require global coordination
The challenge is reinforced by geopolitics. AI is a strategic technology on par with energy, defence, and finance, but it directly affects decision-making and control of information. It can therefore create strategic dependencies on other states or actors, and states will structurally prioritise control over ethics. Cooperation becomes more difficult, and standards fragment.
At the same time, the technology is global, while regulation remains national. This creates a structural mismatch in which no single actor can control development, yet all have incentives to accelerate.
Ethics is necessary, but insufficient, …
Ethical AI therefore requires both national and international frameworks, as well as active state involvement. Companies cannot enforce ethical constraints on their own.
However, states have incentives to bypass such constraints if others are willing to supply the technology when one actor refuses. In the Anthropic case above, OpenAI stepped in immediately when Anthropic declined.
The problem therefore cannot be solved through voluntary measures or isolated regulation.
… because alignment is the deepest challenge
The fundamental question is therefore: who should AI be aligned with? Neither ethics nor current legal frameworks provide a clear answer as to whether it should be the state, the company, the user, or something else entirely.
At the same time, history shows that blind obedience, even to legitimate authority, often leads to severe consequences. “Perfect alignment” may, in practice, reflect an absence of resistance, and is therefore not necessarily desirable.
… making the core risk structural
In summary, three central risks emerge:
- Concentration of power, as AI may consolidate economic and political control among a small number of actors
- Erosion of autonomy, as as AI can influence decisions and behaviour at scale
- Institutional lag, as as governance struggles to keep pace with technological development
The key point is that these risks arise from structure, not necessarily from intent.
In conclusion, simple solutions are lacking
AGI fundamentally challenges the preconditions for democracy. It also challenges the economic assumption that individuals make informed and independent decisions. If surveillance becomes pervasive, influence becomes precise, and decision-making becomes automated, democracy risks remaining formally intact but substantively hollow.
The central challenge is therefore not the technology itself, but how it is embedded within power structures. At present, there are no simple solutions. The priority must be to protect human autonomy and institutional legitimacy.
Democracies are designed for actors that are broadly comparable across key dimensions. AGI is superior across all such dimensions and can simultaneously shape human decision-making.
- Can power in an AI-driven society remain accountable?
- And can AI, as an agent of power, itself be held accountable?