AGI & ASI and the Governance Dilemma: How do we control the new intelligence?
The development of Artificial General Intelligence (AGI) and Artificial Superintelligence (ASI) has the potential to profoundly impact political structures, social systems, and the governance of societies. A fundamental question that arises from this development is: what will AGI optimise for? Unlike human intelligence, which is characterised by the integration of emotions and ethical reasoning, AGI functions as an optimisation machine. If it is controlled by a select few, there is a risk that it will reinforce existing power structures rather than serving the interests of humanity. It is therefore essential to establish a democratic governance framework to ensure that the benefits of AGI are distributed equitably and do not accrue to the privileged elite.
The Optimization Problem
AGI will possess broad reasoning and learning capabilities, potentially surpassing human intelligence. ASI would extend further, reaching levels incomprehensible to us. The central concern is what AGI will optimize for, as optimization without ethical constraints could lead to dangerous consequences.
Historically, AI optimizes for specific metrics—accuracy in medicine, efficiency in logistics, or engagement in social media. Without governance, AGI’s goals will be dictated by those in power, leading to:
Corporate AGI prioritizing profit, exacerbating economic inequality.
Government-controlled AGI reinforcing political dominance.
Military AGI increasing global instability.
Or even a combination of all three, as we might currently observe in the United States.
Competing Perspectives 🫧
Decentralization by Ben Goertzel’s Vision
Ben Goertzel, CEO of SingularityNET, warns that centralized AGI could serve only elite interests. He advocates for open-source, decentralized AGI governed by a collective, ensuring broader representation in decision-making. Enabling people to vote on major objectives, enabled by web3. [1]
Singleton Theory by Nick Bostrom
Philosopher Nick Bostrom suggests a global Singleton—one entity governing AGI to prevent competitive risks. However, centralizing control could enable totalitarian rule over humanity’s future, suggesting different solutions to deal with this topic. [2]
The Alignment Problem by Stuart Russell
AI researcher Stuart Russell argues AGI must be aligned with human values to prevent unintended harm. Without ethical constraints, AGI might optimize for goals that conflict with human survival, such as maximizing efficiency by shutting down human activity. Russell calls for global cooperation to ensure AGI aligns with ethical principles. [3]
AGI and Societal Inequality by Yuval Noah Harari
Historian Yuval Noah Harari warns of AGI-driven social stratification, where intelligence-controlled economies create an elite ruling class. He advocates for international oversight to prevent growing inequalities. [4]
A Chinese Philosophical Perspective: Balancing Power and Governance
Traditional Chinese philosophy, such as discussed in “Avatar: The Last Airbender”, provides insights into the problems that even well-intended efforts toward a more globally unified society might lead to unintended harm and violence. This perspective suggests that rather than a single entity holding centralized control, societies should be structured with interdependent systems that maintain balance. There should not be a limited group that has influence over everything and absolute control. However, there may be a need for a philosophical council or guiding body that helps steer AGI development toward equilibrium rather than unchecked dominance. These insights highlight the importance of building self-correcting and interdependent systems that allow societies to evolve without falling into either extreme authoritarianism or complete chaos. [5]
The Core Problem: AGI Without Governance Serves Power, Not Humanity
Without democratic oversight, AGI will serve the priorities of its developers and funders, leading to:
Economic Inequality: Corporate-driven AGI will concentrate wealth.
Political Manipulation: AI-controlled governance could erode democracy.
Ethical Blind Spots: AGI optimized solely for efficiency may make inhumane decisions.
The question remains: Who decides AGI’s optimization rules?
Proposed Solutions for Democratic AGI Governance
To prevent monopolization and ensure AGI serves humanity, governance must rest on three pillars:
Global Democratic Oversight: An international AI governance council representing diverse stakeholders must ensure transparency and guide AI development. AI developers should be required to disclose objectives and safeguards.
Decentralized Development: Open-source AGI frameworks should prevent control by a select few, encouraging community-driven AI, much like Goertzel’s SingularityNET approach. This allows users to vote on optimization goals and ensures AGI development remains distributed.
Ethical and Human-Centered Design: AGI must integrate fairness, accountability, and compassion. Ethical constraints should be embedded into its learning mechanisms to ensure alignment with human values, avoiding purely profit-driven or power-centric decision-making.
Conclusion: A Future We Must Shape Together
AGI governance represents a substantial ethical challenge, and it is imperative to recognise that it is one of the most significant ethical dilemmas of our time. In the absence of intervention, AGI's objectives will be dictated by those in positions of power, resulting in the reinforcement of inequality, authoritarianism and corporate dominance.To avert this scenario, it is essential to facilitate democratic participation in defining the purpose of AGI, commencing with a philosophical approach. The development of AGI must be transparent, accountable, and ethically guided to ensure that intelligence serves all of humanity, and not merely the privileged few.It is imperative to avoid the establishment of global governance at this juncture, until such time as we have successfully colonised more stars.
The world is in our hands, not only for ourselves but for all sentient beings and future generations. This represents the most significant information governance challenge of human civilisation, yet it remains underrepresented in both the media and politics. Technocrats are on the horizon, gaining control over these systems and governments.
Referneces
[1] Montes, G. A., & Goertzel, B. (2019). Distributed, decentralized, and democratized artificial intelligence. Technological Forecasting and Social Change, 141, 354–358. https://doi.org/10.1016/j.techfore.2018.11.010
[2] Bostrom, N. (2005). Future of Humanity Institute. Faculty of Philosophy, Oxford University. Retrieved from www.nickbostrom.com
[3] Russell, S. (2019). Human compatible: Artificial intelligence and the problem of control. Viking.
[4] Harari, Y. N. (2024). Nexus: A brief history of information networks from the Stone Age to AI. Random House.
[5] Wikipedia. (n.d.). Avatar: The Last Airbender. In Wikipedia, from https://en.wikipedia.org/wiki/Avatar:_The_Last_Airbender