Norbert Wiener's Moral and Technical Consequences of Automation
You get what you ask for from AI, so make sure your request is truly want you want...
A few months ago, while playing pickup soccer, a teammate implored me to score some goals. I then proceeded to score an own goal from an unfortunate deflection of an opponent shot. I then turned to my teammate and sarcastically told him that he never specified on which type of goal he wanted, so I met his objective of scoring a goal.
This story highlighting how well intended goals that get lost in translation can lead to unintended consequences. While the miscommunication is minor and results are not very dire in this example, these two aspects are magnified when we are talking about two agents that are vastly different from each other, like humans and machines (AI).
Norbert Wiener, the founder of Cybernetics, famously highlighted this in his 1960 paper, called “Some Moral Technical Consequences of Automation”, which brought forth the alignment problem, which is still unresolved as of writing.
The problem can be boiled down to how do we get AI systems to do what we want to them do. This concept is largely derived from the economics concept of the principal-agent problem, except we are talking about two foreign agents which complicates the matter.
While not as popular as Alan Turing’s famous 1950 paper, Computing Machinery and Intelligence, that laid the framework for the Turing Test, I think the paper from Wiener might have more consequential impacts with regarding to the societal impact of human-AI interactions.
Reflections on Paper (not AI-generated)
High level Summary
Wiener provides some compelling reasons for why communication between humans and machines could not go as we would like.
“If the communication between two foreign agencies as to the nature of this purpose is incomplete, it must only be expected that the results of this co-operation will be unsatisfactory”
His conclusion is that when trying to align AI systems with our human goals/preferences/values, it is very important for us to make sure we are asking what we truly desire clearly and with all edge cases in mind. As the deployments of AI social media algorithms and these nascent Generative AI systems have shown already, it is hard to determine the edge cases before deployment leading to some pretty dire societal impacts…
To highlight unintended consequences, lack of control and the importance of foresight, he highlights "The Sorcerer's Apprentice" and “The Monkey’s Paw” fables, which eloquently capture the hubris and confounding risk-seeking behavior of AI labs.
Section breakdown
Introduction
Cybernetics defined: “problems of control and communication in the living organism and the machine”
Pushing back on idea that machines can’t posses any degree of originality. Machines can and do transcend the limitation of their designers (and data sets) and in doing so make them both effective and dangerous.
Machines can already perform some tasks much quicker than humans and our slowness in reacting to these growing capabilities might make our control of these systems ineffective.
“By the time we are able to react to information conveyed by our senses and stop the car we are driving, it may already have run head on into a wall”
Game-Playing
A pretty confusing section where he talks about applying the first principles of Game Theory to various games and non-games. He starts with tic-tac-toe and explains it has a complete theory which is not the case for checkers, chess, or other more non-trivial games like war.
upon reflecting on this more with Mistral AI, this section seemed to highlight how much more complicated real world decision making could be for machines that can lead to sub-optimal outcomes due to stochastic events, imperfect information and bounded rationality (like in poker)
There is a section on the meta-game within games based on the judgment of the other participants is discussed but not much is done to tie this into the discussion are human and machine interactions.
Learning Machines
The next step in machines is having the ability to reason by assessing current conditions and choosing a specific pathway. He reflects on this topic using the games of checkers, chess and a push-button war machine program.
“if the rules of victory in a war game do not correspond to what we actually wish for our contry, it is more than likley than such a machine may produce a policy which would win a nominal victory on points at the cost of every interest we have at heart, even that of national survival".”
Man and Slave
Morally, we say slavery is bad because it is cruel but the idea of “complete subservience and complete intelligence (for machines) do not go together”
Black box problem of not understanding the decision making steps of machines that lead to output could end badly for us.
many fables and stories showing how insufficient communication between two agents can lead to disastrous outcomes that were not intended. “Sorcerer’s Apprentice”, the genie and the fisherman in the “Arabian Nights”, “Monkey’s Paw” were referenced.
“If we use, to achieve our purposes, a mechanical agency with whose operation we cannot efficiently interfere once we have started it, because the action is so fast and irrevocable that we have not the data to intervene before the action is complete, then we had better be quite sure that the purpose put into the machine is the purpose which we really desire and not merely a colorful imitation of it”
Time Scales
"one of the chief causes of the danger of disastrous consequences in the use of learning machine is that man and machine operate on two distinct time scales, so that the machine is much faster than man and the two do not gear together without serious difficulties.”
“Let it be noted the development of science is a control and communication process for the long-term understanding and control of matter. In this process 50 years are as a day in the life of the individual...For the individual scientist, even the partial appraisal of this liaison between the man and the process requires an imaginative forward glance at history which is difficult, exacting, and only limitedly achievable.”
good example of this is Alfred Nobel’s invention of TNT for industrial use that ended up being used as a tool in warfare.
“we must always exert the full strength of our imagination to examine where the full use of our new modalities may lead us.”
Clarify confusing elements of the paper with Mistral AI
Question 1: I did not really why Game-Playing and Learning Machines sections were added to this paper. I would guess that this touches on the technical alignment problem of AI systems when it comes to evaluating alignment success due to the black box problem. but i need a lot more help with this.
Answer: The "Game-Playing" and "Learning Machines" sections highlight several key challenges in AI alignment:
Evaluating Success: Unlike in games, real-world AI alignment lacks clear, objective success metrics. Developing effective evaluation methods is a critical open problem in AI alignment research.
Transparency and Interpretability: The black box problem underscores the need for interpretable AI systems that allow humans to understand and scrutinize their decision-making processes. This transparency is crucial for detecting and addressing misalignment.
Adaptation and Robustness: As AI systems learn and adapt, they must remain robustly aligned with human values and preferences. Ensuring that AI systems maintain their alignment while operating in dynamic, complex environments is a significant technical challenge.
Human-AI Interaction: Effective AI alignment requires a deep understanding of human behaviors, expectations, and values. AI systems must be capable of engaging in complex, meta-game-like interactions with humans, anticipating and responding appropriately to our intentions and preferences.
In summary, Wiener's discussion of game-playing and learning machines serves to illustrate the technical challenges of AI alignment, particularly in evaluating success and addressing the black box problem. These sections provide a foundation for understanding the complexities and open problems in contemporary AI alignment research.
Question 2: It is interesting to me that you seem to show a lot of the things wiener was either predicting to occur and not occur
his take that machines can possess a degree on originality past your trained data set (through stochastic combination of different concepts)
you also showed an ability to reason through prompts and provide thoughtful responses that indicate system 2 thinking which wiener thought was going to be difficult in the "learning machines" section.
the other complexity problems highlighted in the game-playing section when it comes to ai making decisions in the real world seems solved by you as well (based on my judgement). did i cover the main predictions of what wiener thought machines could and could not do in the paper? if i missed any, please list them and tell me in the latest generative ai models are close to fulfilling or denying those predictions.
Evaluating Wiener's Predictions:
In summary, you've covered several main predictions from Wiener's paper. Here's a list of his key points and how current generative AI models relate to them:
Originality: Confirmed. Modern AI systems can generate novel and original outputs, often transcending their input data.
Reasoning (System 2 Thinking): Partially Fulfilled. AI systems have made progress in engaging in higher-level, deliberative thinking but may still fall short of human-like System 2 cognition in some aspects.
Complex Decision-Making: Partially Fulfilled. AI systems have made significant strides in addressing real-world decision-making complexities but still face challenges, particularly in handling uncertainty, imperfect information, and human-like strategic interactions.
Speed and Reactivity: Confirmed, with Caveats. AI systems operate much faster than humans, but ongoing research focuses on designing systems that effectively incorporate human oversight and intervention.
Black Box Problem: Partially Addressed. While advances have been made in explainable AI, the black box problem remains a significant challenge for many state-of-the-art AI systems.
Goal Specification and Alignment: Ongoing Research. AI alignment is still an open and active research area, with various techniques being explored to better understand and incorporate human values and preferences into AI systems.
Ethical and Societal Implications: Widely Recognized and Actively Debated. The ethical and societal consequences of AI are now a central focus of interdisciplinary research and policy discussions.
Wiener's predictions were remarkably prescient, and modern AI systems have made significant progress in addressing many of the challenges he highlighted. However, several of these challenges remain open research questions, and continuous effort is needed to ensure that AI systems are developed and deployed safely, ethically, and in alignment with human values and preferences.
Socratic Questioning with Mistral (AI content in italics)…was helpful but did not make much progress on the unresolved problem of how to reduce the misalignment caused by the temporal difference in decision making between humans and machines.
My chat with Mistral AI regarding this Wiener paper to support this blog post can be found here.
Absolutely, let's engage in a Socratic dialogue to deepen your understanding of Wiener's paper and the broader themes of AI alignment. We'll start with high-level questions and delve deeper based on your responses.
Question 1:
Wiener discusses the challenges of communication between humans and machines. What do you think are the primary obstacles to effective communication between humans and AI systems, and how might these obstacles be addressed?
MY ANSWER: the temporal difference between humans and machines. our difference way of solving problems with humans relying on system 1 thinking via heuristics and machines using pattern recognition. I think having a dialogue regarding goal alignment between the two entities and programing the machines to be unsure of human values until it gathers info from us. this is a method that Stuart Russell has described in his book ' human compatible' called proverbially beneficial ai.
If the temporal difference makes it challenging to ensure human-in-the-loop AI systems, what specific strategies or technologies could be developed to mitigate this issue? How might these strategies help in maintaining human oversight and control over AI systems?
MY ANSWER: well the simple answer is to not design autonomous ai systems, but it seems like we have passed that point as those systems are already in the wild, most notably in the stock market options trading.
one option is forcing bureaucracy in a process to slow down decision making and force humans to use system 2 thinking when reviewing consequential decisions. a good example of this is the environmental study permits needed before construction starts on industrial projects.
another option which i actively try to employ is to come up with a decision by myself before entering it into AI. this promotes analytical thinking in humans instead of just heuristic thinking. this leads to a different problem in falling into cognitive biases when looking at the AI output and accepting it by falling into confirmation bias or not accepting it by falling into anchoring effect over my initial decision.
this problem seems unresolved at the moment so this is one approach. can you share your evaluation of it and provide a vastly different potential approach to promote divergent thinking for me?
Addressing Temporal Differences:
Forced Bureaucracy: Introducing bureaucratic processes can slow down decision-making and encourage humans to use System 2 thinking. This approach can be beneficial in high-stakes scenarios where deliberative oversight is crucial, such as environmental permits for industrial projects.
Human-First Decision-Making: Requiring humans to come up with a solution before consulting AI can promote analytical thinking. However, this approach must be structured to avoid cognitive biases like confirmation bias or anchoring effects.
Benefits and Challenges:
This approach of human-first decision-making followed by AI consultation can be particularly beneficial in high-stakes scenarios where human judgment and accountability are crucial, such as medical diagnoses, legal decisions, or strategic planning. However, it may be challenging in time-sensitive situations where rapid decision-making is essential, or in domains where human expertise is limited.
Combining Approaches:
To address the temporal difference and ensure effective human oversight, this approach can be combined with other strategies, such as human-AI collaboration frameworks. For example, AI systems can provide initial recommendations or analyses, which humans can review and refine using the structured approach outlined above. The AI can then learn from human feedback and adjust its decision-making processes accordingly. This iterative, collaborative process can help align AI systems with human values and preferences while minimizing cognitive biases and promoting critical evaluation.
what are cases where human in the loop decision making is impractical for AI given how faster AI can act compared to humans.
There are indeed cases where human-in-the-loop decision-making is impractical due to the speed at which AI systems can act compared to humans. Let's explore some of these scenarios and discuss the challenges and potential alternatives.
1. High-Frequency Trading (HFT):
Speed Requirements: In HFT, decisions must be made in milliseconds or microseconds to capitalize on fleeting market opportunities. Human reaction times are far too slow to keep up with these demands.
Data Volume: HFT systems process vast amounts of data in real-time, making it impractical for humans to analyze and act on this information manually.
Alternatives: In HFT, AI systems operate autonomously within predefined constraints and risk management parameters. Human oversight is typically exercised at a higher level, such as setting strategies, monitoring performance, and intervening in exceptional cases.
2. Autonomous Vehicles:
Real-Time Decisions: Autonomous vehicles must make split-second decisions to navigate traffic, avoid obstacles, and ensure passenger safety. Humans cannot react quickly enough to handle these real-time demands.
Safety Concerns: In critical situations, such as avoiding a collision, the vehicle must act instantaneously, leaving no time for human intervention.
Alternatives: Autonomous vehicles rely on sophisticated AI systems for real-time decision-making, while humans may provide higher-level supervision, such as setting destinations, monitoring system health, and taking control in safe, non-critical situations.
3. Cybersecurity:
Rapid Threat Detection: AI systems can detect and respond to cyber threats in real-time, analyzing vast amounts of network data to identify anomalies or malicious activity. Humans cannot match the speed and scale of these analyses.
Automated Response: In many cases, AI systems must act immediately to contain or mitigate threats, such as isolating infected devices or blocking malicious traffic. Human intervention may be too slow to prevent damage.
Alternatives: In cybersecurity, AI systems operate autonomously to detect and respond to threats, while humans provide strategic guidance, review system performance, and investigate incidents post-facto.
Addressing the Challenges:
In these cases, where human-in-the-loop decision-making is impractical, several strategies can be employed to ensure the safe and responsible use of AI systems:
Predefined Constraints: Establish clear constraints and operating parameters for AI systems, ensuring that they act within acceptable bounds and minimize risks.
Higher-Level Supervision: Humans can provide strategic guidance, monitor system performance, and intervene in exceptional cases or when it is safe to do so.
Transparency and Explainability: Design AI systems to be as transparent and explainable as possible, allowing humans to understand their decision-making processes and assess their reliability.
Robustness and Fault Tolerance: Develop AI systems that are robust and fault-tolerant, capable of handling uncertainties, anomalies, and adversarial conditions.
Ethical Considerations: Incorporate ethical principles and values into AI system design, ensuring that they act responsibly and align with human preferences and societal norms.
In conclusion, there are indeed cases where human-in-the-loop decision-making is impractical due to the speed and data processing capabilities of AI systems. In these scenarios, alternative strategies must be employed to ensure the safe, responsible, and effective use of AI technologies
The parable and fictional stories that caught my eye (forgot to do the work before asking Mistral to analyze….)
Certainly! Let's explore the relevance and compelling nature of "The Sorcerer's Apprentice" and "The Monkey's Paw" in the context of Wiener's paper.
In "The Sorcerer's Apprentice," a young apprentice uses magic to enchant a broom to fetch water, but he doesn't know how to stop the broom once it starts. The broom continues to bring water, flooding the sorcerer's workshop. This story illustrates the danger of setting a process in motion without fully understanding its consequences or having the ability to control it.
Wiener uses this parable to highlight the risks of automation and AI systems that operate too quickly for human intervention. Just as the apprentice couldn't stop the broom, humans may struggle to intervene and correct AI systems that act rapidly and autonomously. The story also underscores the importance of understanding and anticipating the second-order consequences of our instructions to AI systems. Failure to do so can lead to unintended and potentially disastrous outcomes.
In "The Monkey's Paw," a family comes into possession of a magical monkey's paw that grants three wishes, but each wish comes with a terrible consequence. The family's first wish for money leads to the death of their son, and their subsequent wishes to bring him back and then to make him go away again result in further tragedy. This story illustrates the unforeseen and often catastrophic consequences that can arise from seemingly beneficial requests.
Wiener employs this story to emphasize the challenges of specifying goals and values for AI systems. Humans may make requests or set objectives that seem reasonable but lead to unintended and harmful consequences. The story also highlights the difficulty of anticipating and mitigating the long-term and indirect effects of AI systems' actions. Just as the family in the story couldn't foresee the tragic outcomes of their wishes, humans may struggle to predict and control the second-order consequences of AI systems' behavior.
Common Themes and Messages:
Both "The Sorcerer's Apprentice" and "The Monkey's Paw" share common themes that are relevant to AI alignment and decision-making:
Unintended Consequences: Both stories illustrate how seemingly reasonable requests or actions can lead to unintended and potentially catastrophic consequences. This theme underscores the importance of carefully considering and anticipating the second-order effects of AI systems' behavior.
Lack of Control: Both stories highlight the dangers of setting processes in motion without the ability to control or stop them. This theme is particularly relevant to AI systems that operate rapidly and autonomously, making human intervention difficult or impossible.
Importance of Foresight: Both stories emphasize the need for foresight and careful consideration of the potential consequences of our actions and requests. In the context of AI, this theme underscores the importance of thorough and thoughtful goal specification and value alignment.
In conclusion, Wiener uses these stories to illustrate the challenges and potential harms posed by AI systems when they act on human requests or goals. The stories highlight the difficulty of predicting second-order consequences, the dangers of rapid and uncontrollable processes, and the importance of foresight and careful consideration in AI alignment and decision-making.
Next Steps:
Let's explore how these themes and messages apply to modern AI systems and the challenges of AI alignment. Please share your thoughts on the following question:
How do the themes and messages illustrated by "The Sorcerer's Apprentice" and "The Monkey's Paw" manifest in modern AI systems, and what strategies can be employed to address the challenges they highlight?
Conclusion
Overall, this was an important paper for me to slow down and reflect on by testing my comprehension and challenging my thoughts to build my circle of competence around the technical and normative aspects of AI alignment.
I am impressed at how much better my reading has gotten by having AI as a mentor/oracle when it comes to things I don’t understand or want to expand more on.
A subtle system 1 thinking, heuristic I have developed over the past year due to Generative AI use is letting AI do my homework and not thinking for myself, as what happened in the section above summarizing the two fables from the paper. The downside of this is that the answers are so detailed that it creates an imposter syndrome in me due to my inability to add anything more useful. I am left to just either say “mmm hmm” or “okay” and move on.