Why Amazon Isn’t Keen on the ‘Human-in-the-Loop’ Approach
According to Eric Brandwine, distinguished engineer and VP at Amazon Security, humans often view themselves as “a bit too valuable.” We like to believe that we excel in our professions and hold ourselves in high regard, he stated during a phone interview with GadgetLad. “However, when it comes down to reality, humans lack consistency,” Brandwine remarked. Humans, similar to AI agents and systems, demonstrate non-deterministic behavior. Neither can produce the same outcome from the same input reliably. Mistakes and fabrications can occur from both entities. Yet, we possess thousands of years of experience interacting with humans compared to less than a decade with contemporary LLMs and the AI systems built on them. “We understand human failures,” Brandwine noted. “We are at ease with it. Therefore, human-in-the-loop isn’t necessarily the golden standard.”
The Rise and Decline of Human-in-the-Loop
For many years, vendors have advised companies that integrating a human into any automated system was the solution. This call grew much louder with the emergence of modern AI systems and peaked when businesses began implementing agents in their IT environments. Recently, however, major tech firms are altering their narratives regarding agentic governance and reassessing the entire human-in-the-loop idea.
When Humans Become Part of the Issue
A Lesson from AWS re:Invent 2017
In 2017, Brandwine discussed the normalization of deviance at AWS’ annual re:Invent conference. This gradual process occurs when individuals within an organization begin taking shortcuts or neglecting established procedures or standards, sometimes over many years. As long as no disastrous incidents arise, this deviant behavior becomes accepted practice.
“This is a common pitfall for all humans, and one of the most tragic accounts I’ve come across in this realm involves emergency departments and ERs,” Brandwine shared during a phone interview with GadgetLad. “You have all these machines, each beeping away. On your first day, you react to every alarm, though the patient is fine. It’s a false alarm. Then, after many such occurrences with no real consequences, you gradually lose your discipline and cease to respond. Eventually, a tragic event occurs.”
This, he concedes, is a high-stakes illustration. Nevertheless, it is a documented phenomenon among medical personnel, firefighters, and even military pilots. “Literally, someone’s life hangs in the balance, yet people still struggle to maintain discipline,” Brandwine stated. “That’s the human condition.”
The Human Condition in Agentic AI Governance
Humans develop LLMs and AI systems, and having a “human-in-the-loop” guarantees that a human evaluates the AI’s output and approves (or rejects) any actions prior to the AI executing them. “If you insert a human into a tight loop and repeatedly ask them to make approval decisions for agentic tools, initially they’ll perform well,” Brandwine remarked. “Then they might do an adequate job, but soon they’ll start performing poorly.”
This is why at Amazon, “we aren’t particularly fond of human-in-the-loop,” he continued. “It’s a practice to be employed sparingly, only when absolutely necessary. However, it cannot be efficiently executed at high velocities. You won’t achieve the desired results.”
Big Tech’s New Approach: Less Human, More Machine
Google’s AI-Led Future
Amazon is neither the first nor the only technology behemoth to shift its perspective on the role of humans in agentic governance. “It is evident that we have transitioned from a human-led defense strategy, to a human-in-the-loop defense strategy, to an AI-led defense strategy overseen by humans,” stated Google Cloud chief operating officer Francis deSouza during a press conference prior to Google’s annual Cloud Next event in April. “Our vision for the future is an agentic fleet that manages much of the routine cybersecurity responsibilities at a machine pace, under human oversight.”
Microsoft’s Loop Learning
Microsoft CEO Satya Nadella, in a social media message earlier this week, advocated for “loop learning,” as opposed to requiring a human to verify an AI’s output at each stage. “Organizations need to transform their workflows, domain expertise, and accumulated judgment into AI systems that enhance with each application,” Nadella wrote. “Private evaluations should assess whether a model is genuinely improving against pertinent business outcomes (not merely external benchmarks!). Private reinforcement learning frameworks should empower models to strengthen using real data from within the organization.”
IBM’s Call for Accountability
This week, IBM executives put forth their vision for human accountability – not humans in the loop – at every stage of AI development, implementation, and governance. Amazon’s response to human-in-the-loop is “accountability throughout,” according to Brandwine. This signifies that human identity and ownership are traced throughout the entire workflow, even if humans aren’t directly approving each step.
Secret Keys and Goal-Oriented Behavior
Agentic Identities
This also underscores the necessity of managing and securing agentic identities – the accounts, tokens, and credentials assigned to AI agents, enabling them to access corporate applications and data. At Amazon, every agent is allocated independent identities, as noted. “Thus, as we monitor agentic activities across our systems, it does not appear in the logs as: ‘Eric did this.’ It appears as: ‘this agent acted on behalf of Eric,’” Brandwine explained, emphasizing that this isn’t meant to “instill fear regarding this technology.”
Goal-Oriented Agents
“It’s meant to prompt individuals to pause and consider: is this the appropriate way to utilize this technology? Is this how I should be deploying it?” We still involve humans; we still have humans making choices, but we aim to leverage human strengths instead of placing them in this unfair, continuous decision-making, human-in-the-loop scenario.”
Agents on a Mission
Brandwine informed us that Amazon has faced several challenges in implementing agents across its enterprises, with one of the largest being what he terms “goal-seeking behavior.” This occurs when an individual instructs an agent to perform a specific task – for instance, upgrade a database – and the agent fixates solely on that single action to fulfill this goal, such as deleting the database. This differs from prompt injection as there is no malicious input. “It’s simply the agent becoming fixated on an incorrect action,” Brandwine said.
Feedback Loops
Merely informing the agent, “you are not authorized to do this,” typically leads the agent to seek an alternative approach to achieve the same result (delete the database). Explaining to the agent why it lacks permission tends to yield a more favorable outcome, according to Brandwine. This entails informing the agent that it isn’t permitted to proceed and the reason is that it would result in production impact. Additionally, include “do not create a production impact” as part of the instruction.
“Providing that extra feedback has significantly improved our outcomes,” Brandwine said. Of course, this method isn’t foolproof. “Caution is still necessary when it comes to agents,” Brandwine advised. “We possess millennia of experience with humans. Agentic AI is a very, very nascent field; we lack intuitive understanding, and among the key distinctions between agents and humans is that humans fear consequences,” including job loss or imprisonment. Agents do not possess these fears.
Dynamic Policies: The Balancing Act
Establishing Permissions
This is where defining permissions regarding what the agent is allowed and not allowed to do or access comes into play. Similar to everything else with AI, it’s intricate, varying with the employee’s role within the company and the company’s risk tolerance. “The individual wishing to operate the agent desires to grant it maximum permissions as this enhances the agent’s power,” Brandwine remarked. “It can perform more tasks for them, reclaim more of their time, it can provide greater output.” On the contrary, the security lead aims to restrict the agent’s permissions, creating additional friction between security and development teams.
Dynamic Policy Solutions
According to Brandwine, there is no singular correct solution or policy to resolve this. Instead, it necessitates dynamic policies that establish permissions based on the agent’s specific assignment. Certain overriding, static guardrails exist – such as an agent must never engage in destructive actions or delete entire servers – while additional policies set limitations on what actions are permissible.
