When AI Agents Start Acting, Who Is Really in Control?

For the past few years, much of the conversation around enterprise AI has focused on what AI can tell us. The next phase will be defined by what AI can do: AI agents are moving beyond answering questions, summarizing information or recommending next steps. Increasingly, they will be able to interact with systems, coordinate with other agents and initiate actions themselves.

That changes the nature of the risk. When an AI system generates a poor answer, a human can ignore it. When an AI agent changes a security policy, reroutes network traffic or modifies a production environment, the consequences can be immediate.

The question is therefore no longer simply how intelligent AI systems will become. It is: How much freedom should we give them to act, and how do we remain in control when they do?

Autonomy Is Not Binary

The debate around autonomous AI often implies a binary choice: either humans are in control, or machines are. Technically, there is a large spectrum between the two.

An AI system might independently analyze a problem but require human approval before taking action. It might be permitted to perform a limited set of reversible actions autonomously. Or it might operate independently within a clearly defined environment while escalating anything outside those boundaries. This is what I think of as bounded autonomy.

The objective should not be to make an AI system as autonomous as technically possible. It should be to give it precisely the amount of autonomy required to perform a task effectively and safely. That distinction will become increasingly important as agents gain access to operational systems.

Guardrails first. Autonomy second.

A Human in the Loop Does Not Necessarily Mean Human Control

One of the most common answers to the risks of autonomous AI is simple: keep a human in the loop. That sounds reassuring. But it can also create an illusion of control.

Imagine an AI system making hundreds of recommendations every day. A human operator is required to approve every one of them. Initially, that person may scrutinize every recommendation. But if the system proves reliable again and again, behavior inevitably changes. Approval becomes routine.

Click. Approve. Click. Approve.

The human is technically still in the loop, but is that meaningful oversight? Writer Cory Doctorow has used the term “reverse centaur” to describe a similar relationship between humans and machines. Instead of technology augmenting human judgment, the relationship is reversed: the machine determines the decisions and pace, while the human increasingly executes or approves what it proposes.

That is why human oversight cannot simply mean putting an approval button at the end of an automated process. Humans need to control the decision space: What is the agent allowed to do? Which systems can it access? Which actions must always be escalated? What thresholds apply? Which decisions are reversible? And where is human judgment mandatory?

This moves human oversight from approving every individual action towards defining and governing the boundaries within which autonomous systems operate.

We Also Need to Rethink Observability

There is another challenge that becomes more important as AI systems gain autonomy: understanding what actually happened. Traditional observability tells us a great deal about systems. We can record inputs, outputs, events, dependencies and actions.

With AI agents, that is necessary, but it may no longer be sufficient. An agent may have received all the correct information and still reach the wrong conclusion; it may misinterpret context; it may hallucinate. So knowing which information the system accessed does not necessarily explain why it arrived at a particular decision.

And this becomes even more complicated in multi-agent environments. Imagine Agent A delegates part of a task to Agent B. Agent B consults Agent C. Its result is returned to Agent A and becomes one of the inputs for an operational decision. If that decision turns out to be wrong, observing Agent A alone tells us very little.

We need to understand the entire chain: which agents interacted, what information they exchanged, which conclusions they reached, which recommendations were accepted and ultimately which actions were executed. In other words, observability needs to extend from individual agents to the agent network itself.

And even that is only part of the answer. If an agent can have all the necessary information and still make a poor decision, observability needs to be combined with guardrails that can identify implausible or impermissible outcomes before they have consequences.

Auditability, observability and guardrails therefore need to evolve together.

Agent Security Will Become a Discipline of Its Own

Once AI agents can act, they also create a new security surface.

The questions are familiar, but the context is new: Which agent is allowed to access which data? How does it authenticate itself? Which systems can it interact with? What actions is it authorized to perform? Can another agent manipulate it? How do we detect abnormal behavior? And can we reconstruct afterwards what happened?

Many of the principles required to answer these questions already exist in cybersecurity. Zero Trust taught us not to trust a user or device simply because it is inside a network. Identity and access management taught us to grant only the permissions required for a particular task. Modern security architectures assume that access must be continuously verified.

We now need to apply similar principles to AI agents. As agents become participants in enterprise environments alongside humans, applications and devices, Agent Security and Agent Governance will become necessary disciplines for organizations deploying autonomous AI.

From Assistance to Action, One Controlled Step at a Time

This is also how we think about AI at Open Systems: Our AI Matrix is based on specialized agents for different networking and security tasks. Rather than asking one generic AI system to solve every problem, individual agents can contribute specialized expertise, while Lucy, our AI operator, provides the interface to the user and orchestrates those capabilities behind the scenes.

The intelligence behind these agents is grounded in something we have accumulated over 35 years of operating business-critical network and security infrastructure: operational knowledge from incidents, support cases, playbooks, troubleshooting processes and millions of operational decisions.

Today, Lucy can already analyze network and security problems, explain configurations and policies, and support troubleshooting. More than 60 percent of our customers used Lucy within its first months in production.

The next step is not to simply remove the human and let the agents take over.

Our north star is to relieve people of more operational routine work. But we want to get there incrementally, expanding an agent’s ability to act only where we can understand the impact, control the boundaries and reverse an action if necessary.

Technologies such as digital twins can play an important role here. Before making a critical change to a production environment, an agent can test the proposed action in a simulated environment. Approval thresholds, rollback mechanisms and comprehensive audit logs provide additional layers of control.

This is central to what we mean by Autonomous SASE: not autonomy for autonomy’s sake, but the gradual evolution from AI-assisted operations towards controlled autonomy.

The Goal Is Not Maximum Autonomy

There is understandable concern that we may give increasingly powerful AI systems too much freedom too quickly. There is an opposite risk as well: that fear leads us to build extremely capable AI systems that can analyze everything but are permitted to do almost nothing. Neither extreme makes much sense.

The right level of autonomy depends on the consequences of the decision. Where actions are clearly bounded, observable and reversible, agents can be given greater freedom. As the potential impact increases, so should the safeguards and the role of human judgment.

The challenge ahead is therefore not choosing between autonomy and control; it is designing systems in which autonomy can increase without control decreasing. That, much more than simply building increasingly capable AI models, may be one of the defining engineering challenges of the agentic era.