Agent UX
The AI Read It Correctly and Still Got It Wrong
A camera can be right about what it sees and wrong about what should happen next. That gap is a design problem.
Year :
2026
Industry :
Agentic AI, Industrial Operations
Client :
Self-initiated concept
Project Duration :
One week
The Premise
The premise was straightforward. AI agents are starting to take actions, not just answer questions, so put a human in front of the agent and let them approve or reject what it proposes. Oversight solved.
I built that and it did not survive contact with a realistic scenario.
If approving takes one keystroke and disagreeing takes six, people approve. All shift. You have not built oversight; you have built a rubber stamp that converts an automated decision into a human-approved one without any judgement being added.
That reframed the whole project. I stopped designing an approval screen and started designing the cost of disagreement.
Context
Agent Action is a self-initiated concept: a review console for a distribution depot where an AI agent watches camera feeds and proposes operational actions. Dispatch a tow. Radio a supervisor. Hold a barrier. Stop a conveyor.
I designed and built it end to end in a week: concept, agent workflow, interaction model, interface, copy, and a deployed front-end prototype. No team, no client, no users.
I worked directly in code rather than building a Figma prototype first, because the parts of this problem that mattered are timing, keyboard behaviour, and what happens after a decision. None of those exist in a static screen.
Live prototype: agent-action.vercel.app
Discovery
A camera sees a trailer stopped across a fire lane. Six minutes. The site rule says anything blocking the lane past five minutes gets towed. The agent recommends dispatching a tow.
Forty minutes earlier, a supervisor released that same trailer as a scheduled delivery. It is permitted to stay until 14:30. That permission lives in a log the agent cannot read.
The camera did not fail. The model was not uncertain. The agent understood the scene correctly and confidently recommended the wrong action.
The agent reads the lane. It cannot read the permit log.
This is the failure mode I wanted to design for. It is not a perception problem; it is an institutional knowledge problem. The context that makes the recommendation wrong exists, it is just somewhere the system cannot see. Most operational AI will fail this way, and almost nobody designs for it.
What We Found
Confidence is not one number.
The agent can be completely clear that a trailer is stationary across a lane while the recommended tow is entirely wrong. Detection confidence and action confidence are different questions, and collapsing them into a single score hides the thing the operator needs.
So the console separates them. A second check reads the permit and decision records the agent cannot access. When it finds a contradiction, it demotes the proposed action without pretending the visual detection was wrong. The agent stays confident about what it saw. The interface becomes cautious about what to do.
Some actions cannot be taken back, and some can be corrected instead.
Undo is not a single concept. Logging a note can be withdrawn with no trace. A radio call has already reached a person, so the only remedy is a second call, which has its own cost. Dispatching a tow removes a vehicle and bills a haulier, and nothing recovers that.
Treating those three as one behaviour is where most agent interfaces quietly mislead.
A percentage is close to meaningless under time pressure.
An operator has about eight seconds per event. Nobody in that position can interpret what 87% is calibrated against.
The Assumptions the Research Killed
I wrote these down before building and held the interface to them.
Disagreeing must cost less than agreeing.
Alternatives sit one keystroke away. Approving a final action that contradicts an earlier human decision requires opening that record first. Friction goes where the consequence is, not everywhere.
State the consequence before the commitment, not after.
Every option says what will happen and whether it can be taken back. The operator learns that towing bills the haulier before they approve it, not in a confirmation dialog.
Three action classes, three different interfaces.
Final offers nothing and says so, with no disabled undo button. Reversible holds an eight-second window inside the console. Correctable has already happened and promotes a correcting action with its own stated cost, never labelled as Undo.
Confidence changes the layout, not just a label.
A low-confidence proposal is wider, slower to approve, and shows its evidence without a click. A clear one is compact and moves fast.
Taking no action is still a decision.
Rejecting asks which of four reasons applies, and each reason states what happens next to the event, the camera, or the queue.
Reasoning is written in the operator’s words.
“Trailer stationary in fire lane, 6 min” rather than object classes and confidence thresholds.
What We Designed Instead
A review console built around the moment of disagreement rather than the moment of approval.
The operator works a prioritised queue. Selecting an event opens the camera evidence, the agent’s assessment, the site rule it applied, and any record that contradicts it. When a contradiction exists, it sits above the evidence and the risky approval is held back behind the record that justifies it.
Every option carries its consequence and its reversibility class. After a decision, the console shows what actually happened in the world, then writes it to the trail as a sentence a future reader can follow.
The activity trail opens filtered to overrides, because overrides are where the agent and the operation disagreed. Corrections stay nested under the action they correct. Reversed decisions are marked as reversed. Repeated overrides on the same vehicle surface in the proposal itself, which is the operator’s real complaint made visible.
Designing past the happy path
Nine states, each loadable directly by URL so they could be tested repeatedly rather than demonstrated once.
The one worth naming: when the agent’s reasoning fails to load, the operator can still override or take no action, because those choices do not depend on the explanation. Only approval is held back, and the approve slot retries the load instead. The asymmetry survives even when the system breaks.
The empty queue says what is still true, six cameras watching and where the last decision sits, rather than reporting an absence.
Deliberate omissions
No settings, no dashboard, no onboarding, no mobile layout. An operator console is a single-purpose tool used for eight hours by someone trained on it. Every screen that is not the work is a screen that is not the work.
Where It Stands, and What I’d Do Differently
Where It Stands, Honestly
Agent Action is a fictional concept. There is no depot, no camera, no operator, no database. I did not conduct research, test with users, or measure anything. Nothing here is validated.
What it is: a design hypothesis expressed as working software. Three camera events, a review queue, consequence-aware decisions, undo and correction behaviour, failure states, and an audit trail, deployed and usable.
I built it in a week using AI throughout, for exploring scenarios, challenging the interaction rules, and implementing directly in code. The speed came from AI. The framing, the authority model, the consequence language, and the decision to rebuild when the first version did not feel credible did not.
What I’d Do Differently
The first build looked like a developer console. It communicated the logic and failed as a product: one card floating in space, no queue, no shell, no sense of a shift in progress. I rebuilt it into a workspace with the queue visible and the operator’s context persistent. I should have started there. A tool that does not look like it belongs on a control-room desk does not get evaluated on its ideas.
I would test the reversal window rather than assume it. Eight seconds is a guess. The window also closes on the operator’s next decision, which means moving quickly forfeits the undo. That is a defensible rule and it might be the wrong one.
Overrides should feed back into the agent. Every override is a signal that the agent is missing an integration, a rule, or a record. The console records them and stops there. Repeated overrides on the same plate should change what the agent proposes, and that belongs in the product rather than in a future phase.
I would test with real operators next: whether people notice the contradiction quickly, whether the difference between final and correctable lands, whether the cheaper override path actually prevents default approval, and what belongs in the downstream detection review and camera maintenance queues that this console currently only names.
Explore the working prototype: agent-action.vercel.app
More Projects
Agent UX
The AI Read It Correctly and Still Got It Wrong
A camera can be right about what it sees and wrong about what should happen next. That gap is a design problem.
Year :
2026
Industry :
Agentic AI, Industrial Operations
Client :
Self-initiated concept
Project Duration :
One week
The Premise
The premise was straightforward. AI agents are starting to take actions, not just answer questions, so put a human in front of the agent and let them approve or reject what it proposes. Oversight solved.
I built that and it did not survive contact with a realistic scenario.
If approving takes one keystroke and disagreeing takes six, people approve. All shift. You have not built oversight; you have built a rubber stamp that converts an automated decision into a human-approved one without any judgement being added.
That reframed the whole project. I stopped designing an approval screen and started designing the cost of disagreement.
Context
Agent Action is a self-initiated concept: a review console for a distribution depot where an AI agent watches camera feeds and proposes operational actions. Dispatch a tow. Radio a supervisor. Hold a barrier. Stop a conveyor.
I designed and built it end to end in a week: concept, agent workflow, interaction model, interface, copy, and a deployed front-end prototype. No team, no client, no users.
I worked directly in code rather than building a Figma prototype first, because the parts of this problem that mattered are timing, keyboard behaviour, and what happens after a decision. None of those exist in a static screen.
Live prototype: agent-action.vercel.app
Discovery
A camera sees a trailer stopped across a fire lane. Six minutes. The site rule says anything blocking the lane past five minutes gets towed. The agent recommends dispatching a tow.
Forty minutes earlier, a supervisor released that same trailer as a scheduled delivery. It is permitted to stay until 14:30. That permission lives in a log the agent cannot read.
The camera did not fail. The model was not uncertain. The agent understood the scene correctly and confidently recommended the wrong action.
The agent reads the lane. It cannot read the permit log.
This is the failure mode I wanted to design for. It is not a perception problem; it is an institutional knowledge problem. The context that makes the recommendation wrong exists, it is just somewhere the system cannot see. Most operational AI will fail this way, and almost nobody designs for it.
What We Found
Confidence is not one number.
The agent can be completely clear that a trailer is stationary across a lane while the recommended tow is entirely wrong. Detection confidence and action confidence are different questions, and collapsing them into a single score hides the thing the operator needs.
So the console separates them. A second check reads the permit and decision records the agent cannot access. When it finds a contradiction, it demotes the proposed action without pretending the visual detection was wrong. The agent stays confident about what it saw. The interface becomes cautious about what to do.
Some actions cannot be taken back, and some can be corrected instead.
Undo is not a single concept. Logging a note can be withdrawn with no trace. A radio call has already reached a person, so the only remedy is a second call, which has its own cost. Dispatching a tow removes a vehicle and bills a haulier, and nothing recovers that.
Treating those three as one behaviour is where most agent interfaces quietly mislead.
A percentage is close to meaningless under time pressure.
An operator has about eight seconds per event. Nobody in that position can interpret what 87% is calibrated against.
The Assumptions the Research Killed
I wrote these down before building and held the interface to them.
Disagreeing must cost less than agreeing.
Alternatives sit one keystroke away. Approving a final action that contradicts an earlier human decision requires opening that record first. Friction goes where the consequence is, not everywhere.
State the consequence before the commitment, not after.
Every option says what will happen and whether it can be taken back. The operator learns that towing bills the haulier before they approve it, not in a confirmation dialog.
Three action classes, three different interfaces.
Final offers nothing and says so, with no disabled undo button. Reversible holds an eight-second window inside the console. Correctable has already happened and promotes a correcting action with its own stated cost, never labelled as Undo.
Confidence changes the layout, not just a label.
A low-confidence proposal is wider, slower to approve, and shows its evidence without a click. A clear one is compact and moves fast.
Taking no action is still a decision.
Rejecting asks which of four reasons applies, and each reason states what happens next to the event, the camera, or the queue.
Reasoning is written in the operator’s words.
“Trailer stationary in fire lane, 6 min” rather than object classes and confidence thresholds.
What We Designed Instead
A review console built around the moment of disagreement rather than the moment of approval.
The operator works a prioritised queue. Selecting an event opens the camera evidence, the agent’s assessment, the site rule it applied, and any record that contradicts it. When a contradiction exists, it sits above the evidence and the risky approval is held back behind the record that justifies it.
Every option carries its consequence and its reversibility class. After a decision, the console shows what actually happened in the world, then writes it to the trail as a sentence a future reader can follow.
The activity trail opens filtered to overrides, because overrides are where the agent and the operation disagreed. Corrections stay nested under the action they correct. Reversed decisions are marked as reversed. Repeated overrides on the same vehicle surface in the proposal itself, which is the operator’s real complaint made visible.
Designing past the happy path
Nine states, each loadable directly by URL so they could be tested repeatedly rather than demonstrated once.
The one worth naming: when the agent’s reasoning fails to load, the operator can still override or take no action, because those choices do not depend on the explanation. Only approval is held back, and the approve slot retries the load instead. The asymmetry survives even when the system breaks.
The empty queue says what is still true, six cameras watching and where the last decision sits, rather than reporting an absence.
Deliberate omissions
No settings, no dashboard, no onboarding, no mobile layout. An operator console is a single-purpose tool used for eight hours by someone trained on it. Every screen that is not the work is a screen that is not the work.
Where It Stands, and What I’d Do Differently
Where It Stands, Honestly
Agent Action is a fictional concept. There is no depot, no camera, no operator, no database. I did not conduct research, test with users, or measure anything. Nothing here is validated.
What it is: a design hypothesis expressed as working software. Three camera events, a review queue, consequence-aware decisions, undo and correction behaviour, failure states, and an audit trail, deployed and usable.
I built it in a week using AI throughout, for exploring scenarios, challenging the interaction rules, and implementing directly in code. The speed came from AI. The framing, the authority model, the consequence language, and the decision to rebuild when the first version did not feel credible did not.
What I’d Do Differently
The first build looked like a developer console. It communicated the logic and failed as a product: one card floating in space, no queue, no shell, no sense of a shift in progress. I rebuilt it into a workspace with the queue visible and the operator’s context persistent. I should have started there. A tool that does not look like it belongs on a control-room desk does not get evaluated on its ideas.
I would test the reversal window rather than assume it. Eight seconds is a guess. The window also closes on the operator’s next decision, which means moving quickly forfeits the undo. That is a defensible rule and it might be the wrong one.
Overrides should feed back into the agent. Every override is a signal that the agent is missing an integration, a rule, or a record. The console records them and stops there. Repeated overrides on the same plate should change what the agent proposes, and that belongs in the product rather than in a future phase.
I would test with real operators next: whether people notice the contradiction quickly, whether the difference between final and correctable lands, whether the cheaper override path actually prevents default approval, and what belongs in the downstream detection review and camera maintenance queues that this console currently only names.
Explore the working prototype: agent-action.vercel.app
More Projects
Agent UX
The AI Read It Correctly and Still Got It Wrong
A camera can be right about what it sees and wrong about what should happen next. That gap is a design problem.
Year :
2026
Industry :
Agentic AI, Industrial Operations
Client :
Self-initiated concept
Project Duration :
One week
The Premise
The premise was straightforward. AI agents are starting to take actions, not just answer questions, so put a human in front of the agent and let them approve or reject what it proposes. Oversight solved.
I built that and it did not survive contact with a realistic scenario.
If approving takes one keystroke and disagreeing takes six, people approve. All shift. You have not built oversight; you have built a rubber stamp that converts an automated decision into a human-approved one without any judgement being added.
That reframed the whole project. I stopped designing an approval screen and started designing the cost of disagreement.
Context
Agent Action is a self-initiated concept: a review console for a distribution depot where an AI agent watches camera feeds and proposes operational actions. Dispatch a tow. Radio a supervisor. Hold a barrier. Stop a conveyor.
I designed and built it end to end in a week: concept, agent workflow, interaction model, interface, copy, and a deployed front-end prototype. No team, no client, no users.
I worked directly in code rather than building a Figma prototype first, because the parts of this problem that mattered are timing, keyboard behaviour, and what happens after a decision. None of those exist in a static screen.
Live prototype: agent-action.vercel.app
Discovery
A camera sees a trailer stopped across a fire lane. Six minutes. The site rule says anything blocking the lane past five minutes gets towed. The agent recommends dispatching a tow.
Forty minutes earlier, a supervisor released that same trailer as a scheduled delivery. It is permitted to stay until 14:30. That permission lives in a log the agent cannot read.
The camera did not fail. The model was not uncertain. The agent understood the scene correctly and confidently recommended the wrong action.
The agent reads the lane. It cannot read the permit log.
This is the failure mode I wanted to design for. It is not a perception problem; it is an institutional knowledge problem. The context that makes the recommendation wrong exists, it is just somewhere the system cannot see. Most operational AI will fail this way, and almost nobody designs for it.
What We Found
Confidence is not one number.
The agent can be completely clear that a trailer is stationary across a lane while the recommended tow is entirely wrong. Detection confidence and action confidence are different questions, and collapsing them into a single score hides the thing the operator needs.
So the console separates them. A second check reads the permit and decision records the agent cannot access. When it finds a contradiction, it demotes the proposed action without pretending the visual detection was wrong. The agent stays confident about what it saw. The interface becomes cautious about what to do.
Some actions cannot be taken back, and some can be corrected instead.
Undo is not a single concept. Logging a note can be withdrawn with no trace. A radio call has already reached a person, so the only remedy is a second call, which has its own cost. Dispatching a tow removes a vehicle and bills a haulier, and nothing recovers that.
Treating those three as one behaviour is where most agent interfaces quietly mislead.
A percentage is close to meaningless under time pressure.
An operator has about eight seconds per event. Nobody in that position can interpret what 87% is calibrated against.
The Assumptions the Research Killed
I wrote these down before building and held the interface to them.
Disagreeing must cost less than agreeing.
Alternatives sit one keystroke away. Approving a final action that contradicts an earlier human decision requires opening that record first. Friction goes where the consequence is, not everywhere.
State the consequence before the commitment, not after.
Every option says what will happen and whether it can be taken back. The operator learns that towing bills the haulier before they approve it, not in a confirmation dialog.
Three action classes, three different interfaces.
Final offers nothing and says so, with no disabled undo button. Reversible holds an eight-second window inside the console. Correctable has already happened and promotes a correcting action with its own stated cost, never labelled as Undo.
Confidence changes the layout, not just a label.
A low-confidence proposal is wider, slower to approve, and shows its evidence without a click. A clear one is compact and moves fast.
Taking no action is still a decision.
Rejecting asks which of four reasons applies, and each reason states what happens next to the event, the camera, or the queue.
Reasoning is written in the operator’s words.
“Trailer stationary in fire lane, 6 min” rather than object classes and confidence thresholds.
What We Designed Instead
A review console built around the moment of disagreement rather than the moment of approval.
The operator works a prioritised queue. Selecting an event opens the camera evidence, the agent’s assessment, the site rule it applied, and any record that contradicts it. When a contradiction exists, it sits above the evidence and the risky approval is held back behind the record that justifies it.
Every option carries its consequence and its reversibility class. After a decision, the console shows what actually happened in the world, then writes it to the trail as a sentence a future reader can follow.
The activity trail opens filtered to overrides, because overrides are where the agent and the operation disagreed. Corrections stay nested under the action they correct. Reversed decisions are marked as reversed. Repeated overrides on the same vehicle surface in the proposal itself, which is the operator’s real complaint made visible.
Designing past the happy path
Nine states, each loadable directly by URL so they could be tested repeatedly rather than demonstrated once.
The one worth naming: when the agent’s reasoning fails to load, the operator can still override or take no action, because those choices do not depend on the explanation. Only approval is held back, and the approve slot retries the load instead. The asymmetry survives even when the system breaks.
The empty queue says what is still true, six cameras watching and where the last decision sits, rather than reporting an absence.
Deliberate omissions
No settings, no dashboard, no onboarding, no mobile layout. An operator console is a single-purpose tool used for eight hours by someone trained on it. Every screen that is not the work is a screen that is not the work.
Where It Stands, and What I’d Do Differently
Where It Stands, Honestly
Agent Action is a fictional concept. There is no depot, no camera, no operator, no database. I did not conduct research, test with users, or measure anything. Nothing here is validated.
What it is: a design hypothesis expressed as working software. Three camera events, a review queue, consequence-aware decisions, undo and correction behaviour, failure states, and an audit trail, deployed and usable.
I built it in a week using AI throughout, for exploring scenarios, challenging the interaction rules, and implementing directly in code. The speed came from AI. The framing, the authority model, the consequence language, and the decision to rebuild when the first version did not feel credible did not.
What I’d Do Differently
The first build looked like a developer console. It communicated the logic and failed as a product: one card floating in space, no queue, no shell, no sense of a shift in progress. I rebuilt it into a workspace with the queue visible and the operator’s context persistent. I should have started there. A tool that does not look like it belongs on a control-room desk does not get evaluated on its ideas.
I would test the reversal window rather than assume it. Eight seconds is a guess. The window also closes on the operator’s next decision, which means moving quickly forfeits the undo. That is a defensible rule and it might be the wrong one.
Overrides should feed back into the agent. Every override is a signal that the agent is missing an integration, a rule, or a record. The console records them and stops there. Repeated overrides on the same plate should change what the agent proposes, and that belongs in the product rather than in a future phase.
I would test with real operators next: whether people notice the contradiction quickly, whether the difference between final and correctable lands, whether the cheaper override path actually prevents default approval, and what belongs in the downstream detection review and camera maintenance queues that this console currently only names.
Explore the working prototype: agent-action.vercel.app