The fix often happens somewhere else
A failing compliance test is a starting point, not a task someone can always finish inside Vanta. Resolving it may require changing a cloud configuration or infrastructure code, often with help from an engineer who is not a Vanta user.
That creates two different kinds of context. Vanta knows the test, the affected resources, and the compliance requirement. An engineer or coding assistant knows the repository and how the infrastructure is built. A useful remediation experience has to connect those contexts without pretending either tool knows everything.
The old experience offered generic instructions maintained by hand. People still had to translate those steps into a change that made sense for their environment, then return to check whether the test passed.
Put the work where the context is
I shaped the experience around the choice of where remediation should happen. Guided instructions could help with configuration changes. For repository-specific work, a handoff to the customer's coding assistant could bring Vanta's test context into the environment where the code was available.
Our team owned the Claude Code plugin, with remediation skills designed to work across coding assistants. The team used MCP to make compliance context available to connected tools; my focus was the experience around that capability, from understanding the test to choosing an action and knowing what to verify.
The tradeoff was explicit: keeping every step inside Vanta would look more self-contained, but would not give the agent the codebase context it needed. The handoff was part of the product, not an escape hatch from it.
Make the next action clear
Context-aware output still needs an understandable interaction. While instructions generated, I gave people the test and scope to read instead of a blank waiting state. I separated the remediation steps visually so the next action was easier to find.
I replaced ambiguous free-text completion with explicit actions and a prompt to rerun the test after the fix. A person saying they are done is not the same as a passing check. The return to verification needed to be part of the flow.
These decisions focused on the moments around the generated answer: what someone understands while waiting, what they do next, and how they establish whether the work is complete.
Tool I built / LangSmith
Building a tool to understand agent behavior
The question was where the remediation experience lost people: while the agent generated instructions, when they acted outside Vanta, or when they returned to verify a fix.
I built a dashboard on the LangSmith API to isolate relevant production sessions and inspect the sequence of guidance, actions, and follow-up. The team used it during recurring trace reviews.
Those reviews exposed a gap between receiving instructions and confirming what happened next. A blank waiting state also gave people little context while instructions generated. Receiving an answer was not evidence that a test had been fixed.
I used those findings to show the test and affected scope during generation, replace free-text completion with explicit actions, and prompt people to rerun the test. The dashboard connected an observed interaction problem to a specific design change.
A workflow that could be reused
I also worked on making remediation available beyond one guided entry point, including starting from a filtered test list. That required thinking of the skill as a reusable capability rather than a single conversation.
The durable contribution was the combination of context-aware guidance, an intentional developer-tool handoff, and a repeatable way to examine agent behavior. Product design extended from the interface into how the team learned about the system.
The interface is not always where the agent works
This work changed how I think about AI interaction. A helpful response is only one part of a successful workflow. People still need to understand the scope, choose where work happens, and verify the result.
The unresolved design challenge is continuity across tools. A handoff can carry useful instructions without proving that a change was made. That distinction matters whenever a product helps people delegate technical work.