AI Agent Auditor
A practical publication for evaluating and auditing AI agents, covering tools and methods for evaluation, observability, testing, safety, and governance.
Safety & Governance
Build an Excel register for AI-agent risks by mapping each entry to NIST AI RMF functions, assigning an owner, recording evidence, and tracking a response and review date.
Safety & Governance
Configure an approval workflow so an agent must receive human sign-off before executing writes, deletions, or configuration changes in production systems.
Testing Tools
Record real agent-LLM exchanges, replay them offline, and assert correct tool selection with a repeatable Pytest suite.
Buying Guides
A practical guide to testing agent safety tools against prompt injection attacks. Learn to craft adversarial prompts, measure detection accuracy, assess false positive rates, and evaluate latency impact to make an informed purchasing decision.
Observability
Add span attributes and error boundaries to agent workflows so failures in tool calls or sub-agents surface with enough context to debug quickly.
Audit Methods
Compare tools and policy workflows for examining cloud permissions assigned to AI-agent service identities, with attention to provider coverage, audit setup, and useful evidence for reviewers.
Observability
Compare tracing platforms that capture agent spans, tool-call inputs/outputs, token usage, and errors across multi-step runs.
Agent Evaluation
Compare LLM-judge tools that score agent responses for accuracy, groundedness, and relevance in multi-step workflows.
Testing Tools
Choose a mock server that simulates third-party API responses so agent tests can run offline and assert correct request payloads.