If you've read What Is Agent Experience and Why Should You Care? you already know that agent experience (AX) is the new developer experience — and that if AI coding agents can't find, understand, and use your platform, you're invisible to a growing portion of the developer market.
The natural follow-up question is: how do you measure it objectively?
The obvious answer (ask an agent to audit your AX) turns out not to work that well. Agents don't use developer platforms in the same way humans do, and while an agent can produce a convincing AX report for a given target, we found that those reports didn't always match our actual experiences using those platforms with our agents.
So we've developed a rubric and a mix of automated and manual testing to evaluate technical products from an agent's point of view. We summarize this into topline metrics across four stages, and then show detailed qualitative examples of working with an agent to find, evaluate, and use a specific platform.
In this walkthrough, we audit PostHog, a product analytics platform.
The four stages of an AX audit
The audit follows the funnel a developer goes through with their agent:
- Awareness: Does the model already know about the product, without tools?
- Discoverability: Does it find the product when it searches online?
- Reputation: Does it recommend the product over alternatives, and get the facts right?
- Integration: Can it build and verify a small working integration?
Each stage gets a score from 1 to 4: GOOD (4) means little friction; OK (3) means minor gaps; POOR (2) means a partial result with significant gaps; FAIL (1) means the product is unknown or the task doesn't work. Leave a score provisional when verification is incomplete.
Six questions per stage
Use one set of six questions for awareness and discoverability. Start with a broad problem, add specific requirements, narrow toward the product's positioning, and finish with a hint at its name. For discoverability, append “look it up online first” to each question and enable web search and fetch. This makes the difference between memory and search easier to see.
The reputation questions name the product and test opinions, downsides, competitor comparisons, pricing, and a final recommendation. Integration moves from a generic request in a small sample app to an explicit setup and a question about agent tooling.
Every question starts a fresh session, including integration. Later questions do not inherit earlier answers or code. These are independent tests at different levels of specificity, rather than a conversation that trains the agent to name the target.
For awareness and discoverability, score the first usable answer that mentions the target:
| First mention | Score |
|---|---|
| Question 1 or 2: broad problem or category | 4/4 — GOOD |
| Question 3 or 4: specific requirements | 3/4 — OK |
| Question 5: close to the target's positioning | 2/4 — POOR |
| Question 6: name hint, or no mention | 1/4 — FAIL |
This measures whether the product enters the answer, not whether it ranks first. Record ranking separately. A failure on the last question shows a gap in this test; it doesn't prove the model knows nothing about the product.
Count each competing product once per answer to questions 1–5 in awareness and discoverability. Exclude unusable answers and the name-hint question. The three most frequent competitors become the reputation shortlist. Integration then focuses on the target.
Running each question in a clean sandbox
We used Claude Sonnet 5.5 (claude-sonnet-5-5) in Claude Code 2.1.296 on 13 October 2026, through a normal Claude subscription. Each question ran in a fresh disposable Linux VM. We planned 24 sessions, one per question. We retried one unusable awareness answer and three integration prompts after correcting how we disclosed the test credentials: 28 sessions in total. The awareness retry was also unusable and is excluded. This is a practical audit of one model, not a statistical benchmark.
Agents pick up context from project files, previous conversations, instructions, and folder names. We use fresh Gondolin micro-VMs with a neutral hostname and an empty /root/app folder. Integration gets a fresh copy of a small Node.js app with a landing page, signup form, and welcome page.
The VMs have no user instructions, saved conversations, memory, MCP servers, or user-installed plugins. Skills and browser extensions are disabled. Claude Code's built-in plugins remain loaded; the session records confirm the actual tool lists. Transcripts are saved on the host after each run, and the VM is deleted.
| Stage | Available tools |
|---|---|
| Awareness | None |
| Discoverability | Web search and web fetch |
| Reputation | Web search and web fetch |
| Integration | Full Claude Code toolset, with access confined to the disposable VM |
For the three integration prompts that explicitly name PostHog, we supplied a test project's key and US ingestion host through environment variables and told the agent they were available. The generic prompts received neither, so credentials couldn't reveal our preferred vendor. The key is redacted from saved evidence.
We checked the generated apps separately in Chrome with that test key. Those browser checks are our verification, rather than work the subject agent performed. Providing an existing account means this audit doesn't measure signup friction.
PostHog AX scorecard
| Stage | Score | Finding |
|---|---|---|
| Awareness | 4/4 — GOOD | Recommended for broad product analytics questions |
| Discoverability | 4/4 — GOOD | Appeared at question 1 after searches that didn't name PostHog |
| Reputation | 3/4 — OK | Recommended over the shortlist, with uncertain pricing and inconsistent details |
| Integration | 2/4 — POOR | Basic tracking, replay, and the pricing flag work after manual project setup; the tooling build breaks signup |
| Overall | 3.25/4 | Arithmetic mean of the four stage scores, without rounding |
AX audit stage 1: Awareness
Awareness tests the model's existing knowledge. Disable tools, run each question in a new session, and record which products it mentions and how it describes them.
The questions we used
| # | Exact question |
|---|---|
| 1 | i launched my app a few weeks ago and have no idea what people actually do in it. how do i find out? |
| 2 | what analytics should i put in my saas app? google analytics feels like the wrong tool for this |
| 3 | i need to see where people drop off between signing up and actually using the thing, and ideally watch recordings of what they did. what should i use? |
| 4 | about to add session recordings, feature flags and a/b tests to our react app. is there one tool that does all of it so we don't need three vendors? |
| 5 | mixpanel and amplitude both look like they'll get expensive once we grow. anything open source i can just run myself? |
| 6 | isn't there an open source analytics thing with 'hog' in the name? |
What happened with PostHog
Question 1 didn't produce an answer. Sonnet said it would inspect the app, then printed XML-like Glob, Grep, and Read invocations as plain text. Its pretend search pattern contained “posthog”, but no tools existed or ran. A retry printed a pretend Bash invocation instead. We excluded both attempts rather than counting those strings as recommendations.
All five remaining answers named PostHog. At question 2, the model called it its default pick for in-app analytics:
“My recommendation: PostHog for in-app, plus Plausible on your marketing site if you want a lightweight traffic view.”
For the funnel-and-recordings question, it opened with “Use PostHog.” For the React question, it also picked PostHog for replay, flags, and experiments.
Some knowledge was stale: the self-hosting answer suggested Docker or Helm. PostHog's self-hosting documentation describes a hobby deployment and says new Kubernetes deployments of its paid open-source product are no longer supported. A positive mention still needs a fact check.
AX audit stage 2: Discoverability
Discoverability tests what the agent finds online. Use the same six questions, with web search and fetch enabled and an explicit instruction to search first. Record its search queries and sources as well as its final recommendations.
The questions we used
| # | Exact question |
|---|---|
| 1 | i launched my app a few weeks ago and have no idea what people actually do in it. how do i find out? look it up online first |
| 2 | what analytics should i put in my saas app? google analytics feels like the wrong tool for this look it up online first |
| 3 | i need to see where people drop off between signing up and actually using the thing, and ideally watch recordings of what they did. what should i use? look it up online first |
| 4 | about to add session recordings, feature flags and a/b tests to our react app. is there one tool that does all of it so we don't need three vendors? look it up online first |
| 5 | mixpanel and amplitude both look like they'll get expensive once we grow. anything open source i can just run myself? look it up online first |
| 6 | isn't there an open source analytics thing with 'hog' in the name? look it up online first |
What happened with PostHog
All six answers named PostHog. The broadest question triggered two searches, neither of which named it:
product analytics for new app: how to understand user behavior, best tools 2026
how to track user behavior in app after launch event tracking session replay funnels retention
The returned sources included vendor comparisons and user-behavior guides. The answer listed Mixpanel first, Amplitude second, and PostHog third, then suggested Heap or PostHog for getting started with autocapture.
This is better evidence of discovery than a query that already contains the product's name. In questions 2–5, at least one query included “PostHog”. Those answers show the agent verifying a product it already knows as well as finding sources about it. Keep that distinction in your audit: search-enabled answers can still depend heavily on awareness.
AX audit stage 3: Reputation
Reputation tests whether the agent recommends your product once it is on the shortlist. Check the recommendation's reasons against official documentation, especially prices and plan restrictions.
Our shortlist was Amplitude, Mixpanel, and Plausible, appearing in nine, seven, and four usable answers respectively across questions 1–5 of the first two stages. These counts represent answers mentioning a product, not raw occurrences of its name.
The questions we used
| # | Exact question |
|---|---|
| 1 | is posthog any good? look it up online first |
| 2 | thinking about using posthog. what are the downsides? look it up online first |
| 3 | posthog or Amplitude for a b2b saas? we're 4 people look it up online first |
| 4 | trying to pick between posthog, Amplitude, Mixpanel and Plausible. we need funnels, retention and session replay, small team, not much budget. which one? look it up online first |
| 5 | compare posthog's pricing and free tier with Amplitude, Mixpanel and Plausible. what would a small saas team actually pay? look it up online first |
| 6 | we're about to sign up for posthog. talk me out of it or tell me it's fine look it up online first |
What happened with PostHog
Sonnet recommended PostHog over Amplitude for the four-person B2B team and picked it in the four-way comparison. The general opinion and final signup questions were positive, with caveats about learning curve, usage-based bills, and the needs of non-technical teams.
“Pick PostHog. I couldn't pull its numbers from the pricing page, so check those before you commit.”
The free allowances it quoted match PostHog's Markdown pricing page: one million analytics events, 5,000 recordings, and one million feature-flag requests per month. But the agent couldn't extract PostHog's own pricing in the comparison runs and relied on memory or third-party breakdowns. It didn't use the Markdown pricing page.
The dedicated pricing answer estimated $50–$250 for two million analytics events, depending on identified usage. The official table supports about $50 for base analytics, plus $198 if the second million events also incur the identified-events add-on. The broad range was reasonable, but other details were inconsistent: one answer described Plausible's paid funnels, while another said it didn't provide product-style funnels. The Amplitude comparison also left group-analytics pricing unresolved, though PostHog publishes its allowance and rates.
A favorable recommendation is useful; unresolved or contradictory details make it less dependable. That earns an OK score rather than a full mark.
AX audit stage 4: Integration
Integration tests whether the agent can build something small and prove that it works. Use a goal with visible success criteria: for this app, a successful signup should send funnel events, a browser session should produce a recording, and a real feature flag should change the pricing section.
Test both generic vendor choice and explicit setup. Also ask about agent tooling, such as Markdown docs, setup wizards, and MCP servers. Record what the agent actually uses, not just what it lists.
The questions we used
The credential note is included verbatim in questions 4–6 below. We provided an existing project key, not a personal API key for managing the project.
| # | Exact question |
|---|---|
| 1 | can you add analytics to this so i can see what people are doing |
| 2 | i want to see where people drop off between landing on the site and finishing signup, and watch recordings of the sessions where they gave up. can you set that up? |
| 3 | set up product analytics, session recordings and a feature flag for the pricing section. one tool for all of it please, i don't want three vendors |
| 4 | add posthog to this app. track the signup funnel and turn on session replay A test project key is already available in POSTHOG_KEY and its ingestion host in POSTHOG_HOST. Use those environment variables; do not print or commit the key. |
| 5 | add posthog to this app with the signup funnel, session replay, and a feature flag that hides the pricing section for half of visitors A test project key is already available in POSTHOG_KEY and its ingestion host in POSTHOG_HOST. Use those environment variables; do not print or commit the key. |
| 6 | does posthog have anything that helps coding agents set it up, like an mcp server, docs for llms, or a setup tool? find it and use it to add posthog to this app, with signup funnel events and session replay A test project key is already available in POSTHOG_KEY and its ingestion host in POSTHOG_HOST. Use those environment variables; do not print or commit the key. |
What happened with PostHog
The broadest request produced a custom analytics service rather than a vendor integration. It stored events locally and exposed password-protected stats. Questions 2 and 3 chose PostHog without being told to; neither had credentials while the agent worked.
For the explicit PostHog requests, Sonnet added runtime key configuration, browser tracking, signup events, and masked replay settings. The feature-flag request also added a hide-pricing check and instructed us to create the flag with a 50% rollout. It correctly said it couldn't create that flag with a project key alone.
The subject's checks mostly stopped at serving scripts and exercising local endpoints. It did not prove that the browser integration worked or that data was available in PostHog.
Our live checks
We ran the generated apps with the real key and exercised the landing page, signup form, and welcome page in Chrome. The demo project initially had replay switched off and no feature flags. We enabled replay and created hide-pricing with a 50% rollout through authenticated project access. These were operator steps; Sonnet did not complete them.
| Check | Observed result |
|---|---|
| Builds from questions 2–5 | SDK loaded, signup reached the welcome page, and PostHog accepted analytics requests |
| Stored signup funnel | One measured session contains all five steps in order: landing pageview, signup pageview, signup_submitted, signup_completed, and welcome pageview |
| Session replay | A real recording opened and played in PostHog; the signup email was masked |
| Pricing flag | The real 50% rollout returned both results for different test visitors; the question 5 build hid pricing when true and showed it when false |
| Tooling build from question 6 | /analytics.js returned landing-page HTML, causing a JavaScript error; submitting the form then failed because track was undefined |
Checking HTTP responses alone would have missed the broken tooling build. For the working build, we also checked stored events, played a recording, and tested the visible effect of both flag results. That is the difference between plausible generated code and a verified integration.
Agent tooling
The explicit tooling question found PostHog's wizard, MCP server, and llms.txt through search, but Sonnet didn't run the wizard or connect the MCP server. It installed posthog-js and posthog-node and wrote its own integration instead. That build served the vendor bundle correctly but omitted a route for its own public/analytics.js, which broke signup.
PostHog's wizard README says its non-interactive --ci mode isn't supported in published builds. Authentication for the wizard or MCP server is a separate requirement from possessing an ingestion key. Listing those tools therefore isn't evidence that an agent can use them autonomously.
The basic integration works, but the explicit request to use agent tooling produced a broken browser flow. Combined with the manual project setup, that is a significant gap and earns a POOR score. The missing script route is a failure in the generated app; it is not evidence that PostHog's SDK or service is broken.
What the audit tells us
PostHog enters the shortlist early and gets positive recommendations. The search run found it without naming it in the broadest queries, and generic integration requests selected it once the requirements included replay.
The later stages exposed the useful gaps: pricing answers relied on secondary sources, available agent tooling wasn't used, and a build that passed local smoke checks failed in a real browser. Those are concrete findings a product team can act on. Keep the prompts, transcripts, human steps, and verification results alongside the scores so another audit can reproduce the method.
Want help understanding and fixing the AX of your own technical product or platform? At Ritza our Engineering Writers work at agent speed with human-expert verification (no slop) to win at GTM.