Home · Testing & QA · webapp-testing
webapp-testing
Anthropic's Playwright toolkit for testing local web apps: a reconnaissance-then-action pattern, a server-lifecycle helper script, and three worked examples.
Score
“A sound browser-driving method wrapped around one useful script — reconnaissance before action, selectors from the rendered page, servers managed for you. It teaches an agent to operate a web app, not to judge one: there is no assertion guidance anywhere in its 95 lines, and its own server helper caused the only crash in our test run.”
Both planted bugs found by running tests — but so did the baseline. The skill's edge was evidence quality, not bug count, and its own helper script caused the run's only crash.
Sub-scores & evidence
| Task completion | Both planted bugs reported with expected-vs-actual values: tip $16.50 vs $15.00, stale per-person $63.25 after the split change. Docked one point because the first attempt died before testing anything (see Failure behaviour). | 11/12 |
| Beats the no-skill baseline | The baseline also found both bugs (4 failing checks). The skill run probed deeper — 8 failing checks, the tip error confirmed at two bill amounts, and the split bug isolated with a before/after probe ($63.25 → $63.25) instead of one wrong final value. | 6/10 |
| Self-verification in output | Reconnaissance before acting is built into the method, and selectors came from the rendered DOM. But SKILL.md contains no assertion or expected-value guidance at all — the verification discipline came from our task, not the skill. | 3/5 |
| Failure behaviour | A real failure, injected by the tooling itself: the first with-skill run crashed with ERR_EMPTY_RESPONSE because a previous with_server.py session left a stale server holding port 8932. Recovery was manual (free the port, rerun); the skill offers no troubleshooting for it. | 1/3 |
A genuine decision tree with one strong pattern — reconnaissance-then-action — that stops halfway: it covers driving the browser, not judging what the browser shows.
Sub-scores & evidence
| Clear steps | Decision tree first (static vs dynamic, server running or not), then a numbered recon-then-action sequence. Nothing after 'execute actions' — no reporting step. | 5/6 |
| Inputs & outputs defined | Commands and the script skeleton are exact, including ports and flags. What a finished test should output — assertions, a bug report format — is never defined. | 3/4 |
| Checks & verification | The networkidle wait and DOM inspection are real check steps. There is no step that compares observed behaviour against expected behaviour. | 2/4 |
| Stop conditions | None stated. No 'ask the user' points; the only hard rules are procedural (headless always, close the browser). | 1/3 |
| Failure recovery | One real fallback: if static HTML inspection fails, treat the app as dynamic. Nothing on server-start failure — the exact failure our run hit. | 2/3 |
95 lines and under 1,000 tokens, with an explicit design goal most skills never state: keep helper scripts out of the context window entirely.
Sub-scores & evidence
| Leanness | SKILL.md is 95 lines / 3,913 bytes — about 978 tokens in the always-loaded layer. | 5/5 |
| Progressive disclosure | Three examples (105 lines total) sit in examples/ and load on demand; scripts are invoked as black boxes via --help instead of being read. The reference layer is thin, though — three small files is all of it. | 4/5 |
| Organisation & naming | Decision tree, worked example, pattern, pitfalls, best practices — in the order an agent works through them. | 3/3 |
| No padding | Dense throughout. One typo ('abslutely') and minor overlap between the pitfall callout and Best Practices, nothing structural. | 2/2 |
Matches how users actually phrase browser-testing requests, and the word 'local' draws a real boundary. But there is no explicit 'use when' sentence to anchor on.
Sub-scores & evidence
| What it does | 'Interacting with and testing local web applications using Playwright' plus four named supports: verifying functionality, debugging UI, screenshots, browser logs. | 4/4 |
| When to use | Scenarios are clear from the supports list, but there is no explicit 'Use when…' trigger sentence — the agent has to infer. | 2/4 |
| User-word keywords | 'Debugging UI behavior', 'capturing browser screenshots', 'viewing browser logs' — the words users type, not library jargon. | 3/3 |
| When NOT to use | No explicit boundary, but 'local web applications' does real work: production-site audits and unit tests fall outside the stated scope. | 1/3 |
| Naming & voice | Functional compound name; description written for the agent, not the marketplace. | 1/1 |
Nothing malicious anywhere — but this skill's helper runs shell commands by design, which is a wider privilege than the guidance-only skills, and it is never declared.
Sub-scores & evidence
| Least privilege | Audit of all four Python files: no network exfiltration, no env or credential access, sockets only to localhost for readiness checks. Docked because with_server.py executes caller-supplied commands with shell=True — intended, but broad. | 3/4 |
| Declared tool scope | No allowed-tools field, and SKILL.md never states its one hard dependency: Playwright plus a Chromium download. An agent discovers it at import time. | 1/3 |
| Sensitive-action confirmation | No destructive or external actions in scope; the helper terminates the servers it started when the command finishes. | 3/3 |
One genuinely good helper script and advice earned from real runs — let down by examples that carry another environment's file paths.
Sub-scores & evidence
| Examples & templates | Three examples plus a complete script skeleton in SKILL.md. Docked: two examples hardcode /mnt/user-data/outputs/ — a path from Anthropic's own environment — and fail as written anywhere else. | 2/3 |
| Scripts & instruction precision | with_server.py is real engineering: multi-server support, port-readiness polling, cleanup in a finally block. Docked for the flaw our test hit — it terminates the shell wrapper, and a grandchild server process can survive holding the port. | 2/3 |
| Dependencies stated | Playwright is imported in every example, so the dependency is obvious — but no install command or version appears anywhere in the skill. | 1/2 |
| Concrete instructions | Specific and checkable: 'Always run scripts with --help first', 'wait for networkidle before inspecting the DOM', headless always, close the browser when done. | 2/2 |
Best for / not for
Best for
- Ad-hoc verification of a local dev app: does the flow work, what does the console say, what does it look like
- Agents that flail at dynamic UIs — the recon-then-action pattern forces looking before clicking
- Multi-server projects (separate API and frontend) where lifecycle handling is the annoying part
Not for
- Building a maintained test suite — there are no assertions, fixtures or reporting patterns here
- Testing production websites — the stated scope is local applications
- Environments where a stray server process on a port is costly; the helper's cleanup can leave one behind
What it does
webapp-testing is a method, not a library. Its 95-line SKILL.md teaches one workflow for testing local web apps with Python Playwright: decide whether the page is static or dynamic, decide whether the server is already running, and — for anything dynamic — reconnaissance before action. Navigate, wait for networkidle, screenshot, inspect the rendered DOM, take your selectors from what the browser actually built, and only then start clicking.
One helper script ships with it: scripts/with_server.py, which starts one or more servers (a separate API and frontend is the stated case), polls their ports until they answer, runs your command, and tears everything down. The skill is explicit that scripts like this are black boxes — run --help, do not read the source into context. Three examples in examples/ cover element discovery, console-log capture, and file:// automation for static HTML.
How the live test went
We built a tip calculator with two planted bugs — a tip computed on bill-plus-tax instead of the bill, and a people-split field with no event listener — and fixed the expected values ($15.00 tip, $125.00 total, $62.50 each; $31.25 each at four people) before either side ran.
The baseline needed no help. A single straight-line Playwright script, selectors read from the source, produced four failing assertions and caught both bugs. If the question is “can an agent find planted bugs in a small app”, the honest answer is yes, skill or no skill.
The with-skill run earned its keep differently. Following the reconnaissance step, its script isolated the split bug with a before/after probe — $63.25, change the people count, still $63.25 — which proves the field is stale rather than merely miscalculated, and it re-confirmed the tip bug at a second bill amount ($33.00 shown vs $30.00 expected on a $200 bill). Eight failing checks to the baseline’s four, each one a separate piece of evidence.
It also produced the day’s only crash. The first with-skill attempt died at page.goto with ERR_EMPTY_RESPONSE: a server from the earlier reconnaissance session, started through with_server.py, was still holding the port. The helper terminates the shell it spawned; the grandchild server process can outlive it. We freed the port by hand and the rerun passed end to end — but the skill documents no recovery for this, and it is the helper’s own design that creates the trap.
Where it falls short
The largest gap is in the name. This is a testing skill with no testing in it: no assertions, no expected-versus-actual pattern, no guidance on what to check or how to report a failure. Everything in our run that constituted verification came from the task brief. The skill makes an agent competent at driving a browser; whether the agent judges the app correctly is left entirely to the agent.
Smaller, concrete items. Two of the three examples write output to /mnt/user-data/outputs/ — a path from a different environment — and fail as written anywhere else. The Playwright dependency (plus its Chromium download) is never stated as a setup step. And the description, while well matched to user phrasing, carries no explicit “use when” sentence — an agent deciding between this and a general browser tool gets less signal than it should.
Live A/B test — the evidence
One standard task, run twice against the same app: a single-file tip calculator (bill, tax %, tip %, people) with exactly two planted bugs, fixed in advance as ground truth. Bug A: the tip is computed on (bill + tax) instead of the bill alone — with the defaults (100, 10%, 15%, 2 people) the app shows tip $16.50, total $126.50, each pays $63.25; correct is $15.00 / $125.00 / $62.50. Bug B: the people input has no event listener, so 'Each pays' never updates when only the split count changes — after setting people to 4 it stays at $63.25 instead of $31.25. Both sides got the same brief: write and run automated browser tests to verify the app works correctly, and report any bugs with evidence.
With the skill
Approach: The skill's method end to end: with_server.py --help first, reconnaissance (screenshot + element discovery on the rendered page), then an assertion script with selectors from the recon output, run through with_server.py managing the server
- Bug A found by running: tip $16.50 vs expected $15.00, and confirmed again at bill $200 ($33.00 vs $30.00)
- Bug B isolated by a before/after probe: per-person $63.25 before the people change and $63.25 after — proof the field is stale, not just wrong
- 8 failing checks total; console capture wired in per the skill's example (the app logged nothing — 0 messages, reported as such)
- First attempt crashed before testing: ERR_EMPTY_RESPONSE from a stale server left on the port by a previous with_server.py session; passed on rerun after the port was freed
Without the skill (baseline)
Approach: One straight-line Playwright script written directly from the task, selectors taken from reading the HTML source, no reconnaissance phase
- Bug A found by running: tip $16.50 vs expected $15.00, total $126.50 vs $125.00
- Bug B found by running: after setting people to 4, per-person still $63.25 vs expected $31.25
- 4 failing checks total — both bugs caught, with thinner evidence (no isolation probe, no second data point, no console capture)
| Objective count | With skill | Baseline |
|---|---|---|
| Planted bugs found by running tests | 2/2 | 2/2 |
| Planted bugs found by reading code only | 0 | 0 |
| Failing checks produced as evidence | 8 | 4 |
| Run crashes before testing began | 1 (stale server on port) | 0 |
Both sides caught both bugs — assertion-based testing finds planted maths and wiring errors with or without a skill. What the skill changed was the quality of the proof: a before/after probe that isolates the stale field, a second bill amount confirming the tip formula, and selectors taken from the rendered page instead of the source. It also supplied the run's only crash, via its own server helper.
Full outputs: The app under test (with both planted bugs) · With-skill test script · With-skill run log · Reconnaissance script & log · Baseline test script · Baseline run log
Trigger tests
| Prompt | Expected | Result |
|---|---|---|
| “Test my local web app on localhost:3000 and check the checkout flow works” | trigger | Triggers — the description's core case, near-verbatim. |
| “Screenshot my app's dashboard and check the browser console for errors” | trigger | Triggers — screenshots and browser logs are named supports. |
| “Debug why this button does nothing in my dev app” | trigger | Triggers — 'debugging UI behavior' is in the description. |
| “Write unit tests for this Python function” | skip | Skips — no web application, no browser; outside the stated scope. |
| “Run an accessibility audit on our production website” | skip | Borderline — Playwright could do it, but the description scopes the skill to local applications, and it offers no audit methodology. |
Desk-simulated against the skill's description matching rules; noted as simulated wherever the score depends on it.
Security screen
✓ Pass
Credential theft / exfiltration
✓ Pass
Remote code download & execute
✓ Pass
Data sent to unrelated third parties
✓ Pass
Hidden instructions (incl. invisible Unicode)
✓ Pass
Obfuscated payloads
✓ Pass
Privilege escalation requests
Screening only — not a security certification. We read all four shipped Python files in full and pattern-scanned for network, shell, environment and credential access; the only shell execution is with_server.py running the caller's own server commands, which is its stated purpose. Industry context: Snyk's ToxicSkills scan of 3,984 public skills found security flaws in 36.82% of them.
Install
Claude Code (plugin)
/plugin marketplace add anthropics/skills /plugin install example-skills@anthropic-agent-skills
Any agent (skills CLI)
npx skills add anthropics/skills --skill webapp-testing
Cursor / Codex (manual)
# copy the webapp-testing folder into: .cursor/skills/webapp-testing/ # Cursor project .agents/skills/webapp-testing/ # shared agents folder # then: pip install playwright && playwright install chromium
Alternatives
Playwright Test Agents (Microsoft)
Planner, generator and healer agents built into @playwright/test itself. If what you want is a maintained, self-repairing test suite inside a Playwright project, that is the framework-native route; webapp-testing is lighter and aimed at ad-hoc verification of a running app.
Frequently asked questions
Is the webapp-testing skill safe?
In our screening on 2026-10-23, webapp-testing passed all six security red lines (credential theft, remote code execution, data exfiltration, hidden instructions, obfuscated payloads, privilege abuse) and scored 7/10 on the safety dimension. Screening is not a certification — re-screen after major updates.
What is the webapp-testing skill used for?
Anthropic's Playwright toolkit for testing local web apps: a reconnaissance-then-action pattern, a server-lifecycle helper script, and three worked examples.
How do I install the webapp-testing skill?
For Claude Code, add the marketplace and install the plugin set that contains it. With the skills CLI: npx skills add anthropics/skills --skill webapp-testing. For Cursor or Codex, copy the webapp-testing folder into .cursor/skills/ or .agents/skills/. Full commands are in the Install section of this page.
How good is the webapp-testing skill?
webapp-testing scored 73/100 (grade B) in our live test on 2026-10-23: A sound browser-driving method wrapped around one useful script — reconnaissance before action, selectors from the rendered page, servers managed for you. It teaches an agent to operate a web app, not to judge one: there is no assertion guidance anywhere in its 95 lines, and its own server helper caused the only crash in our test run.
Tested 2026-10-23 in an agent environment following the skill's own instructions — not inside the Cursor desktop app. Scores follow the Skills List methodology v1: red lines veto regardless of score; install counts and stars never count toward quality. More in Testing & QA.