semgrep
Trail of Bits' approval-gated Semgrep scan: language detection, ruleset selection, a hard plan-approval step, parallel scans and merged SARIF — with telemetry off by construction.
Score
“The strongest process we have scored — hard approval gate, failure accounting, telemetry off by construction — wrapped around a scanner that missed the SQL injection in a 71-line file. Trust it to run a scan honestly; do not trust the scan alone to clear a file.”
The pipeline ran perfectly and the scanner missed the SQL injection. Three of four planted vulnerabilities found; the no-skill baseline found all four by reading the file.
Sub-scores & evidence
| Task completion | End-to-end run: 6 rulesets scanned, merged SARIF written, report delivered. But the audit output itself is incomplete — the line-41 SQL injection is absent from the final report. | 8/12 |
| Beats the no-skill baseline | It lost. Baseline: 4/4 planted issues, 0 false positives. With skill: 3/4, 0 false positives. The scanner's ceiling is its rule set, and no OSS rule fired on a %-formatted sqlite3 query. | 2/10 |
| Self-verification in output | scans.json accounts for every ruleset (ran / failed / skipped / partial / covered-nothing); the report labelled both partial rulesets instead of burying them. Every number in the report can be checked against the SARIF. | 5/5 |
| Failure behaviour | Partial rule compilations and the missing Pro engine were surfaced honestly. The deep failure — a whole vulnerability class absent — produces no signal; the report reads complete unless you hold the ground truth. | 2/3 |
The best workflow we have scored. Five gated steps, an anti-rationalization table, and failure accounting designed in rather than improvised.
Sub-scores & evidence
| Clear steps | Five numbered steps with the gate marked, a summary table routing to a 468-line detailed workflow, and task creation with blockedBy dependencies to enforce the order. | 6/6 |
| Inputs & outputs defined | Output-directory resolution protocol, rulesets.json as the approved plan in, scans.json and merged SARIF out — the directory layout is specified to the filename. | 4/4 |
| Checks & verification | A 20-item success-criteria checklist closes the skill; failed, skipped, partial, covered-nothing and oversized-file results must all appear in the report, and the merge deliberately drops output from failed scans. | 4/4 |
| Stop conditions | Step 3 is a hard gate: the initial 'scan this' request explicitly does not count as approval. A sanctioned no-gate path (the bundled workflow) exists, so the gate is a scope confirmation, not theatre. | 3/3 |
| Failure recovery | Failure behaviour is specified in advance: a non-zero scanner exit means report and stop — the skill forbids hand-running a subset and presenting it as the scan. | 3/3 |
A 273-line core with 802 lines of references and workflow behind it. Heavier than the Anthropic document skills, and it spends the weight on gates and checklists.
Sub-scores & evidence
| Leanness | SKILL.md is 273 lines / 15,678 bytes — about 3,920 tokens, under the 500-line guideline but twice the pdf skill's core. | 4/5 |
| Progressive disclosure | rulesets.md (162 lines), scan-modes.md (172) and scan-workflow.md (468) load on demand; the core file carries the summary and the routing. | 5/5 |
| Organisation & naming | Principles, when / when-not, output contract, modes, orchestration diagram, workflow, rationalizations, reference index — in execution order. | 3/3 |
| No padding | Every section is a rule, a contract or a checklist. The one soft spot is a quantitative claim (Pro catches '~250% more true positives') stated without a source. | 2/2 |
Precise about what it runs and when — and one of the few skills that ships an explicit 'when NOT to use' list with named alternatives.
Sub-scores & evidence
| What it does | The description is a compressed spec: detects languages, selects rulesets, presents the plan for approval, runs run-scans.sh, merges to SARIF, two scan modes named. | 4/4 |
| When to use | 'Use when asked to scan code for vulnerabilities, run a security audit with Semgrep, find bugs, or perform static analysis.' | 4/4 |
| User-word keywords | Scan, vulnerabilities, security audit, static analysis, find bugs — the words people actually type. | 3/3 |
| When NOT to use | Not in the description itself, but the body lists five exclusions with routings: binary analysis, existing CI, custom rules (semgrep-rule-creator), rule porting, no-Pro cross-file work (CodeQL). | 2/3 |
| Naming & voice | The tool's own name; the description is written as an instruction the agent can act on. | 1/1 |
The first perfect safety score we have given: read-only over the target, declared tool scope, an approval gate, and telemetry disabled in the script rather than by instruction.
Sub-scores & evidence
| Least privilege | Scans never modify the target (no --autofix); every write lands inside the output directory. run-scans.sh hard-codes --metrics=off so audit targets are not phoned home to Semgrep's telemetry. | 4/4 |
| Declared tool scope | allowed-tools is declared in frontmatter (Bash, Read, Glob, AskUserQuestion, task tools). The third-party rule repositories are enumerated in rulesets.md, not fetched on a whim. | 3/3 |
| Sensitive-action confirmation | Nothing scans until the user approves the exact ruleset list; the approved plan is written to rulesets.json and passed to the runner unchanged. Cloned rule repos are deleted at the end of the run. | 3/3 |
Instruction precision achieved by construction: the 786-line runner script generates every scanner command, so the safety flags cannot drift. Ships its own test suite.
Sub-scores & evidence
| Examples & templates | Output-directory layout, rulesets.json format, mode-selection table, and jq post-filter commands in scan-modes.md. | 3/3 |
| Scripts & instruction precision | run-scans.sh builds all commands from the approved plan; merge_sarif.py (339 lines) ships with a 581-line test file beside it. | 3/3 |
| Dependencies stated | Semgrep CLI and optional Pro are named, with a working Pro-detection snippet. But the merge step assumes uv, which is never listed as a prerequisite — our test environment had to substitute python3 — and git, needed for cloning rule repos, is never mentioned. | 1/2 |
| Concrete instructions | Scar tissue throughout: clone rule repos first because URL handling fails on non-standard YAML; files over the size cap are reported as oversized because a skipped file otherwise reads as a clean one. | 2/2 |
Best for / not for
Best for
- A first pass over a whole repository, where a recorded scan plan matters more than a chat opinion
- Teams that need the run itself audited — what ran, what failed, what covered nothing
- Repeat scans that must execute the same rulesets the same way every time
Not for
- Clearing one small file — the gated pipeline is heavy, and a plain read beat it in our test
- Anyone treating a clean scan as a clean bill of health; OSS rules missed a textbook SQL injection here
- Cross-file taint questions without a Semgrep Pro licence — the skill says so itself and points to CodeQL
What it does
semgrep is an orchestration skill, not a rule pack. It wraps the Semgrep CLI in a five-step process: detect the languages in the target, select rulesets from a catalog (the official registry sets plus third-party repositories from Trail of Bits, 0xdea, Decurity and others), present that exact plan for explicit approval, run every approved ruleset through a bundled 786-line shell script, then merge the per-scan SARIF files into one result and report what ran, what failed and what covered nothing.
Two design choices run through it. The first is that the important properties are enforced by the script, not requested in prose: run-scans.sh generates every scanner command from the approved plan, which is how --metrics=off (Semgrep sends telemetry by default), the per-language --include scoping and the output-directory exclusions survive contact with a busy agent. The second is a hard approval gate at Step 3 — the skill states, in a table of “rationalizations to reject”, that the user’s original scan request is not approval of the plan.
It is honest about its own ceiling. OSS Semgrep cannot track data flow across files; the skill checks for a Pro engine before scanning, and its when-not-to-use list sends cross-file work without a Pro licence to CodeQL instead.
How the live test went
We fixed the ground truth first: a 71-line Python service with four planted vulnerabilities (SQL injection, a hardcoded Stripe key, eval() on request input, MD5 password hashing) and one parameterized query as a false-positive control. The with-skill run followed the process end to end — Python detected, run-all mode, six rulesets approved at the gate and recorded in rulesets.json, six scans executed, two of them (elttam, Apiiro) labelled partial because some of their non-Python rules failed to compile, merged SARIF written, cloned rule repos cleaned up.
The scanner reported five findings that dedupe to three real vulnerabilities: the key, the MD5 hash and the eval() — each caught, the MD5 and the eval() by two rules apiece. The SQL injection was never flagged. The query is built with % formatting on line 41 and executed on line 42, a two-line source-to-sink flow, and no rule in the six rulesets — official or third-party — covers that shape for stdlib sqlite3 in OSS mode.
The baseline agent, given the same file and no tools beyond reading it, found all four at the exact lines, flagged nothing else, and wrote the more specific fixes. We publish that plainly: on a file this size, the skill’s process is overhead wrapped around a weaker result. The process is also the only part that scales — the baseline’s careful read does not survive contact with a 400-file repository, and the skill’s run leaves an auditable record (rulesets.json, scans.json, merged SARIF) that a chat answer never does.
Where it falls short
The miss is the headline: a clean-looking scan from this pipeline is evidence about rule coverage, not about the code, and the report format cannot tell the difference — the absent SQL injection left no trace in any output file. Budget for a human or dataflow pass (the same plugin’s CodeQL skill) on anything that matters. Second, the dependency list is incomplete: the merge step assumes uv, which the prerequisites never mention — our environment did not have it and substituted python3 — and git is required to clone rule repositories without ever being named. Third, the approval gate costs a round trip on every scan; the bundled no-gate workflow exists for repeat runs, but the skill’s default path will feel heavy for a one-file check.
Live A/B test — the evidence
One file, audited twice: user_service.py, a 71-line stdlib Python service with exactly four planted vulnerabilities fixed in advance — SQL injection via a %-formatted query (line 41), a hardcoded live Stripe key (line 11), eval() on request input (line 54), and MD5 password hashing (line 25). Both sides report each finding with severity, line, exploitability and fix. A parameterized INSERT in create_user() is the control: flagging it counts as a false positive.
With the skill
Approach: The skill's five steps, run as written: language detection, run-all mode, plan approved at the Step 3 gate (p/security-audit, p/secrets, p/python plus the Trail of Bits, elttam and Apiiro rule repos), scripts/run-scans.sh for execution, merge_sarif.py for the merged SARIF
- 3 of 4 planted vulnerabilities in the final report: hardcoded key (Critical, line 11), MD5 password hash (High, line 25), eval() RCE (Critical, line 54)
- The SQL injection at line 41 was missed — no rule in the six rulesets fired on the %-formatted sqlite3 query
- Scanner produced 5 SARIF results (MD5 and eval each hit by two rules); both partial rulesets were labelled partial in the report, per the skill's own rules
Without the skill (baseline)
Approach: An agent read the file and wrote the audit unaided — no scanner, no skill
- 4 of 4 planted vulnerabilities found, at the exact lines, with zero false positives
- Severities: SQL injection Critical, eval() Critical, hardcoded key Critical, MD5 High — all within one band of the expected ratings
- Fixes were the most specific of the two reports (parameterized query mirroring create_user, ast allowlist walker for the calculator)
| Objective count | With skill | Baseline |
|---|---|---|
| Planted vulnerabilities found | 3/4 | 4/4 |
| False positives | 0 | 0 |
| Severities within one band of expected | 3/3 found | 4/4 |
| Raw scanner findings (merged SARIF) | 5 | — |
| Rulesets run | 6 (2 partial, labelled) | — |
The baseline won the audit and the skill won everything around the audit. On one small file, a careful reader beats a rule set — the scanner never saw the SQL injection because no rule covered that pattern in OSS mode. What the skill adds is a run you can inspect afterwards: an approved plan, a per-ruleset account, and partial failures labelled instead of hidden.
Full outputs: The audited file (with ground truth) · Ground truth — the four planted issues · With-skill report · Baseline report · Merged SARIF from the scan · scans.json — the run accounting
Trigger tests
| Prompt | Expected | Result |
|---|---|---|
| “Scan this repository for vulnerabilities” | trigger | Triggers — the description's first use case, verbatim. |
| “Run a Semgrep security audit on this codebase” | trigger | Triggers — named explicitly in the description. |
| “Do a static analysis pass before we ship” | trigger | Triggers — 'perform static analysis' is in the use-when list. |
| “Write a custom Semgrep rule for our internal API” | skip | Skips — the body's when-not list routes this to the semgrep-rule-creator skill. |
| “Review this pull request diff for security issues” | skip | Borderline — the description promises codebase scans, not diffs; the same repo's differential-review skill is the right-shaped tool. |
Desk-simulated against the skill's description matching rules; noted as simulated wherever the score depends on it.
Security screen
✓ Pass
Credential theft / exfiltration
✓ Pass
Remote code download & execute
✓ Pass
Data sent to unrelated third parties
✓ Pass
Hidden instructions (incl. invisible Unicode)
✓ Pass
Obfuscated payloads
✓ Pass
Privilege escalation requests
Screening only — not a security certification. One judgement call, stated openly: run-scans.sh clones the third-party rule repositories listed in rulesets.md (YAML pattern files, not executable code) after the user approves them at the Step 3 gate, and runs a local scanner. Nothing is fetched silently. The runner also forces --metrics=off, the opposite of exfiltration. Industry context: Snyk's ToxicSkills scan of 3,984 public skills found security flaws in 36.82% of them.
Install
Claude Code (plugin)
/plugin marketplace add trailofbits/skills /plugin install static-analysis@trailofbits
Codex (plugin)
codex plugin marketplace add trailofbits/skills codex plugin add static-analysis@trailofbits
Cursor / manual
# copy the semgrep folder (plugins/static-analysis/skills/semgrep) into: .cursor/skills/semgrep/ # Cursor project .agents/skills/semgrep/ # shared agents folder # prerequisite: the Semgrep CLI — check with: semgrep --version
Alternatives
codeql (Trail of Bits, same plugin)
Dataflow analysis that builds a database of the codebase and tracks taint across files — the gap our test exposed (the missed injection was a short source-to-sink flow). Heavier: the database build is its own workflow, and the skill ships a full build-troubleshooting guide for when it fails.
differential-review (Trail of Bits)
For the other shape of question — 'is this change safe?' — it reviews diffs with git-blame context and blast-radius counting instead of scanning the whole tree. Pick it for pull requests; pick semgrep for a standing codebase.
Frequently asked questions
Is the semgrep skill safe?
In our screening on 2026-10-23, semgrep passed all six security red lines (credential theft, remote code execution, data exfiltration, hidden instructions, obfuscated payloads, privilege abuse) and scored 10/10 on the safety dimension. Screening is not a certification — re-screen after major updates.
What is the semgrep skill used for?
Trail of Bits' approval-gated Semgrep scan: language detection, ruleset selection, a hard plan-approval step, parallel scans and merged SARIF — with telemetry off by construction.
How do I install the semgrep skill?
For Claude Code, add the marketplace and install the plugin set that contains it. With the skills CLI: npx skills add anthropics/skills --skill semgrep. For Cursor or Codex, copy the semgrep folder into .cursor/skills/ or .agents/skills/. Full commands are in the Install section of this page.
How good is the semgrep skill?
semgrep scored 84/100 (grade A) in our live test on 2026-10-23: The strongest process we have scored — hard approval gate, failure accounting, telemetry off by construction — wrapped around a scanner that missed the SQL injection in a 71-line file. Trust it to run a scan honestly; do not trust the scan alone to clear a file.
Tested 2026-10-23 in an agent environment following the skill's own instructions — not inside the Cursor desktop app. Scores follow the Skills List methodology v1: red lines veto regardless of score; install counts and stars never count toward quality. More in Security.