JMeter
Real checkouts.
Executable task pass rate
- DeepSeek V4 Pro66.7
- DeepSeek Flash66.7
PERFORMANCE, UNDER TEST.
An open benchmark for AI on performance test scripting and engineering.
01 / THE LEADERBOARD
Every segment is earned. Explore where each model picked up its points.
Points = category score × weight ÷ present weights. Five active categories; 75 total weight. Hover, focus or tap a segment.
02 / THE COST FRONTIER
A full seed run, priced in dollars. Up and left wins.
A SMALL BUDGET. A REAL CHOICE.
Find the highest score within your per-run budget.
DeepSeek V4 Pro
82.8 points · $0.0367 / run
Two models. One run each. A starting signal, not a final verdict.
03 / UNDER THE HOOD
The same task, the same prompt. Focus or tap a result to inspect its grading detail.
| Category / task | DeepSeek V4 Pro | DeepSeek Flash |
|---|---|---|
| authoring1 tasks | ||
| shop-flowjmeter | Verified completions: 1/1 | Verified completions: 1/1 |
| correlation2 tasks | ||
| session-isolationjmeter | Empty run or failed samples | Empty run or failed samples |
| token-refreshjmeter | Verified completions: 2/2 | Verified completions: 2/2 |
| workload4 tasks | ||
| arrivalreasoning | numeric comparison | numeric comparison |
| concurrencyreasoning | numeric comparison | numeric comparison |
| little-lawreasoning | numeric comparison | numeric comparison |
| pacingreasoning | numeric comparison | numeric comparison |
| analysis2 tasks | ||
| errorsreasoning | numeric comparison | numeric comparison |
| meanreasoning | numeric comparison | numeric comparison |
| concepts3 tasks | ||
| closedreasoning | exact choice comparison | exact choice comparison |
| correlationreasoning | exact choice comparison | exact choice comparison |
| percentilereasoning | exact choice comparison | exact choice comparison |
Repair and migration: not yet benchmarked. Results are binary, with no partial credit.
04 / PICK YOUR INSTRUMENT
Executable plans first. More engines next.
Executable task pass rate
Graders coming
Graders coming
Graders coming
JMeter score = percentage of executable tasks passed. No scores are assigned to tools without graders.
Direct provider APIs. Temperature 0. One fixed system prompt and identical task prompts for every model. No routing layer.
Generated JMeter plans run against a loopback demo shop. Passing requires zero sample errors and verified session-aware checkouts. Numeric and choice tasks use deterministic answer keys.
Category means are weighted, then renormalized over categories with tasks. Unmeasured categories stay unmeasured, never zero.
Cost is estimated from published peak list prices, including cache-hit pricing. It is not an invoice. Latency measures provider calls, excluding grading.
This public seed pack is small: one run per model, not a confidence interval. DeepSeek thinking is disabled. Output cap: 8,192 tokens. Tool scores and overall scores measure different slices.