How we test

Rankings are earned, never bought. A tool gets a rank only after we've run it ourselves.

1. Clean install

We install the exact released version in a fresh environment (for example, a clean Python virtualenv) on our own machine and record anything that breaks.

2. Run it end to end

We build a minimal agent or task and run it end to end. Where possible we first use an offline stand-in model, so the test is reproducible without API keys, then run a live model when we can.

3. Score on a fixed rubric

Fit 25%, output quality 25%, ease of start 15%, value 15%, reliability 10%, momentum 10%. Anything we couldn't check is scored conservatively and noted. Full details are in How we rank.

4. Publish the notes

Each tested tool shows the test date, what we did and its weak spots. Every claim links to its public source.

What money can't buy

Featured slots, if we ever sell them, are labelled "Featured" and never change a score or rank. Untested tools are listed, not ranked.

Tested so far (2)

Pydantic AI 2.56.0: 7.7/10, tested 2026-10-15
LangChain Deep Agents 0.7.23: 7.2/10, tested 2026-10-14