Evaluating the Stacksift verdict pipeline

The test run

Listing candidates used gpt-4.1-mini. The final product-or-not decision used gpt-4.1, with no automatic tuning. The code was at Stacksift commit a8b6cb9. I ran it once and didn't re-run it to get better numbers.

Dialpad has three products in my answer sheet. Stacksift found two (Support and Sell) and missed Meetings. It also named "Dialpad AI Agents" and plain "Dialpad", and my answer sheet lists both as not products: traps I'd already written down. Its score for Dialpad was 0.571 (precision 0.500, recall 0.667). Checking candidates one at a time didn't rescue Meetings, and it still fell for two known traps.

Dialpad's score changed from run to run (0.333, then 0.667, then 0.571), so a single company's score from a single run is noisy.

The worst score was 0. For demandbase.com it returned only the brand name instead of the three products in my answer sheet. For c3.ai it returned a completely different product list from mine. Both zeros are included in the overall score. They aren't one-off glitches; that's simply how the tool does on those sites.

Other features it wrongly called products this run: Zapier Tables and Algolia AI Browse. For HubSpot it missed Content Hub and Operations Hub, and made up a "Revenue Hub".

How scoring works

scripts/eval.py runs the full five-step tool on the 20 checked companies only. Names are matched exactly first, then loosely when one name contains the other as whole words. A one-word guess can't match a longer product name just because they share the brand: "Dialpad" doesn't count as "Dialpad Support".

Precision is the share of the tool's answers that were correct. Recall is the share of the real products it found. A wrong answer that's also on my "not a product" list is tagged separately (matches_known_not_product). Those are worse than a made-up name, because the tool fell for a trap I had already written down.

Only companies where search actually ran count toward the overall score. If every search call failed, the company is listed separately as "search unavailable" instead of counting as a zero. In this run, all 20 were scored and none were left out. If search did run and the tool returned nothing, that still counts as a total miss.

How the answer sheet is built

data/eval_labels.json has room for 32 companies. Twenty were checked by hand and marked verified on 18 April 2026. The other twelve aren't checked yet and are skipped when scoring.

Each checked company has two lists, not one. The first is the real products: things a customer can buy on their own. The second is the traps: names that appear on the website, and often on review sites like G2 or Capterra, but are really features, pricing tiers, AI add-ons or just the brand name. The second list is the point. Without it, a tool that lists every product-sounding name would look accurate.

Examples from the answer sheet:

• Dialpad: Support, Sell and Meetings are products. Plain "Dialpad", Dialpad AI, DialpadGPT and several "enterprise call center" variations are not.
• HubSpot: the seven Hubs are products. Breeze / Breeze AI, Free CRM and "Smart CRM" are not. HubSpot renames its Hubs (CMS Hub became Content Hub in 2024), so this one needs re-checking.
• Zapier: one product. Tables, Interfaces, Canvas and Chatbots are parts of it, sold on a single pricing page.
• Salesloft: one product. Rhythm is an AI feature, not something you buy separately, even though it has review-site listings. That makes it exactly the kind of tricky example the training data needs.
• Loom: Loom is the product. Loom AI is an add-on.

My rule: if you can't buy it on its own, it's not a product, even if a review site has a page for it. That's the mistake I care about most, calling a feature a product, because it corrupts a database record in a way that's expensive to fix.

The two main failures

Calling features products is still the most common mistake. Marketing pages describe Dialpad AI Agents, Zapier Tables, Algolia AI Browse, HubSpot Breeze, Salesloft Rhythm and Loom AI as if they were products, and some have review-site listings. The decision step still falls for them when a page looks like a product page. My answer sheet says otherwise, and this test measures that gap.

Automatic tuning made it worse. I used a tool (DSPy's MIPROv2) to automatically rewrite the instructions for the decision step. The score went down, so I undid it and shipped the untuned version. The README records three causes, to fix before trying again:

• The scoring rule used during tuning took off only 0.25 points for calling a feature a product. My notes say it should be 0.5. The tuning script still uses 0.25 for that mistake.
• The training examples used links instead of the page text the model actually sees when it runs.
• The training examples didn't include tricky "not a product" cases that also have review-site listings, like Salesloft Rhythm.

I've since changed the tuning script to build its examples from real page text. I haven't published a new tuned version yet. The results above are the untuned version.

Earlier that day: search was failing, and the test kept scoring anyway

Before this run, every call to the search service (Serper) failed with "Not enough credits". With no search results, the tool still read the websites and ran the models, and five companies were scored as complete misses. Those numbers aren't published. The test now checks the search account has credit before it starts, and stops if it doesn't, so an outage partway through can't drag the score down. The tool itself still treats a failed search as "no results" and carries on. That's the gap still left to fix.

Check it yourself

The answer sheet, the tool's frozen output and python3 rescore.py are in irfanalidv/stacksift-eval. The results are in eval_results.json. You don't need any API keys. The tool itself stays private.