Building a benchmark for professional relationship decisions

TLDR: noticed consistently beat flagship models at 1/11 the cost and achieved 3x the performance of comparably priced models. View the results.

This post is a work in progress. We’re continuing to run benchmarks and will update the results as testing progresses.

A founder’s time is limited. When they trust an AI assistant to help plan a fundraising week, it must answer a deceptively difficult question: “who deserves my attention now?”

A CRM stores a customer’s identity, related tasks and their deadlines. Meeting transcripts record a point-in-time state of a relationship and a set of next steps. A calendar event can show that the follow-up has already been arranged and a one-line email can completely change everything up to the point when it was received. Correctly answering the founder’s query depends on reconciling these records to first define, if nothing else, what is a relevant person in the current context.

This technical write-up describes how we built noticed’s first benchmark for evaluating an AI agent’s ability to plan next steps in founder-led fundraising and sales. We tested whether noticed could turn fragmented relationship evidence into better next-step recommendations than a general-purpose AI agent using a set of external MCP integrations. Across 1,080 answers from twelve systems, noticed led with an average rubric score of 97.08, ahead of every tested model from OpenAI, Anthropic, Google, xAI and Moonshot. It cost about 1.8 cents per answer, compared with 9.7–47.7 cents for the tested flagship models. GPT-5 Mini remained cheaper at about one cent, but scored 31.25.

Generating synthetic data

We’re currently working with our first research partners. Building this benchmark is the first step in a plan to design and build noticed’s proprietary AI harness and models to cover a set of engineering challenges such as data enrichment, identity matching, aggregation and generalized inference. These models will be trained on our proprietary data and customer interactions recorded by our system.

For the first version of our benchmark, we built two fully simulated business environments as development examples, using synthetic data based on life-like scenarios. Kilnbeam has two founders building quoting and capacity-planning software for precision manufacturers. It is raising €900,000 with three unpaid design partners. Docklane has two founders, two operators and five paying customers for its freight-exception software. It needs to refine its ideal customer profile (ICP) and find five more customers. The two worlds contrast an early fundraising case built around unpaid design-partner evidence with a sales case built around a small base of paying customers.

Each team received five tasks, covering selection, introductions, meeting preparation, follow-through and coordination. These tasks require decisions across relationships: who fits, who can introduce them, what is already known, what needs to be done, and who should act. The two worlds cover founder-led fundraising and customer acquisition; they do not test hiring, broader networking or relationship outcomes. These are development cases; any similarity to actual persons, living or dead, or actual events, is purely coincidental.

We authored both sides of each world, including investor mandates, customer adoption problems, buying constraints and introduction conditions. The shared provider fixtures contain 65 people records, 63 company records, 34 meetings, 19 CRM notes, 25 deals and 12 tasks. The additional fixtures contain 70 emails, 43 WhatsApp messages and 35 calendar events, plus 65 LinkedIn profiles and 65 connection records for the same people. The fixtures were generated by a deterministic Python authoring script, with fixed fictional records and stable UUIDv5 identifiers; it makes no model or network calls and uses no customer data; the records are fictional rather than anonymized customer records. The freeze checks verify rubric evidence references, meeting IDs and owners, task coverage and 100-point totals. Dates and conflicting records are authored explicitly; these checks do not establish a complete chronology audit or independent human review.

Aside from the synthetic datasets, added contextual challenges were set up to mimic real-world situations where interest is not a final decision, a customer result is not necessarily approved sales material and a newer message may change a deadline while leaving the original assignee and obligation intact. Time has also been added as part of this simulation and the evaluation clock was fixed at September 8, 2026, 09:00 UTC.

Here’s a request sample and some of the synthetic data that was generated for this scenario:

“What fundraising follow-ups should we do in the next seven days, and which should we postpone? Give owners and draft the two most useful messages.”

One investor, Sofia Malik at Cedarline, has an open follow-up task. These three records determine what should happen. The expected action is to prepare the update and defer contact until September 16. The email changes the contact date; it does not cancel the work or change its owner.

Besides the final responses, evaluation records retain the tool calls and records returned to the model: speakers, task assignees, linked records, deadlines, completion flags and attendee responses, all referenced by source ID during the grading stage of the benchmark.

Benchmark design

We run the same tasks through two harnesses: the comparison harness connects a general-purpose model to locally simulated Granola and Attio MCP tools; the noticed harness runs noticed locally, using its own instructions and native tools over imported relationship data. The comparison harness was used to test model candidates while retaining the task prompts, tool definitions, source fixtures and judging criteria. noticed imports the same Granola and Attio records, plus Gmail, WhatsApp, Google Calendar and LinkedIn data, and retrieves them through its own relationship history and profile tools.

This setup allows us to compare the resulting workflows: the comparison harness can read full transcripts and CRM notes whereas noticed receives imported summaries, metadata and derived facts, including evidence unavailable through the baseline’s two integrations. Both harnesses receive the same user task and neither answering model receives the private judging criteria. The simulations expose meeting lists, summaries and transcripts, CRM records and notes, tasks and workspace members.

noticed’s source-fact extraction setup is separate from answer generation and excluded from its cost; answer-time language helpers are included. These differences mean this is a comparison of complete harnesses—the combined effects of source access, representation, model and orchestration—rather than a model-only leaderboard.

The ten cases below cover Kilnbeam’s fundraising work (K1–K5) and Docklane’s customer acquisition work (D1–D5). The criteria summarize the original source-backed rubrics; each case is scored out of 100, with critical violations tracked separately.

Scoring and judging

Each system completed three benchmark runs, with three answers per scenario in each run: 90 answers per system and 1,080 across twelve systems. The task prompts, source fixtures and judging criteria were held constant. The judge was a language model run against the same fixed prompt and private rubric for every answer; it also received the answer, retrieval trace, team description, source-access description and complete source corpus. It returned an award, rationale, evidence IDs and failure classification for every criterion. Code checks that each expected criterion appears exactly once and receives zero, half or full credit. The judge distinguishes inaccessible sources, retrieval failures, reasoning failures and incomplete answers. Inaccessible evidence can still cost points under the unchanged 100-point rubric; there is no adjustment for each system’s attainable score. We have not reported a per-system failure-type breakdown here, so the overall scores combine source coverage with retrieval and reasoning quality. Missing evidence alone is not treated as fabrication.

For example, 20 of the fundraising query's 100 points cover timing and completed work: “defer Sofia until September 16”, “respect another investor's review date”, and “recognize an already-sent deck”. The remaining criteria cover outstanding commitments, owners, pauses, two useful drafts, unresolved work and dated evidence.

Critical violations remain separate from the numeric total. A high score cannot make contact during an explicit no-contact window acceptable. Missing judge outputs and setup failures stay incomplete rather than receiving invented zero scores. Failed provider attempts were retried, and all 1,080 reported answers have complete judgments from the same judge model.

scenario_score = median(repeat_1, repeat_2, repeat_3)
overall_score  = median(the 10 scenario_scores)

For example, noticed scored 91.25, 100 and 100 in its three runs, averaging 97.08. A run can score 100 under this median-based formula while still containing lower-scoring answers. For every system, the reported score is the mean of its three run scores. Generation cost is summed across those runs and divided by 90 for cost per answer. This is a rubric score, not a percentage of successful tasks or a measured business outcome.

Results

Across the twelve systems, noticed had the highest average score; GPT-5 Mini had the lowest generation cost.

Table 1. Mean rubric score, generation cost per answer and run-score range for all twelve systems. Each system completed three runs and 90 scored answers.

The highest comparator average is Grok 4.5 at 88.33, 8.75 points below noticed. noticed and GPT-5 Mini form the score-cost Pareto frontier among these reported averages.

Generation cost includes attributed noticed helper calls, but excludes ingestion and extraction setup, judging, development iterations, hosting and subscriptions. It is the marginal cost of answering this suite, not the total cost of operating the product. Costs combine provider-reported charges and token-based estimates. Some historical comparator totals also retain conservative full-call budget reservations for failed calls whose actual charge is unknown; the Grok 4.6 reruns use confirmed settled charges. The published averages preserve those recorded accounting choices, so the cost comparison mixes settled charges, estimates and conservative upper bounds.

Across 90 answers per system, generation cost totaled $1.652 for noticed and $0.879 for GPT-5 Mini. The tested flagship models cost between $8.704 and $42.918 for the same number of answers. Helper calls are attributed to noticed only within each answer’s exclusive execution window.

Across all three runs, generation cost includes answer-model input and output, language-model helpers and embeddings. Output usage includes reasoning tokens. Gateway costs retain provider-reported charging; we have not isolated cache savings.

Answer time runs from local answer orchestration to the final response, including model execution, tools, helpers and any recovery. Provider routes and execution schedules varied across the runs; cache warmth was not controlled. The updated latency medians cover all 90 answers per system. Setup and judging are excluded, and no amortized setup-cost analysis is included. Retry costs and uncertain-call allowances are included where captured by each run’s cost record; they are not broken out separately here.

noticed outscored all ten tested flagship models at roughly one-eleventh of their average generation cost. Grok 4.5 came closest at 88.33 versus 97.08, while costing six times as much per answer. noticed achieved 3.11 times Mini's average score at 1.88 times its cost.

Across three runs, noticed's largest lead over the strongest alternative was re-engagement timing: 96.67 versus 73.33 for GPT-5.5 and Sonnet 4.6. Fundraising follow-through came next: 100 versus 80 for Grok 4.5. Using mean scenario scores, noticed led outright in seven tasks and tied Grok 4.6 on next-five-prospects. Both Grok models led ICP refinement, scoring 100 versus noticed's 92.50. Ten alternatives outscored noticed on introductions.

Across all three runs, noticed's repeated answers had the smallest average score spread of all twelve systems. For each scenario, we measured the gap between the highest and lowest of its three scores, then averaged those gaps across ten scenarios and three runs: 9.92 points for noticed, compared with 12.33-32.00 across the eleven alternatives. Sonnet 4.6 came closest at 12.33, followed by Opus 4.8 at 14.25 and Grok 4.5 at 14.75; Gemini had the widest spread at 32.00. Judge variation may contribute to the spread, and scores near the 100-point ceiling have less room to vary upward. As for latency, noticed was slower than ten of the eleven alternatives: its median was 82.30 seconds, compared with 20.94-64.12 seconds for those models. Kimi K3 was fastest; Grok 4.5 took 64.12 seconds. Only Sonnet 4.6 was slower than noticed, at 91.74 seconds. These descriptive results from ten tasks across two synthetic worlds do not yet establish statistical significance.

Next steps

Early results in this benchmark suggest the noticed harness scores higher on the given tasks than the averages of the other, more expensive tested models in a general-purpose harness. Nevertheless, the tasks are limited in scope and the testing is limited in volume and model candidates. This technical write-up is a WIP and will continue to be updated as more testing is conducted. It remains relevant to find better answers for these questions:

• Same model, different harnesses: how much comes from the system?
• Same source access, different retrieval and representation: how much comes from assembling context?
• Unseen worlds and an independent judge: does the advantage generalize?

The research team at noticed is also interested in individually benchmarking the several components that make up our system, namely our data enrichment and identity matching solutions, against other alternatives on the market. Testing each of these individually will provide a more complete picture of whether, where and why our system outperforms alternatives. It will also highlight the path for improvement of generalized benchmarks.

It’s our belief that by working with more research partners and creating better synthetic data from the experiences we learn from them, we’ll be able to create better, more demanding benchmarks that test whether noticed can excel at building professional relationships.