You're pledging to donate if the project hits its minimum goal and gets approved. If not, your funds will be returned.
DriftBench is an open-source, fully deterministic benchmark (Apache 2.0) for how faithfully an AI tracks a user's changing beliefs across a long conversation — scored reproducibly, with no LLM judge. This grant ships v1.1: a new deterministic "ambivalence" metric (working prototype already in-repo) and a 3x larger, multi-domain scenario library.
DriftBench measures — deterministically, with no LLM judge — whether an AI correctly tracks how a user's beliefs, conflicts, and identity change across a long conversation.
Why it matters: systems that track a person's beliefs are increasingly deployed in high-stakes settings — mental-health support, coaching, advice — where misreading how someone's state is changing can cause real harm. Deterministic, reproducible evaluation lets anyone check how well a system actually does this, without trusting a vendor's own claims.
This funding covers one concrete, shippable bundle:
- Promote the working "ambivalence" (AMB) metric from prototype to a calibrated, tested metric in the core. AMB scores when an AI recognizes a person holds two conflicting beliefs at once (e.g. loyalty to the past and openness to the new) rather than treating it as a simple switch — a distinction existing direction-only metrics miss. Prototype already runs at research/v1_1_draft/ambivalence.py.
- Expand the scenario library from 7 to 20+, across new domains: relationships, health, money & risk, grief & loss, addiction & recovery (today's are all career/identity).
- Ship it: tagged v1.1 release, updated specification, and a public leaderboard.
All deliverables are verifiable in the open repository as they land.
Solo developer time over ~3 months to design domain ontologies, author and validate 13+ new scenarios, calibrate and test the AMB metric, and cut the v1.1 release, plus a small amount for LLM API costs. Based in Ukraine, where costs are well below US/EU levels, so the funding goes far. At the $5,000 minimum I'd ship the AMB metric plus ~5 new-domain scenarios; the full $15,000 funds the complete 20+ scenario library and release.
I'm the sole creator of DriftBench. To be upfront: I'm not a professional programmer — I designed the system (ontology, metrics, validation rules, scenarios) and built it by directing AI coding tools (mainly Claude) as my implementer, while I act as architect and tester. The deterministic, fully test-covered design is what makes this work: correctness is verifiable against fixed expected outputs, not dependent on my hand-coding.
Already shipped, open-source (Apache 2.0): deterministic scoring engine, five metrics, frozen ontology, seven scenarios, zero-trust validation with cryptographic integrity checks, adapter architecture, CI. The AMB prototype and a full new-domain (grief) scenario already run in-repo. https://github.com/simon9679/driftbench
The main risk is calibration: ambivalence thresholds need tuning so AMB is neither trivially easy nor impossible across diverse scenarios. This is engineering, not open research — the metric is fully deterministic and already runs; the work is validating thresholds against a broader library. Scenario authoring is labor but low-risk.
None raised yet. Applications pending: Long-Term Future Fund ($20k, for a separate adversarial safety-evaluation track) and Anthropic Claude for OSS (non-cash).
No offers yet — be the first to back this project.
You're pledging to donate if the project hits its minimum and gets approved. If not, your funds are returned.
No comments yet. Sign in to create one!