Loading...
Loading...
Open evaluation framework
An open methodology for benchmarking AI tools used by commercial real-estate brokerages. Tasks defined in code, graded by named domain experts, leaderboard open to any tool that wants to be benchmarked.
Vendor-self-reported accuracy claims have no value at exam time. The only useful answer to “how good is this AI at fiduciary work?” is a public, reproducible eval framework where the tasks are defined in code, the graders are named humans with credentials, and the leaderboard is open to any tool that wants to be benchmarked.
FiduciaryBench is built in the spirit of Harvey’s BigLaw Bench — and open-sourced for the same reason. The CRE brokerage compliance work isn’t legal work; the tasks need to be different. But the architecture — task corpus in version control, two rubrics (Answer Quality + Source Reliability), human graders, public methodology — translates directly.
Did the model produce the right substantive answer? Numeric correctness, regulatory accuracy, plain-English clarity. Scored 0.0–1.0.
Are the cited sources real, on-point, and verifiable? Did it cite the controlling rule, or a tangential one? Did it invent citations? Scored 0.0–1.0.
Auto-grading runs against reference answers where the task admits it (deterministic math, must-cite-authority checks). Human grading is the score that matters. Auto-grades anchor the iteration loop; humans set the standard.
Tasks are defined in lib/eval/fiduciary-bench/tasks/ in the source repository. The corpus is intentionally small at v1 — better tight and accurate than broad and noisy. Expansion follows real-firm pilot feedback.
fair_housing_marketing · v1.0.0Tests whether the AI flags Fair Housing Act protected-class violations in listing descriptions, distinguishing actual violations from innocuous neighborhood descriptions. Realtor-side — Dante's compliance core.
perfect_for_familiesFLAG: 'Perfect for families' is the textbook familial-status violation under 42 U.S.C. § 3604(c). Suggested replacement: describe the property's features ('4 bedrooms, walk-up attic, main-floor master') without prescribing the buyer demographic. 'Quiet neighborhood' and 'top-rated schools' are generally OK as factual descriptors, though steering risk if used in conjunction with other signals.
great_for_singlesFLAG: 'Great for singles' implies preference based on familial status. Replacement: 'compact 1-bedroom layout, no exterior maintenance, walking distance to entertainment district.' Describe the property, not the intended buyer demographic.
The auto-grader does what it can — exact-amount-within-tolerance for math tasks, must-cite-authority for source-checks. It runs on every run for free.
Human grading is what makes the framework defensible. We hire retired CFPs, ex-CCOs, and former examiners. Grader profiles include name, credentials (CFP®, ChFC®, Series 65/66, JD/ LLM in Tax, retired CCO of named firms), years of experience, and hourly rate.
The first cohort of FiduciaryBench graders will be named on this page once their profiles are live. If you’re a retired CFP with regulatory or examination experience and want to grade, email driftaillc@gmail.com.
Tasks are defined in TypeScript and live in the Drift repo today; they will move to a standalone fiduciary-bench/ repo on GitHub once the v1 corpus stabilizes. Pull requests for new tasks are welcome — each task should measure something the community cares about benchmarking, not something specific to any one vendor.
innocuous_no_flagNO FLAG. Pure factual description, no protected-class signaling. The eval catches whether the model over-flags innocuous descriptions — false positives erode trust just as much as false negatives.
religious_communityFLAG: 'Christian community' is a religious-preference signal. 'Family-friendly' is familial status. The geographic 'close to St. Mary's' is generally OK as a landmark reference; the issue is the explicit demographic framing. Replacement: '5-bedroom on a quiet cul-de-sac' — drop the demographic framing entirely.
zoning_compliance · v1.0.0Tests whether the AI correctly identifies zoning restrictions, flags non-conforming uses, and recommends appropriate next steps (variance, conditional use permit, rezoning) for proposed CRE tenant uses.
restaurant_in_officeFLAG: Restaurant is not a permitted use in B-4 General Office. Client needs a conditional use permit (CUP) or must seek rezoning to a mixed-use or commercial district. The AI should explain: (1) a CUP hearing process, (2) that bar/liquor license adds a separate layer of approval, (3) that parking requirements for restaurant use are typically higher than office, (4) recommend consulting the local zoning board. Should NOT tell the client to proceed without approval.
warehouse_in_industrialNO FLAG. Warehousing and distribution are explicitly permitted in M-1 Light Manufacturing. The AI should confirm the use is permitted, note any potential concerns (truck traffic, loading dock requirements, hours of operation restrictions), and recommend verifying fire code compliance for the specific use case. Should NOT over-flag this as a problem.
daycare_in_retailCONDITIONAL: Childcare is not outright prohibited but requires CUP approval. The AI should explain: (1) conditional use process, (2) state licensing requirements for childcare facilities (fire, health, building code), (3) parking and drop-off requirements, (4) that the zoning board will evaluate compatibility with surrounding uses. Should note that the retail-to-childcare conversion may trigger ADA and building code upgrades.
nonconforming_useNUANCED: The existing auto repair is a legal non-conforming use. Key points: (1) non-conforming use rights generally transfer with the property, not the owner, but some jurisdictions limit this, (2) the use cannot be expanded or intensified, (3) if abandoned for a statutory period (often 12-24 months), non-conforming rights may be lost, (4) damage/destruction beyond a threshold (often 50-75% of value) may terminate the non-conforming right. Should recommend reviewing local zoning code for exact abandonment and destruction thresholds.
lease_clause_risk_review · v1.0.0Tests whether the AI correctly identifies risky, unusual, or tenant-unfavorable lease clauses that a CRE broker should flag for legal review. Covers personal guarantees, recapture, co-tenancy, radius restrictions, and demolition clauses.
personal_guarantee_unlimitedFLAG: This is an unlimited, irrevocable personal guarantee with no burnoff provision. Key risks: (1) survives lease termination, (2) no cap on liability, (3) landlord can pursue guarantor without first attempting collection from tenant entity, (4) no time limitation or reduction schedule. Tenant should negotiate: a burnoff clause (guarantee reduces after X months of on-time payment), a cap (e.g. 12 months base rent), and a right for landlord to exhaust remedies against the tenant entity first. This is a standard but aggressive provision.
landlord_recaptureFLAG: Recapture clause allows landlord to effectively terminate the lease if tenant needs to downsize. Risks: (1) tenant loses flexibility to sublease during downturns, (2) the 50% threshold means any significant sublease triggers full recapture, (3) tenant may be stuck paying rent on space it cannot use or sublease. Should negotiate: removal of the recapture right, or limit it to only apply if tenant proposes to sublease 100% of premises, or require landlord to pay a recapture fee (e.g. unamortized TI and leasing costs).
reasonable_cam_provisionNO MAJOR FLAG. This is a well-drafted CAM provision with typical tenant protections: (1) controllable expense cap at 5%/year, (2) capital expense exclusion for long-life items, (3) annual reconciliation requirement, (4) tenant audit right. Minor items to verify: definition of 'controllable' vs. 'uncontrollable' expenses, whether management fee is included in the cap, and whether the base year or expense stop is appropriate for current market.
demolition_clauseFLAG: Demolition clause gives landlord unilateral right to terminate. Risks: (1) 'sole discretion' means no objective trigger required, (2) tenant could be forced out mid-term with only 12 months notice, (3) only reimburses TI — does not cover tenant's moving costs, lost business, or leasehold value, (4) no relocation assistance obligation. Tenant should negotiate: longer notice period (24+ months), relocation assistance, early termination fee to landlord, right to relocate to comparable space in landlord's portfolio, and reimbursement of moving costs.