Results & maturity

Two separate things to be honest about: which use cases work today, and how good the compliance engine’s answers are on the latest audit.

Feature status

The production test checked features (FRs) across three areas. Headline status:

Area	Available	Partial	Not available
Framework Compliance & Gap Analysis	FR2, FR4	FR5, FR7	FR3, FR6, FR7-ITSCM
Local Health & Safety	FR11	FR14	—
SAF Assessment & Evidence	FR15, FR16, FR18, FR19, FR20, FR21, FR22	—	—

“Available” means the capability exists and the tester reached the screen — not that a full end-to-end run was demonstrated. The test org had no adopted regulations and no answered controls, and the SAF workspace was not accessible for the test user, so several SAF items were confirmed present but not exercised live. See All use cases for the per-FR detail.

Answer-quality benchmark (2026-08-13)

The real audit (audit_compliance_service.mts all --compact) scored five compliance services on an 8-dimension review rubric, 0–16, against NYDFS/DORA and two internal policies (Access Control, Authentication).

Service	Score	Verdict
evidence_check	13/16	Useful as a review draft — strongest
gap_analysis	8/16	Fail — treats missing selected evidence as a verified gap
draft_response	8/16	Fail — includes irrelevant/invented content
compare	7/16	Fail — misses the central MFA provision
requirement_analysis	7/16	Fail — omits Section 500.12 MFA analysis

Caution

The honest headline. In this audit, none of the five sampled outputs passed the production-acceptance gate. evidence_check is the strongest and is useful as a reviewed draft; the others have material omissions or unsupported inference. The engine now executes through its intended routes and returns source manifests — a real improvement — but that is execution, not certified answer correctness.

Note

On “coverage %”. The harness also reports a dimensionCoveragePct (often 100%). That only detects whether expected headings/columns appear — it is not a quality or completeness score, and is not shown here as one.

Caution

Source honesty. These benchmarks ran against seeded regulation records (NYDFS, DORA, and GDPR/“GD” for cross-reg analytics) plus tagged internal policy PDFs — not a full set of AXA production documents. SAF (FR15–FR22) draws on internal Document Hub evidence; there is no live SharePoint/OneDrive connector.