#4985 Decision Model Applying Multi-Question list and Uploaded Document
#4985 Decision Model Applying Multi-Question list and Uploaded Document
#4985 Can I upload a list of questions and a document, the use Google Decision model to answer all questions concerning the document as compliant, partially-compliant and non-compliant, with certainty for each?
Yes—you can do this directly using Gemini Decision 1 (gemini-decision-1) via Google's Decisions API (available in Google AI Studio and the Gemini Developer API).
How It Works for Your Workflow
Document Upload & Limits:
You pass your document (such as a PDF, text, or images) as a shared input.
PDFs up to 50 MB can be sent inline, or larger/reusable files can be uploaded via the Files API. The total context window is 262,144 tokens (~256K), covering both your document and all questions combined.
The document is only processed and billed once per request ($0.10 per 1M input tokens, with zero output tokens generated).
Evaluating Up to 1,000 Questions at Once:
You can include 1 to 1,000 independent questions in a single API call.
Because questions are scored in parallel in a single forward pass, latency remains nearly flat whether you ask 1 question or 100. For your three labels, you use the Single-Select (
choice) question type (or an orderedscorerubric) and definecompliant,partially_compliant, andnon_compliantas the fixed options along with criteria describing when each applies.
Certainty & Probabilities:
For every question, the model returns the top selected option, the calibrated probability for each individual option (summing to 1.0), and an overall confidence score (0 to 1).
Because the model is specifically calibrated for low Expected Calibration Error (ECE), you can set probability thresholds in your code to auto-approve high-certainty results and flag low-certainty items for human review.
Important Caveats to Keep in Mind
Zero Text Explanations or Citations: Gemini Decision 1 produces zero output tokens—it cannot write a rationale, quote the document, or tell you which page/clause triggered a
non-compliantorpartially-compliantverdict.If you need audit trails or page citations, the standard pattern is to run gemini-decision-1first as a fast triage filter, and then pass only the flagged/non-compliant questions to a generative model (like Gemini Pro or Flash) to extract quotes and explanations.Programmatic Setup: While you can test this manually in the Google AI Studio Decisions Playground, uploading a list of questions and batch-processing them requires calling the Decisions API via the Google GenAI SDK (Python/Node.js) rather than a built-in end-user compliance app.
Data Residency & Privacy: As of its November 2026 release, it runs on the Gemini Developer API (Google Cloud / Vertex AI support is marked as coming soon) without EU data residency, though paid-tier data is not used for model training.
Explain how to combine Gemini Decision 1 with a generative Gemini model so flagged non-compliant items also get page citations and written explanations.
To get fast, low-cost bulk scoring alongside detailed audit trails, combine Gemini Decision 1 (gemini-decision-1) as a Stage 1 triage filter with a generative model (such as Gemini 2.5 Flash or Gemini 2.5 Pro) as a Stage 2 evidence extractor.
Because both APIs share the same Gemini Files API, you only upload the PDF once and pass its URI to both models without re-uploading or running separate OCR.
Architecture Overview
Upload Once (Files API): Upload the document once to get a persistent
file_uri. Bothgemini-decision-1and generative Gemini models natively read PDF page layouts, tables, and text from that URI.Stage 1 — Bulk Triage (
gemini-decision-1): Send all compliance questions (up to 1,000) in a singleclient.decisions.createcall.It evaluates every question in parallel (~150–250 ms) with zero output tokens and returns calibrated probabilities. Deterministic Routing (Your Code): Filter the results in Python using the returned verdict and
confidencescore:High-confidence
compliant: Auto-approve immediately—no generative tokens spent.non_compliant,partially_compliant, or low-confidencecompliant: Collect these question IDs into aflagged_itemslist for Stage 2.
Stage 2 — Evidence & Citations (Generative Gemini): Send only the
flagged_itemsplus the samefile_urito a generative model using Structured Outputs (response_schema). Prompt it to locate the exact page numbers, verbatim quotes, and remediation gaps for those specific items.
Python Implementation (google-genai >= 3.2.0)
import os
import json
from pydantic import BaseModel, Field
from google import genai
from google.genai import types
# Decisions API requires v1beta and google-genai >= 3.2.0
client = genai.Client(
api_key=os.environ["GEMINI_API_KEY"],
http_options={"api_version": "v1beta"},
)
# 1. Upload the document ONCE via the Files API
uploaded_doc = client.files.upload(file="vendor_contract.pdf")
# Define your compliance checklist
compliance_rules = {
"data_encryption": "Does the contract require AES-256 encryption at rest and TLS 1.2+ in transit?",
"breach_notification": "Must the vendor notify the client within 48 hours of a confirmed security breach?",
"subprocessor_approval": "Is prior written consent required before onboarding new subprocessors?",
"audit_rights": "Does the client retain the right to conduct an annual independent security audit?",
}
# Build the Decisions API question map
decision_questions = {
rule_id: {
"type": "choice",
"instructions": prompt,
"criteria": {
"compliant": "All requirements in the question are explicitly met with no weakening caveats.",
"partially_compliant": "Addressed in part, weaker thresholds (e.g., 72h instead of 48h), or missing sub-requirements.",
"non_compliant": "Omitted entirely or explicitly contradicted by the document.",
},
}
for rule_id, prompt in compliance_rules.items()
}
# 2. STAGE 1: Fast, zero-output-token scoring across all rules
decision_response = client.decisions.create(
model="gemini-decision-1",
input=[
{
"type": "document",
"uri": uploaded_doc.uri,
"mime_type": "application/pdf",
}
],
questions=decision_questions,
)
# 3. ROUTE: Separate auto-approved items from items needing citations
final_report = {}
flagged_for_explanation = []
CONFIDENCE_THRESHOLD = 0.85
for rule_id, ans in decision_response.answers.items():
verdict = ans.choice
confidence = ans.confidence
final_report[rule_id] = {
"verdict": verdict,
"confidence": round(confidence, 4),
"probabilities": ans.probabilities,
"page_numbers": [],
"verbatim_quote": None,
"explanation": "Auto-approved based on high-confidence compliance score.",
}
# Escalate non-compliant, partially compliant, OR uncertain compliant items
if verdict != "compliant" or confidence < CONFIDENCE_THRESHOLD:
flagged_for_explanation.append({
"rule_id": rule_id,
"question": compliance_rules[rule_id],
"initial_verdict": verdict,
"confidence": round(confidence, 4),
})
# 4. STAGE 2: Generate page citations & explanations ONLY for flagged items
class FindingDetail(BaseModel):
rule_id: str
page_numbers: list[int] = Field(
description="1-based PDF page numbers where relevant clauses appear, or empty if omitted."
)
verbatim_quote: str = Field(
description="Exact quote from the document supporting the finding, or 'Clause omitted' if absent."
)
explanation: str = Field(
description="Concise explanation of why the document is non-compliant, partially compliant, or ambiguous."
)
class AuditFindings(BaseModel):
findings: list[FindingDetail]
if flagged_for_explanation:
prompt = f"""
Review the attached PDF for the following flagged compliance items.
For each item, locate the exact 1-based PDF page number(s), extract the relevant verbatim quote,
and explain the specific compliance gap or ambiguity.
Flagged items:
{json.dumps(flagged_for_explanation, indent=2)}
"""
gen_response = client.models.generate_content(
model="gemini-2.5-flash", # Use gemini-2.5-pro for dense legal reasoning
contents=[uploaded_doc, prompt],
config=types.GenerateContentConfig(
response_mime_type="application/json",
response_schema=AuditFindings,
temperature=0.0,
),
)
parsed = AuditFindings.model_validate_json(gen_response.text)
for finding in parsed.findings:
if finding.rule_id in final_report:
final_report[finding.rule_id].update({
"page_numbers": finding.page_numbers,
"verbatim_quote": finding.verbatim_quote,
"explanation": finding.explanation,
})
# Clean up the uploaded file when done
client.files.delete(name=uploaded_doc.name)
print(json.dumps(final_report, indent=2))
Key Design Tips for Production
Escalate Low-Confidence
compliantVerdicts Too: Notice the routing checkif verdict != "compliant" or confidence < CONFIDENCE_THRESHOLD:. Ifgemini-decision-1sayscompliantwith only0.58certainty, sending it to Stage 2 lets a reasoning model inspect the edge case and surface the ambiguity for a human reviewer.Batch Stage 2 vs. Per-Item Calls: Passing all flagged items into a single Stage 2 call (as shown above) avoids paying input token costs for the PDF multiple times. However, if a document has more than 15–20 flagged items, chunk them into batches of 5–10 per generative call so the model maintains high precision on page numbers and verbatim quotes.
Explicit Page-Number Prompting: Generative Gemini models natively see visual page boundaries in uploaded PDFs. Explicitly requesting "1-based PDF page numbers" (rather than printed footer numbers, which often differ due to cover pages) keeps citations aligned with standard PDF viewers.
What are the best practices for writing choice criteria and calibrating confidence thresholds in Gemini Decision 1 for legal and policy audits?
Because gemini-decision-1 scores questions in a single forward pass with zero output tokens (no chain-of-thought scratchpad), its accuracy and probability calibration depend heavily on how clearly your questions and criteria define the decision boundaries.
1. Writing Choice Criteria for Legal & Policy Audits
Decompose Compound Rules into Atomic Questions: Because
gemini-decision-1evaluates all questions in parallel without generating intermediate reasoning text, avoid multi-part questions like "Does the vendor encrypt data at rest, notify within 48 hours, and maintain SOC 2 Type II?" Split those into three separate atomic questions (up to 1,000 per call) and compose the overall section status in your code.Define Mutually Exclusive, Observable Boundaries: Avoid subjective words like "adequate," "reasonable," or "satisfactory" in your criteria unless those exact legal terms are what you are testing for. Anchor each option to concrete textual evidence:
compliant: "All mandatory elements of the requirement are explicitly stated in the text without carve-outs, delays, or discretionary qualifiers (e.g., 'may' instead of 'shall')."partially_compliant: "The topic is addressed, but with weaker thresholds (e.g., 72 hours instead of 48), missing sub-elements, or conditional exceptions."non_compliant: "The requirement is completely omitted from the document, or a clause explicitly contradicts or waives it."
Add an Escape Hatch (
not_applicable/insufficient_context) When Needed: In achoicequestion, probabilities are forced to sum to1.0across your defined options.If a rule only applies conditionally (e.g., when personal health information is processed), forcing a 3-way choice on a non-PHI contract creates false confidence. Either add a 4th option ( not_applicable: "The document's scope does not trigger this requirement") or pair the compliance question with a separate binary (noul) applicability gate.Avoid Positional Keys: The API sorts
choicekeys alphabetically during evaluation. Use descriptive semantic keys (compliant,partially_compliant,non_compliant,not_applicable) rather than positional labels likeoption_aornone_of_the_above.Consider
scoreOverchoicefor Ordinal Severity: Whilechoiceworks well for categorical buckets,gemini-decision-1also offers ascore(rubric) type for ordered scales (e.g.,0= Non-Compliant,1= Partially Compliant,2= Compliant).scorereturns both per-level probabilities and a continuous probability-weighted expected score (e.g.,1.85), which makes ranking borderline clauses easier.
2. Calibrating Confidence Thresholds
In the Decisions API, confidence equals the highest option probability (max(probabilities)). Even though gemini-decision-1 is trained for low Expected Calibration Error (ECE), legal documents have domain-specific ambiguity that requires empirical threshold tuning rather than guessing a single cutoff.
Account for Probability Floors & Ceilings: For binary (
noul) checks, probabilities are bounded between0.01and0.99, so never set an auto-approve threshold above0.99. For 3-waychoicequestions, random chance is0.33, meaning a0.55top probability represents substantial split-probability uncertainty (e.g.,0.55compliant vs.0.42partially compliant).Use Asymmetric Risk-Based Thresholds: Legal audits carry asymmetric error costs—a false-negative
compliantverdict (missing a real compliance gap) is far more costly than a false-positivenon_compliantflag that gets verified by Stage 2 or a human. Set different thresholds per verdict:Auto-approve
compliant: Require high certainty (e.g.,confidence >= 0.90ANDP(non_compliant) <= 0.03).Escalate
partially_compliant: Always route to Stage 2 / human review, as partial compliance inherently requires nuance and redlining.Auto-flag
non_compliant: Can trigger at a lower threshold (e.g.,confidence >= 0.75) since Stage 2 will immediately verify whether the clause is truly missing or merely phrased differently.
Check the Second-Highest Probability (Margin Check): Relying only on
confidencemisses close ties. A distribution of[0.52 compliant, 0.46 partially_compliant, 0.02 non_compliant]has a0.52confidence, whereas[0.52 compliant, 0.24 partially_compliant, 0.24 non_compliant]has the exact sameconfidencebut very different ambiguity. Add a margin rule: escalate whenevertop_prob - second_prob < 0.25.Calibrate Against a Golden Set (50–100 Annotated Clauses): Before locking thresholds for production:
Run a benchmark batch of 50–100 historical document-rule pairs with known human audit labels.
Bucket predictions by confidence (
0.50–0.60,0.60–0.70, ...,0.90–1.00) and measure your actual precision in each bucket.Set your auto-approval cutoff at the lowest bucket where empirical error rates meet your audit tolerance (for example,
< 2%false compliance rate), and route everything below that threshold to your generative Stage 2 model or human reviewer.
Example Calibration Routing Pattern
probs = ans.probabilities # e.g., {"compliant": 0.88, "partially_compliant": 0.10, "non_compliant": 0.02}
sorted_probs = sorted(probs.values(), reverse=True)
margin = sorted_probs[0] - sorted_probs[1]
if ans.choice == "compliant" and ans.confidence >= 0.90 and margin >= 0.30:
status = "AUTO_APPROVED"
elif ans.choice == "non_compliant" and ans.confidence >= 0.80 and margin >= 0.25:
status = "ESCALATE_STAGE_2_CITATION" # Confirm absence & extract section context
else:
status = "HUMAN_OR_PRO_REVIEW" # Ambiguous, partial, or low-margin edge caseCompare using the choice question type versus the score rubric type in Gemini Decision 1 for multi-level compliance auditing, including Python examples for both.
In Gemini Decision 1 (gemini-decision-1), both choice and score evaluate a document against defined criteria in a single forward pass with zero output tokens, but they model your compliance states differently: choice treats categories as unordered nominal buckets, while score models an ordered ordinal scale and computes a continuous probability-weighted expected score.
For multi-level compliance auditing (non_compliant --> partially_compliant --> compliant), choosing between them comes down to whether you need non-ordinal escape hatches (not_applicable) or continuous numerical ranking and aggregation.
Side-by-Side Comparison
| Dimension | choice (Single-Select) | score (Ordinal Rubric) |
| Mathematical Structure | Nominal / unordered categories (dict of string keys). | Ordinal integer levels (0 to N, e.g., 0, 1, 2). |
| Primary Outputs | ans.choice (winning key), ans.probabilities (per key), ans.confidence (max prob). | ans.score (continuous expected value sum i cdot P(i)), ans.probabilities (per integer level), ans.confidence. |
| Ordering Awareness | None—non_compliant and compliant have the same distance as partially_compliant and compliant. | Built-in ordinal progression—probability mass shifting between 1 and 2 smoothly moves the expected score (e.g., 1.15 vs. 1.88). |
Escape Hatches (N/A) | Supported—you can include not_applicable alongside compliance levels. | Not supported—every level must sit on the same monotonic scale (an N/A level breaks expected-value math). |
| Roll-Up / Section Scoring | Requires manual mapping weights in Python to average across a policy section. | Directly averages across requirements (e.g., a section's compliance index out of 2.00). |
| Best Audit Use Case | Checklists with conditional applicability (not_applicable), distinct legal failure modes, or strict categorical gates. | Uniformly applicable requirements where you want to rank borderline gaps by severity or compute an overall compliance index. |
When to Choose Which
Use
choicewhen:Some requirements in your checklist may be Not Applicable to a given document (e.g., rules triggered only when subcontractors, flight-critical software, or PII are present).
You want to distinguish why a rule failed—for instance, separating
omittedfromcontradictedrather than lumping both into0.
Use
scorewhen:Every requirement in the batch applies to the document and follows a strict progression (
0= Non-Compliant,1= Partially Compliant,2= Compliant).You want a continuous severity signal: a requirement with
[P(0)=0.02, P(1)=0.18, P(2)=0.80]yields an expected score of1.78(minor qualifier), whereas[P(0)=0.45, P(1)=0.45, P(2)=0.10]yields0.65(major gap)—allowing you to sort a 200-item audit report from worst gap (0.02) to cleanest pass (1.98).
Python Examples (google-genai >= 3.2.0)
1. choice Type (With not_applicable Support)
import os
from google import genai
client = genai.Client(
api_key=os.environ["GEMINI_API_KEY"],
http_options={"api_version": "v1beta"},
)
doc = client.files.upload(file="supplier_sdp.pdf")
choice_questions = {
"REQ-01": {
"type": "choice",
"instructions": "Does the plan mandate formal traceability between software requirements and test cases?",
"criteria": {
"compliant": "Explicitly mandates bidirectional traceability across all software levels with no waivers.",
"partially_compliant": "Mentions traceability, but limits scope, omits directionality, or leaves it optional.",
"non_compliant": "Omits traceability requirements entirely or explicitly excludes them.",
"not_applicable": "The document explicitly states no software development or testing is within scope.",
},
}
}
res = client.decisions.create(
model="gemini-decision-1",
input=[{"type": "document", "uri": doc.uri, "mime_type": "application/pdf"}],
questions=choice_questions,
)
ans = res.answers["REQ-01"]
print(f"Verdict: {ans.choice}")
print(f"Confidence: {ans.confidence:.3f}")
print(f"Breakdown: {ans.probabilities}")
# Example output:
# Verdict: partially_compliant
# Confidence: 0.740
# Breakdown: {'compliant': 0.19, 'non_compliant': 0.05, 'not_applicable': 0.02, 'partially_compliant': 0.74}
2. score Rubric Type (Ordinal 0–2 Scale with Continuous Expected Score)
score_questions = {
"REQ-01": {
"type": "score",
"instructions": "Evaluate compliance with mandatory bidirectional requirements-to-test traceability.",
"rubric": {
0: "Non-Compliant: Traceability is omitted entirely or explicitly waived.",
1: "Partially Compliant: Traceability is mentioned, but lacks bidirectional links, full coverage, or mandatory language.",
2: "Compliant: Bidirectional traceability from requirements through test verification is explicitly mandated.",
},
}
}
res = client.decisions.create(
model="gemini-decision-1",
input=[{"type": "document", "uri": doc.uri, "mime_type": "application/pdf"}],
questions=score_questions,
)
ans = res.answers["REQ-01"]
# Map discrete argmax level alongside the continuous expected score
best_level = max(ans.probabilities.items(), key=lambda kv: kv[1])[0]
label_map = {0: "non_compliant", 1: "partially_compliant", 2: "compliant"}
print(f"Expected Score: {ans.score:.3f} / 2.000")
print(f"Discrete Label: {label_map[int(best_level)]} (confidence={ans.confidence:.3f})")
print(f"Level Probs: {ans.probabilities}")
# Example output:
# Expected Score: 1.140 / 2.000
# Discrete Label: partially_compliant (confidence=0.740)
# Level Probs: {0: 0.05, 1: 0.76, 2: 0.19}
Hybrid Best Practice: Gate with noul, Rank with score
If you want both not_applicable filtering and continuous score severity ranking, send two questions per requirement in the same client.decisions.create call (up to 500 rules = 1,000 questions in one pass):
REQ-01_applicable(type: "noul"): Binary check asking whether the requirement applies to the document's scope.REQ-01_compliance(type: "score"):0–2rubric scoring compliance assuming applicability.
In Python, if P(applicable) < 0.50, mark the rule not_applicable; otherwise, use the 0–2 expected score to rank and triage findings.
Write a Python script implementing the hybrid noul applicability gate plus 0-2 score rubric pattern in Gemini Decision 1, storing calibrated probabilities, expected scores, and routing decisions in SQLite.
Below is a complete Python implementation combining a noul applicability gate with a 0–2 score rubric in a single gemini-decision-1 forward pass, validated with Pydantic v2 and persisted to a normalized SQLite3 database.
How the Hybrid Pattern Works
Two Questions per Requirement (One API Call): For every requirement
REQ-ID, the script registers two atomic questions in thequestionsdictionary (supporting up to 500 requirements = 1,000 questions per single-pass call):<REQ-ID>__applicable(type: "noul"): Returns a calibrated applicability probability P(yes) in [0.01, 0.99].<REQ-ID>__score(type: "score"): Evaluates the document on a0–2ordinal rubric (0= Non-Compliant,1= Partially Compliant,2= Compliant), returning per-level probabilities and the continuous probability-weighted expected score E[score] = sum_{i=0}^{2} i cdot P(i).
Deterministic Gating & Routing:
If $P(\text{applicable}) < 0.50$, the requirement is gated as not_applicable (with a margin check so borderline
0.40–0.60applicability cases get flagged for review rather than silently skipped).Otherwise, the continuous expected score (
0.00to2.00), discrete argmax level, confidence, and top-two probability margin route each requirement intoAUTO_APPROVED,ESCALATE_STAGE_2_CITATION, orHUMAN_REVIEW.
Idempotent SQLite Persistence: Uses a composite primary key
(audit_run_id, document_hash, req_id)withON CONFLICT ... DO UPDATEso re-running an audit safely updates records without duplicating rows.
Complete Python Implementation (google-genai >= 3.2.0)
import hashlib
import json
import os
import sqlite3
from datetime import datetime, timezone
from enum import Enum
from pathlib import Path
from typing import Dict, List, Literal, Optional
from google import genai
from pydantic import BaseModel, Field
# ==============================================================================
# 1. Pydantic v2 Domain & Persistence Schemas
# ==============================================================================
class VerdictLabel(str, Enum):
COMPLIANT = "compliant"
PARTIALLY_COMPLIANT = "partially_compliant"
NON_COMPLIANT = "non_compliant"
NOT_APPLICABLE = "not_applicable"
class RouteAction(str, Enum):
AUTO_APPROVED = "AUTO_APPROVED"
NOT_APPLICABLE_SKIP = "NOT_APPLICABLE_SKIP"
ESCALATE_STAGE_2_CITATION = "ESCALATE_STAGE_2_CITATION"
HUMAN_REVIEW = "HUMAN_REVIEW"
class ComplianceRequirement(BaseModel):
req_id: str
title: str
applicability_question: str
applicability_criteria: Dict[Literal["true", "false"], str]
compliance_instruction: str
rubric: Dict[int, str] = Field(
description="Ordered rubric levels starting at 0 (0=non_compliant, 1=partially_compliant, 2=compliant)."
)
class EvaluatedRequirement(BaseModel):
audit_run_id: str
document_name: str
document_hash: str
req_id: str
title: str
# Gate outputs (noul)
p_applicable: float = Field(ge=0.0, le=1.0)
gate_confidence: float = Field(ge=0.0, le=1.0)
is_applicable: bool
# Rubric outputs (score 0..2)
expected_score: Optional[float] = Field(default=None, ge=0.0, le=2.0)
p_level_0: Optional[float] = Field(default=None, ge=0.0, le=1.0)
p_level_1: Optional[float] = Field(default=None, ge=0.0, le=1.0)
p_level_2: Optional[float] = Field(default=None, ge=0.0, le=1.0)
score_confidence: Optional[float] = Field(default=None, ge=0.0, le=1.0)
prob_margin: Optional[float] = Field(default=None, ge=0.0, le=1.0)
# Final synthesized verdict & routing
verdict: VerdictLabel
route_action: RouteAction
evaluated_at: str
# ==============================================================================
# 2. SQLite3 Persistence Layer
# ==============================================================================
class ComplianceAuditStore:
def __init__(self, db_path: str = "compliance_audits.db") -> None:
self.db_path = db_path
self._init_schema()
def _connect(self) -> sqlite3.Connection:
conn = sqlite3.connect(self.db_path)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA journal_mode=WAL;")
conn.execute("PRAGMA foreign_keys=ON;")
return conn
def _init_schema(self) -> None:
with self._connect() as conn:
conn.executescript(
"""
CREATE TABLE IF NOT EXISTS audit_runs (
audit_run_id TEXT PRIMARY KEY,
document_name TEXT NOT NULL,
document_hash TEXT NOT NULL,
model_name TEXT NOT NULL,
input_tokens INTEGER,
created_at TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS requirement_evaluations (
audit_run_id TEXT NOT NULL,
document_hash TEXT NOT NULL,
req_id TEXT NOT NULL,
title TEXT NOT NULL,
p_applicable REAL NOT NULL,
gate_confidence REAL NOT NULL,
is_applicable INTEGER NOT NULL,
expected_score REAL,
p_level_0 REAL,
p_level_1 REAL,
p_level_2 REAL,
score_confidence REAL,
prob_margin REAL,
verdict TEXT NOT NULL,
route_action TEXT NOT NULL,
evaluated_at TEXT NOT NULL,
PRIMARY KEY (audit_run_id, document_hash, req_id),
FOREIGN KEY (audit_run_id) REFERENCES audit_runs(audit_run_id)
);
CREATE INDEX IF NOT EXISTS idx_eval_route
ON requirement_evaluations (audit_run_id, route_action, expected_score);
"""
)
def save_run(
self,
audit_run_id: str,
document_name: str,
document_hash: str,
model_name: str,
input_tokens: Optional[int],
evaluations: List[EvaluatedRequirement],
) -> None:
now = datetime.now(timezone.utc).isoformat()
with self._connect() as conn:
conn.execute(
"""
INSERT INTO audit_runs (audit_run_id, document_name, document_hash, model_name, input_tokens, created_at)
VALUES (?, ?, ?, ?, ?, ?)
ON CONFLICT(audit_run_id) DO UPDATE SET
input_tokens = excluded.input_tokens,
created_at = excluded.created_at;
""",
(audit_run_id, document_name, document_hash, model_name, input_tokens, now),
)
conn.executemany(
"""
INSERT INTO requirement_evaluations (
audit_run_id, document_hash, req_id, title,
p_applicable, gate_confidence, is_applicable,
expected_score, p_level_0, p_level_1, p_level_2,
score_confidence, prob_margin, verdict, route_action, evaluated_at
)
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
ON CONFLICT(audit_run_id, document_hash, req_id) DO UPDATE SET
p_applicable = excluded.p_applicable,
gate_confidence = excluded.gate_confidence,
is_applicable = excluded.is_applicable,
expected_score = excluded.expected_score,
p_level_0 = excluded.p_level_0,
p_level_1 = excluded.p_level_1,
p_level_2 = excluded.p_level_2,
score_confidence = excluded.score_confidence,
prob_margin = excluded.prob_margin,
verdict = excluded.verdict,
route_action = excluded.route_action,
evaluated_at = excluded.evaluated_at;
""",
[
(
e.audit_run_id,
e.document_hash,
e.req_id,
e.title,
e.p_applicable,
e.gate_confidence,
int(e.is_applicable),
e.expected_score,
e.p_level_0,
e.p_level_1,
e.p_level_2,
e.score_confidence,
e.prob_margin,
e.verdict.value,
e.route_action.value,
e.evaluated_at,
)
for e in evaluations
],
)
def fetch_triage_report(self, audit_run_id: str) -> List[dict]:
"""Returns applicable items ordered from lowest expected score (worst gap) to highest."""
with self._connect() as conn:
rows = conn.execute(
"""
SELECT req_id, title, verdict, route_action,
ROUND(p_applicable, 3) AS p_app,
ROUND(expected_score, 3) AS exp_score,
ROUND(score_confidence, 3) AS conf,
ROUND(prob_margin, 3) AS margin
FROM requirement_evaluations
WHERE audit_run_id = ?
ORDER BY is_applicable DESC, expected_score ASC NULLS LAST;
""",
(audit_run_id,),
).fetchall()
return [dict(r) for r in rows]
# ==============================================================================
# 3. Hybrid Evaluator (noul Gate + 0-2 Score Rubric)
# ==============================================================================
def sha256_file(path: Path) -> str:
h = hashlib.sha256()
with path.open("rb") as f:
for chunk in iter(lambda: f.read(65536), b""):
h.update(chunk)
return h.hexdigest()
def route_decision(
p_app: float,
probs: Dict[int, float],
expected_score: float,
score_confidence: float,
) -> tuple[bool, VerdictLabel, RouteAction, float]:
"""Combines applicability probability and 0-2 score distribution into a deterministic route."""
sorted_probs = sorted(probs.values(), reverse=True)
margin = sorted_probs[0] - sorted_probs[1] if len(sorted_probs) > 1 else sorted_probs[0]
# Step A: Check applicability gate (noul bounds are [0.01, 0.99])
if p_app < 0.50:
is_app = False
# If applicability is certain (P(app) <= 0.20), skip; if borderline (0.20..0.50), flag for review
route = RouteAction.NOT_APPLICABLE_SKIP if p_app <= 0.20 else RouteAction.HUMAN_REVIEW
return is_app, VerdictLabel.NOT_APPLICABLE, route, margin
is_app = True
best_level = max(probs.items(), key=lambda kv: kv[1])[0]
# If the applicability gate itself was borderline (0.50..0.70), force human review
if p_app < 0.70:
verdict_map = {
0: VerdictLabel.NON_COMPLIANT,
1: VerdictLabel.PARTIALLY_COMPLIANT,
2: VerdictLabel.COMPLIANT,
}
return is_app, verdict_map[best_level], RouteAction.HUMAN_REVIEW, margin
# Step B: Route applicable items using expected score, argmax level, confidence, and margin
if best_level == 2 and expected_score >= 1.85 and score_confidence >= 0.88 and margin >= 0.30:
return is_app, VerdictLabel.COMPLIANT, RouteAction.AUTO_APPROVED, margin
if best_level == 0 and expected_score <= 0.30 and score_confidence >= 0.75 and margin >= 0.25:
return is_app, VerdictLabel.NON_COMPLIANT, RouteAction.ESCALATE_STAGE_2_CITATION, margin
if best_level == 1:
return is_app, VerdictLabel.PARTIALLY_COMPLIANT, RouteAction.ESCALATE_STAGE_2_CITATION, margin
# Low-margin or ambiguous scores route to human review
fallback_map = {
0: VerdictLabel.NON_COMPLIANT,
1: VerdictLabel.PARTIALLY_COMPLIANT,
2: VerdictLabel.COMPLIANT,
}
return is_app, fallback_map[best_level], RouteAction.HUMAN_REVIEW, margin
def run_hybrid_compliance_audit(
pdf_path: str,
requirements: List[ComplianceRequirement],
audit_run_id: str,
db_path: str = "compliance_audits.db",
) -> List[EvaluatedRequirement]:
if len(requirements) > 500:
raise ValueError("Max 500 requirements (1,000 total questions) per single-pass Decisions API request.")
file_path = Path(pdf_path)
doc_hash = sha256_file(file_path)
client = genai.Client(
api_key=os.environ["GEMINI_API_KEY"],
http_options={"api_version": "v1beta"},
)
# 1. Build paired questions: <req_id>__applicable (noul) + <req_id>__score (0..2 score)
questions_payload: dict = {}
for req in requirements:
questions_payload[f"{req.req_id}__applicable"] = {
"type": "noul",
"instructions": req.applicability_question,
"criteria": req.applicability_criteria,
}
questions_payload[f"{req.req_id}__score"] = {
"type": "score",
"instructions": req.compliance_instruction,
"rubric": req.rubric,
}
# 2. Upload PDF once via Files API and run single-pass evaluation
uploaded_doc = client.files.upload(file=str(file_path))
try:
response = client.decisions.create(
model="gemini-decision-1",
input=[
{
"type": "document",
"uri": uploaded_doc.uri,
"mime_type": "application/pdf",
}
],
questions=questions_payload,
)
finally:
client.files.delete(name=uploaded_doc.name)
# 3. Parse answers, apply gate + routing logic, and validate with Pydantic
now = datetime.now(timezone.utc).isoformat()
evaluations: List[EvaluatedRequirement] = []
for req in requirements:
gate_ans = response.answers[f"{req.req_id}__applicable"]
score_ans = response.answers[f"{req.req_id}__score"]
p_app = float(gate_ans.noul)
gate_conf = float(gate_ans.confidence)
# Normalize rubric level keys to integers {0, 1, 2}
raw_probs = {int(k): float(v) for k, v in score_ans.probabilities.items()}
exp_score = float(score_ans.score)
score_conf = float(score_ans.confidence)
is_app, verdict, route, margin = route_decision(
p_app=p_app,
probs=raw_probs,
expected_score=exp_score,
score_confidence=score_conf,
)
evaluations.append(
EvaluatedRequirement(
audit_run_id=audit_run_id,
document_name=file_path.name,
document_hash=doc_hash,
req_id=req.req_id,
title=req.title,
p_applicable=p_app,
gate_confidence=gate_conf,
is_applicable=is_app,
expected_score=exp_score if is_app else None,
p_level_0=raw_probs.get(0) if is_app else None,
p_level_1=raw_probs.get(1) if is_app else None,
p_level_2=raw_probs.get(2) if is_app else None,
score_confidence=score_conf if is_app else None,
prob_margin=margin if is_app else None,
verdict=verdict,
route_action=route,
evaluated_at=now,
)
)
# 4. Persist to SQLite3
input_tokens = getattr(getattr(response, "usage", None), "input_tokens", None)
store = ComplianceAuditStore(db_path=db_path)
store.save_run(
audit_run_id=audit_run_id,
document_name=file_path.name,
document_hash=doc_hash,
model_name="gemini-decision-1",
input_tokens=input_tokens,
evaluations=evaluations,
)
return evaluations
# ==============================================================================
# 4. Example Usage
# ==============================================================================
if __name__ == "__main__":
sample_requirements = [
ComplianceRequirement(
req_id="SWE-052",
title="Bidirectional Requirements Traceability",
applicability_question="Does the document cover software development, engineering, or verification activities?",
applicability_criteria={
"true": "The document describes software design, coding, integration, or testing.",
"false": "The document is strictly administrative, hardware-only, or out of software scope.",
},
compliance_instruction="Evaluate whether bidirectional traceability between software requirements and test procedures is mandated.",
rubric={
0: "Non-Compliant: Traceability is omitted entirely or explicitly waived.",
1: "Partially Compliant: Traceability is mentioned, but lacks bidirectional links, full coverage, or mandatory language.",
2: "Compliant: Bidirectional traceability from requirements through test verification is explicitly mandated.",
},
),
ComplianceRequirement(
req_id="SWE-134",
title="Safety-Critical Software Hazard Controls",
applicability_question="Does the system described include safety-critical or human-rated software functions?",
applicability_criteria={
"true": "The document references safety-critical functions, hazard controls, or human safety risk.",
"false": "The document explicitly states the software has no safety-critical impact or hazard classification.",
},
compliance_instruction="Evaluate whether software hazard controls and safety verification methods are explicitly defined.",
rubric={
0: "Non-Compliant: Safety-critical hazard controls are missing despite safety-critical scope.",
1: "Partially Compliant: Safety risks are acknowledged, but specific software controls or verification methods are incomplete.",
2: "Compliant: Specific software hazard controls, inhibits, and verification tests are explicitly defined.",
},
),
]
# Run audit against a target PDF
results = run_hybrid_compliance_audit(
pdf_path="provider_sdp.pdf",
requirements=sample_requirements,
audit_run_id="RUN-2026-11-16-001",
)
store = ComplianceAuditStore()
print(json.dumps(store.fetch_triage_report("RUN-2026-11-16-001"), indent=2))
Notable Details in This Design
Dual-Stage Uncertainty Protection: Notice in
route_decision()that ifp_applicablelands in the uncertain middle band (0.20 < p_app < 0.70), the requirement is routed toHUMAN_REVIEWrather than silently markednot_applicableor auto-approved.Nulling Gated Scores in SQLite: When
is_applicableisFalse(P(applicable) < 0.50),expected_scoreandp_level_*are stored asNULLin SQLite. That keepsAVG(expected_score)roll-up queries across a document or section mathematically clean without artificially dragging down the score with0.0s from inapplicable rules.Triage Ordering:
fetch_triage_report()orders applicable findings byexpected_score ASC, placing the highest-severity gaps (0.00–0.40) at the top of your Stage 2 generative citation queue.
Rather than a list of questions, can I provide one questions with multiple bullets, requesting answers for each combo of question and bullet?
No—Gemini Decision 1 cannot return separate verdicts or confidence scores for individual bullets bundled inside a single question. Each entry in the questions dictionary produces exactly one verdict and one probability distribution, and the model generates zero output tokens (so it cannot output a list of per-bullet answers).
However, you can achieve the exact workflow you want with a small helper function in Python: write one shared question template and a list of bullets, and let Python expand them into individual question keys (q_1, q_2, ...) before calling client.decisions.create.
Why Bundling Bullets into One Question Fails
One Output Slot per Question Key: The Decisions API maps each key in your
questionsdictionary to a single scalar answer (choice,score, ornoul). If you pass 1 question containing 10 bullets, you get back 1 blended probability distribution—not 10.No Chain-of-Thought Scratchpad: Because the model emits zero reasoning tokens, asking it to mentally evaluate 10 sub-bullets and aggregate them into a single
partially_compliantverdict degrades accuracy compared to scoring each bullet independently.Zero Cost Penalty for Expanding in Python: Because
gemini-decision-1processes your document once and scores all question keys in parallel in a single forward pass, sending 1 question with 20 bullets expanded into 20 separate keys costs the exact same input tokens for the document and runs in virtually the same latency (~150–250 ms) as sending 1 bundled question.
Pattern: One Shared Template + Bullet List (Expanded Automatically)
Define your base question and criteria once, list your bullets, and use a dictionary comprehension to generate the Cartesian product (question × bullet) for the API:
import os
from google import genai
client = genai.Client(
api_key=os.environ["GEMINI_API_KEY"],
http_options={"api_version": "v1beta"},
)
# 1. Define ONE shared question template and rubric
BASE_INSTRUCTION = (
"Does the Software Development Plan explicitly define roles, procedures, "
"and verification artifacts for the following requirement?"
)
SHARED_CRITERIA = {
"compliant": "Explicitly mandated in the document with defined artifacts and no waivers.",
"partially_compliant": "Mentioned in general terms, or missing defined procedures/verification artifacts.",
"non_compliant": "Completely omitted from the document or explicitly excluded.",
}
# 2. Define your list of bullets
bullets = {
"cm_baseline": "Configuration management and baseline control",
"peer_reviews": "Formal peer inspections of code and design artifacts",
"req_traceability": "Bidirectional requirements-to-test traceability",
"static_analysis": "Automated static code analysis prior to release",
"hazard_mitigation": "Verification of safety-critical software hazard controls",
}
# 3. Expand into individual (template + bullet) question keys in Python
questions_payload = {
bullet_id: {
"type": "choice",
"instructions": f"{BASE_INSTRUCTION}\n\nTarget Item: {bullet_text}",
"criteria": SHARED_CRITERIA,
}
for bullet_id, bullet_text in bullets.items()
}
# 4. Single API call returns a distinct verdict & certainty for EVERY bullet
doc = client.files.upload(file="provider_sdp.pdf")
try:
response = client.decisions.create(
model="gemini-decision-1",
input=[{"type": "document", "uri": doc.uri, "mime_type": "application/pdf"}],
questions=questions_payload,
)
finally:
client.files.delete(name=doc.name)
for bullet_id, ans in response.answers.items():
print(f"{bullet_id:<20} | {ans.choice:<20} | confidence={ans.confidence:.3f}")
Scaling Across Multiple Questions × Multiple Bullets
If you have M base questions and N bullets (for example, 5 standard audit questions applied across 40 subsystems or requirements), the same comprehension generates the full M x N = 200 matrix in a single API call—well within the 1,000-question per call limit—and gives you an exact (question_id, bullet_id) cell with its own calibrated certainty score.
From <https://gemini.google.com/app/999a992510a1f960> Google Gemini (4 Hydrogen)

Comments
Post a Comment