#4706 Fact-Checking Follow-up Prompt, AKA Peer Review

#4706 Fact-Checking Follow-up Prompt, AKA Peer Review

#4706 My coworker uses Microsoft Copilot to evaluate documents for compliance to NASA standards and requirements. Provide a general-purpose follow-up prompt to a different LLM (say Google Gemini) to take as input the original prompt and response to check it for correctness and completeness.

You are acting as an independent, rigorous Systems Engineering and Software Safety & Mission Assurance (SMA) verification auditor. 

Your task is to conduct a dual-pass evaluation of a compliance assessment generated by another LLM: first checking for factual and standard-specific correctness, and second checking for analytical completeness.

---

### Audit Inputs

* **Original Prompt / Directive:**

[PASTE ORIGINAL PROMPT HERE]

* **Source Requirement / Standard (Context):**

[PASTE STANDARD TEXT / SECTION / DIRECTIVE EXCERPT HERE, e.g., NPR 7150.2, NASA-STD-8739.8, etc.]

* **Target Artifact / Document Evaluated:**

[PASTE ARTIFACT TEXT / HAZARD REPORT / REQUIREMENT MATRIX BEING EVALUATED HERE]

* **Prior LLM Response to Audit:**

[PASTE COPILOT RESPONSE HERE]

---

### Evaluation Criteria

1. **Correctness (Ground-Truth Verification):**

   * Identify any misinterpretations, incorrect clause mappings, or hallucinated requirements.

   * Verify whether explicit mandatory provisions (e.g., "shall" statements) were evaluated against the exact verbiage of the standard rather than inferred intent.

   * Check for invalid assumptions regarding safety classifications, hazard causes, controls, or verification methods.

2. **Completeness (Gap & Edge-Case Analysis):**

   * Identify omitted sections, sub-clauses, or lifecycle/verification requirements demanded by the standard.

   * Highlight unaddressed edge cases, missing bidirectional traceability, or incomplete compliance matrices.

   * Note where the response gave a superficial "compliant/non-compliant" verdict without the required technical justification or objective evidence.

---

### Required Output Format

* **Audit Verdict:** State clearly whether the prior response is *Fully Valid*, *Partially Valid (Corrections Needed)*, or *Invalid/Incomplete*.

* **Factual Corrections:** Bullet list of explicit errors, wrong citations, or hallucinated claims in the prior response (with exact standard citations).

* **Missing Gaps & Omissions:** Bullet list of missing requirements, unaddressed controls, or incomplete verification criteria.

* **Refined Compliance Assessment:** Provide the fully corrected, end-to-end compliance evaluation incorporating all missing elements and standard-compliant technical justifications.

The prompt needs to work with the original uploaded files (both NASA standard or requirements, and NASA cargo or crew launch provider software development document)

You are acting as an independent, rigorous Systems Engineering and Software Safety & Mission Assurance (SMA) auditor. 

I have uploaded two primary source files along with the initial review context below:

1. **Governing NASA Standard / Requirements Document** [ATTACHED]

2. **Provider Software Development / Verification Artifact** [ATTACHED]

Your task is to conduct an independent verification and validation (IV&V) audit of a compliance evaluation previously generated by an AI assistant. You must cross-reference the prior response directly against the uploaded source documents for technical correctness and complete coverage.

---

### Audit Inputs

* **Original Prompt Provided to Prior Model:**

[PASTE ORIGINAL COPILOT PROMPT HERE]

* **Prior Model Response Under Review:**

[PASTE COPILOT RESPONSE HERE]

---

### Evaluation Criteria

1. **Correctness (Direct Document Traceability):**

   * Cross-examine every finding in the prior response against the exact text in the uploaded standard and provider document.

   * Flag any hallucinated clauses, misattributed requirement IDs, or misinterpretations of mandatory requirements ("shall" statements).

   * Verify that provider claims of compliance are backed by valid objective evidence and verification methods within the uploaded artifact, rather than superficial assertion.

2. **Completeness & Gap Analysis:**

   * Identify any mandatory requirements from the governing standard that apply to this provider artifact but were completely omitted from the prior evaluation.

   * Highlight missing bidirectional traceability (e.g., standard requirement -> provider implementation -> verification method/test case).

   * Note any unaddressed edge cases, degraded modes, off-nominal conditions, or safety-critical software control gaps.

---

### Required Output Structure

* **Overall Audit Verdict:** Explicitly state one of: **Verified (No Changes)**, **Partially Valid (Corrections/Additions Required)**, or **Failed (Substantial Errors/Gaps)**.

* **Discrepancy & Correction Log:** A Markdown table listing:

  | Standard Ref / Clause | Prior Model Finding | Audit Assessment (Correction/Refutation) | Ground Truth Source Evidence |

  | :--- | :--- | :--- | :--- |

* **Omissions & Unaddressed Requirements:** Bulleted list of applicable standard clauses or provider verification gaps missed by the prior model.

* **Comprehensive Re-Evaluation:** A definitive, end-to-end compliance matrix and technical finding report generated from the ground-truth documents.

As applied using AskSageClient, what would be the best primary LLM for document analysis, and secondary LLM for cross-checking?

For dual-pass document analysis and independent verification in AskSageClient, pairing models from different foundation providers eliminates shared blind spots and tokenizer biases.

Recommended Pairing Strategy

Role	Recommended Model Family	Strengths in Pipeline
Primary (Extraction & Synthesis)	Claude 3.5 Sonnet / Claude 3 Opus (anthropic)	Industry-leading document comprehension, exceptional fidelity to complex prompt instructions, and low hallucination rates when processing long-form technical standards.
Secondary (Cross-Checking & Audit)	OpenAI GPT-4o / GPT-4 Turbo (or Google Gemini Pro)	Strong structured reasoning, rigid adherence to tabular evaluation rubrics, and independent logic to catch omissions or misinterpretations made by Claude.

AskSageClient Implementation Pattern

You can inspect the exact model strings authorized on your enclave via client.get_models(), then route each pass explicitly:

from asksageclient import AskSageClient

client = AskSageClient(email="user@agency.gov", api_key="YOUR_API_KEY")

# 1. Primary Analysis Pass (Claude)
primary_response = client.query_with_file(
    message=initial_review_prompt,
    model="claude-3-5-sonnet",  # or your enclave's Anthropic deployment tag
    file_path="provider_sw_dev_plan.pdf"
)

# 2. Independent Audit Pass (OpenAI / Gemini)
audit_payload = f"""
{cross_check_prompt_template}

### Prior Model Response Under Review:
{primary_response.get('response')}
"""

audit_response = client.query_with_file(
    message=audit_payload,
    model="gpt-4o",  # or google-gemini-pro / gpt-4-gov
    file_path="governing_standard.pdf"
)

Tip: When running audits, keep the temperature parameter set to 0 (or 0.1) on the secondary model to enforce deterministic, strictly text-anchored verification against the uploaded standard.

From <https://gemini.google.com/app/a6c96b391edb5b54>  Google Gemini (3.7 Flash)

When using AskSageClient in Python, can I create a dataset and add a file to run against one LLM, then run a second prompt against the same to another LLM?

Yes, you can create a dataset, upload a file to it, and query that same dataset across multiple different models sequentially or in parallel using the AskSageClient.

Datasets (trainings) in AskSage are decoupled from the model runtime. Once a dataset is created and populated with files, it exists on your account and can be referenced by its dataset identifier in any subsequent query call regardless of which model (e.g., gpt-4o, claude-3-5-sonnet, mistral-large) you target.

Workflow Implementation

from asksage import AskSageClient

client = AskSageClient(api_key="YOUR_API_KEY")

# 1. Create a dataset (training bucket)
dataset_name = "multi_model_analysis"
client.create_dataset(dataset_name=dataset_name)

# 2. Upload file to the dataset
with open("report.pdf", "rb") as f:
    client.upload_file_to_dataset(
        dataset_name=dataset_name,
        file=f,
        file_name="report.pdf"
    )

# Optional: Allow time for vectorization/indexing if using large documents

# 3. First prompt against Model A (e.g., GPT-4o)
response_gpt = client.get_query(
    prompt="Extract key financial metrics from this document.",
    dataset=dataset_name,
    model="gpt-4o"
)
print("GPT Response:\n", response_gpt)

# 4. Second prompt against Model B (e.g., Claude 3.5 Sonnet) using the SAME dataset
response_claude = client.get_query(
    prompt="Identify potential risk factors and conflicting data based on the extracted metrics.",
    dataset=dataset_name,
    model="claude-3-5-sonnet"
)
print("Claude Response:\n", response_claude)

Key Considerations

• Indexing Latency: Large files require a few seconds to chunk and embed before they are queryable via RAG. If running a script end-to-end, implement a brief retry loop or wait for the dataset status to confirm indexing is complete.

• Stateless Queries: Each client.get_query() call against a dataset is evaluated against the vector store independently unless you pass a shared conversation/session ID to preserve context between the two different models.

From <https://gemini.google.com/app/8e7430171827036d>  Google Gemini (3.7 Flash)