#4712 Prompt for Analyzing Provide Software Safety Assessment Reports

#4712 We review NASA cargo and crew launch provider software process and safety documents against NASA standards and requirements, including Computer Based Control Systems Requirements. Provider documents include Software Hazard Reports and Software Safety Analysis Reports. Given an uploaded Software Safety Analysis Report, provide LLM prompts to analyze the document structure and content an provide follow-up prompts for analyzing the document

A systematic prompt workflow aligned with NASA Software Safety standards (such as NASA-STD-8719.13, NASA-STD-8739.8, and NPR 7150.2) allows an LLM to effectively evaluate structural completeness, safety traceability, and control system rigor.

Initial Document Ingestion & Structural Prompts

Prompt 1: Baseline Structural & Compliance Assessment

Role: Act as a NASA Software Safety and Mission Assurance (SMA) Engineer.
Task: Review the attached provider Software Safety Analysis Report (SSAR).
Context: The document must align with NASA Software Safety Standards (NASA-STD-8719.13 / NASA-STD-8739.8) and Computer-Based Control System (CBCS) requirements.
Instructions:
Extract and summarize the core document architecture (e.g., Executive Summary, System/Software Description, Hazard Identification, Safety Criticality Assessment, Software Fault Tree / FMEA, Safety Requirements Traceability, Verification & Validation strategies).
Identify missing structural sections, ambiguous scopes, or standard boilerplate text that fails to reference specific provider subsystems.
Generate a structural compliance matrix highlighting: Required Element, Document Section Found, Status (Compliant / Partial / Missing), and Reviewer Notes.

Prompt 2: Scope, Architecture, and Criticality Mapping

Task: Audit the provider's software architecture and Software Safety Criticality Assessment (SSCA).
Instructions:
List all software components/CSIs (Computer Software Items) classified as Safety-Critical (Catastrophic or Critical severity).
Identify any autonomous flight control, abort triggers, propulsion management, separation sequencing, or CBCS command pathways.
Verify whether the provider established clear segregation/isolation boundaries between safety-critical and non-safety-critical software items. Flag any shared resources (memory, bus communications, CPU scheduling) lacking detailed safety controls.

Targeted Follow-Up Prompts for Deep-Dive Analysis

Follow-Up 1: Hazard Control & Mitigation Verification (NASA-STD-8719.13)

Task: Perform an in-depth audit of the Hazard Controls and Causes documented in this SSAR.
Instructions:
For every identified software cause leading to a system hazard (e.g., inadvertent command, untimely execution, loss of function, corrupted telemetry):
Extract the specific software safety control/mitigation.
Determine if the mitigation relies purely on software logic or includes independent hardware interlocks/inhibits.
Evaluate whether the control satisfies NASA two-fault tolerance for catastrophic hazards or single-fault tolerance for critical hazards.
Output any hazard causes where software is a single point of failure (SPOF) without an independent inhibit or manual override.

Follow-Up 2: Computer-Based Control Systems (CBCS) & Command Inhibit Analysis

Task: Audit the SSAR against NASA Computer-Based Control Systems (CBCS) requirements for crew/cargo safety.
Instructions:
Extract all sections detailing command authorization, health monitoring, fault detection, isolation, and recovery (FDIR).
Review the command handling architecture: Are critical commands protected against bit-flips, stale telemetry, out-of-order execution, and unauthorized uplink?
Verify the independence and validity of software inhibits for critical pyrotechnic, propulsion, or docking events. Flag any shared state variables or dependencies that invalidate inhibit independence.

Follow-Up 3: Bi-Directional Traceability & V&V Strategy

Task: Evaluate the Verification and Validation (V&V) matrix and requirement traceability within the SSAR.
Instructions:
Cross-reference stated software safety requirements against proposed V&V methods (e.g., Unit Testing, Static Code Analysis, Fault Injection, Hardware-in-the-Loop (HITL), Integrated Avionics Testing).
Flag any high-criticality software safety requirement verified solely by "Analysis" or "Inspection" where "Test" or "Fault Injection" should be mandatory.
Highlight any orphaned requirements (safety requirements without a linked test case or hazard report reference).

Follow-Up 4: Discrepancy & Finding Generator (NASA RID Format)

Task: Synthesize all identified gaps, ambiguities, and non-compliances from the previous analysis into standard NASA Review Item Discrepancies (RIDs).
Format each finding as follows:
RID Title / ID: Concise title
Document Reference: Page / Section number
Standard / Requirement Reference: (e.g., NASA-STD-8719.13 §X.X, CBCS Req ID)
Category: Major Non-Conformance / Minor Non-Conformance / Recommendation
Description of Finding: Clear, fact-based explanation of the technical gap
Safety Impact / Risk Statement: Potential consequence during launch, rendezvous, or docking
Recommended Provider Action: Specific resolution or data required from the provider

Write a Python script using LangChain to automate chunking and running this multi-stage SSAR review pipeline on PDF documents.

The Python pipeline below uses LangChain, PyPDF, and modern LCEL (LangChain Expression Language) runnable chains to ingest a Software Safety Analysis Report (SSAR) PDF, construct an in-memory vector store for contextual retrieval, and execute the multi-stage NASA SMA review pipeline.

Prerequisites

pip install langchain langchain-openai langchain-community chromadb pypdf tiktoken

export OPENAI_API_KEY="your-api-key"

SSAR Automated Review Script

import os
import sys
from typing import Dict, Any
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser


def build_vector_store(pdf_path: str, chunk_size: int = 1500, chunk_overlap: int = 250) -> Chroma:
    """Loads a PDF report, splits it into semantic chunks, and builds a vector store."""
    if not os.path.exists(pdf_path):
        raise FileNotFoundError(f"SSAR document not found at {pdf_path}")

    print(f"[+] Loading and parsing SSAR: {pdf_path}...")
    loader = PyPDFLoader(pdf_path)
    docs = loader.load()

    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=chunk_overlap,
        separators=["\n\n", "\n", " ", ""]
    )
    splits = splitter.split_documents(docs)
    print(f"[+] Created {len(splits)} chunks from {len(docs)} pages.")

    embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
    vectorstore = Chroma.from_documents(documents=splits, embedding=embeddings)
    return vectorstore


def format_docs(docs):
    """Concatenates retrieved document text with source metadata."""
    formatted = []
    for doc in docs:
        page = doc.metadata.get("page", "Unknown")
        formatted.append(f"--- [Page {page}] ---\n{doc.page_content}")
    return "\n\n".join(formatted)


def run_ssar_pipeline(pdf_path: str, output_file: str = "NASA_SSAR_Review_Findings.md"):
    """Executes the multi-stage NASA SMA review pipeline on an SSAR document."""
    vectorstore = build_vector_store(pdf_path)
    retriever = vectorstore.as_retriever(search_kwargs={"k": 8})
    llm = ChatOpenAI(model="gpt-4o", temperature=0.0)

    # ---------------------------------------------------------
    # STAGE DEFINITIONS & SYSTEM PROMPTS
    # ---------------------------------------------------------
    stages = [
        {
            "id": "Stage 1: Structural & Baseline Compliance",
            "query": "Table of contents, system description, safety criticality, software hazard analysis, V&V methods, requirements traceability matrix",
            "system_prompt": (
                "You are a NASA Software Safety and Mission Assurance (SMA) Lead. "
                "Audit the attached SSAR excerpts against NASA-STD-8719.13 and NASA-STD-8739.8 baseline structures.\n"
                "1. Summarize the document architecture.\n"
                "2. Identify missing structural sections or vague generic boilerplate text.\n"
                "3. Output a Structural Compliance Matrix (Required Element | Found Section | Status: Compliant/Partial/Missing | Reviewer Notes)."
            )
        },
        {
            "id": "Stage 2: Criticality, Architecture & Segregation",
            "query": "Software Safety Criticality Assessment SSCA, safety critical software items CSIs, flight computer, autonomous abort, propulsion, segregation, memory boundaries",
            "system_prompt": (
                "You are a NASA Avionics Safety Reviewer.\n"
                "1. Identify all software components classified as Safety-Critical (Catastrophic/Critical).\n"
                "2. Review autonomous flight control, abort triggers, and separation pathways.\n"
                "3. Evaluate architectural segregation between safety-critical and non-critical software. Flag shared CPU, bus, or memory resources lacking isolation controls."
            )
        },
        {
            "id": "Stage 3: Hazard Controls & Two-Fault Tolerance",
            "query": "Software hazard causes, hazard controls, mitigations, single point failure SPOF, inhibits, two-fault tolerance, loss of function, inadvertent execution",
            "system_prompt": (
                "You are an expert in NASA Hazard Analysis.\n"
                "1. For each software hazard cause (inadvertent command, untimely execution, loss of control), extract the documented mitigation.\n"
                "2. Verify if mitigations are pure software logic or paired with hardware interlocks.\n"
                "3. Flag single points of failure (SPOF) and assess whether catastrophic hazards meet NASA two-fault tolerance standards."
            )
        },
        {
            "id": "Stage 4: Computer-Based Control Systems (CBCS) & Inhibits",
            "query": "Computer Based Control Systems CBCS, command handling, telemetry integrity, FDIR, software inhibits, pyrotechnic commands, thruster control",
            "system_prompt": (
                "You are a NASA CBCS Safety SME.\n"
                "1. Audit command validation, health monitoring, and Fault Detection, Isolation, and Recovery (FDIR).\n"
                "2. Review protections against bit-flips, stale data, and unauthorized uplink commands.\n"
                "3. Verify software inhibit independence for critical propulsion, docking, and separation mechanisms."
            )
        },
        {
            "id": "Stage 5: Bi-Directional Traceability & V&V Rigor",
            "query": "Verification and Validation V&V matrix, unit test, hardware in the loop HITL, fault injection test, requirements traceability, inspection, analysis",
            "system_prompt": (
                "You are a NASA Software V&V Auditor.\n"
                "1. Cross-reference software safety requirements with their proposed verification methods.\n"
                "2. Flag safety-critical requirements verified only by 'Analysis' or 'Inspection' that require 'Test' or 'Fault Injection'.\n"
                "3. Identify orphaned requirements lacking linked verification cases."
            )
        },
        {
            "id": "Stage 6: NASA Review Item Discrepancies (RID) Generation",
            "query": "Deficiencies, missing mitigations, single points of failure, verification gaps, non-compliance with NASA-STD-8719.13",
            "system_prompt": (
                "You are generating formal NASA Review Item Discrepancies (RIDs) based on the SSAR evaluation.\n"
                "Format every critical deficiency as follows:\n"
                "- **RID ID / Title**:\n"
                "- **Document Reference** (Section / Page):\n"
                "- **Standard / Requirement Reference** (e.g., NASA-STD-8719.13, CBCS Req):\n"
                "- **Category** (Major Non-Conformance / Minor Non-Conformance / Recommendation):\n"
                "- **Description of Finding**:\n"
                "- **Safety Impact / Risk Statement**:\n"
                "- **Recommended Provider Action**:"
            )
        }
    ]

    # ---------------------------------------------------------
    # PIPELINE EXECUTION
    # ---------------------------------------------------------
    full_report = [f"# NASA SMA Review Report: {os.path.basename(pdf_path)}\n"]

    for stage in stages:
        print(f"\n[*] Executing {stage['id']}...")

        prompt_template = ChatPromptTemplate.from_messages([
            ("system", stage["system_prompt"]),
            ("human", "SSAR Context from Document:\n\n{context}\n\nPerform the requested analysis based strictly on the provided context.")
        ])

        rag_chain = (
            {"context": lambda _: format_docs(retriever.invoke(stage["query"]))}
            | prompt_template
            | llm
            | StrOutputParser()
        )

        response = rag_chain.invoke({})
        
        section_header = f"## {stage['id']}\n"
        full_report.append(section_header)
        full_report.append(response + "\n\n---\n")
        print(f"[✓] {stage['id']} complete.")

    # Save aggregated findings to Markdown
    with open(output_file, "w", encoding="utf-8") as f:
        f.write("\n".join(full_report))

    print(f"\n[+] Full review generated successfully: {output_file}")


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("Usage: python ssar_reviewer.py <path_to_ssar.pdf>")
    else:
        run_ssar_pipeline(sys.argv[1])

Pipeline Architecture

Contextual Retrieval per Stage: Rather than dumping the entire PDF into a single prompt, each stage queries the vector database using domain-specific safety terminology (fault injection, CSI, inhibit independence, CBCS).
Source Attribution: The format_docs helper injects page metadata into the retrieval stream so findings can reference document locations.
Deterministic Evaluation: temperature=0.0 ensures audit repeatability across iterative report revisions.

From <https://gemini.google.com/app/7b03639009798e3d> Google Gemini (3.7 Flash)

Search This Blog

Learning from ChatGPT