#4809 Generative-AI Applied for Source Code Analysis

#4809 Generative-AI Applied for Source Code Analysis

#4809 Describe how to use generative-AI to document and provide visualization of source code

Using Generative AI for codebase documentation and visualization bridges the gap between raw implementation and human understanding by converting complex abstract syntax trees (ASTs), logic, and control flows into clear textual specs and dynamic visual artifacts.

1. Generating Source Code Documentation

Generative AI operates across three distinct documentation layers: inline symbol documentation, interface contracts, and high-level architectural narratives.

• Docstrings & Inline Annotations:

  • Prompt Pattern: Instruct the LLM to follow ecosystem-specific schemas (e.g., Google/NumPy docstrings for Python, JSDoc/TSDoc for TypeScript, rustdoc for Rust).

  • Context Strategy: Feed the function definition alongside references to its call sites or type interfaces. Explicitly instruct the model to document intent, side effects, and edge cases rather than rephrasing syntax line by line.

• API & Interface Specifications:

  • LLMs can parse controller code, route handlers, and data models to generate OpenAPI/Swagger specs (JSON/YAML), GraphQL schemas, or gRPC Protobuf definitions.

  • They automate endpoint summaries, expected query parameters, validation constraints, and payload response structures.

• High-Level Architecture & Runbooks:

  • Feed directory structures, dependency manifests (package.json, Cargo.toml, go.mod), and entry-point scripts.

  • Direct the model to construct system overviews, component relationship summaries, configuration guides, and onboarding READMEs.

2. Providing Visualizations via Text-Based Diagramming

Generative AI models excel at generating diagram-as-code syntaxes (like Mermaid.js, PlantUML, Graphviz DOT, or D2), which can be directly previewed or rendered inside Markdown engines, GitHub, or documentation portals.

Visualization Type	Diagram Format	Best Use Case in Code
Control Flow & Logic	Mermaid flowchart TD	Complex decision trees, retry logic, error-handling cascades.
Component & Service Architecture	Mermaid graph LR / C4-PlantUML	Microservice communication, module boundaries, API gateways.
Temporal / Request Cycles	Mermaid sequenceDiagram	Auth handshakes (OAuth), asynchronous worker queues, API roundtrips.
Data Models & Schema	Mermaid erDiagram	ORM entities, SQL relational joins, database tables.
State Transitions	Mermaid stateDiagram-v2	Finite state machines (order processing, connection lifecycles).

Example: Sequence Workflow Prompt

"Analyze this asynchronous payment webhook handler. Produce a Mermaid.js sequence diagram tracking the request from Stripe, through the HMAC verification middleware, database transaction, and Kafka event broadcast."

sequenceDiagram

    autonumber
    actor Stripe as Stripe Gateway
    participant MW as HMAC Middleware
    participant Svc as Payment Service
    participant DB as Postgres DB
    participant Queue as Kafka Broker

    Stripe->>MW: POST /webhook/stripe (Payload + Signature)
    alt Invalid Signature
        MW-->>Stripe: 401 Unauthorized
    else Signature Valid
        MW->>Svc: Forward event payload
        Svc->>DB: UPDATE order SET status = 'PAID'
        DB-->>Svc: Ack (Transaction Committed)
        Svc->>Queue: Emit `order.paid` event
        Queue-->>Svc: Ack
        Svc-->>Stripe: 200 OK
    end

3. End-to-End Implementation Architecture

To operationalize AI documentation across an organization, embed it into the CI/CD and developer tooling loop:

1. Extraction & Context Preparation:

   • Use an AST parser (like Tree-sitter) or language server protocol (LSP) to extract call graphs, function signatures, and imports.

   • Avoid feeding massive unindexed repositories directly; use Retrieval-Augmented Generation (RAG) or symbol graphs to retrieve relevant context files.

2. Automated CI/CD Pipeline:

   • Trigger an AI agent on pull requests or scheduled jobs.

   • If a pull request modifies an API route or schema, the agent generates an updated Mermaid sequence diagram and regenerates docstrings for changed symbols.

3. Verification & Guardrails:

   • Syntax Linters: Validate Mermaid and OpenAPI outputs through CLI linters (mmdc, spectral) before merging to ensure diagrams parse cleanly.

   • Human-in-the-Loop: Flag AI-generated documentation in PR comments for maintainer approval to prevent architectural hallucinations.

Describe unit-level code analysis at the method level

Unit-level code analysis at the method level is the automated or systematic evaluation of individual functions, procedures, or methods in isolation from the rest of the application. Instead of looking at architecture or cross-module interactions, it treats the method as the smallest functional boundary to evaluate correctness, quality, security, and performance.

It generally splits into two complementary approaches: Static Analysis (examining source code or bytecode without executing it) and Dynamic Analysis (observing the method during execution).

1. Static Analysis at the Method Level

Static method analysis parses the method's code into an Abstract Syntax Tree (AST) and constructs models such as Control Flow Graphs (CFGs) to evaluate logic and structure.

Key Metrics & Structural Properties

• Cyclomatic Complexity (M): Measures the number of linearly independent paths through the method:

M = E - N + 2P

Where \(E\) is edges, \(N\) is nodes in the control flow graph, and \(P\) is connected components. High complexity flags methods that are difficult to test, reason about, or maintain.

• Cognitive Complexity: Evaluates readability by weighting nested flow control statements (e.g., deeply nested if/for blocks) more heavily than flat switches.

• Lines of Code (LOC) & Parameter Count: Flags violations of structural limits (e.g., methods exceeding 50 lines or taking more than 4–5 arguments, suggesting a violation of the Single Responsibility Principle).

Semantic and Data-Flow Checks

• Taint Analysis & Variable Scoping: Traces how data moves from parameters to return values or sinks within the method body, detecting unvalidated inputs or uninitialized local variables.

• Defensive Checks & Nullability: Evaluates branch coverage for boundary conditions, missing default cases in switches, null reference dereferences, or unhandled exceptions.

• Dead Code Elimination: Identifies unreachable branches, unused local variables, or redundant assignments.

2. Dynamic Analysis at the Method Level

Dynamic analysis evaluates the method while it runs, typically instrumenting execution during unit testing.

Execution Tracking

• Code Coverage: Measures how thoroughly test suites exercise the method:

  • Line/Statement Coverage: Which statements were executed.

  • Branch Coverage: Whether every boolean decision (e.g., true and false branches of an if) was taken.

  • Path Coverage: Whether all possible end-to-end combinations of execution paths were traversed.

• Mutation Testing: Programmatically introduces small faults (mutants) into the method body (e.g., flipping < to >=, replacing + with -) to verify if existing unit tests fail. If tests still pass, the test suite is inadequate.

Resource & Runtime Behavior

• Micro-benchmarking & Profiling: Measures invocation overhead, CPU time per call, and memory/heap allocations strictly inside the method boundary.

• Invariant Checking: Validates formal preconditions, postconditions, and runtime assertions (Design by Contract) across various inputs.

3. Core Differences: Method-Level vs. Broader Scopes

Dimension	Method Level	Class / Module Level	System / Integration Level
Primary Scope	Function body, inputs, local state, return value	Interactions between methods, internal state encapsulation	Service-to-service communication, network, DB
Complexity Focus	Cyclomatic & cognitive complexity	Coupling (CBO), cohesion (LCOM)	Architectural layering, dependency cycles
State Dependencies	Isolated; relies on mocks/stubs for external calls	State invariants of the object/module	Distributed state, persistent storage
Analysis Speed	Near-instantaneous (ideal for IDE linters & commit hooks)	Fast to moderate	Slow (requires environment spin-up)

4. Primary Use Cases

• IDE-Time Feedback: Linters (e.g., ESLint, SonarLint, Clippy) compute AST-based method metrics in real time to enforce clean code standards.

• Vulnerability Mitigation: Catches issues like off-by-one errors, buffer overflows, integer wrap-arounds, or unhandled exceptions before code is committed.

• Automated Test Generation: Symbolic execution and fuzzing engines (e.g., AFL, EvoSuite) analyze method control flow graphs to generate test inputs that trigger every execution branch.

• Refactoring Triggers: Identifies code smells such as "Long Method," "Feature Envy," or excessive nesting, signaling when a method should be decomposed into smaller helpers.

Describe requirements to source code traceability

Requirements-to-source-code traceability is the ability to establish, follow, and verify a bidirectional link between high-level functional or non-functional requirements and the specific lines, methods, or modules of source code implementing them.

It ensures that every requirement is backed by concrete implementation, and conversely, that every line of production code exists for a documented business or safety purpose.

1. Bidirectional Traceability: Forward vs. Backward

A complete traceability framework functions in two complementary directions:

• Forward Traceability (Requirements $\rightarrow$ Code):

  • Question Answered: "Has this requirement been implemented, and where does it live?"

  • Purpose: Verifies completeness. It ensures no documented feature, business rule, or safety invariant is forgotten or orphaned during development.

• Backward Traceability (Code $\rightarrow$ Requirements):

  • Question Answered: "Why does this piece of code exist?"

  • Purpose: Prevents scope creep and "gold plating." It flags dead, undocumented, or rogue code by requiring every pull request or module to cite a valid originating requirement.

2. The Traceability Chain

Code traceability rarely connects a requirement directly to raw code in one leap; instead, it forms an interconnected lifecycle graph:

Plaintext

Business / System Requirement (e.g., REQ-104)
        │
        ▼
Architectural / Technical Design (e.g., ADR-012)
        │
        ▼
Implementation Issue / Ticket (e.g., JIRA-842)
        │
        ▼
Git Branch / Pull Request (e.g., PR #152)
        │
        ▼
Source Code Symbols (e.g., `PaymentService::processRefund()`)
        │
        ▼
Verification / Unit Tests (e.g., `test_processRefund_exceeds_balance()`)

3. Implementation Techniques

Organizations enforce and maintain traceability using several common technical mechanisms:

Metadata & Semantic Tagging in Code

Developers link requirements directly at the function or module boundary via annotations, decorators, or docstrings:

Python

# @implements REQ-SEC-402: Rate-limit authentication attempts to 5 per minute
@rate_limiter(max_attempts=5, window_seconds=60)
def authenticate_user(credentials: AuthRequest) -> AuthToken:
    ...

Static analysis tools can then parse the Abstract Syntax Tree (AST) to generate an automated Requirements Traceability Matrix (RTM).

VCS & Git-Native Linkage

• Branch Naming & Commit Hooks: Branches and commit messages strictly require requirement or ticket prefixes (e.g., feat/REQ-201-session-timeout).

• Git Blame / Commit Metadata: Engineers auditing a line of code can trace git log $\rightarrow$ merge commit $\rightarrow$ Pull Request $\rightarrow$ Issue Tracker ID $\rightarrow$ Product Requirement Document (PRD).

Programmatic Traceability Matrix (RTM)

An automated table mapping requirements directly to their execution and verification assets:

Requirement ID	Description	Source File / Symbol	Verification Test	Status
REQ-AUTH-01	Support OAuth2 Authorization Code flow	src/auth/oauth.ts::exchangeToken()	tests/auth.test.ts::test_token_exchange	Verified
REQ-PAY-04	Deduct store credit before charging card	src/billing/ledger.go::ApplyCredit()	tests/ledger_test.go::TestCreditDeduction	In Progress
REQ-SEC-09	Enforce TLS 1.3 minimum for API calls	src/config/tls.go::TlsConfig()	tests/security/tls_test.go	Passed

4. Industry Significance & Use Cases

• Regulated & Safety-Critical Engineering: In medical devices (IEC 62304), avionics (DO-178C), and automotive safety (ISO 26262), bidirectional traceability is an explicit legal mandate. Systems cannot achieve certification without proving that every requirement is implemented and verified by test suites.

• Impact Analysis: When a customer or stakeholder alters a requirement, architects query the traceability link to immediately identify which files, API contracts, and tests will be broken or require changes.

• Software Archeology & Maintenance: Reduces cognitive overhead for developers maintaining legacy systems by explaining why an unusual edge case or non-obvious conditional exists.

• Auditing and Compliance: Enables third-party compliance teams (e.g., SOC 2, HIPAA) to verify that privacy and security mandates are continuously enforced in source artifacts.

5. Automated Traceability with Modern Tooling

Maintaining static tables manually in spreadsheets degrades quickly as codebases evolve. Modern engineering teams rely on automated pipelines:

• Application Lifecycle Management (ALM) Platforms: Tools like Polarion, Siemens Teamcenter, Jama Software, or IBM Engineering Requirements Management (DOORS) link requirement baselines directly to Git repositories.

• CI/CD Linting Engines: Automated actions that fail PR merges if new methods lack traceability tags, or if a referenced Requirement ID is missing or invalid in the requirement management system.

• Generative AI & Semantic Graph Analysis: LLMs and code embeddings index requirement specifications alongside repository call graphs to automatically detect unmapped requirements, obsolete comments, or implementation drift.

How can generative AI and RAG be used to build and maintain an automated Requirements Traceability Matrix from source code?

Building and maintaining an automated Requirements Traceability Matrix (RTM) using Generative AI and Retrieval-Augmented Generation (RAG) transforms traceability from a retrospective manual compliance task into a continuous, real-time feedback loop.

The core challenge in software traceability is vocabulary and abstraction mismatch: requirements are expressed in domain and natural-language terms ("authenticate user with MFA"), while source code is defined by control flow, data models, AST structures, and technical identifiers (verify_totp_token(), UserAuthSession).

Below is an end-to-end architecture and operational model for automating RTM generation and maintenance.

1. Dual-Index Ingestion: Code & Requirements Representation

A standard naive RAG approach (raw vector search over text chunks) often struggles with code semantics because it ignores hierarchical structure. A production-grade traceability pipeline uses a hybrid or GraphRAG representation:

[Requirements Source] (Jira, DOORS, Markdown, SRS)
         │
         ▼ (Structural Decomposition: ID, Statement, Safety Class, Constraints)
   Requirement Index (Dense Vectors + Keyword/BM25)
         │
         │  Cross-Modal Semantic Alignment & Link Inference
         ▼
       LLM Link Reasoning Engine (Pydantic / Structured Output)
         ▲
         │
     Code Knowledge Index (AST / Call Graph / Vector Embeddings)
         ▲
         │ (Tree-sitter Parsing: Functions, Methods, Types, Docstrings)
[Source Codebase] (Git Repositories, PR Diffs, Test Suites)

1. Requirements Side:

   • Ingest structured requirements (e.g., from DOORS, Jira, Jama, or System Requirement Specifications).

   • Parse each record into discrete metadata units: Requirement_ID, Title, Text, Verification_Method (e.g., Test, Analysis, Inspection), and Safety/Criticality_Level.

   • Store both semantic embeddings (dense vectors) and structural tags in a hybrid vector/metadata index.

2. Code Side (AST-Aware Chunking):

   • Do not chunk code by raw line count. Use language-aware parsers (e.g., Tree-sitter) to extract granular code units: methods, classes, signatures, docstrings, and call-site edges.

   • Generate code embeddings combining docstring summaries, function names, and enclosing class/module paths to capture semantic purpose alongside syntax.

2. Traceability Link Recovery (TLR) Pipeline

To establish initial candidate mappings (Traceability Link Recovery), the RAG pipeline operates through a multi-stage retrieval and evaluation process:

1. Candidate Retrieval (Broad Filter):

   • For each requirement, run a hybrid query (BM25 lexical + dense vector search) across the codebase index to surface the top $K$ candidate functions or modules.

2. Context Enrichment via Call Graphs:

   • For surfaced candidate methods, pull immediate neighbors from the call graph (e.g., helper methods, database queries, downstream service calls) so the LLM sees the end-to-end execution path rather than an isolated function stub.

3. LLM Chain-of-Thought Link Evaluation:

   • Feed the requirement and the candidate code context into the model with strict structured output schemas.

   • Direct the LLM to assess coverage criteria:

     • Does this code fully implement, partially implement, or only support the requirement?

     • Which specific conditional branches enforce the specified constraints?

   • Output Format: Emit a validated JSON or Pydantic record containing req_id, file_path, symbol_name, confidence_score, and rationale.

{
  "requirement_id": "REQ-AUTH-04",
  "source_target": "src/services/auth_service.py::validate_login_attempt",
  "relationship_type": "IMPLEMENTS",
  "coverage_status": "FULL",
  "confidence": 0.94,
  "rationale": "Method enforces maximum failed attempts (5) and triggers account lockout via redis_client.set_lockout().",
  "verification_target": "tests/test_auth.py::test_lockout_after_five_failed_attempts"
}

3. Continuous RTM Maintenance via CI/CD

Traceability decays when code changes without corresponding updates to the matrix. Embedding the pipeline into CI/CD automates ongoing maintenance:

• PR Diff-Level Impact Analysis:

  • When a developer opens a pull request, an automated action inspects the git diff.

  • For modified methods, the system queries the RTM graph to identify linked requirements.

  • The LLM evaluates whether the code changes alter, satisfy, or break the linked requirement’s acceptance criteria.

• Orphan & Drift Detection:

  • Dead Code / Rogue Features: Flags newly introduced public methods that cannot be traced to any active requirement baseline (detecting scope creep or unapproved changes).

  • Unimplemented Requirements: Scans the requirements baseline to detect any item lacking associated source code or verification tests (coverage gaps).

  • Broken Trace Links: If a function is renamed or refactored, the static analysis parser alerts the RTM store to update symbol pointers automatically.

• Inline Tagging & Verification:

  • Generative AI can propose non-invasive inline annotations (e.g., # @implements REQ-AUTH-04) during PR review, enabling fast deterministic compiler/AST validation on subsequent commits.

4. Graph-Based RTM Storage & Querying

Instead of storing the RTM in a flat spreadsheet, persist traces within a Property Graph (e.g., Neo4j, Apache AGE, or NetworkX):

graph LR

    R[Requirement

REQ-PAY-02] -->|DECOMPOSED_INTO| SR[Sub-Requirement

REQ-PAY-02.1]

SR -->|IMPLEMENTED_BY| C[Source Code

StripeGateway.ts]

C -->|CALLS| H[Helper Method

verifySignature]

T[Test Case

test_webhook_validation] -->|VERIFIES| SR

T -->|EXECUTES_AGAINST| C

**Benefits of Graph-Backed RTM:**
* **Bidirectional Impact Queries:** Run Cypher/graph queries to determine exact ripple effects: *"If `REQ-PAY-02` changes, which code files, helper functions, and test fixtures must be re-baselined?"*
* **Verification Auditing:** Instantly identify sub-requirements where `IMPLEMENTED_BY` exists, but `VERIFIED_BY` (linked unit/integration test) is missing.

5. Architectural Safeguards for Regulated Contexts

In mission-critical or regulated software environments (e.g., NASA NPR 7150.2D, DO-178C, ISO 26262, IEC 62304), AI outputs cannot be treated as authoritative without verification:

1. Deterministic Pre-Filters Over Semantic Guesses:

   • Prioritize explicit tags, ticket linkages, and PR commit references before resorting to semantic embeddings. Use LLMs primarily to reconcile unmapped entities, infer implicit links, and detect semantic drift.

2. Confidence Thresholding & Human Review Triage:

   • High-confidence links (>= 0.90) with matching symbols are auto-staged into the RTM.

   • Borderline matches (0.60 - 0.89) generate an automated review checklist on the PR for the software quality assurance (SQA) team or reviewer.

   • Matches below 0.60 are flagged as unlinked gaps.

3. Immutable Provenance:

   • Store the model version, embedding checkpoint, prompt template, and git commit SHA alongside every generated trace link to maintain auditability during safety compliance audits.

Show a Neo4j Cypher schema and sample queries for mapping requirements, code methods, and unit test suites.

Here is a complete Neo4j property graph schema, constraint setup, synthetic ingest script, and sample analytical Cypher queries for an automated Requirements Traceability Matrix (RTM).

1. Graph Data Model & Schema

The graph tracks bidirectional relationships across four primary domains: requirements decomposition, code structure, test verification, and audit metadata.

graph TD

    R[Requirement] -->|DECOMPOSES_TO| SR[Requirement]
    SR -->|IMPLEMENTED_BY| M[Method]
    M -->|BELONGS_TO| C[Class / Module]
    M -->|CALLS| M2[Method]
    T[TestCase] -->|VERIFIES| SR
    T -->|EXECUTES| M
    A[AuditFinding] -->|FLAGS| SR

Node Labels and Properties

• :Requirement — id, text, type (Functional, Safety, Security), safety_class (A, B, C, D), status (Draft, Approved, Deprecated), version.

• :Method — id (e.g., file_path::class::method), name, file_path, start_line, end_line, cyclomatic_complexity, git_sha.

• :TestCase — id, name, file_path, framework (pytest, JUnit), execution_status (Passed, Failed), coverage_type (Branch, Statement).

• :AuditFinding — id, finding_type (Missing_Test, Unmapped_Code, Low_Confidence), severity, timestamp.

Relationship Types and Properties

• (:Requirement)-[:DECOMPOSES_TO]->(:Requirement)

• (:Requirement)-[:IMPLEMENTED_BY {confidence: Float, source: 'EXPLICIT_TAG' | 'LLM_INFERRED', model_version: String}]->(:Method)

• (:TestCase)-[:VERIFIES {verification_method: 'Automated_Unit' | 'Integration'}]->(:Requirement)

• (:TestCase)-[:EXECUTES {line_coverage: Float}]->(:Method)

• (:Method)-[:CALLS]->(:Method)

2. Constraints and Indexes

Execute these unique constraints and property indexes first to ensure data integrity and optimize traversal speeds during batch ingest.

// Ensure unique identifiers
CREATE CONSTRAINT req_id_unique IF NOT EXISTS
FOR (r:Requirement) REQUIRE r.id IS UNIQUE;

CREATE CONSTRAINT method_id_unique IF NOT EXISTS
FOR (m:Method) REQUIRE m.id IS UNIQUE;

CREATE CONSTRAINT test_id_unique IF NOT EXISTS
FOR (t:TestCase) REQUIRE t.id IS UNIQUE;

// Indexes for fast filtering and joins
CREATE INDEX req_safety_class IF NOT EXISTS
FOR (r:Requirement) ON (r.safety_class);

CREATE INDEX method_file_path IF NOT EXISTS
FOR (m:Method) ON (m.file_path);

CREATE INDEX trace_confidence IF NOT EXISTS
FOR ()-[r:IMPLEMENTED_BY]-() ON (r.confidence);

3. Sample Data Ingestion Script

Run this transaction block to populate a representative cross-section of hierarchical requirements, methods, unit tests, and LLM-inferred links.

// 1. Create Requirements Hierarchy
MERGE (r_parent:Requirement {id: 'REQ-SEC-100'})
ON CREATE SET r_parent.text = 'The system shall ensure secure transaction authentication.',
              r_parent.safety_class = 'Class_A',
              r_parent.type = 'Security';

MERGE (r_child1:Requirement {id: 'REQ-SEC-101'})
ON CREATE SET r_child1.text = 'The system shall rate-limit token validation attempts to 5 per minute per IP.',
              r_child1.safety_class = 'Class_A',
              r_child1.type = 'Security';

MERGE (r_child2:Requirement {id: 'REQ-SEC-102'})
ON CREATE SET r_child2.text = 'The system shall record an audit log entry for every failed authentication.',
              r_child2.safety_class = 'Class_B',
              r_child2.type = 'Security';

MERGE (r_parent)-[:DECOMPOSES_TO]->(r_child1);
MERGE (r_parent)-[:DECOMPOSES_TO]->(r_child2);

// 2. Create Code Methods
MERGE (m1:Method {id: 'auth/ratelimit.py::RateLimiter::check_rate'})
ON CREATE SET m1.name = 'check_rate',
              m1.file_path = 'auth/ratelimit.py',
              m1.cyclomatic_complexity = 4,
              m1.git_sha = '8b3c9a1';

MERGE (m2:Method {id: 'auth/token.py::TokenValidator::validate_token'})
ON CREATE SET m2.name = 'validate_token',
              m2.file_path = 'auth/token.py',
              m2.cyclomatic_complexity = 7,
              m2.git_sha = '8b3c9a1';

MERGE (m3:Method {id: 'telemetry/logger.py::AuditLogger::log_security_event'})
ON CREATE SET m3.name = 'log_security_event',
              m3.file_path = 'telemetry/logger.py',
              m3.cyclomatic_complexity = 2,
              m3.git_sha = '8b3c9a1';

// Method Calls
MERGE (m2)-[:CALLS]->(m1);

// 3. Link Requirements to Code
// Explicit docstring match
MERGE (r_child1)-[:IMPLEMENTED_BY {confidence: 1.0, source: 'EXPLICIT_TAG'}]->(m1);

// LLM Inferred link
MERGE (r_child1)-[:IMPLEMENTED_BY {
  confidence: 0.93,
  source: 'LLM_INFERRED',
  model_version: 'gemini-1.5-pro'
}]->(m2);

// 4. Create Test Cases
MERGE (t1:TestCase {id: 'tests/test_ratelimit.py::test_exceed_five_attempts'})
ON CREATE SET t1.name = 'test_exceed_five_attempts',
              t1.framework = 'pytest',
              t1.execution_status = 'Passed';

// Verification Links
MERGE (t1)-[:VERIFIES {verification_method: 'Automated_Unit'}]->(r_child1);
MERGE (t1)-[:EXECUTES {line_coverage: 0.95}]->(m1);

4. Core Verification & Impact Analysis Queries

Query A: Bidirectional Traceability Matrix Extraction

Extracts the complete flattened matrix showing each leaf requirement, the implementing method, verification test, and link provenance.

MATCH (r:Requirement)
WHERE NOT ()-[:DECOMPOSES_TO]->(r) // Find leaf requirements or top-level items
OPTIONAL MATCH (r)-[imp:IMPLEMENTED_BY]->(m:Method)
OPTIONAL MATCH (t:TestCase)-[:VERIFIES]->(r)
RETURN r.id AS Requirement_ID,
       r.safety_class AS Safety_Class,
       r.text AS Requirement_Text,
       m.id AS Implementing_Method,
       imp.source AS Link_Source,
       imp.confidence AS Link_Confidence,
       t.id AS Verified_By_Test,
       t.execution_status AS Test_Status
ORDER BY r.id, m.id;

Query B: Gap Analysis (Orphan Requirements & Untested Implementations)

Critical for quality audits: identifies requirements that have code but no tests, or requirements missing code implementations entirely.

MATCH (r:Requirement)
WHERE NOT (r)-[:DECOMPOSES_TO]->() // Target operational leaf requirements
OPTIONAL MATCH (r)-[:IMPLEMENTED_BY]->(m:Method)
OPTIONAL MATCH (t:TestCase)-[:VERIFIES]->(r)
WITH r, count(m) AS method_count, count(t) AS test_count
WHERE method_count = 0 OR test_count = 0
RETURN r.id AS Requirement_ID,
       r.safety_class AS Safety_Class,
       CASE 
         WHEN method_count = 0 AND test_count = 0 THEN 'NO_CODE_AND_NO_TEST'
         WHEN method_count = 0 THEN 'UNIMPLEMENTED'
         WHEN test_count = 0 THEN 'UNVERIFIED_CODE'
       END AS Traceability_Gap,
       r.text AS Requirement_Text;

Query C: Change Impact Analysis (Ripple Effect from Code Modification)

Given a changed or refactored method (e.g., during a PR diff review), traverse backward through call trees to find all impacted requirements and the upstream test suites that must be rerun.

MATCH (changed:Method {id: 'auth/ratelimit.py::RateLimiter::check_rate'})
// Match methods calling the changed method up to 2 hops away, or the method itself
MATCH path = (caller:Method)-[:CALLS*0..2]->(changed)
WITH collect(DISTINCT caller) AS affected_methods

UNWIND affected_methods AS m
MATCH (r:Requirement)-[:IMPLEMENTED_BY]->(m)
OPTIONAL MATCH (t:TestCase)-[:EXECUTES]->(m)
RETURN m.id AS Directly_Affected_Method,
       collect(DISTINCT r.id) AS Impacted_Requirements,
       collect(DISTINCT t.id) AS Tests_To_Reexecute;

Query D: High-Risk AI Inferences Requiring Human-in-the-Loop Review

Identifies links inferred by LLMs whose confidence score falls between ambiguous margins (0.60 \le \text{confidence} < 0.90) for Safety-Critical requirements (Class A/B).

MATCH (r:Requirement)-[rel:IMPLEMENTED_BY]->(m:Method)
WHERE rel.source = 'LLM_INFERRED'
  AND rel.confidence >= 0.60 
  AND rel.confidence < 0.90
  AND r.safety_class IN ['Class_A', 'Class_B']
RETURN r.id AS Requirement_ID,
       r.safety_class AS Criticality,
       m.id AS Inferred_Method,
       rel.confidence AS Confidence,
       rel.model_version AS LLM_Engine
ORDER BY rel.confidence ASC;

Would it make sense to add both design and test data to the knowledge graph?

Yes, adding both design and test data to the knowledge graph makes absolute sense. In fact, doing so transforms the graph from a simple code-indexer into a complete digital engineering thread and verification/validation (V&V) engine.

Linking requirements directly to code methods creates a semantic "abstraction cliff." A high-level requirement rarely maps cleanly to a single line or method of code without an intermediate design decision, architectural boundary, or interface definition. Incorporating design artifacts bridges that semantic gap, while integrating dynamic test execution data closes the verification loop.

1. The Value of Incorporating Design Data

Design data—such as architectural decision records (ADRs), interface control documents (ICDs), SysML/UML block decompositions, state machines, and hazard mitigations—serves as the structural glue between intent and implementation.

• Mitigating Vocabulary Drift: LLMs and embeddings struggle to link an abstract operational requirement (e.g., "System shall safe itself upon communication loss") directly to low-level hardware registers or socket code. A design node representing a WatchdogState or FailSafeSubsystem provides the intermediate semantic hop.

• Architectural Conformance Checking: You can detect design erosion or layer violations. For example, if a Method in the data layer calls a Method in the UI layer, or implements a behavior not sanctioned by an ArchitectureComponent node, a Cypher query can instantly flag the architectural anomaly.

• Safety & Hazard Allocation: For mission-critical or safety systems, safety requirements frequently trace to hazard reports and architectural fault-tolerance designs (e.g., redundant buses, triple-modular voting) rather than single functions.

2. The Value of Incorporating Test Data

Test data should include not just static test code declarations, but test cases, test configurations/environments, and runtime execution results.

• Closed-Loop Verification: Static links between code and requirements show implementation coverage, but they do not prove operational satisfaction. Linking test runs and coverage metrics demonstrates whether a requirement is actually verified.

• Targeted Regression & Smoke Testing: When a PR changes a specific design specification or code method, a graph query can traverse downstream to identify the exact subset of unit, integration, and hardware-in-the-loop (HIL) test suites that must run, drastically reducing CI/CD build times.

• Root Cause & Failure Impact Analysis: If an automated test fails in CI or a field test, traversing upstream identifies which specific code commits, architectural interfaces, and mission-level requirements are compromised.

3. The Unified Knowledge Graph Topology

Integrating design and test assets expands the graph into a multi-tiered verification lifecycle:

[Hazard / Mission Goal]
          │
          ▼
    (:Requirement) ◄───────────────┐
          │                        │
     [:SPECIFIED_BY]          [:VERIFIES]
          │                        │
          ▼                        │
   (:DesignElement)           (:TestCase)
   (ADR / SysML Block)             │
          │                   [:EXECUTES]
    [:REALIZED_BY]                 │
          │                        ▼
      (:Method) ◄──────────── (:TestRun)
   (AST / Symbol)             (Passed / Failed / Coverage)

4. Extended Cypher Schema Representation

// 1. Structural Nodes
(:Requirement {id, text, safety_class, status})
(:DesignElement {id, type: 'ADR'|'Block'|'Interface', rationale, author})
(:Method {id, file_path, cyclomatic_complexity, git_sha})
(:TestCase {id, framework, level: 'Unit'|'Integration'|'System'})
(:TestRun {id, execution_timestamp, outcome: 'Passed'|'Failed', line_coverage, env})

// 2. Traversal Relationships
(:Requirement)-[:SPECIFIED_BY]->(:DesignElement)
(:DesignElement)-[:REALIZED_BY]->(:Method)
(:TestCase)-[:VERIFIES]->(:Requirement)
(:TestCase)-[:VALIDATES_DESIGN]->(:DesignElement)
(:TestCase)-[:EXECUTES]->(:Method)
(:TestCase)-[:PRODUCED]->(:TestRun)

5. High-Impact Analytical Capabilities Unlocked

A. Comprehensive Verification Gap Detection

Identify high-criticality requirements that have design specs and code implementation, but lack passing execution tests:

MATCH (r:Requirement {safety_class: 'Class_A'})
MATCH (r)-[:SPECIFIED_BY]->(d:DesignElement)-[:REALIZED_BY]->(m:Method)
OPTIONAL MATCH (t:TestCase)-[:VERIFIES]->(r)
OPTIONAL MATCH (t)-[:PRODUCED]->(tr:TestRun)
WITH r, d, m, collect(tr.outcome) AS test_results
WHERE NOT 'Passed' IN test_results
RETURN r.id AS Requirement,
       d.id AS Design_Spec,
       collect(DISTINCT m.id) AS Implementing_Code,
       CASE 
         WHEN size(test_results) = 0 THEN 'NO_TESTS_EXIST'
         ELSE 'TESTS_FAILING_OR_STALE'
       END AS Verification_Deficit;

B. Full Upstream/Downstream Change Impact

When a design decision or interface contract changes (e.g., an architectural refactor), evaluate the blast radius across both the implementation and verification domains:

MATCH (d:DesignElement {id: 'ARCH-DES-042'})
OPTIONAL MATCH (r:Requirement)-[:SPECIFIED_BY]->(d)
OPTIONAL MATCH (d)-[:REALIZED_BY]->(m:Method)
OPTIONAL MATCH (t:TestCase)-[:EXECUTES]->(m)
RETURN d.id AS Altered_Design,
       collect(DISTINCT r.id) AS Upstream_Requirements_Affected,
       collect(DISTINCT m.id) AS Code_Files_To_Refactor,
       collect(DISTINCT t.id) AS Test_Suites_To_Invalidate;

Summary Checklist for Ingestion

Data Layer	Source System	Ingest Mechanism	Primary Role in Graph
Requirements	Jira / DOORS / Markdown	Metadata parser + Embeddings	Defines What must be achieved
Design	SysML v2 / ADR Markdown / OpenAPI	Text parser + Model AST	Defines How the system is structured
Code	Git Repositories	Tree-sitter AST + Call graph	Defines the Concrete execution logic
Tests	Pytest / JUnit XML / CI logs	JUnit XML parser + Coverage.py	Confirms Correctness & satisfaction

From <https://gemini.google.com/app/e069cfdd1d60245e>  Google Gemini (3.8 Flash)