#4956 DARPA Data Analysis
#4956 DARPA Data Analysis
- Are there online tools to search and analyze DARPA projects
- Provide a Python script to search and extract DARPA contracts, awards, and solicitations using the USAspending and SAM.gov APIs.
- How can I discover hidden or non-obvious relationships between DARPA project needs?
- Show me a Python script using pdfplumber or PyMuPDF to extract Project Codes, Accomplishments, and Planned Programs from DARPA RDT&E R-2 budget PDF documents into structured JSON.
- Show a Python script using the neo4j library to load this DARPA R-2 JSON output into a graph database linking Projects to Capabilities and Planned Programs.
- Show how to compute text embeddings using sentence-transformers and store them as Neo4j vector properties for hybrid semantic search.
- Show a Cypher query using vector cosine similarity to identify where one project's PlannedProgram is already solved by another project's demonstrated Capability.
- Show Cypher queries using Neo4j Graph Data Science (GDS) to project the CAN_REUSE_TECH_FROM relationships into an in-memory graph and execute Louvain community detection.
- Show Cypher queries using Neo4j GDS to compute FastRP node embeddings on the projected DARPA technology graph.
- Show how to concatenate FastRP structural graph embeddings with sentence-transformer narrative embeddings into a hybrid representation for UMAP dimensionality reduction and clustering in Python.
#4956 Are there online tools to search and analyze DARPA projects
Several specialized federal databases, technical repositories, and analytical platforms allow you to search, monitor, and analyze DARPA programs across their lifecycle—from initial solicitation to technical reports and spending records.
1. Solicitations & Program Formulation
SAM.gov (Contract Opportunities): The primary source for DARPA Broad Agency Announcements (BAAs), Requests for Information (RFIs), and Special Notices.
Search tip: Filter by Agency
DEPT OF DEFENSE--> Sub-tierDEFENSE ADVANCED RESEARCH PROJECTS AGENCY (DARPA).Analysis utility: Reading full BAA attachments reveals program architecture, technical metrics, phases, and the Heilmeier Catechism questions DARPA expects performers to answer.
Grants.gov: Covers DARPA research opportunities structured as grants, cooperative agreements, or educational partnerships rather than FAR-based acquisition contracts.
2. Official Agency Portfolios
DARPA Programs Directory (
darpa.mil/research/programs): Searchable index of active and completed programs filterable by technical office (BTO, DSO, I2O, MTO, STO, TTO) and research topic. Each program page outlines core operational concepts, participating Program Managers, and recent milestones.DARPA Open Catalog: Catalogs open-source software, datasets, and publications resulting from DARPA research (particularly heavily leveraged for information processing and data analytics programs).
3. Technical Deliverables & Literature Repositories
DTIC (Defense Technical Information Center -
discover.dtic.mil): The central clearinghouse for DoD scientific and technical information.Search utility: Contains unclassified final contractor technical reports, interim milestone summaries, and technical architecture docs produced by DARPA performers.
Query syntax: Filter results by issuing organization:
Defense Advanced Research Projects Agencyor look up specific BAA solicitation IDs.
Dimensions.ai & Lens.org: Index grant awards and map them to downstream academic papers, patents, and citations, which helps trace commercialization or academic diffusion of DARPA-funded work.
4. Financial Tracking & Award Analysis
USAspending.gov: Essential for tracking which universities, defense primes, and startups received DARPA funding, award amounts, sub-awards, and contract performance periods.
Analysis utility: Filter by Awarding Agency
Department of Defense$\rightarrow$ Awarding Sub-AgencyDefense Advanced Research Projects Agency. You can export raw CSV/JSON records to construct funding graphs and prime-to-subcontractor network maps.
DoD RDT&E Budget Justification Books (Comptroller Reports): Published annually via
comptroller.defense.govunder the R-2 Exhibits (Research, Development, Test & Evaluation).Analysis utility: Contains granular line-item descriptions of DARPA project funding across budget activities (6.1 Basic Research, 6.2 Applied Research, 6.3 Advanced Technology Development), including specific accomplishments from the prior fiscal year and planned goals for the upcoming year.
5. Small Business Innovation
SBIR.gov / DARPA Small Business Portal: Dedicated search engine for DARPA Phase I, II, and Direct-to-Phase-II Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) topics and historical awards.
2. Provide a Python script to search and extract DARPA contracts, awards, and solicitations using the USAspending and SAM.gov APIs.
This modular Python script queries both USAspending.gov (post-award contract and grant funding) and SAM.gov (pre-award solicitations, BAAs, and RFIs).
USAspending API: Completely open and requires no API key.
SAM.gov Opportunities API: Requires a free public API key obtained from your SAM.gov account profile.
import os
import json
import requests
from typing import List, Dict, Any, Optional
from datetime import datetime, timedelta
# =====================================================================
# 1. USAspending.gov Client (Contracts & Awards)
# Endpoint: /api/v2/search/spending_by_award/ (No auth required)
# =====================================================================
class USAspendingClient:
BASE_URL = "https://api.usaspending.gov/api/v2"
def fetch_darpa_awards(
self,
fiscal_year: int = 2026,
award_types: Optional[List[str]] = None,
limit: int = 50,
page: int = 1
) -> List[Dict[str, Any]]:
"""
Extracts DARPA awards.
Award Types:
- ['A', 'B', 'C', 'D'] for FAR contracts (BPA calls, Purchase Orders, Definitive Contracts)
- ['02', '03', '04', '05'] for Grants & Cooperative Agreements
"""
if award_types is None:
award_types = ["A", "B", "C", "D"]
url = f"{self.BASE_URL}/search/spending_by_award/"
# FY start and end dates
start_date = f"{fiscal_year - 1}-10-01"
end_date = f"{fiscal_year}-09-30"
payload = {
"filters": {
"time_period": [{"start_date": start_date, "end_date": end_date}],
"award_type_codes": award_types,
"agencies": [
{
"type": "awarding",
"tier": "subtier",
"name": "Defense Advanced Research Projects Agency",
"toptier_name": "Department of Defense"
}
]
},
"fields": [
"Award ID",
"Recipient Name",
"Award Amount",
"Total Outlays",
"Description",
"Action Date",
"Awarding Agency",
"Awarding Sub Agency",
"Contract Award Type"
],
"page": page,
"limit": limit,
"sort": "Award Amount",
"order": "desc"
}
headers = {"Content-Type": "application/json"}
response = requests.post(url, json=payload, headers=headers, timeout=30)
response.raise_for_status()
data = response.json()
return data.get("results", [])
# =====================================================================
# 2. SAM.gov Opportunities Client (BAAs, Solicitations, RFIs)
# Endpoint: /prod/opportunities/v2/search (Requires API Key)
# =====================================================================
class SAMOpportunitiesClient:
BASE_URL = "https://api.sam.gov/prod/opportunities/v2/search"
def __init__(self, api_key: Optional[str] = None):
self.api_key = api_key or os.getenv("SAM_API_KEY")
def fetch_darpa_opportunities(
self,
days_back: int = 60,
limit: int = 25
) -> List[Dict[str, Any]]:
"""
Searches active solicitations and notices published by DARPA.
Date format required by SAM.gov: MM/dd/yyyy
"""
if not self.api_key:
print("[Warning] No SAM_API_KEY provided. Skipping SAM.gov search.")
return []
now = datetime.now()
posted_from = (now - timedelta(days=days_back)).strftime("%m/%d/%Y")
posted_to = now.strftime("%m/%d/%Y")
params = {
"api_key": self.api_key,
"postedFrom": posted_from,
"postedTo": posted_to,
"deptname": "DEPT OF DEFENSE",
"subtier": "DEFENSE ADVANCED RESEARCH PROJECTS AGENCY (DARPA)",
"limit": limit
}
headers = {"Accept": "application/json"}
response = requests.get(self.BASE_URL, params=params, headers=headers, timeout=30)
response.raise_for_status()
data = response.json()
return data.get("opportunitiesData", [])
# =====================================================================
# Execution & Normalization Pipeline
# =====================================================================
def main():
# 1. Fetch USAspending Awards
print("Fetching top DARPA contracts from USAspending.gov...")
usaspending = USAspendingClient()
try:
awards = usaspending.fetch_darpa_awards(fiscal_year=2026, limit=10)
print(f"Retrieved {len(awards)} award records.\n")
for award in awards[:5]:
print(f"Award ID: {award.get('Award ID')}")
print(f"Recipient: {award.get('Recipient Name')}")
print(f"Amount: ${award.get('Award Amount', 0):,.2f}")
print(f"Description: {award.get('Description')[:110]}...")
print("-" * 60)
except Exception as e:
print(f"Error querying USAspending: {e}")
# 2. Fetch SAM.gov Solicitations (BAAs / Presolicitations)
print("\nFetching recent DARPA solicitations/BAAs from SAM.gov...")
sam_key = os.getenv("SAM_API_KEY") # export SAM_API_KEY="your_key_here"
sam = SAMOpportunitiesClient(api_key=sam_key)
if sam_key:
try:
opps = sam.fetch_darpa_opportunities(days_back=90, limit=10)
print(f"Retrieved {len(opps)} opportunities.\n")
for opp in opps[:5]:
print(f"Solicitation #: {opp.get('solicitationNumber', 'N/A')}")
print(f"Title: {opp.get('title')}")
print(f"Type: {opp.get('type')}")
print(f"Posted Date: {opp.get('postedDate')}")
print(f"UI Link: {opp.get('uiLink')}")
print("-" * 60)
except Exception as e:
print(f"Error querying SAM.gov: {e}")
else:
print("To enable SAM.gov search, set the SAM_API_KEY environment variable.")
if __name__ == "__main__":
main()
Key Query Parameters & Architecture Notes
USAspending Agency Hierarchy: DARPA is classified as Sub-Tier Agency
Defense Advanced Research Projects Agencyunder Top-TierDepartment of Defense.Award Types: FAR-based prime procurement contracts map to
["A", "B", "C", "D"], while university research grants, fellowship agreements, and cooperative agreements map to["02", "03", "04", "05"].Rate Limits: USAspending endpoints allow broad analytical queries without authentication;
SAM.gov enforces daily quotas depending on whether the public account is unregistered/basic (10 calls/day) or entity-authorized (1,000 calls/day).
3. How can I discover hidden or non-obvious relationships between DARPA project needs?
Discovering non-obvious relationships across DARPA programs requires looking past surface-level agency offices (e.g., DSO, I2O, MTO) and keyword taxonomy. DARPA’s structure inherently silos technical thrusts even when multiple programs are solving the same underlying mathematical, materials, or architectural bottlenecks.
The most effective methodology is to construct a computational pipeline combining multi-source document ingestion, semantic graph modeling, and network topology analysis.
1. Ingest Across the Complete Document Lifecycle
To uncover latent links, analyze text across three distinct lifecycle layers rather than relying solely on active website summaries:
Upstream Intent (DoD R-2 Budget Books): Download the Research, Development, Test & Evaluation (RDT&E) budget justifications from the DoD Comptroller (
comptroller.defense.gov). Each R-2 exhibit contains explicit Accomplishments and Planned Programs text broken down by Project Code across Budget Activities 6.1 (Basic), 6.2 (Applied), and 6.3 (Advanced Technology). This is where DARPA explicitly articulates the operational problems they are trying to solve.Solicitation Architecture (SAM.gov BAAs): Ingest full Broad Agency Announcements and RFIs. The critical text is found under Section I (Funding Opportunity Description), specifically the Heilmeier Catechism breakdown (what is the current state of practice, why does it fail, what are the quantitative metrics of success).
Downstream Deliverables (DTIC Technical Reports): Search
discover.dtic.milfor performer final technical reports. Often, performers hit identical physical, thermodynamic, or algorithmic walls across unrelated projects.
2. Extract Latent Functional Primitives (Beyond Keywords)
Traditional keyword search fails because a biology program (BTO) and an RF communications program (MTO) describe equivalent physical mechanisms using totally divergent vocabularies.
Deconstruct document sections into structured functional tuples using an extraction schema:
Function / Transformation: What physical or computational transformation is required? (e.g., attenuate high-frequency thermal noise, extract weak signals from high-dimensional latent space).
Constraint / Bottleneck: What limits current solutions? (e.g., size-weight-power (SWaP), non-stationary drift, phase decoherence, memory bandwidth saturation).
Operating Regime: Environmental or physical parameters (e.g., sub-Kelvin, RF-denied, air-gapped, sub-wavelength).
3. Model as a Bipartite or Multi-Entity Knowledge Graph
Ingest these entities and documents into a graph database (such as Neo4j) to expose paths that are invisible in relational databases.
(:Program)-[:HAS_SOLICITATION]->(:BAA)
(:BAA)-[:TARGETS_BOTTLENECK]->(:Bottleneck)
(:BAA)-[:REQUIRES_TRANSFORMATION]->(:Function)
(:BAA)-[:OPERATES_IN]->(:Environment)
(:Program)-[:FUNDED_UNDER]->(:R2ProjectCode)
(:Award)-[:PERFORMED_BY]->(:Organization)-[:PERFORMS_ON]->(:OtherProgram)
Graph Mining Queries to Run:
Jaccard/Cosine Similarity on Shared Bottlenecks: Calculate projection links between programs that have zero direct citation or office overlap but share >= 2 fundamental bottleneck nodes.
Betweenness Centrality & Bridging Performers: Run community detection (Louvain or Leiden) and Betweenness Centrality on performer networks. Defense primes (Lockheed, Raytheon, Northrop) work on many projects, but specialized subcontractors, academic labs, and boutique engineering firms acting as bridge nodes between two disconnected clusters indicate shared underlying engineering challenges.
Triadic Closure on Shared R-2 Justifications: Query for BAA topics that trace back to different DARPA offices but share identical DoD modernization priority tags or common line items in Comptroller R-2 exhibits.
4. Semantic Dense Retrieval & Dual-Layer Embeddings
Use dense semantic embeddings rather than lexical BM25 search to detect synonymous problem spaces:
Chunking Strategy: Chunk text by structural sections (e.g., "Program Objectives", "Technical Area 1 Requirements", "Metrics Table") rather than arbitrary character lengths.
Asymmetric Embedding: Pass text chunks through dense embedding models fine-tuned on scientific and technical literature (e.g., Specter2, BGE-large, or local domain embedding models).
Cross-Office Nearest Neighbors: Filter your cosine similarity matrix to strictly exclude pairs within the same technical office:
Similarity(P_i, P_j) quad where quad Office(P_i) <> Office(P_j)When an Information Innovation Office (I2O) program's metric definition clusters tightly with a Microsystems Technology Office (MTO) program, you have found an unspoken cross-domain requirement (e.g., sparse graph computing on neuromorphic substrates).
5. Automated Co-Occurrence & Contradiction Triangulation
Feed extracted requirement pairs into an automated contradiction detector:
Group programs where Program A's desired output is Program B's primary operating bottleneck.
Example: A tactical optical communications program requiring a passive thermal management coating at sub-micron tolerances directly intersects an advanced manufacturing program developing roll-to-roll dielectric metamaterial surfaces, even if neither program mentions the other.
4. Show me a Python script using pdfplumber or PyMuPDF to extract Project Codes, Accomplishments, and Planned Programs from DARPA RDT&E R-2 budget PDF documents into structured JSON.
This parser uses PyMuPDF (fitz) and a line-based state machine to extract hierarchical R-2 budget data from DARPA RDT&E budget justification books.
Structural Strategy
DoD R-2 exhibits follow a strict layout:
Program Element (PE) & Project Headers: Look for
Appropriation / Budget Activity / P-1 Reg Item: ...,PE <Number> / <Name>, andProject: <Code> / <Name>.Accomplishments & Plans Block: Usually headed by
Title: Accomplishments/Planned Programsor simplyAccomplishments/Planned Programs.Sub-Item Anchors: Under accomplishments, projects delineate descriptions by fiscal years (e.g.,
FY 2024 Accomplishments:,FY 2025 Planned Programs:,FY 2026 Plans:) followed by narrative descriptions and monetary figures.
import fitz # PyMuPDF
import re
import json
from typing import List, Dict, Any, Optional
def parse_darpa_r2_pdf(pdf_path: str) -> List[Dict[str, Any]]:
doc = fitz.open(pdf_path)
# State tracking
programs = []
current_pe = None
current_pe_title = None
current_proj = None
current_proj_title = None
# Text buffers
current_section = None # "ACCOMPLISHMENTS" or "PLANNED"
current_fy = None
current_text_lines = []
# Compiled regular expressions for structural anchors
re_pe = re.compile(r"R-1\s+Line\s+Item\s+Number:\s*(\w+)|Program\s+Element\s*\(PE\):\s*([0-9A-Za-z]+)", re.I)
re_proj = re.compile(r"Project\s*\(Number/Name\):\s*([0-9A-Za-z_-]+)\s*/\s*(.+)", re.I)
re_fy_acc = re.compile(r"^(?:Description:\s*)?FY\s*(20\d\d)\s*(?:Accomplishments|Plans|Planned\s+Programs)?:\s*(.*)", re.I)
re_sub_title = re.compile(r"^Title:\s*(.+)", re.I)
def flush_entry():
nonlocal current_text_lines, current_section, current_fy
if current_proj and current_section and current_text_lines:
text_block = " ".join(line.strip() for line in current_text_lines if line.strip())
# Clean repeated whitespaces
text_block = re.sub(r"\s+", " ", text_block).strip()
entry = {
"fiscal_year": current_fy,
"category": current_section,
"text": text_block
}
if current_section == "ACCOMPLISHMENTS":
current_proj["accomplishments"].append(entry)
else:
current_proj["planned_programs"].append(entry)
current_text_lines = []
for page_num in range(len(doc)):
page = doc[page_num]
lines = page.get_text("text").split("\n")
for line in lines:
line_str = line.strip()
if not line_str:
continue
# Check for Project code definition
proj_match = re_proj.search(line_str)
if proj_match:
flush_entry()
p_code = proj_match.group(1).strip()
p_name = proj_match.group(2).strip()
# Check if this project already exists
proj_obj = next((p for p in programs if p["project_code"] == p_code), None)
if not proj_obj:
proj_obj = {
"program_element": current_pe,
"program_element_title": current_pe_title,
"project_code": p_code,
"project_title": p_name,
"accomplishments": [],
"planned_programs": []
}
programs.append(proj_obj)
current_proj = proj_obj
current_section = None
continue
# Detect Accomplishment or Planned Program anchor
acc_match = re_fy_acc.match(line_str)
if acc_match:
flush_entry()
year = acc_match.group(1)
remainder = acc_match.group(2).strip()
if "accomplish" in line_str.lower():
current_section = "ACCOMPLISHMENTS"
else:
current_section = "PLANNED"
current_fy = f"FY{year}"
if remainder:
current_text_lines.append(remainder)
continue
# Capture end-of-section terminators (Budget tables, Remarks, Milestones)
if re.match(r"^(C\.\s+Other\s+Program\s+Funding|D\.\s+Acquisition\s+Strategy|B\.\s+Accomplishments)", line_str, re.I):
flush_entry()
current_section = None
continue
# Accumulate narrative text inside the active section
if current_section and current_proj:
current_text_lines.append(line_str)
flush_entry()
return programs
if __name__ == "__main__":
pdf_input = "DARPA_R2_Budget_Justification.pdf"
try:
results = parse_darpa_r2_pdf(pdf_input)
# Save structured output
output_file = "darpa_r2_extracted.json"
with open(output_file, "w", encoding="utf-8") as f:
json.dump(results, f, indent=2)
print(f"Extracted {len(results)} projects to {output_file}")
if results:
print(f"Sample Project: {results[0]['project_code']} - {results[0]['project_title']}")
print(f"Accomplishments count: {len(results[0]['accomplishments'])}")
print(f"Planned programs count: {len(results[0]['planned_programs'])}")
except FileNotFoundError:
print(f"File {pdf_input} not found. Place an R-2 PDF in the script directory.")
Generated JSON Output Schema
[
{
"program_element": "0602715E",
"program_element_title": "MATERIALS AND BIOLOGICAL TECHNOLOGY",
"project_code": "MBT-01",
"project_title": "TACTICAL TECHNOLOGY MATERIALS DEVELOPMENT",
"accomplishments": [
{
"fiscal_year": "FY2025",
"category": "ACCOMPLISHMENTS",
"text": "Fabricated multi-layer optical phase plates and evaluated coherence loss at sub-Kelvin temperatures..."
}
],
"planned_programs": [
{
"fiscal_year": "FY2026",
"category": "PLANNED",
"text": "Scale aperture manufacturing process to continuous roll-to-roll substrates and integrate with tactical transceivers..."
}
]
}
]
5. Show a Python script using the neo4j library to load this DARPA R-2 JSON output into a graph database linking Projects to Capabilities and Planned Programs.
This Python script uses the official neo4j driver to batch-ingest the extracted DARPA R-2 JSON data into Neo4j. It establishes uniqueness constraints and uses idempotent MERGE patterns so you can rerun it across multiple budget cycles without creating duplicate nodes or edges.
Graph Data Model
Nodes:
(:ProgramElement {pe_number, title})(:Project {project_code, title})(:Capability {id, fiscal_year, text})(Accomplishments represent demonstrated technical capabilities)(:PlannedProgram {id, fiscal_year, text})(Planned activities represent targeted future thrusts)(:FiscalYear {year})
Relationships:
(:ProgramElement)-[:CONTAINS_PROJECT]->(:Project)(:Project)-[:DEMONSTRATED]->(:Capability)(:Project)-[:PLANS]->(:PlannedProgram)(:Capability)-[:ACHIEVED_IN]->(:FiscalYear)(:PlannedProgram)-[:TARGETS_YEAR]->(:FiscalYear)
Ingestion Script
import hashlib
import json
import os
from typing import Any, Dict, List
from neo4j import GraphDatabase, Driver
def compute_hash(project_code: str, fy: str, text: str) -> str:
"""Generate a deterministic surrogate key for narrative blocks."""
payload = f"{project_code}|{fy}|{text.strip()}".encode("utf-8")
return hashlib.sha256(payload).hexdigest()[:16]
def create_schema_constraints(driver: Driver, database: str = "neo4j") -> None:
"""Ensure idempotency and index lookups via uniqueness constraints."""
constraints = [
"CREATE CONSTRAINT IF NOT EXISTS FOR (pe:ProgramElement) REQUIRE pe.pe_number IS UNIQUE",
"CREATE CONSTRAINT IF NOT EXISTS FOR (p:Project) REQUIRE p.project_code IS UNIQUE",
"CREATE CONSTRAINT IF NOT EXISTS FOR (c:Capability) REQUIRE c.id IS UNIQUE",
"CREATE CONSTRAINT IF NOT EXISTS FOR (pl:PlannedProgram) REQUIRE pl.id IS UNIQUE",
"CREATE CONSTRAINT IF NOT EXISTS FOR (fy:FiscalYear) REQUIRE fy.year IS UNIQUE",
]
with driver.session(database=database) as session:
for query in constraints:
session.run(query)
print("Schema constraints validated.")
def prepare_records(raw_data: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
"""Transform nested JSON into normalized Cypher parameter structures."""
normalized = []
for item in raw_data:
proj_code = item.get("project_code")
if not proj_code:
continue
pe_num = item.get("program_element") or "UNKNOWN_PE"
pe_title = item.get("program_element_title") or "Unknown PE Title"
proj_title = item.get("project_title") or "Unknown Project Title"
# Transform accomplishments into discrete Capability records
capabilities = []
for acc in item.get("accomplishments", []):
fy = acc.get("fiscal_year", "FY_UNKNOWN")
text = acc.get("text", "")
if text:
capabilities.append({
"id": f"{proj_code}_CAP_{compute_hash(proj_code, fy, text)}",
"fiscal_year": fy,
"text": text,
})
# Transform planned programs into discrete PlannedProgram records
planned = []
for pl in item.get("planned_programs", []):
fy = pl.get("fiscal_year", "FY_UNKNOWN")
text = pl.get("text", "")
if text:
planned.append({
"id": f"{proj_code}_PLAN_{compute_hash(proj_code, fy, text)}",
"fiscal_year": fy,
"text": text,
})
normalized.append({
"pe_number": pe_num,
"pe_title": pe_title,
"project_code": proj_code,
"project_title": proj_title,
"capabilities": capabilities,
"planned_programs": planned,
})
return normalized
CYPHER_BATCH_INGEST = """
UNWIND $batch AS row
// 1. ProgramElement & Project
MERGE (pe:ProgramElement {pe_number: row.pe_number})
ON CREATE SET pe.title = row.pe_title
MERGE (p:Project {project_code: row.project_code})
ON CREATE SET p.title = row.project_title
ON MATCH SET p.title = row.project_title
MERGE (pe)-[:CONTAINS_PROJECT]->(p)
// 2. Capabilities (Past Accomplishments)
FOREACH (c_data IN row.capabilities |
MERGE (c:Capability {id: c_data.id})
ON CREATE SET
c.text = c_data.text,
c.fiscal_year = c_data.fiscal_year
MERGE (p)-[:DEMONSTRATED]->(c)
MERGE (fy_c:FiscalYear {year: c_data.fiscal_year})
MERGE (c)-[:ACHIEVED_IN]->(fy_c)
)
// 3. Planned Programs (Future Thrusts)
FOREACH (pl_data IN row.planned_programs |
MERGE (pl:PlannedProgram {id: pl_data.id})
ON CREATE SET
pl.text = pl_data.text,
pl.fiscal_year = pl_data.fiscal_year
MERGE (p)-[:PLANS]->(pl)
MERGE (fy_p:FiscalYear {year: pl_data.fiscal_year})
MERGE (pl)-[:TARGETS_YEAR]->(fy_p)
)
"""
def load_darpa_json_to_neo4j(
json_path: str,
uri: str = "bolt://localhost:7687",
user: str = "neo4j",
password: str = "password",
database: str = "neo4j",
batch_size: int = 50,
) -> None:
with open(json_path, "r", encoding="utf-8") as f:
raw_data = json.load(f)
records = prepare_records(raw_data)
total_records = len(records)
print(f"Prepared {total_records} project records for ingestion.")
driver = GraphDatabase.driver(uri, auth=(user, password))
try:
create_schema_constraints(driver, database=database)
# Batch execution
with driver.session(database=database) as session:
for i in range(0, total_records, batch_size):
batch = records[i : i + batch_size]
session.run(CYPHER_BATCH_INGEST, batch=batch)
print(f"Ingested batch {i // batch_size + 1} ({len(batch)} projects)...")
print("DARPA R-2 data ingestion complete.")
finally:
driver.close()
if __name__ == "__main__":
JSON_INPUT = "darpa_r2_extracted.json"
NEO4J_URI = os.getenv("NEO4J_URI", "bolt://localhost:7687")
NEO4J_USER = os.getenv("NEO4J_USER", "neo4j")
NEO4J_PASSWORD = os.getenv("NEO4J_PASSWORD", "password")
if os.path.exists(JSON_INPUT):
load_darpa_json_to_neo4j(
json_path=JSON_INPUT,
uri=NEO4J_URI,
user=NEO4J_USER,
password=NEO4J_PASSWORD,
)
else:
print(f"Input file not found: {JSON_INPUT}")
Verification & Graph Analysis Queries
Once ingested, run these Cypher queries in NeoDash or Neo4j Browser to inspect the graph topology:
1. Inspect the Project Hierarchy
MATCH (pe:ProgramElement)-[:CONTAINS_PROJECT]->(p:Project)
OPTIONAL MATCH (p)-[:DEMONSTRATED]->(c:Capability)
OPTIONAL MATCH (p)-[:PLANS]->(pl:PlannedProgram)
RETURN pe.pe_number, p.project_code, p.title, count(DISTINCT c) AS capabilities, count(DISTINCT pl) AS plans
ORDER BY capabilities DESC;
2. Detect Cross-Project Semantic Bridging Opportunities
MATCH (p1:Project)-[:DEMONSTRATED]->(c:Capability)
MATCH (p2:Project)-[:PLANS]->(pl:PlannedProgram)
WHERE p1 <> p2
RETURN p1.project_code AS SourceProject,
c.text AS AchievedCapability,
p2.project_code AS TargetProject,
pl.text AS TargetedThrust
LIMIT 10;6. Show how to compute text embeddings using sentence-transformers and store them as Neo4j vector properties for hybrid semantic search.
This pipeline uses sentence-transformers to compute dense vectors for narrative text nodes (Capability and PlannedProgram), writes them to Neo4j via the official driver, configures native vector and full-text indexes, and performs a Reciprocal Rank Fusion (RRF) hybrid semantic search in Cypher.
1. Vector Index & Schema Setup
Neo4j 5+ natively supports HNSW vector indexes alongside Lucene full-text indexes. The following Cypher statements configure both on dimension 768 (matching standard models like BAAI/bge-base-en-v1.5 or nomic-embed-text):
// 1. Vector index for Capability nodes
CREATE VECTOR INDEX capability_embeddings IF NOT EXISTS
FOR (c:Capability) ON (c.embedding)
OPTIONS {
indexConfig: {
`vector.dimensions`: 768,
`vector.similarity_function`: 'cosine'
}
};
// 2. Vector index for PlannedProgram nodes
CREATE VECTOR INDEX planned_embeddings IF NOT EXISTS
FOR (pl:PlannedProgram) ON (pl.embedding)
OPTIONS {
indexConfig: {
`vector.dimensions`: 768,
`vector.similarity_function`: 'cosine'
}
};
// 3. Full-text index for BM25 lexical keyword matching
CREATE FULLTEXT INDEX capability_fulltext IF NOT EXISTS
FOR (c:Capability) ON EACH [c.text];
2. Python Embedding & Batch Ingestion Script
This script reads nodes lacking an embedding, batches the inference on GPU/CPU, and updates the property via db.create.setNodeVectorProperty or direct array assignment.
import os
from typing import List, Dict, Any
from neo4j import GraphDatabase, Driver
from sentence_transformers import SentenceTransformer
MODEL_NAME = "BAAI/bge-base-en-v1.5" # 768-dim, dense retrieval
BATCH_SIZE = 64
def embed_and_store_neo4j(
driver: Driver,
model: SentenceTransformer,
label: str,
database: str = "neo4j"
) -> None:
"""
Finds nodes of a given label missing an embedding,
computes dense representations, and writes them back.
"""
# 1. Fetch unindexed nodes
fetch_query = f"""
MATCH (n:{label})
WHERE n.embedding IS NULL AND n.text IS NOT NULL
RETURN n.id AS id, n.text AS text
"""
with driver.session(database=database) as session:
records = session.run(fetch_query).data()
if not records:
print(f"No pending `{label}` nodes to embed.")
return
print(f"Computing embeddings for {len(records)} `{label}` nodes using {MODEL_NAME}...")
# 2. Batch inference
update_cypher = f"""
UNWIND $batch AS row
MATCH (n:{label} {{id: row.id}})
SET n.embedding = row.embedding
"""
for i in range(0, len(records), BATCH_SIZE):
batch = records[i : i + BATCH_SIZE]
texts = [b["text"] for b in batch]
# Normalize embeddings for cosine similarity
vectors = model.encode(texts, normalize_embeddings=True, show_progress_bar=False)
payload = [
{"id": item["id"], "embedding": vectors[idx].tolist()}
for idx, item in enumerate(batch)
]
with driver.session(database=database) as session:
session.run(update_cypher, batch=payload)
print(f"Ingested batch {i // BATCH_SIZE + 1} ({len(batch)} nodes)...")
print(f"Finished embedding `{label}` nodes.")
def main():
uri = os.getenv("NEO4J_URI", "bolt://localhost:7687")
user = os.getenv("NEO4J_USER", "neo4j")
password = os.getenv("NEO4J_PASSWORD", "password")
print("Loading embedding model...")
model = SentenceTransformer(MODEL_NAME)
driver = GraphDatabase.driver(uri, auth=(user, password))
try:
# Update both past capabilities and future plans
embed_and_store_neo4j(driver, model, "Capability")
embed_and_store_neo4j(driver, model, "PlannedProgram")
finally:
driver.close()
if __name__ == "__main__":
main()
3. Executing Hybrid Semantic Search (Vector + Full-Text RRF)
A production hybrid query runs vector approximate nearest neighbors ($k$-NN) and Lucene full-text BM25 in parallel, ranking the union using Reciprocal Rank Fusion (RRF) ($k = 60$). It then traverses outward into the surrounding graph context.
def hybrid_graph_search(
query_text: str,
driver: Driver,
model: SentenceTransformer,
top_k: int = 5,
database: str = "neo4j"
) -> List[Dict[str, Any]]:
# Generate dense vector for input query
query_vector = model.encode(query_text, normalize_embeddings=True).tolist()
hybrid_cypher = """
// 1. Vector Search
CALL db.index.vector.queryNodes('capability_embeddings', 20, $query_vector)
YIELD node AS vNode, score AS vectorScore
WITH collect({node: vNode, score: vectorScore}) AS vectorResults
// 2. Full-Text Search
CALL db.index.fulltext.queryNodes('capability_fulltext', $query_text, {limit: 20})
YIELD node AS ftNode, score AS textScore
WITH vectorResults, collect({node: ftNode, score: textScore}) AS textResults
// 3. Reciprocal Rank Fusion (RRF)
WITH vectorResults, textResults,
[i IN range(0, size(vectorResults)-1) | {node: vectorResults[i].node, rrf: 1.0 / (60.0 + i + 1)}] AS vRRF,
[j IN range(0, size(textResults)-1) | {node: textResults[j].node, rrf: 1.0 / (60.0 + j + 1)}] AS ftRRF
UNWIND (vRRF + ftRRF) AS entry
WITH entry.node AS c, sum(entry.rrf) AS combinedScore
ORDER BY combinedScore DESC
// 4. Graph Traversal Context Expansion
MATCH (pe:ProgramElement)-[:CONTAINS_PROJECT]->(p:Project)-[:DEMONSTRATED]->(c)
OPTIONAL MATCH (c)-[:ACHIEVED_IN]->(fy:FiscalYear)
RETURN
pe.pe_number AS ProgramElement,
p.project_code AS ProjectCode,
p.title AS ProjectTitle,
fy.year AS FiscalYear,
c.text AS CapabilityDescription,
combinedScore
LIMIT $top_k
"""
with driver.session(database=database) as session:
result = session.run(
hybrid_cypher,
query_vector=query_vector,
query_text=query_text,
top_k=top_k
)
return result.data()
Why This Hybrid Layout Outperforms Single-Index Search
Lexical Exact Matches: Finds specialized defense acronyms, military MIL-STDs, and program handles that semantic models dilute or tokenize into subwords.
Vector Semantic Paraphrase: Retrieves equivalent mathematical and physical concepts (e.g., matching dielectric phase control when the query asks about optical wavefront modulation).
Graph Grounding: Employs the graph topology (
[:CONTAINS_PROJECT],[:ACHIEVED_IN]) to restrict, elevate, or link the extracted results to adjacent planned programs in other offices without post-processing.
7. Show a Cypher query using vector cosine similarity to identify where one project's PlannedProgram is already solved by another project's demonstrated Capability.
This Cypher query leverages Neo4j's native vector index procedure (db.index.vector.queryNodes) inside a correlated subquery to evaluate every PlannedProgram against demonstrated Capability nodes, filtering for high cosine similarity across different project codes.
1. Vector Search & Cross-Project Resolution
// 1. Iterate over all planned programs with valid embeddings
MATCH (p1:Project)-[:PLANS]->(pl:PlannedProgram)
WHERE pl.embedding IS NOT NULL
// 2. Query the vector index for nearest past capabilities
CALL db.index.vector.queryNodes('capability_embeddings', 10, pl.embedding)
YIELD node AS capNode, score AS cosineSimilarity
// 3. Keep high-confidence semantic matches (e.g., >= 0.85)
WHERE cosineSimilarity >= 0.85
// 4. Resolve the project that already demonstrated this capability
MATCH (p2:Project)-[:DEMONSTRATED]->(capNode)
// 5. Exclude intra-project matches (ensure true cross-program overlap)
WHERE p1 <> p2
RETURN
p1.project_code AS BeneficiaryProject,
p1.title AS BeneficiaryTitle,
pl.fiscal_year AS PlannedFY,
pl.text AS FuturePlannedThrust,
round(cosineSimilarity, 4) AS CosineSimilarity,
p2.project_code AS SolutionProviderProject,
p2.title AS SolutionProviderTitle,
capNode.fiscal_year AS AccomplishedFY,
capNode.text AS ExistingDemonstratedCapability
ORDER BY CosineSimilarity DESC;
2. Materializing Transfer Edges with Metadata
To avoid recomputing these vector lookups during graph traversals or NeoDash dashboard rendering, materialize these matches as explicit directed relationships:
MATCH (p1:Project)-[:PLANS]->(pl:PlannedProgram)
WHERE pl.embedding IS NOT NULL
CALL db.index.vector.queryNodes('capability_embeddings', 5, pl.embedding)
YIELD node AS capNode, score AS cosineSimilarity
WHERE cosineSimilarity >= 0.85
MATCH (p2:Project)-[:DEMONSTRATED]->(capNode)
WHERE p1 <> p2
// Connect the target plan directly to the satisfying prior capability
MERGE (pl)-[r:POTENTIALLY_SATISFIED_BY]->(capNode)
ON CREATE SET
r.cosine_similarity = cosineSimilarity,
r.created_at = datetime()
ON MATCH SET
r.cosine_similarity = cosineSimilarity
// Create a high-level project-to-project portfolio bridge
MERGE (p1)-[bridge:CAN_REUSE_TECH_FROM]->(p2)
ON CREATE SET
bridge.max_similarity = cosineSimilarity,
bridge.match_count = 1
ON MATCH SET
bridge.max_similarity = CASE
WHEN cosineSimilarity > bridge.max_similarity THEN cosineSimilarity
ELSE bridge.max_similarity
END,
bridge.match_count = bridge.match_count + 1
RETURN count(r) AS MaterializedEdges;
3. Aggregating Top Tech-Transfer Opportunities
After materializing the relationships, run this query to rank projects by how much prior DARPA work they can immediately absorb:
MATCH (p1:Project)-[bridge:CAN_REUSE_TECH_FROM]->(p2:Project)
RETURN
p1.project_code AS ProjectNeedingSolutions,
p1.title AS ProjectNeedingSolutionsTitle,
count(p2) AS AvailableSourceProjects,
collect(DISTINCT p2.project_code) AS ProviderProjectCodes,
round(avg(bridge.max_similarity), 4) AS AvgSimilarityScore
ORDER BY AvailableSourceProjects DESC
LIMIT 10;8. Show Cypher queries using Neo4j Graph Data Science (GDS) to project the CAN_REUSE_TECH_FROM relationships into an in-memory graph and execute Louvain community detection.
These Cypher queries use the Neo4j Graph Data Science (GDS) library to construct an in-memory graph projection of the cross-project CAN_REUSE_TECH_FROM transfer network and run Louvain Community Detection using the max_similarity property as relationship weights.
1. In-Memory Graph Projection (gds.graph.project)
Because technology transfer bridges are directed (p1 needs tech from p2), projecting the relationship orientation as UNDIRECTED allows Louvain to detect functional clusters sharing common capability ecosystems regardless of who was the provider or beneficiary.
// Check and drop any existing projection with the same name
CALL gds.graph.drop('darpaTechTransferGraph', false) YIELD graphName;
// Project the Project nodes and CAN_REUSE_TECH_FROM weighted edges
CALL gds.graph.project(
'darpaTechTransferGraph',
'Project',
{
CAN_REUSE_TECH_FROM: {
type: 'CAN_REUSE_TECH_FROM',
orientation: 'UNDIRECTED',
properties: ['max_similarity']
}
}
)
YIELD graphName, nodeCount, relationshipCount, projectMillis;
2. Stream Louvain Communities (Validation & Inspection)
Run gds.louvain.stream using max_similarity as the weight property to inspect the modularity clusters before mutating the database.
CALL gds.louvain.stream('darpaTechTransferGraph', {
relationshipWeightProperty: 'max_similarity',
includeIntermediateCommunities: false
})
YIELD nodeId, communityId, intermediateCommunityIds
WITH gds.util.asNode(nodeId) AS p, communityId
RETURN
communityId,
count(p) AS clusterSize,
collect(p.project_code) AS projectCodes,
collect(p.title)[..3] AS sampleTitles
ORDER BY clusterSize DESC;
3. Write Cluster IDs Back to Node Properties
Mutate the persistent graph by writing the assigned community integer directly onto the Project nodes as techClusterId.
CALL gds.louvain.write('darpaTechTransferGraph', {
relationshipWeightProperty: 'max_similarity',
writeProperty: 'techClusterId'
})
YIELD communityCount, modularity, modularities, ranLevels;
4. Cluster Profile & Functional Theme Synthesis
Once written, this query aggregates the dominant demonstrated capabilities and planned thrusts inside each Louvain community, exposing cross-office thematic clusters:
MATCH (p:Project)
WHERE p.techClusterId IS NOT NULL
OPTIONAL MATCH (p)-[:DEMONSTRATED]->(c:Capability)
OPTIONAL MATCH (p)-[:PLANS]->(pl:PlannedProgram)
RETURN
p.techClusterId AS ClusterId,
count(DISTINCT p) AS TotalProjects,
collect(DISTINCT p.project_code) AS MemberProjects,
count(DISTINCT c) AS SharedCapabilities,
count(DISTINCT pl) AS SharedFuturePlans
ORDER BY TotalProjects DESC;
5. Memory Cleanup
Release the in-memory projection from RAM:
CALL gds.graph.drop('darpaTechTransferGraph')
YIELD graphName;9. Show Cypher queries using Neo4j GDS to compute FastRP node embeddings on the projected DARPA technology graph.
These Cypher queries use Neo4j Graph Data Science (GDS) to project the DARPA technology transfer network, configure and compute Fast Random Projection (FastRP) node embeddings, and write low-dimensional topological vectors directly onto Project nodes.
FastRP captures higher-order structural proximity across the graph by iteratively multiplying projected random feature vectors across edge neighborhoods, weighted by transfer strength (max_similarity).
1. Project the Technology Transfer Graph
Project Project nodes and CAN_REUSE_TECH_FROM relationships as undirected edges so information propagates across both technology providers and consumers.
// 1. Drop existing in-memory projection if present
CALL gds.graph.drop('darpaFastRPGraph', false) YIELD graphName;
// 2. Create the GDS in-memory projection
CALL gds.graph.project(
'darpaFastRPGraph',
'Project',
{
CAN_REUSE_TECH_FROM: {
type: 'CAN_REUSE_TECH_FROM',
orientation: 'UNDIRECTED',
properties: ['max_similarity']
}
}
)
YIELD graphName, nodeCount, relationshipCount, projectMillis;
2. Stream FastRP Embeddings (Validation & Tuning)
Run gds.fastRP.stream to verify embedding dimension, iteration weights, and cosine similarity weighting before persisting vectors to disk.
embeddingDimension: 64 or 128 dimensions preserves structural variance without memory overhead.iterationWeights: Controls the influence of multi-hop neighborhoods.[0.0, 1.0, 0.7, 0.4]downweights the node itself (hop 0) while emphasizing immediate and 2-to-3-hop transfer neighbors.relationshipWeightProperty: Usesmax_similarityto ensure heavily connected programs remain closer in vector space.
CALL gds.fastRP.stream('darpaFastRPGraph', {
embeddingDimension: 64,
iterationWeights: [0.0, 1.0, 0.7, 0.4],
relationshipWeightProperty: 'max_similarity'
})
YIELD nodeId, embedding
WITH gds.util.asNode(nodeId) AS p, embedding
RETURN
p.project_code AS ProjectCode,
p.title AS ProjectTitle,
size(embedding) AS Dimensions,
embedding[0..5] AS SampleComponents
LIMIT 10;
3. Mutate or Write FastRP Embeddings Back to Nodes
Persist the generated embeddings directly onto the persistent database nodes under the property fastRPEmbedding.
CALL gds.fastRP.write('darpaFastRPGraph', {
embeddingDimension: 64,
iterationWeights: [0.0, 1.0, 0.7, 0.4],
relationshipWeightProperty: 'max_similarity',
writeProperty: 'fastRPEmbedding'
})
YIELD nodePropertiesWritten, computeMillis, writeMillis;
4. Configure a Vector Index on Structural Embeddings
Create a native vector index on the FastRP embeddings to enable fast approximate nearest-neighbor (k-NN) retrieval across graph topologies:
CREATE VECTOR INDEX project_fastrp_index IF NOT EXISTS
FOR (p:Project) ON (p.fastRPEmbedding)
OPTIONS {
indexConfig: {
`vector.dimensions`: 64,
`vector.similarity_function`: 'cosine'
}
};
5. Detect Structural Whitespace & Unconnected Analogous Programs
Query the FastRP index to find pairs of projects that occupy identical structural niches (high FastRP cosine similarity) but have no direct transfer edge between them. These identify non-obvious portfolio whitespace and parallel investments across offices.
MATCH (p1:Project)
WHERE p1.fastRPEmbedding IS NOT NULL
CALL db.index.vector.queryNodes('project_fastrp_index', 5, p1.fastRPEmbedding)
YIELD node AS p2, score AS structuralSimilarity
// Filter: High topological alignment, distinct projects, and NO existing transfer link
WHERE p1 <> p2
AND structuralSimilarity >= 0.80
AND NOT (p1)-[:CAN_REUSE_TECH_FROM]-(p2)
RETURN
p1.project_code AS ProjectA,
p1.title AS ProjectATitle,
p2.project_code AS ProjectB,
p2.title AS ProjectBTitle,
round(structuralSimilarity, 4) AS StructuralNicheSimilarity
ORDER BY structuralSimilarity DESC
LIMIT 15;
6. Memory Cleanup
Release the in-memory projection from RAM:
CALL gds.graph.drop('darpaFastRPGraph')
YIELD graphName;10. Show how to concatenate FastRP structural graph embeddings with sentence-transformer narrative embeddings into a hybrid representation for UMAP dimensionality reduction and clustering in Python.
This pipeline extracts structural graph topology vectors (fastRPEmbedding) and semantic text embeddings from Neo4j, normalizes and weights each modality, concatenates them into a unified multimodal latent space, and applies UMAP for dimensionality reduction alongside HDBSCAN clustering.
Methodological Architecture
Topological Representation (X_graph in mathbb{R}^{d_1}): Derived from FastRP random walks over
CAN_REUSE_TECH_FROMedges, encoding multi-hop relational proximity and structural network roles.Semantic Representation (X_text in mathbb{R}^{d_2}): Derived from mean-pooled
BAAI/bge-base-en-v1.5dense vectors over associatedCapabilityandPlannedProgramnarratives, encoding domain vocabulary and physical mechanism descriptions.Harmonization & Normalization: Because raw vector magnitudes differ between graph random projections and transformer embeddings, both matrices are individually L_2-normalized before concatenation. An optional scalar balance parameter (alpha) adjusts the relative influence of graph topology versus narrative semantics:
z_hybrid = begin{bmatrix} alpha cdot {v_graph} / {Vert{}v_graph Vert{}_2} & (1 - alpha) cdot {v_text} / {Vert{}v_{\text{text}} Vert{}_2} end{bmatrix}
Python Script
import os
import numpy as np
import pandas as pd
from neo4j import GraphDatabase
import umap
import hdbscan
import matplotlib.pyplot as plt
import seaborn as sns
from typing import Tuple, List, Dict, Any
# =====================================================================
# 1. Neo4j Data Extraction
# =====================================================================
def extract_hybrid_vectors(
uri: str = "bolt://localhost:7687",
user: str = "neo4j",
password: str = "password",
database: str = "neo4j"
) -> pd.DataFrame:
"""
Extracts Project nodes with their FastRP structural vectors and
aggregates mean-pooled text embeddings across all linked Capabilities and Plans.
"""
query = """
MATCH (p:Project)
WHERE p.fastRPEmbedding IS NOT NULL
// Collect all semantic text embeddings attached to this project
OPTIONAL MATCH (p)-[:DEMONSTRATED]->(c:Capability)
WHERE c.embedding IS NOT NULL
WITH p, collect(c.embedding) AS capEmbeddings
OPTIONAL MATCH (p)-[:PLANS]->(pl:PlannedProgram)
WHERE pl.embedding IS NOT NULL
WITH p, capEmbeddings + collect(pl.embedding) AS allTextEmbeddings
// Filter projects that have both structural and semantic data
WHERE size(allTextEmbeddings) > 0
RETURN
p.project_code AS project_code,
p.title AS title,
p.techClusterId AS louvain_cluster,
p.fastRPEmbedding AS graph_embedding,
allTextEmbeddings AS text_embeddings
"""
driver = GraphDatabase.driver(uri, auth=(user, password))
with driver.session(database=database) as session:
records = session.run(query).data()
driver.close()
data = []
for r in records:
text_vectors = np.array(r["text_embeddings"])
# Mean pool narrative embeddings belonging to the project
mean_text_vector = np.mean(text_vectors, axis=0)
data.append({
"project_code": r["project_code"],
"title": r["title"],
"louvain_cluster": r["louvain_cluster"] if r["louvain_cluster"] is not None else -1,
"graph_vec": np.array(r["graph_embedding"], dtype=np.float32),
"text_vec": mean_text_vector.astype(np.float32)
})
df = pd.DataFrame(data)
print(f"Extracted {len(df)} aligned project records.")
return df
# =====================================================================
# 2. Vector Fusion & Harmonization
# =====================================================================
def build_hybrid_matrix(
df: pd.DataFrame,
alpha: float = 0.5
) -> np.ndarray:
"""
Normalizes and concatenates graph and semantic feature spaces.
alpha = 1.0 -> purely structural (FastRP)
alpha = 0.0 -> purely semantic (SentenceTransformers)
alpha = 0.5 -> equal contribution
"""
X_graph = np.vstack(df["graph_vec"].values)
X_text = np.vstack(df["text_vec"].values)
# L2-normalization along rows
norm_graph = np.linalg.norm(X_graph, axis=1, keepdims=True)
norm_graph[norm_graph == 0] = 1.0
X_graph_unit = X_graph / norm_graph
norm_text = np.linalg.norm(X_text, axis=1, keepdims=True)
norm_text[norm_text == 0] = 1.0
X_text_unit = X_text / norm_text
# Weight and concatenate
X_hybrid = np.hstack([
alpha * X_graph_unit,
(1.0 - alpha) * X_text_unit
])
print(f"Constructed hybrid matrix shape: {X_hybrid.shape} (Graph: {X_graph.shape[1]}d, Text: {X_text.shape[1]}d)")
return X_hybrid
# =====================================================================
# 3. UMAP Projection & Density-Based Clustering
# =====================================================================
def run_reduction_and_clustering(
X: np.ndarray,
n_neighbors: int = 15,
min_dist: float = 0.1,
min_cluster_size: int = 4
) -> Tuple[np.ndarray, np.ndarray]:
"""
Projects the high-dimensional hybrid space to 2D using UMAP
and detects dense portfolio niches using HDBSCAN.
"""
# Use cosine metric for high-dimensional fused vector spaces
reducer = umap.UMAP(
n_neighbors=n_neighbors,
min_dist=min_dist,
n_components=2,
metric="cosine",
random_state=42
)
embedding_2d = reducer.fit_transform(X)
# HDBSCAN clustering on the UMAP projection
clusterer = hdbscan.HDBSCAN(
min_cluster_size=min_cluster_size,
metric="euclidean",
cluster_selection_epsilon=0.05
)
cluster_labels = clusterer.fit_predict(embedding_2d)
return embedding_2d, cluster_labels
# =====================================================================
# 4. Visualization & Exploration
# =====================================================================
def plot_portfolio_landscape(
df: pd.DataFrame,
embedding_2d: np.ndarray,
cluster_labels: np.ndarray
) -> None:
df["umap_x"] = embedding_2d[:, 0]
df["umap_y"] = embedding_2d[:, 1]
df["hybrid_cluster"] = cluster_labels
plt.figure(figsize=(13, 8))
palette = sns.color_palette("tab10", n_colors=len(np.unique(cluster_labels)))
# Plot scatter points
scatter = sns.scatterplot(
x="umap_x",
y="umap_y",
hue="hybrid_cluster",
palette=palette,
data=df,
s=90,
alpha=0.85,
edgecolor="black",
linewidth=0.5
)
# Annotate project codes
for _, row in df.iterrows():
plt.text(
row["umap_x"] + 0.05,
row["umap_y"] + 0.05,
row["project_code"],
fontsize=8,
alpha=0.75,
weight="semibold"
)
plt.title("DARPA Portfolio Topology: Hybrid FastRP (Graph) + BGE (Semantic) Space", fontsize=14, weight="bold")
plt.xlabel("UMAP Dimension 1", fontsize=11)
plt.ylabel("UMAP Dimension 2", fontsize=11)
plt.legend(title="Hybrid Cluster (HDBSCAN)", bbox_to_anchor=(1.02, 1), loc="upper left")
plt.grid(True, linestyle="--", alpha=0.3)
plt.tight_layout()
plt.savefig("darpa_hybrid_landscape.png", dpi=300)
plt.show()
print("Portfolio visualization saved to darpa_hybrid_landscape.png.")
if __name__ == "__main__":
NEO4J_URI = os.getenv("NEO4J_URI", "bolt://localhost:7687")
NEO4J_USER = os.getenv("NEO4J_USER", "neo4j")
NEO4J_PASSWORD = os.getenv("NEO4J_PASSWORD", "password")
df_projects = extract_hybrid_vectors(uri=NEO4J_URI, user=NEO4J_USER, password=NEO4J_PASSWORD)
if len(df_projects) >= 5:
# Fuse with equal weight between structural connectivity and textual meaning
X_fused = build_hybrid_matrix(df_projects, alpha=0.5)
coords_2d, clusters = run_reduction_and_clustering(X_fused)
plot_portfolio_landscape(df_projects, coords_2d, clusters)
else:
print("Insufficient project vectors found in Neo4j to execute UMAP reduction.")
Tuning the Fusion Parameter (alpha)
alpha = 0.5 (Balanced Default): Clusters correspond to technology domains where programs actively share technical solutions or face identical structural bottlenecks.
High alpha (0.7 - 0.8): Emphasizes contract/organizational structure and multi-hop transfer hubs over vocabulary differences, pulling together programs that bridge disparate technical disciplines.
Low alpha (0.2 - 0.3): Prioritizes physics and engineering taxonomy over current transfer links, revealing unbridged projects that work on identical materials, frequencies, or algorithms.
From <https://gemini.google.com/app/418975476f2ce7b2> Google Gemini (3.8 Flash)
Comments
Post a Comment