Every defense strategy team, regional procurement agency, and industrial investor has tried prompting a general-purpose AI assistant or enterprise search terminal to map a sovereign supply chain. The result is always the same: off-the-shelf AI covers roughly 20% of the problem—and half of that 20% is wrong.
The finding: off-the-shelf language models hallucinate industrial geography and miss private sub-tier plants
When an organization asks a general-purpose assistant (such as ChatGPT, Claude, or Microsoft Copilot) or a public-market document terminal (such as AlphaSense or Hebbia) to map domestic suppliers for a patrol submarine, an armored vehicle fleet, or an over-the-horizon radar program, the model retrieves only what is explicitly written in public press releases, broker notes, and SEO-optimized marketing pages.
In sovereign defense manufacturing, the companies that issue press releases are almost exclusively the 5 to 10 Tier-1 prime contractors everybody already knows. The actual physical bottlenecks—the founder-owned precision machine shops, ASME nuclear valve fabricators, NADCAP surface-treatment facilities, and specialty elastomer compounders—are private, operate with minimal web presence, and file under generic commercial industrial codes.
Across head-to-head audits comparing general-purpose AI outputs against bottom-up, analyst-verified industrial graphs across 500+ manufacturers and 170+ defense sub-supply chains, uncalibrated AI workflows exhibited four recurring failure modes:
| Intelligence Architecture | Sub-Tier Private Plant Coverage | Claim & Geographical Accuracy | Auditability for Investment & Procurement Committees | Typical Time to Board-Grade Answer |
|---|---|---|---|---|
| General-Purpose AI Assistants (ChatGPT, Claude, Copilot) | ~15%–20% (repeats well-known Tier-1 primes and vocal startups) | Low (~50% error rate); invents market shares and assigns naval drydocking to landlocked provinces | None; outputs drift across runs and mingle sourced facts with unlabelled model inference | Instant draft, followed by weeks of manual rework |
| Financial Search Terminals (AlphaSense, Hebbia, Rogo) | ~25%–30% (strong on public equities and VDRs; blind to private sub-tier shops) | Moderate-to-High on public filings; misses 4.4x statutory registry revenue distortions | Citation-level only; retrieves document passages without reconciling engineering tolerances | 1 to 2 weeks of analyst assembly |
| Entity-Screening Registries (Exiger, Sayari, D&B) | ~35%–45% (maps corporate ownership trees and sanctions lists) | High on legal ownership; blind to shop-floor 5-axis CNC envelopes, weld certs, and lead times | Compliance-grade, but cannot evaluate commercial-to-defense pivot readiness or should-cost | Ongoing software license + internal team |
| 15Rock Analyst-Verified Industrial Graph | 90%+ of addressable Tier-1 to Tier-3 base (includes non-NAICS private plants via equipment proxies) | >99% verified; enforces hydrological/physical rules and separates Observed, Calculated, and Modeled claims | Decision-grade ledger; every claim linked to timestamped statutory, registry, or equipment proof | 5 to 15 business days (scoped end-to-end delivery) |
The framework: Deterministic Value-Chain Engineering + Verified Evidence Ledgers
Crossing from a 20% first draft to a 100% board-ready or tender-ready supply-chain map requires replacing open-ended text generation with a four-layer engineering and verification architecture:
- Layer 1 — Deterministic Engineering Decomposition (Before Any Search): Never ask an AI model “Who are the defense suppliers in Region X?” First, decompose the physical military asset across its four lifecycle stages into 150 to 200 component sub-supply chains, specifying the exact metallurgical grades, machining envelopes, and certification standards (AS9100D, ASME Section III, Controlled Goods) required at each node.
- Layer 2 — Multi-Signal Shop-Floor Proxy Triangulation: To discover the 80%+ of private Tier-2 and Tier-3 manufacturers that never publish defense marketing copy, query indirect empirical footprints: machine-tool OEM installation records, coordinate-measuring machine (CMM) calibrations, specialized CNC and non-destructive testing (NDT) job requisitions, environmental permits, and customs trade ledgers.
- Layer 3 — Physical, Hydrological, and Corporate-Hierarchy Calibration: Apply hard deterministic rules that prevent model hallucinations:
- Geographical realism: Deep-water hull maintenance and submarine drydocking cannot be allocated to landlocked jurisdictions; inland clusters are mapped to their true structural strengths (avionics, radar signal processing, energetics, precision valves, and composite structures).
- Plant-vs-headquarters separation: Strip out post-office boxes, sales offices, and out-of-region parent consolidations so only verified physical manufacturing floors enter the capability ledger.
- Layer 4 — Explicit Epistemic Labelling & Human Acceptance Gates: Every cell in a decision matrix must carry a permanent provenance tag—distinguishing Observed statutory facts, Calculated ratios, and Modeled engineering estimates—reviewed by domain analysts before a single recommendation reaches an investment committee or procurement board.
The trap: mistaking fluent prose for verified industrial capacity
The danger of general-purpose AI in defense intelligence is not that it produces gibberish; it is that it produces highly fluent, plausible-sounding tables that collapse under the first engineering question.
In recent buyer evaluations, executive teams noted that off-the-shelf language models routinely fabricated competitor market-share percentages to the decimal point, confused academic research labs with commercial production facilities holding security clearances, and produced different supplier shortlists when the exact same prompt was run forty-eight hours later.
The tell: Ask the author of an AI-generated defense market map to show the underlying evidence ledger for three private Tier-2 suppliers on the page—specifically their physical plant address, verified CNC or clean-room envelope, current quality certifications, and the exact remediation step required for them to bid on a prime work package. If the answer relies on company website taglines, the map is a 20% draft.
The decision agenda: what to do differently
- For Corporate Development & Strategy Leaders: Stop asking internal analysts to stitch together ChatGPT prompts and NAICS database exports for high-stakes board decisions. Use general assistants for internal drafting, but require an audited, component-level evidence ledger before committing capital or bid resources.
- For Defence Primes & Offset (ITB) Teams: Audit your supplier discovery pipeline for taxonomy and geography bias. Ensure your regional offset scans verify physical shop-floor equipment and separate commercial production plants from university labs and distribution brokers.
- For Private Equity & Credit Committees: Require every commercial and operational diligence pack to explicitly label Observed, Calculated, and Modeled claims—and reconcile commercial credit-bureau summaries against full statutory filings and physical plant headcount.
What could break this read
- Classified capability blackouts: Above secret and special-access security thresholds, neither public registries nor commercial equipment proxies can observe compartmented production lines; those nodes must be flagged explicitly as sovereign classified exclusions rather than guessed by a model.
- Rapid post-acquisition equipment re-tooling: Private machine shops undergoing private-equity roll-ups can add 5-axis or clean-room capacity faster than annual registry cycles update, requiring direct pre-engagement verification during final supplier down-selection.
The checklist
General-purpose AI summarizes what companies say about themselves in public; defense-grade industrial intelligence verifies what their machines can actually build under sovereign tolerances.
Sources
- 15Rock Competitive Intelligence & Buyer Verification Benchmarks (2026): Head-to-head evaluation of general-purpose LLMs, financial search terminals, entity-screening registries, and analyst-verified industrial graphs across 585+ manufacturing entities and 170+ defense sub-supply chains.
- National Institute of Standards and Technology (NIST) & CMMC / Controlled Goods Frameworks: Verification standards for defense industrial base cybersecurity and technical-data handling.
- UK Companies House, Statistics Canada & Provincial Corporate Registries: Empirical benchmarks on private-company filing exemptions and headquarters-vs-plant reporting variances.
All comparisons and case metrics in this article are derived from anonymized 15Rock research benchmarks and public regulatory standards. No client or private counterparty is named.
Frequently Asked Questions
Why do general-purpose LLMs only capture about 20% of a defense supply chain?
General-purpose LLMs are trained and grounded on publicly indexed web pages, news articles, and public-company filings. While Tier-1 defense primes publish extensive public material, more than 80% of Tier-2 and Tier-3 precision manufacturers are privately held, operate under generic commercial NAICS codes, and avoid publicizing customer relationships or machine-floor specifications online.
How does an industrial capability graph differ from an entity-screening tool like Exiger or Sayari?
Entity-screening registries excel at mapping corporate ownership trees, sanctions exposure, and beneficial ownership for compliance teams. An industrial capability graph maps engineering and operational reality—decomposing weapon systems into sub-assembly tolerances, verifying which private plants own the required 5-axis mills, clean rooms, or nuclear certifications, and quantifying should-cost and conversion timelines.
What does it mean to separate Observed, Calculated, and Modeled claims?
It is an evidentiary standard where Observed claims cite a direct primary source (such as an AS9100D certificate number or statutory filing), Calculated claims apply transparent arithmetic to observed data (such as a 35.37% multi-year capital lapse rate), and Modeled claims represent explicit engineering or financial estimates (such as plant revenue capacity inferred from verified CNC spindle counts and headcount).