Data Pipeline

From raw wire to a settled onchain contract

Ten deterministic stages across four layers. Every headline flows through the same path, every score is reproducible, every attestation is signed and posted to Arc.

Live10 stages4 layersEnd-to-end < 10s typical
P1

Ingestion Layer

Raw wire to structured event
  1. 01~5s
    Ingest
    Pull live headlines across geopolitics, rare earth, macro and crypto.
    NewsAPI · 4 buckets
  2. 02<50ms
    Normalize
    Reshape vendor payloads into one canonical event schema.
    Zod schema
  3. 03<10ms
    Dedupe
    Rolling djb2 hash plus URL fingerprint to suppress duplicates.
    djb2 + URL
P2

Intelligence Layer

LLM classification and risk scoring
  1. 04<5ms
    Prefilter
    Drop anything outside the four tracked narrative classes.
    Heuristic gate
  2. 05~1.2s
    Classify
    Route survivors to llama-3.3-70b for category and stage tagging.
    Groq · 70B
  3. 06<200ms
    Score
    Assign severity, confidence and contribution to the Global Risk Index.
    0-100 scale
P3

Forecast Layer

Falsifiable calls with calibration
  1. 07~800ms
    Predict
    Write a falsifiable forecast with a 48h resolution deadline.
    Deadline bound
  2. 08rolling
    Reflect
    Backtest prior calls onchain and adjust analyst calibration.
    Track record
P4

Settlement Layer

Onchain attestation and market settlement
  1. 09~3s
    Attest
    SHA-256 the event payload, sign the digest, post the attestation to Arc.
    Arc · SHA-256
  2. 10at T+48h
    Resolve
    Main agent referees the analyst duel and settles the event contract.
    Onchain payout