← Recompete Radar
recompete radar · architecture notebook · nov 2026

Recompete Radar Schematics

Five views of the same system. Public procurement data is downloaded, loaded into Snowflake, modeled by dbt, scored, copied into a small Postgres database, and served by an API to the web app. Each view uses a different systems-engineering notation because each answers a different question about that path. The Petri net, lineage graph and score map are interactive.

ibd · internal block diagram

What talks to what

Solid arrows carry data, labeled with the script that moves it. Dashed arrows are control: what starts the work. The data plane is on top and the schedulers sit underneath.

ibd [System] Recompete Radar data path PUBLIC SOURCES SNOWFLAKE · SUNLIGHT_DB SERVING PEOPLE USASpendingaward-summary downloads USASpendingsubaward reports SAM.govnotices + entity registrations DHS APFS · retiredfrozen snapshot · 16 Oct 2026 GSA eLibrarycontract-vehicle lookups ingest_usaspending load_to_snowflake ingest_enrichment load_enrichment 20 agencies × 12 NAICS, contracts only (A–D) RAW RAW_ENRICHMENT DEV_REFERENCEseeds +PIPELINE_RUNS DEV_STAGING DEV_INTERMEDIATE DEV_MARTS DEV_ENRICHMENT dbt dbt seeds staging = views marts = tables enrichment = views sync_marts_to_neon sync_discovery_to_neon (manual --apply) Neon Postgresfree tier, 0.5 GB award table + mart_awards view 3 small marts discovery_signals no raw data SQL FastAPI · Fly.ioread API /summary /recompetes /vendors /agencies /signals export_outreach.pyCSV straight from marts JSON Next.js 14on Vercel agencies, vendors, signals, insights, contracts Usersbrowser + CSV SCHEDULERS · CONTROL GitHub Actionsrun on demand refresh.sh → dbt build → sync_marts_to_neon Airflow · local, DAGs paused sunlight_awards weekly Sun 02:00 ET sunlight_enrichment daily 06:00 ET sunlight_freshness daily 09:00 ET award refresh refresh, enrichment, rows in PIPELINE_RUNS

Why two databases

Snowflake holds everything: about 484k raw award rows plus every dbt model. Neon's free tier is 0.5 GB, so it only gets the tables the API reads. The award table goes over once, and mart_awards is recreated in Neon as a view over it so the same rows aren't stored twice.

Two ways to run a refresh

A GitHub Actions workflow runs the award refresh on demand. Airflow DAGs are built and tested locally for the weekly award refresh, nightly SAM.gov notices, monthly entity registrations and a daily freshness check. The times in the diagram are the schedules written into each DAG. The DAGs are paused, not running on an always-on scheduler. The two paths never run together, because both would run refresh.sh --replace against the same tables. The DHS APFS feed was retired on 30 Oct 2026, so its last snapshot is frozen.

pn · coloured Petri net · interactive

How a refresh runs

This is the sunlight_awards DAG modeled as a coloured Petri net. Circles are states, bars are steps, and dots are tokens. Each token carries a colour: an agency, a download outcome, or the dataset. An orange bar is enabled. Click it to fire, or press Run. Click any circle to see its tokens.

agency gets nothing 0.18%
Firing log
  1. Ready. 20 agency tokens queued, 4 download slots free.
Place · agencies queued
  • The slots place is a resource. xargs -P 4 in refresh.sh lets at most four USASpending downloads generate at once. A slot token goes back only when a download lands or gives up.
  • “download lands” hides a retry loop. Each agency tries up to six fiscal-year windows (2024, 2023, 2025, 2022, 2021, 2026), pausing 60s between them. Any one success is the full all-time history, so only six failures in a row leave an agency with nothing. The slider shows how rare that is.
  • ×20 is a barrier. The load waits for every agency to finish, whatever the outcome. --replace is per-agency safe: a red token means that agency keeps last week's rows instead of being emptied. The delete and COPY run in one transaction.
  • record_* never blocks. Each writes a row to PIPELINE_RUNS through record_run(), wrapped in || true. The GitHub Actions path runs the same chain without the record steps.
dbt · model lineage · click a node

How tables depend on each other

These are the 26 objects in the 17 Oct 2026 inventory, traced through the ref() and source() calls in each model. Click a node to trace everything upstream and downstream of it. The dashed orange edge is the dependency that runs outside dbt: fit_scores.py reads the backtest table and writes the weights seed, which is committed to the repo. This is a dated snapshot; the models have since evolved.

Nothing selected

Click a model. Row counts come from the 2026-10-17 inventory, not a live query.

source table seed (CSV in repo) staging view intermediate core mart table enrichment view copied to Neon
calc · scoring model

How a contract gets scored

USASpending never says which contract replaced which, so the project infers it. An ended award's follow-on is the award from the same office and PSC that starts within −180 to +365 days of the end date and is closest in size. Two logistic regressions are trained on ended awards (about 94k rows in int_score_backtest), then applied to live ones.

act [Scoring] train → apply mart_awardsevery award,one row each int_score_backtestended >$250K, 2012 – 1yr ago+ inferred follow-on fit_scores.py2 binned logistic modelsfit pre-2021, test after score_weights39 log-odds rowscommitted seed mart_recompete_candidatesPOP end −3 … +36 months live contracts joined to the same feature bins
follow-on probability f →
retention probability r →
incumbent_strength = 100 · r
recompete_score = 100 · f · (1 − r)
r = P(incumbent wins the follow-on | there is one)
f = P(a follow-on happens at all)
Hover the map. The score is highest in the top-left corner, where a follow-on is likely and the incumbent is weak.

The score combines the two fitted model outputs to rank contracts for follow-up; it is not a calibrated probability that a new vendor wins. Timing is left out on purpose, so filter on months_to_pop_end separately. The follow-on links are inferred. A placebo window (follow-ons 3–5 years out) measures how much of the signal is matching noise.

f_shareincumbent's share of that office + PSC over the trailing 5 years
f_rivalsdistinct vendors in that office + PSC
f_tenureprior awards the incumbent has there
f_solesole-source vs competed
f_vehiclecontract award type
f_dur_*period-of-performance length
f_size · f_pricingaward size bin, pricing type
stm · two clocks

How staleness is detected

pipelines/freshness.py keeps two separate clocks per source. Recorded is what the scheduler says happened, from PIPELINE_RUNS. Observed is what the data itself shows, from OBSERVED_AT on each record or LAST_ALTERED on each table. Each source becomes stale once it has clearly missed its cadence: SAM after 72 h, awards after 240 h, GSA after 1,440 h. Comparing the two clocks is what makes the check useful.

observed timestamp moved
observed timestamp unchanged
recorded success
healthy
The job ran and new data landed.
ran, fetched nothing
The task succeeded but no rows changed. This is the case the two-clock design exists to catch.
no recent record
loaded outside Airflow
Someone ran the script by hand, or CI did. The data is fine, but the audit trail missed it.
stale
Nothing ran, or the scheduler stopped. A separate daily freshness DAG flags this, because a check inside the same DAG would stop too.