Five views of the same system. Public procurement data is downloaded, loaded into Snowflake, modeled by dbt, scored, copied into a small Postgres database, and served by an API to the web app. Each view uses a different systems-engineering notation because each answers a different question about that path. The Petri net, lineage graph and score map are interactive.
Solid arrows carry data, labeled with the script that moves it. Dashed arrows are control: what starts the work. The data plane is on top and the schedulers sit underneath.
Snowflake holds everything: about 484k raw award rows plus every dbt model. Neon's free tier is 0.5 GB, so it only gets the tables the API reads. The award table goes over once, and mart_awards is recreated in Neon as a view over it so the same rows aren't stored twice.
A GitHub Actions workflow runs the award refresh on demand. Airflow DAGs are built and tested locally for the weekly award refresh, nightly SAM.gov notices, monthly entity registrations and a daily freshness check. The times in the diagram are the schedules written into each DAG. The DAGs are paused, not running on an always-on scheduler. The two paths never run together, because both would run refresh.sh --replace against the same tables. The DHS APFS feed was retired on 30 Oct 2026, so its last snapshot is frozen.
This is the sunlight_awards DAG modeled as a coloured Petri net. Circles are states, bars are steps, and dots are tokens. Each token carries a colour: an agency, a download outcome, or the dataset. An orange bar is enabled. Click it to fire, or press Run. Click any circle to see its tokens.
xargs -P 4 in refresh.sh lets at most four USASpending downloads generate at once. A slot token goes back only when a download lands or gives up.--replace is per-agency safe: a red token means that agency keeps last week's rows instead of being emptied. The delete and COPY run in one transaction.PIPELINE_RUNS through record_run(), wrapped in || true. The GitHub Actions path runs the same chain without the record steps.These are the 26 objects in the 17 Oct 2026 inventory, traced through the ref() and source() calls in each model. Click a node to trace everything upstream and downstream of it. The dashed orange edge is the dependency that runs outside dbt: fit_scores.py reads the backtest table and writes the weights seed, which is committed to the repo. This is a dated snapshot; the models have since evolved.
Click a model. Row counts come from the 2026-10-17 inventory, not a live query.
USASpending never says which contract replaced which, so the project infers it. An ended award's follow-on is the award from the same office and PSC that starts within −180 to +365 days of the end date and is closest in size. Two logistic regressions are trained on ended awards (about 94k rows in int_score_backtest), then applied to live ones.
The score combines the two fitted model outputs to rank contracts for follow-up; it is not a calibrated probability that a new vendor wins. Timing is left out on purpose, so filter on months_to_pop_end separately. The follow-on links are inferred. A placebo window (follow-ons 3–5 years out) measures how much of the signal is matching noise.
| f_share | incumbent's share of that office + PSC over the trailing 5 years |
| f_rivals | distinct vendors in that office + PSC |
| f_tenure | prior awards the incumbent has there |
| f_sole | sole-source vs competed |
| f_vehicle | contract award type |
| f_dur_* | period-of-performance length |
| f_size · f_pricing | award size bin, pricing type |
pipelines/freshness.py keeps two separate clocks per source. Recorded is what the scheduler says happened, from PIPELINE_RUNS. Observed is what the data itself shows, from OBSERVED_AT on each record or LAST_ALTERED on each table. Each source becomes stale once it has clearly missed its cadence: SAM after 72 h, awards after 240 h, GSA after 1,440 h. Comparing the two clocks is what makes the check useful.