Every Friday at 3:30 PM Eastern, the U.S. Commodity Futures Trading Commission publishes the Commitments of Traders report. It contains the full breakdown of open interest across 400+ futures markets — hedge funds, commercial hedgers, swap dealers, non-reportable traders. Who is long, who is short, how many contracts, how concentrated. It is one of the most actionable public datasets in finance. And it is delivered as a ZIP file containing a 1990s-era CSV with 300 columns and no documentation. This article documents the build: how we parsed, structured, and exposed 30 years of COT data as a clean REST API — with Z-scores, pressure metrics, COT Index, and divergence signals computed on the fly.

The CFTC publishes COT data in two formats: a viewable HTML table on their website, and a compressed annual ZIP containing a legacy XLS file. The XLS has 300+ columns with names like NonComm_Postions_Spread_All — note the typo in 'Positions', baked into the official dataset since 1986 and never corrected. There is no API. There is no JSON. There is no way to query 'give me the net Non-Commercial positioning on Gold for the last 5 years' without downloading multiple annual files, concatenating them, filtering by contract code, and computing the net yourself. This is the state of public financial data in 2026.

The bigger problem: the raw numbers are useless without context. Knowing that Non-Commercial traders hold 180,000 long contracts on Gold means nothing without knowing the historical range. Is that elevated? Is it near the all-time high? Has it been increasing for 4 weeks? The COT report requires normalization, and normalization requires history. The only way to do this properly is to ingest the full 30-year dataset and compute relative metrics at query time — which is exactly what this API does.

The database has five core models: Contract (398 markets, each with a CFTC contract code, commodity type, exchange, and region), OpenInterest (weekly positions for Non-Commercial, Commercial, and Non-Reportable traders), Traders (weekly count of distinct reporting traders per category), and Concentration (top 4 and top 8 trader concentration ratios — gross and net). Each OpenInterest record stores raw positions; derived metrics like net_non_comm, open_interest, and all percentage breakdowns are computed as model properties at read time, not stored.

The CFTC publishes annual ZIP files for the Legacy Futures Only format going back to 1986. The ingestion pipeline is a Django management command (sync_cftc) that downloads the relevant years, parses the XLS, and bulk-inserts only the records not already in the database. The key design choices: use report_date as a column name instead of _date (itertuples renames columns starting with underscore), filter on reported_date > latest_db_date for incremental updates, and use bulk_create with ignore_conflicts=True for idempotent reruns.

Four metrics drive the API's analytical value. The Z-score normalizes the net Non-Commercial position (long minus short) relative to its mean and standard deviation over a configurable window. A Z-score of +2 means positioning is 2 standard deviations above its historical mean — historically elevated. The Pressure metric normalizes the Z-score further: |z_current| / max(|z|) over the period, giving a 0-to-1 score where 1 means the market is at its most extreme relative positioning ever recorded in the window. The COT Index is the simplest: (current - min) / (max - min) × 100, a percentile rank. Above 80 means the most stretched long in the window; below 20 means the most stretched short. The Divergence metric tracks the gap between net Non-Commercial and net Commercial — when speculators are maximum long and hedgers are maximum short, that structural tension is historically mean-reverting.

The API exposes 25+ endpoints organized in four layers: reference data (contract list, types, search), raw historical data (open interest, traders, concentration series), per-contract analytics (Z-score series, pressure series, COT index, divergence, full summary), and cross-universe analytics (Z-score ranking across all mainstream contracts, extremes above a threshold, active signal detection). Authentication uses a custom API key header (X-API-KEY) with per-day rate limiting tracked in the database — no Redis required. The mainstream flag on each Contract filters the universe to the ~55 most liquid markets, eliminating noise from obscure basis swaps and regional electricity contracts.

The chart below shows where every mainstream futures contract currently sits on its Z-score — computed since inception on each contract's full available history. The red dot marks the historical minimum (most extreme short positioning ever recorded), the green dot marks the historical maximum (most extreme long). The dark bar is the current Z-score. The light bar is the Z-score from the previous week. This gives full context: not just where positioning is, but where it has been, and whether it is moving toward or away from an extreme.

The current API covers the Legacy Futures Only report — Non-Commercial, Commercial, Non-Reportable. The next build adds the Disaggregated format (Producer/Merchant, Swap Dealers, Managed Money, Other Reportables), which gives a sharper picture of who is actually driving the positioning moves. Managed Money — the COT category that most closely maps to systematic hedge funds — is a cleaner signal than the legacy Non-Commercial bucket, which mixes trend-followers with macro funds and option dealers. The COT Index and Z-score methodology transfers directly; the data model needs four new position categories per record.