Yesterday I worked on redesigning my ForeignAssistance.gov tracker. The version I showed you last month, which worked against this official platform's API, was absurdly slow—there was no other way. But I finally managed to download the 3.5 GB CSV containing the entire ForeignAssistance.gov dataset—updated through July of this year—and designed a new system, later developed with DeepSeek and Claude.
The performance gain is overwhelming, particularly on the main problem I targeted in the previous version: detecting Cuba-related mentions when the island does not appear as the recipient country of certain foreign assistance activity. As I mentioned back then, some of these references appeared attached to countries as distant as Afghanistan. In contrast, others are filed under "World" or "Latin America and the Caribbean". Some activities tagged for Venezuela are meant to affect Cuba as well.
If you go by ForeignAssistance.gov's native search interface, you can't solve this at scale because there's no bulk keyword search across the whole dataset. Everything revolves around country. So knowing this pattern exists, I would have had to download or query every single country's data to find stray or "hidden" Cuba references buried inside. With the CSV, all of that gets resolved in one very short, computationally cheap pass. A first useful filter for this kind of search is a plain word-boundary regex:
CUBA_TEXT_PATTERN = re.compile(r"\bCuban?s?\b", flags=re.IGNORECASE)
Now, the CSV itself is quite scattered — it doesn't condense anything. A single activity (that is, a single award) can have many separate entries, so I designed the pipeline to distinguish, for each activity, between obligations and disbursements, summing each type separately to arrive at one concrete figure per transaction type:
df['is_obligation'] = df['Transaction Type Name'].str.upper().str.strip().str.contains('OBLIGAT', na=False)
df['is_disbursement'] = df['Transaction Type Name'].str.upper().str.strip().str.contains('DISBURS', na=False)
The result of that processing gets presented in an HTML report. On top of this, I prepared another script that now reads from the API, in a targeted way, to update the data as new information is added.
This will let me, down the road, to cross-reference a gainst USASpending data—which will be considerably more complex, because I'm seeing that the Trump administration has been highly disruptive to these disclosure processes. Processes that actually have never been so transparent, least of all when it comes to the democracy programs run against Cuba, handled with the same secrecy you'd attribute to clandestine or covert operations.
Transparency infrastructure like this was never built for people doing the digging I'm doing; it was built to satisfy a statutory disclosure requirement, and it shows in every design choice—the country-centric search, the redacted activity names and descriptions "in accordance with the exceptions outlined in the Foreign Aid Transparency and Accountability Act of 2016," the total absence of a bulk-search option. Getting past those limits implies building our own tools rather than relying on what the platform offers in its frontend.
Thanks for your reading.