Work on Lagging Indicators

Deduplicating 40k SNAP news articles with MinHash LSH

Tue, 10 Feb 2026 00:00:00 +0000

I needed to identify and tag duplicate news articles from a Google Alerts RSS feed which filters the news through keywords while working on topic modeling of news coverage on the Supplemental Nutrition Assistance Program since summer 2024.

I also want to in the future group these on the surfaced SNAP News site I put together for these articles.

After reading it for around a year, I consider duplicates / near duplicates to be an overall rare, but noticeable phenomenon. This post quantifies that feeling, explores why it occurs, and discusses some implications for topic modeling.

Visualizing SNAP Performance Indicators

Mon, 22 Dec 2025 00:00:00 +0000

A slow pace, to be sure, but I have been adding to a dashboard of SNAP Performance Indicators over the past year, based on the SNAP data I’ve been cleaning and compiling over the past 2 years, and built with the Observable Framework library.

You can view the dashboard here.

No extremely tricky visualizations, but that is because I have found that these datasets that are simple tables by state each year mask the complexity of federal and state policy around implementing SNAP benefits. The dashboards become more and more text heavy, I realize, as I absentmindedly pull in context that I believe is valuable to know alongside each chart. Additionally, the recent changes to SNAP in 2025 have complicated any forward-looking projections.

Quickly visualizing some low-effort content on YouTube

Wed, 17 Dec 2025 00:00:00 +0000

Recently on YouTube I searched “population apportionment” to find that many of the recommended videos across different channels looked the same and appeared to be AI-generated, heavily automated, and extremely low effort. Of the 18 Recently Uploaded videos for the search, 10 of theme were all from this same “family” of channels: 2 to 3 minute, political-themed questions with an AI-generated man or women in the thumbnail, leading to an animated PowerPoint presentation.

How political are shutdown alerts?

Mon, 17 Nov 2025 00:00:00 +0000

If you visited a .gov website during the 2025 government shutdown, you may have noticed alert banners on the top of the page that indicate a suspension or interruption of normal operations for the agency.

Some agencies vary from the standard, however, and instead use this opportunity to deliver a political message.

From a cursory analysis, of the 248 shutdown banners I could find from about 1,150 agency websites, 92 were political to some extent (37%).¹

Well, let's just not measure food insecurity then

Mon, 03 Nov 2025 00:00:00 +0000

In September 2025, the Trump administration said it will no longer release the Household Food Security Reports. The USDA also put relevant employees on indefinite leave.

This post provides context to the published press release based on the USDA’s own 2025 Information Collection Request renewal for the Food Security Supplement to the Current Population Survey, hosted on Reginfo.gov.¹

These redundant, costly, politicized, and extraneous studies do nothing more than fear monger.

Is the survey redundant?

Debugging public data - county codes in Texas

Fri, 16 May 2025 00:00:00 +0000

If you sort a list of Texas counties by county name, the counties will not be in order by county number.
Texas Center for Healthcare Statistics

Every county in the United States has a FIPS code. These standards codes are usful for mapping and joining additional datasets.

Texas state agencies do not bother with these FIPS codes. Instead, each state agency publishes data using their own county codes. The county codes of one state agency may not match other agencies’ county codes. I recently spent a few hours extremely confused by this, because I used the formula published by the Texas Department of State Health Services on data published by the Texas Health and Human Services, resulting in incorrect data.

The Trump Admin's data visualizations need some work

Sun, 02 Mar 2025 00:00:00 +0000

The Trump Administration has a famously loose approach to data visualizations. If the data doesn’t match up with what you want, grab a marker and draw over the visualization until it does.

Source: New York Times article showing hand-drawn modifications to official NOAA weather map

The USDA released an announcement, USDA Invests Up To $1 Billion to Combat Avian Flu and Reduce Egg Prices, with two charts that made some questionable decisions worth discussing.

Questions on “How Congress Can Fix Food Stamps”

Sun, 22 Sep 2024 00:00:00 +0000

The National Review in July published a short opinion piece in July ‘24 entitled How Congress Can Fix Food Stamps. Sam Adolphsen, the author, argues that the cost of SNAP benefits is too high. He comments that “about 17.6 million food-stamp recipients are able-bodied adults, and about two-thirds don’t work,” which leads him to list a series of Republican-led, state-level policies that require SNAP recipients to participate in job training or employment programs. These policy decisions, Adolphsen argues throughout the article, are “common-sense”, “wildly popular among voters”, “turning a handout into the hand up that [SNAP benefits were] always supposed to be,” and are “protecting taxpayers while promoting work over welfare”.

Lessons learned on dashboard design for early-stage apps

Sat, 07 Sep 2024 00:00:00 +0000

What follows are a few takeaways from a recent app launch, where I had some successes and some things I had to fix. I also maintain dashboards for a wide set of clients on a different project that will be overhauled soon, and would like to apply these learnings during the redesign.

The three guiding business questions for initial design are:
- How are we getting users?
- What our users are finding value in when using the application?

Daily SNAP News Aggregator launched

Tue, 11 Jun 2024 00:00:00 +0000

Site is accessible here, and the accompanying GitHub repository can be accessed through that site as well. It is based on a Google Alerts RSS feed, built every ~6 hours with Jekyll and GitHub Actions.

Monthly commentary will be included on that site in the future. Immediate next steps are to add in more news aggregators as sources, to try and get articles that are missed. I would like to integrate Google Scholar as well, but it does not appear they offer an RSS feed or any simple method. Article display filtering is currently done on a keyword basis, which has been working fine, but would like to move to a classifier model.

April SNAP Updates

Sun, 05 May 2024 00:00:00 +0000

Note a slightly different format to topic, rather than state-based.

Snap Interview Waivers in some places expiring

I would be very curious to see any study performed exploring differences in program administration and case outcomes between while the waiver was in effect, and the before / after where the waiver is not in place. How has the speed of applications changed? Overpayment or underpayment rates? What do the caseworkers think about this change? I also came across this Reddit thread on the impact of interview waivers, but would like to see a structured case study.

Joint & conditional probabilities with pd.crosstab

Thu, 02 May 2024 00:00:00 +0000

pd.crosstab is one of those built-in functions in the Pandas API that I forget about routinely. I instinctively reached for df.groupby('x')['y'].count().unstack(), but when I wanted to normalize the values, it takes more and more steps to get where I wanted.

This was a nice straightforward overview of the pd.crosstab function. To document for myself, below, create a sample correlated DataFrame with integer columns ActiveUsers and CompletedProfile.

import pandas as pd
import numpy as np

# following code from Github Copilot
np.random.seed(0)
n = 1000 # Number of samples
p = 0.7 # Probability of True in the first column
rho = 0.8 # Correlation

col1 = np.random.choice([True, False], size=n, p=[p, 1-p])
col2 = np.where(col1, np.random.choice([True, False], size=n, p=[rho, 1-rho]),
 np.random.choice([True, False], size=n, p=[1-rho, rho]))

df = pd.DataFrame({'ActiveUsers': col1, 'CompletedProfile': col2})

	ActiveUsers	CompletedProfile
0	True	True
1	False	False
2	True	False
3	...	...

Sample of the constructed DataFrame

Identifying outbound IP addresses of a Azure Function

Wed, 01 May 2024 00:00:00 +0000

Here is Microsoft’s documentation page on the topic. The only reason I am documenting this is because the Azure Resource Explorer failed to load consistently across devices for me. Some tutorials I found, like this one, seem to be outdated and the outbound IP addresses weren’t listed in any Properties menu in the Function App in the Azure Portal. The Azure CLI method was what worked for me:

az functionapp show --resource-group <GROUP_NAME> --name <APP_NAME> --query outboundIpAddresses --output tsv
az functionapp show --resource-group <GROUP_NAME> --name <APP_NAME> --query possibleOutboundIpAddresses --output tsv

I wanted this as I was hoping to whitelist the IP addresses in the database firewall to allow for scheduling some SQL scripts to run daily. The issue with this is the following, copied from the above linked Microsoft documentation: “because of autoscaling behaviors, the outbound IP can change at any time when running on a Consumption plan or in a Premium plan.” While at least a few people say whitelist these IPs anyway, the recommended course of action is to set up a virtual network within Azure.

Questions on "Why We Should Rethink the SNAP Interview Requirement: A Former Caseworker’s Perspective"

Tue, 09 Apr 2024 00:00:00 +0000

Context

Student and supporting organizations wrote a letter to the USDA recommending that the SNAP application interview requirement be removed, with the accompanying press release here. A news article that summarizes the entire P.R. push for this is accessible here through Fortune magazine.

Why We Should Rethink the SNAP Interview Requirement: A Former Caseworker’s Perspective is an op-ed from The Center for Law and Social Policy (CLASP) which argues SNAP interviews should be made optional. Based on the author’s experience as a caseworker, the mandatory interviews are largely a pretense for detecting fraud, and the time could be better spent providing holistic assistance to SNAP participants. These “unnecessary interviews that are ineffective at finding what infinitesimal fraud exists” only serve to increase the stigma associated with accessing food stamps by treating applicants as if they “are thieves that must be watched”.¹ Unfortunately, the lack of any evidence for or against the effectiveness of SNAP interviews (whatever their purpose may be), the confusing use of statistics, and the wider focus on stigmas associated with SNAP participation resulted in an unconvincing argument. While I agree with the author’s wider points that we should take steps to minimize stigmas of government benefits and that a primary purpose of mandatory SNAP interviews is to verify information, further study of the consequences is warranted before SNAP interviews are made optional.

SNAP Benefits News - March 2024

Wed, 03 Apr 2024 00:00:00 +0000

Summary:

many warnings about EBT payment scams and theft, alongside a proposal to improve security
state bills (and national) continue to propose work requirements and refuse additional SNAP funding for the summer
upcoming Farm Bill renewal is stalled by SNAP discussions and funding

National

Jobs and Opportunities for SNAP Act of 2023 proposed by Senator John Kennedy in the U.S. Senate. The companion U.S. House bill is here. The following is copied from the summary:

The Cartoon Guide to Statistics

Wed, 03 Apr 2024 00:00:00 +0000

Written by Larry Gonick & Woollcott Smith

Note: This is based on the “Revised and Updated Edition”.

Cover page of The Cartoon Guide to Statistics

Deriving, and learning when to apply, statistical formulas is always going to be difficult. It turns out that including cartoons does not make it any easier.

I would put statistics writing into three general categories: teaching statistical literacy, e.g., How To Lie With Statistics; deriving the principles of statistical concepts, e.g., any college-level textbook; and applying statistical methods, e.g., an academic study. The Cartoon Guide To Statistics tries to do a bit of all three of these, and is worse off for it. The guide covers probability, random variables, sampling, confidence intervals, hypothesis testing, experimental design, and regression in just over 200 pages.

How To Lie With Statistics

Mon, 04 Mar 2024 00:00:00 +0000

Written by Darrell Huff and illustrated by Irving Geis. Originally published in 1954.

In a sentence: a quick, worthwhile general knowledge read that explains how people misuse statistics through familiar and useful examples (that could use an update).

I’ll face up to the serious purpose that I like to think lurks just beneath the surface of this book: explaining how to look a phony statistic in the eye and face it down; and no less important, how to recognize sound and usable data in that wilderness of fraud…

SNAP Benefits News - Feb 2024

Sat, 02 Mar 2024 00:00:00 +0000

The following is a curated list of news articles by U.S. state and nationally that discussed SNAP benefits in February 2024.

National

Letter from Secretary Vilsack to Governors on SNAP Performance and Operations
- Poor performance on the SNAP efficiency measures meant 44 states got letters from the USDA expecting improvements.
Congressional leaders agree on ag funding as Vilsack talks about SNAP pilot
- The SNAP pilot they discuss is the SNAP-choice pilot, limiting purchasable foods to “nutrient-dense” foods. Unsurprising that a paternalistic policy with the name “SNAP-choice” is being pushed by a Republican. See the next article for the industry’s response.

Viva Insights - How are you feeling?

Sat, 02 Mar 2024 00:00:00 +0000

Microsoft Teams provides an integration with Microsoft Viva, which schedules daily check-ins to record your perception of your mental state. Once a day, at 4pm, I get a notification that asks me to select one of five emojis that reflects how I feel.

Clearly, the last few weeks have not been particularly excellent, but I’m not sure what the value of this type of check-in or chart provides. We can come up with reasonable estimates of our own satisfaction with our work. I frequently journal, and I believe that the self-reflection forced through written inquiries gives way to a better sense of being, rather than an even shorter point-in-time indicator with almost no ambiguity.

SNAP State Performance Indicators Datasets

Wed, 21 Feb 2024 00:00:00 +0000

I started to post SNAP performance indicators from the USDA in machine-readable format, aggregating and cleaning the data from the many PDF tables. This repository includes quick-access to cleaned, FIPS-linked CSV long-format tables of yearly state reported indicators for Application Processing Timeliness, Program Error Rates, and Case and Procedural Error Rates (coming soon).

USDA Descriptions as follows:

Application Processing Timeliness “measures the timeliness of states’ processing of initial SNAP applications. The Food and Nutrition Act of 2008 entitles all eligible households to SNAP benefits within 30 days of application, or within 7 days, if they are eligible for expedited service.”

Using Observable Framework on Windows WSL and quickly deploying to GitHub Pages

Fri, 16 Feb 2024 00:00:00 +0000

If you’ve got the framework installed, and just want to deploy the static pages, skip to below. Note - my first time doing this, but it works. You can always deploy directly to Observable.

Observable Framework is not supported on Windows, according to an issue in their GitHub. (Update Mar 2024: Observable Framework is now supported on Windows). To use it, you need to use WSL.

If you want to use it through WSL, then the following still applies. Fortunately, it is very easy to do. Steps I had to take to do this:

Monday links

Mon, 12 Feb 2024 00:00:00 +0000

- Whenever I do analyses I am nervous about being wrong, but using Excel’s autofill function to impute data would have definitely been on my “this is not correct to do,” at the very least.

- Nice example of a personal project. Contained, clear output, and interesting to the person doing it + delightful to play with and generate maps as a person who has no use for it. It’s fun!

- Helpful and clear description of how XSS (cross site scripting) vulnerabilities work and can be identified.

Friday Links

Sat, 10 Feb 2024 00:00:00 +0000

- A series of YouTube videos on how “practical effects” in movies are highlighted by directors, actors, and viewers, but actually are building blocks to extremely realistic computer generated visual effects. I didn’t even realize much of what he showed was technically possible: it makes sense that performing a stunt in real life can give VFX artists valuable information about how it should look when it is computer generated, but the fact that oftentimes the stunt itself is replaced with CGI is astounding; see, for instance, the Top Gun: Maverick replacement jets.