Elizabeth Warren and Bernie Sanders both unveiled student debt forgiveness plans recently. This has prompted a good deal of think tank and media coverage trying to estimate the precise distributional impact of those plans. Being a think tank man myself, I set out to produce my own distributional analysis over the past few days but learned something very unpleasant in the process: the data source everyone uses for these estimates, the Survey of Consumer Finances (SCF), is not well-suited for the purpose and almost certainly systematically understates how much student debt is carried by low-income individuals.

Everyone uses the SCF for these purposes because it is the only high-quality wealth survey in the US that appears to allow for distributional analysis on student debt. The think tanks that have put out student debt figures based on it include behemoths like Brookings and the Urban Institute. Those figures have then been cited in articles at the New York Times, the Washington Post, the Wall Street Journal, Vox, Slate, and virtually every other media outlet.

But a deep dive into the methods of the SCF, along with comparisons of the SCF to other student debt data sources, clearly show that these figures are off the mark, and probably dramatically so.

Primary Economic Units

The way the SCF constructs family units differs from other surveys like the Census Bureau’s Current Population Survey. Rather than grouping all related people who live in the same household, they instead construct a Primary Economic Unit (PEU) for each household, which “consists of an economically dominant single individual or couple (married or living as partners) in a household and all other individuals in the household who are financially interdependent with that individual or couple.”

Importantly for our purposes here, the financially independent relatives of the economically dominant individual or couple, such as many young adults living with their parents, are not included in the PEU. In fact, they are excluded from the survey altogether.

This is a problem because it means that student debtors who live with their parents, which many do in part because of their student debt, are either absent from the survey sample or being counted as part of their parent’s PEU. Specifically, if the parents tell the survey taker that their co-resident student debtor kid is financially independent, then they are just dropped out of the survey universe. If they tell the survey taker that their kid is financially interdependent, then the kid (and their student debt, if their parents properly estimate it) are included in the parent’s PEU, meaning that it is the parent’s characteristics (age, income, education) that gets assigned to the kid’s student debt.

Understating Debt

Once you understand how the PEU works, it does not take a genius to realize that the SCF must be missing a lot of student debt and especially student debt carried by low and moderate income young adults who are living with their parents. And even when it does pick up that kind of debt, it is assigning it to the older, generally more affluent parents of the student debtors, which should also skew the distribution of student debt up the age and income ladders.

Digging into the SCF appears to confirm that this is exactly what is going on.

For starters, the total amount of student debt picked up in the SCF is well below the aggregate reported by the Federal Reserve’s G.19 publication and well below the aggregate reported by the NY Fed’s Consumer Credit Panel. How much lower? About 25 to 35 percent lower.

The total amount of SCF student debt is not just lower than the other sources. It is also skewed further up the age range. This is exactly what you would expect if student debtors living with parents are mostly being dropped from the survey while the ones that are being included are having their debt counted under their parent’s demographics.

Although I’d like to think I am the first one to realize this systematic bias, it turns out that the Federal Reserve itself acknowledged this problem in a 2015 piece in FEDS Notes:

A “family” in the SCF is defined as the economic core of the sampled household, roughly speaking the person whose name is on the deed or lease at the surveyed address, and all people at that address whose finances are intertwined with those of that person. The published SCF statistics refer only to the debts and assets of this economic core. Thus, student loan information is not collected for members of the household that are outside of the household economic core. It is likely that most of the student loans of these non-core household members are included in G.19 and CCP statistics.

[…]

Due to these considerations, the aggregate amount of student loan debt captured in the SCF will be lower than that captured in either the G.19 or the CCP. Thus, the differences between the SCF and CCP and G.19 aggregate levels suggests that a considerable portion of aggregate student loan debt is held by individuals outside of the economic core of a household.

In 2014, the Federal Reserve put out a paper that also flagged this general issue and specifically warned that people should interpret SCF data on young adults with caution:

Overall, our comparison between SCF and CPS data indicates that young adults captured by the SCF tend to have higher incomes and higher homeownership rates than the overall population of young adults in the CPS. Moreover, because fewer young adults lived independently in 2013 than in 2001, SCF median income declined slightly less than CPS median income over the period. If higher incomes are correlated with greater wealth, the SCF will tend to overstate young adults’ balance sheets on average. Thus, the results presented in this article should be considered with the caveat that they are representative of only young adults living independently. More broadly, our comparisons with CPS data indicate that SCF users should exercise caution when drawing inferences regarding balance sheet items that are primarily held by young people.

Put simply, the only student debt that is being picked up by the SCF and assigned to the person who carries it is the student debt held by adults living independently in a household where they are the economically dominant individual/couple. This means that a lot of the debt held by especially lower income young adults is simply not represented in the survey, which further means that these think tank estimates about the distribution of student debt almost certainly overstate the degree to which it is skewed towards the rich. Without better data, anyone purporting to know with great certainty the average debt burden of the bottom or second quintile is either mistaken or worse.