A Guide for Investment Analysts: Working with Historical Market Data

Q: How far back does the record date for U.S. stocks and bonds?

A: There are good data series from the 1790s.

This is the primary of three posts aimed toward analysts taken with working with older historical data. It is straightforward to download a historical data set and immediately proceed with statistical evaluation. But pitfalls lie in wait for the unwary. The further back in time, the more different the circumstances. When interpreting the outcomes, context matters.

I even have a watch on investors who enjoy reading historical reports. I see many more of those reports within the press and white papers than I did 15 years ago once I began examining the records. These posts lift the hood – or turn the stone – to offer you a greater understanding of what underlies these reports.

I begin by dating and defining the fully modern era after which trace the roots of modernity to the Twenties. Later contributions will go even further back into history.

Complete modernity: the Seventies

• From the top of 1972 the Center for Research on Securities Prices (CRSP) includes stocks in its database which are traded over-the-counter on the NASDAQ. Starting in 1962, stock trading was added on the AMEX.
• Before 1962, there was no true Total Stock Market Index to trace. Indices designated as “Market” include only stocks listed on the New York Stock Exchange. This implies that only the biggest firms which are in a position to meet the NYSE’s strict listing standards are included. Before 1972, many of the smallest firms within the United States and people with the weakest financial records—literally 1000’s—were excluded from the historical record.
• Accordingly, factor analyzes prior to this era are suspect. “Small stocks” were the smallest of the biggest stocks that might qualify for listing on the NYSE.
• Banks and other financial services firms usually are not included within the CRSP data before 1972. These weren’t listed on the NYSE.

• Only at this point will there be regular issuance of presidency bonds and bonds spread across all the range of maturities. As described below, in the beginning of the trendy era within the Twenties, most government bonds were long-term issues, and issuance was irregular. Years could pass without recent government bond problems arising.
• It was not until the Seventies that a complete bond market index appeared that included all traded maturities and combined government and company issues.

Roots of modernity: the Twenties

You can have read the sentence “Since 1926, stocks have returned…” and wondered what was so special about 1926.

The short answer: nothing. The December 1925 anchor for the Standard & Poor’s index and for the general stock market index published by CRSP represents an arbitrary start line set by time and value constraints faced by early data compilers.

Yet accurate data at the person stock level – every day price changes, ex-dividend date, demergers, mergers and acquisitions, other corporate actions – currently only goes back to January 1926. Before this point, the analyst normally has to work with index data, at best over a monthly period.

With this restriction, the actual starting of the trendy era lies around the top of the First World War. Before the war, the markets looked very different, especially the bond market. The data available to interpret market returns, even at index level, can also be becoming increasingly thin. From January 1919, nonetheless, quite a few macroeconomic and microeconomic data series will be present in Federal Reserve publications.

In the Twenties:

• Hundreds of stocks traded on the New York Stock Exchange, which many years earlier had established its dominance over all other U.S. exchanges. Almost all of the biggest firms within the United States were listed on the NYSE.
• These stocks were spread across greater than a dozen different sectors, including transportation, utilities, various industrial sectors – including durable goods and packaged goods manufacturers – and emerging services akin to retail chains.
• After World War I, a powerful and liquid marketplace for U.S. government bonds emerged.

However, from 1926 onwards, some elements that make the twenty first century are still missing^st Century, investors take it as a right.

For stocks:

• Again, banks and most financial services firms weren’t traded on the NYSE and weren’t included in either the CRSP or S&P indexes in the course of the period.
• The Securities and Exchange Commission didn’t yet exist (1935), nor did the Investment Companies Act of 1940. There were few regulations to stop market manipulation or the spread of false or self-serving information.
• The Federal Reserve doesn’t yet regulate the margin required for stock purchases. Depending on the client, stock and brokerage firm, the trade can have required as little as 10% margin.

For bonds:

• There were only a number of maturities available for presidency bonds, most of them long. It wasn’t until the Nineteen Thirties, when the Treasury tried to ease the depression with multiple problems with various lengths, that the maturity spectrum began to fill out.
• There was no regular schedule of offerings for any due date. In fact, for many of the Twenties, the federal government was busy paying off the debts accrued by the war with recent offers aimed primarily at converting these debts, particularly the short-term bonds, right into a form favorable to the federal government refinancing due date schedule.
• The era’s considering viewed national debt as an unlucky necessity of war, to be reduced and paid off when peace conditions permitted.
• The modern treasury bill, defined as a really short-term debt security issued commonly and allowing amounts to be rolled over indefinitely, was not introduced until 1929.

Takeaways

There is now nearly 100 years of information that enables comprehensive evaluation of stock and Treasury returns, not much different than what the analyst has been in a position to do over the past 50 and even 20 years.

But as soon because the analyst dares to look back to the time before the Twenties, the information sets which are now taken as a right begin to thin out and disappear. Above all:

• There was no Treasury bill, subsequently no good indicator of the risk-free rate, subsequently no method to create a Capital Asset Pricing Model (CAPM) regression, and subsequently no method to assess market beta. In fact, the CAPM recently celebrated its sixtieth anniversary.
  • There isn’t any good range of Treasury bond maturities until the Nineteen Thirties, so there’s little opportunity to check the Treasury yield curve or changes in that yield curve. In general, until the later Nineteen Sixties, there was no regular supply of short-term or medium-term bonds in treasuries. There is not even an everyday offering of 10-year government bonds. It had not yet turn into a benchmark. Before the Nineteen Sixties, investing in bonds primarily meant owning long-term bonds.
  • There was little sector diversification in stocks before the Twenties.

In my next post I’ll proceed this story beyond the First World War. In the meantime, in case you’re able to roll up your sleeves and begin working on the information, Here are some sources for contemporary times:

• Monthly data on the whole stock market return (inside specified limits) and the risk-free rate of interest (30-day T-bills) through June 1926. Free to download. Is updated every June.
• Data on quite a lot of subdivisions of the market, including essentially the most common aspects (size, value, etc.) and major industry sectors.

• The S&P index returns monthly through January 1926 (and before, see next post). Separate dividend series and price return series. Profit series for calculating the CAPE (cyclically adjusted price-earnings ratio). Monthly inflation to calculate the true return.
• Two caveats:

• Shiller calculates returns based on the typical of every day prices, not end-of-month prices. This limits volatility and can lead to very different return estimates than the usual month-end estimates over periods of ten years or less.
• Shiller’s Treasury yields, presented as 10-year yields, are based not on the value of 10-year Treasury bonds, but on yield curve interpolations back to 1954 after which extracted from longer bond yields back to 1926.

• Data from 1926 to 1987 will be present in this free online copy of the 1989 SBBI on the CFA Research Foundation website, with rows for giant stocks, small stocks, long Treasuries, intermediate Treasuries, long corporate bonds and T-bills.
• Monthly data through 2023 is on the market behind the paywall at Morningstar.

• Both manage extensive compilations of individual inventory data behind a paywall. CRSP has greater than 25,000 stocks and all Treasury bonds dating back to 1926; GFD has data on a dozen international markets dating back to 1700.
• Gaining access to those databases typically requires access to a university library subscription, most probably that of a big research university.

The Jorda-Schularick-Taylor Macrohistorical Database Tracks a smaller variety of international markets through 1870, with macroeconomic series in addition to asset returns. Dimson, Marsh and Staunton annually publish yearbooks describing international asset returns as much as 1900. Data series is behind the paywall at Morningstar.