It isn’t all that uncommon for someone to get a degree in finance, become a CFA charterholder, and spend decades around the most brilliant financiers on Earth and still underperform the market. In fact, it’s almost the norm—and that’s a reason why many financial professionals stick to low-cost index funds in their 401ks.

There are two reasons for this, and neither of them prove the efficiency market hypothesis or suggest all investors should stick to the Vanguard index funds (VOO, VTI) or something similar.

The first reason is compliance. An analyst specializing in, for instance, internet stocks, may know enough about that particular sector to make very complicated and well-timed bets that would result in a multi-fold return in a couple of years. There’s just one problem: doing so would almost always be illegal. Analysts cannot engage in restricted trades in their private accounts, and very often the well-timed bets that they would make for themselves are the ones that the fund manager will not make because of external concerns. So those potential big gains never actually happen.

If analysts cannot invest in their specialisms, why not use their financial acumen to invest in another sector? This goes to the second reason: superior investment returns are not a product of financial expertise alone. Superior returns come from superior access to information: that is, knowing more than other investors and betting accordingly.

This is partly why insider trading is illegal. However, one can get superior information without having access to insider information. Take, for instance, our internet analyst. If that analyst can find out from, say, Amazon how much they’re spending on ads on Google Search, that analyst will know more than the average investor and can make recommendations accordingly. And since this information isn’t inside to Google, it isn’t insider information (but it would be if used to invest in Amazon—the rules are tricky!)

This is known as primary research, and it is a core component of modern finance even if it is not taught in classes. Tracking, for instance, deactivated accounts on Twitter or Facebook or new subscriptions to Netflix are all part of the primary research methods that are used to make bets on those companies—while similar analysis of foot traffic for retail, store presence for consumer goods, and tanker flows for commodities are all part of the arsenal analysts use to get an edge.

The key is that these tools are not widely available and cannot be automated. An analyst has to carefully sift through data and separate signal from noise. This process is not as simple as, say, doing a discounted cash flow model, which can be very easily automated.

The popularity of primary research is likely to rise as more data becomes available, both increasing the potential for insights and increasing the demand for analysts who can use that data to produce insights. An aspiring financier should learn not only the nuts and bolts of finance but also the nitty gritty of primary research.