What other old school game should have become as big as D&D but didn’t? Why do you think so?
Would it be disingenuous of me to suggest Pendragon? I feel like it would, because - as big a fan of the game as I am - I can also recognize that it's a pretty dang narrowly-focused game. Part of D&D's appeal, of course, is that one can do so much with it, after all.
So I'll broaden my nomination to include Pendragon's root system, Basic Roleplaying. Here is a system that was pretty much in its completed form by 1980, the year it was first released as a standalone system. The core of the system could be expressed in a simple 16-page booklet, and it was explicitly released as a "generic" system, among the first of its kind. It took D&D-style mechanics and made changes a lot of people wanted to see (like having armor subtract from damage rather than make you harder to hit, or adding handy attributes like Size or things like Luck or Idea rolls, or introducing a skill-based system of character building) built on an extremely robust design while still remaining simple and flexible. (Plus, personally, it's just more fun to roll percentile dice than a d20.)
D&D has always benefited from its name-brand recognition, and I think perhaps part of the reason BRP never got as big as it deserved is that its name just isn't as evocative? But in all seriousness, it probably more likely came down to a combination of market forces and behind-the-scenes management (or possibly mis-management...) derailing any sort of momentum the system might have built up. Because, really, in 1980 BRP had everything D&D was offering and more. Its parent system, Runequest, was giving AD&D a serious run for its money in terms of popularity. I'm not suggesting that BRP could've eventually knocked D&D off its pedestal, but it deserved much more of a lasting place than it received.
Fortunately, BRP has survived the vicissitudes of the market and is still around today, still largely the same game you could have played back in 1980. If you want to hear me and my co-host blather on about our love for the system, check out this episode of our podcast.