Diseases face a choice between spreading easily and being severe. If a disease is too hard on its host, killing quickly, it can’t spread. If it’s too easy on its host, it doesn’t much matter if it spreads.But HIV spreads pretty easily and is (or at least was until the "cocktail" for treatment was developed) as severe as can be imagined. If you were designing a disease to wipe out a large portion of the population, HIV would provide a pretty good model for two key reasons.
The first is that victims don't actually die of HIV. They die from something else because their immune system is so weakened that an opportunistic infection (like thrush and Kaposi's sarcoma) can spread and eventually cause death. This enabled HIV to hide for several decades as patients were assumed to have died from the other infections. Even after HIV made it to the United States, it took a decade to detect it.
The second reason is that it has a long incubation period. Klein was right that a disease that kills too quickly can't spread, but one that kills nearly 100% of the time can still spread - as long as it does so slowly. And the median incubation period for HIV is 10 years. The saving grace, if you wish to call it that, of HIV was that it was an STD, which significantly slowed down the rate at which it could be spread.
But imagine if HIV was a vector-spread disease like malaria, or could be transmitted via droplet spread or what if it had been airborne. In other words, if HIV had a Basic Reproductive Number that was closer to that of measles or whooping cough it would have spread very far and infected many many people before we even knew there was a new disease on the loose. And it took us nearly 15 years from HIV identification to effective treatment, during which time the disease continued to spread.
Another thing to consider is a disease that alters the hosts behavior like toxoplasmosis does. The life cycle of Toxoplasma gondii, the parasite that causes toxoplasmosis, requires the disease to be passed between rodents and cats. Rodents infected with the disease will have their behavior modified so that they are less fearful of cats and so that they behave in ways that are unsafe, thereby increasing the chances that they will be eaten by a cat and continue the cycle. Even humans infected by the disease have been shown to be involved in a higher rate of car crashes and other risk-taking related deaths, indicating that their behavior has been changed as well.
Imagine an HIV that changes peoples behavior, making them engage in more sex and/or riskier sex. Such behavior could increase the transmission rate considerably over what was observed. Or imagine a flu-like disease that makes someone become more gregarious and outgoing during their peak contagious period, allowing them to come into contact with more people and spreading the disease more quickly.
It is unlikely, but a disease really could cause a world-wide pandemic that threatens human civilization. What would such a disease look like? It would be a disease, like HIV, with a long incubation period and low survivability rate, and one that masks it impact and is hard to understand, but with a higher transmission rate more like the measles. Such a disease could very well create a movie-like disease-based pandemic that would infect and kill 5% or 10% or 20% of the world's population. And how large an infection would it take to strain the civilization that we have now? Probably lower than 20%. Like I said, it's unlikely, but then so was HIV.
The scary thought is not that something like the AIDS crisis could happen again, but that perhaps, as bad as it was, we got lucky.