I believe that the amount of male Gems was disproportionate to the amount of females, and that the Crystal Gems were likely a matriarchy. The demise of the Gems is connected to the lack of male Gems, as the population didn't grow when the females had to transfer their life energy and gem to their children due to mortal/Gem reproduction (in Gem/Gem reproduction, the energy is combined from both Gems' magic, and while this does take a toll on each parent's strength, they have a far lesser chance of dying in childbirth).
Either that, or maybe some sort of invasion or banishment. Who knows if all the Gems are even dead? Maybe they're on a separate planet.

















