This has probably been discussed to death on here before, but I'm feel like the pact was formed because witches were dying enmasse. Not just ordinary people. Probably why there's huge ass silverwood forests or why there's one at the bottom of the sea (probably a witch trying to escape the chaos and oopsing themselves into being peaceful enough to die). Like i wouldn't be surprised if intentionally ingesting silverwood seeds or using blood in their magic was that common in the era of chaos.
The pact and the surrounding mass memory erasure that happened was probably an attempt at self-preservation even if it came at a cost (like medical spells). I also feel like ordinary people were still very in awe of witches and unaware of how they worked pre-pact, while more powerful ones or people in more higher end settlements were more knowledgeable about it, making the memory/person erasure (cough) more possible.
