There are lots of mentions of castles being burnt down to the ground, but I don't really understand, how does it go on? Weren't castles mostly stone? Even if there are lots of internal structures made of wood, the walls would be mostly stone, how does the fire take them down?
There’s a technical term for this, and it’s called slighting a castle.
While it’s true that castles are stone, a lot of the internal structures made of wood are, in part, load-bearing. Think roof beams and floor beams and joists and the like. Burn those, and the stone walls are just free-standing and can (usually) be more easily pulled down or fall over of their own accord.
Regardless of whether the walls remain standing, burning the wood interiors accomplishes the immediate task of making the structure largely unusable for some time, since A. it doesn’t provide much shelter from the elements, and B. there isn’t anywhere to stand.
However, if you want to do a full job, what you do is dig a tunnel or “gallery” under the walls, and then set fire to the props holding the tunnel up, the tunnel roof collapses, and brings the walls down with it. Once the walls are down, the structure has no more defensive utility. Similarly, you can fill in ditches and moats with earth, brick up or widen arrowslits or tear down crenellations - anything that makes it less useful as a fortification.
