#multiple replacement effects

The short answer is that it's up to your opponent.

When multiple replacement effects want to modify the same event (in this case damage being dealt by a red sou

When multiple replacement effects try to modify the same event, e.g. damage from your Lightning Bolt to your opponent or their creature/planeswalker, and none of those effects take precedence, then the affected player gets to choose which one to apply first.

In this case, the affected player is the opponent being dealt damage, or the opponent who controls the permanent that's being dealt damage. So they get their choice of adding then doubling or vice versa. This means that they'll likely opt to double first since that results in less damage to them or their creature/planeswalker.

When I answered a similar question recently, there were several comments wondering why the effects' controller didn't get to choose. So why does it work this way? A couple of reasons. First, there could be a situation where different players control the effects and disagree on the order in which they should be applied, in which case letting the controller of the effects choose just doesn't work. Additionally, this is how multiple replacement effects are handled for everything in the game and carving out an exception for one case involving less than 50 cards just isn't necessary.

The player (or controller of the creature or planeswalker) that’s being damaged chooses the order in which the replacement effects are applied to the damage. This means that generally, you’re going to get the lesser amount of damage out of this combo.

Both Library of Leng and Madness have replacement effects that want to send your discarded card somewhere other than the graveyard. ‘

As the affected player, you choose whether you want to discard to your library for LoL’s effect or if you want to discard to exile for Madness’s effect.

Once you’ve applied one of the replacement effects, the other one can no longer apply to the card. The card will be either in exile or on top of your library.

If you applied Library of Leng’s effect, then that’s it. The card’s on top of your library and we’re done here.

Probably not. 

Casting a spell for its flashback cost creates a replacement effect that exiles the spell if it tries to go from the stack to any zone other than exile. 

Buyback creates a replacement effect that puts the spell back into its owner’s hand instead of its owner’s graveyard when it resolves.

So we have two replacement effects both trying to modify where the spell goes as it leaves the stack and as that spell’s controller you get to choose the order to apply them in. 

Great... so let’s apply buyback first, since we want the spell back. OK, so now Buyback is sending the spell back to your hand, but we have to check and see if any other replacement effects are trying to modify the event still. Flashback is still waiting there to apply since the spell isn’t sending itself to exile, so Flashback’s effect is then applied, exiling the spell.

Ok, um... let’s apply flashback’s effect first then? OK, sure. So now Flashback is sending the card to exile. But we can apply Buyback still, right? Nope. Buyback can only replace the card going to the graveyard, and that’s not where it’s trying to go anymore, so Buyback’s replacement effect can’t apply anymore.