In a long thread about AI safety, @discoursedrome writes:
Any reasoning that treats utility as linear in something real a fungble is, at best, locally heuristic. If you slide from “utility” to something like “lives saved” or “dollars” or “person-hours” and then raise those values to extreme levels, you’ll always get an absurd result, and a lot of the problems with these scenarios just boil down to that.
This shook loose a thought about derivatives in my head. Most functions aren't linear, but most functions can be locally approximated by a linear function. So if we're studying some complicated relationship, but we expect the inputs to stay within a narrow band, we will just use the linear approximation, but claim we're talking about the original relationship.
Resistance
My favorite example of this is electrical resistance. Ohm's law says that current = voltage / resistance: current is a linear function of voltage. This is false.
TThe current-voltage curve is a curve, so what is the resistance? There's not just one number we can put there. But as long as we know the voltage will be between, say, 100 and 150 volts, we can just use the linear approximation, the derivative of this curve is the "resistance", and everything is fine.
That same principle shows up all sorts of places. Elon Musk "has" 240 billion dollars. The majority of this comes in Tesla stock: he owns 175 million shares, at 943 dollars per share, for a total of about 165 billion dollars.
Wealth
Now, if he wants to sell a hundred shares, he can sell them for 94300 dollars, no problem. If he wants to sell a million shares, he can probably still get something very close to 943,000,000 dollars for them. But he cannot sell 175 million shares for 943 dollars per share and put 165 billion dollars in the bank. If he tried to sell all his Tesla stock, the price would fucking crater.
This isn't to say the price is fake, or made up, or anything. It's a real price. You or I could go buy or sell a share of Tesla for 943 dollars, and so can Elon Musk. It's a real thing. But it's a local thing—you can't sell a hundred million shares at that price. The list price is the derivative of the quantity-price relationship.
Bad extrapolation
When people take these linear approximations and assume they're the real function, silly things happen.
If you assume resistance is constant, you'll melt your lightbulb at high voltages. If you think of prices as constant, you won't understand why buying a company costs more than the stock price times the number of shares. And if you think of mass as constant, your gps won't work.
Back to utility
And maybe utility makes the most sense this way too. If you're comparing comparable quantities of comparable things, utilitarianism basically works. (Yes, I would rather save ten lives than five. Yes, I'd prefer to lose 100 dollars rather than 150 dollars.) And it probably makes sense to say that ten lives are "twice as valuable" as five, for the purposes of doing the math. But that doesn't mean ten billion lives are "twice as valuable" as five billion. We're out of scope of the approximation.
