Something ethically messed up and logically incomplete about treating every potential creation of an experiencing mind (even if you guarantee that it will have an ideally positive life experience) as a "we ought to pursue this" increase in good/value/utility.
The extreme of what I'm criticizing here is the longtermist view that we must pursue a future where we fill as much of the universe as we can with human minds experiencing the best possible experiences. (In simulations! because that way we can have more of them, with reliably positive experiences.)
As if hypothetical minds that don't exist, and would only exist if we actively caused them to exist, have ethically-obligating worth now.
As if it is ethically better to pursue a future in which we create an extra richly experiencing mind having the best possible experiences (even if that mind is in a simulation and doesn't interact with or in any way affect any other minds) than a future where we just don't create that extra mind.
There's something off about this view... a subtle slip/skip (misstep, assumption, error, etc) which feels paperclip-maximizer-y in nature. It's just that the paperclips are very convincingly shaped like actually good values. Like copy-pasting a snapshot of a good-to-achieve result, while missing at least one reason why it was good to achieve (a reason which is contingent on a mind already existing, or contingent on good odds of a new mind having certain effects beyond itself).
I don't have the time to properly drag this out into words, and it's not a priority for me to make it rigorous or convincing. But there's something more-than-one-dimensional about good/value/utility revealed by this. I would assert that even though
the transition from a mind experiencing a worse existence to a better existence is ethically positive, and
the transition from a mind experiencing a good existence to that mind not existing is ethically negative,
the transition from no mind to a mind experiencing its best possible existence is ethically neutral.
To me it's obvious that there's no contradiction to those three things. I think the only way it would seem contradictory is if you implicitly assume that ethical value is a simple integral of qualia over time. There's something at play that is best described by one or more of
assigning ethical values to the derivatives, to some or all of the transitions between states rather than to just snapshots of states, or
recognizing a different kind of ethical value, where its only worth maximizing within the "domain" for it which already exists, but not worth creating more "domain" within which it can be maximized (and perhaps this is the only kind of value);
and so minds which don't exist and have no external reason to exist are in some crucial sense ethically worthless until they start to.
This is of course without even touching the point (which I previously wrote down in my reply to torture vs specks) that when integrating the future, possible states further in the future are increasingly less certain. Arbitrarily far in the future, there are too many possibilities still impossible to rule out, so there is enough possible good and possible bad from any act to cancel each other out, the probabilities are so small that the ethical weight of each possibility is tiny anyway, and the uncertainties have multiplied to the point that all the possible outcomes have vast error bars on tiny likelihoods. This alone is enough to kill the idea that we today can assign significant probability or ethical value to outcomes like "we eventually populate the virgo supercluster and the majority of our descendents have good lives".
