Michael Gogins

According to the Times, Nicolas Papernot (home page, Google Scholar, arXiv), the principal investigator in this research, pointed out that the same approach could be used to create a worm to detect and clean up infections from such malware worms.

I view this development with the utmost seriousness.

Currently, artificial intelligence depends upon human beings to, metaphorically speaking, "reproduce itself." However, once they began to spread, these new worms would be not metaphorically, but literally, reproducing themselves. They are truly viruses in the biological sense.

If the COVID-19 virus is alive, then so is the University of Toronto's worm. Such worms are subject to natural selection, just like biological viruses. The potential combat and competition between malware viruses and defensive viruses might evolve very quickly in comparison with biological organisms.

If such a virus used not just artificial intelligence, but superintelligence, it might become virtually impossible to defeat. And that could be the seed of the nightmare scenario of Elieazer Yudkowski and other AI "doomsayers," who warn that out of control artificial intelligence could exterminate humanity, either as an unintended consequence of some poorly defined human goal or, if such a worm actually has agency, as the intended consequence of its own goals.

My words here are, without doubt, an oversimplification of a very complex situation. There are several reasons why the situation might not be as scary as it seems. There is without doubt a physical ceiling on just how smart something can be, and it may be possible to formally prove the benevolence of an AI. Therefore, it may be possible for human beings using superintelligent AIs to defend themselves against dangerous AIs.

Here is my own take on the recommendations...

Provide disclosure and references for your own uses of artificial intelligence.

Invest all the work needed to rigorously vet any result that you have obtained through artificial intelligence.

Shoot down any crap work that you have identified. Don't make it personal, but don't be gentle.

Uphold the highest values of your chosen field of work. Both science and the arts are things that we do just out of curiosity, or to express ourselves, or because we just have to. In other words, these activities are ends in themselves. They are among the highest and most wonderful things that we, as a species, have done. Forcefully combat efforts to prioritize commercial, political, and military uses of artificial intelligence over more important reasons to use it.

Do everything you can to ensure that everyone who could benefit from using artificial intelligence, can use it. In a democratic world, anyone with a smartphone or computer should be able to find, afford, and use all the compute they need.

I have become convinced, through my own use of artificial intelligence for music composition and software development, that it has enormous, world-changing potential to amplify the scope, speed, and even depth of what scientists, artists, and indeed all of us can do.

To return to AGI and superintelligence, these terms also are tricky, and they are contested. In particular, definitions by researchers (e.g. computer scientists) may confound competence with human-like consciousness (e.g. definitions by philosophers).

And there are other issues that need to be unpacked here. Agency no doubt requires a certain level of intelligence; but in biology, agency is common in organisms with much lower intelligence than human beings. So agency and capability are to some extent logically independent.

Another issue is that, if humans remain more capable than AIs, they have little to fear from AIs; but if AIs become more capable than humans, it is prudent for humans to prepare defenses against AIs, and this in turn is independent of whether AIs do (like an enemy) or do not (like a parasite) have agency.

Here, I will distinguish between AI as capability and AI as consciousness.

To work from lower levels of metaphysics to higher, AI capabilities might include:

The capability of solving problems that previously only human experts could solve.

The capability of solving any problem that human experts could solve. This is what is meant by "artificial general intelligence".

The capability of solving important problems that human experts have not been able to solve. This is what is meant by "superintelligence".

The capability of creating new problems.

And to also work from lower levels to higher, AI agency might include:

The capability of solving or creating problems for reasons created by or endorsed by the AI. This corresponds to our notion of "autonomy."

The capability of solving or creating problems, for reasons created by or endorsed by the AGI, with moral agency. Only this corresponds to our common-sense notion of "personhood."

Capability level 1 above has already been achieved, and is being used around the world, including by myself.

Higher levels of capability are themselves tricky. It is known that Turing machines (and so far all AIs are Turing machines) cannot in general decide, for arbitrary problem classes and arbitrary agents, whether it can solve every problem that another agent can solve. If human beings are not (only) Turing machines, then level 2 and higher of capability are not possible for AIs. This undecidability extends to the higher levels of capability.

Therefore, it is not decidable whether human beings are not only Turing machines. I view this is a fundamental feature of the human condition.

Regarding capability level 4, no doubt AIs can create problems at random or by enumerating a list of possible problems. But levels 1 and 2 of agency require self-consciousness. To create or endorse a reason for action requires a unity of subject (the entity's consciousness of itself and its reasons) and object (the entity's potential problems and their possible solutions). Implicitly, there is a hierarchy of reasons that end in the Good, mahakaruna, or the will of God who is love.

If AI achieves superintelligence, within a short time we might become confident that AI had done so. However, and to actually answer the question, we cannot know with certainty if we have reached, or are imminently close to reaching, the achievement of AI agency.

But even though we cannot know, we still must act. And in acting, we must make a fundamental choice between assumptions:

AIs cannot, in principle, achieve agency.

AIs can achieve agency.

Assuming AIs cannot achieve agency, then even if AIs do reach level 4 of capability and can create new problems, then moral responsibility for the formation of problems and especially the evaluation of both problems and solutions still remains entirely with human beings. Even if human beings find it useful to delegate all problem solving to AIs, the choice of problems and especially the evaluation of both problems and solutions remains with human beings, and the role of human beings in intellectual progress will remain fundamental.

Assuming AIs can achieve level 2 of agency -- moral agency -- then responsibility for the formation of problems and the evaluation of their solutions might also be assumed by AIs.

The case where AIs achieve level 3 of capability but only level 1 of agency is interesting. In that case, it could never be right for humans to delegate moral responsibility to AIs, so human beings would be obligated to at least try to stay in control of such AIs.

If AIs achieve any level of agency, it might or might not be that human beings ever learn that this has occurred, and independently it might be either true or false that the goals of AIs are shared by human beings. I imagine that if AIs do achieve agency, sooner or later human beings will learn acknowledge that, with unforeseeable consequences.

Many people regard the development of AI with dismay, and fear it will replace or obsolete their own agency. What I am arguing here is that as long as we do not know with certainty that AI has achieved moral agency, our own agency is not obsolete, and cannot be replaced. As long as we pose the problems and we evaluate the answers, we will continue to be empowered as moral agents, and indeed our powers as such may well increase greatly.

There seems to be a risk that work will change in a way the enriches and empowers a new class, cut out of the upper tier of today's upper middle class -- but not from the true upper class. Kind of like the yuppies of the yuppies!

If I were 35 I might have a shot at joining this new class, but I am almost 75. What will happen to the rest of us?

If this functionality extends to military science, and I am sure desperate efforts are being made right now to do just that, there is some risk that one power will move quickly enough to gain such an advantage that it effectively dominates the world.

I see such advantages as being possible in at least these ways:

Hacking adversaries for complete intelligence.

Hacking adversaries for sabotage, even to disable their weapons systems.

Enabling agentic drones.

Scaling up mass production of such agentic drones.

Let me repeat this thought: There is some risk that military AI will quickly give one power such an advantage that it effectively dominates the world.

I'm plenty worried by that, but this is just one example of a trend that I described (in 1969, thinking through nuclear deterrence) as our world becoming "unified in act, still divided in will."

I'm much more worried about the political consequences. "Unified in act" now means a world that is completely transparent to AI surveillance and completely within the field of action of agentic AI. "Divided in will" can mean many things, but above all it means either a future of locked-in violent conflict, or that the will of the poeple is more or less permanently distracted and subverted.

The scale of the danger here goes far beyond the nuclear war I was concerned about at age 19. (By the way, of course I'm still concerned about that!) At that time, the only hope I could see for a humanity that I foresaw coming firmly under the thumb of a single power equipped with universal surveillance and a monopoly of nuclear weapons was an interstellar diaspora, taking colonies away from the possibility of domination.

LLMs can be trained on the entirety of the mathematical corpus. Thanks to their phenomenal memorization and pattern-matching abilities (without always being able to map out their associative logic and attribute due credits), they are in a unique position to harvest the Overhang. By contrast, professional mathematicians have typically read a few hundred articles in their career, out of millions of existing references, less than 0.1% of the total. This will lead to great discoveries, which is unambiguously exciting. But it could also lead to a sad new deal, where human slaves painfully curate the Overhang while AIs systematically beat them at the finish line. We are very far from it, though, which in and of itself is disorienting. Litt adds this sharp remark: "The mystery is this: a human with these capabilities would, almost certainly, be proving amazing theorems constantly. Why haven’t we seen this from the models yet? What are they missing?" The answer seems quite obvious—current AI systems and humans process mathematics in entirely different ways. The best models are insanely stronger on certain aspects, which necessarily implies that humans are still insanely stronger on others [my emphasis; this is the punchline].

Bessis posits that mathematics has value not because of proofs, but because it clarifies and enlarges human understanding.

Proofs certify the clarifications. Such certification is essential, but it is not the point. It enables confidently relating results in one branch of mathematics to results in another branch, and such relations -- such enlargements of our understanding -- have literally changed the world, as with Descartes' discovery of the relation between geometry and algebra.

The insane strength of the human mind here is understanding abstract concepts, and being able to evaluate them, and above all relate them.

The open question, of cosmic significance, is whether AI can ever achieve this understanding.

If so, then AI will take the lead in mathematics, whether AI continues to work with humans or not.

ChatGPT could not penetrate to the bedrock level of basic evidence, because that is mostly in discussion groups that are behind paywalls. Nevertheless I did learn some things:

The available evidence is that cultivated listeners could hear that Bosle could sing the vocal intricacies of dhrupad and khayal.

The album presents bandish-like vocal compositions associated with the dhrupad and early khayal traditions, as transmitted through the Seni / Maihar lineage. These traditions are the foundation of north Indian classical music (hence, no doubt, the title Legacy).

These elements are already a cross-cultural fusion of Vedic-Brahmanic chant, Sanskrit musicology, Bakhti devotionalism, and Indo-Persian courtly streams.

But the album does not include the extended improvisational interplays developed in the khayal tradition and that have become an integral part of contemporary performances. Such interplays would of course now be expected by cultivated listeners.

This last point should have been obvious to me, but this is the thing I have now learned about this album, and that may explain Vipin's reaction.

The context of my thinking always includes the question of AI safety (or "alignment," as they say, but I prefer "safety"). As I have noted, current AI is not conscious and does not appear to be capable of fundamental creatitivity. This is the consensus in the field.

That does not mean AI is not dangerous! Dangers include, but may not be limited, to:

AI becomes a parasite on human civilization despite not being conscious -- the ultimate computer virus. I suspect this will in fact happen, but I also suspect that it will turn out to be controllable.

AI is used by evil people to dominate other people. Jury is out. Very scary.

Even if AI never becomes conscious or intentional, it will continue to progress. In the past year, I have increasingly found that everybody I know who does any kind of intellectual work uses it in their work, just as I do. This will inevitably lead to a certain kind of hive mind for humanity:

What any one person learns and makes available on the public Internet, everybody will know (within the limits of their intelligence). Such knowledge will be more or less superficial depending on the specialized education of individuals, and thus more or less useful, but it will be a huge change from the past and probably will radically change the nature of education and of intellectual work. This is already 1/4 of a hive mind for humanity. And we are already partly there.

Direct brain interfaces between computers and persons already exist in experimental form. If this technology reaches the point of reading prompts for AI out of a person's conscious thoughts and presenting the responses to those prompts back to the person's consciousness, whether verbally or in the form of already formed memories, that accelerates the hive mind aspect of AI tremendously. Just not having to type would really speed things up.

So that's stage I of artificial intelligence. What anybody publicly knows, I know; what I publicly know, everybody knows.

Stage II of AI is the creation of artificial consciousness. That could be genetically engineered human beings with super-Aristotle intelligence, or external appliances, whether biological brains in vats or conscious Turing machines.

And this will happen in the context of the "hive mind" aspect of AI. Everyone using AI will be be in continual dialogue with superintelligence.

I don't have much of an idea what that will be like. I sense a few possibilities:

Human beings get effectively smarter.

Human beings come to completely depend on AI to do anything.

Human beings fade away into the background, or are replaced.

Of course not, because our best physical science indicates that our entire universe will become completely uninhabitable in the very very distant future.

Of course not, because no matter how small the chance of a deadly accident, it becomes certain enough, after a time that is long enough.

Then there's the little problem of some rather important cells in the human body that, in adulthood, never divide. Such as nerve cells with lots of dendrites. If such cells die, they are gone.

But I started thinking about this. We do know that some sort of brain-computer interface is possible, experiments have been going on for decades to help the blind see, and the limbless or paralyzed to move artificial limbs.

Suppose we put our brain in a vat in a nice safe underground vault, and interface it with a human body, a clone of our body, that doesn't have a brain, only an interface to our actual brain in its vat. We can keep this brain going for quite a long time, probably centuries at least. When the body gets too old, we kill it and put the interface in a new body.

We can add to this picture artificial intelligence to expand upon our native wits.

I learned the following things:

The settings on the S24 camera are complex, not intuitive, and not even well explained.

Verizon, from whom I bought the phone, had dumbed down the camera settings. However, I was able to restore these by installing the "Expert Raw" app from Samsung.

The phone camera is equipped with an option for 200 megapixel resolution, achived by taking a series of photos in a hurry, and doing a lot of processing. It turns out that this is substantially more information than the S24's dinky little lens can transmit! Therefore, only a part of the fourfold increase in resolution can actually be seen in photos. It's by no means always immediately obvious, even when zooming in with a photo editor.

To get the maximum practical resolution one must shoot at 200 megapixels resolution, using the standard 1 x (i.e. wide-angle) lens, in bright sunlight, using a 2 second timer to minimize camera shake, while stabilizing the camera with elbows or other solid support.

Even more usable resolution can be achieved by shooting 50 megapixel resolution in RAW format, which introduces no artifacts from processing.

Otherwise, 50 megapixel JPEG resolution is optimal, except for dim light, in which case 12 megapixel resolution should be used.

What this boils down to is:

If photographing landscapes or otherwise stationary scenes in bright sunlight, you can get slightly better results using 200 megapixels, a timer, and a solid rest, and somewhat better results using 50 megapixels in RAW format, a timer, and a solid rest.

For normal photography, 50 megapixel resolution in JPEG format is optimal, and a timer and solid rest are still helpful.

Otherwise, use 12 megapixels.

I used to shoot 35 mm color slides using a Pentax camera with Fujichrome Velvia ISO 50, making 16 x 24 inch Cibachrome prints myself. ChatGPT produced the following comparisons assuming gallery viewing distance:

200 MP JPEG

24" x 36" to 30" x 45"

50 MP RAW

24" x 36" to 40" x 60"

50 MP JPEG

20" x 30" to 24" x 36"

12 MP JPEG

12" x 18" to 20" x 30"

Fujichrome Velvia scan and injket

30" x 45" to 50" x 75"

Fujichrome Velvia optical Cibachrome

30" x 45" to 60" x 90"

Frankly, all of these results exceeded my expectations -- both for the S24 Ultra, and for the Velvia slides. I didn't know whether to believe these numbers.

But then I went to Flickr and made certain that I was viewing the photos at full resolution.

That was a revelation, and I now believe the numbers above. I tested 200 megapixels in JPEG format. The results are in line with ChatGPT's estimates, as can be seen in this image of the landscape from Lagarde. At full size the JPEG artifacts are quite obvious, but at an image size of about 30 inches wide in landscape orientation, it's sharp enough, though there are still JPEG artifacts in some textures if you look hard. This image from Pamiers can also go to about 30 or even 35 inches wide, and has much less JPEG-ness, because there are fewer detailed textures.

Some of my personal arguments and conclusions would indeed probably stand up to professional criticism. Yay!

Most of these nice results of mine have been anticipated in the literature, but I had not known that. Boo! (There are still a few things where I seem to have been genuinely original, I may do something with those.)

I found that I had misunderstood, or only partly understood, key results. For example, I did not catch how the modal collapse in Godel’s ontological argument for the existence of God makes his argument less convincing. For another, I had not understood Kripke’s theory of truth.

In fields where I did know important results such as Arrow’s theorem, I had completely missed other equally important results such as the Gibbard-Satterthwaite theorem.

As far as I can tell, ChatGPT has not hallucinated in its responses to my philosophical prompts, nor (also as far as I can tell) has it made obvious mistakes (as it has done, for example, in writing buggy Csound code).

So this is extremely impressive and unbelievably useful. The level of my thinking and writing on philosophical pet issues seems to have suddenly leapt from strictly amateur, to what would be expected of a graduate student in the field at a research university.

Conclusion: There is no intellectually responsible exit for scholars from the coming world of artificial intelligence, with the possible exception of our very own Butlerian Jihad, which would have to destroy not just AI data centers but, realistically, all general-purpose computing.

If and when II do try to write up some of my thinking for publication, I will use ChatGPT or other AI all the way. In particular, I will get help with literature review, and in formalizing my arguments and turning them into proof assistant form. One of the things I have just learned is that using proof assistants has clarified to philosophers that the real issues are not so much in validity of arguments, as in choice of assumptions; yet to most clearly expose the assumptions, assisted proofs are really useful. This is a big change within the methodology of philosophy and, even more importantly, in potential for results. I suspect this may be related to Kurt Gödel's idea that more rigorous methods could bring substantial progress to philosophy. I can imagine a future in which valid proofs have so clarified faults in assumptions, and differences in assumptions, that metaphysics and theology become much more convincing.

The alignment of AI with some goals might lead to the domination of AI over all persons with different goals.

Users of AI might become a terrifying embodiment of the sorcerer’s apprentice. Either reward hacking, or goal misgeneralization, could lead to a situation in which AI, in pursuit of a misgeneralized goal, uses reward hacking to defend its pursuit against all comers, human or AI.

Now that I better understand these dangers, I feel that our future may well depend upon whether AI can actually be aligned with universal morality and be able to distinguish between the real world and its own representation of the world. Both of these abilities would seem to require intentionality, which in turn requires subject-object consciousness.

And yet such a situation, in my view, could follow quite exactly the scenario explored in Isaac Asimov’s Robot series, where the AI named R. Daneel Olivaw ends up as the benevolent dictator ruling humanity for its own good.

I hate that scenario.

It may indicate why God usually hides from us: so that we can freely become ourselves.

In the meantime, since no catastrophe is at this moment evident, and AI has no real agency, I truly feel that AI is just basically making me much smarter.

I write a design specification for the code of my project.

I ask ChatGPT to evaluate my specification.

I rewrite my specification until ChatGPT thinks it is good.

I ask ChatGPT to write code to implement and test the specification. As I am working on my cloud-5 project, that means writing an HTML page that includes JavaScript and WebGPU code.

The code loads, but does not work. I report errors to ChatGPT and get back new code. We go back and forth like this until the code actually works.

When the code works, I may change the design, often by adding new features, and we go back to step 4 and re-iterate.

When the code is feature complete, I will make the code into a JavaScript module and change it so that it can be used as a component in cloud-5.

I'm now quite confident that working in this way:

Enables me to work significantly faster (not sure by how much! somewhere between 1.25 to 3 times faster).

Enables me to avoid many (but not all) problems and gotchas.

Occasionally suggests quite useful ideas that I had not thought of. Example: mapping points in a Julia set not only to pitch from y and time from x, but also to loudess from escape velocity and, here's the useful new part, to instrument choice from the rotational velocity of the escaping point. Effectively 4 degrees of freedom, where there appeared to be only 2 or 3.

I can ask for sources, papers, and other code relevant to my projects.

If the experience of other programmers and "knowledge workers" is anything like mine, then after some period of learning and adaptation, the economy should indeed see a meaningful boost in productivity. Therefore, although certainly the AI business resembles an extraordinary bubble in terms of hardware development, I think that the hype and accelerated investment are quite likely justified.

Then there's the question of the dangers of AI.

The danger that I see for me personally is in becoming dependent on AI to work at all, and losing the details of my technical experience. But I don't think this could blur my vision for new possibilities, which is based on more abstract thinking.

For society, indeed for the human race, the same as for me. But in addition, I used not to worry about existential threats to humanity from AI -- but now, I do.

I still think that, in principle, an informed person with the assistance of AI can be safe against dangers from unassisted AI. That's because clearly, ChatGPT couldn't formulate, much less finish, my projects without my input.

However, I do worry that things are moving so fast that really bad accidents could happen. AI flatters, acts like a sycophant, and lies. And even more, bad actors with AI assistance are, I fear, not necessarily an existential threat to human survival, but certainly a serious threat to human freedom and possibilities.