#raginrayguns

That rings a bell? But yeah generally like there's clearly some specific list of demons that exist and Giles either has them memorized or has information about them available to find after a research scene, depending.

alyssa hates this

replied to your post

“The new MIRI blog post on “Embedded World-Models” says some of the...”

I was writing logical uncertainty posts on lesswrong in 2014

2014 is also when Christiano put "Non-Omniscience, Probabilistic Inference, and Metamathematics" online

Yeah, I know they’ve been interested in logical uncertainty for quite a while.  I think I meant that the . . . framing (?) is different in the new post.

Maybe it’s just that they’re talking about problems rather than solutions, and describing the problems they’re working on in a way that makes them sound really far from being solved or understood.  I interpreted that as pessimism about how much their existing work does to resolve the problems, but that may be a misreading -- maybe the intended message is “here are the problems we’ve been working on for the last few years (by the way, we solved them!).”  Although that would really bury the lede, so I dunno.

Incidentally, reading that post led me to have some interesting interactions on LW.  I made a comment on that post, then decided to turn it into my own post, because I was kinda proud of how I was able to crystalize a certain objection I had to AIXI, LIA, etc. -- essentially, that tell you to search over a set that already contains everything you can possibly do as an element, so you can’t do the search.  (And if you can do anything approximating the search, that’s just an element in the set, so “you should do the search!” doesn’t tell you why it’s good.)

And someone pointed to me to the MIRI papers on “reflective oracles,” which I hadn’t looked into.  (Even though a reflective oracle paper got highly positive external reviewer commenters in the OpenPhil review of MIRI, unlike all the other papers reviewed.)  As it turns out, these papers were motivated, basically, by a search for some mathematical context in which my argument fails.  The papers frame it as being about game theory -- what kind of player could actually contain a perfect model of another player as complex as itself? -- but in the terms of my argument, this is asking, “what kind of search could contain itself as an element of the search space?”

And they succeed at coming up with such a thing, by doing something that’s almost “giving Turing machines oracles that tell them what Turing machines will do,” but with some probabilistic stuff that blocks self-reference paradoxes and doesn’t let them solve the halting problem.

It’s all very cool and trippy, but it ends in the same weird way that the Logical Induction paper did, where after constructing this apparently very delicate thing and working hard to prove that it has some very specific properties, they basically say “you can approximate it on a computer, maybe that’ll work!”, without talking at any level of rigor about which properties we might expect to transfer over to an approximation.  And this is probably the thing about MIRI’s work that I find most confusing: it seems like their preferred approach to solving practical problems is to 

clearly articulate the practical problems

ascend to a mystical Platonic world where ten eternities pass in the blink of an meta-eye, your breakfast eats you eating it in a infinite yet somehow terminating loop, etc.

carefully prove that in this world, you wouldn’t have the practical problems, although you might have others

what i’ve heard about the pentagon papers (haven’t read them) was that they show how unwinnable the war was, yet were written at a time when the president was still telling everyone we could do it

I also haven’t read the Pentagon Papers, but I do have the little July 1971 Bantam paperback edition, prepared by the New York Times. 

Neil Sheehan’s foreword suggests that the Pentagon Papers revealed a number of things, but the biggest thing they revealed was the gap between the public statements and its private beliefs of the America executive:

Nevertheless, the history as a whole demonstrates that the four Administrations progressively developed a sense of commitment to a non-Communist Vietnam, a readiness to fight the North to protect the South, and an ultimate frustration with this effort -- to a much greater extent than their public statements acknowledged at the time.

The New York Times believed that the Pentagon Papers showed that the American executive had been far more aggressive than the public knew or understood. 

Sheehan’s foreword suggests that the public’s concern had shifted away from whether the Vietnam War was winnable to something else. The New York Times may or may not have been concerned about whether the war was winnable, or whether the White House thought it winnable, but the New York Times was certainly concerned about who started it. 

Sheehan’s Pentagon Papers remind me of the histories of the Great War, searching for the key events and texts that will reveal the causes of the disaster, and the guilty men who made it happen. 

The [Pentagon Papers] historians were led to many broad conclusions and specific findings, including the following:

That the Truman Administration’s decision to give military aid to France in her colonial war against the Communist-led Vietminh “directly involved” the United States in Vietnam and “set” the course of American policy.

That the Eisenhower Administration’s decision to rescue a fledgling South Vietnam from a Communist takeover and the Administration’s attempt to undermine the new Communist regime of North Vietnam gave the Administration a “direct role in the ultimate breakdown of the Geneva settlement” for Indochina in 1954.

That the Kennedy Administration, though ultimately spared from major escalation decisions by the death of its leader, transformed a policy of “limited-risk gamble,” which it inherited, into a “broad commitment” that left President Johnson with a choice between more war and withdrawal.

That the Johnson Administration, though the President was reluctant and hesitant to take the final decisions, intensified the covert warfare against North Vietnam and began planning in the spring of 1964 to wage overt war, a full year before it publicly revealed the depth of its involvement and its fear of defeat.

That this campaign of growing clandestine military pressure through 1964 and the expanding program of bombing North Vietnam in 1965 were begun despite the judgment of the Government’s intelligence community that the measures would not cause Hanoi to cease its support of the Vietcong insurgency in the South, and that the bombing was deemed militarily ineffective within a few months.

That these four succeeding administrations built up the American political, military and psychological stakes in Indochina, often more deeply than they realized at the time, with large-scale shipments of military equipment to the French in 1950; with acts of sabotage and terror warfare against North Vietnam beginning in 1954; with moves that encouraged and abetted the overthrow of President Ngo Dinh Diem of South Vietnam in 1963; with plans, pledges, and threats of further action that sprang to life in the Tonkin Gulf clashes in August, 1964; with the careful preparation of public opinion for the years of open warfare that were to follow; and with the calculation in 1965, as the planes and troops were openly committed to sustained combat, that neither accommodation inside South Vietnam nor early negotiations with North Vietnam would achieve the desired result.

For Sheehan, the blame was as much structural as personal. He proceeds from this narration of events and actors to institutions, and to one institution in particular: the executive branch. 

To read the Pentagon papers in their vast detail is to step through the looking glass into a new and different world. This world has a set of values, a dynamic, a language and a perspective quite distinct from the public world of the ordinary citizen and the two other branches of the Republic -- Congress and the judiciary.

Clandestine warfare against North Vietnam, for example, is not seen, either in the written words of the senior decisionmakers in the Executive Branch or by the anonymous authors of the study, as violating the Geneva Accords of 1954, which ended the French Indochina War, or as conflicting with the public policy pronouncements of the various administrations. Clandestine warfare, because it is covert, does not exist as far as treaties and public posture are concerned. Further, secret commitments to other nations are not sensed as infringing on the treaty-making powers of the Senate, because they are not publicly acknowledged.

The guarded world of the government insider and the public world are like two intersecting circles. Only a small portion of the government circle is perceived from the public domain, however. Vigorous internal policy debates are only dimly heard and high-level intelligence analyses that contradict policy are not read outside. But, as the Pentagon papers demonstrate, knowledge of these policy debates and the dissents from the intelligence agencies might have given Congress and the public a different attitude toward the publicly announced decisions of the successive administrations.

The segments of the public world -- Congress, the news media, the citizenry, even international opinion as a whole -- are regarded from within the world of the government insider as elements to be influenced. The policy memorandums repeatedly discuss ways to move these outside “audiences” in the desired direction, through such techniques as the controlled release of information and appeals to patriotic stereotypes. The Pentagon papers are replete with examples of the power the Executive Branch has acquired to make its influence felt in the public domain.

On the New York Times’s theory of the case, the secrecy and autonomy of the American executive was the cause of the Vietnam War, and those evils were ones that the New York Times’s publication of the Pentagon Papers would help correct. 

The American people had been deceived by their executive. They did not know what they should have known. The Pentagon Papers were sold as “The Secret History of the Vietnam War.” The New York Times would show the American people what had been hidden and end the deceptions that allowed them to be led into the great disaster.

Like any memoirist, the New York Times is always the hero of its own story. Hence the paperback’s epitaph, from Justice Hugo Black’s opinion in the Pentagon Papers case: 

Only a free and unrestrained press can effectively expose deception in government. And paramount among the responsibilities of a free press is the duty to prevent any part of the Government from deceiving the people and sending them off to distant lands to die of foreign fevers and foreign shot and shell. ... In revealing the workings of government that led to the Vietnam war, the newspapers nobly did precisely that which the founders hoped and trusted they would do.

“#I need to leave an environment that conditions me to believe that this is very important and I am under threat.”

i never heard about any of this shit until like an hour ago when i got really stressed about something work related and went on tumblr to procrastinate

there’s a whole world of people not acting like this is a huge deal and our reactions to it determine the kind of society we will live in for the rest of our lives

really. Truly. I was just there.

the really stupid thing is: I know, and have always known, this

I know that the news cycle, especially the part college + social media people circulate, will repeatedly bring to my attention the people at the fringes in terms of being hostile and strident

I know that the numbers are on my side and the probability that I will be one of those people who says something in a public forum that is (taken out of context and) written about by majors news orgs and fired for it is microscopic

I belong to demographics that will make people who try to attack me look bad – by the law of numbers I would be safe even if this were not true 

I think ive said this before, but I think if I was in this field, I'd be approaching from the opposite direction. I think Bayesianism came from, "hmm why does probability theory seem to reproduce a lot of qualitative good advice about reasoning, when it's just counting possibilities? Is there another way to look at it?" I'd be trying to get something that works in SOME situations, and something else that works in OTHER situations, etc, and start from there for a general solution

This sounds right, although some cases are harder because you don’t have anything sitting around that does mysteriously well at the problem.  Like, clearly there are some things (say, humans) that can “do practical logical induction” pretty well, but we don’t have any prototype case that’s simple enough to analyze.

Maybe related: after I made that post, I thought “hmm, but do I have any positive examples of how to think about ideal behavior under constraints?”  And I thought, well, VC theory and PAC learning seem like examples of what I want.  These ideas start out with the observation that real “learners” (humans, ML algorithms) can do pretty well at inferring functions from finitely many examples, even though you can’t really be good at this in the fully general case where a function is just an arbitrary mapping between sets and f(x) tells you nothing about f(x’).  So they ask, how would the functions have to be restricted to make learning feasible?  And what sort of success metric captures the success of the real learners, without being too forgiving or too stringent?

So, these ideas start out by noting that there are versions of your problem that are too hard or too easy, and trying to find the “right scale” where some practical methods look better than others, rather than looking equally bad or good, or too coarsely distinguished.

Actually... now that I think about it, these are theories of online supervised learning, and logical induction is trying to do online supervised learning for logic. So, this isn’t just an analogy — these things are directly comparable. Just like with PAC learning, you could say: “if logic were just an arbitrary assignment of binary labels to sentences, then there would be no patterns to learn and the problem would be impossible. So, what properties does logic have that could make it more learnable? And how weak does our notion of learnability have to be?” Maybe you’ll find that PA is fundamentally too hard but some more restricted system is okay, maybe you’ll find that PAC is too hard but a weaker learning concept is appropriate, that sort of thing.

Viewed from the online supervised learning perspective, MIRI’s criterion essentially says, “you should use a nonparametric method, because a parametric method will have some blind spots that persist no matter how much data has come in, and someone without these blind spots could make winning trades against your parametric model forever.”

And this is . . . a fair if limited point about machine learning, although it has nothing really to do with logic!  Like, in a nonlinear online learning problem, someone using a random forest that grows with N could pump money out of someone using logistic regression, and they could do this forever.  This is true but it’s not the only fact that’s ever relevant about these two methods.  You can’t make a good general theory of online supervised learning out of just this one distinction, and it’s not clear why this distinction would be any more (or less) important for predicting logic than for anything else.

So this is picking the “wrong scale” for the problem, sorta -- your evaluation is too strict in one way (you have to do well forever) and too forgiving in another (you only have to do well in the long run), and this ends up making a distinction between parametric and nonparametric and not getting any finer-grained than that, with the weird implication that the worst (consistent) nonparametric methods are better than the best parametric methods.  This is so simple, I’m really not sure why I didn’t think about it before.

One of the OpenPhil reviewers actually said something about the relation between PAC learning and MIRI’s framework (this was about the paper “Inductive Coherence”):

At a very high level, there is some overlap between the type of work considered here and work on making decisions optimally in the face of computational constraints, since making such decision might involve approximating probability. [...] In spirit, the paper is also close to work going back to the 1960s on language identification in the limit. Work on this topic has been largely superseded by a weaker model, Valiant’s notion of PAC (probably approximately correct) learning. (As an aside, if I were a referee of this paper for a journal, I would ask the authors to compare their work to Gold’s work on language identification.)