Wow, what a perfect description of why their probability-logic leads to silly beliefs.
I've been wondering how to argue within their frame for a while, and here's what I've come up with: Is the likelihood that aliens exist, are unfriendly, and AGI will help us beat them higher or lower than the likelihood that the AGI itself that we develop is unfriendly to us and wants to FOOM us? Show your work.
It’s pointless. They aren’t rational. Any argument you come up with that contradicts their personal desires will be successfully “reasoned” away by them because they want it to be. Your mistake was ever thinking they had a rational thought to begin with, they think they are infallible.
"Widespread robots that make their own decisions autonomously will probably be very bad for humans if they make decisions that aren't in our interest" isn't really that much of a stretch is it?
If we were going slower maybe it would seem more theoretical. But there are multiple Manhattan-Project-level or (often, much) larger efforts ongoing to explicitly create software and robotic hardware that makes decisions and takes actions without any human in the loop.
We don't need some kind of 10000 IQ god intelligence if a glitch token causes the majority of the labor force to suddenly and collectively engage in sabotage.
None of those projects are even heading in the direction of "AGI". The state of AI today is something akin to what science fiction would have called an "oracle", a devise that can answer questions intelligently or seemingly-intelligently but otherwise isn't intelligent at all: it can't learn, has no agency, does nothing new. Even if that can be scaled up indefinitely, there's no reason to believe that it will ever become an AGI.
If any of this can make decisions in a way that a human can, then I would start to question what human decision-making really amounts to.
For what it's worth, as the person at the top of this thread, I actually do take AI risk pretty seriously. Not in a singulatarian sense, but in the sense that I would be quite surprised if AI weren't capable of this stuff in ten years.
Even the oracle version is already really dangerous in the wrong hands. A totalitarian government doesn't need to have someone listening to a few specific dissidents if they can have an AI transcriber write down every word of every phone conversation in the country, for example. And while it's certainly not error-proof, asking an LLM to do something like "go through this list of conversations and flag anything that sounds like anti-government sentiment" is going to get plenty of hits, too.
> "Widespread robots that make their own decisions autonomously will probably be very bad for humans if they make decisions that aren't in our interest" isn't really that much of a stretch is it?
We already have widespread humans making their own decisions autonomously that aren’t in the best interest of humans, and we’re all still here.
Much of philosophy throughout history seems to operate this way.
I think philosophy is a noble pursuit, but it's worth noting how often people drew very broad conclusions, and then acted on them, from not very much data. Consider the dozens of theories of the constitution of the world from the time of the Greek thinkers (even the atomic theory doesn't look very much at all like atoms as we now understand them), or the myriad examples of political philosophies that ran up against people simply not acting the way the philosophy needed them to act to cohere.
The investigation of possibility is laudable, but a healthy and regular dose of evidence is important.
> Much of philosophy throughout history seems to operate this way.
“Philosophy is poor at revealing truths but excellent at revealing falsehoods (or at least unsupported arguments)” was the main lesson I took from informally studying it.
Anything about the self bumps into an immediate problem here. For instance I cannot prove to you that I'm conscious and not simply an automaton who's not actually thinking. My evidence for such is strictly personal - I can personally testify to my own experience of consciousness, but you have no reason to believe me since that's not evidence.
And in fact even for myself - perception is not necessarily valid evidence since perception can be distorted. If I am in a compelling VR game I might be more than willing to swear to the fact that I'm flying (if I wasn't otherwise aware of the situation) - while you simply look at me acting a fool standing still while vigorously flapping my arms.
... so at some point, one realizes one has pondered one's way into untestables and goes back to living. Or doesn't, I guess, and then gets kept up at night anxious about the notion that in some as-yet-unrealized future, an AI is forever torturing an identical copy of oneself that one cannot possibly ever meet.
The programming analogy to this kind of philosophy is writing design docs (or building a class hierarchy of abstracts) without ever writing implementation. Lots of work, but why should anyone outside the room care?
It contradicts the ideals of an evidence based system of values. Most of what we believe we believe because we think it is right, and there's always , more or less, viable arguments for most of any remotely reasonable view. And this applies to all people. For instance it was none other than Max Planck that observed, "Science progresses one funeral at a time."
I also think this is for the best. If one looks at the evidence of the skies it's indisputable that humanity lies at the center of the cosmos, with everything revolving around us - which, in turn, naturally leads into religious narratives around Earth. It's only thanks to these weirdos that adopted quite absurd sounding (for the time) systems of views and values, completely without meaningful evidence at first, that we were able to gradually expand, and correct, our understanding of the universe.
And this doesn't just apply to the ancient past. Einstein's observation that the speed of light stays fixed while the rate of passage of time itself is variable, to enable the former, sounds so utterly and absolutely absurd. In many ways the real challenge in breaking through to relativity wasn't the math or anything like that (which, in fact, Einstein himself lacked when first developing the concept) but accepting that a concept that sounds so wrong might actually be right.
At the same time, however, Einstein's theory was confirmed by direct observation of the gravitational lensing of starlight past our sun during an eclipse.
The difference, to my mind, between science and philosophy is that the philosophy of relativity can exist independent of such confirming evidence. But the scientific theory cannot.
It's always important, when dabbling in philosophy, to keep track of when one has wandered so far out on the branch of reason that one is no longer supported by the trunk of evidence.
Excellent point. And it shines a light directly on what I'm saying. There's a great line from the Hitchhiker's Guide to the Galaxy:
""""Oh, that was easy," says Man, and for an encore goes on to prove that black is white and gets himself killed on the next zebra crossing.”"""
... and, of course, the philosopher can argue that we are all of us tumbling through a Pachinko-machine of parallel universes, and it is only our constrained perception that suggests Man is dead; in a parallel universe, Man is just fine and living their days in the grey universe that we, through some as-yet-unphilosophized-but-don't-worry-we're-thinking-hard-about-it process, cannot exist in.
... but does that matter over-much if the observable is that ignoring zebra crossings gets you crossed out of this universe?
very few philosophers dared to live by their theories. the famous failures of aristotle (in the case of alexander) and plato in syracuse (where he saw firsthand that the philosopher-king is at best a book character) are good examples. seneca didn’t live stoically: he was avaricious and didn’t bother to incite civil war over unpaid debt, if ancient sources are to be believed. he failed horribly with nero, who later instructed him to commit suicide for treasonous crimes. again, if ancient sources are to be believed, he fumbled his suicide out of raw fear.
the cynics, though, made a good life but that’s not because they had a better philosophy. it’s because cynicism is base/primitive logic available to the brute as well as the civilized man.
I actually have on my desk, right now, Mary Wollstonecraft's "A Vindication of the Rights of Woman."
To summarize one of her points from memory, she basically lays into Rousseau about having a lot of opinions on domestic life for someone who clearly doesn't even know how to set foot in a kitchen.
AGI would be extremely helpful in navigating clashes with aliens, but taking the time to make sure it's safe is very unlikely to make a difference to whether it's ready in time. Rationalists want AGI to be built, and they're generally very excited about it, e.g. many of them work at Anthropic. They just don't want a Move Fast and Break Things pace of development.
> The mugger argues back that for any low but strictly greater than 0 probability of being able to pay back a large amount of money (or pure utility) there exists a finite amount that makes it rational to take the bet.
This is a basic logic error. It ignores the very obvious fact that increasing the reward amount decreases the probability that it will be returned.
E.g. if the probability of the reward R being returned is (0.5/R) we get "a low by strictly greater than 0 probability", and for that probability there is a (different) finite reward that would make it rational to take the bet, but it's not R.
This is even simpler and more stupid than the "proofs" that 0=1. It does not change my opinion that philosophers (and lesswrong) are idiots.
That's what the person responding to meant - attempts to make human systems "rational" often involves simplifying dependent probabilities and presenting them as independent.
The rationalist community both frequently makes this reasoning error and is aware they frequently make this error, and coined this term to refer to the category of reasoning mistake.
> coined this term to refer to the category of reasoning mistake.
That's not at all what the Wikipedia article for it says. It presents it as an interesting paradox with several potential (and incorrect!) "remedies" rather than a category of basic logical errors.
The “quadrillion days of happiness” offered to a rational person gives away that such allegories are anthropomorphized just for the sake of presentation. For the sake of what the philosophers mean, you should probably imagine this as an algorithm running on a machine (no AGI).
It’s a mental tease, not a manual on how to react when faced with a mugger who forgot his weapon at home but has an interesting proposition to make.
Similarly the trolley problem isn’t really about real humans in that situation, or else the correct answer would always be “do nothing”.
It’s what the comment here [0] says. If you try to analyze everything purely rationally it will lead to dark corners of misunderstanding and madness.
> Similarly the trolley problem isn’t really about real humans in that situation, or else the correct answer would always be “do nothing”.
The correct answer in case of it being about real people, of course, is to switch immediately after the front bogey makes it through. This way, the trolley will derail and make a sharp turn before it runs over anyone, and stop.
The passengers will get shaken, but I don’t remember fatalities being reported when such things happened for real with real trams.
The scenario is set up by an evil philosopher though, so they can tie up the people arbitrarily close to the split in the rails, such that your solution doesn’t work, right?
In this case, it won’t matter, I’m afraid, which way the trolley goes as it will at least mangle both groups of people, and the only winning move is to try to move as many people as possible away from the track.
An Eastern European solution is to get a few buddies to cut the electrical wire that powers the trolleys and sell it for scrap metal, which works on all electrical trolleys. (After the trolley stops, it can be scavenged for parts you can sell as scrap metal, too.)
Made me chuckle. Funny 'cause it's true. About the trolley problem, if taken literally (people on tracks, etc.) pulling the lever exposes you to liability: you operated a mechanism you weren't authorized to use and for which you had no prior training, and you decided to kill one innocent person that was previously safe from harm.
Giving CPR is a very tame/safe version of the trolley problem and in some countries you're still liable for what happens after if you do it. Same when donating food to someone who might starve. Giving help has become a very spiny issue. But consciously harming someone when giving help in real life is a real minefield.
P.S. These philosophical problems are meant to force a decision from the options given. So assume the the problem is just a multiple choice one, 2 answers. You don't get to write a third.
> P.S. These philosophical problems are meant to force a decision from the options given. So assume the the problem is just a multiple choice one, 2 answers. You don't get to write a third.
I know about it. And yet I refuse to play the game. The problem is that even philosophers should be able to acknowledge that in the real universe, no box should be too big to prevent from thinking outside of it.
Otherwise we get people who conflate map with the territory, like what this whole comment thread is about.
> The “quadrillion days of happiness” offered to a rational person gives away that such allegories are anthropomorphized just for the sake of presentation.
So what? It's still presented as if it's a interesting problem that needs to be "remedied", when in fact it's just a basic maths mistake.
If I said "ooo look at this paradox: 1 + 1 = 2, but if I add another one then we get 1 + 1 + 1 = 2, which is clearly false! I call this IshKebab's mugging.", you would rightly say "that is dumb; go away" rather than write a Wikipedia article about the "paradox" and "remedies".
> Similarly the trolley problem isn’t really about real humans in that situation, or else the correct answer would always be “do nothing”.
It absolutely wouldn't. I don't know how anyone with any morals could claim that.
Interestingly, the trolley problem is decided every day, and humanity does not change tracks.
There are people who die waiting for organ donors, and a single donor could match multiple people. We do not find an appropriate donor and harvest them. This is the trolley problem, applied.
I would pull the lever in the trolley problem and don't support murdering people for organs.
The reason is that murdering people for organs has massive second-order effects: public fear, the desire to avoid medical care if harvesting is done in those contexts, disproportionate targeting of the organ harvesting onto the least fortunate, etc.
The fact that forcibly harvesting someone’s organs against their will did not make your list is rather worrying. Most people would have moral hangups around that aspect.
Yea, it doesn’t seem quite right to say that the trolley problem isn’t about really people. I mean the physical mechanical system isn’t there but it is a direct abstraction of decisions we make every day.
My actual words quoted below give one extra detail that makes all the difference, one that I see people silently dropped in a rush to reply. The words were aimed at someone taking these problems in a too literal sense, as extra evidence that they are not to be taken as such but as food for though that has real life applicability.
> the trolley problem isn’t really about real humans in that situation
> We do not find an appropriate donor and harvest them. This is the trolley problem, applied.
I don't think that matches the trolley problem particularly well for all sorts of reasons. But anyway your point is irrelevant - his claim was that the trolley problem isn't about real humans, not that people would pull the lever.
Edit: never mind, I reread your comment and I think you were also agreeing with that.
> his claim was that the trolley problem isn't about real humans
Is it though? Let's look at the comment [0] written 8h before your reply:
> the trolley problem isn’t really about real humans in that situation
As in "don't take things absolutely literally like you were doing, because you'll absolutely be wrong". You found a way to compound the mistake by dropping the critical information then taking absolutely literally what was left.
It seems that you didn't understand the main point of the exposition. I'll summarize the ops comment a bit further.
Points 1 and 2 only explain how they are able to erroneously justify their absurd beliefs, they don't explain why they hold those beliefs.
Points 3 through 5 are the heart of the matter; egotistical and charismatic (to some types of people) leaders, open minded, freethinking and somewhat weird or marginalized people searching for meaning plus a way for them all to congregate around some shared interests.
TLDR: perfect conditions for one or more cults to form.
I've been wondering how to argue within their frame for a while, and here's what I've come up with: Is the likelihood that aliens exist, are unfriendly, and AGI will help us beat them higher or lower than the likelihood that the AGI itself that we develop is unfriendly to us and wants to FOOM us? Show your work.