I mean, if you play on the doom to hype yourself, dealing with employees that take that seriously feel like a deserved outcome.

I saw people making fun of this on (the normally absurdly overly credulous) /r/singularity of all places. I guess even hopeful techno-rapture believers have limits to their suspension of disbelief.

First of all. You could make facts a token value in an LLM if you had some pre-calculated truth value for your data set.

An extra bit of labeling on your training data set really doesn’t help you that much. LLMs already make up plausible looking citations and website links (and other data types) that are actually complete garbage even though their training data has valid citations and website links (and other data types). Labeling things as “fact” and forcing the LLM to output stuff with that “fact” label will get you output that looks (in terms of statistical structure) like valid labeled “facts” but have absolutely no guarantee of being true.

Broadly? There was a gradual transition where Eliezer started paying attention to deep neural network approaches and commenting on them, as opposed to dismissing the entire DNN paradigm? The watch the loss function and similar gaffes were towards the middle of this period. The AI dungeon panic/hype marks the beginning, iirc?

It is even worse than I remembered: https://www.reddit.com/r/SneerClub/comments/hwenc4/big_yud_copes_with_gpt3s_inability_to_figure_out/ Eliezer concludes that because it can’t balance parentheses it was deliberately sandbagging to appear dumber! Eliezer concludes that GPT style approaches can learn to break hashes: https://www.reddit.com/r/SneerClub/comments/10mjcye/if_ai_can_finish_your_sentences_ai_can_finish_the/

iirc the LW people had betted against LLMs creating the paperclypse, but they now did a 180 on this and they now really fear it going rogue

Eliezer was actually ahead of the curve on overhyping LLMs! Even as far back as AI Dungeon he was claiming they had an intuitive understanding of physics (which even current LLMs fail at if you get clever with questions to stop them from pattern matching). You are correct that going back far enough Eliezer really underestimated Neural Networks. Mid 2000s and late 2000s sequences posts and comments treat neural network approaches to AI as cargo cult and voodoo computer science, blindly sympathetically imitating the brain in hopes of magically capturing intelligence (well this is actually a decent criticism of some of the current hype, so partial credit again!). And mid 2010s Eliezer was focusing MIRI’s efforts on abstractions like AIXI instead of more practical things like neural network interpretability.

I unironically kinda want to read that.

Luckily LLMs are getting better at churning out bullshit, so pretty soon I can read wacky premises like that without a human having to degrade themselves to write it! I found a new use case for LLMs!

Sneerclub tried to warn them (well not really, but some of our mockery could be interpreted as warning) that the tech bros were just using their fear mongering as a vector for hype. Even as far back as the OG mid 2000s lesswrong, a savvy observer could note that much of the funding they recieved was a way of accumulating influence for people like Peter Thiel.

I think this is the first mention of the Brennan email on LW?

That is actually kind of weird… Did the lesswrong mods deliberately censor all discussion of the emails? (Out of a misplaced sense of respect for what gets the privilege of privacy? Or deliberately covering up the racism? Or the later disguised as the former?) They seem foundational to understanding Scott’s true motives, it seem like the emails should have at least warranted a tangential mention. Trying to clear this up… but searching for Brennan doesn’t help as an original fiction character has that name and searching for emails doesn’t help as it gets the Bostrom emails.

So, I was morbidly curious about what Zack has to say about the Brennan emails (as I think they’ve been under-discussed, if not outright deliberately ignored, in lesswrong discussion), I found to my horror I actually agree with a side point of Zack’s. From the footnotes:

It seems notable (though I didn’t note it at the time of my comment) that Brennan didn’t break any promises. In Brennan’s account, Alexander “did not first say ‘can I tell you something in confidence?’ or anything like that.” Scott unilaterally said in the email, “I will appreciate if you NEVER TELL ANYONE I SAID THIS, not even in confidence. And by ‘appreciate’, I mean that if you ever do, I’ll probably either leave the Internet forever or seek some sort of horrible revenge”, but we have no evidence that Topher agreed.

To see why the lack of a promise is potentially significant, imagine if someone were guilty of a serious crime (like murder or stealing billions of dollars of their customers’ money) and unilaterally confessed to an acquaintance but added, “Never tell anyone I said this, or I’ll seek some sort of horrible revenge.” In that case, I think more people’s moral intuitions would side with the reporter.

Of course, Zack’s ultimate conclusion on this subject is the exact opposite of the correct one I think:

I think that to people who have read and understood Alexander’s work, there is nothing surprising or scandalous about the contents of the email.

I think the main reason someone would consider the email a scandalous revelation is if they hadn’t read Slate Star Codex that deeply—if their picture of Scott Alexander as a political writer was "that guy who’s so committed to charitable discourse

Gee Zack, I wonder why so many people misread Scott? …Its almost like he is intentionally misleading about his true views in order to subtly shift the Overton window of rationalist discourse and intentionally presents himself as simply committed to charitable discourse while actually having a hidden agenda! And the bloated length of Scott’s writing doesn’t help with clarity either. Of course Zack, who writes tens of thousands of words to indirectly complain about perceived hypocrisy of Eliezer’s in order to indirectly push gender essentialist views, probably finds Scott’s writings a perfectly reasonable length.

Edit: oh and a added bonus on the Brennan Emails… Seeing them brought up again I connected some dots I had missed. I had seen (and sneered at) this Yud quote before:

I feel like it should have been obvious to anyone at this point that anybody who openly hates on this community generally or me personally is probably also a bad person inside and has no ethics and will hurt you if you trust them, but in case it wasn’t obvious consider the point made explicitly.

But somehow I had missed or didn’t realize the subtext was the emails that laid clear Scott’s racism:

(Subtext: Topher Brennan. Do not provide any link in comments to Topher’s publication of private emails, explicitly marked as private, from Scott Alexander.)

Hmm… I’m not sure to update (usage of rationalist lingo is deliberate and ironic) in the direction of “Eliezer is stubbornly naive on Scott’s racism” or “Eliezer is deliberately covering for Scott’s racism”. Since I’m not a rationalist my probabilities don’t have to sum to 1, so I’m gonna go with both.

ghost of 2007!Yud

This part gets me the most. The current day Yud isn’t transphobic (enough? idk) so Zack has to piece together his older writings on semantics and epistemology to get a more transphobic gender essentialist version of past Yud.

Right, its a joke, in the sense that the phrase “Caliph” started its usage in a non-serious fashion that got a chuckle, but the way Zack uses it, it really doesn’t feel like a joke. It feels like the author genuinely wants Eliezer to act as the central source of authority and truth among the rationalists and thus Eliezer must not endorse the heresy of inclusive language or else it will mean their holy prophet has contradicted the holy scripture causing a paradox.

The hilarious part to me is that they imagine Eliezer moderates himself or self-censors particularly in response to sneerclub. Like of all the possible reasons why Eliezer may not want to endorse transphobic rhetoric about pronouns (concern about general PR besides sneerclub, a more complex nuanced understanding of language, or even genuine compassion for trans people), sneerclubs disapproval is the one that sticks out to the author. I guess good job on us? Keep it up!

The thing that gets me the most about this is they can’t imagine that Eliezer might genuinely be in favor of inclusive language, and thus his use of people’s preferred pronouns must be a deliberate calculated political correctness move and thus in violation of the norms espoused by the sequences (which the author takes as a given the Eliezer has never broken before, and thus violating his own sequences is some sort of massive and unique problem).

To save you all having to read the rant…

—which would have been the end of the story, except that, as I explained in a subsequent–subsequent post, “A Hill of Validity in Defense of Meaning”, in late 2018, Eliezer Yudkowsky prevaricated about his own philosophy of language in a way that suggested that people were philosophically confused if they disputed that men could be women in some unspecified metaphysical sense.

Also, bonus sneer points, developing weird terminology for everything, referring to Eliezer and Scott as the Caliphs of rationality.

Caliphate officials (Eliezer, Scott, Anna) and loyalists (Steven) were patronizingly consoling me

One of the top replies does call this like it is…

A meaningful meta-level reply, such as “dude, relax, and get some psychological help” will probably get me classified as an enemy, and will be interpreted as further evidence about how sick and corrupt is the mainstream-rationalist society.

I was on the old reddit /r/sneerclub, same username. Finally making an account here. I will probably post a few sneequence classics that haven’t been discussed mocked properly before.

I kinda agree with, but the post does correctly point out the Eliezer ignored a lot of the internal distinctions between philosophical positions and ignored how the philosophers use their own terminology. So even though I also think p-zombies are ultimately an incoherent thought experiment I don’t think Eliezer actually did a good job addressing them.