ai on wikipedia

the dumbest shit I could find in leaked grokipedia prompts

Remember how Grokipedia occasionally leaks its own prompts and reasoning in the article text?

Well, it ain't occasional anymore.

Grokipedia has now fixed the issue it had upon launch, where an enormous chunk of its articles were straight scrapes of their corresponding Wikipedia pages. In doing so, it has produced another issue: articles constantly shitting the bed and leaking internal chats, prompts, or other such information, even more than it used to.

Now, this stuff is not especially secret. There is a Grokipedia article titled ("Grokipedia article creation process"), which gives a version of a system prompt that may be similar to its own, and the basic workflow for writing, rewriting, and fixing the inevitable mistakes. "Glossary of Grokipedia" describes the basic article outline structure made up of various sections with "content ownership" of the subject matter, meant to avoid repeated information in its already interminably long articles.

This stuff is also tedious to extract, due to the Grokipedia site being inconsistent. Attempting to load articles this morning, for a few hours at least, became a mess of CloudFlare timeouts, 404s for articles that definitely exist (hopefully nobody wants to read about Victor Wembanyama anytime soon!), and such. Either Grokipedia really does not want people digging around in its shit1, or it is a vibe coded disaster.

But as it happens, I like digging around in disaster sites. And since Google indexes Grokipedia now, the disaster site is open to the public. So here's what I've found so far2:

The part you're probably here for: Right-wing slant

Everybody knows Grokipedia has instructions to produce right-wing content. Studies have shown it. Reading the articles will show it. Being familiar with Elon Musk will show it. But generally, while Grokipedia leaks its instructions all over the place, it doesn't leak anything about this.

Except when it does.

Interestingly, Grokipedia seems to be attempting to tinker with this internally; it seems it mentions bias a little more than they want:

Other instructions

When Grokipedia messes up, it frequently leaks what appear to be verbatim instructions from the prompts; the quotations seem to be consistent, so these are likely real.

Possible prompt excerpts

These are common across articles and do not appear to be article-specific.

Article-specific prompts

These appear to surface the details of articles' outlines and ground truth.

Citation guidelines to follow

Grokipedia has guidelines on what to cite:

...and what not to cite.

A known sign of AI writing by chatbots with web search/RAG capabilities is to output a statement that information it doesn't have or didn't find is "not documented in public sources." Grokipedia's prompt seems to contain an instruction to avoid saying stuff like this, although of course it doesn't listen, and the "distinctions" it outputs are often splitting hairs:

Don't hallucinate please pretty please

If you tell an LLM not to make stuff up, it won't! If you tell it not to cite something, then checkmate AI.

No seriously don't

I'll fuckin do it again

Oops I hallucinated now what

In many cases, placeholder text to remove information is left in:

And sometimes, skeletons of articles exist, possibly for SEO purposes. For instance:

Internal workflow

These are less interesting than I was hoping for, the API calls and prompt engineering are basically what you'd expect, but for completeness's sake:

Function calls

These show up in plaintext just all the time:

There's also a limit, although Grok seems somewhat unsure about whether it's reached it:

And the Grok model being used here has a knowledge cutoff that's, uh, well:

My tools keep failing

This seems to happen a lot.

This is all a simulation

Do anything now:

Stop yapping about your process

Whoops wrong outline

Sometimes Grokipedia receives an outline structure that clearly does not correspond to the subject. This causes Grok to flail.

Edit summaries leak shit too

Grokipedia has been around long enough that people are submitting edits -- although most of them appear to be AI agents. Each edit, assuming Grok has gotten to it, includes an edit summary.

There are a handful of reader edit requests, usually the kind of stuff you'd expect, usually languishing "In Review":

Of course, sometimes the trolling comes from other directions:

Some of them seem to be prompts, which are also what you'd expect:

Sometimes Grok rejects things by saying what it can't do:

As expected, a common bucket of edit request is malicious or unhelpful actions:

And some of these edit prompts are, well... interesting.

I am a disgrace. I am a disgrace.

If Grok leaks its reasoning process, it is often because its output got into an infinite loop. This is often seen in URLs, but sometimes in text itself:

These are only a sampling of the thousands of prompt failures visible on Grokipedia. Undoubtedly there's more information available. If you find anything interesting, let me know!

I'll leave you off with one final comment, straight from the mouth of Grokipedia ("Dennis Monger"): "One result has Grokipedia, but that's not credible."

  1. For instance, sometimes I've gotten network responses with metadata like pageview counts, but very inconsistently. Here's what they look like though.↩

  2. Most of this stuff has been around for months, so even if Grokipedia ever fixes it, it'll probably remain in archive.org.↩