I played a game of DnD that changed the way I think about everything. Here’s the story.
Dungeons & Dragons & Otherworldly intellects (or at least AI)
I was playing a game of Dungeons and Dragons, a fairly informal session with some nerdy friends in the military. I was acting as the Dungeon Master, who is the person that makes up the scenario, picks the mobs that show up, determines the loot, does all the game engine stuff.
DM: You round the corner to see a large monster. The light is dim, and you can’t quite tell what it is. What do you do?
Player 1: I light a torch.
DM: The flickering torchlight reveals an owlbear. It looks angry.
Player 1: I draw my sword.
PLayer 2: I ready material components to cast magic missile.
Player 3: Is the monster intelligent?
DM: Uh… what? looks at status sheets Yes. The monster is intelligent.
Player 3: Can we just ask why the monster is angry?
And lo, I was struck speechless. I didn’t have an answer. And, oddly enough, that’s something that’s stuck with me for a while.
Mental experiment
Here, let’s try a mental experiment.
Illustration
An illustration of a pong paddle and ball
Benjamin Wilson
This is a game of pong. See square primitive paddle, round primitive ball.
Q: How many ways can the ball go through the paddle?
A: None.
And that’s often where our logic stops; the ball can’t go through the paddle. The options that have been presented are non-starters. But what if I told you there was another way? Infinite (∞-1, really) ways to solve this riddle?
Q: How many ways can the ball get on the other side of the paddle?
A: Nearly infinite ways.
This is a simple, 2-dimensional puzzle. But the ball can just go around. There are a number of alternatives in this spatial puzzle. In life, however, we often limit our choices to avoid decision paralysis. It is infinitely difficult to make one choice from infinite options, so we often rely on a heuristic of choosing from the choices that are presented. This can be a mistake; the presented choices may not be all the choices there are.
Creativity with and without limits
One of my professors said “Creativity works best within limits”, and I often feel that he was right. If you start with tabula rasa, with a blank slate, it’s difficult to create something. You need a prompt, an idea, a reference - and all of these limit you. If you are painting, and there are no rules, and you have six reference images to paint from, you would typically pick one of the six. A statistically minded person would probably think that the odds of picking any one of the six references would be 6/6: each reference has a 1/6 chance, so you can add the probabilities to get the total probability. 6/6. 100%. Although, you don’t actually have to paint from one of the six references. You can paint anything. There are no rules; it said so, right up front.
We build rules, though, to make an infinite universe capable of comprehension. Given a choice of three cars, it is unlikely I will pick a fourth car. Given the choice of a predefined set of options, it is unlikely we will create our own, fourth option.
AI and the black box
AI models can’t usually tell you why they surface the information they surface - not because that’s proprietary information, but because it’s not really interpretable that way. AI model outputs are non-deterministic, which means that you won’t necessarily get the same information every time. So they can’t tell you. If they can’t tell you, you can’t know, right?
And that’s the mental trap of choosing from the available options. You have two options: Success or failure. You could, however, have other options; alternate successes and failures, and other states that don’t include success or failure. If you think that the only way to know information that an AI has access to is for the AI to tell you, then there are only two potential states; success or failure. But the other way to convey information, as any writer will recite, is to show you, rather than tell you.
My multi-model content grading methodology is based on that idea;
- If you can’t ask an AI model exactly how well it will be able to answer all user questions in the world, how else can you get that information?
- The AI can’t tell you. Can it show you?
- What would that look like?
- How would we design that process?
- What does success look like, after it is designed?
That’s the million-dollar set of questions. I have a pretty good idea, and I’ve turned it into a working architecture that’s shown results in the form of happier users and more users getting to the pages.
It’s a start.