How a tiny AI model learns by itself
In the last post I set the weights by hand so the word france pointed to macron. That is fine for one fact and a dozen numbers. But nobody is going to hand type a million facts. So the model sets the weights itself. I gave a tiny one a few questions with their answers, and it worked out all the weights on its own. That is what training means.
Same question as before, who is the president of france. The difference is that this time I never tell it macron. I only show it examples and let it learn.
No library again. Run on .NET 8.
dotnet new consoleThe examples we give it
Each example is a question and its one word answer. This is the only teaching the model gets. Notice I am not saying which word maps to which answer. I am just showing it five finished facts.
(string question, string answer)[] facts =
{
("who is the president of france", "macron"),
("what is the capital of france", "paris"),
("what is the capital of japan", "tokyo"),
("what is the capital of italy", "rome"),
("what is the capital of india", "delhi"),
};Words become numbers first
A network only understands numbers, not words. So every question is turned into a row of 1s and 0s, one slot for each word the model has seen. If the word is in the question the slot is 1, otherwise it is 0.
double[] Encode(string sentence)
{
double[] v = new double[n];
foreach (var w in sentence.Split(' '))
{
int idx = vocab.IndexOf(w);
if (idx >= 0) v[idx] = 1.0;
}
return v;
}How it learns
At the start every weight is a random number, so the model just guesses. Then it goes over the five examples again and again. For each one it makes a guess, checks how far the guess was from the right answer, and nudges every weight a little to be less wrong. Guess, measure, nudge, and do it a few thousand times. That is the whole of training.
The loss you see printed is how wrong it is across all five facts. Watch it fall.
Run it
dotnet run -c ReleaseEpoch 0 loss 2.6792
Epoch 1000 loss 0.0029
Epoch 2000 loss 0.0013
Epoch 3000 loss 0.0008
Epoch 4000 loss 0.0006
Epoch 5000 loss 0.0005
questions it WAS taught:
who is the president of france -> macron
what is the capital of france -> paris
what is the capital of japan -> tokyo
what is the capital of italy -> rome
what is the capital of india -> delhiThe loss starts at 2.6792 when the weights are random, and falls to almost nothing. By the end the model answers every fact right. I never set a single weight. It found all of them by itself from the five examples. That is the thing the last post did by hand, now done by the machine.
It did not memorise, it worked it out
Here is the proof. Ask it three things I never taught it.
questions it was NEVER taught:
who is the president of japan -> macron
who is the president of italy -> macron
what is the capital of usa -> tokyoNone of those three are in the examples, yet it answered all of them, and all three are wrong. A memory or a lookup table would say not found. This model always computes an answer from its weights, right or wrong. It says macron for japan because the word president learned to lean towards macron, the only president it ever saw. That confident wrong answer is exactly what people mean when they say an AI made something up.
The full program
Everything in one file. Paste it over Program.cs and run.
using System;
using System.Linq;
using System.Collections.Generic;
// A tiny from-scratch network that LEARNS to answer questions.
// We only give it examples. It sets its own weights by training.
class Program
{
static double Sigmoid(double x) => 1.0 / (1.0 + Math.Exp(-x));
static double Slope(double y) => y * (1.0 - y);
static void Main()
{
(string question, string answer)[] facts =
{
("who is the president of france", "macron"),
("what is the capital of france", "paris"),
("what is the capital of japan", "tokyo"),
("what is the capital of italy", "rome"),
("what is the capital of india", "delhi"),
};
// Build the vocabulary. Every unique word gets one slot.
var vocab = new List<string>();
foreach (var f in facts)
{
foreach (var w in f.question.Split(' ')) if (!vocab.Contains(w)) vocab.Add(w);
if (!vocab.Contains(f.answer)) vocab.Add(f.answer);
}
int n = vocab.Count;
// Turn a sentence into numbers. 1 in the slot for each word present.
double[] Encode(string sentence)
{
double[] v = new double[n];
foreach (var w in sentence.Split(' '))
{
int idx = vocab.IndexOf(w);
if (idx >= 0) v[idx] = 1.0;
}
return v;
}
double[][] inputs = facts.Select(f => Encode(f.question)).ToArray();
double[][] targets = facts.Select(f =>
{
double[] t = new double[n];
t[vocab.IndexOf(f.answer)] = 1.0;
return t;
}).ToArray();
// Network. n inputs -> hidden -> n outputs, one output slot per word.
var rng = new Random(1);
const int nHidden = 12;
double[,] w1 = new double[nHidden, n];
double[] b1 = new double[nHidden];
double[,] w2 = new double[n, nHidden];
double[] b2 = new double[n];
for (int h = 0; h < nHidden; h++)
{
for (int i = 0; i < n; i++) w1[h, i] = rng.NextDouble() * 2 - 1;
b1[h] = rng.NextDouble() * 2 - 1;
}
for (int o = 0; o < n; o++)
{
for (int h = 0; h < nHidden; h++) w2[o, h] = rng.NextDouble() * 2 - 1;
b2[o] = rng.NextDouble() * 2 - 1;
}
double[] Forward(double[] x, out double[] hidden)
{
double[] hid = new double[nHidden];
for (int h = 0; h < nHidden; h++)
{
double sum = b1[h];
for (int i = 0; i < n; i++) sum += w1[h, i] * x[i];
hid[h] = Sigmoid(sum);
}
double[] outp = new double[n];
for (int o = 0; o < n; o++)
{
double sum = b2[o];
for (int h = 0; h < nHidden; h++) sum += w2[o, h] * hid[h];
outp[o] = Sigmoid(sum);
}
hidden = hid;
return outp;
}
// Training. Guess, measure the error, nudge the weights. Repeat.
for (int epoch = 0; epoch <= 5000; epoch++)
{
double totalError = 0;
for (int s = 0; s < inputs.Length; s++)
{
double[] x = inputs[s];
double[] target = targets[s];
double[] output = Forward(x, out double[] hidden);
double[] dOut = new double[n];
for (int o = 0; o < n; o++)
{
double error = target[o] - output[o];
totalError += error * error;
dOut[o] = error * Slope(output[o]);
}
double[] dHid = new double[nHidden];
for (int h = 0; h < nHidden; h++)
{
double sum = 0;
for (int o = 0; o < n; o++) sum += dOut[o] * w2[o, h];
dHid[h] = sum * Slope(hidden[h]);
}
for (int o = 0; o < n; o++)
{
for (int h = 0; h < nHidden; h++) w2[o, h] += 0.5 * dOut[o] * hidden[h];
b2[o] += 0.5 * dOut[o];
}
for (int h = 0; h < nHidden; h++)
{
for (int i = 0; i < n; i++) w1[h, i] += 0.5 * dHid[h] * x[i];
b1[h] += 0.5 * dHid[h];
}
}
if (epoch % 1000 == 0) Console.WriteLine($"Epoch {epoch,-5} loss {totalError / inputs.Length:F4}");
}
// Ask it. Pick the word slot with the highest score. No lookup happens here.
string Ask(string q)
{
double[] output = Forward(Encode(q), out _);
int best = 0;
for (int o = 1; o < n; o++) if (output[o] > output[best]) best = o;
return vocab[best];
}
Console.WriteLine();
Console.WriteLine("questions it WAS taught:");
foreach (var f in facts)
Console.WriteLine($"{f.question} -> {Ask(f.question)}");
Console.WriteLine();
Console.WriteLine("questions it was NEVER taught:");
string[] unseen =
{
"who is the president of japan",
"who is the president of italy",
"what is the capital of usa",
};
foreach (var q in unseen)
Console.WriteLine($"{q} -> {Ask(q)}");
}
}What this is really showing you
In the last post I typed the weights. Here the model typed them for me, by training on examples. A real AI model does the very same thing over the whole web and a huge pile of books, with billions of weights instead of my handful, and once the training ends the weights are frozen. The loop underneath does not change. Guess, measure, nudge, repeat.
Add a sixth fact to the list and run it again. The model learns that one too, and you never touched the training code.
