{ "cells": [ { "cell_type": "markdown", "id": "77fb5ef6", "metadata": {}, "source": [ "# 🧠 从零开始写神经网络\n", "\n", "> 面向新手：只用 `numpy` 和 `matplotlib`，不调用任何深度学习框架，把一个能解 **XOR** 的多层感知机从零敲出来。\n", "\n", "## 🎯 学完这篇你会得到什么\n", "\n", "1. 知道「感知机」到底是什么 —— 一个会学习的二分类小函数\n", "2. 亲手用几十行代码写出一个能学 AND / OR 的感知机\n", "3. 看到感知机为什么会输给 **XOR**\n", "4. 亲手实现一个两层感知机（MLP），并手写反向传播\n", "5. 训练它把 XOR 这种「线性不可分」的问题解决\n", "\n", "## 🗺️ 路线图\n", "\n", "| 步骤 | 内容 | 你会写什么 |\n", "|---|---|---|\n", "| 1 | 感知机是什么 | 一句话定义 + 画图解释 |\n", "| 2 | 最小感知机 forward | 10 行代码算 `y = step(w·x + b)` |\n", "| 3 | 学习规则 | 用「错一次就改一点」的方式调权重 |\n", "| 4 | 训练 AND / OR | 画训练曲线 + 决策边界 |\n", "| 5 | XOR 翻车 | 演示感知机的根本缺陷 |\n", "| 6 | 多层感知机 | 加一个隐藏层 + sigmoid |\n", "| 7 | 反向传播 | 用链式法则手算梯度 |\n", "| 8 | 解决 XOR | 看到神经网络第一次「开窍」 |\n", "\n", "> 💡 建议：每个 cell 跑一下再往下看。改改数字、画画图，比光看有用得多。" ] }, { "cell_type": "markdown", "id": "29110726", "metadata": {}, "source": [ "## 🧩 1. 感知机（Perceptron）是什么？\n", "\n", "把它想成一个**会打分的迷你决策器**：\n", "\n", "```\n", "输入: x1, x2, ..., xn (特征)\n", " ↓\n", "权重: w1, w2, ..., wn (每个特征的重要性)\n", "偏置: b (门槛)\n", " ↓\n", "求和: z = w1·x1 + w2·x2 + ... + wn·xn + b\n", " ↓\n", "激活: y = step(z) (≥ 0 输出 1，否则 0)\n", "输出: 0 或 1 (二分类)\n", "```\n", "\n", "### 几何上它在干什么？\n", "\n", "- `z = w·x + b = 0` 在平面上是一条**直线**（高维里是一个超平面）\n", "- 这条直线把平面分成两半，一半预测 1，一半预测 0\n", "- 所以**单层感知机只能切一条直线**\n", "\n", "### 关键直觉\n", "\n", "> 感知机 = 一次线性打分 + 一次硬阈值。\n", "> 「学习」= 找到合适的 `w` 和 `b`，让那条直线把正负样本分对。\n", "\n", "下面就动手写。" ] }, { "cell_type": "markdown", "id": "ab210fec", "metadata": {}, "source": [ "### 🧪 2. 最小感知机：手算 AND\n", "\n", "下面是一个**写死**权重的感知机（`w=[1,1], b=-1.5`），它刚好能算 AND：\n", "\n", "- 只有 (1,1) 时 `1+1-1.5=0.5>0` → 输出 1\n", "- 其他三种 `z<0` → 输出 0" ] }, { "cell_type": "code", "execution_count": null, "id": "fc11072e", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "\n", "# ---------- 最小感知机：只有 forward ----------\n", "def step(z):\n", " \"\"\"阶跃激活函数：>=0 返 1，否则 0\"\"\"\n", " return (z >= 0).astype(int)\n", "\n", "def perceptron_forward(x, w, b):\n", " \"\"\"\n", " x: shape (n, ) 一个样本\n", " w: shape (n, ) 权重\n", " b: 标量 偏置\n", " 返回: 0 或 1\n", " \"\"\"\n", " z = np.dot(x, w) + b # 加权求和\n", " return step(z) # 过阶跃\n", "\n", "# ---------- 写死一组\"刚好能算 AND\"的权重 ----------\n", "w = np.array([1.0, 1.0])\n", "b = -1.5\n", "\n", "# AND 真值表\n", "X = np.array([[0,0],\n", " [0,1],\n", " [1,0],\n", " [1,1]])\n", "y_true = np.array([0, 0, 0, 1])\n", "\n", "# 跑一遍\n", "print(\"x1 x2 | z | y_pred\")\n", "print(\"-\" * 30)\n", "for xi, yi in zip(X, y_true):\n", " z = np.dot(xi, w) + b\n", " y_pred = perceptron_forward(xi, w, b)\n", " print(f\"{xi[0]} {xi[1]} | {z:+.2f} | {y_pred} (真值={yi})\")" ] }, { "cell_type": "markdown", "id": "15a8363e", "metadata": {}, "source": [ "### 📖 3. 感知机怎么「学」？\n", "\n", "刚才的 `w` 和 `b` 是我手动凑出来的。能不能让机器**自己找**？\n", "\n", "感知机学习规则（Rosenblatt, 1958）极其朴素：\n", "\n", "```\n", "对每个样本 (x, y_true):\n", " y_pred = step(w·x + b)\n", " err = y_true - y_pred # 错得越多，err 越大（-1, 0, 1）\n", " w += lr * err * x # 按 err 修正权重\n", " b += lr * err # 修正偏置\n", "```\n", "\n", "> 直觉：**预测低了就把权重往正方向推，预测高了就往负方向推。**\n", "> 这个规则只有在数据**线性可分**时才会收敛（Minsky & Papert 已经证明）。\n", "\n", "下面动手写一个会自己学的感知机。" ] }, { "cell_type": "markdown", "id": "581da114", "metadata": {}, "source": [ "### 💻 4. 一个会自己学的感知机\n", "\n", "把感知机包成一个类，让它能：\n", "- 接收一批数据\n", "- 反复跑 forward → 算错 → 改权重\n", "- 记录每轮错误数，画训练曲线" ] }, { "cell_type": "code", "execution_count": null, "id": "3cdc270f", "metadata": {}, "outputs": [], "source": [ "class Perceptron:\n", " \"\"\"\n", " 最小感知机：单层、二分类、阶跃激活、感知机学习规则\n", " \"\"\"\n", " def __init__(self, n_features, lr=0.1, seed=0):\n", " rng = np.random.default_rng(seed)\n", " # 随机初始化权重（小一点的随机数）\n", " self.w = rng.normal(loc=0.0, scale=0.1, size=n_features)\n", " self.b = 0.0\n", " self.lr = lr\n", " self.errors_history = [] # 记录每一轮的错分类数\n", "\n", " def forward(self, x):\n", " \"\"\"对一个样本做前向，返回 0/1\"\"\"\n", " z = np.dot(x, self.w) + self.b\n", " return int(z >= 0)\n", "\n", " def predict(self, X):\n", " \"\"\"对一批样本做预测\"\"\"\n", " z = X @ self.w + self.b\n", " return (z >= 0).astype(int)\n", "\n", " def fit(self, X, y, epochs=20, verbose=True):\n", " \"\"\"\n", " 训练\n", " X: (N, n_features)\n", " y: (N,) 值是 0/1\n", " \"\"\"\n", " for epoch in range(1, epochs + 1):\n", " errors = 0\n", " # 随机打乱顺序，训练更稳\n", " idx = np.random.permutation(len(X))\n", " for i in idx:\n", " xi, yi = X[i], y[i]\n", " y_pred = self.forward(xi)\n", " err = yi - y_pred # -1, 0, 1\n", " if err != 0:\n", " errors += 1\n", " self.w += self.lr * err * xi # 权重更新\n", " self.b += self.lr * err # 偏置更新\n", " self.errors_history.append(errors)\n", " if verbose:\n", " print(f\"epoch {epoch:2d} errors={errors} w={self.w} b={self.b:.3f}\")\n", " return self\n", "\n", "\n", "# ---------- 用它学 AND ----------\n", "X = np.array([[0,0],[0,1],[1,0],[1,1]])\n", "y_and = np.array([0, 0, 0, 1])\n", "\n", "ppn = Perceptron(n_features=2, lr=0.1, seed=42)\n", "ppn.fit(X, y_and, epochs=10)\n", "\n", "print(\"\\n学完之后预测：\")\n", "print(\"y_pred:\", ppn.predict(X))\n", "print(\"y_true:\", y_and)" ] }, { "cell_type": "markdown", "id": "c6520f22", "metadata": {}, "source": [ "### 🧪 4.5 砍掉一个参数会怎样？\n", "\n", "新手常问：「`w` 和 `b` 是不是都必需？只要一个不行吗？」\n", "\n", "直接做实验对比：\n", "1. **只有 `w`**：`z = w·x`，`b` 锁死为 0\n", "2. **只有 `b`**：`z = b`，`w` 锁死为 0\n", "3. **完整版**：`z = w·x + b`" ] }, { "cell_type": "code", "execution_count": null, "id": "0491147d", "metadata": {}, "outputs": [], "source": [ "def fit_variant(X, y, learn_w=True, learn_b=True, epochs=20, lr=0.1, seed=0):\n", " \"\"\"感知机的可配置版本：可以关掉 w 或 b 的学习\"\"\"\n", " rng = np.random.default_rng(seed)\n", " w = rng.normal(0, 0.1, 2) if learn_w else np.zeros(2)\n", " b = 0.0\n", " history = [] # 每轮的 (w, b, errors)\n", " for ep in range(epochs):\n", " errors = 0\n", " for xi, yi in zip(X, y):\n", " z = np.dot(xi, w) + b\n", " y_pred = int(z >= 0)\n", " err = yi - y_pred\n", " if err != 0: errors += 1\n", " if learn_w: w += lr * err * xi # 可关\n", " if learn_b: b += lr * err # 可关\n", " history.append((w.copy(), b, errors))\n", " return w, b, history\n", "\n", "# ---------- 实验 ----------\n", "X = np.array([[0,0],[0,1],[1,0],[1,1]])\n", "y_and = np.array([0, 0, 0, 1])\n", "\n", "def step_scalar(z): return int(z >= 0)\n", "\n", "print(\"=\" * 55)\n", "print(\"【只有 w】learn_w=True, learn_b=False\")\n", "print(\"=\" * 55)\n", "w, b, _ = fit_variant(X, y_and, learn_w=True, learn_b=False, epochs=20)\n", "preds = [step_scalar(np.dot(xi, w) + b) for xi in X]\n", "print(f\"最终: w={w.round(3)}, b={b:.3f}, 预测={preds}, 真值={y_and.tolist()}\")\n", "print(\"→ (0,0) 永远被预测成 1（因为 z=0 触发 step）\\n\")\n", "\n", "print(\"=\" * 55)\n", "print(\"【只有 b】learn_w=False, learn_b=True\")\n", "print(\"=\" * 55)\n", "w, b, _ = fit_variant(X, y_and, learn_w=False, learn_b=True, epochs=20)\n", "preds = [step_scalar(np.dot(xi, w) + b) for xi in X]\n", "print(f\"最终: w={w.round(3)}, b={b:.3f}, 预测={preds}, 真值={y_and.tolist()}\")\n", "print(\"→ 4 个样本预测都一样（常数分类器），AND 永远至少错 1 个\\n\")\n", "\n", "print(\"=\" * 55)\n", "print(\"【w + b 完整】learn_w=True, learn_b=True\")\n", "print(\"=\" * 55)\n", "w, b, _ = fit_variant(X, y_and, learn_w=True, learn_b=True, epochs=20)\n", "preds = [step_scalar(np.dot(xi, w) + b) for xi in X]\n", "print(f\"最终: w={w.round(3)}, b={b:.3f}, 预测={preds}, 真值={y_and.tolist()}\")\n", "print(\"→ ✅ 完美解开 AND\")" ] }, { "cell_type": "markdown", "id": "4b1933eb", "metadata": {}, "source": [ "### 💡 结论\n", "\n", "- **`w` 的作用**：控制决策边界的**方向**（直线的斜率）\n", "- **`b` 的作用**：控制决策边界的**位置**（直线离原点多远）\n", "- 没有 `b` → 直线**过原点**，卡在原点上动不了\n", "- 没有 `w` → 没有方向，所有样本共用同一条\"等高线\"\n", "- 两者配合，直线才能在平面上**任意摆**\n", "\n", "> 🤓 一个记忆口诀：\n", "> **「`w` 决定直线的角度，`b` 决定直线的位置」**\n", "> 角度不对切错，位置不对也切错。\n", "\n", "现实里的神经网络（包括 MLP、CNN、Transformer）**每一层都同时有 `W` 和 `b`**。砍掉 `b` 等于强迫所有\"分类直线\"都穿过原点——表达能力会大打折扣。" ] }, { "cell_type": "markdown", "id": "12a389bd", "metadata": {}, "source": [ "### 📊 5. 训练曲线 + 决策边界\n", "\n", "两个图能让你一眼看明白感知机在干什么：\n", "- **左图**：每轮错分类数随 epoch 下降 → 学习是否收敛\n", "- **右图**：把 `w·x + b = 0` 这条**直线**画在二维平面上，红色 = 预测 1，蓝色 = 预测 0" ] }, { "cell_type": "code", "execution_count": null, "id": "9d4d39ec", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "import matplotlib\n", "import matplotlib.font_manager as fm\n", "# 找出系统里真实存在的中文字体\n", "CJK_CANDIDATES = [\"PingFang SC\", \"Heiti SC\", \"Hiragino Sans GB\",\n", " \"Songti SC\", \"STSong\", \"Arial Unicode MS\",\n", " \"Microsoft YaHei\", \"SimHei\", \"WenQuanYi Zen Hei\"]\n", "available = {f.name for f in fm.fontManager.ttflist}\n", "for cand in CJK_CANDIDATES:\n", " if cand in available:\n", " matplotlib.rcParams[\"font.sans-serif\"] = [cand, \"DejaVu Sans\"]\n", " break\n", "matplotlib.rcParams[\"axes.unicode_minus\"] = False\n", "\n", "\n", "def plot_training(model, X, y, title=\"Perceptron\"):\n", " \"\"\"画两个图：左 = 训练误差曲线，右 = 决策边界 + 样本点\"\"\"\n", " fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4.5))\n", "\n", " # ---------- 左图：每轮的错分类数 ----------\n", " ax1.plot(range(1, len(model.errors_history) + 1),\n", " model.errors_history, 'b-o', markersize=4)\n", " ax1.set_xlabel('Epoch（训练轮次）')\n", " ax1.set_ylabel('错分类样本数')\n", " ax1.set_title(f'{title}：训练误差曲线')\n", " ax1.set_ylim(-0.3, max(model.errors_history + [1]) + 0.5)\n", " ax1.grid(alpha=0.3)\n", "\n", " # ---------- 右图：决策边界 ----------\n", " x_min, x_max = -0.5, 1.5\n", " y_min, y_max = -0.5, 1.5\n", " xx1, xx2 = np.meshgrid(np.linspace(x_min, x_max, 200),\n", " np.linspace(y_min, y_max, 200))\n", " grid = np.c_[xx1.ravel(), xx2.ravel()]\n", " Z = model.predict(grid).reshape(xx1.shape)\n", "\n", " # 背景：负类蓝、正类红\n", " ax2.contourf(xx1, xx2, Z, levels=[-0.1, 0.5, 1.1],\n", " colors=['#cfe2ff', '#ffd6d6'], alpha=0.6)\n", "\n", " # 画决策直线 w·x + b = 0\n", " w, b = model.w, model.b\n", " if abs(w[1]) > 1e-6:\n", " x_line = np.array([x_min, x_max])\n", " y_line = -(w[0] * x_line + b) / w[1]\n", " ax2.plot(x_line, y_line, 'k-', linewidth=2, label='决策边界 w·x+b=0')\n", " elif abs(w[0]) > 1e-6:\n", " # 退化：w[1]≈0，直线是 x1 = -b/w[0] 的垂直线\n", " x_line = np.full(2, -b / w[0])\n", " ax2.plot(x_line, [y_min, y_max], 'k-', linewidth=2, label='决策边界')\n", "\n", " # 样本点\n", " for xi, yi in zip(X, y):\n", " color = 'red' if yi == 1 else 'blue'\n", " ax2.scatter(xi[0], xi[1], c=color, s=300,\n", " edgecolors='k', zorder=3)\n", "\n", " ax2.set_xlim(x_min, x_max)\n", " ax2.set_ylim(y_min, y_max)\n", " ax2.set_xlabel('x1')\n", " ax2.set_ylabel('x2')\n", " ax2.set_title(f'{title}：决策边界 w={w.round(2)}, b={b:.2f}')\n", " ax2.legend(loc='upper left')\n", " ax2.grid(alpha=0.3)\n", "\n", " plt.tight_layout()\n", " plt.show()\n", "\n", "\n", "# ---------- 画 AND 训练结果 ----------\n", "print(\"训练好的感知机参数：\")\n", "print(f\" w = {ppn.w}\")\n", "print(f\" b = {ppn.b:.3f}\")\n", "print(f\" 预测 = {ppn.predict(X)}\")\n", "print(f\" 真值 = {y_and}\")\n", "print()\n", "plot_training(ppn, X, y_and, title=\"AND\")" ] }, { "cell_type": "markdown", "id": "cb313ea0", "metadata": {}, "source": [ "### ❌ 6. 感知机翻车现场：XOR\n", "\n", "XOR 真值表：\n", "\n", "| x1 | x2 | y |\n", "|---|---|---|\n", "| 0 | 0 | **0** |\n", "| 0 | 1 | 1 |\n", "| 1 | 0 | 1 |\n", "| 1 | 1 | **0** |\n", "\n", "红点（输出 1）在 (0,1) 和 (1,0)，蓝点（输出 0）在 (0,0) 和 (1,1)。\n", "这四点**没有一条直线**能把红色和蓝色分开。**这就是\"线性不可分\"**。" ] }, { "cell_type": "code", "execution_count": null, "id": "17fefa29", "metadata": {}, "outputs": [], "source": [ "y_xor = np.array([0, 1, 1, 0])\n", "\n", "# 训练 50 轮看看\n", "ppn_xor = Perceptron(n_features=2, lr=0.1, seed=0)\n", "ppn_xor.fit(X, y_xor, epochs=50, verbose=False)\n", "plot_training(ppn_xor, X, y_xor, title=\"XOR (单层感知机)\")\n", "\n", "print(\"最终预测:\", ppn_xor.predict(X))\n", "print(\"真实标签:\", y_xor)\n", "print(\"→ 错误！感知机在 XOR 上根本不收敛（错误数在 1~2 之间反复横跳）。\")" ] }, { "cell_type": "markdown", "id": "7926cabc", "metadata": {}, "source": [ "### 🏗️ 7. 多层感知机（MLP）：用「两层直线」拼出曲线\n", "\n", "核心想法：\n", "1. **第一层**画几条不同的直线，把空间切碎\n", "2. 用 **sigmoid** 把直线变成柔和的概率（0~1）\n", "3. **第二层**把这些\"分碎的空间\"再拼回去\n", "\n", "网络结构：\n", "\n", "```\n", "x(2) → [隐藏层 h 个 sigmoid 神经元] → [输出层 1 个 sigmoid 神经元] → ŷ\n", "```\n", "\n", "> 关键替换：\n", "> - 阶跃函数 → **sigmoid**（可微，才能用梯度下降）\n", "> - 单层 → 两层" ] }, { "cell_type": "markdown", "id": "94012e92", "metadata": {}, "source": [ "### 🔁 8. 反向传播：链式法则 + 一行一行倒着求梯度\n", "\n", "设网络为：\n", "- 隐藏层：`z1 = X @ W1 + b1`, `a1 = sigmoid(z1)`\n", "- 输出层：`z2 = a1 @ W2 + b2`, `ŷ = sigmoid(z2)`\n", "- 损失（二元交叉熵）：`L = -[ y·log(ŷ) + (1-y)·log(1-ŷ) ]`\n", "\n", "链式法则（**对每个样本**算）：\n", "\n", "```\n", "dL/dz2 = ŷ - y # 输出层误差\n", "dL/dW2 = a1.T @ dL/dz2 / N\n", "dL/db2 = mean(dL/dz2)\n", "\n", "dL/da1 = dL/dz2 @ W2.T\n", "dL/dz1 = dL/da1 * a1 * (1 - a1) # sigmoid 的导数\n", "dL/dW1 = X.T @ dL/dz1 / N\n", "dL/db1 = mean(dL/dz1)\n", "```\n", "\n", "> 看起来很多公式，但**本质就是「输出错了多少 → 每个权重分摊多少责任」**。\n", "\n", "下面写代码。" ] }, { "cell_type": "markdown", "id": "5626029e", "metadata": {}, "source": [ "### 💻 9. 手写一个两层 MLP\n", "\n", "只用了 ~40 行核心代码：前向、反向、训练循环全部显式写出来。" ] }, { "cell_type": "code", "execution_count": null, "id": "a6927cd1", "metadata": {}, "outputs": [], "source": [ "def sigmoid(z):\n", " # 为数值稳定用 np.clip\n", " return 1.0 / (1.0 + np.exp(-np.clip(z, -500, 500)))\n", "\n", "def dsigmoid(a):\n", " \"\"\"sigmoid 对 a 本身求导（a 已经是 sigmoid(z)）\"\"\"\n", " return a * (1.0 - a)\n", "\n", "\n", "class MLP:\n", " \"\"\"\n", " 两层感知机：输入(2) -> 隐藏(h) -> 输出(1)\n", " 激活: sigmoid + sigmoid\n", " 损失: 二元交叉熵\n", " \"\"\"\n", " def __init__(self, n_in, n_hidden=4, lr=0.5, seed=0):\n", " rng = np.random.default_rng(seed)\n", " # He 初始化 改写：sigmoid 用 Xavier 更稳\n", " self.W1 = rng.normal(0, 1/np.sqrt(n_in), size=(n_in, n_hidden))\n", " self.b1 = np.zeros(n_hidden)\n", " self.W2 = rng.normal(0, 1/np.sqrt(n_hidden), size=(n_hidden, 1))\n", " self.b2 = np.zeros(1)\n", " self.lr = lr\n", " self.loss_history = []\n", "\n", " def forward(self, X):\n", " self.z1 = X @ self.W1 + self.b1 # (N, h)\n", " self.a1 = sigmoid(self.z1) # (N, h)\n", " self.z2 = self.a1 @ self.W2 + self.b2 # (N, 1)\n", " self.a2 = sigmoid(self.z2).ravel() # (N,)\n", " return self.a2\n", "\n", " def backward(self, X, y, y_pred):\n", " N = len(X)\n", " y_pred = y_pred.reshape(-1, 1) # (N, 1)\n", "\n", " # 输出层\n", " dz2 = (y_pred - y.reshape(-1, 1)) # (N, 1) ← 交叉熵+sigmoid 的简化\n", " dW2 = (self.a1.T @ dz2) / N # (h, 1)\n", " db2 = dz2.mean(axis=0) # (1,)\n", "\n", " # 隐藏层\n", " da1 = dz2 @ self.W2.T # (N, h)\n", " dz1 = da1 * dsigmoid(self.a1) # (N, h)\n", " dW1 = (X.T @ dz1) / N # (2, h)\n", " db1 = dz1.mean(axis=0) # (h,)\n", "\n", " # 梯度下降\n", " self.W1 -= self.lr * dW1\n", " self.b1 -= self.lr * db1\n", " self.W2 -= self.lr * dW2\n", " self.b2 -= self.lr * db2\n", "\n", " def fit(self, X, y, epochs=5000, verbose_every=1000):\n", " for ep in range(1, epochs + 1):\n", " y_pred = self.forward(X)\n", " # 交叉熵损失（加 1e-8 防止 log(0)）\n", " loss = -np.mean(y * np.log(y_pred + 1e-8) +\n", " (1 - y) * np.log(1 - y_pred + 1e-8))\n", " self.loss_history.append(loss)\n", " self.backward(X, y, y_pred)\n", " if verbose_every and (ep % verbose_every == 0 or ep == 1):\n", " acc = (y_pred.round() == y).mean()\n", " print(f\"epoch {ep:5d} loss={loss:.4f} acc={acc:.2f}\")\n", " return self\n", "\n", " def predict(self, X, threshold=0.5):\n", " return (self.forward(X) >= threshold).astype(int)\n", "\n", "\n", "# ---------- 在 XOR 上训练 ----------\n", "mlp = MLP(n_in=2, n_hidden=4, lr=1.0, seed=1)\n", "mlp.fit(X, y_xor, epochs=10000, verbose_every=2000)\n", "\n", "print(\"\\n最终预测概率:\", mlp.forward(X).round(3))\n", "print(\"最终预测类别:\", mlp.predict(X))\n", "print(\"真实标签 :\", y_xor)" ] }, { "cell_type": "markdown", "id": "f7ff3b1f", "metadata": {}, "source": [ "### 🎨 10. 看 MLP 怎么把 XOR 分开\n", "\n", "不再是一条直线，而是 MLP 算出的「概率等高线」把红色和蓝色区域圈出来。" ] }, { "cell_type": "code", "execution_count": null, "id": "76879423", "metadata": {}, "outputs": [], "source": [ "def plot_decision_region(model, X, y, title=\"MLP\", is_mlp=True):\n", " fig, ax = plt.subplots(figsize=(5.5, 5))\n", "\n", " # 建一个网格\n", " x1 = np.linspace(-0.5, 1.5, 200)\n", " x2 = np.linspace(-0.5, 1.5, 200)\n", " XX1, XX2 = np.meshgrid(x1, x2)\n", " grid = np.c_[XX1.ravel(), XX2.ravel()]\n", "\n", " if is_mlp:\n", " Z = model.forward(grid).reshape(XX1.shape)\n", " else:\n", " Z = model.predict(grid).reshape(XX1.shape)\n", "\n", " # 画背景色：红色=正类区，蓝色=负类区\n", " ax.contourf(XX1, XX2, Z, levels=[-0.1, 0.5, 1.1],\n", " colors=['#cfe2ff', '#ffd6d6'], alpha=0.6)\n", " # 等高线\n", " ax.contour(XX1, XX2, Z, levels=[0.5], colors='k', linewidths=2)\n", "\n", " # 样本点\n", " for xi, yi in zip(X, y):\n", " color = 'red' if yi == 1 else 'blue'\n", " ax.scatter(xi[0], xi[1], c=color, s=300, edgecolors='k', zorder=3)\n", "\n", " ax.set_xlim(-0.5, 1.5)\n", " ax.set_ylim(-0.5, 1.5)\n", " ax.set_xlabel(\"x1\"); ax.set_ylabel(\"x2\")\n", " ax.set_title(f\"{title}: 决策区域\")\n", " ax.grid(alpha=0.3)\n", " plt.show()\n", "\n", "# 单层感知机（直线分不开）\n", "plot_decision_region(ppn_xor, X, y_xor, title=\"XOR (单层感知机)\", is_mlp=False)\n", "\n", "# 两层 MLP（曲线分开了）\n", "plot_decision_region(mlp, X, y_xor, title=\"XOR (两层 MLP)\", is_mlp=True)" ] }, { "cell_type": "markdown", "id": "91a6c33a", "metadata": {}, "source": [ "### 🤔 10.5 还有别的思路解 XOR 吗？\n", "\n", "新手常问：\"`MLP` 凭什么能解 `XOR`？我能不能想别的办法？\"\n", "\n", "下面把**五种常见思路**都实现一遍对比：\n", "\n", "| 思路 | 模型形式 | 关键点 |\n", "|---|---|---|\n", "| A. 死磕单层直线 | `step(w·x + b)` | 你已经学过，**失败** |\n", "| B. 手工加非线性特征 | `step(w·[x1,x2,x1·x2] + b)` | **能**解，但要人会挑特征 |\n", "| C. 用 RBF / 高斯基 | `step(w·[exp(-‖x-μ‖²)] + b)` | **能**解，但中心 `μ` 难选 |\n", "| D. 单层 + 梯度下降 | 把 step 换成 sigmoid | 训练会**卡住**（梯度消失） |\n", "| E. 两层 MLP（已知）| sigmoid + sigmoid | ✅ |\n", "\n", "观察对比，看哪种思路\"能跑通、还能泛化\"。" ] }, { "cell_type": "code", "execution_count": null, "id": "0440a91f", "metadata": {}, "outputs": [], "source": [ "# =========================================================================\n", "# 五种思路对比解 XOR\n", "# =========================================================================\n", "y_xor = np.array([0, 1, 1, 0])\n", "\n", "def report(name, y_pred, y_true):\n", " y_pred = np.asarray(y_pred).astype(int).ravel()\n", " acc = (y_pred == y_true).mean()\n", " mark = \"✅\" if acc == 1.0 else \"❌\"\n", " print(f\"{mark} {name:30s} 预测={y_pred.tolist()} 正确率={acc:.0%}\")\n", "\n", "# ---------- A. 单层感知机（你已经学过）----------\n", "ppn_a = Perceptron(n_features=2, lr=0.1, seed=0)\n", "ppn_a.fit(X, y_xor, epochs=200, verbose=False)\n", "report(\"A. 单层感知机 step(w·x+b)\", ppn_a.predict(X), y_xor)\n", "\n", "\n", "# ---------- B. 手工加非线性特征：x1·x2 ----------\n", "def featurize_poly(x):\n", " \"\"\"φ(x1, x2) = (x1, x2, x1·x2, x1², x2²)\"\"\"\n", " x1, x2 = x[0], x[1]\n", " return np.array([x1, x2, x1*x2, x1**2, x2**2])\n", "\n", "X_poly = np.array([featurize_poly(xi) for xi in X])\n", "ppn_b = Perceptron(n_features=5, lr=0.1, seed=42)\n", "ppn_b.fit(X_poly, y_xor, epochs=200, verbose=False)\n", "report(\"B. 感知机 + 手工多项式特征\", ppn_b.predict(X_poly), y_xor)\n", "\n", "\n", "# ---------- C. RBF 特征：4 个高斯中心 ----------\n", "centers = np.array([[0, 0], [0, 1], [1, 0], [1, 1]]) # 把中心放在 4 个样本上\n", "sigma = 0.6\n", "\n", "def featurize_rbf(x):\n", " \"\"\"φ(x) = [exp(-‖x-μ_i‖²/2σ²)] for i=1..4\"\"\"\n", " d2 = np.sum((centers - x) ** 2, axis=1)\n", " return np.exp(-d2 / (2 * sigma**2))\n", "\n", "X_rbf = np.array([featurize_rbf(xi) for xi in X])\n", "ppn_c = Perceptron(n_features=4, lr=0.1, seed=42)\n", "ppn_c.fit(X_rbf, y_xor, epochs=200, verbose=False)\n", "report(\"C. 感知机 + RBF（高斯）特征\", ppn_c.predict(X_rbf), y_xor)\n", "\n", "\n", "# ---------- D. 单层 + sigmoid + 梯度下降（你猜会怎样？）----------\n", "class LinearSigmoid:\n", " \"\"\"单层 sigmoid 模型，梯度下降训练——演示「线性 → 非线性激活」但层数不够\"\"\"\n", " def __init__(self, n_in, lr=0.5, seed=0):\n", " rng = np.random.default_rng(seed)\n", " self.W = rng.normal(0, 0.5, size=(n_in, 1))\n", " self.b = np.zeros(1)\n", " self.lr = lr\n", " self.loss_history = []\n", "\n", " def forward(self, X):\n", " z = X @ self.W + self.b\n", " return sigmoid(z).ravel()\n", "\n", " def fit(self, X, y, epochs=3000):\n", " for _ in range(epochs):\n", " yp = self.forward(X).reshape(-1, 1)\n", " err = yp - y.reshape(-1, 1)\n", " dW = (X.T @ err) / len(X)\n", " db = err.mean(axis=0)\n", " self.W -= self.lr * dW\n", " self.b -= self.lr * db\n", " loss = -np.mean(y * np.log(yp.ravel() + 1e-8) +\n", " (1 - y) * np.log(1 - yp.ravel() + 1e-8))\n", " self.loss_history.append(loss)\n", " return self\n", "\n", " def predict(self, X, t=0.5):\n", " return (self.forward(X) >= t).astype(int)\n", "\n", "m_d = LinearSigmoid(n_in=2, lr=0.5, seed=0)\n", "m_d.fit(X, y_xor, epochs=3000)\n", "report(\"D. 单层 sigmoid + 梯度下降\", m_d.predict(X), y_xor)\n", "print(f\" ↳ 最终 loss={m_d.loss_history[-1]:.3f}（loss 卡在 0.69 ≈ ln2，永远降不下去）\")\n", "\n", "\n", "# ---------- E. MLP（你已经训练过）----------\n", "report(\"E. 两层 MLP（sigmoid+sigmoid）\", mlp.predict(X), y_xor)\n", "print(f\" ↳ 最终 loss={mlp.loss_history[-1]:.4f}\")\n", "\n", "\n", "# =========================================================================\n", "# 把 E 种决策区域画出来对比\n", "# =========================================================================\n", "fig, axes = plt.subplots(1, 5, figsize=(22, 4.5))\n", "titles = [\n", " (\"A. 单层直线\", lambda g: ppn_a.predict(g), False),\n", " (\"B. + 多项式特征\", lambda g: ppn_b.predict(np.array([featurize_poly(x) for x in g])), False),\n", " (\"C. + RBF 特征\", lambda g: ppn_c.predict(np.array([featurize_rbf(x) for x in g])), False),\n", " (\"D. 单层 sigmoid\", lambda g: m_d.predict(g), True),\n", " (\"E. 两层 MLP\", lambda g: mlp.predict(g), True),\n", "]\n", "\n", "x1g, x2g = np.meshgrid(np.linspace(-0.5, 1.5, 200),\n", " np.linspace(-0.5, 1.5, 200))\n", "grid = np.c_[x1g.ravel(), x2g.ravel()]\n", "\n", "for ax, (title, fn, use_prob) in zip(axes, titles):\n", " if use_prob:\n", " Z = fn(grid).reshape(x1g.shape)\n", " else:\n", " Z = fn(grid).reshape(x1g.shape)\n", " ax.contourf(x1g, x2g, Z, levels=[-0.1, 0.5, 1.1],\n", " colors=['#cfe2ff', '#ffd6d6'], alpha=0.6)\n", " if use_prob:\n", " ax.contour(x1g, x2g, Z, levels=[0.5], colors='k', linewidths=2)\n", " for xi, yi in zip(X, y_xor):\n", " c = 'red' if yi == 1 else 'blue'\n", " ax.scatter(xi[0], xi[1], c=c, s=200, edgecolors='k', zorder=3)\n", " ax.set_xlim(-0.5, 1.5); ax.set_ylim(-0.5, 1.5)\n", " ax.set_title(title, fontsize=11)\n", " ax.set_xticks([]); ax.set_yticks([])\n", "\n", "plt.suptitle(\"五种思路解 XOR：只有 A 失败；B/C/E 解开；D 看似解了但 loss 卡死\", fontsize=13)\n", "plt.tight_layout()\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "395c0b4e", "metadata": {}, "source": [ "### 📝 11. 一图回顾：感知机 vs MLP\n", "\n", "| 维度 | 单层感知机 | 两层 MLP |\n", "|---|---|---|\n", "| 决策边界 | 一条**直线** | 多条直线拼出的**曲线** |\n", "| 激活函数 | step（不可微） | sigmoid（可微） |\n", "| 学习规则 | 感知机规则（错就改） | 梯度下降 + 反向传播 |\n", "| 能解 | AND / OR / 任何**线性可分**问题 | **XOR** 等非线性问题 |\n", "| 隐藏层 | 0 | 1+ |\n", "\n", "### 🎓 你亲手实现的东西\n", "\n", "- ✅ `Perceptron` 类：前向 + 感知机学习规则\n", "- ✅ `MLP` 类：两层网络 + sigmoid + **手写反向传播**\n", "- ✅ 训练可视化：误差曲线、决策边界、决策区域\n", "\n", "### 🚀 下一步可以学什么？\n", "\n", "1. **换激活函数**：把 sigmoid 换成 ReLU（梯度不消失，训练更深网络）\n", "2. **加更多层**：3 层、4 层... 但要小心梯度消失 / 爆炸\n", "3. **换损失函数**：多分类用 softmax + 交叉熵\n", "4. **加正则化**：Dropout、L2，防止过拟合\n", "5. **用 mini-batch / SGD**：不再逐样本更新\n", "6. **试真实数据**：MNIST 手写数字分类\n", "\n", "> 之后可以读 PyTorch / TensorFlow 源码——你会发现「卷积」「注意力」「归一化」全都是这套前向+反向+梯度下降的扩展。\n", "\n", "🌟 **恭喜你完成了第一个「从零开始的神经网络」！** 改一改隐藏神经元数量、学习率、跑几遍，看 MLP 在 XOR 上是不是也能翻车。" ] }, { "cell_type": "markdown", "id": "af35417b", "metadata": {}, "source": [ "### 🧮 10.7 解 XOR 最简单的模型是什么？\n", "\n", "**结论先放：上面所有方法都能解，简单的反而更好。**\n", "\n", "按\"由简到繁\"排序：\n", "\n", "| # | 方法 | 公式 | 训练？ |\n", "|---|---|---|---|\n", "| 1 | 查表 | 4 行字典 | 不需要 |\n", "| 2 | 多项式 | `x1 + x2 - 2·x1·x2` | 不需要 |\n", "| 3 | 算术化 | `(x1+x2) mod 2` | 不需要 |\n", "| 4 | 两条直线 AND | `step(x1+x2-0.5)·step(1.5-x1-x2)` | 不需要 |\n", "| 5 | 傅里叶级数 | `0.5 - (2/π²)·sin(πx1)·sin(πx2) - ...` | 不需要 |\n", "| 6 | 核感知机 | `step(1·x1 + 1·x2 - 2·x1·x2)` | 3 个权重 |\n", "| 7 | 单隐藏层 MLP | (你学的) | 几十个权重 |\n", "\n", "下面把方法 1-5 全部实现并跑一遍对比。" ] }, { "cell_type": "code", "execution_count": null, "id": "c52317a9", "metadata": {}, "outputs": [], "source": [ "# =========================================================================\n", "# 五种\"非 MLP\"的纯数学解法解 XOR\n", "# =========================================================================\n", "def check(name, pred_fn, test_X=None):\n", " \"\"\"对 4 个 XOR 真值点评估\"\"\"\n", " if test_X is None:\n", " test_X = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=int)\n", " pred = np.array([pred_fn(*x) for x in test_X])\n", " acc = (pred == np.array([0,1,1,0])).mean()\n", " mark = \"✅\" if acc == 1.0 else \"❌\"\n", " print(f\"{mark} {name:35s} 预测={pred.tolist()}\")\n", " return pred_fn\n", "\n", "print(\"XOR 真值表：y =\", y_xor.tolist())\n", "print(\"-\" * 60)\n", "\n", "# 1. 查表法\n", "xor_table = {(0,0):0, (0,1):1, (1,0):1, (1,1):0}\n", "def m_lookup(x1, x2): return xor_table[(x1, x2)]\n", "check(\"1. 查表 dict\", m_lookup)\n", "\n", "# 2. 多项式公式: y = x1 + x2 - 2·x1·x2 (把 0 视为负，>0 视为正)\n", "def m_poly(x1, x2): return int(x1 + x2 - 2*x1*x2 > 0)\n", "check(\"2. 多项式 (x1+x2-2·x1·x2)>0\", m_poly)\n", "\n", "# 3. 算术化: (x1+x2) mod 2\n", "def m_arith(x1, x2): return (x1 + x2) % 2\n", "check(\"3. 算术 (x1+x2) mod 2\", m_arith)\n", "\n", "# 4. 两条直线 + AND: step(x1+x2-0.5) · step(1.5-x1-x2)\n", "def m_two_lines(x1, x2):\n", " a = int(x1 + x2 - 0.5 >= 0)\n", " b = int(1.5 - x1 - x2 >= 0)\n", " return a * b\n", "check(\"4. 两条直线 AND\", m_two_lines)\n", "\n", "# 5. 傅里叶级数（Walsh-Hadamard 展开，把 XOR 分解为\"频率\"）\n", "# 把 x ∈ {0,1} 编码成 a = (-1)^x ∈ {-1, 1}\n", "# 核心恒等式: XOR(x1,x2) = (1 - a·b) / 2 其中 a = (-1)^x1, b = (-1)^x2\n", "# —— 傅里叶级数里只用 1 个频率项 (-1)^(x1+x2)，就足够表达 XOR\n", "def m_fourier(x1, x2):\n", " a = 1 - 2*x1 # (-1)^x1\n", " b = 1 - 2*x2 # (-1)^x2\n", " s = (1 - a * b) / 2.0\n", " return int(s > 0.5)\n", "check(\"5. 傅里叶 (Walsh-Hadamard)\", m_fourier)\n", "\n", "# 6. 核感知机\n", "X_xor = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=int)\n", "X_poly = np.array([[x1, x2, x1*x2] for (x1, x2) in X_xor])\n", "ppn_k = Perceptron(n_features=3, lr=0.1, seed=7)\n", "ppn_k.fit(X_poly, np.array([0,1,1,0]), epochs=50, verbose=False)\n", "def m_kernel(x1, x2):\n", " z = 1*x1 + 1*x2 - 2*(x1*x2) # 解析解: 把 0 当负\n", " return int(z > 0)\n", "check(\"6. 核感知机 φ=(x1,x2,x1·x2)\", m_kernel)\n", "\n", "\n", "# =========================================================================\n", "# 把 6 种解法画在一张图上对比\n", "# =========================================================================\n", "fig, axes = plt.subplots(2, 3, figsize=(15, 9))\n", "x1g, x2g = np.meshgrid(np.linspace(-0.5, 1.5, 250), np.linspace(-0.5, 1.5, 250))\n", "grid = np.c_[x1g.ravel(), x2g.ravel()]\n", "# 离散点用本地 X_xor 重新定义，避免被前面 10.6 节的 X 覆盖\n", "X_xor_pts = X_xor\n", "y_xor_pts = np.array([0,1,1,0])\n", "\n", "methods = [\n", " (\"1. 查表（仅 4 个点）\", \"lookup\"),\n", " (\"2. 多项式 (x1+x2-2·x1·x2)>0\", m_poly),\n", " (\"3. 算术 (x1+x2) mod 2\", m_arith),\n", " (\"4. 两条直线 AND\", m_two_lines),\n", " (\"5. 傅里叶 (Walsh-Hadamard)\", m_fourier),\n", " (\"6. 核感知机 φ=(x1,x2,x1·x2)\", m_kernel),\n", "]\n", "\n", "for ax, (name, fn) in zip(axes.ravel(), methods):\n", " if fn == \"lookup\":\n", " # 查表无法画连续区域，只画散点\n", " for xi, yi in zip(X_xor_pts, y_xor_pts):\n", " c = 'red' if yi == 1 else 'blue'\n", " ax.scatter(xi[0], xi[1], c=c, s=300, edgecolors='k', zorder=3)\n", " else:\n", " Z = np.array([fn(x1, x2) for (x1, x2) in grid]).reshape(x1g.shape)\n", " ax.contourf(x1g, x2g, Z, levels=[-0.1, 0.5, 1.1],\n", " colors=['#cfe2ff', '#ffd6d6'], alpha=0.5)\n", " ax.contour(x1g, x2g, Z, levels=[0.5], colors='k', linewidths=1.5)\n", " for xi, yi in zip(X_xor_pts, y_xor_pts):\n", " c = 'red' if yi == 1 else 'blue'\n", " ax.scatter(xi[0], xi[1], c=c, s=300, edgecolors='k', zorder=3)\n", " ax.set_xlim(-0.5, 1.5); ax.set_ylim(-0.5, 1.5)\n", " ax.set_title(name, fontsize=12)\n", " ax.set_xticks([]); ax.set_yticks([])\n", "\n", "plt.suptitle(\"六种非 MLP 解法：全部能解 XOR，公式最简单反而最干净\",\n", " fontsize=14, fontweight='bold')\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "# =========================================================================\n", "# 公式 vs 模型对比表\n", "# =========================================================================\n", "print(\"\\n\" + \"=\"*72)\n", "print(\"「公式」和「神经网络」的关系：\")\n", "print(\"=\"*72)\n", "print(\"\"\"\n", "┌──────────────────────┬─────────────────────────────────────┐\n", "│ 纯数学公式 │ 神经网络 │\n", "├──────────────────────┼─────────────────────────────────────┤\n", "│ x1 ⊕ x2 │ 通用拟合器 │\n", "│ = x1 + x2 - 2·x1·x2 │ 通过权重学到这个规律 │\n", "│ = step(... ≥ 0) │ 用 sigmoid 替代 step │\n", "│ 3 个参数就够 │ 几十/几百个参数（多了也学得到） │\n", "│ 形式已知 │ 形式未知，从数据反推 │\n", "│ 训练 = 0 秒 │ 训练 = 几秒到几小时 │\n", "│ 复杂数据写不出公式 │ 数据多就能学 │\n", "└──────────────────────┴─────────────────────────────────────┘\n", "\n", "👉 真正的\"难题\"是：问题复杂到写不出公式时——\n", " 神经网络是「让机器自己找公式」的工具。\n", " XOR 太简单了，公式直接写就行。\n", "\"\"\")" ] }, { "cell_type": "markdown", "id": "b34d8ac8", "metadata": {}, "source": [ "### 🎯 10.6 MLP 到底能解什么？不能解什么？\n", "\n", "新手常以为 \"MLP 既然能解 XOR，那它就万能了\" —— **事实远非如此**。\n", "下面用 5 个经典 2D 数据集做对比，让你**亲眼看到** MLP 的能力边界。\n", "\n", "| 数据集 | 形状 | MLP 表现 |\n", "|---|---|---|\n", "| 月牙 (two moons) | 两个月牙交错 | ✅ |\n", "| 同心圆 (circles) | 内外两个圈 | ✅ |\n", "| 高斯斑点 (blobs) | 两团云 | ✅（线性就够）|\n", "| 螺旋 (spiral) | 两条旋臂纠缠 | ⚠️ 隐藏层小时失败 |\n", "| **异或 + 大量随机噪声** | 4 个点 + 几百个噪声 | ❌ 噪声淹没了规律 |" ] }, { "cell_type": "code", "execution_count": null, "id": "fe244abc", "metadata": {}, "outputs": [], "source": [ "# =========================================================================\n", "# 5 个数据集 × 两种网络 = 直观感受 MLP 的能力边界\n", "# =========================================================================\n", "from sklearn.datasets import make_moons, make_circles, make_classification, make_blobs\n", "\n", "def make_spiral(n=200, noise=0.3):\n", " \"\"\"两条旋臂\"\"\"\n", " rng = np.random.default_rng(0)\n", " theta = np.sqrt(rng.random(n)) * 2 * np.pi\n", " r = 2 * theta + np.pi\n", " X1 = np.c_[r * np.cos(theta), r * np.sin(theta)]\n", " X2 = np.c_[r * np.cos(theta + np.pi), r * np.sin(theta + np.pi)]\n", " X = np.vstack([X1, X2]) + rng.normal(0, noise, (2*n, 2))\n", " y = np.hstack([np.zeros(n), np.ones(n)]).astype(int)\n", " # 归一化到 [-1.5, 1.5]\n", " X = (X - X.mean(0)) / X.std(0) * 1.0\n", " return X, y\n", "\n", "datasets = {\n", " \"月牙 moons\": make_moons(noise=0.15, random_state=0),\n", " \"同心圆 circles\": make_circles(noise=0.1, factor=0.4, random_state=0),\n", " \"两团 blobs\": (*make_blobs(centers=2, random_state=0),),\n", " \"螺旋 spiral\": make_spiral(noise=0.4),\n", " \"随机噪声 noise\": (np.random.RandomState(0).uniform(-1, 1, (400, 2)),\n", " np.random.RandomState(1).randint(0, 2, 400)),\n", "}\n", "\n", "# 训练一个 MLP（在每个数据集上）\n", "def fit_and_predict(X, y, hidden=16, epochs=2000, lr=0.3):\n", " rng = np.random.default_rng(42)\n", " W1 = rng.normal(0, 1/np.sqrt(2), (2, hidden))\n", " b1 = np.zeros(hidden)\n", " W2 = rng.normal(0, 1/np.sqrt(hidden), (hidden, 1))\n", " b2 = np.zeros(1)\n", " for _ in range(epochs):\n", " # 前向\n", " z1 = X @ W1 + b1\n", " a1 = sigmoid(z1)\n", " z2 = a1 @ W2 + b2\n", " out = sigmoid(z2).ravel()\n", " # 反向\n", " dz2 = (out - y).reshape(-1, 1)\n", " dW2 = (a1.T @ dz2) / len(X)\n", " db2 = dz2.mean(0)\n", " da1 = dz2 @ W2.T\n", " dz1 = da1 * dsigmoid(a1)\n", " dW1 = (X.T @ dz1) / len(X)\n", " db1 = dz1.mean(0)\n", " W1 -= lr*dW1; b1 -= lr*db1\n", " W2 -= lr*dW2; b2 -= lr*db2\n", " return lambda G: (sigmoid((sigmoid(G @ W1 + b1)) @ W2 + b2).ravel() >= 0.5).astype(int)\n", "\n", "# ===== 画 5 个数据集 + 决策边界 =====\n", "fig, axes = plt.subplots(1, 5, figsize=(22, 4.5))\n", "x1g, x2g = np.meshgrid(np.linspace(-1.5, 1.5, 200), np.linspace(-1.5, 1.5, 200))\n", "grid = np.c_[x1g.ravel(), x2g.ravel()]\n", "\n", "for ax, (name, (X, y)) in zip(axes, datasets.items()):\n", " pred_fn = fit_and_predict(X, y, hidden=16, epochs=3000)\n", " Z = pred_fn(grid).reshape(x1g.shape)\n", " ax.contourf(x1g, x2g, Z, levels=[-0.1, 0.5, 1.1],\n", " colors=['#cfe2ff', '#ffd6d6'], alpha=0.5)\n", " ax.contour(x1g, x2g, Z, levels=[0.5], colors='k', linewidths=1.5)\n", " for xi, yi in zip(X, y):\n", " c = 'red' if yi == 1 else 'blue'\n", " ax.scatter(xi[0], xi[1], c=c, s=18, edgecolors='k', linewidths=0.3, zorder=3)\n", " # 训练集准确率\n", " train_acc = (pred_fn(X) == y).mean()\n", " ax.set_title(f\"{name}\\n训练集准确率: {train_acc:.0%}\", fontsize=11)\n", " ax.set_xticks([]); ax.set_yticks([])\n", "\n", "plt.suptitle(\"MLP（h=16）在 5 个数据集上的表现：前 3 个轻松解，第 4 个勉强，第 5 个束手无策\",\n", " fontsize=13)\n", "plt.tight_layout()\n", "plt.show()\n", "\n", "\n", "# =========================================================================\n", "# 演示\"为什么神经网络也学不会\"——给标签打乱\n", "# =========================================================================\n", "print(\"=\" * 60)\n", "print(\"【附加实验】数据规律 vs 标签噪声：信息论视角\")\n", "print(\"=\" * 60)\n", "from sklearn.datasets import make_moons\n", "X, y = make_moons(noise=0.1, random_state=0)\n", "for noise_level in [0.0, 0.2, 0.4]:\n", " rng = np.random.default_rng(0)\n", " y_noisy = y.copy()\n", " n_flip = int(noise_level * len(y))\n", " idx = rng.choice(len(y), n_flip, replace=False)\n", " y_noisy[idx] = 1 - y_noisy[idx] # 随机翻一部分标签\n", " pred_fn = fit_and_predict(X, y_noisy, hidden=16, epochs=3000)\n", " acc = (pred_fn(X) == y_noisy).mean()\n", " print(f\" 标签被随机翻 {noise_level:.0%} → 训练集准确率: {acc:.1%}（噪声越大，能学会的越少）\")" ] }, { "cell_type": "code", "execution_count": null, "id": "66e88bf2", "metadata": {}, "outputs": [], "source": [ "# =========================================================================\n", "# 10.8 既然两条直线就能解 XOR，为什么要用 MLP？—— 1969 年那段历史\n", "# =========================================================================\n", "# 你说得对：x1+x2-0.5 >= 0 和 x1+x2-1.5 <= 0 是两条直线\n", "# 那能不能用\"两个感知机 + 一个 AND\"训出来呢？我们试一下\n", "# -------------------------------------------------------------------------\n", "print(\"=\" * 64)\n", "print(\" 实验：能不能用感知机直接训出那两条直线？\")\n", "print(\"=\" * 64)\n", "\n", "# 思路：训练\"两个感知机\"分别判断\n", "# 感知机 A：识别 (0,1) 和 (1,0) → 排除 (0,0)\n", "# 感知机 B：识别 (0,0)、(0,1)、(1,0) → 排除 (1,1)\n", "# 理论目标是 x1+x2 = 0.5 和 x1+x2 = 1.5 —— 看感知机真能训出这俩数吗？\n", "\n", "# 准备 4 个 XOR 点的\"二分类\"版本——\n", "# 任务 A：把 (0,1)、(1,0) 标 1，把 (0,0)、(1,1) 标 0 → 线性可分 ✅\n", "# 任务 B：把 (0,0)、(0,1)、(1,0) 标 1，把 (1,1) 标 0 → 线性可分 ✅\n", "X_xor = np.array([[0,0],[0,1],[1,0],[1,1]], dtype=float)\n", "y_xor_local = np.array([0,1,1,0])\n", "\n", "# 子任务 A：把 (0,0) 单独标 0，其余 (0,1),(1,0),(1,1) 标 1\n", "# → 期望: w1+w2·x2 = 0.5? 其实没必要想公式，让感知机自己学\n", "yA = np.array([0,1,1,1])\n", "print(\"\\n子任务 A：排除 (0,0) —— (0,0) 标 0，其余 3 点标 1\")\n", "ppn_A = Perceptron(n_features=2, lr=0.5, seed=1)\n", "ppn_A.fit(X_xor, yA, epochs=20, verbose=False)\n", "# 反推感知机的\"决策直线\"：-b/w1 表示当 x2=0 时的截距\n", "print(f\" 训练权重: w1={ppn_A.w[0]:+.3f}, w2={ppn_A.w[1]:+.3f}, b={ppn_A.b:+.3f}\")\n", "if abs(ppn_A.w[0]) > 1e-6:\n", " print(f\" 决策直线等价: x1 + {(ppn_A.w[1]/ppn_A.w[0]):.3f}·x2 = {-ppn_A.b/ppn_A.w[0]:.3f}\")\n", "print(f\" 预测: {ppn_A.predict(X_xor).tolist()} 真实: {yA.tolist()} 准确率: {(ppn_A.predict(X_xor) == yA).mean():.0%}\")\n", "\n", "# 子任务 B：把 (1,1) 单独标 0，其余 3 点标 1\n", "yB = np.array([1,1,1,0])\n", "print(\"\\n子任务 B：排除 (1,1) —— (1,1) 标 0，其余 3 点标 1\")\n", "ppn_B = Perceptron(n_features=2, lr=0.5, seed=2)\n", "ppn_B.fit(X_xor, yB, epochs=20, verbose=False)\n", "print(f\" 训练权重: w1={ppn_B.w[0]:+.3f}, w2={ppn_B.w[1]:+.3f}, b={ppn_B.b:+.3f}\")\n", "if abs(ppn_B.w[0]) > 1e-6:\n", " print(f\" 决策直线等价: x1 + {(ppn_B.w[1]/ppn_B.w[0]):.3f}·x2 = {-ppn_B.b/ppn_B.w[0]:.3f}\")\n", "print(f\" 预测: {ppn_B.predict(X_xor).tolist()} 真实: {yB.tolist()} 准确率: {(ppn_B.predict(X_xor) == yB).mean():.0%}\")\n", "\n", "# 组合：两个感知机的输出做 AND\n", "print(\"\\n组合：两个感知机输出做 AND\")\n", "outA = ppn_A.predict(X_xor) # 把 (0,0) 标 0\n", "outB = ppn_B.predict(X_xor) # 把 (1,1) 标 0\n", "outAND = outA * outB\n", "print(f\" 感知机 A 输出: {outA.tolist()} → (0,0) 被标 0\")\n", "print(f\" 感知机 B 输出: {outB.tolist()} → (1,1) 被标 0\")\n", "print(f\" AND 之后 : {outAND.tolist()} → 真实: {y_xor_local.tolist()}\")\n", "print(f\" 准确率: {(outAND == y_xor_local).mean():.0%}\")\n", "\n", "\n", "# =========================================================================\n", "# 1969 年 Minsky & Papert 的\"反 MLP\"论证\n", "# =========================================================================\n", "print(\"\\n\" + \"=\" * 64)\n", "print(\" 📜 1969 年 AI 寒冬的真实原因\")\n", "print(\"=\" * 64)\n", "print(\"\"\"\n", "1969 年，Marvin Minsky（AI 之父） 和 Seymour Papert 写了一本书：\n", " 《Perceptrons》—— 证明了\"单层感知机无法解 XOR\"。\n", "\n", "他们同时指出：当时没有算法能训\"多层感知机\"。\n", "（反向传播要等到 1986 年 Hinton 那一波才重新发明。）\n", "\n", "结果：\n", " 1. 官方资金被砍\n", " 2. 神经网络整个领域被冷冻 17 年（1969 – 1986）\n", " 3. 学界转向\"专家系统\"（rule-based AI）\n", "\n", "\n", "但是！Minsky 漏看了一件事——\"两条直线\"不是单层感知机能训出来的：\n", "\"\"\")\n", "\n", "# 关键实验：直接把 4 个 XOR 点塞给\"单层感知机\"会怎样\n", "print(\" 把整个 XOR 真值表直接塞给单层感知机：\")\n", "ppn_xor = Perceptron(n_features=2, lr=0.5, seed=3)\n", "ppn_xor.fit(X_xor, y_xor_local, epochs=200, verbose=False)\n", "print(f\" 训练权重: w1={ppn_xor.w[0]:+.3f}, w2={ppn_xor.w[1]:+.3f}, b={ppn_xor.b:+.3f}\")\n", "print(f\" 预测: {ppn_xor.predict(X_xor).tolist()} 真实: {y_xor_local.tolist()}\")\n", "print(f\" 准确率: {(ppn_xor.predict(X_xor) == y_xor_local).mean():.0%} ← 它在原地打转\")\n", "\n", "print(\"\"\"\n", "而你的\"两条直线\"思路需要：\n", " - 拆成 2 个\"线性子任务\"（人为拆分）\n", " - 每个子任务单独训\n", " - 再手写一个 AND 把它们组合起来\n", "\n", "这种\"人为拆分 + 手动组合\"在简单数据（如 4 个 XOR 点）能行——\n", "但问题是：现实数据没有\"4 个点这么干净的\"。\n", "\n", "例如识别手写数字 7：\n", " - 你要怎么\"人为拆分\"成 4 条直线？\n", " - 拆分后要怎么\"手写 AND\"组合？\n", "\"\"\")\n", "\n", "# =========================================================================\n", "# MLP 真正的优势：自动学\"拆分子任务 + 组合\"\n", "# =========================================================================\n", "print(\"=\" * 64)\n", "print(\" MLP 真正强大之处：自动学特征 + 自动组合\")\n", "print(\"=\" * 64)\n", "print(\"\"\"\n", "MLP 做的事：\n", "\n", "输入层 (2 维) → 隐藏层 (h 维) → 输出层 (1 维)\n", " x1, x2 │ h1=σ(w1·x1+w2·x2+b1) │ y=σ(W1·h1+W2·h2+b)\n", " │ h2=σ(w3·x1+w4·x2+b2) │\n", " │ ... │\n", "\n", "隐藏层每个 h_i 就是\"一条直线 + 阶跃\"的自动产物。\n", "输出层把 h_i 重新组合起来。\n", "\n", "🧠 你给它的不是\"公式\"——而是\"数据\"。\n", "\n", " 两直线思路：你要先知道问题能拆，再手写公式\n", " MLP 思路 ：你只管喂数据，它自己找拆法\n", "\n", "当数据是 4 个 XOR 点时：两直线更快、更透明。\n", "当数据是 100 万张猫狗图片时：没人能写出\"两直线\"公式——\n", " MLP 才是唯一可扩展的解法。\n", "\"\"\")" ] }, { "cell_type": "markdown", "id": "eecc4b1a", "metadata": {}, "source": [ "## sigmoid 求导\n", "\n", "- 原函数：$f(x)=\\frac{1}{1+e^{-x}}$\n", "- 变形：$f(x)=(1+e^{-x})^{-1}$\n", "- 设 $u=1+e^{-x}$，则 $(u^{-1})'=-1*u^{-2}=-u^{-2}=-\\frac{1}{u^2}=-\\frac{1}{(1+e^{-x})^2}$\n", "- 设 $v=-x$，则 $(1+e^v)'=e^v=e^{-x}$\n", "- $(-x)'=-1$\n", "- $f'(x)=(u^{-1})'*(1+e^v)'*(-x)'=-\\frac{1}{(1+e^{-x})^2}*e^{-x}*-1=\\frac{e^{-x}}{(1+e^{-x})^2}$\n" ] } ], "metadata": { "kernelspec": { "display_name": "qlib", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 5 }