{ "cells": [ { "cell_type": "code", "execution_count": null, "id": "5f8cf592-5f77-40c6-a57d-89f4ea5bc25d", "metadata": {}, "outputs": [], "source": [ "import numpy as np" ] }, { "cell_type": "markdown", "id": "8d10e7ac", "metadata": {}, "source": [ "# 用 NumPy 从零实现两层神经网络 — 解决 XOR 问题\n", "\n", "## 1. 任务背景\n", "\n", "我们要学一个二分类函数 **XOR（异或）**：\n", "\n", "| $x_1$ | $x_2$ | $y = x_1 \\oplus x_2$ |\n", "| :--: | :--: | :--: |\n", "| 0 | 0 | 0 |\n", "| 0 | 1 | 1 |\n", "| 1 | 0 | 1 |\n", "| 1 | 1 | 0 |\n", "\n", "观察数据可以发现，**正类 (1) 和负类 (0) 在二维平面上是交叉分布的**，\n", "无论怎么画一条直线都无法把 $\\{ (0,1), (1,0) \\}$ 与 $\\{ (0,0), (1,1) \\}$ 分开。\n", "这说明 XOR **不是线性可分**的，单层感知机（无隐藏层）学不出来。\n", "\n", "解决办法是引入一个**带激活函数的隐藏层**，让网络具有非线性能力。\n", "\n", "## 2. 网络结构\n", "\n", "$$\n", "x \\in \\mathbb{R}^{2}\n", "\\;\\xrightarrow{\\,W_1,\\,b_1\\,}\\;\n", "z_1 \\in \\mathbb{R}^{4}\n", "\\;\\xrightarrow{\\sigma}\\;\n", "a_1 \\in \\mathbb{R}^{4}\n", "\\;\\xrightarrow{\\,W_2,\\,b_2\\,}\\;\n", "z_2 \\in \\mathbb{R}^{1}\n", "\\;\\xrightarrow{\\sigma}\\;\n", "\\hat y \\in (0,1)\n", "$$\n", "\n", "- 隐藏层 4 个神经元 + sigmoid 激活；\n", "- 输出层 1 个神经元 + sigmoid，把 logits 压成概率。\n", "\n", "## 3. 训练流程概览\n", "\n", "1. **前向传播**：算 $z_1 \\to a_1 \\to z_2 \\to \\hat y$，并计算损失 $L$。\n", "2. **反向传播**：用链式法则从输出往回算 $\\partial L/\\partial W_1, \\partial L/\\partial W_2$ 等梯度。\n", "3. **参数更新**：$W \\leftarrow W - \\eta \\cdot \\partial L/\\partial W$。\n", "4. 重复 1~3 若干轮 (epoch)，直到损失不再下降。\n" ] }, { "cell_type": "code", "execution_count": null, "id": "0a502ef2-0f71-417b-87fe-c81710cf91fc", "metadata": {}, "outputs": [], "source": [ "seed = 1\n", "n_in = 2\n", "n_hidden = 4\n", "lr = 1.0" ] }, { "cell_type": "markdown", "id": "d973f4ee", "metadata": {}, "source": [ "### 超参数\n", "\n", "- `seed`：随机种子，保证结果可复现。\n", "- `n_in=2`、`n_hidden=4`：输入维度和隐藏层神经元个数。\n", "- `lr=1.0`（learning rate）：学习率，控制每一步参数更新的\"步长\"。 \n", " 太小收敛慢，太大容易在最优解附近来回震荡甚至发散。\n" ] }, { "cell_type": "code", "execution_count": null, "id": "76b63c6d-6fd0-4f0b-af76-7d658bbadd0d", "metadata": {}, "outputs": [], "source": [ "# 训练数据：4 个样本 (N=4)，每个样本 2 个特征\n", "# X.shape = (N, n_in) = (4, 2)\n", "# y.shape = (N,) = (4,)\n", "X = np.array([[0,0],\n", " [0,1],\n", " [1,0],\n", " [1,1]])\n", "y = np.array([0, 1, 1, 0])" ] }, { "cell_type": "markdown", "id": "c81f2216-1e44-41e8-92a9-c4256ae436c7", "metadata": {}, "source": [ "## 初始化\n", "\n", "### 形状约定\n", "\n", "按\"行=样本\"的布局 (与 PyTorch / NumPy 矩阵乘法一致)：\n", "\n", "$$\n", "W^{(1)} \\in \\mathbb{R}^{n_{\\text{in}} \\times n_{\\text{hidden}}}\n", "\\;=\\; \\mathbb{R}^{2 \\times 4},\n", "\\qquad\n", "W^{(2)} \\in \\mathbb{R}^{n_{\\text{hidden}} \\times 1}\n", "\\;=\\; \\mathbb{R}^{4 \\times 1}\n", "$$\n", "\n", "- $W^{(1)}_{ij}$：从输入第 $i$ 维到隐藏层第 $j$ 个神经元的权重；\n", "- $W^{(2)}_{ij}$：从隐藏层第 $i$ 个神经元到输出第 $j$ 个神经元的权重；\n", "- 偏置 $b^{(1)} \\in \\mathbb{R}^{4}$，$b^{(2)} \\in \\mathbb{R}^{1}$。\n", "\n", "### 初始化方式\n", "\n", "使用均值为 0、标准差为 $1/\\sqrt{n_{\\text{in}}}$（前层神经元数）的高斯分布，\n", "这其实就是 **Xavier 初始化**的简化版本，目的是让每一层激活值的方差大致保持在 1，\n", "从而缓解 sigmoid 带来的梯度消失问题。\n", "\n", "$$\n", "W^{(1)}_{ij} \\sim \\mathcal N\\!\\left(0,\\; \\frac{1}{n_{\\text{in}}}\\right),\n", "\\quad\n", "W^{(2)}_{ij} \\sim \\mathcal N\\!\\left(0,\\; \\frac{1}{n_{\\text{hidden}}}\\right)\n", "$$\n", "\n", "偏置全部初始化为 0。\n" ] }, { "cell_type": "code", "execution_count": null, "id": "afca8e2c-988b-473a-920e-b6452e704a15", "metadata": {}, "outputs": [], "source": [ "rng = np.random.default_rng(seed)" ] }, { "cell_type": "code", "execution_count": null, "id": "4752cd16-a383-4ec4-a3bb-90992aa82a7f", "metadata": {}, "outputs": [], "source": [ "W1 = rng.normal(0, 1/np.sqrt(n_in),size=(n_in, n_hidden))\n", "W1.shape" ] }, { "cell_type": "code", "execution_count": null, "id": "44af3c28-b3d4-4e86-91e4-84c89559c701", "metadata": {}, "outputs": [], "source": [ "W1" ] }, { "cell_type": "code", "execution_count": null, "id": "8e781485-8486-4222-a481-3cb189dcbdeb", "metadata": {}, "outputs": [], "source": [ "b1 = np.zeros(n_hidden)\n", "b1" ] }, { "cell_type": "code", "execution_count": null, "id": "2b3488f2-1601-4f83-b696-f12d9b8c2c03", "metadata": {}, "outputs": [], "source": [ "W2 = rng.normal(0, 1/np.sqrt(n_hidden), size=(n_hidden, 1))\n", "W2.shape" ] }, { "cell_type": "code", "execution_count": null, "id": "d939ef45-1ecc-42bb-91ac-a115fe79fc11", "metadata": {}, "outputs": [], "source": [ "W2" ] }, { "cell_type": "code", "execution_count": null, "id": "2b355f9e-dfdb-40b8-bea6-16a3e636157a", "metadata": {}, "outputs": [], "source": [ "b2 = np.zeros(1)\n", "b2.shape" ] }, { "cell_type": "code", "execution_count": null, "id": "7cc6f562-5c1e-41e8-b630-7229c3b37d0b", "metadata": {}, "outputs": [], "source": [ "b2" ] }, { "cell_type": "markdown", "id": "e72cfdc8-d644-4935-b100-0f2fe4ffb5d7", "metadata": {}, "source": [ "## 前向传播\n", "\n", "### sigmoid 激活函数\n", "\n", "$$\n", "\\sigma(z) \\;=\\; \\frac{1}{1 + e^{-z}}\n", "\\quad\\Rightarrow\\quad\n", "\\sigma'(z) \\;=\\; \\sigma(z)\\bigl(1 - \\sigma(z)\\bigr)\n", "$$\n", "\n", "> 性质：$\\sigma(z) \\in (0,1)$ 单调递增；其导数用\"激活后的值 $a=\\sigma(z)$\" \n", "> 表示为 $a(1-a)$，反向传播时直接用 `a` 算就行，不必再算 $z$。\n", "\n", "### 隐藏层线性变换\n", "\n", "$$\n", "Z^{(1)} \\;=\\; X \\, W^{(1)} + b^{(1)}\n", "\\;=\\;\n", "\\underbrace{X}_{(N,2)}\n", "\\;\\underbrace{W^{(1)}}_{(2,4)}\n", "\\;+\\;\n", "\\underbrace{b^{(1)}}_{(4,)}\n", "$$\n", "\n", "形状对位：\n", "\n", "| 量 | 形状 | 说明 |\n", "| --- | --- | --- |\n", "| $X$ | $(4, 2)$ | 4 个样本，每个 2 维 |\n", "| $W^{(1)}$ | $(2, 4)$ | 输入到隐藏的权重 |\n", "| $b^{(1)}$ | $(4,)$ | 隐藏层偏置（自动广播到 $(4,4)$ 后加到每一行） |\n", "| $Z^{(1)}$ | $(4, 4)$ | 隐藏层\"激活前\"的输出 |\n", "\n", "激活 $A^{(1)} = \\sigma(Z^{(1)})$，形状不变 $(4,4)$。\n", "\n", "### 输出层\n", "\n", "$$\n", "Z^{(2)} \\;=\\; A^{(1)} W^{(2)} + b^{(2)}\n", "\\;=\\;\n", "\\underbrace{A^{(1)}}_{(4,4)}\n", "\\;\\underbrace{W^{(2)}}_{(4,1)}\n", "\\;+\\;\n", "\\underbrace{b^{(2)}}_{(1,)}\n", "$$\n", "\n", "$Z^{(2)}$ 形状 $(4,1)$，再过 sigmoid 得到 $\\hat y = \\sigma(Z^{(2)})$， \n", "最后 `.ravel()` 拉平成 $(4,)$ 与 $y$ 对齐。\n" ] }, { "cell_type": "code", "execution_count": null, "id": "28cc0dcd-2185-4555-8a33-220ca89b43e8", "metadata": { "scrolled": true }, "outputs": [], "source": [ "# 隐藏层线性变换：Z1 = X @ W1 + b1\n", "# (4,2) @ (2,4) + (4,) -> (4,4) (b1 沿行广播)\n", "z1 = X @ W1 + b1\n", "X.shape, W1.shape, b1.shape, z1.shape" ] }, { "cell_type": "code", "execution_count": null, "id": "f24f6d1d-0155-41a9-b867-20ff789f3e9f", "metadata": {}, "outputs": [], "source": [ "z1" ] }, { "cell_type": "code", "execution_count": null, "id": "d2d90c74-62d9-4600-8abf-139f5dcd8685", "metadata": {}, "outputs": [], "source": [ "def sigmoid(z):\n", " # 为数值稳定用 np.clip\n", " return 1.0 / (1.0 + np.exp(-np.clip(z, -500, 500)))\n", "\n", "def dsigmoid(a):\n", " \"\"\"sigmoid 对 a 本身求导（a 已经是 sigmoid(z)）。\n", "\n", " 由 σ'(z) = σ(z)·(1 − σ(z))，把 a = σ(z) 代入即可。\n", " 直接用 a 算比用 z 算更省事，省一次 sigmoid 调用。\n", " \"\"\"\n", " return a * (1.0 - a)" ] }, { "cell_type": "code", "execution_count": null, "id": "af6f4f4c-062e-4cc0-adde-88f0466609fc", "metadata": {}, "outputs": [], "source": [ "# 隐藏层激活：A1 = σ(Z1)，形状仍然是 (4,4)\n", "a1 = sigmoid(z1)\n", "a1.shape" ] }, { "cell_type": "code", "execution_count": null, "id": "b2cba430-23b9-40e0-96a5-37504f89b2ec", "metadata": {}, "outputs": [], "source": [ "a1" ] }, { "cell_type": "code", "execution_count": null, "id": "581d0be9-6807-48dc-bef2-a932e65d25f0", "metadata": {}, "outputs": [], "source": [ "# 输出层线性变换：Z2 = A1 @ W2 + b2\n", "# (4,4) @ (4,1) + (1,) -> (4,1)\n", "z2 = a1 @ W2 + b2\n", "a1.shape, W2.shape, b2.shape, z2.shape" ] }, { "cell_type": "code", "execution_count": null, "id": "ce2fd10f-aabc-4742-a021-c8eb69d1f592", "metadata": {}, "outputs": [], "source": [ "z2" ] }, { "cell_type": "code", "execution_count": null, "id": "75873878-7925-44cb-90f7-a7dce0256322", "metadata": {}, "outputs": [], "source": [ "# ŷ = σ(Z2)，.ravel() 把 (4,1) 拉平成 (4,)，便于和 y 做逐元素运算\n", "y_pred = sigmoid(z2).ravel()\n", "y_pred.shape" ] }, { "cell_type": "code", "execution_count": null, "id": "49a8333f-7b37-4d18-b9fd-753973669dd2", "metadata": {}, "outputs": [], "source": [ "y_pred" ] }, { "cell_type": "code", "execution_count": null, "id": "0dceac39-758b-442d-8b60-70d2e99e8810", "metadata": {}, "outputs": [], "source": [ "# 二分类交叉熵损失（加 1e-8 防止 log(0)）\n", "# L = -mean( y·log ŷ + (1-y)·log(1-ŷ) )\n", "loss = -np.mean(y * np.log(y_pred + 1e-8) +\n", " (1 - y) * np.log(1 - y_pred + 1e-8))" ] }, { "cell_type": "markdown", "id": "cb7d4304", "metadata": {}, "source": [ "### 损失函数：二分类交叉熵 (Binary Cross-Entropy)\n", "\n", "对单个样本 $(x, y)$，$y \\in \\{0,1\\}$，$\\hat y \\in (0,1)$：\n", "\n", "$$\n", "\\ell\\bigl(y, \\hat y\\bigr) \\;=\\;\n", "-\\,\\Bigl[\\,y \\log \\hat y \\;+\\; (1-y)\\,\\log(1-\\hat y)\\,\\Bigr]\n", "$$\n", "\n", "对 $N$ 个样本取平均：\n", "\n", "$$\n", "L \\;=\\; \\frac{1}{N} \\sum_{i=1}^{N} \\ell\\bigl(y^{(i)}, \\hat y^{(i)}\\bigr)\n", "\\;=\\;\n", "-\\frac{1}{N} \\sum_{i=1}^{N}\n", "\\Bigl[\\,y^{(i)} \\log \\hat y^{(i)} + (1-y^{(i)})\\log(1-\\hat y^{(i)})\\,\\Bigr]\n", "$$\n", "\n", "**为什么用交叉熵而不是 MSE？** \n", "当输出层用 sigmoid 时，$\\partial L/\\partial \\hat y$ 在 $\\hat y$ 接近 0 或 1 时 \n", "被 $1/\\hat y$ 或 $1/(1-\\hat y)$ 项放大，**不会**像 MSE 那样因 sigmoid 导数小而梯度消失， \n", "训练更稳定。配合 sigmoid 输出层，交叉熵 + sigmoid 还有一个非常简洁的结论：\n", "\n", "$$\n", "\\frac{\\partial L}{\\partial Z^{(2)}} \\;=\\; \\hat y - y\n", "$$\n", "\n", "这正是后面反向传播第一行 `dz2 = y_pred - y` 的来源。\n" ] }, { "cell_type": "code", "execution_count": null, "id": "228cb275-839c-4a34-9b82-92bf058973f4", "metadata": {}, "outputs": [], "source": [ "loss" ] }, { "cell_type": "markdown", "id": "24ff5a00-f458-42b7-8a05-c06a061f1d8e", "metadata": {}, "source": [ "## 反向传播\n", "\n", "### 整体思路\n", "\n", "我们从损失 $L$ 出发，沿着前向传播的反方向，用**多元链式法则**把梯度从后往前推。\n", "\n", "$$\n", "L(\\hat y, y)\n", "\\;\\xleftarrow{\\text{链式}}\\;\n", "Z^{(2)}\n", "\\;\\xleftarrow{\\text{链式}}\\;\n", "A^{(1)}\n", "\\;\\xleftarrow{\\text{链式}}\\;\n", "Z^{(1)}\n", "\\;\\xleftarrow{\\text{链式}}\\;\n", "W^{(1)}, b^{(1)}\n", "$$\n", "\n", "### 逐项推导\n", "\n", "**(1) 输出层** —— 用\"交叉熵 + sigmoid\"那个极简结论：\n", "\n", "$$\n", "\\frac{\\partial L}{\\partial Z^{(2)}} \\;=\\; \\hat y - y \\quad\\Longrightarrow\\quad\n", "\\boxed{dZ^{(2)} = \\hat y - y}\n", "$$\n", "\n", "- $dZ^{(2)}$ 形状 $(N, 1)$，每行是一个样本的\"误差信号\"。\n", "\n", "**(2) $W^{(2)}$, $b^{(2)}$ 的梯度**（$Z^{(2)} = A^{(1)} W^{(2)} + b^{(2)}$）：\n", "\n", "$$\n", "\\frac{\\partial L}{\\partial W^{(2)}} \\;=\\; \\frac{1}{N} (A^{(1)})^\\top dZ^{(2)}\n", "\\;\\in\\; \\mathbb R^{4 \\times 1}\n", "$$\n", "\n", "$$\n", "\\frac{\\partial L}{\\partial b^{(2)}} \\;=\\; \\frac{1}{N} \\mathbf 1^\\top dZ^{(2)}\n", "\\;\\in\\; \\mathbb R^{1}\n", "$$\n", "\n", "**(3) 误差信号向隐藏层回传**：\n", "\n", "$$\n", "\\frac{\\partial L}{\\partial A^{(1)}} \\;=\\; dZ^{(2)} \\, (W^{(2)})^\\top \\quad \\in\\; \\mathbb R^{N \\times 4}\n", "$$\n", "\n", "**(4) 隐藏层激活导数**（$A^{(1)} = \\sigma(Z^{(1)})$，$a(1-a)$ 是元素级）：\n", "\n", "$$\n", "dZ^{(1)} \\;=\\; dA^{(1)} \\odot \\sigma'(Z^{(1)}) \\;=\\; dA^{(1)} \\odot A^{(1)} \\odot \\bigl(1 - A^{(1)}\\bigr)\n", "$$\n", "\n", "**(5) $W^{(1)}$, $b^{(1)}$ 的梯度**（$Z^{(1)} = X W^{(1)} + b^{(1)}$）：\n", "\n", "$$\n", "\\frac{\\partial L}{\\partial W^{(1)}} \\;=\\; \\frac{1}{N} X^\\top dZ^{(1)} \\quad \\in\\; \\mathbb R^{2 \\times 4}\n", "$$\n", "\n", "$$\n", "\\frac{\\partial L}{\\partial b^{(1)}} \\;=\\; \\frac{1}{N} \\mathbf 1^\\top dZ^{(1)} \\quad \\in\\; \\mathbb R^{4}\n", "$$\n", "\n", "### 为什么所有权重梯度公式长得都像 $X^\\top dZ$？\n", "\n", "在线性层 $Z = XW + b$ 里，$\\partial L/\\partial W$ 的第 $(i, j)$ 个元素 \n", "$= \\sum_n X_{n,i} \\cdot dZ_{n,j}$，这正是 $(X^\\top dZ)_{ij}$。 \n", "所以\"输入转置 × 输出梯度\"是一个通用模式，在 PyTorch 里就是 `X.T @ dZ`。\n" ] }, { "cell_type": "code", "execution_count": null, "id": "f473cbb2-f8e7-4fb7-9d48-e2b634a7120f", "metadata": {}, "outputs": [], "source": [ "# 样本数，后面求平均梯度用\n", "N = len(X)\n", "N" ] }, { "cell_type": "code", "execution_count": null, "id": "afa7aaff-9d9b-4d47-a148-e5b994a96be2", "metadata": {}, "outputs": [], "source": [ "# 把 y_pred 从 (N,) 变回 (N,1)，便于和 W2 做矩阵乘法\n", "y_pred = y_pred.reshape(-1, 1)\n", "y_pred" ] }, { "cell_type": "code", "execution_count": null, "id": "1ef9527b-e5a3-43ad-b28e-d85438d1a0f8", "metadata": {}, "outputs": [], "source": [ "# dZ2 = ∂L/∂Z2 = ŷ - y （交叉熵 + sigmoid 的经典结论）\n", "# 形状 (N,1)\n", "dz2 = (y_pred - y.reshape(-1, 1))\n", "dz2" ] }, { "cell_type": "markdown", "id": "ea04a7f9", "metadata": {}, "source": [ "### 关键推导：为什么 dZ2 = ŷ − y ？\n", "\n", "这个结论并不是直接套公式得到的，而是**链式法则**与两个偏导\"相消\"后的巧合。\n", "下面用单个样本把推导完整走一遍（多样本只是按行独立地重复）。\n", "\n", "**设定**（对单个样本，下标省略）：\n", "\n", "$$\n", "z \\;=\\; z^{(2)}, \\qquad\n", "\\hat y \\;=\\; \\sigma(z) \\;=\\; \\frac{1}{1 + e^{-z}}, \\qquad\n", "L \\;=\\; -\\,\\bigl[\\,y \\log \\hat y + (1-y)\\log(1-\\hat y)\\,\\bigr]\n", "$$\n", "\n", "**目标**：求 $\\dfrac{\\partial L}{\\partial z}$。\n", "\n", "---\n", "\n", "**第 1 步 —— 链式法则拆开**\n", "\n", "$$\n", "\\frac{\\partial L}{\\partial z} \\;=\\; \\frac{\\partial L}{\\partial \\hat y} \\cdot \\frac{\\partial \\hat y}{\\partial z}\n", "$$\n", "\n", "---\n", "\n", "**第 2 步 —— 算 $\\partial L / \\partial \\hat y$**\n", "\n", "把 $L$ 看作 $\\hat y$ 的函数（$y$ 是常量）：\n", "\n", "$$\n", "L = -y \\log \\hat y - (1-y) \\log(1-\\hat y)\n", "$$\n", "\n", "$$\n", "\\boxed{\\;\n", "\\frac{\\partial L}{\\partial \\hat y}\n", "\\;=\\; -\\frac{y}{\\hat y} \\;-\\; (1-y)\\cdot\\frac{-1}{1-\\hat y}\n", "\\;=\\; -\\frac{y}{\\hat y} \\;+\\; \\frac{1-y}{1-\\hat y}\n", "\\;}\n", "$$\n", "\n", "> 直觉：$\\hat y$ 越大（越接近 1），第一项 $-\\log\\hat y$ 越小，损失越小； \n", "> 所以 $-\\partial L/\\partial \\hat y$ 的第一项 $y/\\hat y$ 应当随 $\\hat y$ 增大而减小。 \n", "> 这正和公式一致。\n", "\n", "---\n", "\n", "**第 3 步 —— 算 $\\partial \\hat y / \\partial z$（sigmoid 的导数）**\n", "\n", "$$\n", "\\hat y = \\frac{1}{1+e^{-z}} = (1+e^{-z})^{-1}\n", "$$\n", "\n", "$$\n", "\\frac{\\partial \\hat y}{\\partial z}\n", "\\;=\\; -(1+e^{-z})^{-2}\\cdot(-e^{-z})\n", "\\;=\\; \\frac{e^{-z}}{(1+e^{-z})^{2}}\n", "$$\n", "\n", "把它改写成含 $\\hat y$ 的形式：\n", "\n", "$$\n", "\\frac{e^{-z}}{(1+e^{-z})^{2}}\n", "\\;=\\; \\underbrace{\\frac{1}{1+e^{-z}}}_{\\hat y} \\cdot \\underbrace{\\frac{e^{-z}}{1+e^{-z}}}_{1 - \\hat y}\n", "\\;=\\; \\hat y\\,(1-\\hat y)\n", "$$\n", "\n", "$$\n", "\\boxed{\\;\n", "\\frac{\\partial \\hat y}{\\partial z} \\;=\\; \\hat y\\,(1-\\hat y)\n", "\\;}\n", "$$\n", "\n", "> 另一种\"对数求导\"更短：$\\log \\hat y = -\\log(1+e^{-z})$， \n", "> 两边对 $z$ 求导：$\\dfrac{1}{\\hat y}\\hat y' = \\dfrac{e^{-z}}{1+e^{-z}} = 1-\\hat y$， \n", "> 于是 $\\hat y' = \\hat y(1-\\hat y)$。\n", "\n", "---\n", "\n", "**第 4 步 —— 把两段乘起来，并\"通分\"消项**\n", "\n", "$$\n", "\\frac{\\partial L}{\\partial z}\n", "\\;=\\;\n", "\\underbrace{\\left(-\\frac{y}{\\hat y} + \\frac{1-y}{1-\\hat y}\\right)}_{\\partial L/\\partial\\hat y}\n", "\\cdot\n", "\\underbrace{\\hat y\\,(1-\\hat y)}_{\\partial \\hat y/\\partial z}\n", "$$\n", "\n", "把括号里的两项分别乘：\n", "\n", "$$\n", "= \\;-\\frac{y}{\\hat y}\\cdot \\hat y(1-\\hat y) \\;+\\; \\frac{1-y}{1-\\hat y}\\cdot \\hat y(1-\\hat y)\n", "$$\n", "\n", "$$\n", "= \\;-y(1-\\hat y) \\;+\\; (1-y)\\hat y\n", "$$\n", "\n", "展开两项：\n", "\n", "$$\n", "= \\;-y + y\\hat y \\;+\\; \\hat y - y\\hat y\n", "$$\n", "\n", "中间两项 $y\\hat y$ 抵消：\n", "\n", "$$\n", "= \\;-y + \\hat y \\;=\\; \\hat y - y\n", "$$\n", "\n", "$$\n", "\\boxed{\\;\n", "\\frac{\\partial L}{\\partial z} \\;=\\; \\hat y - y\n", "\\;}\n", "$$\n", "\n", "---\n", "\n", "**结论与意义**\n", "\n", "- $N$ 个样本独立，于是 $\\partial L/\\partial Z^{(2)} = \\hat y - y$ 逐元素成立，形状 $(N,1)$。\n", "- 整个反传最复杂的\"激活+损失\"复合导数，**化简成了一个超干净的\"预测减真值\"**。\n", "- 这就是为什么\"sigmoid 输出 + 交叉熵\"是教科书的经典组合—— \n", " 不光梯度形式简洁，而且当 $\\hat y \\to y$ 时梯度 $\\to 0$，**预测越准、步子越小**，训练天然稳定。\n", "- 推广到 $K$ 类的 softmax + 多元交叉熵，同样有 $\\partial L/\\partial Z = \\hat y - y$（one-hot 形式），是同一思想的延伸。\n" ] }, { "cell_type": "markdown", "id": "5ebab610", "source": "### 进一步：$\\partial L/\\partial W^{(2)}$ 和 $\\partial L/\\partial b^{(2)}$ 的求解\n\n上一步我们得到了 $dZ^{(2)} = \\partial L/\\partial Z^{(2)} = \\hat y - y$（形状 $(N,1)$）， \n现在要沿着 $Z^{(2)} = A^{(1)} W^{(2)} + b^{(2)}$ 继续往回算 $W^{(2)}, b^{(2)}$ 的梯度。\n\n**记号约定**（与代码里的形状一致）：\n\n| 量 | 形状 | 含义 |\n| --- | --- | --- |\n| $A^{(1)}$ | $(N, 4)$ | 隐藏层激活值，4 个特征 |\n| $W^{(2)}$ | $(4, 1)$ | 隐藏→输出的权重 |\n| $b^{(2)}$ | $(1,)$ | 输出层偏置（标量）|\n| $Z^{(2)}$ | $(N, 1)$ | 输出层激活前 |\n| $dZ^{(2)}$ | $(N, 1)$ | 已经算好的\"误差信号\" |\n\n---\n\n**先看单样本**（便于看清每个元素是怎么来的）\n\n去掉 $N$ 这个 batch 维度，对第 $n$ 个样本：\n\n$$\nz_n^{(2)} \\;=\\; \\sum_{i=1}^{4} a_{n,i}^{(1)}\\,w_i^{(2)} \\;+\\; b^{(2)}\n\\qquad\\in\\;\\mathbb R\n$$\n\n把它对 $w_i^{(2)}$ 求偏导（链式法则）：\n\n$$\n\\frac{\\partial L}{\\partial w_i^{(2)}}\n\\;=\\; \\frac{\\partial L}{\\partial z_n^{(2)}}\\cdot \\frac{\\partial z_n^{(2)}}{\\partial w_i^{(2)}}\n\\;=\\; (\\hat y_n - y_n)\\cdot a_{n,i}^{(1)}\n\\;=\\; a_{n,i}^{(1)} \\cdot dZ^{(2)}_n\n$$\n\n同理对 $b^{(2)}$（它对 $z_n^{(2)}$ 的偏导是 1）：\n\n$$\n\\frac{\\partial L}{\\partial b^{(2)}}\\bigg|_{\\text{样本 }n}\n\\;=\\; (\\hat y_n - y_n) \\cdot 1\n\\;=\\; dZ^{(2)}_n\n$$\n\n---\n\n**再把 $N$ 个样本合起来**（注意 $L$ 是样本平均，所以要再除以 $N$）\n\n把上面\"单样本\"的偏导对所有 $n$ 求平均，就是 $\\partial L/\\partial W^{(2)}$ 的第 $i$ 行：\n\n$$\n\\frac{\\partial L}{\\partial w_i^{(2)}}\n\\;=\\; \\frac{1}{N}\\sum_{n=1}^{N} a_{n,i}^{(1)} \\cdot dZ^{(2)}_n\n$$\n\n把它写成矩阵形式：左边是 $(4, 1)$ 的列向量，右边把 $n$ 折叠到矩阵乘法里， \n**正好就是 $A^{(1)}$ 的转置乘 $dZ^{(2)}$**：\n\n$$\n\\boxed{\\;\n\\frac{\\partial L}{\\partial W^{(2)}}\n\\;=\\; \\frac{1}{N}\\,(A^{(1)})^\\top \\,dZ^{(2)}\n\\;}\n$$\n\n形状对位：$(4,N) @ (N,1) \\to (4,1)$，与 $W^{(2)}$ 同形 ✓\n\n---\n\n**对 $b^{(2)}$，把 $N$ 个 $dZ^{(2)}_n$ 累加并平均**：\n\n$$\n\\frac{\\partial L}{\\partial b^{(2)}}\n\\;=\\; \\frac{1}{N}\\sum_{n=1}^{N} dZ^{(2)}_n\n\\;=\\; \\overline{dZ^{(2)}}\n$$\n\n$$\n\\boxed{\\;\n\\frac{\\partial L}{\\partial b^{(2)}}\n\\;=\\; \\frac{1}{N}\\,\\mathbf 1^\\top \\,dZ^{(2)}\n\\;}\n$$\n\n形状对位：$(1,N) @ (N,1) \\to (1,1)$，与 $b^{(2)}$ 同形 ✓\n\n在 NumPy 里，`dZ.sum(axis=0)` 就是把每一列对所有样本求和（这里列只有 1 个）， \n再 ` / N` 就是 `dZ.mean(axis=0)`。\n\n---\n\n**换个角度理解\"为什么是 $A^\\top dZ$\"**\n\n把公式拆开看元素：\n\n$$\n\\left(\\frac{\\partial L}{\\partial W^{(2)}}\\right)_{i}\n\\;=\\; \\frac{1}{N}\\sum_{n=1}^{N} \\underbrace{a_{n,i}^{(1)}}_{\\text{第 } n \\text{ 个样本的第 } i \\text{ 个特征}} \\cdot \\underbrace{dZ^{(2)}_n}_{\\text{该样本的误差}}\n$$\n\n- $a_{n,i}^{(1)}$：当特征 $i$ 越大、误差 $dZ^{(2)}_n$ 越大时，权重 $w_i^{(2)}$ 就该被调得越多；\n- 对所有样本取平均：单个样本的噪声被抹平，得到的梯度方向才是\"全局最优\"。\n\n这就是为什么线性层 $Z = AW + b$ 永远有 $\\partial L/\\partial W = A^\\top\\,dZ/N$—— \n**它本质上是\"输入特征\"和\"输出误差\"的相关性矩阵**。 \n在 PyTorch 里同样的公式就是 `A.T @ dZ / N`，只是写成 `A.mT @ dZ / batch_size`。\n", "metadata": {} }, { "cell_type": "code", "execution_count": 48, "id": "1de29b41-23e6-4616-bc40-cc886e0299e7", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[ 0.003297 ],\n", " [-0.00116781],\n", " [-0.00139055],\n", " [-0.00396119]])" ] }, "execution_count": 48, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# dW2 = (1/N) · A1.T @ dZ2\n", "# (4,4).T @ (4,1) -> (4,1)，与 W2 同形\n", "dW2 = (a1.T @ dz2) / N\n", "dW2" ] }, { "cell_type": "code", "execution_count": 49, "id": "329b0c69-1116-4b3a-aad6-26b43643f683", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([-9.09967091e-05])" ] }, "execution_count": 49, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# db2 = dZ2 在样本维 (axis=0) 上的平均，形状 (1,)\n", "db2 = dz2.mean(axis=0) \n", "db2" ] }, { "cell_type": "code", "execution_count": 50, "id": "3f8c2282-25c2-451c-b74b-dd5576307ea3", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[-0.05641362, 0.03935358, 0.01839352, 0.06595786],\n", " [ 0.1368688 , -0.09547831, -0.04462573, -0.1600247 ],\n", " [ 0.13615014, -0.09497698, -0.04439141, -0.15918445],\n", " [-0.21383315, 0.14916786, 0.06971976, 0.25001011]])" ] }, "execution_count": 50, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# dA1 = dZ2 @ W2.T\n", "# (N,1) @ (1,4) -> (N,4)，即 ∂L/∂A1\n", "da1 = dz2 @ W2.T\n", "da1" ] }, { "cell_type": "code", "execution_count": 51, "id": "5874a782-205a-4d6a-ad01-a97316bc049e", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[-9.49892516e-03, 7.35097981e-03, 4.59581061e-03,\n", " 3.97953501e-03],\n", " [ 1.42958966e-04, -4.28259398e-03, -3.07761916e-03,\n", " -8.78734555e-03],\n", " [ 1.26651581e-02, -2.19809458e-03, -9.99359631e-03,\n", " -1.48915174e-05],\n", " [-6.63763154e-03, 5.91403977e-05, 8.37281982e-03,\n", " 5.24705414e-03]])" ] }, "execution_count": 51, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# dZ1 = dA1 ⊙ σ'(A1) (逐元素乘)\n", "# dsigmoid(a) = a * (1 - a) 就是 σ'(Z1) 的\"以 a 表达\"形式\n", "dz1 = da1 * dsigmoid(a1)\n", "dz1" ] }, { "cell_type": "code", "execution_count": 52, "id": "786c37c8-1f3f-4fff-bc22-1de524cc2073", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([[ 0.00150688, -0.00053474, -0.00040519, 0.00130804],\n", " [-0.00162367, -0.00105586, 0.0013238 , -0.00088507]])" ] }, "execution_count": 52, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# dW1 = (1/N) · X.T @ dZ1\n", "# (2,4).T @ (4,4) -> (2,4)，与 W1 同形\n", "dW1 = (X.T @ dz1) / N\n", "dW1" ] }, { "cell_type": "code", "execution_count": 53, "id": "b92593d9-b701-42a8-b3c4-d2bd586ae620", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "array([-8.32109898e-04, 2.32357913e-04, -2.56462589e-05, 1.06088022e-04])" ] }, "execution_count": 53, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# db1 = dZ1 在样本维 (axis=0) 上的平均，形状 (4,)，与 b1 同形\n", "db1 = dz1.mean(axis=0)\n", "db1" ] }, { "cell_type": "markdown", "id": "df40851f-fcb9-4a75-b0fc-8ba217b0ff79", "metadata": {}, "source": [ "## 梯度下降\n", "\n", "有了梯度之后，沿着**损失下降最快的反方向**走一小步：\n", "\n", "$$\n", "W \\;\\leftarrow\\; W - \\eta\\, \\frac{\\partial L}{\\partial W},\n", "\\qquad\n", "b \\;\\leftarrow\\; b - \\eta\\, \\frac{\\partial L}{\\partial b}\n", "$$\n", "\n", "其中 $\\eta$ 就是学习率 `lr`。 \n", "直观理解：\n", "\n", "- $\\partial L/\\partial W$ 告诉我们在每个参数方向上\"山坡有多陡\"；\n", "- 减去 $\\eta$ 倍的坡度，就是\"下山一步\"；\n", "- 反复迭代就能逐步逼近局部最优点（对凸问题就是全局最优点）。\n", "\n", "> 提示：这里的\"梯度\"是**平均梯度**（已经除以 $N$），所以学习率的选择 \n", "> 与 batch size 关系不大，便于不同数据量下复用同样的 `lr`。\n" ] }, { "cell_type": "code", "execution_count": null, "id": "fcdcbc8a-2736-459e-967d-e3f8be911839", "metadata": {}, "outputs": [], "source": [ "# 沿负梯度方向更新参数\n", "W1 -= lr * dW1\n", "b1 -= lr * db1\n", "W2 -= lr * dW2\n", "b2 -= lr * db2\n", "W1, b1, W2, b2" ] }, { "cell_type": "code", "execution_count": null, "id": "2d4f35bf-9b8f-4e49-918c-fa960c490dae", "metadata": {}, "outputs": [], "source": [ "z1 = X @ W1 + b1\n", "a1 = sigmoid(z1)\n", "z2 = a1 @ W2 + b2\n", "a2 = sigmoid(z2).ravel()\n", "z1, a1, z2, a2" ] }, { "cell_type": "code", "execution_count": null, "id": "ca69abae-ba36-4eb5-a26d-271e586767f0", "metadata": {}, "outputs": [], "source": [ "loss = -np.mean(y * np.log(y_pred + 1e-8) +\n", " (1 - y) * np.log(1 - y_pred + 1e-8))\n", "loss" ] }, { "cell_type": "code", "execution_count": null, "id": "44b78f91-18e2-437b-95e4-aa3153ce5212", "metadata": {}, "outputs": [], "source": [ "for ep in range(1, 1000 + 1):\n", " # ---- 前向 ----\n", " z1 = X @ W1 + b1 # (4,4)\n", " a1 = sigmoid(z1) # (4,4)\n", " z2 = a1 @ W2 + b2 # (4,1)\n", " a2 = sigmoid(z2).ravel() # (4,)\n", " y_pred = a2\n", " \n", " # ---- 计算损失 ----\n", " loss = -np.mean(y * np.log(y_pred + 1e-8) +\n", " (1 - y) * np.log(1 - y_pred + 1e-8))\n", " \n", " # 打印第 1 轮和每 100 轮的进度\n", " if ep % 100 == 0 or ep == 1:\n", " acc = (y_pred.round() == y).mean()\n", " print(f\"epoch {ep:5d} loss={loss:.4f} acc={acc:.2f}\")\n", " \n", " # ---- 反向 ----\n", " N = len(X)\n", " y_pred = y_pred.reshape(-1, 1)\n", "\n", " dz2 = (y_pred - y.reshape(-1, 1)) # 交叉熵+sigmoid 的极简结论\n", " dW2 = (a1.T @ dz2) / N\n", " db2 = dz2.mean(axis=0)\n", "\n", " da1 = dz2 @ W2.T\n", " dz1 = da1 * dsigmoid(a1)\n", " dW1 = (X.T @ dz1) / N\n", " db1 = dz1.mean(axis=0)\n", "\n", " # ---- 参数更新 (梯度下降) ----\n", " W1 -= lr * dW1\n", " b1 -= lr * db1\n", " W2 -= lr * dW2\n", " b2 -= lr * db2 " ] }, { "cell_type": "markdown", "id": "f3bb886c", "metadata": {}, "source": [ "## 训练循环\n", "\n", "把前向 + 反向 + 更新打包成一个循环，跑若干个 **epoch**（一轮 = 把全部样本看一遍）。 \n", "由于这里总共只有 4 个样本，相当于每次迭代都是 full-batch 梯度下降。" ] }, { "cell_type": "code", "execution_count": null, "id": "79f627d6-6f89-4abd-bf76-664fe449fb9c", "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "qlib", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.9" } }, "nbformat": 4, "nbformat_minor": 5 }