神经网络学习笔记 - 损失函数的定义和微分证明

\[L_t(y_t, \hat{y_t}) = - y_t \log \hat{y_t} \\L(y, \hat{y}) = - \sum_{t} y_t \log \hat{y_t} \\\frac{ \partial L_t } { \partial z_t } = \hat{y_t} - y_t \\\text{where} \\z_t = s_tV \\\hat{y_t} = softmax(z_t) \\y_t \text{ : for training data x, the expected result y at time t. which are from training data}\]

\[\begin{align}\frac{ \partial L_t } { \partial z_t } & = \frac{ \partial \left ( - \sum_{k} y_k \log \hat{y_k} \right ) } { \partial z_t } \\ & = - \sum_{k} y_k \frac{ \partial \log \hat{y_k} } { \partial z_t } \\ & = - \sum_{k} y_k \frac {1} {\hat{y_k}} \cdot \frac{ \partial \hat{y_k} } { \partial z_t } \\ & = - \left ( y_t \frac {1} {\hat{y_t}} \cdot \frac{ \partial \hat{y_t} } { \partial z_t } \right ) - \left ( \sum_{k \ne t} y_k \frac {1} {\hat{y_k}} \cdot \frac{ \partial \hat{y_k} } { \partial z_t } \right ) \\ & \because \text{softmax differentiation formula } \\ & = - \left ( y_t \frac {1} {\hat{y_t}} \cdot ( 1 - \hat{y_t} ) \hat{y_t} \right ) - \left ( \sum_{k \ne t} y_k \frac {1} {\hat{y_k}} \cdot (-\hat{y_t} \hat{y_k}) \right ) \\ & = - \left ( y_t \cdot ( 1 - \hat{y_t} ) \right ) - \left ( \sum_{k \ne t} y_k \cdot (-\hat{y_t}) \right ) \\ & = - y_t + y_t \hat{y_t} + \left ( \sum_{k \ne t} y_k \hat{y_t} \right ) \\ & = - y_t + \hat{y_t} \left ( \sum_{k} y_k \right ) \\ & \because \sum_{k} y_k = 1 \\ & = \hat{y_t} - y_t\end{align}\]