Backpropagation

Detailed Description

Backpropagation computes the gradient for a fixed input–output pair ( x i , y i ) {\displaystyle (x_{i},y_{i})}, where the weights w j k l {\displaystyle w_{jk}^{l}} can vary. Each individual component of the gradient, ∂ C / ∂ w j k l , {\displaystyle \partial C/\partial w_{jk}^{l},} can be computed by the chain rule; but doing this separately for each weight is inefficient. Backpropagation efficiently computes the gradient by avoiding duplicate calculations and not computing unnecessary intermediate values, by computing the gradient of each layer – specifically the gradient of the weighted input of each layer, denoted by δ l {\displaystyle \delta ^{l}} – from back to front.

Informally, the key point is that since the only way a weight in W l {\displaystyle W^{l}} affects the loss is through its effect on the next layer, and it does so linearly, δ l {\displaystyle \delta ^{l}} are the only data you need to compute the gradients of the weights at layer l {\displaystyle l}, and then the gradients of weights of previous layer can be computed by δ l − 1 {\displaystyle \delta ^{l-1}} and repeated recursively. This avoids inefficiency in two ways. First, it avoids duplication because when computing the gradient at layer l {\displaystyle l}, it is unnecessary to recompute all derivatives on later layers l + 1 , l + 2 , … {\displaystyle l+1,l+2,\ldots } each time. Second, it avoids unnecessary intermediate calculations, because at each stage it directly computes the gradient of the weights with respect to the ultimate output (the loss), rather than unnecessarily computing the derivatives of the values of hidden layers with respect to changes in weights ∂ a j ′ l ′ / ∂ w j k l {\displaystyle \partial a_{j’}^{l’}/\partial w_{jk}^{l}}.

About the Intelladex Team

Intelladex is a non-partisan project dedicated to meticulously indexing the world’s Artificial Intelligence tools. Our mission is to foster transparency, innovation, and understanding in the AI ecosystem. With a passion for demystifying complex software, this site is built on a foundation of hands-on, independent research. Every review, guide, and article is crafted to provide you with clear, actionable insights.