Why back propagation goes backward

This explains why gradient computation for a neural network is done with a backward pass rather than a forward one. It frames a network as a directed computational graph and shows that each weight’s gradient can be decomposed locally at its node using the chain rule: ∂f/∂θ = (∂f/∂v)(∂v/∂u)(∂u/∂θ). Local terms like ∂v/∂u and ∂u/∂θ are easy to compute (activation derivative and input value), but the upstream sensitivity ∂f/∂v depends on downstream nodes. The multivariable chain rule expresses ∂f/∂v as a sum of contributions from its children, so knowing those downstream derivatives is necessary to compute the local gradient.

A naive forward scheme would pass partial products forward and recompute the same downstream sums many times, causing repeated terms and quadratic blowup when multiple weights share upstream nodes. Instead, passing each node’s local derivative messages backward lets each upstream node sum its children’s contributions once, yielding the correct ∂f/∂v = Σj (∂f/∂wj)(∂wj/∂v) and allowing gradients for all weights to be obtained in time linear in the graph size. Framing backprop as backward message-passing clarifies it as an efficient solution to the credit-assignment problem in compositional functions.