We characterize neural networks as approximators of functions and dynamic systems. Neural approximations, leading to nonlinear minimization in highly dimensional spaces, require effective gradient calculation typically realized by gradient backpropagation. We discuss the use of gradient backpropagation for static and for dynamic systems. We also show the essential difference between the common chain rule and backpropagation, which is rarely acknowledged.

PACS numbers: 84.35.+i