Neural Ordinary Differential Equations

1 Neural Ordinary Differential EquationsRicky T. Q. Chen*, Yulia Rubanova*, Jesse Bettencourt*, David DuvenaudUniversity of TorontoBackground: Ordinary Differential Equations (ODEs)-Model the instantaneous change of a state.(explicit form)-Solving an initial value problem (IVP) corresponds to integration.(solution is a trajectory)-Euler method approximates with small steps:Residual Networks interpreted as an ODE Solver-Hidden units look like:-Final output is the composition:Haber & Ruthotto (2017). E (2017). Residual Networks interpreted as an ODE Solver-Hidden units look like:-Final output is the composition:-This can be interpreted as an Euler discretization of an & Ruthotto (2017). E (2017). -In the limit of smaller steps:Deep Learning as Discretized Differential EquationsMany deep learning networks can be interpreted as ODE Numerical SchemeResNet, RevNet, ResNeXt, EulerPolyNetApproximation to Backward EulerFractalNetRunge-KuttaDenseNetRunge- KuttaLu et al.

2 (2017) Chang et al. (2018)Zhu et al. (2018)Deep Learning as Discretized Differential EquationsMany deep learning networks can be interpreted as ODE Numerical SchemeResNet, RevNet, ResNeXt, EulerPolyNetApproximation to Backward EulerFractalNetRunge-KuttaDenseNetRunge- KuttaLu et al. (2017) Chang et al. (2018)Zhu et al. (2018)But:(1)What is the underlying dynamics?(2)Adaptive-step size solvers provide better error handling. Neural Ordinary Differential EquationsInstead of y = F(x),Parameterize Neural Ordinary Differential EquationsInstead of y = F(x), solve y = z(T) given the initial condition z(0) = Neural Ordinary Differential EquationsSolve the dynamic using any black-box ODE step (1) memory of y = F(x), solve y = z(T) given the initial condition z(0) = without knowledge of the ODE SolverUltimately want to optimize some lossBackprop without knowledge of the ODE SolverUltimately want to optimize some lossNaive approach: Know the solver.

3 Backprop through the of implicit solvers perform inner without knowledge of the ODE SolverUltimately want to optimize some lossNaive approach: Know the solver. Backprop through the of implicit solvers perform inner approach: Adjoint sensitivity analysis. (Reverse-mode Autodiff.)-Pontryagin (1962).+Automatic differentiation.+O(1) memory in backward BackpropagationResidual :Backward:Params: Define:Adjoint BackpropagationResidual :Backward:Params: Adjoint :Define:Continuous-time BackpropagationResidual :Backward:Params: Adjoint :Backward:Adjoint DiffEqAdjoint StateDefine:Continuous-time BackpropagationResidual :Backward:Params: Adjoint :Backward:Params:Adjoint DiffEqAdjoint StateDefine:A Differentiable Primitive for AutoDiffForward:Backward:A Differentiable Primitive for AutoDiffForward:Backward:A Differentiable Primitive for AutoDiffReversible networks (Gomez et al.)

4 2018) also only require O(1)-memory, but require very specific Neural network architectures with partitioned t need to store layer activations for reverse pass - just follow dynamics in reverse!Reverse versus Forward Cost-Empirically, reverse pass roughly half as expensive as forward to instance evaluations can be viewed as number of layers in Neural = Number of Function Become Increasingly Complex-Dynamics become more demanding to compute during computation time according to complexity of contrast, Chang et al. (ICLR 2018) explicitly add layers during RNNs for Time Series Modeling-We often want arbitrary measurement times, ie. irregular time do VAE-style inference with a latent vs Recurrent Neural Networks (RNNs)-RNNs learn very stiff dynamics, have exploding gradients.

5 --Whereas ODEs are guaranteed to be Normalizing FlowsInstantaneous Change of variables (iCOV):-For a Lipschitz continuous function Continuous Normalizing FlowsInstantaneous Change of variables (iCOV):-For a Lipschitz continuous function -In other words,Continuous Normalizing FlowsInstantaneous Change of variables (iCOV):-For a Lipschitz continuous function -In other words,With an invertible F:Continuous Normalizing Flows1D:2D:DataDiscrete-NFCNFIs the ODE being correctly solved? stochastic Unbiased Log DensityStochastic Unbiased Log DensityCan further reduce time complexity using stochastic et al. (2019)FFJORD - stochastic Continuous FlowsGrathwohl et al. (2019)MNIST - Model SamplesCIFAR10 - Model SamplesVariational Autoencoders with FFJORD ODE Solving as a Modeling PrimitiveAdaptive-step solvers with O(1) memory directions we re currently working on:-Latent stochastic Differential architectures suited for of dynamics to require fewer !

6 Yulia Rubanova Jesse Bettencourt David DuvenaudCo-authors:Extra SlidesLatent Space Visualizations Released an implementation of reverse-mode autodiff through black-box ODE solvers. Solves a system of size 2D + K + 1. In contrast, forward-mode implementation solves a system of size D^2 + KD. Tensorflow has Dormand-Prince-Shampine Runge-Kutta 5(4) implemented, but uses naive autodiff for much precision is needed?Explicit Error Control-More fine-grained control than low-precision scales with instance = Number of Function Depends on Complexity of Dynamics-Time cost is dominated by evaluation of dynamics = Number of Function not use an ODE solver as modeling primitive?-Solving an ODE is Directions- stochastic Differential Equations and Random ODEs.

7 Approximates stochastic gradient up ODE solvers with machine Differential , physics, simulations.