Neural Ordinary Differential Equations

1 Neural Ordinary Differential EquationsRicky T. Q. Chen*, Yulia Rubanova*, Jesse Bettencourt*, David DuvenaudUniversity of TorontoBackground: Ordinary Differential Equations (ODEs)-Model the instantaneous change of a state.(explicit form)-Solving an initial value problem (IVP) corresponds to integration.(solution is a trajectory)-Euler method approximates with small steps:Residual Networks interpreted as an ODE Solver-Hidden units look like:-Final output is the composition:Haber & Ruthotto (2017). E (2017). Residual Networks interpreted as an ODE Solver-Hidden units look like:-Final output is the composition:-This can be interpreted as an Euler discretization of an & Ruthotto (2017).

2 E (2017). -In the limit of smaller steps:Deep Learning as Discretized Differential EquationsMany deep learning networks can be interpreted as ODE Numerical SchemeResNet, RevNet, ResNeXt, EulerPolyNetApproximation to Backward EulerFractalNetRunge-KuttaDenseNetRunge- KuttaLu et al. (2017) Chang et al. (2018)Zhu et al. (2018)Deep Learning as Discretized Differential EquationsMany deep learning networks can be interpreted as ODE Numerical SchemeResNet, RevNet, ResNeXt, EulerPolyNetApproximation to Backward EulerFractalNetRunge-KuttaDenseNetRunge- KuttaLu et al. (2017) Chang et al. (2018)Zhu et al. (2018)But:(1)What is the underlying dynamics?(2)Adaptive-step size solvers provide better error handling.

3 Neural Ordinary Differential EquationsInstead of y = F(x),Parameterize Neural Ordinary Differential EquationsInstead of y = F(x), solve y = z(T) given the initial condition z(0) = Neural Ordinary Differential EquationsSolve the dynamic using any black-box ODE step (1) memory of y = F(x), solve y = z(T) given the initial condition z(0) = without knowledge of the ODE SolverUltimately want to optimize some lossBackprop without knowledge of the ODE SolverUltimately want to optimize some lossNaive approach: Know the solver. Backprop through the of implicit solvers perform inner without knowledge of the ODE SolverUltimately want to optimize some lossNaive approach: Know the solver.

4 Backprop through the of implicit solvers perform inner approach: Adjoint sensitivity analysis. (Reverse-mode Autodiff.)-Pontryagin (1962).+Automatic differentiation.+O(1) memory in backward BackpropagationResidual :Backward:Params: Define:Adjoint BackpropagationResidual :Backward:Params: Adjoint :Define:Continuous-time BackpropagationResidual :Backward:Params: Adjoint :Backward:Adjoint DiffEqAdjoint StateDefine:Continuous-time BackpropagationResidual :Backward:Params: Adjoint :Backward:Params:Adjoint DiffEqAdjoint StateDefine:A Differentiable Primitive for AutoDiffForward:Backward:A Differentiable Primitive for AutoDiffForward:Backward:A Differentiable Primitive for AutoDiffReversible networks (Gomez et al.)

5 2018) also only require O(1)-memory, but require very specific Neural network architectures with partitioned t need to store layer activations for reverse pass - just follow dynamics in reverse!Reverse versus Forward Cost-Empirically, reverse pass roughly half as expensive as forward to instance evaluations can be viewed as number of layers in Neural = Number of Function Become Increasingly Complex-Dynamics become more demanding to compute during computation time according to complexity of contrast, Chang et al. (ICLR 2018) explicitly add layers during RNNs for Time Series Modeling-We often want arbitrary measurement times, ie. irregular time do VAE-style inference with a latent vs Recurrent Neural Networks (RNNs)-RNNs learn very stiff dynamics, have exploding gradients.

6 --Whereas ODEs are guaranteed to be Normalizing FlowsInstantaneous Change of variables (iCOV):-For a Lipschitz continuous function Continuous Normalizing FlowsInstantaneous Change of variables (iCOV):-For a Lipschitz continuous function -In other words,Continuous Normalizing FlowsInstantaneous Change of variables (iCOV):-For a Lipschitz continuous function -In other words,With an invertible F:Continuous Normalizing Flows1D:2D:DataDiscrete-NFCNFIs the ODE being correctly solved?Stochastic Unbiased Log DensityStochastic Unbiased Log DensityCan further reduce time complexity using stochastic et al. (2019)FFJORD - Stochastic Continuous FlowsGrathwohl et al. (2019)MNIST - Model SamplesCIFAR10 - Model SamplesVariational Autoencoders with FFJORD ODE Solving as a Modeling PrimitiveAdaptive-step solvers with O(1) memory directions we re currently working on:-Latent Stochastic Differential architectures suited for of dynamics to require fewer !

7 Yulia Rubanova Jesse Bettencourt David DuvenaudCo-authors:Extra SlidesLatent Space Visualizations Released an implementation of reverse-mode autodiff through black-box ODE solvers. Solves a system of size 2D + K + 1. In contrast, forward-mode implementation solves a system of size D^2 + KD. Tensorflow has Dormand-Prince-Shampine Runge-Kutta 5(4) implemented, but uses naive autodiff for much precision is needed?Explicit Error Control-More fine-grained control than low-precision scales with instance = Number of Function Depends on Complexity of Dynamics-Time cost is dominated by evaluation of dynamics = Number of Function not use an ODE solver as modeling primitive?

8 -Solving an ODE is Directions-Stochastic Differential Equations and Random ODEs. Approximates stochastic gradient up ODE solvers with machine Differential , physics, simulations.