# neural_ordinary_differential_equations__5fb2c83e.pdf

Neural Ordinary Differential Equations

Ricky T. Q. Chen*, Yulia Rubanova*, Jesse Bettencourt*, David Duvenaud University of Toronto, Vector Institute

We introduce a new family of deep neural network models. Instead of specifying a discrete sequence of hidden layers, we parameterize the derivative of the hidden state using a neural network. The output of the network is computed using a blackbox differential equation solver. These continuous-depth models have constant memory cost, adapt their evaluation strategy to each input, and can explicitly trade numerical precision for speed. We demonstrate these properties in continuous-depth residual networks and continuous-time latent variable models. We also construct continuous normalizing ﬂows, a generative model that can train by maximum likelihood, without partitioning or ordering the data dimensions. For training, we show how to scalably backpropagate through any ODE solver, without access to its internal operations. This allows end-to-end training of ODEs within larger models.

1 Introduction Residual Network ODE Network

Figure 1: Left: A Residual network deﬁnes a discrete sequence of ﬁnite transformations. Right: A ODE network deﬁnes a vector ﬁeld, which continuously transforms the state. Both: Circles represent evaluation locations.

Models such as residual networks, recurrent neural network decoders, and normalizing ﬂows build complicated transformations by composing a sequence of transformations to a hidden state:

ht+1 = ht + f(ht, θt) (1)

where t {0 . . . T} and ht RD. These iterative updates can be seen as an Euler discretization of a continuous transformation (Lu et al., 2017; Haber and Ruthotto, 2017; Ruthotto and Haber, 2018).

What happens as we add more layers and take smaller steps? In the limit, we parameterize the continuous dynamics of hidden units using an ordinary differential equation (ODE) speciﬁed by a neural network:

dt = f(h(t), t, θ) (2)

Starting from the input layer h(0), we can deﬁne the output layer h(T) to be the solution to this ODE initial value problem at some time T. This value can be computed by a black-box differential equation solver, which evaluates the hidden unit dynamics f wherever necessary to determine the solution with the desired accuracy. Figure 1 contrasts these two approaches.

Deﬁning and evaluating models using ODE solvers has several beneﬁts:

Memory efﬁciency In Section 2, we show how to compute gradients of a scalar-valued loss with respect to all inputs of any ODE solver, without backpropagating through the operations of the solver. Not storing any intermediate quantities of the forward pass allows us to train our models with constant memory cost as a function of depth, a major bottleneck of training deep models.

32nd Conference on Neural Information Processing Systems (Neur IPS 2018), Montréal, Canada.

Adaptive computation Euler s method is perhaps the simplest method for solving ODEs. There have since been more than 120 years of development of efﬁcient and accurate ODE solvers (Runge, 1895; Kutta, 1901; Hairer et al., 1987). Modern ODE solvers provide guarantees about the growth of approximation error, monitor the level of error, and adapt their evaluation strategy on the ﬂy to achieve the requested level of accuracy. This allows the cost of evaluating a model to scale with problem complexity. After training, accuracy can be reduced for real-time or low-power applications.

Parameter efﬁciency When the hidden unit dynamics are parameterized as a continuous function of time, the parameters of nearby layers are automatically tied together. In Section 3, we show that this reduces the number of parameters required on a supervised learning task.

Scalable and invertible normalizing ﬂows An unexpected side-beneﬁt of continuous transformations is that the change of variables formula becomes easier to compute. In Section 4, we derive this result and use it to construct a new class of invertible density models that avoids the single-unit bottleneck of normalizing ﬂows, and can be trained directly by maximum likelihood.

Continuous time-series models Unlike recurrent neural networks, which require discretizing observation and emission intervals, continuously-deﬁned dynamics can naturally incorporate data which arrives at arbitrary times. In Section 5, we construct and demonstrate such a model.

2 Reverse-mode automatic differentiation of ODE solutions

The main technical difﬁculty in training continuous-depth networks is performing reverse-mode differentiation (also known as backpropagation) through the ODE solver. Differentiating through the operations of the forward pass is straightforward, but incurs a high memory cost and introduces additional numerical error.

We treat the ODE solver as a black box, and compute gradients using the adjoint sensitivity method (Pontryagin et al., 1962). This approach computes gradients by solving a second, augmented ODE backwards in time, and is applicable to all ODE solvers. This approach scales linearly with problem size, has low memory cost, and explicitly controls numerical error.

Consider optimizing a scalar-valued loss function L(), whose input is the result of an ODE solver:

L(z(t1)) = L z(t0) + t1

t0 f(z(t), t, θ)dt = L (ODESolve(z(t0), f, t0, t1, θ)) (3)

Figure 2: Reverse-mode differentiation of an ODE solution. The adjoint sensitivity method solves an augmented ODE backwards in time. The augmented system contains both the original state and the sensitivity of the loss with respect to the state. If the loss depends directly on the state at multiple observation times, the adjoint state must be updated in the direction of the partial derivative of the loss with respect to each observation.

To optimize L, we require gradients with respect to θ. The ﬁrst step is to determining how the gradient of the loss depends on the hidden state z(t) at each instant. This quantity is called the adjoint a(t) = L/ z(t). Its dynamics are given by another ODE, which can be thought of as the instantaneous analog of the chain rule: da(t)

dt = a(t)T f(z(t), t, θ)

We can compute L/ z(t0) by another call to an ODE solver. This solver must run backwards, starting from the initial value of L/ z(t1). One complication is that solving this ODE requires the knowing value of z(t) along its entire trajectory. However, we can simply recompute z(t) backwards in time together with the adjoint, starting from its ﬁnal value z(t1).

Computing the gradients with respect to the parameters θ requires evaluating a third integral, which depends on both z(t) and a(t): d L

t1 a(t)T f(z(t), t, θ)

The vector-Jacobian products a(t)T f

z and a(t)T f

θ in (4) and (5) can be efﬁciently evaluated by automatic differentiation, at a time cost similar to that of evaluating f. All integrals for solving z, a and L

θ can be computed in a single call to an ODE solver, which concatenates the original state, the adjoint, and the other partial derivatives into a single vector. Algorithm 1 shows how to construct the necessary dynamics, and call an ODE solver to compute all gradients at once.

Algorithm 1 Reverse-mode derivative of an ODE initial value problem

Input: dynamics parameters θ, start time t0, stop time t1, ﬁnal state z(t1), loss gradient L/ z(t1) s0 = [z(t1), L z(t1), 0|θ|] Deﬁne initial augmented state def aug_dynamics([z(t), a(t), ], t, θ): Deﬁne dynamics on augmented state return [f(z(t), t, θ), a(t)T f

z , a(t)T f

θ ] Compute vector-Jacobian products [z(t0), L z(t0), L

θ ] = ODESolve(s0, aug_dynamics, t1, t0, θ) Solve reverse-time ODE return L z(t0), L

θ Return gradients

Most ODE solvers have the option to output the state z(t) at multiple times. When the loss depends on these intermediate states, the reverse-mode derivative must be broken into a sequence of separate solves, one between each consecutive pair of output times (Figure 2). At each observation, the adjoint must be adjusted in the direction of the corresponding partial derivative L/ z(ti).

The results above extend those of Stapor et al. (2018, section 2.4.2). An extended version of Algorithm 1 including derivatives w.r.t. t0 and t1 can be found in Appendix C. Detailed derivations are provided in Appendix B. Appendix D provides Python code which computes all derivatives for by extending the automatic differentiation package. This code also supports all higher-order derivatives. We have since released a Py Torch (Paszke et al., 2017) implementation, including GPU-based implementations of several standard ODE solvers at .

3 Replacing residual networks with ODEs for supervised learning

In this section, we experimentally investigate the training of neural ODEs for supervised learning.

Software To solve ODE initial value problems numerically, we use the implicit Adams method implemented in LSODE and VODE and interfaced through the package. Being an implicit method, it has better guarantees than explicit methods such as Runge-Kutta but requires solving a nonlinear optimization problem at every step. This setup makes direct backpropagation through the integrator difﬁcult. We implement the adjoint sensitivity method in Python s framework (Maclaurin et al., 2015). For the experiments in this section, we evaluated the hidden state dynamics and their derivatives on the GPU using Tensorﬂow, which were then called from the Fortran ODE solvers, which were called from Python code.

Table 1: Performance on MNIST. From Le Cun et al. (1998).

Test Error # Params Memory Time

1-Layer MLP 1.60% 0.24 M - - Res Net 0.41% 0.60 M O(L) O(L) RK-Net 0.47% 0.22 M O( L) O( L) ODE-Net 0.42% 0.22 M O(1) O( L)

Model Architectures We experiment with a small residual network which downsamples the input twice then applies 6 standard residual blocks He et al. (2016b), which are replaced by an ODESolve module in the ODE-Net variant. We also test a network with the same architecture but where gradients are backpropagated directly through a Runge-Kutta integrator, referred to as RK-Net. Table 1 shows test error, number of parameters, and memory cost. L denotes the number of layers in the Res Net, and L is the number of function evaluations that the ODE solver requests in a single forward pass, which can be interpreted as an implicit number of layers.

We ﬁnd that ODE-Nets and RK-Nets can achieve around the same performance as the Res Net, while using fewer parameters. For reference, a neural net with a single hidden layer of 300 units has around the same number of parameters as the ODE-Net and RK-Net architecture that we tested.

Error Control in ODE-Nets ODE solvers can approximately ensure that the output is within a given tolerance of the true solution. Changing this tolerance changes the behavior of the network. We ﬁrst verify that error can indeed be controlled in Figure 3a. The time spent by the forward call is proportional to the number of function evaluations (Figure 3b), so tuning the tolerance gives us a trade-off between accuracy and computational cost. One could train with high accuracy, but switch to a lower accuracy at test time.

Figure 3: Statistics of a trained ODE-Net. (NFE = number of function evaluations.)

Figure 3c) shows a surprising result: the number of evaluations in the backward pass is roughly half of the forward pass. This suggests that the adjoint sensitivity method is not only more memory efﬁcient, but also more computationally efﬁcient than directly backpropagating through the integrator, because the latter approach will need to backprop through each function evaluation in the forward pass.

Network Depth It s not clear how to deﬁne the depth of an ODE solution. A related quantity is the number of evaluations of the hidden state dynamics required, a detail delegated to the ODE solver and dependent on the initial state or input. Figure 3d shows that he number of function evaluations increases throughout training, presumably adapting to increasing complexity of the model.

4 Continuous Normalizing Flows

The discretized equation (1) also appears in normalizing ﬂows (Rezende and Mohamed, 2015) and the NICE framework (Dinh et al., 2014). These methods use the change of variables theorem to compute exact changes in probability if samples are transformed through a bijective function f:

z1 = f(z0) = log p(z1) = log p(z0) log det f

An example is the planar normalizing ﬂow (Rezende and Mohamed, 2015):

z(t + 1) = z(t) + uh(w Tz(t) + b), log p(z(t + 1)) = log p(z(t)) log 1 + u T h

Generally, the main bottleneck to using the change of variables formula is computing of the determinant of the Jacobian f/ z, which has a cubic cost in either the dimension of z, or the number of hidden units. Recent work explores the tradeoff between the expressiveness of normalizing ﬂow layers and computational cost (Kingma et al., 2016; Tomczak and Welling, 2016; Berg et al., 2018).

Surprisingly, moving from a discrete set of layers to a continuous transformation simpliﬁes the computation of the change in normalizing constant: Theorem 1 (Instantaneous Change of Variables). Let z(t) be a ﬁnite continuous random variable with probability p(z(t)) dependent on time. Let dz

dt = f(z(t), t) be a differential equation describing a continuous-in-time transformation of z(t). Assuming that f is uniformly Lipschitz continuous in z and continuous in t, then the change in log probability also follows a differential equation,

log p(z(t))

Proof in Appendix A. Instead of the log determinant in (6), we now only require a trace operation. Also unlike standard ﬁnite ﬂows, the differential equation f does not need to be bijective, since if uniqueness is satisﬁed, then the entire transformation is automatically bijective.

As an example application of the instantaneous change of variables, we can examine the continuous analog of the planar ﬂow, and its change in normalization constant:

dt = uh(w Tz(t) + b), log p(z(t))

Given an initial distribution p(z(0)), we can sample from p(z(t)) and evaluate its density by solving this combined ODE.

Using multiple hidden units with linear cost While det is not a linear function, the trace function is, which implies tr(

n tr(Jn). Thus if our dynamics is given by a sum of functions then the differential equation for the log density is also a sum:

n=1 fn(z(t)), d log p(z(t))

This means we can cheaply evaluate ﬂow models having many hidden units, with a cost only linear in the number of hidden units M. Evaluating such wide ﬂow layers using standard normalizing ﬂows costs O(M 3), meaning that standard NF architectures use many layers of only a single hidden unit.

Time-dependent dynamics We can specify the parameters of a ﬂow as a function of t, making the differential equation f(z(t), t) change with t. This is parameterization is a kind of hypernetwork (Ha et al., 2016). We also introduce a gating mechanism for each hidden unit, dz

n σn(t)fn(z) where σn(t) (0, 1) is a neural network that learns when the dynamic fn(z) should be applied. We call these models continuous normalizing ﬂows (CNF).

4.1 Experiments with Continuous Normalizing Flows

We ﬁrst compare continuous and discrete planar ﬂows at learning to sample from a known distribution. We show that a planar CNF with M hidden units can be at least as expressive as a planar NF with K = M layers, and sometimes much more expressive.

Density matching We conﬁgure the CNF as described above, and train for 10,000 iterations using Adam (Kingma and Ba, 2014). In contrast, the NF is trained for 500,000 iterations using RMSprop (Hinton et al., 2012), as suggested by Rezende and Mohamed (2015). For this task, we minimize KL (q(x) p(x)) as the loss function where q is the ﬂow model and the target density p( ) can be evaluated. Figure 4 shows that CNF generally achieves lower loss.

Maximum Likelihood Training A useful property of continuous-time normalizing ﬂows is that we can compute the reverse transformation for about the same cost as the forward pass, which cannot be said for normalizing ﬂows. This lets us train the ﬂow on a density estimation task by performing

K=2 K=8 K=32 M=2 M=8 M=32

(d) Loss vs. K/M

Figure 4: Comparison of normalizing ﬂows versus continuous normalizing ﬂows. The model capacity of normalizing ﬂows is determined by their depth (K), while continuous normalizing ﬂows can also increase capacity by increasing width (M), making them easier to train.

(a) Two Circles

(b) Two Moons

Figure 5: Visualizing the transformation from noise to data. Continuous-time normalizing ﬂows are reversible, so we can train on a density estimation task and still be able to sample from the learned density efﬁciently.

maximum likelihood estimation, which maximizes Ep(x)[log q(x)] where q( ) is computed using the appropriate change of variables theorem, then afterwards reverse the CNF to generate random samples from q(x).

For this task, we use 64 hidden units for CNF, and 64 stacked one-hidden-unit layers for NF. Figure 5 shows the learned dynamics. Instead of showing the initial Gaussian distribution, we display the transformed distribution after a small amount of time which shows the locations of the initial planar ﬂows. Interestingly, to ﬁt the Two Circles distribution, the CNF rotates the planar ﬂows so that the particles can be evenly spread into circles. While the CNF transformations are smooth and interpretable, we ﬁnd that NF transformations are very unintuitive and this model has difﬁculty ﬁtting the two moons dataset in Figure 5b.

5 A generative latent function time-series model

Applying neural networks to irregularly-sampled data such as medical records, network trafﬁc, or neural spiking data is difﬁcult. Typically, observations are put into bins of ﬁxed duration, and the latent dynamics are discretized in the same way. This leads to difﬁculties with missing data and illdeﬁned latent variables. Missing data can be addressed using generative time-series models (Álvarez and Lawrence, 2011; Futoma et al., 2017; Mei and Eisner, 2017; Soleimani et al., 2017a) or data imputation (Che et al., 2018). Another approach concatenates time-stamp information to the input of an RNN (Choi et al., 2016; Lipton et al., 2016; Du et al., 2016; Li, 2017).

We present a continuous-time, generative approach to modeling time series. Our model represents each time series by a latent trajectory. Each trajectory is determined from a local initial state, zt0, and a global set of latent dynamics shared across all time series. Given observation times t0, t1, . . . , t N and an initial state zt0, an ODE solver produces zt1, . . . , zt N , which describe the latent state at each observation.We deﬁne this generative model formally through a sampling procedure:

zt0 p(zt0) (11) zt1, zt2, . . . , zt N = ODESolve(zt0, f, θf, t0, . . . , t N) (12) each xti p(x|zti, θx) (13)

Function f is a time-invariant function that takes the value z at the current time step and outputs the gradient: z(t)/ t = f(z(t), θf). We parametrize this function using a neural net. Because f is timeinvariant, given any latent state z(t), the entire latent trajectory is uniquely deﬁned. Extrapolating this latent trajectory lets us make predictions arbitrarily far forwards or backwards in time.

Training and Prediction We can train this latent-variable model as a variational autoencoder (Kingma and Welling, 2014; Rezende et al., 2014), with sequence-valued observations. Our recognition net is an RNN, which consumes the data sequentially backwards in time, and outputs qφ(z0|x1, x2, . . . , x N). A detailed algorithm can be found in Appendix E. Using ODEs as a generative model allows us to make predictions for arbitrary time points t1...t M on a continuous timeline.

q(zt0|xt0...xt N) ht0 ht1 ht N

ODE Solve(zt0, f, f, t0, ..., t M)

zt N zt N+1

t0 t1 t N t N+1 t M ˆx(t)

Figure 6: Computation graph of the latent ODE model.

t Figure 7: Fitting a latent ODE dynamics model with a Poisson process likelihood. Dots show event times. The line is the learned intensity λ(t) of the Poisson process.

Poisson Process likelihoods The fact that an observation occurred often tells us something about the latent state. For example, a patient may be more likely to take a medical test if they are sick. The rate of events can be parameterized by a function of the latent state: p(event at time t| z(t)) = λ(z(t)). Given this rate function, the likelihood of a set of independent observation times in the interval [tstart, tend] is given by an inhomogeneous Poisson process (Palm, 1943):

log p(t1 . . . t N| tstart, tend) =

i=1 log λ(z(ti)) tend

tstart λ(z(t))dt

We can parameterize λ( ) using another neural network. Conveniently, we can evaluate both the latent trajectory and the Poisson process likelihood together in a single call to an ODE solver. Figure 7 shows the event rate learned by such a model on a toy dataset.

A Poisson process likelihood on observation times can be combined with a data likelihood to jointly model all observations and the times at which they were made.

5.1 Time-series Latent ODE Experiments

We investigate the ability of the latent ODE model to ﬁt and extrapolate time series. The recognition network is an RNN with 25 hidden units. We use a 4-dimensional latent space. We parameterize the dynamics function f with a one-hidden-layer network with 20 hidden units. The decoder computing p(xti|zti) is another neural network with one hidden layer with 20 hidden units. Our baseline was a recurrent neural net with 25 hidden units trained to minimize negative Gaussian log-likelihood. We trained a second version of this RNN whose inputs were concatenated with the time difference to the next observation to aid RNN with irregular observations.

Table 2: Predictive RMSE on test set

# Observations 30/100 50/100 100/100

RNN 0.3937 0.3202 0.1813 Latent ODE 0.1642 0.1502 0.1346

Bi-directional spiral dataset We generated a dataset of 1000 2-dimensional spirals, each starting at a different point, sampled at 100 equallyspaced timesteps. The dataset contains two types of spirals: half are clockwise while the other half counter-clockwise. To make the task more realistic, we add gaussian noise to the observations.

Time series with irregular time points To generate irregular timestamps, we randomly sample points from each trajectory without replacement (n = {30, 50, 100}). We report predictive rootmean-squared error (RMSE) on 100 time points extending beyond those that were used for training. Table 2 shows that the latent ODE has substantially lower predictive RMSE.

Figure 8 shows examples of spiral reconstructions with 30 sub-sampled points. Reconstructions from the latent ODE were obtained by sampling from the posterior over latent trajectories and decoding it

Figure 9: Data-space trajectories decoded from varying one dimension of zt0. Color indicates progression through time, starting at purple and ending at red. Note that the trajectories on the left are counter-clockwise, while the trajectories on the right are clockwise.

to data-space. Examples with varying number of time points are shown in Appendix F. We observed that reconstructions and extrapolations are consistent with the ground truth regardless of number of observed points and despite the noise.

(a) Recurrent Neural Network

(b) Latent Neural Ordinary Differential Equation

(c) Latent Trajectories

Figure 8: (a): Reconstruction and extrapolation of spirals with irregular time points by a recurrent neural network. (b): Reconstructions and extrapolations by a latent neural ODE. Blue curve shows model prediction. Red shows extrapolation. (c) A projection of inferred 4-dimensional latent ODE trajectories onto their ﬁrst two dimensions. Color indicates the direction of the corresponding trajectory. The model has learned latent dynamics which distinguishes the two directions.

Latent space interpolation Figure 8c shows latent trajectories projected onto the ﬁrst two dimensions of the latent space. The trajectories form two separate clusters of trajectories, one decoding to clockwise spirals, the other to counter-clockwise. Figure 9 shows that the latent trajectories change smoothly as a function of the initial point z(t0), switching from a clockwise to a counter-clockwise spiral.

6 Scope and Limitations

Minibatching The use of mini-batches is less straightforward than for standard neural networks. One can still batch together evaluations through the ODE solver by concatenating the states of each batch element together, creating a combined ODE with dimension D K. In some cases, controlling error on all batch elements together might require evaluating the combined system K times more often than if each system was solved individually. However, in practice the number of evaluations did not increase substantially when using minibatches.

Uniqueness When do continuous dynamics have a unique solution? Picard s existence theorem (Coddington and Levinson, 1955) states that the solution to an initial value problem exists and is unique if the differential equation is uniformly Lipschitz continuous in z and continuous in t. This theorem holds for our model if the neural network has ﬁnite weights and uses Lipshitz nonlinearities, such as or .

Setting tolerances Our framework allows the user to trade off speed for precision, but requires the user to choose an error tolerance on both the forward and reverse passes during training. For sequence modeling, the default value of was used. In the classiﬁcation and density estimation experiments, we were able to reduce the tolerance to and , respectively, without degrading performance.

Reconstructing forward trajectories Reconstructing the state trajectory by running the dynamics backwards can introduce extra numerical error if the reconstructed trajectory diverges from the original. This problem can be addressed by checkpointing: storing intermediate values of z on the

forward pass, and reconstructing the exact forward trajectory by re-integrating from those points. We did not ﬁnd this to be a practical problem, and we informally checked that reversing many layers of continuous normalizing ﬂows with default tolerances recovered the initial states.

7 Related Work

The use of the adjoint method for training continuous-time neural networks was previously proposed (Le Cun et al., 1988; Pearlmutter, 1995), though was not demonstrated practically. The interpretation of residual networks He et al. (2016a) as approximate ODE solvers spurred research into exploiting reversibility and approximate computation in Res Nets (Chang et al., 2017; Lu et al., 2017). We demonstrate these same properties in more generality by directly using an ODE solver.

Adaptive computation One can adapt computation time by training secondary neural networks to choose the number of evaluations of recurrent or residual networks (Graves, 2016; Jernite et al., 2016; Figurnov et al., 2017; Chang et al., 2018). However, this introduces overhead both at training and test time, and extra parameters that need to be ﬁt. In contrast, ODE solvers offer well-studied, computationally cheap, and generalizable rules for adapting the amount of computation.

Constant memory backprop through reversibility Recent work developed reversible versions of residual networks (Gomez et al., 2017; Haber and Ruthotto, 2017; Chang et al., 2017), which gives the same constant memory advantage as our approach. However, these methods require restricted architectures, which partition the hidden units. Our approach does not have these restrictions.

Learning differential equations Much recent work has proposed learning differential equations from data. One can train feed-forward or recurrent neural networks to approximate a differential equation (Raissi and Karniadakis, 2018; Raissi et al., 2018a; Long et al., 2017), with applications such as ﬂuid simulation (Wiewel et al., 2018). There is also signiﬁcant work on connecting Gaussian Processes (GPs) and ODE solvers (Schober et al., 2014). GPs have been adapted to ﬁt differential equations (Raissi et al., 2018b) and can naturally model continuous-time effects and interventions (Soleimani et al., 2017b; Schulam and Saria, 2017). Ryder et al. (2018) use stochastic variational inference to recover the solution of a given stochastic differential equation.

Differentiating through ODE solvers The library (Farrell et al., 2013) implements adjoint computation for general ODE and PDE solutions, but only by backpropagating through the individual operations of the forward solver. The Stan library (Carpenter et al., 2015) implements gradient estimation through ODE solutions using forward sensitivity analysis. However, forward sensitivity analysis is quadratic-time in the number of variables, whereas the adjoint sensitivity analysis is linear (Carpenter et al., 2015; Zhang and Sandu, 2014). Melicher et al. (2017) used the adjoint method to train bespoke latent dynamic models.

In contrast, by providing a generic vector-Jacobian product, we allow an ODE solver to be trained end-to-end with any other differentiable model components. While use of vector-Jacobian products for solving the adjoint method has been explored in optimal control (Andersson, 2013; Andersson et al., In Press, 2018), we highlight the potential of a general integration of black-box ODE solvers into automatic differentiation (Baydin et al., 2018) for deep learning and generative modeling.

8 Conclusion

We investigated the use of black-box ODE solvers as a model component, developing new models for time-series modeling, supervised learning, and density estimation. These models are evaluated adaptively, and allow explicit control of the tradeoff between computation speed and accuracy. Finally, we derived an instantaneous version of the change of variables formula, and developed continuous-time normalizing ﬂows, which can scale to large layer sizes.

9 Acknowledgements

We thank Wenyi Wang and Geoff Roeder for help with proofs, and Daniel Duckworth, Ethan Fetaya, Hossein Soleimani, Eldad Haber, Ken Caluwaerts, and Daniel Flam-Shepherd for feedback. We thank Chris Rackauckas, Dougal Maclaurin, and Matthew James Johnson for helpful discussions.

Mauricio A Álvarez and Neil D Lawrence. Computationally efﬁcient convolved multiple output Gaussian processes. Journal of Machine Learning Research, 12(May):1459 1500, 2011.

Brandon Amos and J Zico Kolter. Opt Net: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning, pages 136 145, 2017.

Joel Andersson. A general-purpose software framework for dynamic optimization. Ph D thesis, 2013.

Joel A E Andersson, Joris Gillis, Greg Horn, James B Rawlings, and Moritz Diehl. Cas ADi A software framework for nonlinear optimization and optimal control. Mathematical Programming Computation, In Press, 2018.

Atilim Gunes Baydin, Barak A Pearlmutter, Alexey Andreyevich Radul, and Jeffrey Mark Siskind. Automatic differentiation in machine learning: a survey. Journal of machine learning research, 18 (153):1 153, 2018.

Rianne van den Berg, Leonard Hasenclever, Jakub M Tomczak, and Max Welling. Sylvester normalizing ﬂows for variational inference. ar Xiv preprint ar Xiv:1803.05649, 2018.

Bob Carpenter, Matthew D Hoffman, Marcus Brubaker, Daniel Lee, Peter Li, and Michael Betancourt. The Stan math library: Reverse-mode automatic differentiation in c++. ar Xiv preprint ar Xiv:1509.07164, 2015.

Bo Chang, Lili Meng, Eldad Haber, Lars Ruthotto, David Begert, and Elliot Holtham. Reversible architectures for arbitrarily deep residual neural networks. ar Xiv preprint ar Xiv:1709.03698, 2017.

Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert. Multi-level residual networks from dynamical systems view. In International Conference on Learning Representations, 2018. URL .

Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientiﬁc Reports, 8(1):6085, 2018. URL ! ! " # ! $"$% #.

Edward Choi, Mohammad Taha Bahadori, Andy Schuetz, Walter F. Stewart, and Jimeng Sun. Doctor AI: Predicting clinical events via recurrent neural networks. In Proceedings of the 1st Machine Learning for Healthcare Conference, volume 56 of Proceedings of Machine Learning Research, pages 301 318. PMLR, 18 19 Aug 2016. URL & ' ( ' .

Earl A Coddington and Norman Levinson. Theory of ordinary differential equations. Tata Mc Graw Hill Education, 1955.

Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. ar Xiv preprint ar Xiv:1410.8516, 2014.

Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. Recurrent marked temporal point processes: Embedding event history to vector. In International Conference on Knowledge Discovery and Data Mining, pages 1555 1564. ACM, 2016.

Patrick Farrell, David Ham, Simon Funke, and Marie Rognes. Automated derivation of the adjoint of high-level transient ﬁnite element programs. SIAM Journal on Scientiﬁc Computing, 2013.

Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. ar Xiv preprint, 2017.

J. Futoma, S. Hariharan, and K. Heller. Learning to Detect Sepsis with a Multitask Gaussian Process RNN Classiﬁer. Ar Xiv e-prints, 2017.

Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Advances in Neural Information Processing Systems, pages 2211 2221, 2017.

Alex Graves. Adaptive computation time for recurrent neural networks. ar Xiv preprint ar Xiv:1603.08983, 2016.

David Ha, Andrew Dai, and Quoc V Le. Hypernetworks. ar Xiv preprint ar Xiv:1609.09106, 2016.

Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 34 (1):014004, 2017.

E. Hairer, S.P. Nørsett, and G. Wanner. Solving Ordinary Differential Equations I Nonstiff Problems. Springer, 1987.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770 778, 2016a.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630 645. Springer, 2016b.

Geoffrey Hinton, Nitish Srivastava, and Kevin Swersky. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent, 2012.

Yacine Jernite, Edouard Grave, Armand Joulin, and Tomas Mikolov. Variable computation in recurrent neural networks. ar Xiv preprint ar Xiv:1611.06188, 2016.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ar Xiv preprint ar Xiv:1412.6980, 2014.

Diederik P. Kingma and Max Welling. Auto-encoding variational Bayes. International Conference on Learning Representations, 2014.

Diederik P Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. Improved variational inference with inverse autoregressive ﬂow. In Advances in Neural Information Processing Systems, pages 4743 4751, 2016.

W. Kutta. Beitrag zur näherungsweisen Integration totaler Differentialgleichungen. Zeitschrift für Mathematik und Physik, 46:435 453, 1901.

Yann Le Cun, D Touresky, G Hinton, and T Sejnowski. A theoretical framework for back-propagation. In Proceedings of the 1988 connectionist models summer school, volume 1, pages 21 28. CMU, Pittsburgh, Pa: Morgan Kaufmann, 1988.

Yann Le Cun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.

Yang Li. Time-dependent representation for neural event sequence prediction. ar Xiv preprint ar Xiv:1708.00065, 2017.

Zachary C Lipton, David Kale, and Randall Wetzel. Directly modeling missing data in sequences with RNNs: Improved classiﬁcation of clinical time series. In Proceedings of the 1st Machine Learning for Healthcare Conference, volume 56 of Proceedings of Machine Learning Research, pages 253 270. PMLR, 18 19 Aug 2016. URL & ' ) ' .

Z. Long, Y. Lu, X. Ma, and B. Dong. PDE-Net: Learning PDEs from Data. Ar Xiv e-prints, 2017.

Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond ﬁnite layer neural networks: Bridging deep architectures and numerical differential equations. ar Xiv preprint ar Xiv:1710.10121, 2017.

Dougal Maclaurin, David Duvenaud, and Ryan P Adams. Autograd: Reverse-mode differentiation of native Python. In ICML workshop on Automatic Machine Learning, 2015.

Hongyuan Mei and Jason M Eisner. The neural Hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems, pages 6757 6767, 2017.

Valdemar Melicher, Tom Haber, and Wim Vanroose. Fast derivatives of likelihood functionals for ODE based models using adjoint-state method. Computational Statistics, 32(4):1621 1643, 2017.

Conny Palm. Intensitätsschwankungen im fernsprechverker. Ericsson Technics, 1943.

Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary De Vito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.

Barak A Pearlmutter. Gradient calculations for dynamic recurrent neural networks: A survey. IEEE Transactions on Neural networks, 6(5):1212 1228, 1995.

Lev Semenovich Pontryagin, EF Mishchenko, VG Boltyanskii, and RV Gamkrelidze. The mathematical theory of optimal processes. 1962.

M. Raissi and G. E. Karniadakis. Hidden physics models: Machine learning of nonlinear partial differential equations. Journal of Computational Physics, pages 125 141, 2018.

Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Multistep neural networks for datadriven discovery of nonlinear dynamical systems. ar Xiv preprint ar Xiv:1801.01236, 2018a.

Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Numerical Gaussian processes for time-dependent and nonlinear partial differential equations. SIAM Journal on Scientiﬁc Computing, 40(1):A172 A198, 2018b.

Danilo J Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, pages 1278 1286, 2014.

Danilo Jimenez Rezende and Shakir Mohamed. Variational inference with normalizing ﬂows. ar Xiv preprint ar Xiv:1505.05770, 2015.

C. Runge. Über die numerische Auﬂösung von Differentialgleichungen. Mathematische Annalen, 46: 167 178, 1895.

Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. ar Xiv preprint ar Xiv:1804.04272, 2018.

T. Ryder, A. Golightly, A. S. Mc Gough, and D. Prangle. Black-box Variational Inference for Stochastic Differential Equations. Ar Xiv e-prints, 2018.

Michael Schober, David Duvenaud, and Philipp Hennig. Probabilistic ODE solvers with Runge-Kutta means. In Advances in Neural Information Processing Systems 25, 2014.

Peter Schulam and Suchi Saria. What-if reasoning with counterfactual Gaussian processes. ar Xiv preprint ar Xiv:1703.10651, 2017.

Hossein Soleimani, James Hensman, and Suchi Saria. Scalable joint models for reliable uncertaintyaware event prediction. IEEE transactions on pattern analysis and machine intelligence, 2017a.

Hossein Soleimani, Adarsh Subbaswamy, and Suchi Saria. Treatment-response models for counterfactual reasoning with continuous-time, continuous-valued interventions. ar Xiv preprint ar Xiv:1704.02038, 2017b.

Jos Stam. Stable ﬂuids. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pages 121 128. ACM Press/Addison-Wesley Publishing Co., 1999.

Paul Stapor, Fabian Froehlich, and Jan Hasenauer. Optimization and uncertainty analysis of ODE models using second order adjoint sensitivity analysis. bio Rxiv, page 272005, 2018.

Jakub M Tomczak and Max Welling. Improving variational auto-encoders using Householder ﬂow. ar Xiv preprint ar Xiv:1611.09630, 2016.

Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent-space physics: Towards learning the temporal evolution of ﬂuid ﬂow. ar Xiv preprint ar Xiv:1802.10123, 2018.

Hong Zhang and Adrian Sandu. Fatode: a library for forward, adjoint, and tangent linear integration of ODEs. SIAM Journal on Scientiﬁc Computing, 36(5):C504 C523, 2014.