# dynamic_capacity_networks__aa2183e3.pdf

Dynamic Capacity Networks

Amjad Almahairi AMJAD.ALMAHAIRI@UMONTREAL.CA Nicolas Ballas NICOLAS.BALLAS@UMONTREAL.CA Tim Cooijmans TIM.COOIJMANS@UMONTREAL.CA Yin Zheng YIN.ZHENG@HULU.COM Hugo Larochelle HLAROCHELLE@TWITTER.COM Aaron Courville AARON.COURVILLE@UMONTREAL.CA

MILA, Universit e de Montr eal, Qu ebec, Canada Hulu LLC. Beijing, China Twitter, Cambridge, MA, USA

We introduce the Dynamic Capacity Network (DCN), a neural network that can adaptively assign its capacity across different portions of the input data. This is achieved by combining modules of two types: low-capacity subnetworks and high-capacity sub-networks. The low-capacity sub-networks are applied across most of the input, but also provide a guide to select a few portions of the input on which to apply the high-capacity sub-networks. The selection is made using a novel gradient-based attention mechanism, that efﬁciently identiﬁes input regions for which the DCN s output is most sensitive and to which we should devote more capacity. We focus our empirical evaluation on the Cluttered MNIST and SVHN image datasets. Our ﬁndings indicate that DCNs are able to drastically reduce the number of computations, compared to traditional convolutional neural networks, while maintaining similar or even better performance.

1. Introduction

Deep neural networks have recently exhibited state-of-theart performance across a wide range of tasks, including object recognition (Szegedy et al., 2014) and speech recognition (Graves & Jaitly, 2014). Top-performing systems, however, are based on very deep and wide networks that are computationally intensive. One underlying assumption of many deep models is that all input regions contain the

Proceedings of the 33 rd International Conference on Machine Learning, New York, NY, USA, 2016. JMLR: W&CP volume 48. Copyright 2016 by the author(s).

same amount of information. Indeed, convolutional neural networks apply the same set of ﬁlters uniformly across the spatial input (Szegedy et al., 2014), while recurrent neural networks apply the same transformation at every time step (Graves & Jaitly, 2014). Those networks lead to timeconsuming training and inference (prediction), in large part because they require a large number of weight/activation multiplications.

Task-relevant information, however, is often not uniformly distributed across input data. For example, objects in images are spatially localized, i.e. they exist only in speciﬁc regions of the image. This observation has been exploited in attention-based systems (Mnih et al., 2014), which can reduce computations signiﬁcantly by learning to selectively focus or attend to few, task-relevant, input regions. Attention employed in such systems is often referred to as hard-attention , as opposed to soft-attention which integrates smoothly all input regions. Models of hard-attention proposed so far, however, require deﬁning an explicit predictive model, whose training can pose challenges due to its non-differentiable cost.

In this work we introduce the Dynamic Capacity Network (DCN) that can adaptively assign its capacity across different portions of the input, using a gradient-based hardattention process. The DCN combines two types of modules: small, low-capacity, sub-networks, and large, highcapacity, sub-networks. The low-capacity sub-networks are active on the whole input, but are used to direct the high-capacity sub-networks, via our attention mechanism, to task-relevant regions of the input.

A key property of the DCN s hard-attention mechanism is that it does not require a policy network trained by reinforcement learning. Instead, we can train DCNs end-to-end with backpropagation. We evaluate a DCN model on the attention benchmark task Cluttered MNIST (Mnih et al.,

Dynamic Capacity Networks

2014), and show that it outperforms the state of the art.

In addition, we show that the DCN s attention mechanism can deal with situations where it is difﬁcult to learn a taskspeciﬁc attention policy due to the lack of appropriate data. This is often the case when training data is mostly canonicalized, while at test-time the system is effectively required to perform transfer learning and deal with substantially different, noisy real-world images. The Street View House Numbers (SVHN) dataset (Netzer et al., 2011) is an example of such a dataset. The task here is to recognize multidigit sequences from real-world pictures of house fronts; however, most digit sequences in training images are wellcentred and tightly cropped, while digit sequences of test images are surrounded by large and cluttered backgrounds. Learning an attention policy that focuses only on a small portion of the input can be challenging in this case, unless test images are pre-processed to deal with this discrepancy 1. DCNs, on the other hand, can be leveraged in such transfer learning scenarios, where we learn low and high capacity modules independently and only combine them using our attention mechanism at test-time. In particular, we show that a DCN model is able to efﬁciently recognize multi-digit sequences, directly from the original images, without using any prior information on the location of the digits.

Finally, we show that DCNs can perform efﬁcient region selection, in both Cluttered MNIST and SVHN, which leads to signiﬁcant computational advantages over standard convolutional models.

2. Dynamic Capacity Networks

In this section, we describe the Dynamic Capacity Network (DCN) that dynamically distributes its network capacity across an input.

We consider a deep neural network h, which we decompose into two parts: h(x) = g(f(x)) where f and g represent respectively the bottom layers and top layers of the network h while x is some input data. Bottom layers f operate directly on the input and output a representation, which is composed of a collection of vectors each of which represents a region in the input. For example, f can output a feature map, i.e. vectors of features each with a speciﬁc spatial location, or a probability map outputting probability distributions at each different spatial location. Top layers g consider as input the bottom layers representations f(x) and output a distribution over labels.

DCN introduces the use of two alternative sub-networks for the bottom layers f: the coarse layers fc or the ﬁne

1This is the common practice in previous work on this dataset, e.g. (Goodfellow et al., 2013; Ba et al., 2014; Jaderberg et al., 2015)

Figure 1. DCN overview. Our model applies the coarse layers on the whole image to get fc(x), chooses a set of salient patches Xs, applies the ﬁne layers only on the salient patches Xs to obtain a set of few ﬁne representation vectors ff(Xs), and ﬁnally combines them to make its prediction.

layers ff, which differ in their capacity. The ﬁne layers correspond to a high-capacity sub-network which has a high-computational requirement, while the coarse layers constitute a low-capacity sub-network. Consider applying the top layers only on the ﬁne representation, i.e. hf(x) = g(ff(x)). We refer to the composition hf = g ff as the ﬁne model. We assume that the ﬁne model can achieve very good performance, but is computationally expensive. Alternatively, consider applying the top layers only on the coarse representation, i.e. hc(x) = g(fc(x)). We refer to this composition hc = g fc as the coarse model. Conceptually, the coarse model can be much more computationally efﬁcient, but is expected to have worse performance than the ﬁne model.

The key idea behind DCN is to have g use representations from either the coarse or ﬁne layers in an adaptive, dynamic way. Speciﬁcally, we apply the coarse layers fc on the whole input x, and leverage the ﬁne layers ff only at a few important input regions. This way, the DCN can leverage the capacity of ff, but at a lower computational cost, by applying the ﬁne layers only on a small portion of the input. To achieve this, DCN requires an attentional mechanism, whose task is to identify good input locations on which to apply ff. In the remainder of this section, we focus on 2-dimensional inputs. However, our DCN model can be easily extended to be applied to any type of N-dimensional data.

2.1. Attention-based Inference

In DCN, we would like to obtain better predictions than those made by the coarse model fc while keeping the computational requirement reasonable. This can be done by selecting a few salient input regions on which we use the ﬁne representations instead of the coarse ones. DCN inference therefore needs to identify the important regions in the input with respect to the task at hand. For this, we use a novel approach for attention that uses backpropagation in

Dynamic Capacity Networks

the coarse model hc to identify few vectors in the coarse representation to which the distribution over the class label is most sensitive. These vectors correspond to input regions which we identify as salient or task-relevant.

Given an input image x, we ﬁrst apply the coarse layers on all input regions to compute the coarse representation vectors:

fc(x) = {ci,j | (i, j) [1, s1] [1, s2]}, (1)

where s1 and s2 are spatial dimensions that depend on the image size and ci,j = fc(xi,j) RD is a representation vector associated with the input region (i, j) in x, i.e. corresponds to a speciﬁc receptive ﬁeld or a patch in the input image. We then compute the output of the model based completely on the coarse vectors, i.e. the coarse model s output hc(x) = g(fc(x)).

Next, we identify a few salient input regions using an attentional mechanism that exploits a saliency map generated using the coarse model s output. The speciﬁc measure of saliency we choose is based on the entropy of the coarse model s output, deﬁned as:

l=1 o(l) c log o(l) c , (2)

where oc = g(fc(x)) is the vector output of the coarse model and C is the number of class labels. The saliency M of an input region position (i, j) is given by the norm of the gradient of the entropy H with respect to the coarse vector ci,j:

Mi,j = || ci,j H||2

l=1 o(l) c log o(l) c

where M Rs1 s2. The use of the entropy gradient as a saliency measure encourages selecting input regions that could affect the uncertainty in the model s predictions the most. In addition, computing the entropy of the output distribution does not require observing the true label, hence the measure is available at inference time. Note that computing all entries in matrix M can be done using a single backward pass of backpropagation through the top layers and is thus efﬁcient and simple to implement.

Using the saliency map M, we select a set of k input region positions with the highest saliency values. We denote the selected set of positions by Is [1, s1] [1, s2], such that |Is| = k. We denote the set of selected input regions by Xs = {xi,j | (i, j) Is} where each xi,j is a patch in x. Next we apply the ﬁne layers ff only on the selected

patches and obtain a small set of ﬁne representation vectors: ff(Xs) = {fi,j | (i, j) Is}, (4)

where fi,j = ff(xi,j). This requires that fi,j RD, i.e. the ﬁne vectors have the same dimensionality as the coarse vectors, allowing the model to use both of them interchangeably.

We denote the representation resulting from combining vectors from both fc(x) and ff(Xs) as the reﬁned representation fr(x). We discuss in Section 4 different ways in which they can be combined in practice. Finally, the DCN output is obtained by feeding the reﬁned representation into the top layers, g(fr(x)). We denote the composition g fr by the reﬁned model.

2.2. End-to-End Training

In this section, we describe an end-to-end procedure for training the DCN model that leverages our attention mechanism to learn ff and fc jointly. We emphasize, however, that DCN modules can be trained independently, by training a coarse and a ﬁne model independently and combining them only at test-time using our attention based inference. In Section 4.2 we show an example of how this modular training can be used for transfer learning.

In the context of image classiﬁcation, suppose we have a training set D = {(x(i), y(i)); i = 1 . . . m}, where each x(i) Rh w is an image, and y(i) {1, . . . , C} is its corresponding label. We denote the parameters of the coarse, ﬁne and top layers by θc, θf, and θt respectively. We learn all of these parameters (denoted as θ) by minimizing the cross-entropy objective function (which is equivalent to maximizing the log-likelihood of the correct labels):

i=1 log p y(i) | x(i); θ , (5)

where p( | x(i); θ) = g(fr(x(i))) is the conditional multinomial distribution deﬁned over the C labels given by the reﬁned model (Figure 1). Gradients are computed by standard back-propagation through the reﬁned model, i.e. propagating gradients at each position into either the coarse or ﬁne features, depending on which was used.

An important aspect of the DCN model is that the ﬁnal prediction is based on combining representations from two different sets of layers, namely the coarse layers fc and the ﬁne layers ff. Intuitively, we would like those representations to have close values such that they can be interchangeable. This is important for two reasons. First, we expect the top layers to have more success in correctly classifying the input if the transition from coarse to ﬁne representations is smooth. The second is that, since the saliency map is based on the gradient at the coarse representation values

Dynamic Capacity Networks

and since the gradient is a local measure of variation, it is less likely to reﬂect the beneﬁt of using the ﬁne features if the latter is very different from the former.

To encourage similarity between the coarse and ﬁne representations while training, we use a hint-based training approach inspired by Romero et al. (2014). Speciﬁcally, we add an additional term to the training objective that minimizes the squared distance between coarse and ﬁne representations: X

xi,j Xs fc(xi,j) ff(xi,j) 2 2. (6)

There are two important points to note here. First, we use this term to optimize only the coarse layers θc. That is, we encourage the coarse layers to mimic the ﬁne ones, and let the ﬁne layers focus only on the signal coming from the top layers. Secondly, computing the above hint objective over representations at all positions would be as expensive as computing the full ﬁne model; therefore, we encourage in this term similarity only over the selected salient patches.

3. Related Work

This work can be classiﬁed as a conditional computation approach. The goal of conditional computation, as put forward by Bengio (2013), is to train very large models for the same computational cost as smaller ones, by avoiding certain computation paths depending on the input. There have been several contributions in this direction. Bengio et al. (2013) use stochastic neurons as gating units that activate speciﬁc parts of a neural network. Our approach, on the other hand, uses a hard-attention mechanism that helps the model to focus its computationally expensive paths only on important input regions, which helps in both scaling to larger effective models and larger input sizes.

Several recent contributions use attention mechanisms to capture visual structure with biologically inspired, foveation-like methods, e.g. (Larochelle & Hinton, 2010; Denil et al., 2012; Ranzato, 2014; Mnih et al., 2014; Ba et al., 2014; Gregor et al., 2015). In Mnih et al. (2014); Ba et al. (2014), a learned sequential attention model is used to make a hard decision as to where to look in the image, i.e. which region of the image is considered in each time step. This so-called hard-attention mechanism can reduce computation for inference. The attention mechanism is trained by reinforcement learning using policy search. In practice, this approach can be computationally expensive during training, due to the need to sample multiple interaction sequences with the environment. On the other hand, the DRAW model (Gregor et al., 2015) uses a softattention mechanism that is fully differentiable, but requires processing the whole input at each time step. Our approach provides a simpler hard-attention mechanism with

computational advantages in both inference and learning.

The saliency measure employed by DCN s attention mechanism is related to pixel-wise saliency measures used in visualizing neural networks (Simonyan et al., 2013). These measures, however, are based on the gradient of the classiﬁcation loss, which is not applicable at test-time. Moreover, our saliency measure is deﬁned over contiguous regions of the input rather than on individual pixels. It is also taskdependent, as a result of deﬁning it using a coarse model trained on the same task.

Other works such as matrix factorization (Jaderberg et al., 2014; Denton et al., 2014) and quantization schemes (Chen et al., 2010; J egou et al., 2011; Gong et al., 2014) take the same computational shortcuts for all instances of the data. In contrast, the shortcuts taken by DCN specialize to the input, avoiding costly computation except where needed. However, the two approaches are orthogonal and could be combined to yield further savings.

Our use of a regression cost for enforcing representations to be similar is related to previous work on model compression (Bucilu et al., 2006; Hinton et al., 2015; Romero et al., 2014). The goal of model compression is to train a small model (which is faster in deployment) to imitate a much larger model or an ensemble of models. Furthermore, Romero et al. (2014) have shown that middle layer hints can improve learning in deep and thin neural networks. Our DCN model can be interpreted as performing model compression on the ﬂy, without the need to train a large model up front.

4. Experiments

In this section, we present an experimental evaluation of the proposed DCN model. To validate the effectiveness of our approach, we ﬁrst investigate the Cluttered MNIST dataset (Mnih et al., 2014). We then apply our model in a transfer learning setting to a real-world object recognition task using the Street View House Numbers (SVHN) dataset (Netzer et al., 2011).

4.1. Cluttered MNIST

We use the 100 100 Cluttered MNIST digit classiﬁcation dataset (Mnih et al., 2014). Each image in this dataset is a hand-written MNIST digit located randomly on a 100 100 black canvas and cluttered with digit-like fragments. Therefore, the dataset has the same size of MNIST: 60000 images for training and 10000 for testing.

4.1.1. MODEL SPECIFICATION

In this experiment we train a DCN model end-to-end, where we learn coarse and ﬁne layers jointly. We use 2

Dynamic Capacity Networks

Model Test Error

RAM 8.11% DRAW 3.36% Coarse Model 3.69% Fine Model 1.70% DCN w/o hints 1.71% DCN with hints 1.39%

Table 1. Results on Cluttered MNIST

Figure 2. The effect of using the hints objective. We show the squared distance between coarse and ﬁne features over salient regions during training in two cases: with and without using the hints objective. We observe that this regularizer helps in minimizing the distance and improves the model s generalization.

convolutional layers as coarse layers, 5 convolutional layers as ﬁne layers and one convolutional layer followed by global max pooling and a softmax as the top layers. Details of their architectures can be found in the Appendix 6.1. The coarse and ﬁne layers produce feature maps, i.e. feature vectors each with a speciﬁc spatial location. The set of selected patches Xs is composed of eight patches of size 14 14 pixels. We use here a reﬁned representation of the full input fr(x) in which ﬁne feature vectors are swapped in place of coarse ones:

fr(x) = {ri,j | (i, j) [1, s1] [1, s2]} (7)

( ff(xi,j), if xi,j Xs

fc(xi,j), otherwise. (8)

4.1.2. BASELINES

We use as baselines for our evaluation the coarse model (top layers applied only on coarse representations), the ﬁne model (top layers applied only on ﬁne representations), and we compare with previous attention-based models RAM (Mnih et al., 2014) and DRAW (Gregor et al., 2015).

4.1.3. EMPIRICAL EVALUATION

Results of our experiments are shown in Table 1. We get our best DCN result when we add the hint term in Eq. (6)

Figure 3. Patch selection on Cluttered MNIST. (left) Sample of selected patches. (right) Test error vs. number of patches: taking more patches yields lower error, but with diminishing returns.

in the training objective, which we observe to have a regularization effect on DCN. We can see that the DCN model performs signiﬁcantly better than the previous state-of-theart result achieved by RAM and DRAW models. It also outperforms the ﬁne model, which is a result of being able to focus only on the digit and ignore clutter. In Figure 2 we explore more the effect of the hint objective during training, and conﬁrm that it can indeed minimize the squared distance between coarse and ﬁne representations. To show how the attention mechanism of the DCN model can help it focus on the digit, we plot in Figure 3(a) the patches it ﬁnds in some images from the validation set, after only 9 epochs of training.

The DCN model is also more computationally efﬁcient. A forward pass of the ﬁne model requires the computation of the ﬁne layers representations on whole inputs and a forward pass of the top layers leading to 84.5M multiplications. On the other hand, DCN applies only the coarse layers on the whole input. It also requires the computation of the ﬁne representations for 8 input patches and a forward pass of the top layers. The attention mechanism of the DCN model requires an additional forward and backward pass through the top layers which leads to approximately 27.7M multiplications in total. As a result, the DCN model here has 3 times fewer multiplications than the ﬁne model. In practice we observed a time speed-up by a factor of about 2.9. Figure 3(b) shows how the test error behaves when we increase the number of patches. While taking additional patches improves accuracy, the marginal improvement becomes insigniﬁcant beyond 10 or so patches. The number of patches effectively controls a trade-off between accuracy and computational cost.

We tackle in this section a more challenging task of transcribing multi-digit sequences from natural images using the Street View House Numbers (SVHN) dataset (Netzer et al., 2011). SVHN is composed of real-world pictures

Dynamic Capacity Networks

Figure 4. The 4 left images are samples from the extra subset, and the 4 right images are samples from the test subset. We notice that extra images are well-centred and have much less background compared to test images.

containing house numbers and taken from house fronts. The task is to recognize the full digit sequence corresponding to a house number, which can be of length 1 to 5 digits. The dataset has three subsets: train (33k), extra (202k) and test (13k). In the following, we trained our models on 230k images from both the train and extra subsets, where we take a 5k random sample as a validation set for choosing hyperparameters.

The typical experimental setting in previous literature, e.g. (Goodfellow et al., 2013; Ba et al., 2014; Jaderberg et al., 2015), uses the location of digit bounding boxes as extra information. Input images are generally cropped, such that digit sequences are centred and most of the background and clutter information is pruned. We argue that our DCN model can deal effectively with real-world noisy images having large portions of clutter or background information. To demonstrate this ability, we investigate a more general problem setting where the images are uncropped and the digits locations are unknown. We apply our models on SVHN images in their original sizes and we do not use any extra bounding box information. 2

An important property of the SVHN dataset is the large discrepancy between the train/extra sets and the test set. Most of the extra subset images (which dominate the training data) have their digits well-centred with little cluttered background, while test images have more variety in terms of digit location and background clutter. Figure 4 shows samples of these images. We can tackle this training/test dataset discrepancy by training a DCN model in a transfer learning setting. We train the coarse and ﬁne layers of the DCN independently on the training images that have little background-clutter, and then combine them using our attention mechanism, which does not require explicit training, to decide on which subsets of the input to apply the ﬁne layers.

4.2.1. MULTI-DIGIT RECOGNITION MODEL

We follow the model proposed in (Goodfellow et al., 2013) for learning a probabilistic model of the digit sequence

2The only pre-processing we perform on the data is converting images to grayscale.

given an input image x. The output sequence S is deﬁned using a collection of N random variables, S1, . . . , SN, representing the elements of the sequence and an extra random variable S0 representing its length. The probability of a given sequence s = {s1, . . . , sn} is given by:

p(S = s | x) = p(S0 = n | x)

i=1 p(Si = si | x), (9)

where p(S0 = n | x) is the conditional distribution of the sequence length and p(Si = si | x) is the conditional distribution of the i-th digit in the sequence. In particular, our model on SVHN has 6 softmaxes: 1 for the length of the sequence (from 1 to 5), and 5 for the identity of each digit or a null character if no digit is present (11 categories).

4.2.2. MODEL SPECIFICATION

The coarse and ﬁne bottom layers, fc and ff, are fullyconvolutional, composed of respectively 7 and 11 layers. The representation, produced by either the ﬁne or coarse layers, is a probability map, which is a collection of independent full-sequence prediction vectors, each vector corresponding to a speciﬁc region of the input. We denote the prediction for the i-th output at position (j, k) by p(j,k)(Si | x).

The top layer g is composed of one global average pooling layer which combines predictions from various spatial locations to produce the ﬁnal prediction p(S | x).

Since we have multiple outputs in this task, we modify the saliency measure used by the DCN s attention mechanism to be the sum of the entropy of the 5 digit softmaxes:

j=1 p(Si = sj | x) log p(Si = sj | x). (10)

When constructing the saliency, instead of using the gradient with respect to the probability map, we use the gradient with respect to the feature map below it. This is necessary to avoid identical gradients as g, the top function, is composed by only one average pooling.

We also use a reﬁned model that computes its output by applying the pooling top layer g only on the k independent predictions from ﬁne layers, ignoring the coarse layers. We have found empirically that this results in a better model, and suspect that otherwise the predictions from the salient regions are drowned out by the noisy predictions from uninformative regions.

We train the coarse and ﬁne layers of DCN independently in this experiment, minimizing log p(S | x) using SGD. For the purposes of training only, we resize images to 64 128. Details on the coarse and ﬁne architectures are found in Appendix 6.2.

Dynamic Capacity Networks

4.2.3. BASELINES

As mentioned in the previous section, each of the coarse representation vectors in this experiment corresponds to multi-digit recognition probabilities computed at a given region, which the top layer g simply averages to obtain the baseline coarse model:

p(Si | x) = 1 d1 d2

j,k p(j,k)(Si | x). (11)

The baseline ﬁne model is deﬁned similarly.

As an additional baseline, we consider a soft-attention coarse model, which takes the coarse representation vectors over all input regions, but uses a top layer that performs a weighted average of the resulting location-speciﬁc predictions. We leverage the entropy to deﬁne a weighting scheme which emphasizes important locations:

p(Si | x) = X

j,k wi,j,kp(j,k)(Si | x). (12)

The weight wi,j,k is deﬁned as the normalized inverse entropy of the i-th prediction by the (j, k)-th vector, i.e. :

H 1 i,j,k H 1 i,q,r , (13)

where Hi,j,k is deﬁned as:

l=1 pj,k(Si = sl | x) log pj,k(Si = sl | x),

(14) and C is either 5 for S0 or 11 for all other Si. As we ll see, this weighting improves the coarse model s performance in our SVHN experiments. We incorporate this weighting in DCN to aggregate predictions from the salient regions.

To address scale variations in the data, we extend all models to multi-scale by processing each image several times at multiple resolutions. Predictions made at different scales are considered independent and averaged to produce the ﬁnal prediction.

It is worth noting that all previous literature on SVHN dealt with a simpler task where images are cropped and resized. In this experiment we deal with a more general setting, and our results cannot be directly compared with these results.

4.2.4. EMPIRICAL EVALUATION

Table 2 shows results of our experiment on SVHN. The coarse model has an error rate of 40.6%, while by using our proposed soft-attention mechanism, we decrease the error rate to 31.4%. This conﬁrms that the entropy is a good measure for identifying important regions when task-relevant information is not uniformly distributed across input data.

Model Test Error

Coarse model, 1 scale 40.6% Coarse model, 2 scales 40.0% Coarse model, 3 scales 40.0%

Fine model, 1 scale 25.2% Fine model, 2 scales 23.7% Fine model, 3 scales 23.3%

Soft-attention, 1 scale 31.4% Soft-attention, 2 scales 31.1% Soft-attention, 3 scales 30.8%

DCN, 6 patches, 1 scale 20.0% DCN, 6 patches, 2 scales 18.2% DCN, 9 patches, 3 scales 16.6%

Table 2. Results on SVHN dataset without using bounding box information.

Figure 5. Number of multiplications in the Coarse, Fine and DCN models given different image input sizes.

The ﬁne model, on the other hand, achieves a better error rate of 25.2%, but is more computationally expensive. Our DCN model, which selects only 6 regions on which to apply the high-capacity ﬁne layers, achieves an error rate of 20.0%. The DCN model can therefore outperform, in terms of classiﬁcation accuracy, the other baselines. This veriﬁes our assumption that by applying high capacity subnetworks only on the inputs most informative regions, we are able to obtain high classiﬁcation performance. Figure 6 shows a sample of the selected patches by our attention mechanism.

An additional decrease of the test errors can be obtained by increasing the number of processed scales. In the DCN model, taking 3 patches at 2 scales (original and 0.75 scales), leads to 18.2% error, while taking 3 patches at 3 scales (original, 0.75 and 0.5 scales) leads to an error rate of 16.6%. Our DCN model can reach its best performance of 11.6% by taking all possible patches at 3 scales, but it does not offer any computational beneﬁt over the ﬁne model.

We also investigate the computational beneﬁts of the DCN approach as the dimensions of the input data increase. Ta-

Dynamic Capacity Networks

Figure 6. A sample of the selected patches in SVHN images. The images are processed by the DCN inference procedure in their original sizes. They are resized here for illustration purposes.

ble 5 reports the number of multiplications the ﬁne model, coarse model and the DCN model require, given different input sizes. We also verify the actual computational time of these models by taking the largest 100 images in the SVHN test set, and computing the average inference time taken by all the models. 3 The smallest of these images has a size of 363 735 pixels, while the largest has a size of 442 1083 pixels. On average, the coarse and the soft-attention models take 8.6 milliseconds, while the ﬁne model takes 62.6 milliseconds. On the largest 100 SVHN test images, the DCN requires on average 10.8 milliseconds for inference.

5. Conclusions

We have presented the DCN model, which is a novel approach for conditional computation. We have shown that using our visual attention mechanism, our network can adaptively assign its capacity across different portions of the input data, focusing on important regions of the input. Our model achieved state-of-the-art performance on the Cluttered MNIST digit classiﬁcation task, and provided computational beneﬁts over traditional convolutional network architectures. We have also validated our model in a transfer learning setting using the SVHN dataset, where we tackled the multi-digit recognition problem without using any a priori information on the digits location. We have shown that our model outperforms other baselines, yet remains tractable for inputs with large spatial dimensions.

6. Appendix

6.1. Cluttered MNIST Experiment Details

Coarse layers: 2 convolutional layers, with 7 7 and 3 3 ﬁlter sizes, 12 and 24 ﬁlters, respectively, and a 2 2 stride. Each feature in the coarse feature maps covers a patch of size 11 11 pixels, which we extend by 3 pixels in each side to give the ﬁne layers more context. The size of the coarse feature map is 23 23.

Fine layers: 5 convolutional layers, each with 3 3

3We evaluate all models on an NVIDIA Titan Black GPU card.

ﬁlter sizes, 1 1 strides, and 24 ﬁlters. We apply 2 2 pooling with 2 2 stride after the second and fourth layers. We also use 1 1 zero padding in all layers except for the ﬁrst and last layers. This architecture was chosen so that it maps a 14 14 patch into one spatial location.

Top layers: one convolutional layer with 4 4 ﬁlter size, 2 2 stride and 96 ﬁlters, followed by global max pooling. The result is fed into a 10-output softmax layer.

We use rectiﬁer non-linearities in all layers. We use Batch Normalization (Ioffe & Szegedy, 2015) and Adam (Kingma & Ba, 2014) for training our models. In DCN we train the coarse layers with a convex combination of cross entropy objective and hints.

6.2. SVHN Experiment Details

Coarse layers: the model is fully convolutional with 7 convolutional layers. First three layers have 24, 48, 128 ﬁlters respectively with size 5 5 and stride 2 2. Layer 4 has 192 ﬁlters with 4 5 and stride 1 2. Layer 5 has 192 ﬁlters with size 1 4. Finally, the last two layers are 1 1 convolutions with 1024 ﬁlters. We use stride of 1 1 in the last 3 layers and do not use zero padding in any of the coarse layers. The corresponding patch size here is 54 110.

Fine layers: 11 convolutional layers. The ﬁrst 5 convolutional layers have 48, 64, 128, 160 and 192 ﬁlters respectively, with size 5 5 and zero-padding. After layers 1, 3, and 5 we use 2 2 max pooling with stride 2 2. The following layers have 3 3 convolution with 192 ﬁlters. The 3 last layers are 1 1 convolution with 1024 hidden units.

Here we use SGD with momentum and exponential learning rate decay. While training, we take 54 110 random crop from images, and we use 0.2 dropout on convolutional layers and 0.5 dropout on fully connected layers.

Acknowledgements

The authors would like to acknowledge the support of the following organizations for research funding and computing support: Nuance Foundation, Compute Canada and Calcul Qu ebec. We would like to thank the developers of Theano (Bergstra et al., 2011; Bastien et al., 2012) and Blocks/Fuel (Van Merri enboer et al., 2015) for developing such powerful tools for scientiﬁc computing, and our reviews for their useful comments.

Dynamic Capacity Networks

Ba, Jimmy, Mnih, Volodymyr, and Kavukcuoglu, Koray. Multiple object recognition with visual attention. ar Xiv preprint ar Xiv:1412.7755, 2014.

Bastien, Fr ed eric, Lamblin, Pascal, Pascanu, Razvan, Bergstra, James, Goodfellow, Ian J., Bergeron, Arnaud, Bouchard, Nicolas, and Bengio, Yoshua. Theano: new features and speed improvements. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop, 2012.

Bengio, Yoshua. Deep learning of representations: Looking forward. In Statistical Language and Speech Processing, pp. 1 37. Springer, 2013.

Bengio, Yoshua, L eonard, Nicholas, and Courville, Aaron. Estimating or propagating gradients through stochastic neurons for conditional computation. ar Xiv preprint ar Xiv:1308.3432, 2013.

Bergstra, James, Bastien, Fr ed eric, Breuleux, Olivier, Lamblin, Pascal, Pascanu, Razvan, Delalleau, Olivier, Desjardins, Guillaume, Warde-Farley, David, Goodfellow, Ian J., Bergeron, Arnaud, and Bengio, Yoshua. Theano: Deep learning on gpus with python. In Big Learn workshop, NIPS 11, 2011.

Bucilu, Cristian, Caruana, Rich, and Niculescu-Mizil, Alexandru. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535 541. ACM, 2006.

Chen, Yongjian, Guan, Tao, and Wang, Cheng. Approximate nearest neighbor search by residual vector quantization. Sensors, 10(12):11259 11273, 2010.

Denil, Misha, Bazzani, Loris, Larochelle, Hugo, and de Freitas, Nando. Learning where to attend with deep architectures for image tracking. Neural computation, 24(8):2151 2184, 2012.

Denton, Emily L, Zaremba, Wojciech, Bruna, Joan, Le Cun, Yann, and Fergus, Rob. Exploiting linear structure within convolutional networks for efﬁcient evaluation. In NIPS, pp. 1269 1277. 2014.

Gong, Yunchao, Liu, Liu, Yang, Min, and Bourdev, Lubomir. Compressing deep convolutional networks using vector quantization. Co RR, abs/1412.6115, 2014.

Goodfellow, Ian J, Bulatov, Yaroslav, Ibarz, Julian, Arnoud, Sacha, and Shet, Vinay. Multi-digit number recognition from street view imagery using deep convolutional neural networks. ar Xiv preprint ar Xiv:1312.6082, 2013.

Graves, Alex and Jaitly, Navdeep. Towards end-to-end speech recognition with recurrent neural networks. In ICML), 2014.

Gregor, Karol, Danihelka, Ivo, Graves, Alex, and Wierstra, Daan. Draw: A recurrent neural network for image generation. ar Xiv preprint ar Xiv:1502.04623, 2015.

Hinton, Geoffrey, Vinyals, Oriol, and Dean, Jeff. Distilling the knowledge in a neural network. ar Xiv preprint ar Xiv:1503.02531, 2015.

Ioffe, Sergey and Szegedy, Christian. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ar Xiv preprint ar Xiv:1502.03167, 2015.

Jaderberg, M., Vedaldi, A., and Zisserman, A. Speeding up convolutional neural networks with low rank expansions. In BMVC, 2014.

Jaderberg, Max, Simonyan, Karen, Zisserman, Andrew, and Kavukcuoglu, Koray. Spatial transformer networks. ar Xiv preprint ar Xiv:1506.02025, 2015.

J egou, Herv e, Douze, Matthijs, and Schmid, Cordelia. Product quantization for nearest neighbor search. IEEE TPAMI, 33(1):117 128, 2011.

Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. ar Xiv preprint ar Xiv:1412.6980, 2014.

Larochelle, Hugo and Hinton, Geoffrey E. Learning to combine foveal glimpses with a third-order boltzmann machine. In Advances in neural information processing systems, pp. 1243 1251, 2010.

Mnih, Volodymyr, Heess, Nicolas, Graves, Alex, et al. Recurrent models of visual attention. In Advances in Neural Information Processing Systems, pp. 2204 2212, 2014.

Netzer, Yuval, Wang, Tao, Coates, Adam, Bissacco, Alessandro, Wu, Bo, and Ng, Andrew Y. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 5. Granada, Spain, 2011.

Ranzato, Marc Aurelio. On learning where to look. ar Xiv preprint ar Xiv:1405.5488, 2014.

Romero, Adriana, Ballas, Nicolas, Kahou, Samira Ebrahimi, Chassang, Antoine, Gatta, Carlo, and Bengio, Yoshua. Fitnets: Hints for thin deep nets. ar Xiv preprint ar Xiv:1412.6550, 2014.

Simonyan, Karen, Vedaldi, Andrea, and Zisserman, Andrew. Deep inside convolutional networks: Visualising image classiﬁcation models and saliency maps. ar Xiv preprint ar Xiv:1312.6034, 2013.

Dynamic Capacity Networks

Szegedy, Christian, Liu, Wei, Jia, Yangqing, Sermanet, Pierre, Reed, Scott, Anguelov, Dragomir, Erhan, Dumitru, Vanhoucke, Vincent, and Rabinovich, Andrew. Going deeper with convolutions. ar Xiv preprint ar Xiv:1409.4842, 2014.

Van Merri enboer, Bart, Bahdanau, Dzmitry, Dumoulin, Vincent, Serdyuk, Dmitriy, Warde-Farley, David, Chorowski, Jan, and Bengio, Yoshua. Blocks and fuel: Frameworks for deep learning. Ar Xiv e-prints, (1506.00619), June 2015. URL http://adsabs. harvard.edu/abs/2015ar Xiv150600619V.