# diffused_redundancy_in_pretrained_representations__6a7648a2.pdf Diffused Redundancy in Pre-trained Representations Vedant Nanda University of Maryland & MPI-SWS Till Speicher MPI-SWS John P. Dickerson University of Maryland Krishna P. Gummadi MPI-SWS Soheil Feizi University of Maryland Adrian Weller The Alan Turing Institute & University of Cambridge Representations learned by pre-training a neural network on a large dataset are increasingly used successfully to perform a variety of downstream tasks. In this work, we take a closer look at how features are encoded in such pre-trained representations. We find that learned representations in a given layer exhibit a degree of diffuse redundancy, i.e., any randomly chosen subset of neurons in the layer that is larger than a threshold size shares a large degree of similarity with the full layer and is able to perform similarly as the whole layer on a variety of downstream tasks. For example, a linear probe trained on 20% of randomly picked neurons from the penultimate layer of a Res Net50 pre-trained on Image Net1k achieves an accuracy within 5% of a linear probe trained on the full layer of neurons for downstream CIFAR10 classification. We conduct experiments on different neural architectures (including CNNs and Transformers) pretrained on both Image Net1k and Image Net21k and evaluate a variety of downstream tasks taken from the VTAB benchmark. We find that the loss & dataset used during pre-training largely govern the degree of diffuse redundancy and the critical mass of neurons needed often depends on the downstream task, suggesting that there is a task-inherent redundancy-performance Pareto frontier. Our findings shed light on the nature of representations learned by pre-trained deep neural networks and suggest that entire layers might not be necessary to perform many downstream tasks. We investigate the potential for exploiting this redundancy to achieve efficient generalization for downstream tasks and also draw caution to certain possible unintended consequences. Our code is available at https://github.com/nvedant07/diffused-redundancy. 1 Introduction Over the years, many architectures have been proposed (such as [50, 16, 23]) that achieve competitive accuracies on many benchmarks [45]. A key reason for the success of these models is their ability to learn useful representations of data [28]. While these models continue to get better, understanding the properties of underlying learned representations continues to be a challenge. Prior works have attempted to understand representations learned by deep neural networks through the lens of mutual information between the representations, inputs, and outputs [49] and hypothesize that neural networks perform well because of a compression phase where mutual information between inputs and representations decreases. Additionally, recent works have found that many neurons are polysemantic, i.e., one neuron can encode multiple concepts [10, 39], and that one can Correspondence to: vnanda@mpi-sws.org 37th Conference on Neural Information Processing Systems (Neur IPS 2023). Table 1: Different model architectures with varying penultimate layer lengths trained on Image Net1k. WRN502 stands for Wide Res Net50-2. Implementation of architectures is taken from timm [59]. Diffused redundancy here measures what fractions of neurons (randomly picked) can be discarded to achieve within δ = 90% performance of the full layer. Model Feature Length Image Net1k Top-1 Accuracy Diffused Redundancy for δ = 0.9 CIFAR10 CIFAR100 Flowers Oxford-IIIT-Pets Vi T S-16 384 64.82% 0.70 0.50 0.50 0.80 Vi T S-32 384 55.73% 0.70 0.50 0.50 0.70 Res Net18 512 69.23% 0.80 0.50 0.50 0.90 Res Net50 2048 80.07% 0.90 0.50 0.20 0.90 WRN50-2 2048 77.00% 0.95 0.80 0.50 0.95 VGG16 4096 73.36% 0.95 0.80 0.80 0.95 then train sparse linear models on such concepts to do explainable classification [61]. However, it is not well understood if or how extracted features are concentrated or spread across the entire representation. While the length of the feature vectors extracted from state-of-the-art networks 2 can vary greatly, their accuracy on downstream tasks are not correlated to the size of the representation (see Table 1), but rather depend mostly on the inductive biases and training recipes [60, 53]. In all cases, the size of extracted feature vector (i.e. number of neurons) is orders of magnitude less than the dimensions of the input and thus allows for efficient transfer to many downstream tasks [22, 4, 41, 55]. We show that even using a random subset of these extracted neurons is enough to achieve downstream transfer accuracy close to that achieved by the full layer, thus showing that learned representations exhibit a degree of diffused redundancy (Table 1). Early works in perception suggest that there are many redundant neurons in the human visual cortex [3] and some later works argued that a similar redundancy in artificial neural networks should help in faster convergence [19]. In this paper, we revisit redundancy in the context of modern DNN architectures that are trained on large-scale datasets. In particular, we propose the diffused redundancy hypothesis and systematically measure its prevalence across different pre-training datasets, losses, model architectures, and downstream tasks. We also show how this kind of redundancy can be exploited to obtain desirable properties such as generalization performance but at the same time draw caution to certain drawbacks of such an approach in increasing disparity in inter-class performance. We highlight the following contributions: We present the diffused redundancy hypothesis which states that learned representations exhibit redundancy that is diffused throughout the layer, i.e., many random subsets (of sufficient size) of neurons can perform as well as the entire layer. Our work aims to better understand the nature of representations learned by DNNs. We present an initial analysis of why diffused redundancy exists in deep neural networks, we show that a randomly chosen subset of neurons of size k performs as well as the projection of the entire layer on the first k principal components. Intuitively this means that in DNNs representations, many random subsets of size k roughly capture all the variation in data that is possible with k dimensions (since PCA represents directions of maximum variance). We propose a measure of diffused redundancy and systematically test our hypothesis across various architectures, pre-training datasets & losses, and downstream tasks. We find that diffused redundancy is significantly impacted by pre-training datasets & loss and downstream datasets. We find that models that are explicitly trained such that particular parts of the full representation perform as well as the full layer, i.e., these models have structured redundancy (e.g. [26]), also exhibit a significant amount of diffused redundancy. Further, we also evaluate models trained with regularization that decorrelates activation of neurons and again find that these regularizations surprisingly do not affect diffused redundancy. These results suggest that this phenomenon is perhaps inevitable when DNNs have a wide enough final layer. 2Extracted features for the purpose of this paper refers to the representation recorded on the penultimate layer, but the larger concept applies to any layer We quantify the degree of diffused redundancy as a function of the number of neurons in a given layer. As we reduce the dimension of the extracted feature vector and re-train the model, the degree of diffused redundancy decreases significantly, implying that diffused redundancy only appears when the layer is wide enough to accommodate redundancy. Finally we draw caution to some potential undesirable side-effects of exploiting diffused redundancy for efficient transfer learning that have implications for fairness. 1.1 Related Work Closest to our work is that of Dalvi et al. [7] who also investigate neuron redundancy but in the context of pre-trained language models. They analyze two language models and find that they can achieve good downstream performance with a significantly smaller subset of neurons. However, there are two key differences to our work. First, their analysis of neuron redundancy uses neurons from all layers (by concatenating each layer), whereas we show that such redundancy exists even at the level of a single (penultimate) layer. Second, and perhaps more importantly, they use feature selection to choose the subset of neurons, whereas we show that features are diffused throughout and that even a random pick of neurons suffices. Our work also differs by analyzing vision models (instead of language models) and using a diverse set of 30 pre-trained models (as opposed to testing only two models) which allows us to better understand the causes of such redundancy. Efficient Representation Learning These works aim to learn representations which are slim , with the goal of efficient deployment on edge devices [65, 64, 5]. Recently proposed paradigm of Matryoshka Representation Learning [26] aims to learn nested representations where one can perform downstream tasks with only a small portion of the representation. The goal of such representations is to allow quick, adaptive deployment without having to perform multiple, often expensive, forward passes. These works could be seen as inducing structured redundancy on the learned representations, where pre-specified parts of the representation are made to perform similar to the full representation. Our work, instead, aims to look at diffused redundancy that arises naturally in the training of DNNs. We carefully highlight the tradeoffs involved in exploiting this redundancy. Pruning and Compression Many prior works focus on pruning weights [29, 13, 11, 15, 31, 9, 30] and how it can lead to sparse neural networks with many weights turned off. Our focus, however, is on understanding redundancy at the neuron level, without changing the weights. Work on structured pruning is more closely related to our work [33, 17], however, a key focus of these works is to prune channels/filters from convolution layers. Our work is more focused on understanding the nature of learned features and is more broadly applicable to all kinds of layers and models. We additionally focus on randomly pruning neurons, whereas structured pruning methods perform magnitude or feature-selection-based pruning. Explainability/Interpretability Many works aim to understand learned representations with the goal of better explainability [35, 63, 2, 20, 40, 39, 10, 67]. Two works in this space are especially related to our work: sparse linear layers [61] which show that one can train sparse linear layers on top of extracted features from DNNs; and concept bottleneck models [21] which explicitly introduce a layer in which each neuron corresponds to a meaningful semantic concept. Both these works explicitly optimize for small/sparse layers, whereas our work shows that similar small layers already exist in pre-trained networks, and in fact, can be found simply with random sampling. Understanding Deep Learning A related concept is that of instrinsic dimensionality of DNN landscapes [32]. Similar to our work, intrinsic dimensionality also requires dropping random parameters (weights) of the network. We, however, are concerned with dropping individual neurons. Other works on understanding deep learning [49, 1] have also looked at the learned features, however, none of these works analyze the redundancy at the neuron level. Another related phenomenon to diffused redundancy is that of neural collapse [42], which states that representations of the penultimate layer collapse to K points (where K = number of classes in the pre-training dataset). This implies that for perfectly collapsed representations we need to store just enough information in the final layer activations to be able to represent K different points. We interestingly show that this information is spread throughout the layer with a significant degree of redundancy. 0.0 0.2 0.4 0.6 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy cifar10 cifar100 flowers oxford-iiit-pets (a) Downstream Acc; Standard 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) cifar10 cifar100 flowers oxford-iiit-pets (b) Diff. Redundancy; Standard 0.0 0.2 0.4 0.6 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy cifar10 cifar100 flowers oxford-iiit-pets (c) Downstream Acc; AT 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) cifar10 cifar100 flowers oxford-iiit-pets (d) Diff. Redundancy; AT 0.0 0.2 0.4 0.6 0.8 Fraction of Neurons (Total=2048) cifar10 cifar100 flowers oxford-iiit-pets (e) CKA; Standard 0.0 0.2 0.4 0.6 0.8 Fraction of Neurons (Total=2048) cifar10 cifar100 flowers oxford-iiit-pets (f) CKA; AT Figure 1: [Testing For Diffused Redundancy in Res Net50 Pre-trained on Image Net1k] Top: transfer accuracies + Diffused Redundancy (DR) measure (Eq 1) on different downstream datasets, dotted horizontal line shows accuracy obtained using the full layer. We see that accuracy obtained using parts of representation varies greatly with pre-training loss (much more diffused redundancy in Adversarially Trained (AT) Res Net), but also depends on the downstream dataset. Bottom: comparing CKA between a randomly chosen fraction of neurons to the whole layer. Here we evaluate CKA on samples from different datasets and find that similarity of a subset of layer rapidly increases, reaching a similarity of greater than 90% on the adversarially trained Res Net with only 10% randomly chosen neurons. Values are averaged over 5 random picks and error bars show std. dev. 2 The Diffused Redundancy Phenomenon Prior observations about a compression phase [49] and neural collapse [42] suggest that the representations need not store a lot of information about the input. These findings imply that all neurons in a learned representation might not be necessary to capture all the information in a particular layer. Extending these observations, we propose the diffused redundancy hypothesis: Learned features are diffused throughout a given layer with redundancy such that there exist many randomly chosen subsets of neurons that can achieve similar performance to the whole layer for a variety of downstream tasks. Note that our hypothesis has two related but distinct parts to it: 1) redundancy in learned features, and 2) diffusion of this redundancy throughout the extracted feature vector. Redundancy refers to features being replicated in parts of the representation so that one can perform downstream tasks with parts of representation as well as with the full representation. Diffusion refers to this redundancy being spread all over the feature vector (as opposed to being structured), i.e., many random subsets (of sufficient size) of the feature vector perform equally well. In order to evaluate the redundancy part of the diffused redundancy hypothesis we use two tasks: 1) representation similarity between randomly chosen subsets of a representation with the whole representation, and 2) transfer accuracy on out-of-distribution datasets (using a linear probe) of randomly chosen subsets of the representation compared to the whole representation. To estimate diffusion, we run each check for redundancy over multiple random seeds and plot the standard deviation over these runs. Representation Similarity of Part vs Whole Centered Kernel Alignment (CKA) is a widely used representation similarity measure and takes in two representations of n data points Z Rn d1 and Y Rn d2 and gives a similarity score between 0 and 1 [24]. Intuitively, CKA (with linear kernel, see Appendix A for details about CKA) measures if the two representations rank the n points similarly (where similarity is based on cosine distances). For a given neural network g and n samples drawn from a given data distribution, i.e., X D, let g(X) be the (penultimate) layer representation. If m is a boolean vector representing a subset of neurons in g(X), then we aim to measure CKA(m g(X), g(X)) to estimate how much redundancy exists in the layer. If indeed CKA(m g(X), g(X)) is high (i.e. close to 1) then it s a strong indication that the diffused redundancy hypothesis holds. Downstream Transfer Performance of Part vs Whole A commonly used paradigm to measure the quality of learned representations is to measure their performance on a variety of downstream tasks [68, 22]. Here, we attach a linear layer (h) on top of the extracted features of a network (g) to do classification. This layer is then trained using the training dataset of the particular task (keeping g frozen). If features were to be diffused redundantly then accuracy obtained using h g, i.e.linear layer attached to the entire feature vector, should be roughly the same as h (m g); where m is a boolean vector representing a subset of neurons extracted by g, and h & h are independently trained linear probes. For both tasks, i.e. representation similarity and downstream transfer performance, we evaluate on CIFAR10/100 [25], Oxford-IIIT-Pets [43] and Flowers [38] datasets, from the VTAB benchmark [68]. Training and pre-processing details are included in Appendix B. Measure of Diffused Redundancy In order to rigorously test our hypothesis, we define a measure of diffused redundancy (DR) for a given model (g) with M being a set of all possible boolean vectors of size |g|, i.e. size of the representation extracted from g. Each vector m M represents a possible subset of neurons from the entire layer. This measure is defined on a particular task (T) as follows: DR(g, T, δ) = 1 min f , s.t. 1 |Mf | P m Mf T (m g) Mf = m M| P i mi = f ; m {0, 1}|g| Here T(.) denotes the performance of the model inside () for the particular task and δ is a userdefined tolerance level. For the task of representation similarity T(m g) is CKA between a subset of neurons denoted by m g and g, and T(g) is always 1, since it denotes CKA between g and g. For downstream transfer performance, T(m g) is the test accuracy obtained by training a linear probe on the portion of representation denoted by m g and T(g) is the test accuracy obtained using the full representation. For δ = 1, this measure tells what fraction of neurons could be discarded to exactly match the performance of the entire set of neurons. A higher value of DR denotes that only a few random neurons were needed to match the task performance of the full set of neurons, and thus indicates higher redundancy. Since M contains an exponential number of vectors (2|g|), precisely estimating this quantity is hard. Thus, we first choose a few f (number of neurons to be chosen) to define subsets of M. Then for each Mf we randomly select 5 samples. 2.1 Prevalence of Diffused Redundancy in Pre-Trained Models Figure 1 checks for diffused redundancy in the penultimate layer representation of two types of Res Net50 pre-trained on Image Net1k: one using the standard cross-entropy loss and another trained using adversarial training [34] (with ℓ2 threat model and ϵ = 3) 3. We check for diffused redundancy using both tasks of representation similarity and downstream transfer performance. Redundancy This is indicated along the x-axis of Fig 1, i.e., redundancy is shown when some small subset of the full set of neurons can achieve almost as good performance as the full set of neurons. When looking at downstream task performance (Figs 1a&1c), in order to obtain performance within some δ% of the full layer accuracy (dotted lines), the fraction of neurons that can be discarded are task-dependent, e.g. across both training types we see that flowers (102 classes) and CIFAR100 (100 classes) require more fraction of neurons than CIFAR10 (10 classes) and oxford-iiit-pets (37 classes), perhaps because both these tasks have more classes. Additionally, across all datasets, the model trained with adversarial training exhibits more diffused redundancy than the one trained with standard loss (Fig 1d& 1b respectively), meaning we can discard far more neurons for the adversarially trained model to reach close to the full layer accuracy. Interestingly when looking at CKA between part of the feature vector with the full extracted vector (Figs 1e&1f), we do not see a significant difference in trends when evaluating CKA on samples from different datasets. However, we still see that we can achieve a given level of CKA with far fewer fraction of neurons in the adversarially trained Res Net50 as compared to the usually trained Res Net50. Diffused Redundancy This is indicated by small error bars in Figs 1a&1c&1e&1f. If redundancy were instead very structured, then different random picks of neurons would have high variance, 3We also report results for ℓ , ϵ = 4/255 in Appendix D and generally find that the ℓ2 models exhibit higher diffused redundancy. We choose to study adversarially robust models since they ve been shown to transfer better [46]. however, the error bars here are very low, showing that performance is very stable across different random picks, thus indicating that redundancy is diffused throughout the layer. While both tasks of downstream transfer and CKA between part and whole indicate higher diffused redundancy for the adversarially trained model, we see that downstream transfer performance can differ substantially based on the dataset (while CKA remains fairly stable across the same datasets), indicating that downstream performance turns out to be a harder test for diffused redundancy. Thus, in the rest of the paper, we examine diffused redundancy through the lens of downstream transfer performance and include CKA results in Appendix A. 2.2 Understanding Why Many Random Subsets Work 0.0 0.2 0.4 0.6 0.8 Fraction of Neurons (Total=2048) cifar10 cifar100 flowers oxford-iiit-pets (a) Standard Training; CKA between two random subsets 0.0 0.2 0.4 0.6 0.8 Fraction of Neurons (Total=2048) cifar10 cifar100 flowers oxford-iiit-pets (b) Adversarial Training; CKA between two random subsets 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Neurons Transfer Accuracy cifar10 cifar100 flowers oxford-iiit-pets (c) Standard Training; PCA top (dotted) and bottom (dashed) 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Neurons Transfer Accuracy cifar10 cifar100 flowers oxford-iiit-pets (d) Adversarial Training; PCA topk (dotted) and bottom-k (dashed) Figure 2: [Why Many Random Subsets Work] (2a& 2b) Similarity between any two randomly picked sets of k% neurons becomes fairly high (for a critical mass of k%), thus showing that any random pick beyond this threshold is likely to perform similarly; (2c& 2d) Performance of a linear probe trained for a downstream task on randomly chosen neurons (solid lines) closely follows that of a linear probe trained on a projection on top k principal components (dotted lines) for a sufficiently large value of k. 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Neurons Transfer Accuracy cifar10 cifar100 flowers oxford-iiit-pets 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Neurons Transfer Accuracy cifar10 cifar100 flowers oxford-iiit-pets Figure 3: [Random Projection (dotted) vs Randomly Choosing Neurons (solid)] We find that diffused redundancy (randomly choosing neurons) performs significantly better than random projections. This indicates that the constrained projection offered by diffused redundancy is crucial in achieving good downstream performance. Many prior works explicitly train models to have small representations (e.g. [26, 65, 64, 5] with the goal of efficient downstream learning. These works show that, when explicitly optimized, networks can perform downstream classification with fewer neurons than typically used in state-of-the-art architectures. We show, however, that such subsets already exist in models that are not explicitly trained for this goal, and in fact, one doesn t even have to try hard to find this subset; it can be randomly chosen. Later in section 3.3 we compare some of these efficient representation learning methods to randomly chosen subsets and carefully analyze the tradeoffs involved. Here, however, we seek to better understand why there exist so many randomly selected subsets that work just as well as the whole layer. High CKA between two random subsets of the same size. First, we use representation similarity (CKA) [24] to get a sense of how two random picks of neurons (of the same size) relate to each other. Concretely, we calculate CKA between two random picks of k% neurons in the penultimate layer (averaged over 10 such randomly picked pairs) on samples taken from different datasets. Fig 2a& 2b show CKA results averaged over these different picks of pairs of subsets of the full set of neurons. We see that after picking a certain threshold, i.e. for a large enough value of k, the similarity between any two randomly picked pairs of heads is fairly high. For example, for the adversarially trained Res Net50 (Fig 2b), we observe that any 10% of neurons picked from the penultimate layer are highly similar (CKA of about 0.8), with very low error bars. A similar value of CKA is obtained with 20% of neurons for the standard Res Net50 model. These results indicate that, given a sufficient size, picking any random subset of that size has very similar representations and thus provides an initial intuition for why almost any random subset, with high probability, could work equally well. Downstream performance on k randomly chosen neurons closely follows performance on top k principal components. While high representation similarity is a necessary condition for two representations to have similar downstream performance, it s not a sufficient one. As seen earlier in Fig 1, similar values of CKA can still show different downstream accuracy. Thus, to further understand why any random pick of neurons achieves similar accuracy, we compare a randomly chosen subset of neurons with a projection on the same number of top k principal components. Fig 2c& 2d shows that the performance of a linear probe trained on a random subset of neurons initially lags behind the top PCA dimensions for small values of k(< 20%), but for higher values (> 20%) closely follows the performance on a probe trained on the projection on the same number of top principal components. This indicates that a sufficiently sized random subset of size k (roughly) captures the maximum variance in data that can be captured in k dimensions (upper bound by top k principal components since they are designed to capture maximum variance). As a sanity check, we also show the results for the bottom k principal components (directions of least variance) and see that randomly chosen neurons perform significantly better. Random projections into lower dimensions do not perform well on downstream tasks. The presence of diffused redundancy in a particular layer indicates that a dimension lower than the layer s size suffices to perform many downstream tasks. However, randomly sampling neurons (as we do in our experiments on diffused redundancy) can be seen as one very particular way of reducing dimension that is restricted by what the network has learned. This raises a natural question, would our observation extend to less restricted ways of reducing dimensions? To test this, we compare diffused redundancy i.e., a linear probe trained on a random sample of k neurons to a linear probe trained on a random projection of the layer s activations. To project a d dimensional layer into a lower dimension (k) we multiply it by a (normalized) randomly sampled matrix from a normal distribution Rd k. We report our results over 5 seeds in Fig 3 and find that diffused redundancy significantly outperforms random projection. 3 Factors Influencing The Degree of Diffused Redundancy 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (a) CIFAR10 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (b) CIFAR100 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (c) Flowers 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (d) Oxford-IIIT-Pets vgg16_bn-nonrob vgg16_bn-robustl2eps3 resnet18-nonrob resnet18-robustl2eps3 resnet50-nonrob resnet50-robustl2eps3 wide_resnet50_2-nonrob wide_resnet50_2-robustl2eps3 vit_small_patch16_224-nonrob vit_small_patch32_224-nonrob Figure 4: [Comparisons Across Architectures For Downstream Task Accuracy] All models shown here are pre-trained on Image Net1k. We see that diffused redundancy exists across architectures, and the trend observed in Figure 1c&1a regarding adversarially trained models also holds here as models curves that are more inside are the ones trained with standard loss. In order to better understand the phenomenon of diffused redundancy we analyze 30 different pretrained models, with different architectures, pre-training datasets and losses. We then evaluate each model for transfer accuracy on 4 datasets mentioned in Section 2. While all results in this section are on the penultimate layer, we also report results on intermediate layers in Appendix E and find similar trends of diffused redundancy. All details for reproducibility can be found in Appendix B. Architectures We consider VGG16 [50], Res Net18, Res Net50, Wide Res Net50-2 [16], Vi T-S16 & Vi T-S32 [23]. Additionally, we consider Res Net50 with varying widths of the final layer (denoted by Res Net50 ffx where x denotes the number of neurons in the final layer). Upstream Datasets Image Net-1k & Image Net-21k [45]. Upstream Losses Standard cross-entropy, adversarial training (ℓ2 threat model, ϵ = 3 and ℓ threat model, ϵ = 4/255 4) [34], MRL Loss [26], De Cov [6], and varying strengths of dropout [52]. Downstream Datasets CIFAR10/1000, Oxford-IIIT-Pets, and Flowers, same as Section 2. Additionally, we also report performance on harder datasets such as Image Netv2 [44] and Places365 [69] 4Results for ℓ robust models can be found in Appendix D 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (a) CIFAR10 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (b) CIFAR100 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (c) Flowers 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (d) Oxford-IIIT-Pets vit_small_patch16_224-imagenet vit_small_patch16_224-imagenet21k vit_small_patch32_224-imagenet vit_small_patch32_224-imagenet21k Figure 5: [Comparison Across Upstream Datasets] We see that degree of diffused redundancy depends a great deal on the upstream training dataset, in particular, models trained on Image Net21k exhibit a higher degree of diffused redundancy, although the differences in the degree of diffused redundancy are downstream task dependent in Appendix F. For all analyses in this section, we also report corresponding approximations for DR (Eq 1) in Appendix D. 3.1 Effects of Architecture, Upstream Loss, Upstream Datasets, and Downstream Datasets Extending the analysis in Section 2, we evaluate the diffused redundancy hypothesis on other architectures. Fig 4 shows transfer performance for different architectures. All architectures shown in Fig 4 are trained on Image Net1k. We find that our takeaways from Section 2 also extend to other architectures. Fig 5 compares two instances each of Vi T-S16 and Vi T-S32, one trained on a bigger upstream dataset (Image Net21k) and another on a smaller dataset (Image Net1k) Note that the nature of all curves in both Figs 4&5 highly depends on downstream datasets. This is also consistent with the initial observation of Section 2 about diffused redundancy being downstream dataset dependent. Additionally, we also report results on Image Net V2 and Places365 in Appendix F showing that diffused redundancy also holds on harder datasets. 3.2 Diffused Redundancy as a Function of Layer Width We take the usual Res Net50 with a penultimate layer consisting of 2048 neurons and compare it with variants that are pre-trained with a much smaller penultimate layer, these are denoted by Res Net50 ffx where x (< 2048) is the number of neurons in the penultimate layer. Fig 6 shows how diffused redundancy slowly fades away as we squeeze the layer to be smaller. In fact, for Res Net50 ff8, we see that across all datasets we need > 90% of the full layer to achieve performance close to the full layer. This shows that diffused redundancy only appears in DNNs when the layer is sufficiently wide to encode redundancy. 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (a) CIFAR10 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (b) CIFAR100 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (c) Flowers 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (d) Oxford-IIIT-Pets resnet50 resnet50_ff1024 resnet50_ff512 resnet50_ff256 resnet50_ff128 resnet50_ff64 resnet50_ff32 resnet50_ff16 resnet50_ff8 Figure 6: [Diffused Redundancy as Function of Layer Width] As we make the length of the layer smaller, the degree of redundancy becomes lesser. For Res Net50 ff8, i.e.Res Net50 with only 8 neurons in the final layer, we see that we need almost 90% of neurons to achieve similar accuracy as the full layer. 3.3 Comparison With Methods That Optimize For Lesser Neurons Matryoshka Representation Learning (MRL) is a recently proposed paradigm that learns nested representations such that the first k, 2k, 4k, ..., N (where N = size of the full layer) dimensions of the extracted feature vector are all explicitly made to be good at minimizing upstream loss, with the intuition of learning coarse-to-fine representations. This ensures that one can flexibly use these smaller parts of the representation for downstream tasks. MRL, thus, ensures that redundancy shows up in learned representations in a structured way, i.e., we know the first k, 2k, ... neurons can be picked and used for downstream tasks and should perform reasonably. Here we investigate two questions regarding Matryoshka representations: 1) do these representations also exhibit the phenomenon of diffused redundancy? i.e. if we were to ignore the structure imposed by MRL-type training and instead just pick random neurons from all over the layer, do we still get reasonable performance?, and 2) how do they compare to representations learned by other kinds of losses? Figure 7 investigates these questions by comparing Res Net50 representations learned using MRL loss to other losses. resnet50 mrl nonrob first (red line) denotes a Res Net50 trained using MRL loss and evaluated on parts of the representation that were optimized to have low upstream loss (i.e. first k, 2k, ...N neurons, here k = 8 and N = 2048) and resnet50 mrl nonrob random (green line) refers to the same model with the same number of neurons chosen for evaluation, except they re chosen at random from the entire layer. First, we interestingly see that even the Res Net50 trained with MRL loss exhibits diffused redundancy (denoted by green line spiking very quickly for most datasets in Fig 7), despite having been trained to only have structured redundancy. Based on this observation, we conjecture that diffused redundancy is a natural consequence of having a wide layer. Second, we see that Res Net50 trained on MRL indeed does better in the low neuron regime across datasets (red line on the extreme left part of the plots in Fig 7), but other models quickly catch up as we pick more neurons, thus indicating that major efficiency benefits of MRL-type models are best realized when using an extremely low number of neurons, else one can obtain similar downstream performances by simply picking random samples from existing pre-trained models. 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (a) CIFAR10 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (b) CIFAR100 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (c) Flowers 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (d) Oxford-IIIT-Pets resnet50-nonrob-random resnet50-robustl2eps3-random resnet50_mrl-nonrob-random resnet50_mrl-nonrob-first Figure 7: [Comparison of Diffused Redundancy in MRL vs other losses] Here we compare Res Net50 trained using multiple losses including MRL [26]. nonrob indicates that the model was trained with the standard crossentropy loss while robustl2eps3 indicates adversarial training with ℓ2 threat model and ϵ = 3. Red line shows results for part of the representation explicitly optimized in MRL, whereas the green line shows results for parts that are picked randomly from the same representation. Even the MRL model shows a significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy. 3.4 Methods That Prevent Co-adaptation of Neurons Also Exhibit Diffused Redundancy We consider two additional modifications to upstream pre-training: we add regularization to the usual cross-entropy loss that decorrelates neurons in the feature representation using De Cov [6] and Dropout [52]. Dropout does this implicitly by randomly dropping neurons during training and De Cov does this explicitly by putting a loss on the activations of a given layer. We pre-train different Res Net50 on Image Net1k by varying strengths of dropout ranging from 0.1 all the way up to 0.8 and also by separately adding the De Cov regularizer (regularization strength = 1e 4) to give 9 additional pre-trained models. Intuitively these methods should lead to increased diffused redundancy since by design these methods force the model to disperse similar information in different parts of a layer to perform the same task downstream task. Interestingly, we see that diffused redundancy in these models is almost completely independent of the regularization strength. This suggests that diffused redundancy might be an inevitable property of DNNs when the layer has sufficient width. Due to space constraints, we include these results in Appendix G. 4 Possible Fairness-Efficiency Tradeoffs in Efficient Downstream Transfer One natural use case for diffused redundancy is efficient transfer to downstream datasets. As defined in Eq 1 and as also seen in Sections 2& 3, dropping neurons comes at a small cost (δ in Eq 1) in 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Accuracy resnet50-nonrob resnet50-robustl2eps3 resnet50_mrl-nonrob-first (a) CIFAR10 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Accuracy resnet50-nonrob resnet50-robustl2eps3 resnet50_mrl-nonrob-first (b) CIFAR100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Accuracy resnet50-nonrob resnet50-robustl2eps3 resnet50_mrl-nonrob-first (c) Flowers 0.0 0.2 0.4 0.6 0.8 Accuracy resnet50-nonrob resnet50-robustl2eps3 resnet50_mrl-nonrob-first (d) Oxford-IIIT-Pets Figure 8: [Gini Coefficient of Class-Wise Accuracies as we Drop Neurons] Higher value of Gini coefficient indicates higher inequality [12]. We see that for all models gini coefficients become higher as the accuracy reduces (as a result of dropping neurons). Additionally, in some regions (highlighted in the plots), the model explicitly optimized for efficient transfer (resnet50 mrl) can give rise to higher gini values, resulting in a more unequal spread of accuracy over classes. performance as compared to the full set of neurons. Here we take a deeper look into this drop in overall performance and investigate how it is distributed across classes. If the drop affects only a few classes, then dropping neurons although efficient for downstream tasks could have implications for fairness, which is not only of concern to ML researchers and practitioners [66, 14, 18], but also to lawyers [56] and policymakers [57]. We compare the spread of accuracies across classes using inequality indices, which are commonly used in economics to study income inequality [8, 47] and have also recently been adopted in the fair ML literature [51]. We use gini index [12] and coefficient of variation [27] to quantify the spread of performance across classes. For a perfect spread, both gini and coefficient of variation are 0, and higher values indicate higher inequality. Figure 8 compares the gini index for various models at varying levels of accuracy (note that accuracy monotonically increases with more neurons, hence the right most point for a model represents the model with all neurons). We make two observations: across all datasets and all models we find that a loss in accuracy (compared to the full layer) comes at the cost of a few classes, as opposed to being smeared throughout classes, as indicated by high gini values on the left of each plot. Additionally, we observe that the model trained using MRL loss tends to have slightly higher gini values in the regions where the drop in accuracy is slightly higher (highlighted on the plots). To ensure that this trend is not simply due to lower accuracy, we investigate the error distributions across classes (Appendix C) and find that predictions become more homogeneous as we drop more neurons. Similar trends are also observed with coeff. of variation as shown in Appendix C. These results draw caution to the potential unintended side-effects of exploiting diffused redundancy and suggest that there could be a possible fairness-efficiency tradeoff involved. Connections to Robustness-Fairness Tradeoffs There are well-established tradeoffs between robustness and fairness in both standard [36] and adversarial training [62, 58]. One major difference between these works and our setup is that the task they train on is also the task they test on. We instead operate in the transfer learning setup where we train a linear probe on the representation of a pre-trained model. While it s not immediately clear whether the discrepancy between class accuracies observed on the pre-training task (e.g. as in [62]) also leads to an inter-class accuracy discrepancy on downstream tasks when dropping neurons, it would be very interesting future work to establish more formal connections between the robustness-fairness tradeoff and its effects on the fairness-efficiency tradeoff presented in this paper. 5 Conclusion and Broader Impacts We introduce the diffused redundancy hypothesis and analyze a wide range of models with different upstream training datasets, losses, and architectures. We carefully analyze the causes of such redundancy and find that upstream training (both loss and datasets) plays a crucial role and that this redundancy also depends on the downstream dataset. One direct practical consequence of our observation is increased efficiency for downstream training times which can have many positive impacts in terms of reduced energy costs [54] which is crucial in moving towards green AI [48]. We, however, also draw caution to potential pitfalls of such efficiency gains , which might hurt the accuracy of certain classes more than others, thus having direct consequences for fairness. Acknowledgements VN and JPD were supported in part by NSF CAREER Award IIS-1846237, NSF D-ISN Award #2039862, NSF Award CCF-1852352, NIH R01 Award NLM013039-01, NIST MSE Award #20126334, DARPA GARD #HR00112020007, Do D WHS Award #HQ003420F0035, ARPA-E Award #4334192 and a Google Faculty Research Award. VN, TS, and KPG were supported in part by an ERC Advanced Grant Foundations for Fair Social Computing (no. 789373). AW acknowledges support from a Turing AI Fellowship under grant EP/V025279/1 and the Leverhulme Trust via CFI. SF was supported in part by a grant from an NSF CAREER AWARD 1942230, ONR YIP award N00014-22-1-2271, ARO s Early Career Program Award 310902-00001, Meta grant 23010098, HR00112090132 (DARPA/RED), HR001119S0026 (DARPA/GARD), Army Grant No. W911NF2120076, the NSF award CCF2212458, an Amazon Research Award, and an award from Capital One. [1] Alessandro Achille and Stefano Soatto. On the emergence of invariance and disentangling in deep representations. ar Xiv preprint ar Xiv:1706.01350, 125:126 127, 2017. [2] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes, 2018. [3] Fred Attneave. Some informational aspects of visual perception. Psychological review, 61(3):183, 1954. [4] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798 1828, 2013. [5] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. ar Xiv preprint ar Xiv:1908.09791, 2019. [6] Michael Cogswell, Faruk Ahmed, Ross Girshick, Larry Zitnick, and Dhruv Batra. Reducing overfitting in deep networks by decorrelating representations. ar Xiv preprint ar Xiv:1511.06068, 2015. [7] Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4908 4926, Online, November 2020. Association for Computational Linguistics. [8] Fernando G De Maio. Income inequality measures. Journal of Epidemiology & Community Health, 61(10):849 852, 2007. [9] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layerwise optimal brain surgeon. Advances in Neural Information Processing Systems, 30, 2017. [10] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam Mc Candlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. Toy models of superposition. Transformer Circuits Thread, 2022. [11] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. Open Review.net, 2019. [12] Corrado Gini. Measurement of inequality of incomes. The economic journal, 31(121):124 126, 1921. [13] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015. [14] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016. [15] Babak Hassibi and David Stork. Second order derivatives for network pruning: Optimal brain surgeon. Advances in neural information processing systems, 5, 1992. [16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770 778, 2016. [17] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389 1397, 2017. [18] Kenneth Holstein, Jennifer Wortman Vaughan, Hal Daum e III, Miro Dudik, and Hanna Wallach. Improving fairness in machine learning systems: What do industry practitioners need? In Proceedings of the 2019 CHI conference on human factors in computing systems, pages 1 16, 2019. [19] Yoshio Izui and Alex Pentland. Analysis of neural networks with redundancy. Neural Computation, 2(2):226 238, 1990. [20] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pages 2668 2677. PMLR, 2018. [21] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages 5338 5348. PMLR, 2020. [22] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision, pages 491 507. Springer, 2020. [23] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. 2021. [24] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pages 3519 3529. PMLR, 2019. [25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. [26] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representations for adaptive deployment. ar Xiv preprint ar Xiv:2205.13147, 2022. [27] Barbara S Lawrence. Perspective the black box of organizational demography. Organization science, 8(1):1 22, 1997. [28] Yann Le Cun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 444, 2015. [29] Yann Le Cun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989. [30] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip HS Torr. Snip: Single-shot network pruning based on connection sensitivity. ar Xiv preprint ar Xiv:1810.02340, 2018. [31] Asriel Levin, Todd Leen, and John Moody. Fast pruning using principal components. Advances in neural information processing systems, 6, 1993. [32] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018. [33] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. ar Xiv preprint ar Xiv:1608.08710, 2016. [34] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks, 2019. [35] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them, 2014. [36] Vedant Nanda, Samuel Dooley, Sahil Singla, Soheil Feizi, and John P Dickerson. Fairness through robustness: Investigating robustness disparity in deep learning. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 466 477, 2021. [37] Vedant Nanda, Till Speicher, Camilla Kolling, John P. Dickerson, Krishna P. Gummadi, and Adrian Weller. Measuring representational robustness of neural networks through shared invariances. In ICML, 2022. [38] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008. [39] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. https://distill.pub/2020/circuits/zoom-in. [40] Chris Olah, Alexander Mordvintsev, and Ludwig Schubert. Feature visualization. Distill, 2017. https://distill.pub/2017/feature-visualization. [41] Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10):1345 1359, 2009. [42] Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40):24652 24663, 2020. [43] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012. [44] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389 5400. PMLR, 2019. [45] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 252, 2015. [46] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? Advances in Neural Information Processing Systems, 33:3533 3545, 2020. [47] Robert R Schutz. On the measurement of income inequality. The American Economic Review, 41(1):107 122, 1951. [48] Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(12):54 63, 2020. [49] Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. ar Xiv preprint ar Xiv:1703.00810, 2017. [50] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. ar Xiv preprint ar Xiv:1409.1556, 2014. [51] Till Speicher, Hoda Heidari, Nina Grgic-Hlaca, Krishna P Gummadi, Adish Singla, Adrian Weller, and Muhammad Bilal Zafar. A unified approach to quantifying algorithmic unfairness: Measuring individual &group unfairness via inequality indices. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 2239 2248, 2018. [52] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929 1958, 2014. [53] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. ar Xiv preprint ar Xiv:2106.10270, 2021. [54] Emma Strubell, Ananya Ganesh, and Andrew Mc Callum. Energy and policy considerations for deep learning in NLP. In Anna Korhonen, David R. Traum, and Llu ıs M arquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28August 2, 2019, Volume 1: Long Papers, pages 3645 3650. Association for Computational Linguistics, 2019. [55] Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang, and Chunfang Liu. A survey on deep transfer learning. In International conference on artificial neural networks, pages 270 279. Springer, 2018. [56] Song ul Tolan, Marius Miron, Emilia G omez, and Carlos Castillo. Why machine learning may lead to unfairness: Evidence from risk assessment for juvenile justice in catalonia. In Proceedings of the Seventeenth International Conference on Artificial Intelligence and Law, pages 83 92, 2019. [57] Michael Veale, Max Van Kleek, and Reuben Binns. Fairness and accountability design needs for algorithmic support in high-stakes public sector decision-making. In Proceedings of the 2018 chi conference on human factors in computing systems, pages 1 14, 2018. [58] Zeming Wei, Yifei Wang, Yiwen Guo, and Yisen Wang. Cfa: Class-wise calibrated fair adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8193 8201, 2023. [59] Ross Wightman. Pytorch image models. https://github.com/rwightman/ pytorch-image-models, 2019. [60] Ross Wightman, Hugo Touvron, and Herv e J egou. Resnet strikes back: An improved training procedure in timm. abs/2110.00476, 2021. [61] Eric Wong, Shibani Santurkar, and Aleksander Madry. Leveraging sparse linear layers for debuggable deep networks. In International Conference on Machine Learning, pages 11205 11216. PMLR, 2021. [62] Han Xu, Xiaorui Liu, Yaxin Li, Anil Jain, and Jiliang Tang. To be robust or to be fair: Towards fairness in adversarial training. In International conference on machine learning, pages 11492 11501. PMLR, 2021. [63] Jason Yosinski, Jeff Clune, Anh Nguyen, Thomas Fuchs, and Hod Lipson. Understanding neural networks through deep visualization, 2015. [64] Jiahui Yu and Thomas S Huang. Universally slimmable networks and improved training techniques. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1803 1811, 2019. [65] Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas S. Huang. Slimmable neural networks. abs/1812.08928, 2018. [66] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rogriguez, and Krishna P Gummadi. Fairness constraints: Mechanisms for fair classification. In Artificial intelligence and statistics, pages 962 970. PMLR, 2017. [67] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks, 2013. [68] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. ar Xiv preprint ar Xiv:1910.04867, 2019. [69] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. A Measuring Diffused Redundancy A.1 CKA Definition In all our evaluations we use CKA with a linear kernel [24] which essentially amounts to the following steps: 1. Take two representations Y Rn d1 and Z Rn d2 2. Compute dot product similarity within these representation, i.e.compute K = Y Y T , L = ZZT 3. Normalize K and L to get K = HKH, L = HLH where H = In 1 4. Return CKA(Y, Z) = HSIC(K,L) HSIC(K,K)HSIC(L,L), where HSIC(K, L) = 1 (n 1)2 (flatten(K ) flatten(L )) We use the publicly available implementation of [37], which provides an implementation that can be calcuated over multiple mini-batches: https://github.com/nvedant07/STIR A.2 Additional CKA results Fig 9 shows CKA comparison between randomly chosen parts of the layer and the full layer for different kinds of Res Net50. We observe that even Res Net50 trained with MRL loss shows a significant amount of diffused redundancy. 0.0 0.1 0.2 0.3 0.4 0.5 Fraction of Neurons (Total=2048) resnet50-nonrob-random resnet50-robustl2eps3-random resnet50_mrl-nonrob-first resnet50_mrl-nonrob-random (a) CIFAR10 0.0 0.1 0.2 0.3 0.4 0.5 Fraction of Neurons (Total=2048) resnet50-nonrob-random resnet50-robustl2eps3-random resnet50_mrl-nonrob-first resnet50_mrl-nonrob-random (b) CIFAR100 0.0 0.1 0.2 0.3 0.4 0.5 Fraction of Neurons (Total=2048) resnet50-nonrob-random resnet50-robustl2eps3-random resnet50_mrl-nonrob-first resnet50_mrl-nonrob-random (c) Flowers 0.0 0.1 0.2 0.3 0.4 0.5 Fraction of Neurons (Total=2048) resnet50-nonrob-random resnet50-robustl2eps3-random resnet50_mrl-nonrob-first resnet50_mrl-nonrob-random (d) Oxford-IIIT-Pets Figure 9: [Comparison of Diffused Redundancy in MRL vs other losses, through the lens of CKA] We see a similar trend as reported in Fig 7 in the main paper, where even the MRL model shows a significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy. The amount of diffused redundancy however is much lesser than the resnets trained using the standard loss and adv. training as denoted by a much lower red line across all datasets. B Training and Pre-Processing Details for Reproducibility Here we list the sources of weights for the various pre-trained models used in our experiments: Res Net18 trained on Image Net1k using standard loss: taken from timm v0.6.1. Res Net18 trained on Image Net1k with adv training: taken from Salman et al. [46]: Res Net50 trained on Image Net1k using standard loss: taken from timm v0.6.1. Res Net50 trained on Image Net1k with adv training: taken from Salman et al. [46]: https: //github.com/microsoft/robust-models-transfer. Res Net50 trained on Image Net1k using MRL and with different final layer widths (resnet50 ffx): taken from released weights of by Kusupati et al. [26]: https:// github.com/RAIVNLab/MRL. Wide Res Net50-2 on Image Net1k both standard and avd. training: taken from Salman et al. [46]: https://github.com/microsoft/robust-models-transfer. VGG16 trained on Image Net1k with standard loss: taken from timm v0.6.1. VGG16 trained on Image Net1k with adv training: taken from Salman et al. [46]: https: //github.com/microsoft/robust-models-transfer. Vi TS32 & Vi TS16 trained on Image Net21k & Image Net1k: taken from weights released by Steiner et al. [53]: https://github.com/google-research/vision_transformer. All linear probes trained on the representations of these models are trained using SGD with a learning rate of 0.1, momentum of 0.9, batch size of 256, weight decay of 1e 4. The probe is trained for 50 epochs with a learning rate scheduler that decays the learning rate by 0.1 every 10 epochs. Scripts for training can also be found in the attached code. For pre-processing, we re-size all inputs to 224x224 (size used for pre-training) and apply the usual composition of Random Horizontal Flip, Color Jitter(brightness=0.25, contrast=0.25, saturation=0.25, hue=0.25), Random Rotation(degrees=2). All inputs were mean normalized. For imagenet1k pre-trained models: mean = [0.485, 0.456, 0.406] and std = [0.229, 0.224, 0.225]. For imagenet21k pre-trained models: mean = [0.5,0.5,0.5], std = [0.5,0.5,0.5]. C Deeper Analysis of Fairness-Efficiency Tradeoff in Section 4 Analyzing Error Distributions To ensure that the higher gini coefficient shown in Fig 8 as we drop more neurons is not merely an artifact of lower overall accuracy, we plot class-wise accuracies as we drop neurons (Figs 11, 12 & 13). We find that for the entire layer, accuracy starts at an almost uniform distribution, and while overall accuracy deteriorates as we drop neurons, the drop comes at a larger cost for a few classes resulting in disparate inter-class accuracies. Coeff of Variation for Measuring Inequality in Inter-Class Accuracy Fig 10 shows results for the same analysis shown in Fig 8 of the main paper and we find similar takeaways even when using the coefficient of variation as a measure of inequality. D Corresponding Diffused Redundancy Estimates For Analyses in Section 3 & ℓ Robust Model Results Corresponding diffused redundancy (DR) ablations for Figures 4,5,7. These are shown in Figures 15,16,17 respectively. This should allow for easy comparison of diffused redundancy (lines that are more outside have higher DR). For example, Figure 16 clearly shows higher diffused redundancy in models trained on larger upstream datasets (here Image Net21k) since these curves lie more on the outside of the same model s curves for Image Net1k. Additionally, we show numbers on x-axis for Figure 4 in Figure 18. Figures 5 and 7 compare models with same number of neurons in the final layer and hence trends shown with fraction on the x-axis will be exactly the same with absolute numbers on the x-axis. However, Figure 18 allows a direct comparison of the performance of the same absolute number of neurons across different models. We report results for ℓ robust models (with ϵ = 4/255) in Fig 14 and find that ℓ2 model generally shows a greater degree of diffused redundancy. E Results on Intermediate Layers We additionally ran our experiment on other intermediate layers and report the results in Fig 19. We present results for a Res Net50 pretrained on Image Net1k using the standard Cross Entropy loss. The 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Accuracy Coeff. of Variation resnet50-nonrob resnet50-robustl2eps3 resnet50_mrl-nonrob-first (a) CIFAR10 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Accuracy Coeff. of Variation resnet50-nonrob resnet50-robustl2eps3 resnet50_mrl-nonrob-first (b) CIFAR100 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Accuracy Coeff. of Variation resnet50-nonrob resnet50-robustl2eps3 resnet50_mrl-nonrob-first (c) Flowers 0.0 0.2 0.4 0.6 0.8 Accuracy Coeff. of Variation resnet50-nonrob resnet50-robustl2eps3 resnet50_mrl-nonrob-first (d) Oxford-IIIT-Pets Figure 10: [Coefficient of Variation As We Drop Neurons] We see a similar trend as reported in Fig 8 of the main paper where inequality increases as we drop neurons for all models on all datasets. intermediate layers considered are characterized as activations following each residual connection within distinct Res Net blocks. layer X.Y.act3 means the Y th residual connection in the Xth Res Net block and act3 indicates that we re taking the value after the activation (Re LU) has been applied. F Results on Harder Downstream Tasks: Image Net V2 and Places365 We report results on harder downstream tasks such as Image Net1k, Image Net V2, and Places365 in Figure 20. We find that when randomly dropping neurons, the model is still able to generalize to Image Net1k and Places365 with very few neurons, i.e., the phenomena of diffused redundancy observed for smaller datasets, also holds for harder datasets. Interestingly we also observe that the accuracy gap between Image Net1k and Image Net V2 is maintained even as we drop neurons. G Effects of Explicitly Preventing Co-adaptation of Neurons: Analysis of Dropout and De Cov Regularizers such as dropout and De Cov, force different parts of the representations to not be correlated. Thus these regularizers can be seen as explicitly requiring different, compact parts of the representation to be self-contained for the downstream task. Thus, intuitively, such methods should increase diffused redundancy. Here we investigate if our observation about diffused redundancy is influenced by such regularizers. We evaluate Res Net50 pre-trained on Image Net1k with dropout in the penultimate layer ranging from a strength of 0.1 all the way to 0.8. We also train another Res Net50 model with the De Cov regularizer added to the usual crossentropy loss and put a weight of 0.0001 on the regularizer to ensure that its numerical range is similar to that of the cross entropy loss term. Results in Figures 21 & 22 suggest that such regularizers have almost no effect on diffused redundancy. Trends across datasets remain consistent regardless of the strength of dropout or the weight given to De Cov regularizer. This observation further adds to the evidence that diffused redundancy is likely to be a natural property of representations learned by DNNs. Fraction/number of neurons airplane automobile dog frog horse Classwise Accuracies (mean) 0.08 0.06 0.30 0.42 0.38 0.39 0.39 0.50 0.63 0.65 0.73 0.74 0.80 0.81 0.85 0.83 0.88 0.87 0.90 0.91 0.92 0.92 0.00 0.02 0.02 0.15 0.22 0.23 0.36 0.56 0.68 0.73 0.79 0.86 0.87 0.89 0.91 0.90 0.92 0.91 0.93 0.94 0.94 0.94 0.19 0.09 0.28 0.31 0.34 0.33 0.40 0.45 0.54 0.55 0.62 0.63 0.71 0.70 0.76 0.75 0.79 0.78 0.81 0.83 0.84 0.83 0.17 0.26 0.17 0.13 0.16 0.17 0.23 0.27 0.36 0.40 0.50 0.49 0.59 0.60 0.69 0.65 0.72 0.71 0.76 0.79 0.79 0.80 0.01 0.17 0.19 0.19 0.24 0.24 0.27 0.32 0.40 0.44 0.57 0.58 0.68 0.69 0.77 0.75 0.80 0.79 0.83 0.85 0.85 0.86 0.06 0.04 0.07 0.23 0.23 0.30 0.38 0.43 0.55 0.62 0.68 0.72 0.75 0.77 0.80 0.80 0.82 0.83 0.85 0.86 0.86 0.86 0.01 0.34 0.40 0.33 0.36 0.35 0.51 0.65 0.68 0.72 0.77 0.81 0.86 0.87 0.91 0.91 0.92 0.92 0.94 0.95 0.95 0.95 0.10 0.09 0.10 0.25 0.25 0.29 0.39 0.62 0.64 0.69 0.74 0.74 0.80 0.82 0.85 0.84 0.87 0.87 0.89 0.89 0.90 0.90 0.00 0.03 0.14 0.09 0.12 0.18 0.37 0.53 0.58 0.61 0.73 0.79 0.84 0.85 0.87 0.87 0.89 0.90 0.92 0.93 0.93 0.93 0.60 0.42 0.19 0.22 0.22 0.26 0.40 0.49 0.66 0.74 0.82 0.83 0.87 0.87 0.90 0.89 0.91 0.91 0.91 0.92 0.93 0.93 (a) CIFAR10, Standard Fraction/number of neurons airplane automobile dog frog horse Classwise Accuracies (mean) 0.11 0.02 0.14 0.14 0.24 0.31 0.48 0.62 0.71 0.76 0.77 0.81 0.84 0.85 0.87 0.87 0.88 0.88 0.90 0.90 0.91 0.91 0.00 0.04 0.19 0.28 0.38 0.37 0.61 0.73 0.79 0.82 0.84 0.86 0.88 0.89 0.90 0.90 0.91 0.91 0.92 0.92 0.92 0.92 0.14 0.11 0.04 0.04 0.09 0.11 0.27 0.42 0.52 0.57 0.63 0.65 0.72 0.73 0.78 0.78 0.80 0.80 0.82 0.84 0.84 0.84 0.01 0.03 0.07 0.17 0.17 0.16 0.27 0.38 0.45 0.49 0.54 0.57 0.61 0.64 0.66 0.67 0.69 0.69 0.72 0.73 0.73 0.73 0.00 0.02 0.16 0.14 0.16 0.21 0.32 0.42 0.52 0.61 0.65 0.71 0.76 0.78 0.81 0.81 0.82 0.83 0.84 0.85 0.85 0.86 0.14 0.27 0.20 0.24 0.32 0.36 0.50 0.63 0.64 0.68 0.71 0.74 0.76 0.77 0.79 0.79 0.81 0.81 0.82 0.83 0.83 0.83 0.34 0.32 0.27 0.39 0.45 0.48 0.56 0.74 0.80 0.82 0.84 0.88 0.89 0.90 0.91 0.91 0.92 0.93 0.93 0.93 0.93 0.94 0.18 0.41 0.35 0.36 0.34 0.38 0.50 0.64 0.69 0.74 0.78 0.82 0.84 0.86 0.87 0.87 0.88 0.87 0.88 0.89 0.90 0.89 0.07 0.02 0.15 0.17 0.20 0.37 0.53 0.71 0.79 0.86 0.88 0.90 0.90 0.92 0.92 0.93 0.92 0.93 0.93 0.93 0.93 0.93 0.33 0.34 0.65 0.61 0.59 0.54 0.60 0.73 0.78 0.78 0.81 0.85 0.87 0.88 0.89 0.89 0.90 0.90 0.92 0.92 0.92 0.92 (b) CIFAR10, Adversarial Training Fraction/number of neurons apple aquarium_fish bear beaver bed bee beetle bicycle bus butterfly can castle caterpillar chair chimpanzee clock cloud cockroach crab crocodile cup dinosaur dolphin elephant fox girl hamster house kangaroo keyboard lamp lawn_mower lion lizard lobster man maple_tree motorcycle mouse mushroom otter palm_tree pear pickup_truck plain plate poppy porcupine rabbit raccoon ray road rocket sea seal shark shrew skunk skyscraper snail snake spider squirrel streetcar sunflower sweet_pepper tank telephone tiger tractor train trout tulip turtle wardrobe whale willow_tree Classwise Accuracies (mean) 0.000.000.120.110.120.120.120.270.290.320.450.480.610.600.720.700.780.810.830.850.840.84 0.000.000.000.020.020.010.180.170.240.230.370.350.480.480.600.610.670.670.700.740.750.73 0.000.000.000.000.000.000.040.140.170.190.340.380.460.430.520.490.540.550.560.580.590.56 0.000.000.000.000.000.000.050.020.030.030.130.100.220.160.350.270.400.340.510.580.600.62 0.000.000.000.030.040.010.010.010.010.020.080.040.100.100.190.160.260.230.370.450.470.47 0.000.000.010.030.030.010.050.070.090.090.210.190.400.330.540.480.580.570.630.660.660.68 0.000.000.000.000.000.020.020.040.060.070.110.080.260.190.360.310.510.500.600.620.630.63 0.000.000.000.000.040.050.110.100.090.120.240.170.410.330.610.510.660.610.700.730.730.71 0.000.000.000.060.070.090.140.070.060.100.320.230.540.480.710.730.800.810.840.860.870.85 0.000.000.000.000.000.000.070.060.250.320.590.580.710.700.760.760.760.780.790.800.810.79 0.000.000.000.000.000.000.020.020.010.030.080.070.250.250.410.320.450.430.520.560.580.61 0.000.000.000.000.000.000.040.030.090.040.170.150.270.280.350.350.390.390.430.440.450.45 0.000.110.100.080.090.080.030.040.040.100.210.280.540.500.630.580.690.660.750.790.800.79 0.000.000.000.000.000.000.000.070.050.170.260.180.400.240.450.360.520.470.640.670.660.66 0.000.000.000.000.000.000.020.040.120.110.200.250.400.330.500.380.530.460.620.690.710.68 0.000.000.000.000.010.020.030.060.060.070.260.270.410.360.500.380.610.580.680.710.720.71 0.000.000.020.020.020.020.040.110.320.320.470.410.620.620.740.680.770.780.800.810.810.80 0.030.100.190.130.130.090.120.330.330.400.550.460.630.630.720.740.740.740.790.800.810.83 0.000.000.000.000.000.000.040.070.150.210.360.390.490.510.570.580.590.620.620.660.670.68 0.080.050.040.040.060.030.040.170.220.260.310.390.520.530.620.590.650.630.710.750.760.77 0.010.000.080.060.160.210.380.310.550.580.750.720.800.850.840.860.850.870.850.870.870.88 0.420.150.210.110.070.060.160.290.300.390.520.510.680.660.730.730.740.770.790.810.810.80 0.000.000.150.230.180.230.130.370.510.500.530.520.630.610.700.680.750.720.790.800.800.80 0.000.000.020.040.050.020.090.050.140.130.230.170.270.230.360.330.450.450.530.560.560.56 0.000.040.010.000.040.010.040.170.290.340.490.560.740.800.770.800.770.800.790.790.800.82 0.000.000.010.010.010.010.010.000.010.010.030.000.040.030.230.090.340.210.450.540.580.55 0.000.020.030.030.020.020.120.090.100.100.190.180.260.200.400.360.500.450.550.590.600.60 0.000.000.000.000.000.020.010.020.020.070.120.100.190.170.290.250.360.310.430.480.510.47 0.000.000.000.010.090.090.230.260.270.320.450.520.640.740.750.780.760.790.790.840.840.84 0.000.000.000.000.000.000.000.050.170.210.290.310.440.440.550.510.560.550.610.660.660.66 0.000.000.010.020.030.030.060.020.060.090.180.190.280.240.360.330.420.400.440.470.490.47 0.000.000.000.000.000.000.000.010.060.080.140.150.390.360.590.550.630.670.690.700.710.70 0.000.000.000.000.000.000.000.000.000.000.010.000.050.050.160.090.260.190.390.470.490.49 0.000.000.000.020.030.010.020.030.090.120.150.230.230.250.270.270.330.310.400.460.470.46 0.000.000.000.000.000.020.020.290.230.260.400.420.540.540.620.680.680.680.710.740.750.73 0.000.000.000.000.000.000.070.040.040.040.110.100.190.230.290.310.320.330.370.410.420.40 0.000.000.000.130.140.140.240.230.350.390.430.470.620.720.700.760.700.740.700.710.720.71 0.000.010.000.010.000.050.110.080.110.150.200.200.280.190.440.370.530.490.620.660.660.65 0.000.000.050.080.100.100.090.100.140.120.220.180.290.260.390.380.500.450.560.570.560.56 0.140.140.140.140.150.180.340.370.430.430.600.610.730.740.780.760.820.800.840.860.870.87 0.000.000.000.000.000.000.020.030.020.030.080.200.340.280.490.410.560.540.590.620.630.61 0.000.040.030.030.130.120.310.350.390.370.550.520.650.610.720.670.760.770.800.800.800.80 0.000.000.090.070.040.020.010.010.070.150.330.330.470.410.590.600.640.640.680.760.780.79 0.000.000.030.000.050.030.190.120.240.290.410.560.590.650.630.630.640.670.680.700.700.70 0.000.000.010.010.000.000.010.020.010.020.070.110.190.140.320.240.450.450.520.570.560.57 0.000.040.020.030.030.020.040.040.060.120.140.140.140.150.210.170.260.230.370.440.460.47 0.000.000.000.000.050.040.060.050.080.180.250.340.500.530.560.550.560.580.620.650.640.64 0.000.000.000.010.010.010.020.000.010.060.120.070.220.170.350.260.390.340.420.450.450.44 0.000.000.020.190.310.300.260.330.480.460.540.590.610.720.740.770.760.750.780.840.850.87 0.000.000.000.000.020.040.060.040.050.070.410.340.570.550.690.670.720.720.770.790.790.79 0.000.000.010.000.000.000.000.000.000.000.000.000.020.050.150.080.240.140.320.360.390.40 0.000.000.000.000.000.000.040.160.180.260.320.340.500.520.620.580.660.630.690.730.730.71 0.000.000.000.020.090.110.170.260.290.280.350.380.370.430.550.610.620.630.620.630.640.66 0.000.020.020.020.020.070.100.280.450.460.620.760.800.780.890.890.930.920.960.970.980.98 0.000.000.020.070.080.050.300.440.390.400.490.580.600.580.640.660.710.720.760.810.820.81 0.000.000.000.000.000.000.000.010.000.000.030.010.030.010.080.040.140.090.240.310.320.28 0.000.000.000.000.010.010.080.040.100.110.290.160.350.420.600.490.630.570.710.780.790.80 0.000.000.000.000.000.000.000.060.130.200.220.200.320.240.480.410.580.530.660.710.720.73 0.000.000.000.000.000.000.100.110.310.390.610.710.750.810.730.790.730.740.720.750.750.74 0.000.000.000.000.000.000.050.040.040.040.060.030.080.090.190.120.250.190.350.430.430.41 0.000.000.000.000.000.120.190.230.460.500.490.570.530.520.600.620.640.620.670.680.690.68 0.090.080.110.150.160.170.370.320.370.410.500.520.610.700.650.720.660.690.650.650.660.66 0.000.030.040.050.040.030.120.130.220.300.310.360.430.470.540.500.560.540.630.700.690.66 0.000.000.010.010.040.020.030.000.070.090.170.110.210.400.470.420.490.460.560.610.620.60 0.000.000.030.000.010.000.020.020.040.050.080.060.150.170.300.230.360.310.430.470.470.46 0.000.000.000.000.000.000.000.000.000.000.000.000.110.110.250.190.340.260.490.570.590.59 0.000.130.000.130.080.080.060.050.100.100.190.220.330.280.390.350.460.440.530.590.610.59 0.000.000.070.020.000.000.050.010.010.020.050.040.160.080.270.220.330.280.430.490.510.51 0.000.150.150.140.140.150.160.160.190.190.250.260.510.450.720.710.760.770.840.870.870.87 0.000.000.000.060.000.000.090.180.200.240.470.460.600.590.650.640.690.680.720.730.720.72 0.000.000.000.060.060.040.110.130.100.180.240.340.410.430.600.540.630.600.690.740.750.77 0.000.160.030.020.210.100.230.450.420.400.440.480.460.390.570.530.620.610.670.690.710.71 0.000.000.160.290.290.280.270.200.240.220.290.170.240.130.230.170.280.230.330.350.360.35 0.000.000.000.000.000.000.000.120.260.250.270.400.390.400.380.380.430.430.500.540.560.55 0.000.230.000.040.060.030.020.010.010.070.090.090.140.120.190.120.240.190.300.340.360.34 0.000.000.000.000.010.000.090.050.180.220.410.400.610.520.690.650.750.750.800.820.830.82 0.010.010.040.010.010.170.040.050.110.110.280.180.430.390.600.570.680.660.750.770.780.77 0.000.000.000.000.000.000.000.160.170.170.200.320.380.340.500.430.570.510.610.650.650.63 0.000.000.000.000.000.010.040.130.090.120.280.320.490.610.680.690.710.720.740.750.750.73 0.000.000.000.000.000.000.100.120.110.130.250.200.450.470.560.520.610.570.600.630.640.63 0.240.210.080.130.070.040.030.110.100.090.150.120.210.170.360.280.390.350.420.460.470.48 0.080.040.050.080.060.040.100.200.240.410.400.430.450.440.520.450.610.590.660.700.700.71 0.020.250.300.060.040.170.300.350.340.340.530.600.730.790.820.830.850.850.870.880.880.89 0.010.000.010.060.010.060.020.050.040.140.240.210.380.480.570.540.630.610.660.690.690.69 0.000.020.060.050.050.100.030.030.120.150.150.130.180.180.310.200.400.300.550.630.630.61 0.000.000.000.000.010.040.010.030.060.050.200.080.330.380.550.450.620.580.670.690.710.71 0.000.000.240.220.250.240.420.520.480.540.570.600.700.700.730.710.750.750.790.820.820.82 0.000.000.010.040.020.030.160.130.270.290.460.480.670.720.820.780.830.790.860.880.890.87 0.000.000.000.010.010.010.090.150.260.290.430.430.520.510.610.570.610.600.640.670.670.67 0.000.000.000.000.000.070.090.130.190.220.370.260.570.510.770.650.810.750.840.860.860.87 0.000.000.000.000.000.000.000.000.000.000.080.040.190.110.390.250.460.440.610.660.670.67 0.000.000.000.010.000.000.010.130.110.110.170.190.310.250.450.310.520.450.620.680.680.67 0.000.000.000.000.000.000.040.060.090.140.180.180.270.270.390.320.440.400.470.530.520.53 0.000.000.000.000.000.000.030.030.060.060.130.080.230.140.300.190.360.290.460.560.560.53 0.170.190.230.220.300.340.420.610.620.590.580.640.660.700.690.730.740.720.780.830.840.83 0.160.000.100.070.080.090.130.220.220.250.350.430.500.520.570.590.610.610.640.660.660.63 0.000.020.060.150.120.250.150.090.140.180.120.100.130.100.130.170.180.180.280.320.310.30 0.000.010.090.050.070.110.110.150.190.250.310.290.420.380.560.550.630.620.700.710.710.72 0.000.000.000.000.000.000.010.010.010.020.100.050.170.170.310.240.360.320.450.530.550.55 0.000.000.000.010.010.010.130.130.170.290.500.460.650.640.720.710.740.740.790.810.810.81 (c) CIFAR100, Standard Fraction/number of neurons apple aquarium_fish bear beaver bed bee beetle bicycle bus butterfly can castle caterpillar chair chimpanzee clock cloud cockroach crab crocodile cup dinosaur dolphin elephant fox girl hamster house kangaroo keyboard lamp lawn_mower lion lizard lobster man maple_tree motorcycle mouse mushroom otter palm_tree pear pickup_truck plain plate poppy porcupine rabbit raccoon ray road rocket sea seal shark shrew skunk skyscraper snail snake spider squirrel streetcar sunflower sweet_pepper tank telephone tiger tractor train trout tulip turtle wardrobe whale willow_tree Classwise Accuracies (mean) 0.000.000.020.010.050.050.320.440.570.640.700.730.790.800.830.820.840.840.850.860.850.86 0.060.060.050.050.090.090.230.310.480.510.560.600.650.630.700.690.730.720.750.760.770.75 0.050.070.060.080.100.100.190.250.260.340.430.420.530.520.580.580.600.610.600.620.620.64 0.000.000.000.000.000.000.030.030.100.230.300.330.430.420.490.430.530.460.530.540.540.53 0.000.000.000.000.000.000.030.010.060.080.190.180.280.260.340.310.380.350.430.470.490.49 0.000.000.010.030.050.050.120.180.270.340.480.460.560.550.630.580.650.620.670.680.690.70 0.000.000.050.050.040.100.090.120.190.340.490.500.600.600.660.650.690.660.730.740.750.74 0.010.020.070.170.250.220.310.350.570.590.690.650.710.700.770.730.780.780.810.810.840.86 0.000.000.000.000.000.010.050.110.290.390.540.560.660.670.720.710.740.720.750.760.770.76 0.000.140.140.190.170.170.300.570.670.680.700.720.740.760.770.770.770.780.780.780.780.78 0.000.000.000.000.000.010.080.150.280.300.410.400.450.440.460.460.450.470.460.450.460.47 0.000.010.030.060.060.080.230.230.330.330.350.370.390.410.430.430.460.450.490.490.500.49 0.000.000.000.010.000.000.010.260.310.380.520.540.650.640.730.700.760.710.780.800.800.80 0.000.040.100.110.100.120.200.270.290.340.390.400.470.450.520.480.550.510.560.580.600.57 0.000.000.070.050.080.090.170.170.330.430.500.550.650.650.710.680.730.710.740.750.750.75 0.000.000.000.000.000.000.050.070.200.260.430.480.600.590.660.650.680.650.690.710.710.72 0.000.000.000.020.030.040.280.440.610.610.670.670.740.730.780.750.810.780.810.830.840.86 0.000.170.130.070.120.140.310.360.540.630.650.700.720.730.730.750.750.740.750.750.750.75 0.000.000.020.030.040.110.260.330.410.420.430.490.530.560.590.590.590.610.620.620.620.63 0.010.000.010.000.000.010.030.150.250.370.470.480.560.570.620.610.640.610.660.670.670.67 0.000.120.080.160.170.380.540.780.830.820.800.830.810.830.840.850.840.840.830.830.830.84 0.140.070.030.030.130.200.380.580.640.680.690.740.770.790.810.810.820.810.810.840.850.84 0.000.000.020.050.070.070.240.430.550.560.590.610.680.680.730.720.750.740.760.790.780.78 0.000.060.030.080.160.180.270.320.410.390.530.530.640.640.660.660.660.670.670.670.680.67 0.130.130.260.250.350.420.550.760.770.740.720.740.740.750.750.760.760.760.760.750.760.76 0.000.000.000.000.000.000.070.080.100.150.310.280.440.400.530.460.560.540.580.580.560.57 0.010.060.060.090.030.040.160.260.290.370.450.470.540.560.620.590.670.650.700.710.720.71 0.000.000.010.010.010.010.030.030.090.190.320.340.430.430.490.490.540.550.540.530.520.54 0.000.010.020.020.060.080.390.510.590.590.650.700.720.730.780.760.790.790.770.790.790.79 0.000.020.050.070.020.160.240.390.420.470.520.500.540.540.600.580.620.600.650.660.680.67 0.000.000.010.020.050.050.240.260.330.360.380.390.460.450.510.470.520.490.520.530.550.54 0.000.000.000.010.010.000.050.120.200.270.360.400.530.520.610.560.630.610.640.660.660.66 0.030.000.000.010.000.000.010.010.060.120.270.260.400.360.480.410.510.500.520.550.560.54 0.000.120.150.140.160.170.100.140.290.340.390.400.450.470.510.480.530.490.540.560.590.57 0.020.000.000.100.140.120.230.280.400.480.550.570.660.630.720.680.740.720.750.750.750.74 0.000.000.010.010.030.020.060.060.080.080.190.170.310.280.380.310.400.370.440.440.450.46 0.000.000.050.100.100.100.220.300.370.430.450.460.520.520.550.520.570.540.590.600.600.60 0.000.000.050.030.000.000.100.160.260.300.420.430.570.570.620.640.640.650.690.690.690.66 0.000.000.000.000.000.000.030.090.160.240.360.350.450.430.490.480.500.520.540.520.540.56 0.000.090.080.090.020.050.350.500.570.630.690.700.740.750.780.780.790.780.800.800.810.80 0.000.000.000.000.000.000.090.260.390.410.510.520.600.610.670.650.680.680.700.690.690.68 0.000.000.110.200.310.300.570.630.710.710.750.780.790.800.820.820.830.820.840.840.840.84 0.000.000.000.000.010.010.070.140.350.400.460.530.560.590.610.620.630.640.650.640.650.64 0.000.000.020.040.120.140.280.350.440.470.470.500.570.540.590.570.600.580.600.620.620.62 0.000.000.000.000.000.000.070.150.180.230.320.390.460.440.500.490.530.510.560.580.580.57 0.000.000.010.020.000.000.030.090.160.210.250.270.340.320.430.400.470.410.490.520.530.55 0.170.320.250.210.190.220.270.430.480.540.500.540.520.540.520.540.520.510.500.500.500.51 0.000.020.010.010.010.020.060.050.180.250.350.360.420.390.490.440.530.470.540.560.580.59 0.030.040.150.180.220.190.360.460.600.680.690.700.750.750.800.780.810.780.840.850.850.85 0.000.000.000.010.000.020.160.230.460.540.690.710.800.810.830.830.830.840.840.840.840.83 0.010.000.000.000.000.000.000.010.020.020.140.110.230.220.310.260.350.300.370.400.420.40 0.000.000.010.040.050.080.110.240.270.330.430.470.620.600.710.680.730.690.760.770.780.77 0.150.000.000.010.170.220.490.480.560.610.710.730.710.720.710.720.710.720.710.690.680.66 0.000.010.070.100.160.270.450.620.700.750.830.840.860.880.890.900.900.920.930.920.930.92 0.000.020.040.100.080.150.250.330.390.510.610.570.670.680.720.700.740.750.760.740.750.74 0.000.000.000.000.000.000.000.010.020.040.100.130.210.190.270.220.300.280.310.310.310.32 0.000.000.010.030.000.050.110.210.370.370.460.480.600.590.640.620.660.630.680.720.730.73 0.000.000.010.000.010.010.080.190.300.340.460.470.630.600.700.680.720.700.720.730.730.73 0.000.000.030.040.150.120.410.550.590.600.620.660.700.720.760.730.780.770.790.800.800.79 0.000.010.000.010.010.020.040.050.070.100.230.210.380.350.470.420.510.480.530.530.550.53 0.000.000.070.050.010.050.310.420.570.630.710.730.800.790.840.830.850.860.860.870.870.88 0.000.000.000.010.050.050.190.340.440.470.520.540.580.600.610.610.640.640.640.670.650.66 0.000.050.110.130.180.270.380.480.510.500.530.570.640.610.690.670.710.690.740.750.750.75 0.000.000.000.000.000.000.040.020.080.190.290.250.430.360.480.420.520.500.540.570.560.56 0.030.000.000.040.020.020.080.100.160.240.240.290.330.300.380.340.400.400.410.430.440.42 0.000.000.000.000.000.000.030.020.040.100.180.150.290.260.350.320.390.340.410.440.430.44 0.000.000.000.000.030.030.090.140.280.370.430.430.530.520.580.580.600.590.620.650.640.64 0.000.000.070.050.040.080.140.240.280.340.380.400.480.450.530.510.550.540.560.570.570.57 0.000.000.020.040.060.050.180.290.410.550.710.670.790.760.830.800.850.820.860.850.860.86 0.000.000.040.120.090.120.210.400.520.640.670.700.750.790.780.790.790.790.780.780.780.78 0.000.300.210.230.220.200.240.300.400.470.620.620.670.690.720.730.730.730.740.740.740.75 0.000.020.040.030.050.080.120.310.420.500.590.650.690.720.720.730.740.730.720.730.730.71 0.000.000.000.000.000.000.000.010.030.060.140.130.220.220.300.260.340.280.380.430.430.44 0.000.000.070.130.260.240.290.300.380.410.440.420.510.460.580.490.620.560.640.650.630.66 0.000.050.000.010.010.030.050.020.070.130.230.230.300.310.380.370.410.390.440.440.440.42 0.000.000.000.130.370.410.540.650.720.750.790.790.830.850.860.850.870.870.870.890.900.89 0.010.050.060.110.190.190.240.400.430.480.580.560.670.660.730.710.760.760.800.800.800.79 0.000.000.000.000.000.000.010.040.090.170.340.370.480.460.540.510.560.560.560.570.560.56 0.180.030.070.220.240.250.320.380.400.490.610.610.650.660.680.670.680.680.680.690.700.70 0.090.000.000.000.000.000.010.070.170.250.400.330.480.440.530.500.570.520.590.610.600.60 0.100.010.020.010.000.000.020.010.010.020.080.060.160.120.270.190.330.260.370.400.420.40 0.190.110.200.200.110.140.220.310.400.450.490.490.570.550.610.620.640.600.650.650.650.64 0.020.020.060.070.110.310.530.610.700.770.770.800.860.860.880.880.880.880.900.890.880.89 0.020.000.000.040.010.080.240.340.410.420.510.530.570.580.630.600.670.650.710.710.730.72 0.000.000.000.000.000.000.060.120.180.210.350.330.460.450.540.510.570.540.590.610.600.59 0.000.000.000.010.000.010.110.200.400.490.640.660.740.720.800.770.810.800.820.820.820.84 0.000.000.110.150.130.160.340.470.570.590.610.630.680.680.710.720.720.730.730.730.730.72 0.000.020.070.140.160.200.340.440.530.620.660.690.750.760.800.790.810.820.810.820.820.83 0.000.000.020.130.120.130.190.310.450.480.490.540.570.570.610.590.630.600.640.640.630.63 0.000.120.070.050.050.050.120.170.240.390.570.500.720.640.770.720.800.770.800.830.840.83 0.000.000.000.000.010.010.060.100.230.310.450.390.600.540.690.640.720.720.740.750.750.74 0.000.000.000.000.010.020.040.170.290.420.530.510.620.610.680.630.710.670.730.750.750.75 0.000.000.020.060.020.040.130.190.210.210.310.300.410.400.490.430.520.490.540.580.590.61 0.000.000.000.000.010.020.080.110.200.270.370.430.550.540.640.630.660.670.700.700.710.70 0.180.050.250.230.340.410.500.510.630.710.770.830.860.860.860.860.860.870.850.860.850.82 0.000.000.010.090.160.130.330.470.530.550.600.630.640.660.690.660.700.670.730.750.750.78 0.000.170.030.010.010.020.040.030.080.090.150.140.240.200.280.260.300.280.340.350.350.41 0.000.000.190.210.190.260.260.350.420.450.500.480.570.570.640.590.680.630.720.740.750.74 0.000.000.000.030.050.050.160.200.210.220.290.270.320.310.360.320.370.350.390.420.420.42 0.160.070.240.290.250.280.440.540.610.660.720.730.730.760.790.770.790.790.800.810.810.80 (d) CIFAR100, Adversarial Training Figure 11: [Error Distributions As A Function of Fraction of Neurons] We see that accuracy deteriorates as we drop neurons, however, this drop comes at a larger cost for a few classes and results in near homogenous predictions for the least number of neurons on the left. 0.0005 0.001 0.002 0.003 0.004 0.005 0.01 0.02 0.03 0.04 0.05 0.075 0.1 0.15 0.2 0.25 0.3 0.4 0.5 0.8 0.9 1.0 Fraction/number of neurons pink primrose hard-leaved pocket orchid canterbury bells sweet pea english marigold tiger lily moon orchid bird of paradise monkshood globe thistle snapdragon colt's foot king protea spear thistle yellow iris globe-flower purple coneflower peruvian lily balloon flower giant white arum lily fire lily pincushion flower fritillary red ginger grape hyacinth corn poppy prince of wales feathers stemless gentian artichoke sweet william carnation garden phlox love in the mist mexican aster alpine sea holly ruby-lipped cattleya cape flower great masterwort siam tulip lenten rose barbeton daisy daffodil sword lily poinsettia bolero deep blue wallflower marigold buttercup oxeye daisy common dandelion petunia wild pansy primula sunflower pelargonium bishop of llandaff gaura geranium orange dahlia pink-yellow dahlia? cautleya spicata japanese anemone black-eyed susan silverbush californian poppy osteospermum spring crocus bearded iris windflower tree poppy gazania azalea water lily rose thorn apple morning glory passion flower lotus toad lily anthurium frangipani clematis hibiscus columbine desert-rose tree mallow magnolia cyclamen watercress canna lily hippeastrum bee balm ball moss foxglove bougainvillea camellia mallow mexican petunia bromelia blanket flower trumpet creeper blackberry lily Classwise Accuracies (mean) 0.000.000.000.000.000.000.010.060.030.360.040.000.130.030.140.160.230.290.470.540.520.55 0.000.000.000.000.010.000.000.000.050.030.050.020.210.170.560.370.720.690.860.880.910.95 0.000.000.000.000.000.000.000.000.000.030.000.000.000.000.000.000.000.000.000.000.000.00 0.000.000.000.000.000.000.010.000.000.100.100.020.060.010.130.130.190.140.220.270.240.22 0.000.000.000.000.000.000.000.000.000.000.030.000.050.030.060.070.110.120.160.200.200.29 0.000.000.000.000.000.000.000.000.000.000.430.300.340.500.780.670.870.900.940.940.940.96 0.000.000.000.000.000.000.080.000.000.000.020.160.120.030.430.500.610.630.800.880.880.90 0.000.000.000.000.000.070.000.000.000.000.090.010.470.340.720.400.720.710.910.981.001.00 0.000.080.000.000.000.000.000.000.020.020.020.070.310.130.360.230.380.550.680.730.750.73 0.000.000.000.000.000.000.000.000.000.000.130.080.070.050.200.110.200.180.390.440.430.48 0.000.000.000.000.000.000.000.000.000.000.070.120.060.010.070.070.070.110.140.240.260.27 0.000.000.000.000.000.070.000.000.000.000.250.000.330.250.360.190.320.370.410.550.650.67 0.000.000.000.260.000.100.000.000.000.390.070.000.030.080.180.060.260.120.480.770.820.86 0.000.000.000.000.650.000.120.000.000.000.180.150.660.270.890.560.480.770.840.920.870.93 0.000.000.000.000.170.000.100.000.000.110.030.010.060.020.280.250.460.430.790.880.910.93 0.070.000.000.000.000.000.000.000.000.040.170.000.020.250.160.360.420.240.540.620.630.62 0.000.000.000.000.000.000.000.000.000.030.180.210.260.180.390.220.330.310.700.720.750.77 0.000.000.060.000.000.000.000.000.000.000.020.170.060.030.050.070.040.020.100.280.330.40 0.000.000.000.000.000.000.000.000.000.010.000.010.060.000.060.120.120.120.370.570.590.55 0.000.000.090.000.160.000.000.010.000.020.020.000.160.150.200.090.430.200.540.630.670.75 0.000.000.000.000.000.000.020.000.000.000.000.020.040.020.010.050.240.140.220.360.430.50 0.000.000.000.000.000.000.000.000.000.010.000.010.250.100.340.340.630.480.660.860.870.90 0.600.000.000.000.000.000.000.000.000.000.340.200.680.240.750.580.820.750.860.860.860.87 0.040.000.000.000.000.000.040.000.000.000.050.000.090.110.140.190.230.250.480.510.550.59 0.000.000.000.000.000.000.000.000.050.000.020.030.190.140.500.350.570.460.760.780.860.86 0.000.000.000.000.090.140.010.020.000.000.030.050.160.150.500.250.590.490.640.700.740.81 0.000.000.000.080.000.000.000.000.000.000.100.120.210.170.520.480.630.400.750.991.001.00 0.000.000.000.000.000.000.000.000.000.000.000.000.090.040.320.280.230.280.370.370.410.43 0.000.000.000.000.000.000.010.000.000.170.120.040.400.480.830.850.970.890.980.980.990.98 0.000.000.000.000.000.120.220.040.290.070.370.400.700.580.770.730.850.890.920.900.890.89 0.000.000.000.000.000.000.000.000.000.030.020.040.010.060.240.170.320.190.430.750.770.78 0.000.000.000.000.000.380.060.200.000.010.000.000.150.180.140.050.120.140.260.400.470.48 0.000.000.000.000.000.000.000.070.010.000.060.000.120.030.230.360.400.380.500.650.700.81 0.000.000.000.000.000.000.000.000.000.010.000.000.010.000.020.010.030.020.080.140.100.20 0.000.000.000.000.000.000.000.000.000.000.190.000.230.170.540.290.720.660.790.830.830.83 0.000.000.000.080.000.000.000.000.000.000.000.000.010.000.050.000.100.020.120.230.340.36 0.000.000.000.000.000.000.000.000.000.000.090.000.270.320.560.470.740.600.910.980.980.99 0.000.000.000.000.000.000.000.000.020.000.000.000.040.010.070.090.190.050.250.430.480.58 0.000.000.000.000.010.000.000.000.000.000.010.000.030.000.000.000.020.000.010.010.010.00 0.000.000.000.000.000.000.020.200.010.000.000.000.000.000.000.000.030.010.060.160.190.23 0.000.000.000.000.000.000.000.140.000.000.230.000.090.400.460.240.650.780.800.800.790.79 0.000.000.000.000.000.000.000.000.040.020.000.000.040.020.180.000.110.090.650.570.580.54 0.000.000.000.000.000.000.110.010.060.010.000.000.080.080.080.140.080.110.170.240.260.31 0.000.000.000.000.000.000.180.000.070.000.110.060.230.460.560.630.700.620.870.950.960.96 0.000.000.000.000.000.000.000.020.000.010.000.000.000.000.000.000.010.030.020.110.150.20 0.000.000.000.000.000.000.000.000.000.020.000.000.110.080.210.070.290.260.560.690.750.78 0.000.000.000.000.000.020.000.000.000.030.030.000.250.060.370.160.490.430.940.990.990.98 0.000.000.000.780.000.000.000.000.070.000.000.000.050.010.020.140.380.170.450.540.560.55 0.000.000.000.000.000.000.000.000.000.000.000.210.390.150.810.770.850.831.001.001.001.00 0.000.000.000.000.000.000.000.000.000.000.070.110.140.240.220.330.520.490.800.940.920.94 0.000.000.000.000.030.070.000.230.000.040.090.110.110.040.210.190.310.280.370.450.440.45 0.000.000.000.000.000.000.030.000.000.000.000.000.000.000.020.030.010.070.220.250.320.28 0.000.000.000.000.000.000.000.000.000.000.070.080.020.210.070.020.100.040.290.360.380.32 0.000.000.000.000.000.000.000.000.000.000.020.000.100.150.170.050.170.060.250.350.360.37 0.000.000.650.000.000.450.000.000.000.000.040.010.110.070.190.070.320.090.470.710.760.82 0.000.000.000.000.000.000.010.000.090.000.050.070.390.360.520.480.580.510.860.990.990.99 0.000.000.000.000.000.010.000.020.000.230.000.040.140.040.200.110.310.290.560.660.710.74 0.000.000.000.000.000.000.140.000.000.000.100.030.300.040.460.410.700.500.890.970.980.98 0.000.000.000.000.000.000.100.000.000.000.160.000.090.200.310.140.580.290.670.850.880.89 0.000.000.000.000.000.000.000.000.020.160.480.630.640.330.870.690.960.980.980.980.990.99 0.000.000.000.000.000.000.000.000.000.010.290.000.350.500.790.680.850.830.991.001.001.00 0.000.000.000.000.000.190.060.000.430.000.010.010.020.010.000.020.020.030.110.100.130.20 0.000.000.000.000.000.000.000.000.000.000.060.040.200.450.090.130.310.190.250.390.400.50 0.000.000.000.000.000.000.000.000.000.080.090.040.210.260.560.440.960.780.941.001.001.00 0.000.000.000.000.000.000.000.030.000.060.020.000.020.040.100.070.360.340.600.710.730.74 0.000.000.000.000.000.000.000.000.000.000.030.010.110.020.320.010.430.410.670.960.970.95 0.000.000.000.000.000.000.000.000.000.000.000.000.010.020.100.020.140.040.220.350.330.36 0.000.000.000.000.000.000.000.000.000.000.000.000.030.010.030.000.060.040.130.180.210.18 0.000.000.000.000.000.000.000.020.000.000.190.030.360.120.690.410.750.710.880.890.890.88 0.000.000.190.000.000.000.000.000.000.000.110.010.170.130.460.230.530.500.690.860.880.90 0.000.000.000.000.000.000.000.000.010.000.010.000.110.050.220.010.270.110.270.380.420.40 0.000.000.000.000.000.000.000.000.000.000.020.010.010.030.070.090.090.060.180.270.280.41 0.000.000.000.000.000.000.010.000.000.000.010.000.070.070.170.160.220.190.500.620.690.74 0.000.000.000.140.000.000.000.000.000.000.010.000.000.000.000.010.020.010.020.040.060.06 0.000.000.000.000.000.010.000.000.000.000.090.080.260.110.310.230.470.460.720.780.790.78 0.000.000.050.000.000.000.060.000.000.000.100.000.020.010.180.040.340.090.520.610.630.67 0.510.000.000.000.000.000.000.000.000.000.020.000.160.210.300.120.480.290.550.700.770.82 0.000.000.000.000.310.000.000.000.000.000.000.110.010.010.020.030.080.040.130.350.350.38 0.000.000.000.000.000.010.040.120.000.000.100.000.330.070.440.380.520.550.800.930.910.95 0.000.000.000.000.000.000.000.000.000.000.120.010.030.070.090.140.300.340.480.680.670.66 0.000.000.000.000.000.000.000.000.020.000.010.070.060.060.120.120.240.260.590.700.730.68 0.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.000.010.000.030.020.03 0.000.000.000.000.000.000.000.300.010.180.010.000.010.060.010.060.040.140.210.310.370.40 0.000.000.000.000.000.000.000.000.000.070.000.000.010.050.020.030.030.050.100.250.280.24 0.000.000.000.000.000.000.190.060.110.050.010.000.230.400.240.100.190.290.560.520.540.49 0.000.950.000.000.000.000.020.010.000.000.100.000.140.250.350.180.310.170.430.690.720.74 0.000.000.000.000.000.000.000.000.190.000.080.050.040.030.240.140.310.240.540.660.730.79 0.000.000.000.000.000.000.000.000.000.000.000.000.010.020.010.000.040.040.160.220.250.26 0.000.000.000.000.000.010.000.000.000.000.000.030.010.120.080.210.210.230.490.600.650.66 0.000.000.000.000.000.000.040.000.000.010.140.000.200.250.380.050.380.310.500.600.580.60 0.000.000.000.000.000.000.000.000.000.000.000.040.020.080.130.070.220.310.480.600.610.64 0.000.000.000.000.000.000.000.000.000.000.010.000.070.000.080.020.090.090.150.250.290.33 0.000.000.000.000.000.000.040.000.000.230.130.090.150.070.400.350.480.250.620.810.850.85 0.000.000.000.000.000.000.000.000.180.000.020.000.040.010.160.070.320.160.450.580.640.68 0.000.000.000.000.000.000.000.000.000.010.000.000.000.000.020.070.090.060.100.160.180.20 0.000.000.000.000.000.000.000.000.000.000.000.130.020.010.030.050.030.050.070.120.130.11 0.000.000.000.000.000.000.000.020.000.000.010.000.010.000.170.020.220.180.390.490.520.57 0.000.000.000.000.000.000.000.000.000.000.040.000.010.000.040.000.050.130.150.290.290.34 0.000.000.000.000.250.000.000.000.000.000.140.000.260.230.350.290.390.590.810.870.880.88 0.000.000.000.000.000.000.000.000.000.000.120.000.170.170.240.180.300.210.420.460.430.41 0.000.000.000.000.000.000.030.000.040.000.000.000.000.010.020.020.020.060.170.180.300.32 0.000.000.000.000.000.000.000.000.000.010.390.040.490.410.690.370.810.690.910.960.960.96 (a) Flowers, Standard 0.0005 0.001 0.002 0.003 0.004 0.005 0.01 0.02 0.03 0.04 0.05 0.075 0.1 0.15 0.2 0.25 0.3 0.4 0.5 0.8 0.9 1.0 Fraction/number of neurons pink primrose hard-leaved pocket orchid canterbury bells sweet pea english marigold tiger lily moon orchid bird of paradise monkshood globe thistle snapdragon colt's foot king protea spear thistle yellow iris globe-flower purple coneflower peruvian lily balloon flower giant white arum lily fire lily pincushion flower fritillary red ginger grape hyacinth corn poppy prince of wales feathers stemless gentian artichoke sweet william carnation garden phlox love in the mist mexican aster alpine sea holly ruby-lipped cattleya cape flower great masterwort siam tulip lenten rose barbeton daisy daffodil sword lily poinsettia bolero deep blue wallflower marigold buttercup oxeye daisy common dandelion petunia wild pansy primula sunflower pelargonium bishop of llandaff gaura geranium orange dahlia pink-yellow dahlia? cautleya spicata japanese anemone black-eyed susan silverbush californian poppy osteospermum spring crocus bearded iris windflower tree poppy gazania azalea water lily rose thorn apple morning glory passion flower lotus toad lily anthurium frangipani clematis hibiscus columbine desert-rose tree mallow magnolia cyclamen watercress canna lily hippeastrum bee balm ball moss foxglove bougainvillea camellia mallow mexican petunia bromelia blanket flower trumpet creeper blackberry lily Classwise Accuracies (mean) 0.000.000.000.010.000.000.080.050.050.100.350.230.630.540.640.640.630.630.640.630.670.65 0.000.000.000.140.100.120.490.540.690.730.910.870.890.950.840.950.840.880.830.840.830.80 0.000.000.000.000.010.000.000.000.060.000.060.030.070.020.150.060.210.090.270.320.310.35 0.000.000.060.000.000.070.010.010.010.030.060.040.170.090.180.140.180.170.170.210.210.19 0.130.010.000.020.000.080.010.010.030.110.260.250.400.330.640.560.690.620.720.730.740.69 0.000.020.000.040.050.000.040.130.120.590.790.610.860.800.890.890.900.920.920.930.920.92 0.000.000.020.030.000.050.020.050.300.270.640.470.770.710.740.750.770.780.750.700.690.70 0.000.000.000.000.100.070.010.110.250.570.950.960.980.980.980.990.980.980.980.980.980.98 0.000.320.000.000.050.000.000.040.120.290.270.510.480.330.630.550.700.580.690.720.720.73 0.000.030.010.000.130.200.200.320.340.550.630.620.860.880.870.900.850.870.890.870.890.92 0.000.000.000.000.000.000.090.000.020.010.060.030.220.160.360.250.410.340.500.550.540.54 0.000.000.000.040.020.010.020.010.000.190.450.480.690.670.660.680.670.800.770.790.810.81 0.000.000.000.010.000.010.160.100.540.110.790.630.890.900.900.880.920.920.930.930.930.93 0.000.000.090.070.030.200.100.040.420.080.760.510.910.670.900.820.920.880.960.981.001.00 0.000.000.000.000.030.000.120.030.070.380.610.700.700.740.760.860.770.850.780.790.820.79 0.230.000.000.000.000.000.000.000.210.120.540.370.760.780.840.880.860.860.860.860.860.86 0.000.000.000.000.020.010.040.120.140.250.620.480.790.770.830.810.830.830.830.830.820.83 0.000.000.040.000.020.000.020.000.010.050.040.020.160.100.310.180.390.290.490.560.590.60 0.000.000.000.000.000.000.030.000.110.170.370.230.480.500.480.540.450.480.460.460.420.41 0.000.000.090.000.030.010.040.060.070.180.480.240.550.510.630.590.690.620.710.720.720.72 0.000.130.000.000.000.110.060.060.080.220.490.480.750.660.860.850.850.900.880.940.880.90 0.010.000.000.010.000.040.010.010.010.030.300.080.660.510.870.700.910.880.930.940.950.95 0.310.000.000.000.010.000.020.050.160.310.690.580.830.850.860.870.870.870.860.870.870.87 0.180.000.000.190.200.230.250.140.410.670.880.700.850.870.860.880.850.890.920.970.950.95 0.000.000.000.030.150.190.360.360.200.210.860.850.930.920.920.910.900.910.900.940.940.95 0.000.000.000.080.090.030.110.210.410.500.790.730.810.790.760.800.820.770.810.780.770.76 0.000.000.000.030.010.000.000.020.060.210.420.540.900.820.990.931.000.971.001.001.001.00 0.000.000.000.000.000.000.010.000.340.630.740.470.830.770.850.900.890.920.870.880.890.89 0.000.000.020.150.000.010.150.110.260.330.710.680.890.890.940.890.920.920.920.910.900.91 0.000.040.000.000.000.000.070.140.260.320.630.540.700.760.630.720.620.640.590.590.580.58 0.000.020.000.020.000.080.060.010.010.000.160.090.310.240.560.400.670.510.750.820.840.88 0.000.000.000.000.000.020.020.100.020.050.170.180.300.230.380.290.430.390.400.550.580.56 0.000.000.000.000.000.040.060.120.050.020.370.070.580.510.760.710.820.850.900.920.920.92 0.000.000.000.000.000.020.000.000.040.080.040.040.060.020.350.050.470.220.630.650.660.65 0.000.000.140.210.100.040.380.270.320.570.750.670.940.900.960.970.970.970.970.940.940.91 0.000.190.000.130.000.000.000.010.020.090.190.080.360.210.440.240.570.320.680.730.750.76 0.000.000.000.000.120.000.120.020.130.120.680.510.940.910.970.970.980.980.970.960.950.95 0.000.000.010.000.020.000.060.050.040.160.320.190.480.510.720.680.740.720.810.820.810.83 0.000.050.000.000.030.000.000.020.010.040.060.030.160.140.450.170.550.350.700.740.730.76 0.000.000.000.010.010.000.060.070.070.090.200.090.430.310.600.550.660.570.700.730.720.77 0.000.000.000.000.010.040.010.010.000.060.030.070.160.200.320.240.420.330.460.520.510.52 0.000.000.000.010.000.000.020.030.010.100.380.420.650.350.720.500.790.610.790.820.770.77 0.000.000.000.000.000.000.040.020.020.010.020.070.040.060.130.120.210.170.280.310.320.32 0.000.000.000.000.000.000.000.010.020.010.370.070.500.540.710.690.760.760.810.820.840.85 0.000.000.000.000.000.000.000.020.070.120.310.220.340.260.540.470.620.550.640.620.660.65 0.000.010.000.000.000.000.060.050.210.070.440.290.780.710.890.810.900.870.930.950.950.95 0.000.000.000.150.010.170.060.010.010.050.660.390.950.980.980.980.980.980.980.980.980.98 0.000.000.000.080.000.000.090.020.010.330.670.340.580.600.730.890.840.830.850.910.930.90 0.000.000.000.170.190.040.230.030.190.250.970.921.001.001.001.001.001.001.001.001.001.00 0.000.000.000.000.040.220.000.000.220.230.710.170.850.710.950.890.960.910.950.940.940.94 0.000.000.000.000.010.000.010.080.030.010.040.050.240.230.380.330.430.440.450.430.430.43 0.000.000.000.000.000.000.000.020.050.070.280.250.520.440.700.600.720.610.800.820.830.83 0.000.000.000.000.020.000.020.040.000.040.100.070.220.210.280.220.290.250.360.390.420.41 0.000.000.000.000.000.000.000.020.060.260.180.240.600.410.750.640.810.770.870.900.900.90 0.000.000.020.000.000.130.110.110.180.230.490.440.640.700.840.730.860.870.920.920.910.92 0.000.000.000.010.000.040.200.080.110.300.660.570.950.930.980.980.980.980.970.970.970.97 0.000.000.000.000.000.020.010.060.030.040.340.330.720.650.830.800.870.850.890.910.910.91 0.000.000.080.000.100.050.090.070.010.010.680.160.880.850.910.890.920.910.950.960.960.97 0.000.000.160.000.040.000.010.000.130.090.380.110.880.661.000.921.001.001.001.001.001.00 0.000.000.000.000.280.020.150.060.070.180.760.290.920.960.970.970.960.970.970.980.980.98 0.000.000.000.000.010.010.320.090.210.340.780.850.980.970.990.990.991.000.990.991.001.00 0.000.000.000.000.000.010.010.040.030.010.110.030.300.170.410.300.590.440.640.730.750.74 0.000.000.150.000.050.010.180.220.280.440.960.711.001.001.000.991.001.001.001.001.001.00 0.000.000.000.000.000.000.320.470.060.360.940.801.000.990.990.990.980.990.950.940.940.94 0.000.000.000.000.000.000.020.040.090.040.180.270.500.240.740.530.810.780.870.900.890.87 0.000.000.000.000.000.100.470.530.630.800.990.980.991.001.001.001.000.991.001.001.001.00 0.000.000.000.000.000.010.020.000.030.050.130.130.380.200.680.400.780.660.860.950.950.91 0.000.000.000.000.000.000.000.050.040.090.140.020.250.190.480.340.550.410.620.680.680.71 0.000.000.020.000.040.010.230.090.190.320.760.500.980.920.991.000.990.990.980.960.940.97 0.000.000.190.030.060.000.050.020.130.400.600.900.880.820.910.970.900.980.900.910.910.90 0.000.000.020.000.000.000.250.220.350.170.810.700.940.860.980.940.970.960.990.990.991.00 0.000.150.000.000.000.000.010.020.030.040.140.030.120.080.270.170.330.270.390.450.440.49 0.000.040.000.000.000.000.010.030.050.090.340.120.390.330.680.530.790.640.860.910.920.92 0.000.000.000.010.000.000.010.020.010.040.060.030.190.150.390.240.500.300.620.670.680.70 0.000.000.000.000.040.010.030.130.200.530.740.600.790.860.840.870.800.830.760.750.750.75 0.000.010.110.000.030.000.080.070.120.160.430.340.670.650.750.720.740.750.760.750.740.74 0.450.000.000.000.000.000.000.010.020.030.080.060.510.420.710.570.750.710.810.830.840.84 0.000.000.000.000.000.010.020.050.070.090.250.060.480.400.500.480.510.500.490.500.530.53 0.000.000.000.000.000.000.010.010.100.270.490.430.850.790.960.900.940.960.940.950.950.95 0.000.000.000.010.000.010.040.010.030.110.400.240.670.640.780.750.780.780.780.780.790.79 0.000.000.000.000.000.000.070.050.130.270.630.240.800.780.900.900.930.930.930.940.950.95 0.000.000.000.000.000.000.000.000.000.010.020.010.050.020.120.050.190.150.300.350.390.38 0.000.000.000.000.060.030.010.000.000.020.070.010.100.080.290.100.400.240.500.520.520.52 0.000.000.000.000.000.000.010.010.010.000.040.050.100.040.310.140.440.290.510.550.550.53 0.000.000.000.000.000.000.000.030.070.160.200.230.450.430.650.580.690.660.740.770.770.79 0.000.190.000.000.010.020.020.080.090.070.240.190.420.280.490.460.520.500.590.630.640.66 0.000.000.000.000.000.150.000.040.020.060.190.120.450.430.590.530.670.580.730.820.870.86 0.000.000.000.000.000.000.000.020.010.010.040.030.150.070.190.150.200.200.300.390.430.45 0.000.000.000.000.000.000.000.010.060.040.210.090.280.320.400.380.420.430.460.460.450.46 0.000.000.000.000.000.000.030.000.040.050.160.110.320.160.430.310.540.410.590.610.630.65 0.000.000.000.000.000.000.030.050.020.010.150.050.310.230.430.360.480.410.530.610.640.64 0.000.000.000.010.000.000.000.040.070.000.100.030.310.210.680.490.770.700.800.820.830.83 0.000.000.000.000.000.010.000.000.020.020.380.080.530.400.720.580.830.850.880.880.890.92 0.000.000.000.000.000.000.000.030.000.010.070.030.120.050.280.160.430.310.570.670.680.68 0.000.000.000.000.000.010.000.020.020.010.020.060.060.030.160.070.250.140.340.350.360.37 0.000.000.000.000.000.000.000.020.000.020.010.000.030.020.080.030.110.070.150.210.210.21 0.000.000.000.000.000.000.040.000.070.020.100.020.300.160.430.290.500.440.530.550.570.57 0.000.000.000.000.000.000.020.010.060.050.170.060.190.210.430.290.450.360.500.530.540.58 0.000.000.000.010.070.000.120.100.190.220.410.280.670.610.790.730.830.820.810.800.820.79 0.000.000.000.000.040.010.020.120.170.500.630.590.880.860.940.900.950.940.960.970.970.97 0.000.000.010.000.000.010.010.010.030.050.110.050.290.250.480.310.490.410.580.680.710.68 0.000.000.000.010.000.130.250.110.150.140.620.420.860.820.930.970.960.970.981.001.001.00 (b) Flowers, Adversarial Training Figure 12: [Error Distributions As A Function of Fraction of Neurons continued] We see that accuracy deteriorates as we drop neurons, however, this drop comes at a larger cost for a few classes and results in near homogenous predictions for the least number of neurons on the left. 0.0005 0.001 0.002 0.003 0.004 0.005 0.01 0.02 0.03 0.04 0.05 0.075 0.1 0.15 0.2 0.25 0.3 0.4 0.5 0.8 0.9 1.0 Fraction/number of neurons Abyssinian American Bulldog American Pit Bull Terrier Basset Hound Beagle Bengal Birman Bombay Boxer British Shorthair Chihuahua Egyptian Mau English Cocker Spaniel English Setter German Shorthaired Great Pyrenees Havanese Japanese Chin Keeshond Leonberger Maine Coon Miniature Pinscher Newfoundland Persian Pomeranian Pug Ragdoll Russian Blue Saint Bernard Samoyed Scottish Terrier Shiba Inu Siamese Sphynx Staffordshire Bull Terrier Wheaten Terrier Yorkshire Terrier Classwise Accuracies (mean) 0.000.000.000.000.040.000.020.150.100.100.220.200.540.570.770.690.830.820.870.900.910.93 0.000.000.000.010.020.110.150.380.470.430.580.670.770.790.870.900.890.890.900.910.910.92 0.000.000.000.000.030.000.010.050.080.090.240.330.520.460.580.570.610.560.610.620.620.63 0.000.000.070.050.040.290.400.270.580.660.820.850.930.920.920.920.930.930.930.940.930.93 0.000.000.000.010.010.010.150.150.200.210.510.540.820.760.960.970.991.000.990.990.990.99 0.000.080.080.000.090.290.330.230.400.370.340.340.490.530.720.660.790.750.840.910.910.91 0.000.000.000.000.000.000.170.000.090.120.170.220.360.330.540.400.670.650.730.780.790.81 0.000.000.000.180.180.260.320.260.550.660.740.750.870.850.900.940.950.950.950.970.960.95 0.000.000.000.130.040.160.220.210.350.430.680.730.890.880.910.900.920.910.930.930.920.92 0.000.010.000.000.000.000.200.260.220.230.380.360.480.510.570.590.580.600.620.650.660.67 0.930.200.260.300.290.390.170.560.910.780.800.840.920.920.930.930.940.940.950.960.970.97 0.000.000.000.000.000.000.030.030.160.560.750.840.820.860.790.820.790.800.790.800.810.80 0.000.000.000.000.100.010.320.270.450.490.730.740.940.960.970.970.970.970.980.980.980.98 0.000.000.000.000.000.030.090.060.240.260.640.730.950.950.970.950.980.980.980.980.980.98 0.000.030.000.020.000.060.110.560.740.800.950.950.980.980.990.991.001.001.001.001.001.00 0.000.000.000.000.000.000.140.230.290.540.700.800.940.970.970.970.970.980.970.980.980.98 0.240.490.270.450.150.270.220.580.630.670.860.810.960.960.970.980.980.991.001.001.001.00 0.000.000.000.000.000.000.000.110.350.310.720.730.960.970.970.980.990.991.001.001.001.00 0.000.000.000.000.110.230.300.310.370.500.870.830.960.960.960.970.970.970.980.980.980.98 0.200.200.200.220.380.290.200.290.520.630.850.840.970.930.980.980.990.980.991.001.001.00 0.000.010.010.000.010.060.060.040.140.390.580.650.760.730.810.840.840.850.830.860.850.84 0.000.000.000.190.210.260.320.240.280.450.770.720.910.920.930.930.940.940.950.940.940.94 0.010.020.010.070.060.110.380.350.510.810.930.970.980.990.991.000.990.991.001.001.001.00 0.000.000.000.010.000.000.000.350.600.540.680.650.830.870.900.900.900.910.900.910.910.91 0.000.000.000.010.190.210.290.200.290.370.670.800.910.940.980.980.980.990.991.001.001.00 0.000.000.050.000.010.000.150.190.470.530.790.680.950.930.980.970.980.980.990.990.990.99 0.000.000.070.000.010.080.150.130.190.360.600.610.680.750.690.710.730.700.720.750.730.75 0.000.240.170.110.150.030.080.170.150.240.290.320.530.480.630.590.710.690.750.770.770.77 0.010.000.070.160.290.070.670.760.770.900.950.980.970.980.980.980.990.990.990.990.990.99 0.000.000.000.000.010.010.240.220.290.630.891.001.001.001.001.000.990.990.990.990.990.99 0.000.000.370.160.150.190.290.330.550.570.660.880.960.980.990.990.991.001.001.001.001.00 0.000.000.050.010.030.010.190.250.180.310.600.720.930.930.991.001.001.001.001.001.001.00 0.000.000.000.000.000.010.350.440.630.580.710.680.870.890.950.950.950.950.960.960.960.96 0.000.010.020.310.300.510.590.390.580.530.690.660.810.870.920.940.930.930.930.930.940.96 0.000.000.000.040.000.180.250.320.450.380.600.540.660.640.720.730.740.760.760.770.790.80 0.000.000.000.230.140.310.440.450.450.490.760.770.850.890.880.890.910.900.930.960.970.97 0.000.000.000.200.350.370.490.350.630.590.810.820.930.950.970.970.990.990.990.990.990.99 (a) Oxford-IIIT-Pets, Standard 0.0005 0.001 0.002 0.003 0.004 0.005 0.01 0.02 0.03 0.04 0.05 0.075 0.1 0.15 0.2 0.25 0.3 0.4 0.5 0.8 0.9 1.0 Fraction/number of neurons Abyssinian American Bulldog American Pit Bull Terrier Basset Hound Beagle Bengal Birman Bombay Boxer British Shorthair Chihuahua Egyptian Mau English Cocker Spaniel English Setter German Shorthaired Great Pyrenees Havanese Japanese Chin Keeshond Leonberger Maine Coon Miniature Pinscher Newfoundland Persian Pomeranian Pug Ragdoll Russian Blue Saint Bernard Samoyed Scottish Terrier Shiba Inu Siamese Sphynx Staffordshire Bull Terrier Wheaten Terrier Yorkshire Terrier Classwise Accuracies (mean) 0.000.000.020.030.010.060.180.280.440.490.650.670.720.740.770.740.770.780.770.780.770.78 0.070.090.080.090.140.150.290.420.500.590.660.660.720.740.760.770.780.790.780.750.750.74 0.000.000.000.000.010.010.090.160.230.320.420.400.480.480.530.500.540.510.530.520.520.53 0.030.090.030.040.150.280.500.570.740.780.800.800.820.820.830.820.820.810.810.820.810.81 0.090.000.010.020.130.130.280.320.450.600.730.720.780.800.840.830.840.860.850.850.850.85 0.070.160.220.180.170.190.250.490.600.630.700.700.780.780.790.800.810.770.800.790.790.78 0.000.060.060.060.110.120.190.320.390.440.570.560.650.650.700.690.710.770.730.700.710.69 0.000.070.010.000.050.070.320.530.660.710.730.800.800.820.810.820.820.810.820.840.830.83 0.000.000.070.250.140.090.240.310.540.630.720.770.800.780.820.820.840.860.870.870.860.87 0.000.000.000.090.070.120.260.310.460.430.470.510.540.550.600.600.610.640.620.680.690.70 0.360.040.060.100.060.080.230.400.480.610.680.680.710.710.740.740.760.750.770.760.760.76 0.120.000.110.210.280.230.370.550.620.640.740.760.800.810.840.830.850.870.870.870.860.87 0.000.000.000.010.030.040.120.200.340.470.700.690.830.830.880.870.900.880.880.870.860.86 0.000.000.080.090.040.050.120.310.360.490.690.730.780.780.850.820.850.850.870.890.900.91 0.000.000.030.100.110.160.240.630.790.820.870.900.920.940.960.970.980.960.970.960.960.96 0.000.040.080.160.270.260.460.590.640.720.830.780.870.890.920.910.910.920.920.910.910.91 0.050.210.110.150.100.090.260.390.470.520.700.710.860.870.890.890.910.900.930.930.940.93 0.070.000.210.370.370.410.530.650.770.790.850.850.900.900.920.920.940.930.940.950.950.95 0.000.260.180.230.380.520.580.790.800.860.900.930.940.940.950.950.960.960.960.960.950.96 0.000.000.000.020.050.170.380.570.670.740.850.880.940.940.960.950.960.940.960.950.950.94 0.000.020.000.090.150.170.250.270.410.460.550.550.610.600.680.660.700.730.720.740.750.74 0.000.050.070.170.200.160.460.520.550.620.720.720.790.790.840.820.850.850.850.870.870.88 0.000.080.120.150.290.400.480.600.770.830.860.870.900.910.910.910.920.910.910.900.900.90 0.000.180.250.170.080.140.230.320.470.510.570.570.600.610.650.620.660.640.690.700.700.71 0.000.000.000.050.090.120.370.550.690.750.810.820.840.850.880.870.870.870.890.890.890.89 0.000.010.050.110.070.200.510.690.820.850.870.900.910.920.920.950.950.950.950.950.950.94 0.080.040.040.100.170.090.200.290.330.420.580.550.640.620.670.660.690.650.690.720.720.74 0.000.160.140.080.150.160.290.320.450.490.580.660.700.700.730.730.760.730.770.740.740.73 0.180.190.220.390.550.530.640.820.820.850.880.910.900.910.920.920.920.930.930.930.940.94 0.190.160.260.400.430.560.770.860.890.900.930.920.940.950.960.960.960.950.950.950.950.95 0.000.040.110.190.240.140.400.430.590.660.760.770.890.860.920.940.950.940.950.950.950.95 0.000.000.100.100.230.260.340.500.700.800.830.870.870.900.900.900.890.890.880.880.890.89 0.000.130.120.110.200.210.520.720.760.770.800.830.850.850.880.870.870.840.870.870.870.87 0.010.030.060.250.260.360.400.510.580.650.720.730.770.790.830.840.850.850.850.870.890.87 0.210.030.170.100.250.240.220.320.350.340.400.390.410.420.450.440.470.480.480.490.490.49 0.000.020.170.180.180.250.410.460.550.620.720.760.810.810.820.830.830.830.830.830.830.84 0.040.000.040.090.100.220.550.720.790.790.830.860.870.860.880.870.890.880.890.910.910.91 (b) Oxford-IIIT-Pets, Adversarial Training Figure 13: [Error Distributions As A Function of Fraction of Neurons continued] We see that accuracy deteriorates as we drop neurons, however, this drop comes at a larger cost for a few classes and results in near homogenous predictions for the least number of neurons on the left. 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (a) CIFAR10 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (b) CIFAR10 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (c) Flowers 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (d) Oxford-IIIT-Pets resnet50-nonrob-random resnet50-robustl2eps3-random resnet50-robustlinfeps4-random Figure 14: [Results for ℓ robust model] We show results for Res Net50 trained with 3 different losses: Cross Entropy (nonrob), Adversarial Training with ℓ2 threat model (robl2eps3), and with the ℓ threat model (roblinfeps4). We see that the ℓ2 threat model shows the most diffused redundancy. All models are trained on Image Net1k. 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (a) CIFAR10 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (b) CIFAR100 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (c) Flowers 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (d) Oxford-IIIT-Pets vgg16_bn-nonrob-random vgg16_bn-robustl2eps3-random resnet18-nonrob-random resnet18-robustl2eps3-random resnet50-nonrob-random resnet50-robustl2eps3-random wide_resnet50_2-nonrob-random wide_resnet50_2-robustl2eps3-random vit_small_patch16_224-nonrob-random vit_small_patch32_224-nonrob-random Figure 15: [Comparisons Across Architectures For Downstream Task Accuracy] All models shown here are pre-trained on Image Net1k. This Figure shows corresponding diffused redundancy values for Figure 4 different δ values. We see that diffused redundancy exists across architectures, and the trend observed in Figure 1c&1a regarding adversarially trained models also holds here as models curves that are more inside are the ones trained with standard loss. 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (a) CIFAR10 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (b) CIFAR100 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (c) Flowers 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (d) Oxford-IIIT-Pets vit_small_patch16_224-imagenet vit_small_patch16_224-imagenet21k vit_small_patch32_224-imagenet vit_small_patch32_224-imagenet21k Figure 16: [Comparison Across Upstream Datasets] We see that degree of diffused redundancy depends a great deal on the upstream training dataset, in particular models trained on Image Net21k exhibit a higher degree of diffused redundanacy, although the differences in the degree of diffused redundanacy are downstream task dependent 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (a) CIFAR10 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (b) CIFAR100 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (c) Flowers 0.75 0.80 0.85 0.90 0.95 1.00 0.0 Diffused Redundancy (DR) (d) Oxford-IIIT-Pets resnet50-nonrob-random resnet50-robustl2eps3-random resnet50_mrl-nonrob-random resnet50_mrl-nonrob-first Figure 17: [Comparison of Diffused Redundancy in MRL vs other losses] Here we compare Res Net50 trained using multiple losses including MRL [26]. Red line shows results for part of the representation explicitly optimized in MRL, whereas green line shows results for parts that are picked randomly from the same representation. Even the MRL model shows a significant amount of diffused redundancy despite being explicitly trained to instead have structured redundancy. This figure shows diffused redundancy (DR) for all plots in Figure 7. 0 1000 2000 3000 4000 Number of neurons Transfer Accuracy (a) CIFAR10 0 1000 2000 3000 4000 Number of neurons Transfer Accuracy (b) CIFAR100 0 1000 2000 3000 4000 Number of neurons Transfer Accuracy (c) Flowers 0 1000 2000 3000 4000 Number of neurons Transfer Accuracy (d) Oxford-IIIT-Pets vgg16_bn-nonrob-random vgg16_bn-robustl2eps3-random resnet18-nonrob-random resnet18-robustl2eps3-random resnet50-nonrob-random resnet50-robustl2eps3-random wide_resnet50_2-nonrob-random wide_resnet50_2-robustl2eps3-random vit_small_patch16_224-nonrob-random vit_small_patch32_224-nonrob-random Figure 18: [Comparisons Across Architectures For Downstream Task Accuracy] This shows the same plots as Figure 4, except showing absolute number of neurons on the x-axis 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (a) layer1.0.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (b) layer1.1.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (c) layer1.2.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (d) layer2.0.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (e) layer2.1.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (f) layer2.2.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (g) layer2.3.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (h) layer3.0.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (i) layer3.1.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (j) layer3.2.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (k) layer3.3.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (l) layer3.4.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (m) layer3.5.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (n) layer4.0.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (o) layer4.1.act3 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of neurons Transfer Accuracy (p) layer4.2.act3 cifar10 cifar100 flowers oxford-iiit-pets Figure 19: [Middle Layers; Res Net50, trained with Cross Entropy loss on Image Net1k] We see that as we go deeper in the network, accuracy progressively increases. We see even middle layers exhibit diffused redundancy, and accuracy plateaus very quickly for earlier layers. layer X.Y.act3 refers to the Y th residual connection in the Xth Res Net block and act3 indicates that we re taking the value after the activation (Re LU) has been applied. 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Neurons (a) Res Net50 nonrobust 0.0 0.2 0.4 0.6 0.8 1.0 Fraction of Neurons (b) Res Net50 robust ℓ2ϵ = 3 Imagenet Imagenet V2 Places365 Figure 20: [Performance on Image Net1k, Image Net V2, and Places365] We check for the performance of randomly chosen subsets of neurons on harder tasks like Image Net1k, Image Net V2, and Places365. We find that diffused redundancy holds for all these harder tasks as well. Additionally, we see that randomly dropping neurons still preserves the accuracy gap between Image Net1k and Image Net V2. 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy (a) Dropout = 0.1 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy (b) Dropout = 0.2 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy (c) Dropout = 0.3 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy (d) Dropout = 0.4 cifar10 cifar100 flowers oxford-iiit-pets 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy (e) Dropout = 0.5 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy (f) Dropout = 0.6 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy (g) Dropout = 0.7 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy (h) Dropout = 0.8 cifar10 cifar100 flowers oxford-iiit-pets 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Fraction of Neurons (Total=2048) Transfer Accuracy (i) De Cov Regularization (0.0001) cifar10 cifar100 flowers oxford-iiit-pets Figure 21: [Dropout and De Cov regularizer s effect on Diffused Redundancy] 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) (a) Dropout = 0.1 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) (b) Dropout = 0.2 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) (c) Dropout = 0.3 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) (d) Dropout = 0.4 cifar10 cifar100 flowers oxford-iiit-pets 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) (e) Dropout = 0.5 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) (f) Dropout = 0.6 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) (g) Dropout = 0.7 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) (h) Dropout = 0.8 cifar10 cifar100 flowers oxford-iiit-pets 0.0 0.2 0.4 0.6 0.8 1.0 0.0 Diffused Redundancy (DR) (i) De Cov Regularization (0.0001) cifar10 cifar100 flowers oxford-iiit-pets Figure 22: [Dropout and De Cov regularizer s effect on Diffused Redundancy] Same results as Figure 21, but showing DR estimates (Eq 1). Lines that are more towards the right (i.e.more on the outside ) mean they exhibit more diffused redundancy.