# kymatio_scattering_transforms_in_python__ea2ed399.pdf

Journal of Machine Learning Research 21 (2020) 1-6 Submitted 1/19; Revised 10/19; Published 1/20

Kymatio: Scattering Transforms in Python

Mathieu Andreux mathieu.andreux@owkin.com Tom as Angles tomas.angles@ens.fr Georgios Exarchakis georgios.exarchakis@ens.fr Roberto Leonarduzzi roberto.leonarduzzi@ens-lyon.fr Gaspar Rochette gaspar.rochette@ens.fr Louis Thiry louis.thiry@outlook.fr John Zarka johnzarka@gmail.com Ecole normale sup erieure, CNRS, PSL Research University, 45, rue d Ulm, 75005 Paris, France

St ephane Mallat stephane.mallat@ens.fr Ecole normale sup erieure, CNRS, PSL Research University, 45, rue d Ulm, 75005 Paris, France Coll ege de France, 11, place Marcelin-Berthelot 75231 Paris, France Flatiron Institute, 162 5th Avenue, New York, NY 10010, USA

Joakim And en janden@flatironinstitute.org Flatiron Institute, 162 5th Avenue, New York, NY 10010, USA

Eugene Belilovsky eugene.belilovsky@umontreal.ca Mila, Universit e de Montr eal, 6666 St Urbain Street, Montreal, Quebec H2S 3H1, Canada

Joan Bruna bruna@cims.nyu.edu Vincent Lostanlen vl1019@nyu.edu New York University, 70 Washington Square South, New York, NY 10012, USA

Muawiz Chaudhary chaudhm@wwu.edu Western Washington University, 516 High Street, Bellingham, WA 98225, USA

Matthew J. Hirn mhirn@msu.edu Michigan State University, 426 Auditorium Road East Lansing, MI 48824, USA

Edouard Oyallon edouard.oyallon@lip6.fr CNRS, LIP6, Sorbonne University, 4 place Jussieu, 75252 Paris, France

Sixin Zhang sixin.zhang@pku.edu.cn Peking University, No. 5 Yiheyuan Road, Haidian District, Beijing 100871, China

Carmine Cella carmine.cella@berkeley.edu University of California, Berkeley, 101 Sproul Hall, Berkeley, CA 94720, USA

Michael Eickenberg meickenberg@flatironinstitute.org Flatiron Institute, 162 5th Avenue, New York, NY 10010, USA

Editor: Balazs Kegl

The wavelet scattering transform is an invariant and stable signal representation suitable for many signal processing and machine learning applications. We present the Kymatio software package, an easy-to-use, high-performance Python implementation of the scattering transform in 1D, 2D, and 3D that is compatible with modern deep learning frameworks, including Py Torch and Tensor Flow/Keras. The transforms are implemented on both CPUs

c 2020 Mathieu Andreux, Tom as Angles, Georgios Exarchakis, Roberto Leonarduzzi, Gaspar Rochette, Louis Thiry, John Zarka, St ephane Mallat, Joakim And en, Eugene Belilovsky, Joan Bruna, Vincent Lostanlen, Muawiz Chaudhary, Matthew J. Hirn, Edouard Oyallon, Sixin Zhang, Carmine Cella, Michael Eickenberg.

License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. Attribution requirements are provided at http://jmlr.org/papers/v21/19-047.html.

Andreux et al.

and GPUs, the latter offering a significant speedup over the former. The package also has a small memory footprint. Source code, documentation, and examples are available under a BSD license at https://www.kymat.io. Keywords: Scattering Transform; GPUs; Wavelets; Convolutional Networks; Invariance

1. Introduction

Many classification and regression tasks have a degree of invariance to translations and deformations, such as those relating to images, audio recordings, and electronic densities. The scattering transform was introduced in Mallat (2012) to build a signal representation that is invariant to such transformations while preserving as much as possible the information relevant to the task at hand. It is defined as a convolutional network whose filters are fixed to be wavelet and lowpass averaging filters coupled with modulus nonlinearities. It has many favorable theoretical properties (Mallat, 2012; Bruna et al., 2015; Waldspurger, 2017) and enjoys considerable success as a powerful tool in modern signal processing (Adel et al., 2017; Bruna and Mallat, 2013; And en and Mallat, 2014; Chud aˇcek et al., 2014; Sifre and Mallat, 2013; Eickenberg et al., 2017). It is also effective in combination with modern representation learning approaches (Oyallon et al., 2018; Sainath et al., 2014; Zeghidour et al., 2016). This article presents Kymatio, a scattering transform implementation that is user-friendly, well-documented, fast, and compatible with existing automatic differentiation libraries. It brings together transforms in 1D, 2D, and 3D under a unified application programming interface (API). The scattering network is also traversed depth-first to reduce memory requirements, enabling efficient processing in limited-memory environments, such as GPUs.

2. Implementing the Scattering Transform

Definition We consider signals defined on a grid of size N1 Nd for d = 1, 2, 3. Given two signals x[n] and y[n] on this grid, we denote their periodic convolution by x y[n]. The second-order scattering transform is defined using two wavelet filter banks {ψ(1) λ1 [n]}λ1 Λ1 and {ψ(2) λ2 [n]}λ2 Λ2, where λ1 and λ2 are frequency indices in the sets Λ1 and Λ2. It also includes a lowpass filter φJ[n], where the integer J > 0 specifies the averaging scale 2J of the filter. Together with a non-linearity ρ(t), these filters define the scattering transform. The zeroth-order scattering coefficient S0x[n] is the local average given by S0x[n] = x φJ[n]. Convolving x[n] with the first-order wavelet filter bank {ψ(1) λ1 [n]}λ1 Λ1, applying ρ(t), and convolving with φJ[n], we obtain the first-order scattering coefficients

S1x[n, λ1] = ρ x ψ(1) λ1

φJ[n], λ1 Λ1. (1)

The modulus of the first wavelet transform acts as a demodulation, shifting its energy to the low frequencies. However, only some of these frequencies are covered by the low-pass filter φJ. We recover the remaining frequencies by decomposing ρ(x ψ(1) λ1 [n]) using the

second filter bank, but this is done only for a subset Λ2(λ1) of Λ2 since ρ(x ψ(1) λ1 [n]) is a low-frequency signal. Typically, we have Λ2(λ1) = {λ Λ2, |λ| > |λ1|}. The result is then passed through ρ(t) and averaged, yielding the second-order coefficients

S2x[n, λ1, λ2] = ρ ρ x ψ(1) λ1

φJ[n], λ1 Λ1, λ2 Λ2(λ1). (2)

Kymatio: Scattering Transforms in Python

dimension gpu diff. core devs. license language Scat Net 1D, 2D 5 Apache 2.0 MATLAB Scat Net Light 2D 2 GPLv2 MATLAB Py Scat Wave 2D 3 BSD-3 Python Scattering.m 1D 1 GPLv3 MATLAB Py Scat Harm 3D 1 BSD-3 Python Wavelet Toolbox 1D N/A Proprietary MATLAB Kymatio 1D, 2D, 3D 15 BSD-3 Python

Table 1: Comparison to existing scattering transform packages.

The energy of higher-order scattering coefficients is typically small and does not greatly influence results (Waldspurger, 2017; Bruna and Mallat, 2013; And en and Mallat, 2014). On the other hand they can be computationally intensive. We have thus chosen to restrict our scope to second-order coefficients, which is what is used in most works.

Implementation Signals obtained by filtering and applying ρ(t) are low-frequency, so intermediate results are downsampled to reduce computational load as in And en et al. (2014). In 1D and 2D, we use Morlet wavelets which are close to analytic (i.e., complex-valued with low energy in the negative frequencies) and the non-linearity is the complex modulus ρ(t) = |t| for t C (And en and Mallat, 2014; Bruna and Mallat, 2013). The 3D transform is calculated using solid harmonic wavelets ψλ1 = ψj,ℓ,m, where j indexes the scale, and ℓ, m are the azimuthal and magnetic quantum numbers. In this case the non-linearity ρ : C2ℓ+1 R is defined, with a slight abuse of notation, as ρ(x ψj,ℓ) = p P m |x ψj,ℓ,m|2 (Eickenberg et al., 2017). Following Oyallon et al. (2018), we compute the scattering transform in a depth-first manner, reducing the number of intermediate signals stored at a given time. Since convolutions are all periodic, they may be efficiently calculated using fast Fourier transforms.

3. Project vision

Code quality Adopting the philosophy of scikit-learn (Pedregosa et al., 2011), the goal of Kymatio is not to maximize the number of features, but to provide a stable and easyto-use framework. To this end, we make heavy use of unit tests, minimize the number of dependencies, and strive for intuitive interfaces inspired by modern deep learning paradigms. Kymatio also provides an extensive user guide, including an API reference, a tutorial, installation instructions, and easy-to-understand examples, several of which feature realworld applications.

Community and bug tracking Kymatio is free and open-source software with a 3-clause BSD license. The members of its core development team all have experience implementing scattering transforms in other packages. A key goal of Kymatio is to combine these efforts and foster a community effort in order to produce high-quality software and maintain a critical mass of contributors for its maintenance. The package was released publicly on Git Hub November 17th, 2018. The main communication channel is the Git Hub page for questions, bug reports, and feature requests. There is also a dedicated Slack channel.

Andreux et al.

Relation to previous software Aside from the emphasis on code quality and usability, Kymatio provides several improvements over previous scattering implementations:

Python is the de facto standard for data science software, but most existing scattering packages are implemented in MATLAB. In contrast, Kymatio provides a completely Pythonic implementation, enabling integration with the scientific Python ecosystem.

GPU compatibility is critical to many data science workloads. Kymatio offers an easy-to-use GPU implementation for scattering transforms in 1D, 2D, and 3D.

Frontends are provided for many frameworks, including Num Py, scikit-learn, Py Torch, and Tensor Flow/Keras, allowing for seamless integrating scattering transforms in a variety of pipelines. In particular, the Py Torch, and Tensor Flow/Keras frontends allow for inclusion into many deep learning workflows.

Differentiability of the scattering transform simplifies applications in reconstruction and generative modeling, among others.

Table 1 provides a detailed comparison of existing implementations: Scat Net (And en et al., 2014), Scat Net Light (Oyallon and Mallat, 2015), Py Scat Wave (Oyallon et al., 2018), Scattering.m (Lostanlen and Mallat, 2015), Py Scat Harm (Eickenberg et al., 2018), and the scattering transform implemented in the MATLAB Wavelet Toolbox.

4. User Interface and Documentation

Interface The interface is designed to be flexible and consistent across inputs and frontends. Let us consider the Py Torch frontend. We first create a scattering object by specifying the averaging scale J and the input signal shape.

from kymatio.torch import Scattering1D, Scattering2D, Harmonic Scattering3D

S = Scattering1D(J, shape=(length,)) S = Scattering2D(J, shape=(height, width)) S = Harmonic Scattering3D(J, shape=(height, width, depth))

The resulting object S acts like a nn.Module object in Py Torch. The scattering transform S is applied through calls of the form

x = torch.randn((28, 28)) output = S(x)

Switching from GPU or CPU functionality also follows the API of nn.module.

S.cuda() # Run on GPU S.cpu() # Run on CPU

Documentation and examples Several examples are provided with the code, illustrating the power of Kymatio. These include image reconstruction and generation from scattering (Angles and Mallat, 2018), hybrid scattering and CNN training on CIFAR and MNIST (Oyallon et al., 2018), regression of molecular properties on QM7/QM9 using solid harmonic scattering (Eickenberg et al., 2017), and classifying recordings of spoken digits.

Kymatio: Scattering Transforms in Python

5. Conclusion

Kymatio provides a well documented, user-friendly, and fast implementation for the scattering transform. It can be used with the Py Torch and Tensor Flow/Keras deep learning frameworks and supports a variety of applications that have been previously inaccessible to non-experts including hybrid deep learning, generative modeling, and 3D chemistry applications. Future work includes further optimization for speed, flexibility, and backend support.

Acknowledgments

We thank Laurent Sifre, Sergey Zagoruyko and Gabriel Huang for their helpful comments. The project was supported by ERC Invariant Class 320959. EB is funded by a Google Focused Research Award and IVADO. MJH is partially supported by Alfred P. Sloan Fellowship #FG-2016-6607, DARPA Young Faculty Award #D16AP00117, and NSF grant #1620216. The Flatiron Institute is a division of the Simons Foundation.

T. Adel, T. Cohen, M. Caan, M. Welling, et al. 3D scattering transforms for disease classification in neuroimaging. Neuro Image: Clinical, 14:506 517, 2017. doi: 10.1016/j. nicl.2017.02.004.

J. And en et al. Scatnet. Computer Software, 2014. URL http://www.di.ens.fr/data/ software/scatnet.

J. And en and S. Mallat. Deep scattering spectrum. IEEE Trans. Signal Process., 62(16): 4114 4128, Aug 2014. doi: 10.1109/TSP.2014.2326991.

T. Angles and S. Mallat. Generative networks as inverse problems with scattering transforms. In Proc. ICLR, 2018.

J. Bruna and S. Mallat. Invariant scattering convolution networks. IEEE Trans. Pattern Anal. Mach. Intell., 35(8):1872 1886, 2013. doi: 10.1109/TPAMI.2012.230.

J. Bruna, S. Mallat, E. Bacry, and J.-F. Muzy. Intermittent process analysis with scattering moments. Ann. Statist., 43(1):323 351, 02 2015. doi: 10.1214/14-AOS1276.

V. Chud aˇcek et al. Scattering transform for intrapartum fetal heart rate variability fractal analysis: A case-control study. IEEE Trans. Biomed. Eng., 61(4):1100 1108, 2014. doi: 10.1109/TBME.2013.2294324.

M. Eickenberg et al. Solid harmonic wavelet scattering: Predicting quantum molecular energy from invariant descriptors of 3D electronic densities. In Proc. NIPS, pages 6540 6549, 2017.

M. Eickenberg et al. Solid harmonic wavelet scattering for predictions of molecule properties. The Journal of Chemical Physics, 148(24):241732, 2018. doi: 10.1063/1.5023798.

V. Lostanlen and S. Mallat. Wavelet scattering on the pitch spiral. In Proc. DAFx, 2015.

Andreux et al.

S. Mallat. Group invariant scattering. Comm. Pure Appl. Math., 65(10):1331 1398, 2012. doi: 10.1002/cpa.21413.

E. Oyallon and S. Mallat. Deep roto-translation scattering for object classification. In Proc. CVPR, June 2015.

E. Oyallon et al. Scattering networks for hybrid representation learning. IEEE Trans. Pattern Anal. Mach. Intell., 41(9):2208 2221, 2018. doi: 10.1109/TPAMI.2018.2855738.

F. Pedregosa et al. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res., 12(Oct): 2825 2830, 2011.

T. N. Sainath et al. Deep scattering spectra with deep neural networks for LVCSR tasks. In Proc. Interspeech, 2014.

L. Sifre and S. Mallat. Rotation, scaling and deformation invariant scattering for texture discrimination. In Proc. CVPR, 2013. doi: 10.1109/CVPR.2013.163.

I. Waldspurger. Exponential decay of scattering coefficients. In Proc. Samp TA, pages 143 146, 2017. doi: 10.1109/SAMPTA.2017.8024473.

N. Zeghidour et al. A deep scattering spectrum deep siamese network pipeline for unsupervised acoustic modeling. In Proc. ICASSP, pages 4965 4969. IEEE, 2016. doi: 10.1109/ICASSP.2016.7472622.