# learning_deep_features_in_instrumental_variable_regression__22b651bd.pdf

Published as a conference paper at ICLR 2021

Learning Deep Features in Instrumental Variable Regression

Liyuan Xu Gatsby Unit liyuan.jo.19@ucl.ac.uk

Yutian Chen Deep Mind yutianc@google.com

Siddarth Srinivasan University of Washington sidsrini@cs.washington.edu

Nando de Freitas Deep Mind nandodefreitas@google.com

Arnaud Doucet Deep Mind arnauddoucet@google.com

Arthur Gretton Gatsby Unit arthur.gretton@gmail.com

Instrumental variable (IV) regression is a standard strategy for learning causal relationships between confounded treatment and outcome variables from observational data by using an instrumental variable, which aﬀects the outcome only through the treatment. In classical IV regression, learning proceeds in two stages: stage 1 performs linear regression from the instrument to the treatment; and stage 2 performs linear regression from the treatment to the outcome, conditioned on the instrument. We propose a novel method, deep feature instrumental variable regression (DFIV), to address the case where relations between instruments, treatments, and outcomes may be nonlinear. In this case, deep neural nets are trained to deﬁne informative nonlinear features on the instruments and treatments. We propose an alternating training regime for these features to ensure good end-to-end performance when composing stages 1 and 2, thus obtaining highly ﬂexible feature maps in a computationally eﬃcient manner. DFIV outperforms recent state-of-the-art methods on challenging IV benchmarks, including settings involving high dimensional image data. DFIV also exhibits competitive performance in oﬀ-policy policy evaluation for reinforcement learning, which can be understood as an IV regression task.

1 Introduction

The aim of supervised learning is to obtain a model based on samples observed from some data generating process, and to then make predictions about new samples generated from the same distribution. If our goal is to predict the eﬀect of our actions on the world, however, our aim becomes to assess the inﬂuence of interventions on this data generating process. To answer such causal questions, a supervised learning approach is inappropriate, since our interventions, called treatments, may aﬀect the underlying distribution of the variable of interest, which is called the outcome.

To answer these counterfactual questions, we need to learn how treatment variables causally aﬀect the distribution process of outcomes, which is expressed in a structural function. Learning a structural function from observational data (that is, data where we can observe, but not intervene) is known to be challenging if there exists an unmeasured confounder, which inﬂuences both treatment and outcome. To illustrate: suppose we are interested in predicting sales of airplane tickets given price. During the holiday season, we would observe the simultaneous increase in sales and prices. This does not mean that raising the price

Published as a conference paper at ICLR 2021

causes the sales to increase. In this context, the time of the year is a confounder, since it aﬀects both the sales and the prices, and we need to correct the bias caused by it.

One way of correcting such bias is via instrumental variable (IV) regression (Stock and Trebbi, 2003). Here, the structural function is learned using instrumental variables, which only aﬀect the treatment directly but not the outcome. In the sales prediction scenario, we can use supply cost shifters as the instrumental variable since they only aﬀect the price (Wright, 1928; Blundell et al., 2012). Instrumental variables can be found in many contexts, and IV regression is extensively used by economists and epidemiologists. For example, IV regression is used for measuring the eﬀect of a drug in the scenario of imperfect compliance (Angrist et al., 1996), or the inﬂuence of military service on lifetime earnings (Angrist, 1990). In this work, we propose a novel IV regression method, which can discover non-linear causal relationships using deep neural networks.

Classically, IV regression is solved by the two-stage least squares (2SLS) algorithm; we learn a mapping from the instrument to the treatment in the ﬁrst stage, and learn the structural function in the second stage as the mapping from the conditional expectation of the treatment given the instrument (obtained from stage 1) to the outcome. Originally, 2SLS assumes linear relationships in both stages, but this has been recently extended to non-linear settings.

One approach has been to use non-linear feature maps. Sieve IV performs regression using a dictionary of nonlinear basis functions, which increases in size as the number of samples increases (Newey and Powell, 2003; Blundell et al., 2007; Chen and Pouzo, 2012; Chen and Christensen, 2018). Kernel IV (KIV) (Singh et al., 2019) and Dual IV regression (Muandet et al., 2020) use diﬀerent (and potentially inﬁnite) dictionaries of basis functions from reproducing kernel Hibert spaces (RKHS). Although these methods enjoy desirable theoretical properties, the ﬂexibility of the model is limited, since all existing work uses pre-speciﬁed features.

Another approach is to perform the stage 1 regression through conditional density estimation (Carrasco et al., 2007; Darolles et al., 2011; Hartford et al., 2017). One advantage of this approach is that it allows for ﬂexible models, including deep neural nets, as proposed in the Deep IV algorithm of (Hartford et al., 2017). It is known, however, that conditional density estimation is costly and often suﬀers from high variance when the treatment is high-dimensional.

More recently, Bennett et al. (2019) have proposed Deep GMM, a method inspired by the optimally weighted Generalized Method of Moments (GMM) (Hansen, 1982) to ﬁnd a structural function ensuring that the regression residual and the instrument are independent. Although this approach can handle high-dimensional treatment variables and deep NNs as feature extractors, the learning procedure might not be as stable as the 2SLS approach, since it involves solving a smooth zero-sum game, as when training Generative Adversarial Networks (Wiatrak et al., 2019).

In this paper, we propose Deep Feature Instrumental Variable Regression (DFIV), which aims to combine the advantages of all previous approaches, while avoiding their limitations. In DFIV, we use deep neural nets to adaptively learn feature maps in the 2SLS approach, which allows us to ﬁt highly nonlinear structural functions, as in Deep GMM and Deep IV. Unlike Deep IV, DFIV does not rely on conditional density estimation. Like sieve IV and KIV, DFIV learns the conditional expectation of the feature maps in stage 1 and uses the predicted features in stage 2, but with the additional advantage of learned features. We empirically show that DFIV performs better than other methods on several IV benchmarks, and apply DFIV successfully to oﬀ-policy policy evaluation, which is a fundamental problem in Reinforcement Learning (RL).

The paper is structured as follows. In Section 2, we formulate the IV regression problem and introduce two-stage least-squares regression. In Section 3, we give a detailed description of our DFIV method. We demonstrate the empirical performance of DFIV in Section 4, covering three settings: a classical demand prediction example from econometrics, a challenging IV setting where the treatment consists of high-dimensional image data, and the problem of oﬀ-policy policy evaluation in reinforcement learning.

Published as a conference paper at ICLR 2021

2 Preliminaries

2.1 Problem Setting of Instrumental Variable Regression

Figure 1: Causal Graph.

We begin with a description of the IV setting. We observe a treatment X X, where X Rd X, and an outcome Y Y, where Y R. We also have an unobserved confounder that aﬀects both X and Y . This causal relationship can be represented with the following structural causal model:

Y = fstruct(X) + ε, E [ε] = 0, E [ε|X] = 0, (1)

where fstruct is called the structural function, which we assume to be continuous, and ε is an additive noise term. This speciﬁc confounding assumption is necessary for the IV problem. In Bareinboim and Pearl (2012), it is shown that we cannot learn fstruct if we allow any type of confounders. The challenge is that E [ε|X] = 0, which reﬂects the existence of a confounder. Hence, we cannot use ordinary supervised learning techniques since fstruct(x) = E [Y |X = x]. Here, we assume there is no observable confounder but we may easily include this, as discussed in Appendix C.

To deal with the hidden confounder ε, we assume to have access to an instrumental variable Z Z which satisﬁes the following assumption.

Assumption 1. The conditional distribution P(X|Z) is not constant in Z and E [ε|Z] = 0.

Intuitively, Assumption 1 means that the instrument Z induces variation in the treatment X but is uncorrelated with the hidden confounder ε. Again, for simplicity, we assume Z Rd Z. The causal graph describing these relationships is shown in Figure 1.1. Note that the instrument Z cannot have an incoming edge from the latent confounder that is also a parent of the outcome.

Given Assumption 1, we can see that the function fstruct satisﬁes the operator equation E [Y |Z] = E [fstruct(X)|Z] by taking expectation conditional on Z of both sides of (1). Newey and Powell (2003) provide necessary and suﬃcient conditions, known as completeness assumptions, to ensure identiﬁability of fstruct(X). Solving this equation, however, is known to be ill-posed (Nashed and Wahba, 1974). To address this, recent works (Carrasco et al., 2007; Darolles et al., 2011; Muandet et al., 2020; Singh et al., 2019) minimize the following regularized loss L to obtain the estimate ˆfstruct:

ˆfstruct = arg min f F L(f), L(f) = EY Z (Y EX|Z [f(X)])2 + Ω(f), (2)

where F is an arbitrary space of continuous functions and Ω(f) is a regularizer on f.

2.2 Two Stage Least Squares Regression

A number of works (Newey and Powell, 2003; Singh et al., 2019) tackle the minimization problem (2) using two-stage least squares (2SLS) regression, in which the structural function is modeled as fstruct(x) = u ψ(x), where u is a learnable weight vector and ψ(x) is a vector of ﬁxed basis functions. For example, linear 2SLS used the identity map ψ(x) = x, while sieve IV (Newey and Powell, 2003) uses Hermite polynomials.

In the 2SLS approach, an estimate ˆu is obtained by solving two regression problems successively. In stage 1, we estimate the conditional expectation EX|z [ψ(X)] as a function of z. Then in stage 2, as EX|z [f(X)] = u EX|z [ψ(X)], we minimize L with EX|z [f(X)] being replaced by the estimate obtained in stage 1.

Speciﬁcally, we model the conditional expectation as EX|z [ψ(X)] = V φ(z), where φ(z) is another vector of basis functions and V is a matrix to be learned. Again, there exist many

1We show the simplest causal graph in Figure 1 It entails Z ε, but we only require Z and ε to be uncorrelated in Assumption 1. Of course, this graph also says that Z is not independent of ε when conditioned on observations X.

Published as a conference paper at ICLR 2021

choices for φ(z), which can be inﬁnite-dimensional, but we assume the dimensions of ψ(x) and φ(z) to be d1, d2 < respectively.

In stage 1, the matrix V is learned by minimizing the following loss,

ˆV = arg min V Rd1 d2 L1(V ), L1(V ) = EX,Z ψ(X) V φ(Z) 2 + λ1 V 2, (3)

where λ1 > 0 is a regularization parameter. This is a linear ridge regression problem with multiple targets, which can be solved analytically. In stage 2, given ˆV , we can obtain u by minimizing the loss

ˆu = arg min u Rd1 L2(u), L2(u) = EY,Z h Y u ˆV φ(Z) 2i + λ2 u 2, (4)

where λ2 > 0 is another regularization parameter. Stage 2 corresponds to a ridge linear regression from ˆV φ(Z) to Y , and also enjoys a closed-form solution. Given the learned weights ˆu, the estimated structural function is ˆfstruct(x) = ˆu ψ(x).

3 DFIV Algorithm

In this section, we develop the DFIV algorithm. Similarly to Singh et al. (2019), we assume that we do not necessarily have access to observations from the joint distribution of (X, Y, Z). Instead, we are given m observations of (X, Z) for stage 1 and n observations of (Y, Z) for stage 2. We denote the stage 1 observations as (xi, zi) and the stage 2 observations as ( yi, zi). If observations of (X, Y, Z) are given for both stages, we can evaluate the out-of-sample losses, and these losses can be used for hyper-parameter tuning of λ1, λ2 (Appendix A).

DFIV uses the following models

fstruct(x) = u ψθX(x) and EX|z [ψθX(X)] = V φθZ(z), (5)

where u Rd1 and V Rd1 d2 are the parameters, and ψθX(x) Rd1 and φθZ(z) Rd2 are the neural nets parameterised by θX ΘX and θZ ΘZ, respectively. As in the original 2SLS algorithm, we learn EX|z [ψθX(X)] in stage 1 and fstruct(x) in stage 2. In addition to the weights u and V , however, we also learn the parameters of the feature maps, θX and θZ. Hence, we need to alternate between stages 1 and 2, since the conditional expectation EX|z [ψθX(X)] changes during training.

Stage 1 Regression The goal of stage 1 is to estimate the conditional expectation EX|z [ψθX(X)] V φθZ(z) by learning the matrix V and parameter θZ, with θX = ˆθX given and ﬁxed. Given the stage 1 data (xi, zi), this can be done by minimizing the empirical estimate of L1,

ˆV (m), ˆθZ = arg min V Rd1 d2,θZ ΘZ L(m) 1 (V , θZ), L(m) 1 = 1

i=1 ψˆθX(xi) V φθZ(zi) 2 + λ1 V 2.

Note that the feature map ψˆθX(X) is ﬁxed during stage 1, since this is the target variable. If we ﬁx θZ, the minimization problem (6) reduces to a linear ridge regression problem with multiple targets, whose solution as a function of θX and θZ is given analytically by

ˆV (m)(θX, θZ) = Ψ 1 Φ1(Φ 1 Φ1 + mλ1I) 1, (7)

where Φ1, Ψ1 are feature matrices deﬁned as Ψ1 = [ψθX(x1), . . . , ψθX(xm)] Rm d1 and Φ1 = [φθZ(z1), . . . , φθZ(zm)] Rm d2. We can then learn the parameters θZ of the adaptive features ψθZ by minimizing the loss L(m) 1 at V = ˆV (m)(ˆθX, θZ) using gradient descent. For simplicity, we introduce a small abuse of notation by denoting as ˆθZ the result of a user-chosen number of gradient descent steps on the loss (6) with ˆV (m)(ˆθX, θZ) from (7), even though ˆθZ need not attain the minimum of the non-convex loss (6). We then write

Published as a conference paper at ICLR 2021

ˆV (m) := ˆV (m)(ˆθX, ˆθZ). While this trick of using an analytical estimate of the linear output weights of a deep neural network might not lead to signiﬁcant gains in standard supervised learning, it turns out to be very important in the development of our 2SLS algorithm. As shown in the following section, the analytical estimate ˆV (m)(θX, ˆθZ) (now considered as a function of θX) will be used to backpropagate to θX in stage 2.

Stage 2 Regression In stage 2, we learn the structural function by computing the weight vector u and parameter θX while ﬁxing θZ = ˆθZ, and thus the corresponding feature map φˆθZ(z). Given the data ( yi, zi), we can minimize the empirical version of L2, deﬁned as

ˆu(n), ˆθX = arg min u Rd1,θX ΘX L(n) 2 (u, θX), L(n) 2 = 1

i=1 ( yi u ˆV (m)φˆθZ( zi))2 + λ2 u 2. (8)

Again, for a given θX, we can solve the minimization problem (8) for u as a function of ˆV (m) := ˆV (m)(θX, ˆθZ) by a linear ridge regression

ˆu(n)(θX, ˆθZ) = ˆV (m)Φ 2 Φ2( ˆV (m)) + nλ2I 1 ˆV (m)Φ 2 y2, (9)

where Φ2 = [φˆθZ( z1), . . . , φˆθZ( zn)] Rn d2 and y2 = [ y1, . . . , yn] Rn.

The loss L(n) 2 explicitly depends on the parameters θX and we can backpropagate it to θX via ˆV (m)(θX, ˆθZ), even though the samples of the treatment variable X do not appear in stage 2 regression. We again introduce a small abuse of notation for simplicity, and denote by ˆθX the estimate obtained after a few gradient steps on (8) with ˆu(n)(θX, ˆθZ) from (9), even though ˆθX need not minimize the non-convex loss (8). We then have ˆu(n) = ˆu(n)(ˆθX, ˆθZ). After updating ˆθX, we need to update ˆθZ accordingly. We do not attempt to backpropagate through the estimate ˆθZ to do this, however, as this would be too computationally expensive; instead, we alternate stages 1 and 2. We also considered updating ˆθX and ˆθZ jointly to optimize the loss L(n) 2 , but this fails, as discussed in Appendix F.

Computational Complexity and Convergence The computational complexity of the algorithm is O(md1d2 + d3 2) for stage 1, while stage 2 requires additional O(nd1d2 + d3 1) computations. This is small compared to KIV (Singh et al., 2019), which takes O(m3) and O(n3), respectively. We can further speed up the learning by using mini-batch training as shown in Algorithm 1. Here, ˆV (mb) and ˆu(nb) are the functions given by (7) and (9) calculated using mini-batches of data. Similarly, L(mb) 1 and L(nb) 1 are the stage 1 and 2 losses for the mini-batches. We recommend setting the batch size large enough so that ˆV (mb), ˆu(nb)

do not diverge from ˆV (m), ˆu(n) computed on the entire dataset. Furthermore, we observe that setting T1 > T2, i.e. updating θZ more frequently than θX, stabilizes the learning process.

In Appendix B, we provide regularity conditions under which the function learned by DFIV converges to the true structural function in probability. The derivation is based on Rademacher complexity bounds (Mohri et al., 2012).

4 Experiments

In this section, we report the empirical performance of the DFIV method. The evaluation considers both low and high-dimensional treatment variables. We used the demand design dataset of Hartford et al. (2017) for benchmarking in the low and high-dimensional cases, and we propose a new setting for the high-dimensional case based on the d Sprites dataset (Matthey et al., 2017). In the deep RL context, we also apply DFIV to perform oﬀ-policy policy evaluation (OPE). The network architecture and hyper-parameters are provided in Appendix G. The algorithms in the ﬁrst two experiments are implemented using Py Torch (Paszke et al., 2019) and the OPE experiments are implemented using Tensor Flow (Abadi et al., 2015) and the Acme RL framework (Hoﬀman et al., 2020). The code is included in the supplemental material.

Published as a conference paper at ICLR 2021

Algorithm 1 Deep Feature Instrumental Variable Regression

Input: Stage 1 data (xi, zi), Stage 2 data ( yi zi), Regularization parameters (λ1, λ2). Initial values ˆθX, ˆθZ. Mini-batch size (mb, nb). Number of updates in each stage (T1, T2). Output: Estimated structural function ˆfstruct(x)

1: repeat 2: Sample mb stage 1 data (x(b) i , z(b) i ) and nb stage 2 data ( y(b) i , z(b) i ). 3: for t = 1 to T1 do 4: Return function ˆV (mb)(ˆθX, θZ) in (7) using (x(b) i , z(b) i )

5: Update ˆθZ ˆθZ α θZL(mb) 1 ( ˆV (mb)(ˆθX, θZ), θZ)|θZ=ˆθZ \\ Stage 1 learning 6: end for 7: for t = 1 to T2 do 8: Return function ˆu(nb)(θX, ˆθZ) in (9) using ( y(b) i , z(b) i ) and function ˆV (mb)(θX, ˆθZ)

9: Update ˆθX ˆθX α θXL(nb) 2 (ˆu(nb)(θX, ˆθZ), θX)|θX=ˆθX \\ Stage 2 learning 10: end for 11: until convergence 12: Compute ˆu(n) := ˆu(n)(ˆθX, ˆθZ) from (9) using entire dataset. 13: return ˆfstruct(x) = (ˆu(n)) ψˆθX(x)

4.1 Demand Design Experiments

The demand design dataset is a synthetic dataset introduced by Hartford et al. (2017) that is now a standard benchmarking dataset for testing nonlinear IV methods. In this dataset, we aim to predict the demands on airplane tickets Y given the price of the tickets P. The dataset contains two observable confounders, which are the time of year T [0, 10] and customer groups S {1, ..., 7} that are categorized by the levels of price sensitivity. Further, the noise in Y and P is correlated, which indicates the existence of an unobserved confounder. The strength of the correlation is represented by ρ [0, 1]. To correct the bias caused by this hidden confounder, the fuel price C is introduced as an instrumental variable. Details of the data generation process can be found in Appendix E.1. In DFIV notation, the treatment is X = P, the instrument is Z = C, and (T, S) are the observable confounders.

We compare the DFIV method to three leading modern competitors, namely KIV (Singh et al., 2019), Deep IV (Hartford et al., 2017), and Deep GMM (Bennett et al., 2019). We used the DFIV method with observable confounders, as introduced in Appendix C. Note that Deep GMM does not have an explicit mechanism for incorporating observable confounders. The solution we use, proposed by Bennett et al. (2019, p. 2), is to incorporate these observables in both instrument and treatment; hence we apply Deep GMM with treatment X = (P, T, S) and instrumental variable Z = (C, T, S). Although this approach is theoretically sound, this makes the problem unnecessary diﬃcult since it ignores the fact that we only need to consider the conditional expectation of P given Z.

We used a network with a similar number of parameters to Deep IV as the feature maps in DFIV and models in Deep GMM. We tuned the regularizers λ1, λ2 as discussed in Appendix A, with the data evenly split for stage 1 and stage 2. We varied the correlation parameter ρ and dataset size, and ran 20 simulations for each setting. Results are summarized in Figure 2. We also evaluated the performance via the estimation of average treatment eﬀect and conditional average treatment eﬀect, which is presented in Appendix E.2

Next, we consider a case, introduced by Hartford et al. (2017), where the customer type S {1, . . . , 7} is replaced with an image of the corresponding handwritten digit from the MNIST dataset (Le Cun and Cortes, 2010). This reﬂects the fact that we cannot know the exact customer type, and thus we need to estimate it from noisy high-dimensional data. Note that although the confounder is high-dimensional, the treatment variable is still realvalued, i.e. the price P of the tickets. Figure 3 presents the results for this high-dimensional confounding case. Again, we train the networks with a similar number of learnable parameters to Deep IV in DFIV and Deep GMM, and hyper-parameters are set in the way discussed in

Published as a conference paper at ICLR 2021

1000 5000 10000

Out-of-Sample MSE

1000 5000 10000

1000 5000 10000

1000 5000 10000

1000 5000 10000 Training Sample Size

Figure 2: MSE for demand design dataset with low dimensional confounders.

0.1 0.25 0.5 0.75 0.9 Confounding Level

Out-of-Sample MSE ( 104)

Figure 3: MSE for demand design dataset with high dimensional observed confounders.

5000 10000 Training Sample Size

Out-of-Sample MSE

Figure 4: MSE for d Sprite dataset. Deep IV did not yield meaningful predictions for this experiment.

Appendix A. We ran 20 simulations with data size n + m = 5000 and report the mean and standard error.

Our ﬁrst observation from Figure 2 and 3 is that the level ρ of correlation has no signiﬁcant impact on the error under any of the IV methods, indicating that all approaches correctly account for the eﬀect of the hidden confounder. This is consistent with earlier results on this dataset using Deep IV and KIV (Hartford et al., 2017; Singh et al., 2019). We note that Deep GMM does not perform well in this demand design problem. This may be due to the current Deep GMM approach to handling observable confounders, which might not be optimal. KIV performed reasonably well for small sample sizes and low-dimensional data, but it did less well in the high-dimensional MNIST case due to its less expressive features. In high dimensions, Deep IV performed well, since the treatment variable is unidimensional. However, DFIV performed consistently better than all other methods in both low and high dimensions, which suggests it can learn a ﬂexible structural function in a stable manner.

4.2 d Sprites Experiments

Figure 5: d Sprite image

To test the performance of DFIV methods for a high dimensional treatment variable, we utilized the d Sprites dataset (Matthey et al., 2017). This is an image dataset described by ﬁve latent parameters (shape, scale, rotation, pos X and pos Y). The images are 64 64 = 4096-dimensional. In this experiment, we ﬁxed the shape parameter to heart, i.e. we only used heartshaped images. An example is shown in Figure 5.

From this dataset, we generated data for IV regression in which we use each ﬁgure as treatment variable X. Hence, the treatment variable is 4096-dimensional in this experiment. To make the task more challenging, we used pos Y as the hidden confounder, which is not revealed to the model. We used the other three latent variables as the instrument variables Z. The structural function fstruct and outcome Y are deﬁned as

fstruct(X) = AX 2 2 5000 1000 , Y = fstruct(X) + 32(pos Y 0.5) + ε, ε N(0, 0.5),

Published as a conference paper at ICLR 2021

where each element of the matrix A R10 4096 is generated from Unif(0.0, 1.0) and ﬁxed throughout the experiment. See Appendix E.3 for the detailed data generation process.

We tested the performance of DFIV with KIV and Deep GMM, where the hyper-parameters are determined as in the demand design problem. The results are displayed in Figure 4. DFIV consistently yields the best performance of all the methods. Deep IV is not included in the ﬁgure because it fails to give meaningful predictions due to the diﬃculty of performing conditional density estimation for the high-dimensional treatment variable. The performance of KIV suﬀers since it lacks the feature richness to express a high-dimensional complex structural function. Although Deep GMM performs comparatively to DFIV, we observe some instability during the training, see Appendix E.4.

4.3 Off-Policy Policy Evaluation Experiments

We apply our IV methods to the oﬀ-policy policy evaluation (OPE) problem (Sutton and Barto, 2018), which is one of the fundamental problems of deep RL. In particular, it has been realized by Bradtke and Barto (1996) that 2SLS could be used to estimate a linearly parameterized value function, and we use this reasoning as the basis of our approach. Let us consider the RL environment S, A, P, R, ρ0, γ , where S is the state space, A is the action space, P : S A S [0, 1] is the transition function, R : S A S R R is the reward distribution, ρ0 : S [0, 1] is the initial state distribution, and discount factor γ (0, 1]. Let π be a policy, and we denote π(a|s) as the probability of selecting action a in stage s S. Given policy π, the Q-function is deﬁned as

Qπ(s, a) = E

s0 = s, a0 = a

with at π( | st), st+1 P( |st, at), rt R( |st, at, st+1). The goal of OPE is to evaluate the expectation of the Q-function with respect to the initial state distribution for a given target policy π, Es ρ0,a|s π[Qπ(s, a)], learned from a ﬁxed dataset of transitions (s, a, r, s ), where s and a are sampled from some potentially unknown distribution µ and behavioral policy πb( |s) respectively. Using the Bellman equation satisﬁed by Qπ, we obtain a structural causal model of the form (1),

structural function fstruct(s,a,s ,a ) z }| { Qπ(s, a) γQπ(s , a ) (10)

+ γ Qπ(s , a ) Es P ( |s,a),a π( |s ) [Qπ(s , a )] + r Er R( |s,a,s ) [r] | {z } confounder ε

where X = (s, a, s , a ), Z = (s, a), Y = r. We have that E [ε] = 0, E [ε|X] = 0, and Assumption 1 is veriﬁed. Minimizing the loss (2) for the structural causal model (10) corresponds to minimizing the following loss LOPE

LOPE = Es,a,r h r + γEs P ( |s,a),a π( |s ) [Qπ(s , a )] Qπ(s, a) 2i , (11)

and we can apply any IV regression method to achieve this. In Appendix D, we show that minimizing LOPE corresponds to minimizing the mean squared Bellman error (MSBE) (Sutton and Barto, 2018, p. 268) and we detail the DFIV algorithm for OPE. Note that MBSE is also the loss minimized by the residual gradient (RG) method proposed in (Baird, 1995) to estimate Q-functions. However, this method suﬀers from the double-sample issue, i.e. it requires two independent samples of s starting from the same (s, a) due to the inner conditional expectation (Baird, 1995), whereas IV regression methods do not suﬀer from this issue.

We evaluate DFIV on three BSuite (Osband et al., 2019) tasks: catch, mountain car, and cartpole. See Section E.6.1 for a description of those tasks. The original system dynamics are deterministic. To create a stochastic environment, we randomly replace the agent action by a uniformly sampled action with probability p [0, 0.5]. The noise level p controls the level of confounding eﬀect. The target policy is trained using DQN (Mnih et al., 2015), and we subsequently generate an oﬄine dataset for OPE by executing the policy in the

Published as a conference paper at ICLR 2021

0.0 0.1 0.2 0.3 0.4 0.5

Absolute Error

0.0 0.1 0.2 0.3 0.4 0.5

0.0 0.1 0.2 0.3 0.4 0.5 Noise Level

mountain_car

Figure 6: Error of oﬄine policy evaluation.

same environment with a random action probability of 0.2 (on top of the environment s random action probability p). We compare DFIV with KIV, Deep IV, and Deep GMM; as well as Fitted Q Evaluation (FQE) (Le et al., 2019; Voloshin et al., 2019), a specialized approach designed for the OPE setting, which serves as our gold standard baseline (Paine et al., 2020) (see Section E.6.2 for details). All methods use the same network for value estimation. Figure 6 shows the absolute error of the estimated policy value by each method with a standard deviation from 5 runs. In catch and mountain car, DFIV comes closest in performance to FQE, and even matches it for some noise settings, whereas Deep GMM is somewhat worse in catch, and signiﬁcantly worse in mountain car. In the case of cartpole, Deep GMM performs somewhat better than DFIV, although both are slightly worse than FQE. Deep IV and KIV both do poorly across all RL benchmarks.

5 Conclusion

We have proposed a novel method for instrumental variable regression, Deep Feature IV (DFIV), which performs two-stage least squares regression on ﬂexible and expressive features of the instrument and treatment. As a contribution to the IV literature, we showed how to adaptively learn these feature maps with deep neural networks. We also showed that the oﬀ-policy policy evaluation (OPE) problem in deep RL can be interpreted as a nonlinear IV regression, and that DFIV performs competitively in this domain. This work thus brings the research worlds of deep oﬄine RL and causality from observational data closer.

In terms of future work, it would be interesting to adapt the ideas from (Angrist and Krueger, 1995; Angrist et al., 1999; Hansen and Kozbur, 2014) to select the regularization hyperparameters of DFIV as well as investigate generalizations of DFIV beyond the additive model (1) as considered in (Carrasco et al., 2007, Section 5.5). In RL, problems with additional confounders are common, see e.g. (Namkoong et al., 2020; Shang et al., 2019), and we believe that adapting DFIV to this setting will be of great value.

M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng. Tensor Flow: Large-scale machine learning on heterogeneous systems, 2015. URL http://tensorflow.org/.

J. D. Angrist. Lifetime earnings and the Vietnam era draft lottery: Evidence from social security administrative records. The American Economic Review, 80(3):313 336, 1990.

J. D. Angrist and A. B. Krueger. Split-sample instrumental variables estimates of the return to schooling. Journal of Business & Economic Statistics, 13(2):225 235, 1995.

Published as a conference paper at ICLR 2021

J. D. Angrist, G. W. Imbens, and D. B. Rubin. Identiﬁcation of causal eﬀects using instrumental variables. Journal of the American Statistical Association, 91(434):444 455, 1996.

J. D. Angrist, G. W. Imbens, and A. B. Krueger. Jackknife instrumental variables estimation. Journal of Applied Econometrics, 14(1):57 67, 1999.

L. Baird. Residual algorithms: Reinforcement learning with function approximation. In Proceedings of the 12th International Conference on Machine Learning, 1995.

E. Bareinboim and J. Pearl. Causal inference by surrogate experiments: Z-identiﬁability. In Proceedings of the 28th Conference on Uncertainty in Artiﬁcial Intelligence, page 113 120, 2012.

A. G. Barto, R. S. Sutton, and C. W. Anderson. Neuronlike adaptive elements that can solve diﬃcult learning control problems. IEEE transactions on Systems, Man, and Cybernetics, (5):834 846, 1983.

A. Bennett, N. Kallus, and T. Schnabel. Deep generalized method of moments for instrumental variable analysis. In Advances in Neural Information Processing Systems 32, pages 3564 3574. 2019.

R. Blundell, X. Chen, and D. Kristensen. Semi-nonparametric IV estimation of shapeinvariant engel curves. Econometrica, 75(6):1613 1669, 2007.

R. Blundell, J. Horowitz, and M. Parey. Measuring the price responsiveness of gasoline demand: Economic shape restrictions and nonparametric demand estimation. Quantitative Economics, 3:29 51, 2012.

S. J. Bradtke and A. G. Barto. Linear least-squares algorithms for temporal diﬀerence learning. Machine Learning, 22(1-3):33 57, 1996.

M. Carrasco, J.-P. Florens, and E. Renault. Linear inverse problems in structural econometrics estimation based on spectral decomposition and regularization. In Handbook of Econometrics, volume 6B, chapter 77. 2007.

X. Chen and T. M. Christensen. Optimal sup-norm rates and uniform inference on nonlinear functionals of nonparametric IV regression: Nonlinear functionals of nonparametric IV. Quantitative Economics, 9:39 84, 2018.

X. Chen and D. Pouzo. Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals. Econometrica, 80(1):277 321, 2012.

S. Darolles, Y. Fan, J. P. Florens, and E. Renault. Nonparametric instrumental regression. Econometrica, 79(5):1541 1565, 2011.

D. Ernst, P. Geurts, and L. Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research, 6:503 556, 2005.

C. Hansen and D. Kozbur. Instrumental variables estimation with many weak instruments using regularized jive. Journal of Econometrics, 182(2):290 308, 2014.

L. P. Hansen. Large sample properties of generalized method of moments estimators. Econometrica, 50(4):1029 1054, 1982.

J. Hartford, G. Lewis, K. Leyton-Brown, and M. Taddy. Deep IV: A ﬂexible approach for counterfactual prediction. In International Conference on Machine Learning, 2017.

M. Hoﬀman, B. Shahriari, J. Aslanides, G. Barth-Maron, F. Behbahani, T. Norman, A. Abdolmaleki, A. Cassirer, F. Yang, K. Baumli, S. Henderson, A. Novikov, S. G. Colmenarejo, S. Cabi, C. Gulcehre, T. L. Paine, A. Cowie, Z. Wang, B. Piot, and N. de Freitas. Acme: A research framework for distributed reinforcement learning. ar Xiv preprint ar Xiv:2006.00979, 2020.

Published as a conference paper at ICLR 2021

H. Le, C. Voloshin, and Y. Yue. Batch policy learning under constraints. In International Conference on Machine Learning, 2019.

Y. Le Cun and C. Cortes. MNIST handwritten digit database. 2010. URL http://yann. lecun.com/exdb/mnist/.

L. Matthey, I. Higgins, D. Hassabis, and A. Lerchner. d Sprites: Disentanglement testing sprites dataset, 2017. URL https://github.com/deepmind/dsprites-dataset/.

T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.

V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.

M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning. MIT Press, 2012.

A. W. Moore. Eﬃcient Memory-Based Learning for Robot Control. Ph D thesis, Cambridge University, 1990.

K. Muandet, A. Mehrjou, S. K. Lee, and A. Raj. Dual IV: A single stage instrumental variable regression. In Advances in Neural Information Processing Systems 34, 2020.

H. Namkoong, R. Keramati, S. Yadlowsky, and E. Brunskill. Oﬀ-policy policy evaluation for sequential decisions under unobserved confounding. In Advances in Neural Information Processing Systems 34, 2020.

M. Z. Nashed and G. Wahba. Generalized inverses in reproducing kernel spaces: An approach to regularization of linear operator equations. SIAM Journal on Mathematical Analysis, 5 (6):974 987, 1974.

W. K. Newey and J. L. Powell. Instrumental variable estimation of nonparametric models. Econometrica, 71(5):1565 1578, 2003.

I. Osband, Y. Doron, M. Hessel, J. Aslanides, E. Sezener, A. Saraiva, K. Mc Kinney, T. Lattimore, C. Szepesvari, S. Singh, et al. Behaviour suite for reinforcement learning. In International Conference on Learning Representations, 2019.

T. L. Paine, C. Paduraru, A. Michi, C. Gulcehre, K. Zolna, A. Novikov, Z. Wang, and N. de Freitas. Hyperparameter selection for oﬄine reinforcement learning. ar Xiv preprint ar Xiv:2007.09055, 2020.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. De Vito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024 8035. 2019.

A. Rahimi and B. Recht. Random features for large-scale kernel machines. In Advances in Neural Information Processing Systems 20, pages 1177 1184. 2008.

W. Shang, Y. Yu, Q. Li, Z. Qin, Y. Meng, and J. Ye. Environment reconstruction with hidden confounders for reinforcement learning based recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 566 576, 2019.

R. Singh, M. Sahani, and A. Gretton. Kernel instrumental variable regression. In Advances in Neural Information Processing Systems 32, pages 4593 4605. 2019.

J. H. Stock and F. Trebbi. Retrospectives: Who invented instrumental variable regression? Journal of Economic Perspectives, 17(3):177 194, 2003.

Published as a conference paper at ICLR 2021

R. S. Sutton and A. G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018.

C. Voloshin, H. M. Le, N. Jiang, and Y. Yue. Empirical study of oﬀ-policy policy evaluation for reinforcement learning. ar Xiv preprint ar Xiv:1911.06854, 2019.

M. Wiatrak, S. V. Albrecht, and A. Nystrom. Stabilizing generative adversarial networks: A survey. ar Xiv preprint ar Xiv:1910.00927, 2019.

P. Wright. The Tariﬀon Animal and Vegetable Oils. Investigations in International Commercial Policies. Macmillan Company, 1928.

Published as a conference paper at ICLR 2021

A Hyper-Parameter Tuning

If observations from the joint distribution of (X, Y, Z) are available in both stages, we can tune the regularization parameters λ1, λ2 using the approach proposed in Singh et al. (2019). Let the complete data of stage 1 and stage 2 be denoted as (xi, yi, zi) and ( xi, yi, zi). Then, we can use the data not used in each stage to evaluate the out-of-sample performance of the other stage. Speciﬁcally, the regularizing parameters are given by

λ 1 = arg min L(n) 1-oos, L(n) 1-oos = 1

j=1 ψˆθX( xi) ˆV (m)φˆθZ( zi) 2,

λ 2 = arg min L(m) 2-oos, L(m) 2-oos = 1

i=1 (yi (ˆu(n)) ˆV (m)φˆθZ(zi))2,

where ˆu(n), ˆV (m), ˆθθX, ˆθθZ are the parameters learned in (6) and (8).

B Consistency of DFIV

In this appendix, we prove consistency of the DFIV approach. Our contribution is to establish consistency of the end-to-end procedure incorporating Stages 1 and 2, which we achieve by ﬁrst showing a Stage 1 consistency result (Lemma 1), and then establishing the consistency of Stage 2 when the empirical Stage 1 solution is used as input (Lemma 2). The desired result then follows in Theorem 1.

Consistency results will be expressed in terms of the complexity of the function classes used in Stages 1 and 2, as encoded in the Rademacher complexity of functionals of these functions (see Proposition 1 below). Consistency for particular function classes can then be shown by establishing that the respective Rademacher complexities vanish. We leave for future work the task of demonstrating this property for function classes of interest.

B.1 Operator view of DFIV

The goal of DFIV is to learn structural function fstruct, which satisﬁes EY |Z [Y ] = EX|Z [fstruct(X)] . (12)

We model fstruct as fstruct(X) = u ψθX(X) and denote the hypothesis spaces for ψθX and fstruct as follows: Hψ = {ψθX : X Rd1 | θX ΘX},

F = {u ψθX : X Y | ψθX Hψ, u Rd1}. To learn the parameters, we minimize the following stage 2 loss:

ˆu(n), ˆθX = arg min u Rd1,θX L(n) 2 (u, θX), L(n) 2 = 1

i=1 ( yi u ˆEX|Z [ψθX(X)] ( zi))2.

We denote the resulting estimated structural function as ˆfstruct(X) = (ˆu(n)) ψˆθX(X). For simplicity, we set all regularization terms to zero. Here, ˆEX|Z [ψθX(X)] is the empirical conditional expectation operator, which maps an element of Hψ to some function g : Z 7 Rd1 G that

ˆEX|Z [ψθX(X)] = arg min g G L(m) 1 (V , θZ), L(m) 1 = 1

i=1 ψθX(xi) g(zi) 2.

In DFIV, we deﬁne G as G = {V φθZ : Z Rd1 | V Rd2 d1, θZ ΘZ}. This formulation is equivalent to the one introduced in Section 3. With a slight abuse of notation, for f(X) = u ψθX(X) F, we deﬁne ˆEX|Z [f(X)] to be ˆEX|Z [f(X)] = u ˆEX|Z [ψθX(X)]

since this is the empirical estimate of EX|Z [f(X)].

Published as a conference paper at ICLR 2021

B.2 Generalization errors for regression

Here, we bound the generalization errors of both stages using Rademacher complexity bounds (Mohri et al., 2012).

Proposition 1. (Theorem 3.3 Mohri et al., 2012, with slight modiﬁcation) Let S be a measurable space and H be a family of functions mapping from S to [0, M]. Given ﬁxed dataset S = (s1, s2, . . . , sn) Sn, empirical Rademacher complexity is given by

ˆRS(H) = Eσ

i=1 σih(si)

where σ = (σ1, . . . , σn), with σi is independent uniform random variables taking values in { 1, +1}. Then, for any δ > 0, with probability at least 1 δ over the draw of an i.i.d sample S of size n, each of following holds for all h H:

i=1 h(si) + 2 ˆRS + 3M

i=1 h(si) E [h(s)] + 2 ˆRS + 3M

We list the assumptions below.

Assumption 2. The following hold:

1. Bounded outcome variable |Y | M.

2. Bounded stage 1 hypothesis space: z Z, g(z) 1.

3. Bounded stage 2 feature map x X, ψθX(x) 1.

4. Bounded stage 2 weight: u M.

5. Identiﬁable stage 1 hypothesis space: ψθX Hψ, g G, EX|Z [ψθX] = g.

6. Identiﬁable stage 2 hypothesis space: fstruct F.

Note that we leave aside questions of optimization. Thus, we assume that the optimization procedure over (θZ, V ) is suﬃcient to recover ˆEX|Z, and that the optimization procedure over (u, θX) is suﬃcient to recover ˆfstruct (which requires, in turn, the correct ˆEX|Z, for this ψθX). We emphasize that Algorithm 1 does not guarantee these properties.

Based on these assumptions, we derive the generalization error in terms of L2-norm P (Z) deﬁned as

h P (Z) = Z h(Z) 2d P(Z) 1

The following lemma proves the generalization error for stage 1 regression.

Lemma 1. Under Assumption 2, and given stage 1 data S1 = {(xi, zi)}m i=1, for any δ > 0, with at least probability 1 2δ, we have

ˆEX|Z [f(X)] EX|Z [f(X)] P (Z) M

4 ˆRS1(H1) + 24

for any f = u ψθX F, where hypothesis space H1 is deﬁned as

H1 = {(x, z) X Z 7 ψθX(x) g(z) 2 R | g G, ψθX Hψ}. (13)

Published as a conference paper at ICLR 2021

Proof. From Cauchy Schwarz inequality, we have E [f(X)|Z] ˆEX|Z [f(X)] P (Z) = (u EX|Z [ψθX(X)] ˆEX|Z [ψθX(X)] P (Z)

M EX|Z [ψθX(X)] ˆEX|Z [ψθX(X)] P (Z)

By applying Proposition 1 to hypothesis space H1, we have

ψθX(X) ˆEX|Z [ψθX(X)] 2

ψθX(xi) ˆEX|Z [ψθX(X)] (zi) 2 + 2 ˆRS1(H1) + 12

with probability 1 δ. Indeed all functions h H1 satisfy h 4 since ψθX(X) 1, g(Z) 1 from Assumption 2.

Also, since ψθX Hψ, g G, EX|Z [ψθX] = g, again from Proposition 1, we have

ψθX(xi) EX|Z [ψθX(X)] (zi) 2

EXZ h ψθX(X) EX|Z [ψθX(X)] 2i + 2 ˆRS1(H1) + 12

with probability 1 δ. From the optimality of ˆEX|Z [ψθX(X)], we have

ψθX(xi) ˆEX|Z [ψθX] (zi) 2 1

ψθX(xi) EX|Z [ψθX(X)] (zi) 2.

Hence, we have

ψθX(X) ˆEX|Z [ψθX(X)] 2

EXZ h ψθX(X) EX|Z [ψθX(X)] 2i + 4 ˆRS1(H1) + 24

with probability 1 2δ. Now we have

ψθX(X) ˆEX|Z [ψθX(X)] 2

= EXZ h ψθX(X) EX|Z [ψθX(X)] 2i + EZ

EX|Z [ψθX(X)] ˆEX|Z [ψθX(X)] 2 ,

EX|Z [ψθX(X)] ˆEX|Z [ψθX(X)] 2 4 ˆRS1(H1) + 24

Therefore, by taking the square root of both sides, we can see

ˆEX|Z [f(X)] EX|Z [f(X)] P (Z) M

4 ˆRS1(H1) + 24

The generalization error for stage 2 is given in the following lemma.

Published as a conference paper at ICLR 2021

Lemma 2. Under Assumption 2, and given stage 1 data S1 = {(xi, zi)}m i=1, stage 2 data S2 = {( yi, zi)}n i=1, and the estimated structural function ˆfstruct(x) = (ˆu(n)) ψˆθX(x), then for any δ > 0, with at least probability 1 4δ, we have EY |Z [Y ] ˆEX|Z h ˆfstruct(X) i P (Z)

4 ˆRS2(H2) + 24M 2 r

4 ˆRS1(H1) + 24

where H1 is deﬁned in (13) and H2 is deﬁned as

H2 = {(y, z) Y Z 7 (y u g(z))2 R | g G, u Rd1, u M}. (14)

Proof. Since ˆEX|Z h ψˆθX

i G, by applying Proposition 1 to hypothesis space H2, we have

EY Z h (Y ˆEX|Z h ˆfstruct(X) i )2i

yi ˆEX|Z h ˆfstruct(X) i ( zi) 2 + 2 ˆRS2(H2) + 12M 2 r

with probability of 1 δ. Note that all functions h H2 are bounded in h 4M 2 since |Y | M, ψθX(X) 1, u M from Assumption 2. Similarly, since fstruct F from Assumption 2, we have

yi ˆEX|Z [fstruct(X)] ( zi) 2

EY Z h (Y ˆEX|Z [fstruct(X)])2i + 2 ˆRS2(H2) + 12M 2 r

with probability 1 δ. From the minimality of ˆfstruct, we have

yi ˆEX|Z h ˆfstruct(X) i ( zi) 2 1

i=1 ( yi ˆEX|Z [fstruct(X)] ( zi))2

Hence, we have

Y ˆEX|Z h ˆfstruct(X) i 2

EY Z h (Y ˆEX|Z [fstruct(X)])2i + 4 ˆRS2(H2) + 24M 2 r

with probability 1 2δ. Now we also have

Y ˆEX|Z h ˆfstruct i 2 = EY Z (Y EY |Z [Y ])2 + EZ h (ˆEX|Z h ˆfstruct i EY |Z [Y ])2i ,

EY Z h (Y ˆEX|Z [fstruct])2i = EY Z (Y EY |Z [Y ])2 + EZ h (ˆEX|Z [fstruct] EY |Z [Y ])2i .

Therefore, with probability 1 2δ,

EZ h (EY |Z [Y ] ˆEX|Z h ˆfstruct(X) i )2i

4 ˆRS2(H2) + 24M 2 r

2n + EZ h (ˆEX|Z [fstruct(X)] EY |Z [Y ])2i .

From Lemma 1 and (12), with probability at least 1 2δ,

EZ h (ˆEX|Z [fstruct(X)] EY |Z [Y ])2i 4M 2 ˆRS1(H1) + 24M 2 r

Published as a conference paper at ICLR 2021

By combining them, we can see that with probability 1 4δ, EY |Z [Y ] ˆEX|Z h ˆfstruct(X) i P (Z)

4 ˆRS2(H2) + 24M 2 r

2n + 4M 2 ˆRS1(H1) + 24M 2 r

4 ˆRS2(H2) + 24M 2 r

4 ˆRS1(H1) + 24

B.3 Consistency Proof of DFIV

The goal is to bound the deviation between ˆfstruct and fstruct. This discrepancy is measured by L2-norm with respect to P(X), deﬁned as

h(X) P (X) = Z h(X) 2d P(X) 1

However, we used the norm P (Z) in Lemmas 1 and 2. To bridge this gap, we state the necessary condition for identiﬁcation introduced in (Newey and Powell, 2003). Proposition 2. (Proposition 2.1 Newey and Powell, 2003) For all δ(x) with ﬁnite expectation, E [δ(X)|Z] = 0 a.e. on Z implies δ(x) = 0.

Proposition 2 is the minimum requirement for identiﬁcation and Assumption 1 is a suﬃcient condition for it. Given Proposition 2, we can consider a constant τ < deﬁned as

τ = sup f F,f =fstruct

fstruct(X) f(X) P (X) E [fstruct(X) f(X)|Z] P (Z) . (15)

Note that τ 1 from deﬁnition and τ = 1 if and only if X is measurable with respect to Z. This measures the ill-posedness of IV problem. A similar quantity is introduced in (Blundell et al., 2007). Given this quantity, we derive the convergence rate of DFIV as follows. Theorem 1. Let Assumption 2 hold. Given stage 1 data S1 = {(xi, zi)}m i=1, stage 2 data S2 = {( yi, zi)}m i=1 and τ deﬁned in (15), for any δ > 0, with at least probabilty of 1 6δ, we have

fstruct(X) ˆfstruct(X) P (X)

4 ˆRS1(H1) + 24

4 ˆRS2(H2) + 24M 2 r

where H1 and H2 are deﬁned in (13) and (14), respectively.

Proof. Using τ in (15), we have

fstruct(X) ˆfstruct(X) P (X) τ E h fstruct(X) ˆfstruct(X) Z i P (Z)

τ E h ˆfstruct(X)|Z i ˆEX|Z h ˆfstruct(X) i P (Z)

+ τ E [fstruct(X)|Z] ˆEX|Z h ˆfstruct(X) i P (Z)

= τ E h ˆfstruct(X)|Z i ˆEX|Z h ˆfstruct(X) i P (Z)

+ τ E [Y |Z] ˆEX|Z h ˆfstruct(X) i P (Z) ,

where the last inequality holds from (12). Using Lemmas 1 and 2, the result thus follows.

Published as a conference paper at ICLR 2021

Figure 7: Causal graph with observable confounder

From this result, we obtain directly the following corollary.

Corollary 1. Let Assumption 2 hold. If ˆRS1(H1) 0 and ˆRS2(H2) 0 in probability as datasize increases, ˆfstruct converges to fstruct in probability.

The proof of vanishing Rademacher complexities for particular Stage 1 and Stage 2 function classes is a topic for future work.

C 2SLS algorithm with observable confounders

In this appendix, we formulate the DFIV method when observable confounders are available. Here, we consider the causal graph given in Figure 7. In addition to treatment X X, outcome Y Y, and instrument Z Z, we have an observable confounder O O. The structural function fstruct we aim to learn is now X O Y, and the structural causal model is represented as

Y = fstruct(X, O) + ε, E [ε] = 0, E [ε|X] = 0.

For hidden confounders, we rely on Assumption 1. For observable confounders, we introduce a similar assumption. Assumption 3. The conditional distribution P(X|Z, O) is not constant in Z, O and E [ε|Z, O] = 0.

Following a similar reasoning as in Section 2, we can estimate the structural function ˆfstruct by minimizing the following loss: ˆfstruct = arg min f F L(f), L(f) = EY ZO (Y EX|Z,O [f(X, O)])2 + Ω(f).

One universal way to deal with the observable confounder is to augment both the treatment and instrumental variables. Let us introduce the new treatment X = (X, O) and instrument Z = (Z, O), then the loss L becomes

L(f) = EY Z

Y E X| Z h f( X) i 2 + Ω(f),

which is equivalent to the original loss L. This approach is adopted in KIV (Singh et al., 2019), and we used it here for Deep GMM method (Bennett et al., 2019) in the demand design experiment. However, this ignores the fact that we only have to consider the conditional expectation of X given Z = (Z, O). Hence, we introduce another approach which is to model f(X, O) = u (ψ(X) ξ(O)), where ψ(X) and ξ(O) are feature maps and denotes the tensor product deﬁned as a b = vec(ab ). It follows that EX|Z,O [f(X, O)] = u (EX|Z,O [ψ(X)] ξ(O)), which yields the following two-stage regression procedure.

In stage 1, we learn the matrix ˆV that ˆV = arg min V Rd1 d2 L1(V ) L1(V ) = EX,Z,O ψ(X) V φ(Z, O) 2 + λ1 V 2,

which estimates the conditional expectation EX|Z,O [ψ(X)]. Then, in stage 2, we learn ˆu using

ˆu = arg min u Rd1 L2(w) L2(w) = EY,Z,O h Y u ( ˆV φ(Z, O) ξ(O)) 2i + λ2 u 2.

Published as a conference paper at ICLR 2021

Again, both stages can be formulated as ridge regressions, and thus enjoy closed-form solutions. We can further extend this to learn deep feature maps. Let φθZ(Z, O), ψθX(X), ξθO(O) be the feature maps parameterized by θZ, θX, θO, respectively. Using notation similar to Section 3, the corresponding DFIV algorithm with observable confounders is shown in Algorithm 2. Note that in this algorithm, steps 3, 5, 6 are run until convergence, unlike for Algorithm 1.

Algorithm 2 Deep Feature Instrumental Variable with Observable Confounder

Input: Stage 1 data (xi, zi, oi), Stage 2 data ( yi, zi, oi). Initial values ˆθO, ˆθX, ˆθZ. Regularizing parameters (λ1, λ2) Output: Estimated structural function ˆfstruct(x).

1: while L(n) 2 has not converged do

2: while L(m) 1 has not converged do

3: Update ˆθZ by ˆθZ ˆθZ α θZ L(m) 1 ( ˆV (m)(ˆθO, ˆθX, θZ), θZ)|θZ=ˆθZ \\ Stage 1 4: end while 5: Update ˆθX by ˆθX ˆθX α θX L(n) 2 (ˆu(n)(ˆθO, θX, ˆθZ), ˆθO, θX)|θX=ˆθX \\ Stage 2

6: Update ˆθO by ˆθO ˆθO α θO L(n) 2 (ˆu(n)(θO, ˆθX, ˆθZ), θO, ˆθX)|θO=ˆθO \\ Stage 2 7: end while 8: Compute ˆu(n) from (9) 9: return ˆfstruct(x) = (ˆu(n)) ψˆθX(x)

D Application of 2SLS and DFIV to Off-Policy Policy Evaluation

Here we ﬁrst show that the optimal solution of (2) in the OPE problem is equivalent to that of mean squared Bellman error, and then describe how we apply 2SLS algorithm.

We interpret the Bellman equation in (10) as IV regression problem, where X = (s, a, s , a ), Z = (s, a) and Y = r and

fstruct(s, a, s , a ) = Qπ(s, a) γQπ(s , a ).

Let R(s, a) be the conditional expectation of reward given s, a deﬁned as

R(s, a) = Er|s,a [r] = Z r P(s |s, a)R(r|s, a, s )ds dr.

Then, we can prove that the solutions of (11) and MSBE are equivalent. Indeed, we have

arg min Qπ Es,a,r h r Qπ(s, a) + γEs ,a |a,s [Qπ(s , a )] 2i // (11)

= arg min Qπ Es,a,r h r R(s, a) + R(s, a) Qπ(s, a) + γEs ,a |a,s [Qπ(s , a )] 2i

= arg min Qπ Es,a h Var(r|s, a) // const wrt Qπ

+ 2Er|s,a (r R(s, a)) ( R(s, a) Qπ(s, a) + γEs ,a |a,s [Qπ(s , a )]) // = 0

+ R(s, a) Qπ(s, a) + γEs ,a |a,s [Qπ(s , a )] 2

= arg min Qπ Es,a h R(s, a) Qπ(s, a) + γEs ,a |a,s [Qπ(s , a )] 2i . (16)

In this context, we model Qπ(s, a) = u ψ(s, a) so that

fstruct(s, a, s , a ) = u (ψ(s, a) γψ(s , a )). (17)

It follows that

EX|z [f(X)] = u (ψ(s, a) γEs P ( |s,a),a π( |s ) [ψ(s , a )]).

Published as a conference paper at ICLR 2021

We will model Es P ( |s,a),a π( |s ) [ψ(s , a )] = V φ(s, a). In this case, given stage 1 data (si, ai, s i), stage 1 regression becomes

ˆV (m) = arg min V Rd1 d2 L(m) 1 , L(m) 1 = 1

i=1 ψ(si, ai) V φ(s i, a i) 2 + λ1 V 2,

where we sample a i π( |s i). However, given the speciﬁc form (17) of the structural function, stage 2 is slightly modiﬁed and requires minimizing the loss

ˆu(n) = arg min u Rd1 L(n) 2 , L(n) 2 = 1

i=1 ( ri u (ψ( si, ai) γ ˆV (m)φ( si, ai)))2 + λ2 u 2,

given stage 2 data ( si, ai, ri). We can further learn deep feature maps by parameterized feature maps ψ and φ as described in Section 3 to obtain the DFIV algorithm for OPE.

E Experiment Details and Additional Results

E.1 Details of Demand Design Experiments

Here, we introduce the details of demand design experiments. We follow the procedure in Singh et al. (2019). The observations are generated from the IV model,

Y = fstruct(P, T, S) + ε, E [ε|C, T, S] = 0,

where Y is sales, P is the treatment variable (price) instrumented by supply cost-shifter C. T, S are the observable confounder, interpretable as time of year and customer sentiment. The true structural function is

fstruct(P, T, S) = 100 + (10 + P)Sh(T) 2P,

h(t) = 2 (t 5)4

600 + exp( 4(t 5)2) + t

Data is sampled as

S Unif{1, . . . , 7} T Unif[0, 10] C N(0, 1) V N(0, 1)

ε N(ρV, 1 ρ2) P = 25 + (C + 3)h(T) + V

From observations of (Y, P, T, S, C), we estimate ˆfstruct by several methods. For each estimated ˆfstruct, we measure out-of-sample error as the mean square error of ˆf versus true fstruct applied to 2800 values of (p, t, s). Speciﬁcally, we consider 20 evenly spaced values of p [10, 25], 20 evenly spaced values of t [0, 10], and all 7 values s {1, . . . , 7}.

E.2 Effect Estimation in Demand Design Experiments

In this section, we report the result of causal eﬀect estimation based on demand design experiments. Speciﬁcally, we consider two eﬀects: one is average treatment eﬀect (ATE) and another is conditional average treatment eﬀect (CATE).

ATE Estimation ATE is a central target quantity in causal inference deﬁned as

ATE = E [Y |do(X = 1)] E [Y |do(X = 0)]

for binary treatment X {0, 1}. ATE thus requires estimating the counterfactual mean outcomes E [Y |do(X = 1)] , E [Y |do(X = 0)]. Here, we generalize this to continuous treatment

Published as a conference paper at ICLR 2021

10 12 14 16 18 20 22 24 Price p

Algorithm Ground Truth DFIV KIV Deep IV Deep GMM

Figure 8: Estimation of ATE.

0 2 4 6 8 10 Time of the Year t

Algorithm Ground Truth DFIV KIV Deep IV Deep GMM

Figure 9: Estimation of CATE conditioned on T given P = 25.

and consider estimating E [Y |do(X = x)]. This is also known as continuous treatment eﬀect or dose-response curve.

In the demand design problem, the ground truth is given by

E [Y |do(P = p)] = ET,S [fstruct(p, T, S)] = (G 2)p + 100 + 10G,

where G = ES [S] ET [h(T)]. The important point here is that this quantity must be monotonically decreasing with respect to price P, since we should observe drop of demand as we increase the ticket price.

We estimate ATE by averaging the estimated structural function ˆfstruct over S and T. The estimated ATE given by Deep GMM, KIV, Deep IV, DFIV are shown in Figure 8. From Figure 8, we can see that KIV and Deep GMM fail to recover the monotonic structure in ATE. Deep IV and DFIV are able to capture this structure, but DFIV estimation is consistently better than Deep IV.

CATE Estimation Although ATE captures the treatment eﬀect for the entire population, treatment eﬀects may be heterogeneous for diﬀerent sub-populations, which we might be interested in. In such a case, we can consider CATE deﬁned as

CATE( o) = E h Y do(X = 1), O = o i E h Y do(X = 0), O = o i

for binary treatment X {0, 1}. Here, O is a sub-vector of observable confounder O. Again, we generalize this idea and consider E h Y do(X = x), O = o i , which allows treatment X to be continuous. This quantity is also known as the heterogeneous treatment eﬀect.

In demand design experiment, we can consider the CATE conditioned on the time of the year T. The true CATE is deﬁned as

E [Y | do(P = p), T = t] = ES [fstruct(p, t, S)] = 100 + (10 + p)E [S] h(t) 2p,

which can be obtained by averaging the estimated structural function ˆfstruct over S. Figure 9 shows the prediction of CATE with respect to T where treatment P is ﬁxed to P = 25. Again, we can observe that KIV and Deep GMM fail to capture the shape of the ground truth curve. Deep IV performs signiﬁcantly better but is less accurate than DFIV.

E.3 Data Generation Process in d Sprites Experiments

Here, we describe the data generation process for the d Sprites dataset experiment. This is an image dataset described via ﬁve latent parameters (shape, scale, rotation, pos X and pos Y). The images are 64 64 = 4096-dimensional. In this experiment, we ﬁxed shape parameter to heart, i.e. we only used the heart-shaped images. The other latent parameters take values of scale [0.5, 1], rotation [0, 2π], pos X [0, 1], pos Y [0, 1].

From this dataset, we generate the treatment variable X and outcome Y as follows.

Published as a conference paper at ICLR 2021

0 20 40 60 80 100 Epoch

Out-of-Sample MSE

Deep GMM DFIV

Figure 10: Out-of-Sample MSE in d Sprite experiment during the training

1. Uniformly samples latent parameters (scale, rotation, pos X, pos Y).

2. Generate treatment variable X as

X = Fig(scale, rotation, pos X, pos Y) + η.

3. Generate outcome variable Y as

Y = AX 2 2 5000 1000 + 32(pos Y 0.5) + ε.

Here, function Fig returns the corresponding image to the latent parameters, and η, ε are noise variables generated from η N(0.0, 0.1I) and ε N(0.0, 0.5). Each element of the matrix A R10 4096 is generated from Unif(0.0, 1.0) and ﬁxed throughout the experiment. From the data generation process, we can see that X and Y are confounded by pos Y. We use the instrumental variable Z = (scale, rotation, pos X) R3, and ﬁgures with random noise as treatment variable X. The variable pos Y is not revealed to the model, and there is no observable confounder. The structural function for this setting is

fstruct(X) = AX 2 2 5000 1000 .

We use 588 test points for measuring out-of-sample error, which is generated from the grid points of latent variables. The grids consist of 7 evenly spaced values for pos X, pos Y, 3 evenly spaced values for scale, and 4 evenly spaced values for orientation.

E.4 Instability of Deep GMM in d Sprites Experiments

In the d Sprite experiments, we observed that the training procedure of Deep GMM can be unstable. Figure 10 displays the MSE for the models learned in the ﬁrst 100 epochs. Here, we can see that Deep GMM performs poorly in the early stage of the learning. Furthermore, even after it appears to have converged, we observe a sudden increase of MSE around 80th epoch, which makes diﬃcult to determine when to stop. We conjecture that this is due to the instability of the smooth zero-sum game solved by Deep GMM. By contrast, DFIV converges quickly and performs consistently better than Deep GMM on this task.

E.5 MNIST Experiments

Here, we report the result of MNIST experiments proposed by Bennett et al. (2019), who consider the following datasets:

Z Unif([ 3, 3]2) e N(0, 1), γ, δ N(0, 0.1) X = Z1 + e + γ Y = |X| + e + δ

Published as a conference paper at ICLR 2021

MNISTx MNISTz MNISTxz Deep GMM .15 .02 .07 .02 .14 .02 DFIV .18 .01 .07 .001 .10 .003 Table 1: Out-of-Sample MSE in MNIST experiment of Bennett et al. (2019)

Catch Mountain Car Cartpole D 50 3 6 A 3 3 3 Table 2: Dimensions of BSuite tasks

Here, the structural function we aim to learn is fstruct(X) = |X|. Additionally, we map Z, X, or both X and Z to MNIST images to see whether the model can handle the highdimensional treatment and instrumental variables. Let the output of original IV problem above to be Xlow, Zlow and π(x) = round(min(max(1.5x + 5, 0), 9)) be a transformation function that maps inputs to an integer between 0 and 9, and let Random Image(d) be a function that selects a random MNIST image from the digit class d. The images are 28 28 = 784-dimensional. We consider the three following scenarios.

MNISTx: X Xlow, Z Random Image(π(Zlow))

MNISTz: X Random Image(π(Xlow)), Z Zlow MNISTxz: X Random Image(π(Xlow)), Z Random Image(π(Zlow))

We refer the reader to Bennett et al. (2019) for a detailed description. We applied DFIV using the same architecture as Deep GMM. In Table 1, we present the mean and standard error of out-of-sample MSE for DFIV and report the results obtained for Deep GMM in Bennett et al. (2019). From Table 1, we can see that DFIV performs essentially like Deep GMM.

E.6 OPE Experiment Details

E.6.1 BSuite tasks

We provide a brief description below of the three behavior suite (BSuite) reinforcement learning tasks in Osband et al. (2019):

1. Catch: A 10x5 Tetris-grid with single block falling per column. The agent can move left/right in the bottom row to catch the block. Illustrated in Figure 11a.

2. Mountain Car: The agent drives an underpowered car up a hill (Moore, 1990). Illustrated in Figure 11b.

3. Cartpole: The agent can move a cart left/right on a plane to keep a balanced pole upright (Barto et al., 1983). Illustrated in Figure 11c.

(b) Mountain Car

(c) Cartpole Figure 11: Three BSuite tasks. Figures are from Osband et al. (2019).

All tasks have a real-valued state space S RD and a discrete action space A = {0, 1, 2, . . . , A 1}. The state dimension and number of actions are provided in Table 2.

Published as a conference paper at ICLR 2021

E.6.2 Fitted Q Evaluation

Fitted Q Evaluation (FQE) (Le et al., 2019) is a simple variant of the Fitted Q Iteration (FQI) algorithm (Ernst et al., 2005). Instead of learning the Q function of an optimal policy as FQI, FQE estimates the Q function of a ﬁxed policy π. It is an iterative algorithm with randomly initialized the Q function parameters θ0. At iteration k 1 with the current estimate of Q function Qπ(s, a|θk 1), it updates parameters θk by solving the following regression problem using a least squares approach:

r + γQπ(s , a |θk 1) =Qπ(s, a|θk) (18)

+ γ Qπ(s , a |θk 1) Es P ( |s,a),a π( |s ) [Qπ(s , a |θk 1)]

+ r Er R( |s,a,s ) [r] ,

where the regression function to estimate is Qπ(s, a|θk), the observed outcome is the term on the LHS of (18) and the residual is the sum of terms in the second and third lines of (18). Comparing the equation above with (10), FQE reformulates the regression problem and moves the confounded part in the treatment of (10), that is γQπ(s , a ), to the outcome. The regression problem at each iteration is therefore not confounded any more. If FQE converges, it ﬁnds a solution of the following equation

Qπ(s, a|θ) = P(Er|s,a [r] + γEs ,a |s,a [Qπ(s , a |θ)]) , (19)

where P( ) is the L2 projection operator that maps a function of (s, a) onto the (parameterized) Q function space.

F Failure of Joint Optimization

One might want to jointly minimize the loss L(n) 2 , which is the empirical approximation of L. However, this fails to learn the true structural function fstruct, as shown in this appendix.

Figure 12 shows the learning curve obtained when we jointly minimize θX and θZ with respect to L(n) 2 , where Ltest is the empirical approximation of

Ltest = EX h fstruct(X) (ˆu(n)) ψθX(X) 2i . (20)

We observe in Figure 12 that the decrease of stage 2 loss does not improve the performance of the learned structural function. This is because the model focuses on learning the relationship between instrument Z and outcome Y , while ignoring treatment X. We can see this from the fact that stage 1 loss becomes large and unstable. This is against the goal of IV regression, which is to learn a causal relationship between X and Y .

On the other hand, Figure 13 shows the learning curve obtained using DFIV. Now, we can see that stage 2 loss matches Ltest. Also, we can conﬁrm that stage 1 loss stays small and stable.

Published as a conference paper at ICLR 2021

0 200 400 600 800 1000 Epochs

Stage 1 loss Stage 2 loss

Figure 12: Learning curve of joint minimization

0 200 400 600 800 1000 Epochs

Stage 1 loss Stage 2 loss

Figure 13: Learning curve of DFIV

Table 3: Network structures of Deep IV for demand design dataset. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions. For mixture Gaussian output, we report the number of components. Dropout rate is given in the main text.

Instrument Net Layer Conﬁguration 1 Input(C, T, S) 2 FC(3, 128), Re LU 3 Dropout 4 FC(128, 64), Re LU 5 Dropout 6 FC(64, 32), Re LU 7 Dropout 8 Mixture Gaussian(10)

Treatment Net Layer Conﬁguration 1 Input(P, T, S) 2 FC(3, 128), Re LU 3 Dropout 4 FC(128, 64), Re LU 5 Dropout 6 FC(64, 32), Re LU 7 Dropout 8 FC(32, 1)

G Network Structures and Hyper-Parameters

Here, we describe the network architecture and hyper-parameters of all experiments. Unless otherwise speciﬁed, all neural network-based algorithms are optimized using Adam with learning rate = 0.001, β1 = 0.9, β2 = 0.999 and ε = 10 8.

Demand Design For Deep IV, we used the original structure proposed in Hartford et al. (2017), which is described in Table 3. We follow the default dropout rate in Hartford et al. (2017), which depends on the data size. For DFIV, we used the structure described in Table 4. The regularizer λ1, λ2 are both set to 0.1 as a result of the tuning procedure described in Appendix A. For KIV, we used the Gaussian kernel where the bandwidth is determined by the median trick described by Singh et al. (2019). We used random Fourier feature trick (Rahimi and Recht, 2008) with 100-dimensions. For Deep GMM, we used the same structure as Deep IV but no dropout is applied and the last layer of the Instrument Net is changed to a fully-connected layer which maps 32 dimensions to 1 dimension.

Demand Design with MNIST The feature extractor for MNIST image data is given in Table 5, which is used for both stage 1 and 2. For Deep IV, we used the original structure proposed in Hartford et al. (2017), which is described in Table 6. We follow the default dropout rate in Hartford et al. (2017), which depends on the data size. For DFIV, we used the structure described in Table 7. The regularizer λ1, λ2 are both set to 0.1 as a result of the tuning procedure described in Appendix A. For KIV, we used the Gaussian kernel where the bandwidth is determined by the median trick and also used random Fourier features with 100-dimensions. For Deep GMM, we used the same structure as Deep IV but no dropout

Published as a conference paper at ICLR 2021

Table 4: Network structures of DFIV for demand design datasets. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions.

Instrument Feature φθZ Layer Conﬁguration 1 Input(C, T, S) 2 FC(3, 128), Re LU 3 FC(128, 64), Re LU 4 FC(64, 32), Re LU

Treatment Feature ψθX Layer Conﬁguration 1 Input(P) 2 FC(1, 16), Re LU 3 FC(16, 1)

Observable Feature ξθO Layer Conﬁguration 1 Input(T, S) 2 FC(2, 128), Re LU 4 FC(128, 64), Re LU 6 FC(64, 32), BN, Re LU

Table 5: Network structures of feature extractor used in demand design experiment with MNIST. For each convolution layer, we list the input dimension, output dimension, kernel size, stride, and padding. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions. For max-pool, we list the size of the kernel. Dropout rate here is set to 0.1. SN denotes Spectral Normalization (Miyato et al., 2018).

Image Feature Layer Conﬁguration 1 Input(S) 2 Conv2D (1, 64, 3, 1, 1), Re LU, SN 3 Conv2D (64, 64, 3, 1, 1), Re LU, SN 4 Max Pool(2, 2) 5 Dropout 6 FC(9216, 64), Re LU 7 Dropout 8 FC(64, 32), Re LU

is applied and the last layer of instrument net is changed to a fully connected layer which maps 32 dimensions to 1 dimension.

d Sprite experiment For Deep GMM, we used the structure described in Table 8. For DFIV, we used the structure described in Table 9. The regularizer λ1, λ2 are both set to 0.01 as a result of the tuning procedure described in Appendix A. For KIV, we used the Gaussian kernel where the bandwidth is determined by the median trick. We used random Fourier feature trick (Rahimi and Recht, 2008) with 100 dimensions.

OPE experiment For Deep IV, we used the structure described in Table 12. For DFIV, we used the structure described in Table 10. The regularizer λ1, λ2 are both set to 10 5 as a result of the tuning procedure described in Appendix A. For KIV, we used the Gaussian kernel where the bandwidth is determined by the median trick. We used random Fourier features (Rahimi and Recht, 2008) with 100 dimensions. For Deep GMM, we use the structure described in Table 11.

Published as a conference paper at ICLR 2021

Table 6: Network structures of Deep IV in demand design with MNIST data. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions. For mixure Gaussian output, we report the number of components. Image Feature denotes the module given in Table 5. Dropout rate is described in the main text.

Instrument Net Layer Conﬁguration 1 Input(C, T, Image Feature(S)) 2 FC(66, 32), Re LU 3 Dropout 4 Mixture Gaussian(10)

Treatment Net Layer Conﬁguration 1 Input(P, T, Image Feature(S)) 2 FC(66, 32), Re LU 3 Dropout 4 FC(32, 1), Re LU

Table 7: Network structures of DFIV in demand design with MNIST. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions. Image Feature denotes the module given in Table 5.

Instrumental Feature φθZ Layer Conﬁguration 1 Input(C, T, Image Feature(S)) 2 FC(66, 32), BN, Re LU

Treatment Feature ψθX Layer Conﬁguration 1 Input(P) 2 FC(1, 16), Re LU 3 FC(16, 1)

Obsevable Feature Net ξθO Layer Conﬁguration 1 Input(T, Image Feature(S)) 2 FC(65, 32), BN, Re LU

Table 8: Network structures of Deep GMM in d Sprite experiment. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions. SN denotes Spectral Normalization (Miyato et al., 2018).

Instrument Net Layer Conﬁguration 1 Input(Z) 2 FC(3, 256), SN, Re LU 3 FC(256, 128), SN, Re LU, BN 4 FC(128, 128), SN, Re LU, BN 5 FC(128, 32), SN, BN, Re LU 6 FC(32, 1)

Treatment Net Layer Conﬁguration 1 Input(X) 2 FC(4096, 1024), SN, Re LU 3 FC(1024, 512), SN, Re LU, BN 4 FC(512, 128), SN, Re LU 5 FC(128, 32), SN, BN, Tanh 6 FC(32,1)

Table 9: Network structures of DFIV in d Sprite experiment. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions. SN denotes Spectral Normalization (Miyato et al., 2018).

Instrument Feature φθZ Layer Conﬁguration 1 Input(Z) 2 FC(3, 256), SN, Re LU 3 FC(256, 128), SN, Re LU, BN 4 FC(128, 128), SN, Re LU, BN 5 FC(128, 32), SN, BN, Re LU

Treatment Feature ψθX Layer Conﬁguration 1 Input(X) 2 FC(4096, 1024), SN, Re LU 3 FC(1024, 512), SN, Re LU, BN 4 FC(512, 128), SN, Re LU 5 FC(128, 32), SN, BN, Tanh

Published as a conference paper at ICLR 2021

Table 10: Network structures of DFIV in OPE experiment. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions.

Instrument Feature φθZ Layer Conﬁguration 1 Input(s, one-hot(a)) 2 FC(*, 150), Re LU 3 FC(150, 100), Re LU 4 FC(100, 50), Re LU

Treatment Feature ψθX Layer Conﬁguration 1 Input(s, one-hot(a)) 2 FC(*, 50), Re LU 3 FC(50, 50), Re LU

Table 11: Network structures of Deep GMM in OPE experiment. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions.

Instrument Net Layer Conﬁguration 1 Input(s, one-hot(a)) 2 FC(*, 150), Re LU 3 FC(150, 100), Re LU 4 FC(100, 50), Re LU 5 FC(50, 1)

Treatment Net Layer Conﬁguration 1 Input(s, one-hot(a)) 2 FC(*, 50), Re LU 3 FC(50, 50), Re LU 4 FC(50, 1), Re LU

Table 12: Network structures of Deep IV in OPE experiment. For the input layer, we provide the input variable. For the fully-connected layers (FC), we provide the input and output dimensions. The Mixuture Gaussian layer maps the input linearly to required parameter dimensions for a mixture of Gaussian distribution with diagonal covariance matrices. The Bernoulli layer maps the input linearly to a single dimension to represent the logit of a Bernoulli distribution to predict if the next state is a terminating state.

Instrument Net Layer Conﬁguration 1 Input(s, one-hot(a)) 2 FC(*, 50), Re LU 3 FC(50, 50), Re LU 4 Mixture Gaussian(3) and Bernoulli

Treatment Net Layer Conﬁguration 1 Input(s, one-hot(a)) 2 FC(*, 50), Re LU 3 FC(50, 50), Re LU 4 FC(50, 1), Re LU