# hierarchical_dirichlet_scaling_process__972df118.pdf

Hierarchical Dirichlet Scaling Process

Dongwoo Kim DW.KIM@KAIST.AC.KR KAIST, Daejeon, Korea

Alice Oh ALICE.OH@KAIST.EDU KAIST, Daejeon, Korea

We present the hierarchical Dirichlet scaling process (HDSP), a Bayesian nonparametric mixed membership model for multi-labeled data. We construct the HDSP based on the gamma representation of the hierarchical Dirichlet process (HDP) which allows scaling the mixture components. With such construction, HDSP allocates a latent location to each label and mixture component in a space, and uses the distance between them to guide membership probabilities. We develop a variational Bayes algorithm for the approximate posterior inference of the HDSP. Through experiments on synthetic datasets as well as datasets of newswire, medical journal articles, and Wikipedia, we show that the HDSP results in better predictive performance than HDP, labeled LDA and partially labeled LDA.

1. Introduction

The Hierarchical Dirichlet process (HDP) is an important nonparametric Bayesian prior for mixed membership models, and the HDP topic model is useful for a wide variety of tasks involving unstructured text (Teh et al., 2006). To extend the HDP topic model, there has been active research in dependent random probability measures as priors for modeling the underlying association between the latent semantic structure and explanatory variables, such as time stamps and spatial coordinates (Ahmed & Xing, 2010; Ren et al., 2011).

A large body of this research is rooted in the dependent Dirichlet process (DP) (Mac Eachern, 1999) where the probabilistic random measure is defined as a function of some covariate. Most dependent DP approaches rely on the generalization of Sethuraman s stick breaking represen-

Proceedings of the 31 st International Conference on Machine Learning, Beijing, China, 2014. JMLR: W&CP volume 32. Copyright 2014 by the author(s).

4 3 2 1 0 1 2 3

ARTS,CULTURE,ENTERTAINMENT

film,movie,music

beat,play,champ

soccer,play,match

clinton,president,dole

tobacco,smoke,cigarette

bank,rate,percent

disease,cow,infect net,loss,share

profit,million,half

Figure 1. Locations of observed labels (capital letters in red) and latent topics (small letters in blue) inferred by HDSP from the Reuters corpus. HDSP uses the distances between labels and topics to scale the topic proportions such that the topics closer to the observed labels in a document are given higher probabilities.

tation of DP (Sethuraman, 1991), incorporating the time difference between two or more data points or the spatial difference among observed data into the predictor dependent stick breaking process (Duan et al., 2007; Dunson & Park, 2008). Some of these priors can be integrated into the hierarchical construction of DP (Srebro & Roweis, 2005), resulting in topic models where temporallyor spatiallyproximate data are more likely to be clustered.

Many datasets, however, come with labels, or categorical side information, which cannot be modeled with these existing dependent DP approaches. Labels, like temporal and spatial information, are correlated with the latent semantics of the documents, but they cannot be used to directly define the distance between two documents. This is because labels are categorical, so there is no simple way to measure distances between labels. Moreover, labels and documents do not have a one-to-one correspondence, as there may be zero, one, or more labels per document.

We develop the hierarchical Dirichlet scaling process

Hierarchical Dirichlet Scaling Process

(HDPS) which models the latent locations of topics and labels in a space and uses the distances between them to guide the topic proportions. Figure 1 visualizes how the HDSP discovers the latent locations of the topics and the labels from the Reuters articles with news categories as labels. In this example, an article under the news category ECONOMICS would be endowed high probabilities for topics <net, loss, share> and <bank, rate, percent>, and low probabilities for topics <beat, play, champ> and <film, movie, music>.

In the next section, we describe the gamma process construction of the HDP and how the scale parameter is used to develop the HDSP. In section 3, we derive a variational inference for the latent variables by directly placing a prior over the distances between the latent locations. In section 4, we verify our approach on a synthetic dataset and demonstrate the improved predictive power of our model on multiand partially-labeled corpora.

Related Work Previously proposed topic models for labeled documents take an approach quite distinct from the dependent DP literature. Labeled LDA (L-LDA) allocates one dimension of the topic simplex per label and generates words from only the topics that correspond to the labels in each document (Ramage et al., 2009). An extension of this model, partially labeled LDA (PLDA), adds more flexibility by allocating a pre-defined number of topics per label and including a background label to handle documents with no labels (Ramage et al., 2011). The Dirichlet process with mixed random measures (DP-MRM) is a nonparametric topic model which generates an unbounded number of topics per label but still excludes topics from labels that are not observed in the document (Kim et al., 2012).

2. Hierarchical Dirichlet Scaling Process

In this section, we describe the hierarchical Dirichlet scaling process (HDSP) for multi-labeled data. First we review the HDP and the gamma process construction of the second level DP. We then present the HDSP where the second level DP incorporates the latent locations for the mixture components and the labels.

2.1. The gamma process construction of HDP

In the HDP1, there are two levels of the DP where the measure drawn from the upper level DP is the base distribution of the lower level DP. The hierarchical representation of the process is

G0 DP(αH), Gm DP(βG0), (1)

1In this paper, we limit our discussions of the HDP to the two level construction of the DP and refer to it simply as the HDP.

where H is a base distribution, α, and β are concentration parameters, and index m represents multiple draws from the second level DP. For the mixed membership model, xmn, observation n in group m, can be drawn from

θmn Gm, xmn f(θmn), (2)

where f( ) is a data distribution parameterized by θ. In the context of topic models, the base distribution H is usually a Dirichlet distribution over the vocabulary, so the atoms of the first level random measure G0 are an infinite set of topics drawn from H. The second level random measure Gm is distributed based on the first level random measure G0, so the second level shares the same set of topics, the atoms of the first level random measure.

The constructive definition of the DP can be represented as a stick breaking process (Sethuraman, 1991), and in the HDP inference algorithm based on stick breaking, the first level DP is given by the following conditional distributions:

Vk Beta(1, α) pk = Vk

k=1 pkδφk, (3)

where Vk defines a corpus level topic distribution for topic φk. The second level random measures are conditionally distributed on the first level discrete random measure G0:

πml Beta(1, β) pml = πml

j=1 (1 πmj)

θml G0 Gm =

l=1 pmlδθml, (4)

where the second level atom θml corresponds to one of the first level atoms φk. This stick breaking construction is the most widely used method for the hierarchical construction (Wang et al., 2011; Teh et al., 2006).

An alternative construction of the HDP is based on the normalized gamma process (Paisley et al., 2012). While the first level construction remains the same, the gamma process changes the second level construction from Eq. 4 to

πmk Gamma(βpk, 1)

πmk P j=1 πmj δφk, (5)

where Gamma(x; a, b) = bax(a 1)e bx/Γ(a). Unlike the stick breaking construction, the atom of the πmk of the gamma process is the same as the atom of the kth stick of the first level. Therefore, during inference, the model does

Hierarchical Dirichlet Scaling Process

not need to keep track of which second level atoms correspond to which first level atoms. Furthermore, by placing a proper random variable on the rate parameter of the gamma distribution, the model can infer the correlations among the topics (Paisley et al., 2012) through the Gaussian process (Rasmussen & Williams, 2005).

2.2. Hierarchical Dirichlet Scaling Process

In the hierarchical Dirichlet scaling process (HDSP), we start with the gamma process construction of the HDP with a proper prior for the rate parameter to guide the topic proportions based on the labels of the document. In the model, each topic and label has a latent location, and the topic proportion of a document is proportional to the distances between the topics and the labels. With the assumption that the locations of topics and labels are drawn from a distribution over the space, the first level DP of HDSP is drawn from the product of two base distributions,

G0 DP(αH L), G0 =

k=1 pkδ{φk,lk} (6)

where H is a distribution over the topic parameter θk, L is a distribution over the latent locations of topic k and label j, and pk is a stick length for topic k, pk = Vk Qk <k k =1(1 Vk ). There are J observable labels, and for each observable label j, a latent location lj is drawn from the distribution L. Through the first level DP, the model draws an infinite number of topics φk as well as their corresponding locations lk.

In the second level DP, the gamma process is used to incorporate the distances between the location of topics and observed labels. Let rmj be an indicator variable, if the label j is observed in document m then rmj is 1 otherwise 0. First, as in the HDP, draw a random measure G m DP(βG0) for each document. Second, scale the topics based on the product of the inverse distances between topics and the observed labels of the document

Gm({φk, lk}) G m({φk, lk})

j=1 d(lj, lk) rmj, (7)

where d(lj, lk) is a distance measure of the location of label lj and the location of topic lk. Through this process, topics that are closely located to the observed labels would have larger proportions in the final random measure Gm.

The constructive definition of HDSP is similar to the HDP, but the difference comes from the location variables and the distance terms. The first level stick breaking for HDSP

Vk Beta(1, α) pk = Vk

φk H, lk L G0 =

k=1 pkδ{φk,lk}. (8)

Also, for each observable label, lj is drawn i.i.d from L.

The second level gamma process for HDSP is

πmk Gamma(βpk,

j=1 d(lj, lk)rmj)

πmk P j=1 πmj δφk. (9)

The distance terms are directly incorporated into the second parameter of the gamma distribution since the scaled gamma random variable y = kx Gamma(a, 1) is equal to y Gamma(a, k 1). For the mixed membership model, nth observation in mth group is drawn as follows:

φk Gm, xmn f(φk). (10)

For topic modeling, Gm and xmn correspond to document m and word n in document m, respectively.

3. Variational Inference for HDSP

The posterior inference for Bayesian nonparametric models is important because it is intractable to compute the posterior over an infinite dimensional space. Approximation algorithms, such as marginalized MCMC (Escobar & West, 1995; Teh et al., 2006) and variational inference (Blei & Jordan, 2006; Teh et al., 2008), have been developed for the Bayesian nonparametric mixture models. We develop a mean field variational inference (Jordan et al., 1999; Wainwright & Jordan, 2008) algorithm for approximate posterior inference of the HDSP topic model. The objective of variational inference is to minimize the KL divergence between a distribution over the hidden variables and the true posterior, which is equivalent to maximizing the lower bound of the marginal log likelihood of observed data.

For simple and efficient inference, we devise a simple model from the same perspective. Since we are interested in the distances between topics and labels, we directly place a prior over the distance between a topic and label d(lj, lk). Let wjk be the inverse distance between the latent location of label j and topic k, i.e. wjk = d(lj, lk) 1, we approximate wjk by placing an inverse-Gamma prior over wjk:

wjk inv Gamma(aw, bw) (11)

Hierarchical Dirichlet Scaling Process

: concentration parameters

: base distribution for topics and locations

: location of topics and labels

: k th topic

: first and second level topic proportion

: topic of n th word in m th document

: j th label of m th document {0, 1}

: observed n th word in m th document

lj φk H L lk

Figure 2. Graphical model of the hierarchical Dirichlet scaling process.

As a consequence, the second level gamma process can be rewritten from Equation 9 to

πmk Gamma(βpk,

j=1 w rmj jk ). (12)

Excluding the locations from the model, it becomes unnecessary to place a proper distribution L over the space, in order to place a proper distance measure and to infer the latent locations of labels and topics. Hence, we can reduce the model complexity and derive a relatively simple and efficient inference algorithm, excluding the need for more complex metric learning problems (Xing et al., 2002) or the kernel based dependent DP construction (Dunson & Park, 2008; Ren et al., 2011). Now, we derive a variational Bayes inference using this approximation approach.

We use a fully factorized variational distribution and perform a mean-field variational inference. There are five latent variables of interest: the corpus level stick proportion Vk, the document level stick proportion πmk, the inverse distance between topic and label wjk, the topic assignment for each word zmn, and the word topic distribution φk. Thus the variational distribution q(z, π, V, w, φ) can be factorized into

q(z, π, V, w, φ) =

n=1 q(zmn)q(πmk)q(Vk)q(φk)q(wjk), (13)

where the variational distributions are

q(zmn) = Multinomial(zmn|γmn)

q(πmk) = Gamma(πmk|aπ mk, bπ mk)

q(wjk) = Inv Gamma(wjk|aw jk, bw jk)

q(φk) = Dirichlet(φk|ηk)

q(Vk) = δVk.

For the corpus level stick proportion Vk, we use the delta function as a variational distribution for simplicity and tractability in inference steps (Liang et al., 2007). Infinite dimensions over the posterior is a key problem in Bayesian nonparametric models and requires an approximation method. In variational treatment, we truncate the unbounded dimensionality to T by letting VT = 1. Thus the model still keeps the infinite dimensionality while allowing approximation to be carried out under the bounded variational distributions.

Using standard variational theory, we derive the lower bound of the marginal log likelihood of the observed data D = (xm, rm)M m=1,

log p(D|α, β, aw, bw, η)

Eq[log p(D, z, π, V, w, φ)] + H(q) = L(q), (14)

where H(q) is the entropy for the variational distribution. By taking the derivative of this lower bound, we derive the following coordinate ascent algorithm.

Document-level Updates: At the document level, we update the variational distribution for the topic assignment zmn and the document level stick proportion πmk. The update for q(zmn|γmn) is

γmnk exp (Eq[ln ηk,xmn] + Eq[ln πmk]) . (15)

Updating q(πmk|aπ mk, bπ mk) requires computing the expectation term E[ln PT k=1 πmk]. Following (Blei & Lafferty, 2007), we approximate the lower bound of the expectation by using the first-order Taylor expansion,

k=1 πmk] ln ξm PT k=1 Eq[πmk] ξm

where the update for ξ = PK k=1 Eq[πmk]. Then, the update

Hierarchical Dirichlet Scaling Process

aπ mk = βpk +

j Eq[w rmj jk ] + Nm

Note again rmj is equal to 1 when jth label is observed in mth document, otherwise 0.

Corpus-level Updates: At the corpus level, we update the variational distribution for the inverse distance wjk, corpus level stick length Vk and word topic distribution ηki.

The optimal form of a variational distribution can be obtained by exponentiating the variational lower bound with all expectations except the parameter of interest (Bishop & Nasrabadi, 2006). For wjk, we can derive the optimal form of variational distribution as follows

q(wjk) Inv Gamma(a , b ) (18)

a = Eq[βpk] X

j /j Eq[w 1 j k]Eq[πm k] + bw,

where m = {m : rmj = 1} and j /j = {j : rmj = 1, j = j}. See the appendix for the complete derivation. There is no closed form update for Vk, instead we use the steepest ascent algorithm to jointly optimize Vk. The gradient of Vk is

m,j rmj Eq[ln πmk] Eq[πmk] + ψ(βpk)}

βpk 1 Vk { X

m,j rmj Eq[ln πmk ] Eq[πmk ] + ψ(βpk )},

where ψ( ) is a digamma function. Finally, the update for the word topic distribution q(φk|ηk) is

ηki = η + X

m,n γmnk1(xmn = i), (20)

where i is a word index, and 1 is an indicator function (Blei et al., 2003).

0 2 4 6 8 words

(a) Synthetic Topics

0 2 4 6 8 words

(b) HDSP Topics

0 2 4 6 8 words

(c) HDP Topics

10 5 0 5 10 15 20 10

topic label

(d) Latent Locations

0 1 2 3 labels

(e) Syn Distances

0 1 2 3 labels

(f) HDSP Distances

Figure 3. Experiments with synthetic data. (a) is the synthetic topic distribution of 5 topics over 10 terms. (b) and (c) are topic distributions inferred by the HDSP and the HDP. Both models recover the original topics. (d) shows the original locations of topics and labels. (e) shows the original distances between the locations of topics and labels. (f) shows that the HDSP recovers the distances. Note that the HDP cannot compute the distances.

The expectations under the variational distribution q are

Eq[πmk] = aπ mk/bπ mk Eq[ln πmk] = ψ(aπ mk) ln bπ mk Eq[wjk] = bw jk/(aw jk 1)

Eq[w 1 jk ] = aw jk/bw jk Eq[ln wjk] = ln bw jk ψ(aw jk)

Eq[ln φki] = ψ(ηki) ψ( X

4. Experiments

We train the HDSP topic model with synthetic and real data to verify the model and show the advantages over existing models.

4.1. Synthetic data

There is no naturally-occurring dataset with observable locations of topics and labels, so we synthesize data based on the model assumptions to verify our model and the approximate inference. First, we check the difference between the original topics and the inferred topics via simple visualization. Then, we focus on the inferred locations and the original locations. For all experiments with synthetic data, we set the truncation level T at twice the number of topics. We terminate variational inference when the fractional change of the lower bound falls below 10 3, and we average all results over 10 individual runs with different initializations.

Hierarchical Dirichlet Scaling Process

0 5 10 15 20 x

Mean absolute error

0 5 10 15 20 x

Spearman s rho

Figure 4. Spearman s correlation coefficient and mean absolute error of the synthetic data with various volume of space (x3). As the volume of space for locations increases, the mean absolute error also increases (left). However, the model preserves the relative distances between topics and labels, shown by the high and stabilized correlation between the original ordering and the recovered ordering of label-topic pairs in terms of the distance between the two (right). This is a key characteristic of the HDSP model which scales the mixture components according to the inverse of the distance.

With the first experiment, we show that HDSP correctly recovers the underlying topics and distances between topics and labels. For the dataset, we generate 500 documents using the following steps. We define five topics over ten terms shown in Figure 3(a) and locations of five topics and four labels shown in Figure 3(d). For each document, we randomly draw Nm from the Poisson distribution and rmj from the Bernoulli distribution. The average length of a document is 20, and the average number of labels per document is 2. We generate topic proportions of corpus and documents by using Equations 8 and 9. For each word in a document, we draw the topic and the word by using Equation 10. We set both α and β to 1.

Figure 3 shows the results of the HDP and the HDSP on the synthetic dataset. Figure 3(b) and Figure 3(c) are the heat maps of topics inferred from each model. We match the inferred topics to the original topics using KL divergence between the two topic distributions. There are no significant differences between the inferred topics of HDSP and HDP. In addition to the topics, HDSP infers the distances between topics and labels, which are shown in Figure 3(f).

With the second experiment, we show that the distances that HDSP infers preserve the relative distances between labels and topics in the dataset. Recall that HDSP does not directly infer the latent locations but instead infers the distances between labels and topics, which are then used to scale the topic proportions.

For this experiment, we generate 1,000 documents with randomly drawn 10 topics from Dirichlet(0.1) with the vocabulary size of 20. The locations of topics and labels are drawn from Uniform(0, x) varying the x value from 1 to 20 for each experiment. We set the dimensionality of locations to 3, thus the volume of space is x3. We compute the

Table 1. Datasets used for the experiments in 4.2. As the last two columns show, we experiment on datasets with a varied number of unique labels, as well as the average number of labels per document, including the Wikipedia corpus with many documents that are unlabeled.

docs vocab labels labels/doc Wikipedia 25,547 7,702 1093 0.6 RCV 23,149 9,911 117 3.2 OHSUMED 7,505 7,056 52 5.2

mean absolute error (MAE) and the spearman s rank correlation coefficient (rho) between the original distances and the inferred distances. The spearman s rho is designed to measure the ranking correlation of two lists.

Figure 4 shows the results. The MAE increases as the volume of the space increases. However, spearman s rho stabilizes, indicating that the relative distances are preserved even when the MAE increases. Since there are an infinite number of configurations of distances that generate the same expectation E[p(πm|βp, wj)] given πm and βp, preserving the relative distances verifies our model s capability of capturing the underlying structure of topics and labels.

4.2. Real data

We evaluate the performance of HDSP and compare it with the HDP, labeled LDA (L-LDA) and partially labeled LDA (PLDA). The L-LDA defines a one-to-one correspondence between latent topics and labels. We use two multi-labeled corpora, RCV 2, newswire from Reuter s, and OHSUMED 3, a subset of the Medline journal articles, and one partially labeled corpus, Wikipedia.

Experimental Settings: For the HDP and HDSP, we initialize the word-topic distribution with three iterations of LDA for fast convergence to the posterior while preventing the posterior from falling into a local mode of LDA, then reorder these topics by the size of the posterior word count. For all experiments, we set the truncation level T to 200. We terminate variational inference when the fractional change of the lower bound falls below 10 3, and we optimize all hyper parameters during inference except η. For the L-LDA and PLDA, we implement the collapsed Gibbs sampling algorithm. For each model, we run 5,000 iterations, the first 3,000 as burn-in and then using the samples thereafter with gaps of 100 iterations. For PLDA, we set the number of topics for each label to two and five (PLDA-2, PLDA-5). We try five different values for the topic Dirichlet parameter η: η = 0.1, 0.25, 0.5, 0.75, 1.0. Finally all results are averaged over 20 runs with different random initialization. We do not report the standard errors because

2http://trec.nist.gov/data/reuters/reuters.html 3http://ir.ohsu.edu/ohsumed/ohsumed.html

Hierarchical Dirichlet Scaling Process

0.0 0.2 0.4 0.6 0.8 1.0 Topic Dirichlet prior

HDSP HDP L-LDA PLDA-2 PLDA-5

(a) OHSUMED

0.0 0.2 0.4 0.6 0.8 1.0 Topic Dirichlet prior

HDSP HDP L-LDA PLDA-2 PLDA-5

0.0 0.2 0.4 0.6 0.8 1.0 Topic Dirichlet prior

HDSP HDP PLDA-2 PLDA-5

(c) Wikipedia

Figure 5. Perplexity of held-out documents. For HDSP, L-LDA and PLDA, the perplexity is measured given documents and observed labels. For HDP, the model only uses the words of the documents. The HDSP which, instead of excluding topics from unobserved labels, scales all topics according to distances to observed labels, shows the best heldout perplexity.

they are small enough to ignore.

Evaluation Metric: The goal of our model is to construct the dependent random probability measure given labels. Therefore, our interest is to see the increments of predictive performance when the label information is given.

The predictive probability given label information for held-out documents are approximated by the conditional marginal,

p(x |r , Dtrain) = (21) Z

k=1 p(x n|φk)p(z n = k|π )p(π |V, r )dq(V, w, φ),

where Dtrain = {xtrain, rtrain} is the training data, x is the vector of N words of a held-out document, r are the labels of the held-out document, z n is the latent topic of word n, and π k is the kth topic proportion of the held-out document. Since the integral is intractable, we approximate the probability

p(x |r , Dtrain)

k=1 πk φk,x n, (22)

where φk and πk are the variational expectation for φk and πk given label r . This approximated likelihood is then used to compute the perplexity of the held-out document

perplexity = exp ln p(x |r , Dtrain)

Lower perplexity indicates better performance. We also take the same approach to compute the perplexity for LLDA, PLDA and HDP, but HDP does not use the labels of held-out documents. To measure the predictive performance, we leave 20% of the documents for testing and use the remaining 80% to train the models.

Multi-labeled Data: We use two multi-labeled corpora, the RCV Reuters news data and the OHSUMED medical journal data. Both are multi-labeled, and every document has at least one label. The average number of labels per article is 3.2 for RCV and 5.2 for OHSUMED. Table 1 contains the details of the datasets.

The HDSP outperforms all comparison models, HDP, LLDA and PLDA, in terms of perplexity as shown in Figure 5. For the OHSUMED data, the performance of L-LDA is worse than the HDP even though L-LDA is trained with label information. PLDA, which relaxes the assumption of LLDA by adding an additional latent label and allowing multiple topics per label, outperforms the HDP and the L-LDA. But it excludes the topics of unobserved labels from modeling the document, and it performs worse than the HDSP. Note that the HDSP also relies on the observed labels to strongly guide the topics, but it still allows all topics to be used, even ones that are not closely located to the observed labels.

Figure 6 shows the expected topic distributions given different sets of labels. As we discussed in Section 2, the scaling effect yields more sharpened distribution given a set of labels. When multiple labels are given, the model expects high probabilities for the topics that are closely located to all given labels. The Appendix provides more examples from the other corpora as well as posterior topic count analysis.

To visualize the latent locations, we embed the inferred topics and the given labels into the two dimensional space by using multidimensional scaling (MDS) on the inferred distances (Kruskal, 1964). In Figure 1, we choose and display a few representative topics and labels.

Partially-labeled Data: We also test our model with partially labeled data which have not been explicitly covered in topic modeling. Many real-world data fall into this cate-

Hierarchical Dirichlet Scaling Process

0 5 10 15 20 0.00

0 5 10 15 20 0.00

label=Market

0 5 10 15 20 0.00

label=Market, Economics

0 5 10 15 20 0.00

label=Econonmics, Domestic Politics

0 5 10 15 20 0.00

label=Sports

bill day week million market

stock rupee exchang

trad market

buy index research

stat plan agree

corp acquir operat

minist govern

party presid

lead polit parlia

bank credit financ

de brazil billion

labor govern

pay percent zimbabw

shar percent

trad million

pric invest compan

stak market

debt common

million compan

offer secur outstand sharehold

sale produc

car stor percent

retail motor industr market

fund foreign percent

sect market

asset mutual

iraq forc kurd

stat fight offic northern

unit turkey

ton export million

crop produc percent

pric harvest

ton grain barley

ship tend deliv soybean

bushel chicag

beat play champ

win seed olymp round

refug burund

rwand student

tuts offic presid

child polic dutroux

girl sex belgian

hous miss charg

polic immigr

afric church

pari govern

team socc england

cup scor club

pric barrel

cent septemb

mark germ currenc

yen franc foreign

french exchang frankfurt

label=Sports, Market

Figure 6. Expected topic distributions given labels from RCV. Topics are sorted by their posterior word counts, and the top 20 topics are displayed with the top 10 words (stemmed). From top to bottom, we compute an expected topic distribution given a set of labels.

Table 2. The most closely located category-topic pairs from Wikipedia. The categories are more specific and narrow than those of RCV. Labels Top words rivers of romania river water area bay north creek valley south cities in iran district population province village county rural area town italian painters van dutch painter italian born netherlands portuguese portrait 2009 albums released song music band single track records songs public high schools school high students schools college education state girls

gory where some of the data are labeled, others are incompletely labeled, and the rest are unlabeled. For this experiment, we randomly sampled Wikipedia articles and use the categories as labels. Because most of the categories only appear once or twice in the dataset, we remove categories that appear in fewer than five articles, and our dataset contains 25,547 articles, 1,093 labels, with 0.6 labels per article on average. The label information is sparse even after removing the infrequent labels.

Figure 5(c) shows the predictive perplexity with Wikipedia. We compare the result with PLDA and HDP. We exclude LLDA which cannot be trained on a dataset containing documents with no labels. HDSP outperforms both HDP and PLDA. We list the top ten most closely located label-topic pairs in Table 2.

5. Conclusion

We have presented the hierarchical Dirichlet scaling process (HDSP), a Bayesian nonparametric prior for a mixed

membership model where categorical side information plays an important role. We discussed how the HDSP models the latent locations of the mixture components and the observed labels, and boosts the membership probability of a mixture component based on the product of the inverse distances to the labels. We showed that the application of HDSP to topic modeling correctly recovers the topics and label-topic distances of synthetic data. Furthermore, we showed the improved predictive performance of the HDSP topic model compared to the HDP, labeled LDA and partially labeled LDA. Future work on this research will explore kernel functions instead of simple products and applications of the HDSP topic model on various text mining problems.

6. Acknowledgement

This work was supported by the IT R&D program of MSIP/KEIT. [10041313, UX-oriented Mobile SW Platform]

Hierarchical Dirichlet Scaling Process

Ahmed, Amr and Xing, Eric P. Timeline: A dynamic hierarchical dirichlet process model for recovering birth/death and evolution of topics in text stream. In Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence (UAI), pp. 20 29, 2010.

Bishop, Christopher M and Nasrabadi, Nasser M. Pattern recognition and machine learning, volume 1. springer New York, 2006.

Blei, D, Ng, A, and Jordan, M. Latent dirichlet allocation. The Journal of Machine Learning Research, pp. 993 1022, Jan 2003.

Blei, David M and Lafferty, John D. A correlated topic model of science. The Annals of Applied Statistics, pp. 17 35, 2007.

Blei, D.M. and Jordan, M.I. Variational inference for dirichlet process mixtures. Bayesian Analysis, 1(1):121 144, 2006.

Duan, Jason A, Guindani, Michele, and Gelfand, Alan E. Generalized spatial dirichlet process models. Biometrika, 94(4):809 825, 2007.

Dunson, David B and Park, Ju-Hyun. Kernel stick-breaking processes. Biometrika, 95(2):307 323, 2008.

Escobar, M.D. and West, M. Bayesian density estimation and inference using mixtures. Journal of the american statistical association, pp. 577 588, 1995.

Jordan, Michael I., Ghahramani, Zoubin, Jaakkola, Tommi S., and Saul, Lawrence K. An introduction to variational methods for graphical models. Machine Learning, 37(2):183 233, November 1999. ISSN 08856125.

Kim, Dongwoo, Kim, Suin, and Oh, Alice. Dirichlet process with mixed random measures: a nonparametric topic model for labeled data. In Proceedings of the 29th International Conference on Machine Learning (ICML), 2012.

Kruskal, Joseph B. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29(1):1 27, 1964.

Liang, Percy, Petrov, Slav, Jordan, Michael I, and Klein, Dan. The infinite pcfg using hierarchical dirichlet processes. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLPCo NLL), pp. 688 697, 2007.

Mac Eachern, S.N. Dependent nonparametric processes. In ASA Proceedings of the Section on Bayesian Statistical Science, pp. 50 55, 1999.

Paisley, John, Wang, Chong, and Blei, David M. The discrete infinite logistic normal distribution. Bayesian Analysis, 7(4):997 1034, 2012.

Ramage, D., Hall, D., Nallapati, R., and Manning, C.D. Labeled lda: A supervised topic model for credit attribution in multi-labeled corpora. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 248 256, 2009.

Ramage, Daniel, Manning, Christopher D., and Dumais, Susan. Partially labeled topic models for interpretable text mining. In Proceedings of the 17th ACM International Conference on Knowledge Discovery and Data Mining (KDD), pp. 457 465, New York, NY, USA, 2011.

Rasmussen, Carl Edward and Williams, Christopher K. I. Gaussian Processes for Machine Learning (Adaptive Computation and Machine Learning). The MIT Press, 2005. ISBN 026218253X.

Ren, Lu, Du, Lan, Carin, Lawrence, and Dunson, David. Logistic stick-breaking process. The Journal of Machine Learning Research, 12:203 239, 2011.

Sethuraman, J. A constructive definition of dirichlet priors. Statistica Sinica, 4:639 650, 1991.

Srebro, Nathan and Roweis, Sam. Time-varying topic models using dependent dirichlet processes. UTML, TR# 2005, 3, 2005.

Teh, Y., Jordan, M., Beal, M., and Blei, D. Hierarchical dirichlet processes. Journal of the American Statistical Association, Jan 2006.

Teh, Y., Kurihara, K., and Welling, M. Collapsed variational inference for HDP. Advances in Neural Information Processsing Systems (NIPS), 20, 2008.

Wainwright, Martin J and Jordan, Michael I. Graphical models, exponential families, and variational inference. Foundations and Trends R in Machine Learning, 1(1-2): 1 305, 2008.

Wang, Chong, Paisley, John W, and Blei, David M. Online variational inference for the hierarchical dirichlet process. In International Conference on Artificial Intelligence and Statistics (AISTATS), pp. 752 760, 2011.

Xing, Eric P, Jordan, Michael I, Russell, Stuart, and Ng, Andrew. Distance metric learning with application to clustering with side-information. In Advances in neural information processing systems (NIPS), pp. 505 512, 2002.