Model

θ˜_d3)^>, where ˜θ_dlis assumed to follow the Dirichlet distribution as prior, ˜θ_dl ∼Dirichlet(θ^˜₀). Therefore, the number of dimensions ofθ_d, or the total number of topics, isK₁+K₂+ K₃ which are the number of negative, neutral, and positive topics, respectively. In this study, unlike LDA2vec, we take topic assignments following the topic distribu-tion that takes into account the polarity of the document.

The proposed model uses the word vectors, topic vectors, and the topic assign-ments to the words to construct the context vectors representing what the word means in the context, which the sum of the word vector and the topic vector cor-responding the topic assignment to the word i, −→

c_i = −→ w_i +−→

t _k, if z_i = k. This formulation stems from the idea of natural language processing that the meaning of a word is expressed on the basis of the original meaning of the word, but it can fluctuate by taking into account the context of the document or sentence in which the word is used. The formulation of the context vectors of this study is almost same as that of Moody (2016)’s LDA2vec.

In the process of text generation model, the proposed model considers its sur-rounding words to define the probability of generating the focal word. We defineS_i as a multiset that contains the surrounding words of the wordiin the dataset. Let IP = {(i,j)| j∈ S_i}andIN = {(i,j)|j∈/S_i}be the positive and negative multisets, respectively, andI_D = I_P∪I_N be the multiset of total vocabulary. Then, we define G= {g_ij |(i,j)∈ I_G}, whereg_ij is a random variable whose value is taken to be 1 if (i,j)∈ I_Por−1 if(i,j)∈ I_N. In the proposed model, this random variable indicating whether the wordiand the wordjare in the same window is assumed to follow the binomial logit model, which the probability of the variable is defined as the stan-dard sigmoid (or logistic) function of the inner product of the context vector and the surrounding word vector,p(g_ij | −→

w_i,−→ w_j,{−→

t _k},z_i) =σ(g_ij· −→ c_i^>−→

w_j), whereσ(x) = 1/1+exp(−x). Because the dependent variable of the logit model (−→

c _i) is also latent variable, this formulation can be seen as the factor model by regarding−→

c _ias factor scores of the wordiand−→

w_j as factor loadings of the word j. However, this factor model has a constraint that factor score−→_c

ican be decomposed into the word vector and the corresponding topic vector,−→

c_i = −→ w_i+−→

t _k.

5.3. Model 111 Therefore, the likelihood of the word embedding part is provided as follows:

p(G,{−→_w

i},{−→

t _k},Z|θ,^˜ π) =

∏

(

p(z_i |θ^˜_di,π_di)

∏

σ(g_ij· −→_c >

i −→_w

j) )

, (5.1)

whereV is the total number of vocabulary in the corpus. In the proposed model, the embedding vectors are trained to maximize the probability of the inner product of the context vector and the surrounding word vector, so that word vectors, which their corresponding words often belong to the same window in a dataset will have similar values. Therefore, we can obtain vector representations taking into account the co-occurrence of words within the window, that is, the context, while LDA uni-formly considers the co-occurrence of words in a document.

However, the computation cost of the likelihood (5.1) is high because the number of total vocabulary is usually over thousands and the total cost of the likelihood is its square. In this study, according to the approach proposed by Mikolov et al. (2013), we also apply a negative sampling technique to approximate the likelihood (5.1) with the following computable formulation:

p(G,{−→ w_i},{−→

t _k},Z|θ,^˜ π)≈

∏

(

p(z_i |θ^˜_di,π_di)

∏

j∈Si

σ(−→ c^>_i −→

w_j)×

∏

σ(−−→ c_i^>−→

w_n) )

, (5.2) where P_n(w) is the noise distribution as a free parameter, and we choose the uni-gram distribution raised to the 3/4 th power according to the Mikolov et al. (2013)’s suggestion. The number of negative samples, N, is suggested to be 5−20 for a small dataset and 2−5 for a large dataset. In machine learning research, since they use a huge dataset containing millions and sometimes billions words for training, our dataset is relatively small. We determine 15 words for the number of negative samples.

5.3.2 Preference Measurement Models Considering Brand Heterogeneity and Consumer Attributes

Another extension of this study from Moody (2016)’s LDA2vec model is to com-bine the word embedding model considering text topic and sentiment as explained above and preference measurement model considering brand heterogeneity and

consumer attributes in a supervised learning fashion. Recalling that the topic as-signments is assumed to follow the categorical distribution of the topic proportion, z_i ∼categorical(θ_di), the topic proportionθ_drepresents the summary of the product attributes mentioned in the reviewd, and in the preference measurement model, it works as dependent variables for explaining the customer satisfaction or the review rating score of the reviewd.

Lety_d be the satisfaction score of the review dthat can take values from 1 toR (Rdepends the e-commerce site platform, for example,R = 5 in Amazon), and we define two ordered probit models to clarify the structure of the satisfaction scores according to the two conceivable processes that consumer attributes directly or indi-rectly affects the satisfaction structure. First, in the direct effect model, the consumer attributes work as dependent variables for satisfaction score to capture their direct effects on the satisfaction structure as follows.

y_d = r ifτ_r−1 ≤y^∗_d< τ_r y^∗_d = α_bd +

∑

β_bdkθ_dk+

∑

X_dqγ_q+e_d, e_d∼ N(_0,σ²)_{, (5.3)}

whereKis the total number of topics including negative, neutral, and positive top-ics, and the thresholds{τ_r}work for realizing discrete satisfaction scoresy_dthrough the latent continuous variabley^∗_d, and the both sides of thresholds, τ₀ andτ_R, and the next two thresholds,τ₁andτ_R−1, are set to−_∞and∞, the points of the empir-ical ratio of the corresponding rating scores in the normal cumulative distribution, respectively, for model identifiability. Also,b_d indicates the brand for which the re-viewdwrote, and we introduce the brand heterogeneity in the brand interceptαand the coefficients of the topic distributionβ. X_d is dummy variables of the reviewer d’s categorical attributes, such as age and status of the reviewer ranking, and γ is the coefficient vector capturing the direct effects of the consumer attributes on the satisfaction structure.

Next, we construct different model from the above direct effect model to under-stand the effects of the consumer attributes on the product attributes that the re-viewers mention in the review; that is to say, it is the indirect effect of the consumer attributes on the satisfaction structure through the hierarchical structure of the topic

5.3. Model 113 distribution. We assume the Dirichlet prior for the topic distribution of each polarity θ˜_dlin the above direct effect model, but the indirect effect model considers the prior hierarchical structure for the topic distributions as follows.

θ˜_dl = softmax ¯θ_dl θ¯_dl =

∑

X_dqγ_lq+λ_dl, λ_dl ∼ MV N(0, 0.1²I). (5.4)

Then the indirect effect model does not consider the direct effects of the consumer attributes on the satisfaction structure,y^∗_d =α_bd +_∑^K_k=1β_bdkθ_dk+e_d.

Therefore, the likelihood of the preference measurement direct effect model for the customer satisfaction is provided as follows:

p(Y,τ,α,β,γ,σ |θ,^˜ π,X) = ( D

d

∏

p(y_d |y^∗_d,τ)p(y^∗_d|θ^˜_d,π_d,α_bd,β_bd,γ,σ,X) )

×p(τ,α,β,γ,σ), (5.5)

and that of the indirect effect model is provided as follows:

p(Y,τ,α,β,γ,σ|θ,^˜ π,X) = ( D

d

∏

p(y_d|y^∗_d,τ)p(y^∗_d |θ^˜_d,π_d,α_bd,β_bd,σ) )

×p(τ,α,β,γ,σ), (5.6)

where p(τ,α,β,γ,σ)is the joint prior distribution for the coefficients, and the defi-nition is explained in the Appendix A.4. Under the assumption of the conditional independence of likelihood (5.1) and (5.5) or (5.6) when the topic distributions are given, the full joint likelihood of the proposed model is obtained by the product of these equations multiplied by the prior density for the topic distribution, which is the the Dirichlet priorp(θ^˜_dl | θ^˜₀) ∼ Dirichlet(θ^˜₀)in the direct effect model and the multivariate normal priorp(θ^˜_dl |X_d,γ_l)∼ MV N(X_d^>γ_l, 0.1²I).

Finally, we briefly introduce the estimation procedure for the proposed models.

In estimation procedure of the proposed model, we take a hybrid approach com-bining the Markov Chain Monte Carlo sampling method and the gradient-based stochastic optimization using Adam (Kingma and Ba, 2015). The stochastic opti-mization gives the optimal point estimates at that iteration for the two embedding

vectors, and then we sample from the posterior distributions using the Metropolis-Hastings algorithm for the topic distributions and the Gibbs sampling for the re-maining parameters, given the point estimates for the embedding vectors updated for every iteration. The estimation procedure and the settings of analysis in the fol-lowing empirical study are explained in more detail in the Appendix A.4.