Examining
the Impact of Ranking on Consumer Behavior and Search Engine
Revenue
Anindya Ghose Department of Information, Operations and Management Sciences,
Department of Marketing, Stern School of Business, New York University, New
York, New York 10012, aghose@stern.nyu.edu Panagiotis G. Ipeirotis Department
of Information, Operations and Management Sciences, Stern School of Business,
New York University, New York, New York 10012, panos@stern.nyu.edu Beibei Li
Information Systems and Management, Heinz College, Carnegie Mellon University,
Pittsburgh, Pennsylvania 15213, beibeili@andrew.cmu.edu
I n this paper, we study
the effects of three different kinds of search engine rankings on consumer
behavior and search engine revenues: direct ranking effect, interaction effect
between ranking and product ratings, and personalized ranking effect. We
combine a hierarchical Bayesian model estimated on approximately one million
online sessions from Travelocity, together with randomized experiments using a
real-world hotel search engine application. Our archival data analysis and
randomized experiments are consistent in demonstrating the following: (1) A
consumer-utility-based ranking mechanism can lead to a significant increase in
overall search engine revenue. (2) Significant interplay occurs between search
engine ranking and product ratings. An inferior position on the search engine
affects “higher-class” hotels more adversely. On the other hand, hotels with a
lower customer rating are more likely to benefit from being placed on the top
of the screen. These findings illustrate that product search engines could
benefit from directly incorporating signals from social media into their
ranking algorithms. (3) Our randomized experiments also reveal that an “active”
personalized ranking system (wherein users can interact with and customize the
ranking algorithm) leads to higher clicks but lower. purchase propensities and lower search engine revenue compared with a “passive” personalized ranking system (wherein users cannot interact with the ranking algorithm). This result suggests that providing more information during the decision-making process may lead to fewer consumer purchases because of information overload. Therefore, product search engines should not adopt personalized ranking systems by default. Overall, our study unravels the economic impact of ranking and its interaction with social media on product search engines. Keywords: travel search engine; randomized experiments; hierarchical Bayesian methods; information systems; IT policy and management; electronic commerce History: Received May 28, 2012; accepted May 12, 2013, by Lorin Hitt, information systems. Published online in Articles in Advance. 1. Introduction Over the last few decades, search engines have emerged as a significant channel for promoting and selling products. In information search engines (e.g., Google) the ranking of the search results is an immediate signal of the relevance of the result to the query. However, in product search engines, the ranking of the displayed products is often based on criteria such as price, product rating, etc. In such a setting, we may often have multiple, potentially conflicting signals given to the customer about the products’ rankings. For example, if we rank by price, then the cheapest products sometimes have low product ratings, or products appearing on top of the list may be too expensive for the customer. Effectively consumers have to observe multiple, competing ranking signals and come up with their own ranking in their minds; in some settings, the product search engine will also generate personalized results, trying to rank the products according to the preferences of the consumer. In such an environment, we want to understand which factors influence the decision-making process of the customers and the magnitude of that influence. Are consumers influenced by the display ranking order, by the product rating, by price, and to what degree? How does this interplay affect the revenue that a search engine can generate?
1.1. Related Work
In the last 10 years,
the literature in e-commerce has shown the existence of a strong primacy effect
in environments wherein consumers make choices among offers displayed in
information search engines such as Google, Yahoo, or Bing. Specifically, we have learned that an online position effect
exists and that rank order has a significant impact on the clickthrough rates
and conversion rates (e.g., Ghose and Yang 2009, Yang and Ghose 2010, Agarwal
et al. 2011, Jerath et al. 2011, Rutz and Trusov 2011, Narayanan and Kalyanam
2011, Animesh et al. 2011, Baye et al. 2012, Jeziorski and Segal 2012, Rutz et
al. 2012, Abhishek et al. 2013, Ghose et al. 2013a). These papers focused
primarily on evaluating the effect of screen position on user behavior,
controlling for the quality of the advertisement. However, in product search
engines, the observed demand patterns can be influenced by the joint variation
in product ratings (either professional rating or user rating) and online
screen position. The first goal of our study is to examine the position effect
in product search engines, conditional on its interaction with product ratings.
Search engines are beginning to adopt signals from social media sites directly
into their ranking mechanism design (e.g., Bing Social Search, TripAdvisor).
Recent work has found that a utility-based ranking mechanism on product search
engines that incorporates multidimensional consumer preferences and social
media signals can lead to significant surplus gain for consumers (Ghose et al.
2012). However, given that price was not the top priority considered in the
ranking recommendation, whether such a mechanism can actually benefit product
search engines is unclear because their revenues are normally commission based.
Therefore, the second goal of our study is to examine the effect of different
ranking mechanisms on product search engine revenue. Outside of search, one of
the most important ways for shoppers to discover products has been through
recommendation engines (Chittor 2010). However, although some online retailers
use recommendation systems, many product-specific search engines (e.g., travel
search engines) still do not provide personalized ranking results in response
to consumer queries, presumably because these product search engine companies
are unsure whether providing extra information to consumers will lead to an
increase in profit. Existing research holds two different opinions on the
effects of personalization. One
stream of work is supportive of personalization (e.g., Rossi et al. 1996,
Ansari and Mela 2003, Arora and Henderson 2007, Yao and Mela 2011), whereas
another stream of work is a bit more skeptical (e.g., Zhang and Wedel 2009,
Aral and Walker 2011, Goldfarb and Tucker 2011, Lambrecht and Tucker 2013),
suggesting that although personalization can lead to higher customer
satisfaction and profits, it will not work as well universally.1 However, none
of these papers have examined the effect of information availability and personalization
in a search engine context. Koulayev (2014) examines consumers’ costly search
behavior on travel search engines through the formation of consideration sets.
Ghose et al. (2013b) build a structural econometric model to predict individual
consumers’ online footprints on product search engines to improve user
experiences under the context of social media overload. Chen and Yao (2012) use
secondary data to examine how the sorting and filtering tools on travel search
engines influence consumer hotel search. They find these tools result in a
significant increase in total search activities, but they also lead to lower
overall welfare because of the disproportional engagement induced by the
refinement tools. With these findings in mind, our third goal is to examine how
different kinds of personalized ranking mechanisms in product search engines
affect consumer behavior and search engine revenues. Specifically, does
allowing users to interact with the ranking algorithm to proactively
personalize their search results lead to more or fewer purchases?
1.1. Contributions and Results
1. Data
Our data set consists of detailed information
on a total of 969,033 online sessions from Travelocity.com, including consumer
searches, clicks, and conversions that occurred within these sessions between
November 2008 and January 2009. In addition, we have hotelrelated information,
such as hotel class, brand, online reviewer rating, and number of reviews. We
collected customer reviews from Travelocity.com. We collected the online
reviews and reviewers’ information on a daily basis up to January 31, 2009 (the
last date of transactions in our database). This process provides us with a
final data set containing 29,222 weekly observations for 2,117 hotels in the
United States.3 We define an “online session” to capture a set of activities by
an online user, identified by a unique cookie. In our data, a starting
indicator and an ending indicator with a corresponding time stamp (provided by
the company) can characterize each unique online session. More specifically, a
typical online session involves the initialization of the session, the search
query, the results (in a particular rank order) returned from that search
query, the sorting method, the click(s) on hotel(s) if any exist, the login and
actual transaction(s) if any conversion occurs, and the termination of the
session. The ending indicator marks the termination of a session.
1. Empirical Model In this section,
we discuss how we develop our simultaneous
model in a hierarchical Bayesian framework. Then we describe how we apply the
Markov chain Monte Carlo (MCMC) methods (Rossi and Allenby 2003) to empirically
identify the effects of product quality and ranking position on consumer search
and purchase behavior. More specifically, our model is motivated by the work of
Ghose and Yang (2009). The general idea is as follows: We propose to build a simultaneous equations
model of clickthrough, conversion, rank, and customer rating. We model the
clickthrough and conversion behavior as a function of hotel brand, price, rank,
page, sorting criteria, customer rating, and hotel characteristics. The rank of
a hotel is modeled as a function of hotel brand, price, sorting criteria,
customer rating, hotel characteristics, and performance metrics such as
previous conversion rate. The customer rating of a hotel is modeled as a function
of hotel brand, price, rank, page, sorting criteria, and hotel characteristics.
Each function contains an unobserved error that is normally distributed with
mean zero. To capture the unobserved covariation among clickthroughs,
conversions, rank, and customer rating, we assume the four error terms are
correlated and follow the multivariate normal distribution with mean zero. We
describe our model next.
1. Empirical Analyses and Results
To estimate our model,
we applied the MCMC methods using a Metropolis–Hastings algorithm with a random
walk chain (Chib and Greenberg 1995). In particular, we ran the MCMC chain for
80,000 iterations and used the last 40,000 iterations to compute the mean and
standard deviation of the posterior distribution of the model parameters. We
provide more details on the MCMC estimation algorithm in Online Appendix D.
rank, suggesting the
negative effect of rank order on conversion rate also increases at a decreasing
rate. As expected, Price has a negative effect on hotel demand, whereas Class
has a positive effect on hotel demand. The online WOM-related variables, Rating
and ReviewCnt, have a statistically significant and positive effect on hotel
demand. We also found similar trends in the interaction effects between Ranking
and Price/Class/Rating, suggesting higher-class hotels and more expensive
hotels are more sensitive to the online ranking effect. And hotels that receive
lower ratings from users benefit more when placed on the top of the screen. The
total number of hotels in a certain market, H, has a negative effect on
hotel-level conversion rate. Intuitively, the higher the number of choices
there are available to consumers, lower the probability of buying from any
given hotel. Thus, on average, the conversion
rate for each hotel decreases.
1. Conclusions and Implications
In this paper, we focus
on investigating three major issues that product search engines are
increasingly facing: the direct effect of ranking mechanism on consumer
behavior and search engine revenue; the interaction effect of ranking and
product ratings; and what kind of personalized ranking mechanism, if any, to
adopt. Toward these objectives, we combine archival data analysis with
randomized experiments based on a hotel search engine application that we designed.
By manipulating the default ranking method and enabling or disabling a variety
of active personalization features on the hotel search engine website, we are
able to analyze consumer behavior and search engine revenue under different
scenarios. Our experimental results on ranking are consistent with those from
the Bayesian model-based archival data analysis, suggesting a significant and
causal effect of search engine ranking on consumer click and purchase behavior.
In addition to a significant surplus gain found by a previous study (Ghose et
al. 2012), a consumer-utility-based ranking mechanism yields the highest
purchase propensity and the highest search engine overall revenue compared with
existing benchmark systems, such as ranking based on price or star ratings.
Moreover, an inferior screen position tends to more adversely affect luxury
hotels and more expensive hotels. Hotels with lower reputations benefit more
from being placed at the top of the search results. This finding illustrates
the need for product search engines to directly incorporate signals from online
social media into the ranking algorithms. We are beginning to see much of this
interplay between search and social media happening in information search
engines. Google began to incorporate tweets and other social media status
updates into its real-time search function and then decided to create its own
version of the Facebook “Like” button— the Google +1—and have it show up in
search results. In another example of the interplay between social media and
search, Microsoft’s search engine Bing is now incorporating Facebook updates in
its results. Our experimental results on personalized ranking show the availability of excess
personalization capabilities during the decision-making process may discourage
consumers from searching, evaluating, and making final choices. In particular,
we find that although active personalized ranking, compared with passive
personalized ranking, can attract more online attention from consumers, it
leads to a lower purchase propensity and lower search engine revenue. This
finding suggests that personalized ranking should not be adopted blindly and
the level of personalization should be carefully designed based on the search
context. Our research sheds light on how consumers search, evaluate choices,
and make purchase decisions in response to differences in product search engine
designs. We provide empirical and experimental evidence for future studies to
build on when designing an efficient ranking system and dynamically modeling
consumer behavior on product shopping sites. A good ranking mechanism can
reduce consumers’ search costs, improve clickthrough rates and conversion rates
of products, and improve revenue for search engines. Our work has some
limitations, some of which we are striving to address in our ongoing work.
First, although the AMT platform provides an efficient and cost-friendly
framework for randomized experimental design, the inherent heterogeneity in the
Internet population makes controlling for subject characteristics across
different treatment groups difficult. The randomization process can alleviate
this concern to a large extent. However, robustness tests based on offline
subjects as well would be helpful. Our current experiments focus on the type of
consumers who can make, at most, one purchase in each online shopping session.
To better understand the counterintuitive finding that an active personalized
ranking mechanism leads to lower conversion rates, one can extend our
experimental design to make a comparison with consumers who are allowed to make
multiple purchases in a given session. In addition, a study of how the content
of consumer search, such as the length and type of search keyword, interacts
with the ranking effect would be interesting. Moreover, with regard to
examining the ranking mechanism, one can expand the research scope by taking into account consumers’
social network neighbors’ search and purchase behavior. This expansion would
allow one to test the impact of social-signal-based ranking mechanisms on
product search engines. Notwithstanding these limitations, we believe our paper
paves the way for future research in this exciting area at the intersection of
social media and search engines.
Acknowledgments
For helpful comments, the authors thank Jason
Chan, Sam Hui, Prasanna Tambe, Duncan Simester, Yong Tan, and Russ Winer, as
well as seminar participants at Carnegie Mellon University, Cheung Kong
Graduate School of Business, Columbia University, Harvard Business School, the
Massachusetts Institute of Technology, University of Texas at Austin,
University of Connecticut, University of Texas at Dallas Marketing Conference,
the Wharton School, University of California, Los Angeles, and New York
University. Anindya Ghose and Panagiotis G. Ipeirotis acknowledge funding from
the Google Faculty Research Award in 2012.
References
Abhishek V, Hosanagar K, Fader P (2013)
Aggregation bias in sponsored search data: The curse and the cure. Working
paper, Carnegie Mellon University, Pittsburgh. Agarwal A, Hosanagar K, Smith M
(2011) Location, location, location: An analysis of profitability of position
in online advertising markets. J. Marketing Res. 48(6):1057–1073. Animesh A,
Viswanathan S, Agarwal R (2011) Competing “creatively” in sponsored search
markets: The effect of rank, differentiation strategy, and competition on
performance. Inform. Systems Res. 22(1):153–169. Ansari A, Mela CF (2003)
E-customization. J. Marketing Res. 40(2): 131–145. Aral S, Walker D (2011)
Creating social contagion through viral product design: A randomized trial of
peer influence in networks. Management Sci. 57(9):1623–1639. Arora N, Henderson
T (2007) Embedded premium promotion: Why it works and how to make it more
effective. Marketing Sci. 26(4):514–531. Baye MR, De los Santos B, Wildenbeest
MR (2012) What’s in a name? Measuring prominence, and its
impact on organic traf- fic from search engines. Working paper, Indiana
University, Bloomington. Baye MR, Gatti JRJ, Kattuman P, Morgan J (2009)
Clicks, discontinuities, and firm demand online. J. Econom. Management Strategy
18(4):935–975. Chen Y, Yao S (2012) Search with refinement. Working paper,
Northwestern University, Evanston, IL.Chevalier J, Mayzlin D (2006) The effect of word of
mouth on sales: Online book reviews. J. Marketing Res. 43(3):345–354. Chib S,
Greenberg E (1995) Understanding the Metropolis–Hastings algorithm. Amer.
Statistician 49(4):327–335. Chittor V (2010) Online retail: Getting the right
product in front of your customers. Inc. (March 8), http://www.inc.com/internet/
articles/201002/chittoor.html. Dellarocas C (2012) Double marginalization in
performancebased advertising: Implications and solutions. Management Sci.
58(6):1178–1195. De los Santos B, Koulayev S (2013) Optimizing click-through in
online rankings for partially anonymous consumers. Working paper, Indiana
University, Bloomington. Dodds WB, Monroe KB, Grewal D (1991) Effects of price,
brand, and store information on buyers’ product evaluations. J. Marketing Res.
28(3):307–319. Dzyabura D (2014) Preference discovery in product search: When
novice consumers don’t know what they don’t know. Working paper, New York
University, New York. Ghose A, Yang S (2009) An empirical analysis of search
engine advertising: Sponsored search in electronic markets. Management Sci.
55(10):1605–1622. Ghose A, Goldfarb A, Han S (2013a) How is the mobile Internet
different? Search costs and local activities. Inform. Systems Res.
24(3):613–631. Ghose A, Ipeirotis PG, Li B (2012) Designing ranking systems for
hotels on travel search engines by mining user-generated and crowd-sourced
content. Marketing Sci. 31(3):493–520. Ghose A, Ipeirotis PG, Li B (2013b)
Surviving social media overload: Predicting consumer footprints on product
search engines. Working paper, Carnegie Mellon University, Pittsburgh. Goldfarb
A, Tucker C (2011) Online display advertising: Targeting and obtrusiveness.
Marketing Sci. 30(3):389–404. Greene W (1999) Econometric Analysis, 4th ed.
(Prentice Hall, Upper Saddle River, NJ). Hausman JA (1975) An instrumental
variable approach to full information estimators for linear and certain
nonlinear econometric models. Econometrica 43(4):727–738. Jerath K, Ma L, Park
Y, Srinivasan K (2011) A “position paradox” in sponsored search auctions.
Marketing Sci. 30(4):612–627.
Latar
belakang Perilaku konsumen
Selama beberapa
dekade terakhir, search engine memiliki muncul sebagai saluran yang signifikan
untuk mempromosikan dan menjual produk. Di mesin pencari informasi (misalnya,
Google) peringkat hasil pencarian adalah langsung sinyal relevansi hasil untuk
query. Namun, dalam mesin pencari produk, peringkat produk yang ditampilkan
sering didasarkan pada kriteria seperti seperti harga, wisatawan produk, dll
Dalam kondisi seperti ini, kami mungkin sering memiliki beberapa, sinyal berpotensi
bertentangan diberikan kepada pelanggan tentang peringkat produk '. Sebagai
contoh, jika kita peringkat berdasarkan harga, maka produk termurah
kadang-kadang memiliki peringkat produk rendah, atau produk muncul di atas
daftar mungkin terlalu mahal bagi pelanggan. Efektif konsumen harus mengamati
beberapa, bersaing sinyal Peringkat dan datang dengan peringkat mereka sendiri
dalam pikiran mereka; dalam beberapa pengaturan, mesin pencari produk juga akan
menghasilkan hasil personalisasi, mencoba untuk peringkat produk menurut
preferensi konsumen.
Dalam lingkungan
seperti itu, kita ingin memahami faktor yang mempengaruhi proses pengambilan
keputusan pelanggan dan besarnya pengaruh itu. Adalah konsumen dipengaruhi oleh
peringkat display order, dengan rating produk, berdasarkan harga, dan untuk apa
Gelar? Bagaimana interaksi ini mempengaruhi pendapatan bahwa sebuah mesin
pencari dapat menghasilkan?
Metode
Perilaku Konsumen
Pada bagian ini,
kita membahas bagaimana kita mengembangkan simultan kami model dalam kerangka
Bayesian hirarkis. Kemudian kita menggambarkan bagaimana kita menerapkan rantai
Markov Monte Carlo (MCMC) metode (Rossi dan Allenby 2003) secara empiris
mengidentifikasi efek dari produk kualitas dan posisi peringkat pada pencarian
konsumen dan membeli perilaku. Lebih khusus, model kami 4 Dalam beberapa kasus,
pengguna dapat memulai sesi dan mencari umum informasi wisata, seperti kawasan
kota, daripada pencarian untuk setiap hotel; dengan demikian, tidak ada hotel
akan ditampilkan pada halaman Web manapun. Kita mengecualikan sesi tersebut
dari analisis kami. 5 Baru-baru ini, Travelocity upgrade desain halaman web
dengan menunjukkan 10 hotel per halaman. Namun, selama periode waktu
pemeriksaan kami, Jumlah itu masih 25. Termotivasi oleh karya Ghose dan Yang
(2009). Ide umum adalah sebagai berikut: Kami mengusulkan untuk membangun
sebuah model persamaan simultan klik per tayang, konversi, rank, dan dari
pelanggan. Kita model klik per tayang dan konversi perilaku sebagai fungsi
merek hotel, harga, peringkat, halaman, kriteria penyortiran, pelanggan
Peringkat, dan hotel karakteristik. Pangkat Hotel dimodelkan sebagai fungsi
hotel merek, harga, pemilahan kriteria, dari pelanggan, karakteristik Hotel,
dan metrik kinerja seperti konversi sebelumnya tingkat. Rating pelanggan dari
sebuah hotel dimodelkan sebagai fungsi dari hotel merek, harga, peringkat,
halaman, menyortir kriteria, dan karakteristik Hotel. Setiap fungsi mengandung
kesalahan tidak teramati yang biasanya didistribusikan dengan maksud nol. Untuk
menangkap covariation teramati antara jumlah klik, konversi, peringkat, dan
pelanggan peringkat, kita asumsikan empat hal error berkorelasi dan mengikuti
distribusi normal multivariat.
Kesimpulan
Perilaku Konsumen
Dalam tulisan
ini, kita fokus pada menyelidiki tiga besar isu bahwa mesin pencari produk yang
semakin menghadapi: efek langsung dari mekanisme pada konsumen perilaku dan
mesin pencari pendapatan; interaksi pengaruh penilaian peringkat dan produk;
dan apa jenis personalisasi mekanisme peringkat, jika ada, untuk mengadopsi.
Menuju tujuan ini, kami menggabungkan arsip analisis data dengan percobaan acak
berdasarkan aplikasi mesin pencari hotel yang kami dirancang. Dengan
memanipulasi metode peringkat default dan mengaktifkan atau menonaktifkan
berbagai personalisasi aktif fitur pada hotel Situs Web mesin pencari, kita mampu
menganalisis perilaku konsumen dan mencari pendapatan mesin di bawah skenario
yang berbeda.
Hasil
eksperimen kami pada personalisasi peringkat menunjukkan ketersediaan kemampuan
personalisasi kelebihan selama proses pengambilan keputusan dapat mencegah konsumen
dari mencari, mengevaluasi, dan membuat pilihan akhir. Secara khusus, kami
menemukan bahwa meskipun aktif peringkat pribadi, dibandingkan dengan pasif
pribadi ranking, dapat menarik lebih banyak secara online perhatian dari
konsumen, itu mengarah ke pembelian lebih rendah kecenderungan dan pendapatan
mesin pencari yang lebih rendah. Ini temuan menunjukkan bahwa peringkat pribadi
seharusnya tidak diadopsi membabi buta dan tingkat personalisasi harus
hati-hati dirancang berdasarkan pencarian konteks. Penelitian kami menyoroti
bagaimana konsumenpencarian, mengevaluasi pilihan, dan membuat keputusan
pembelian dalam menanggapi perbedaan mesin pencari produk desain. Kami
memberikan bukti empiris dan eksperimental untuk studi masa depan untuk
membangun ketika merancang sistem ranking yang efisien dan dinamis pemodelan perilaku
konsumen di situs belanja produk. Mekanisme peringkat yang baik dapat
mengurangi konsumen biaya pencarian, meningkatkan rasio klik per tayang dan
konversi tingkat produk, dan meningkatkan pendapatan untuk mesin pencari.
Pekerjaan
kami memiliki beberapa keterbatasan, beberapa di antaranya kami berusaha untuk
mengatasi dalam pekerjaan berkelanjutan kami. pertama, meskipun platform AMT
menyediakan efisien dan Kerangka ramah-biaya untuk acak eksperimental desain, heterogenitas
yang melekat di Internet . Populasi membuat mengendalikan karakteristik subjek di
kelompok perlakuan yang berbeda sulit. Proses pengacakan dapat meringankan
kekhawatiran ini untuk sebagian besar. Namun, tes ketahanan berdasarkan pada
mata pelajaran offline juga akan sangat membantu. Kami eksperimen saat fokus
pada jenis konsumen yang dapat membuat, paling banyak, satu pembelian secara
online di setiap sesi belanja. Untuk lebih memahami berlawanan menemukan bahwa
peringkat pribadi yang aktif. Mekanisme mengarah untuk menurunkan tingkat
konversi, satu dapat memperpanjang desain eksperimental kami untuk membuat
perbandingan dengan konsumen yang diperbolehkan untuk membuat beberapa pembelian
dalam sesi yang diberikan. Selain itu, studi bagaimana isi pencarian konsumen,
seperti panjang dan jenis pencarian kata kunci, berinteraksi dengan efek
peringkat akan menarik. Selain itu, dengan Berkenaan dengan memeriksa mekanisme
peringkat, satu kaleng memperluas cakupan penelitian dengan memperhatikan
konsumen akun ' jaringan sosial tetangga 'pencarian dan pembelian perilaku.
Ekspansi ini akan memungkinkan seseorang untuk menguji dampak sosial-sinyal
berbasis mekanisme peringkat di search engine produk. Meskipun keterbatasan
ini, kami percaya kami kertas membuka jalan untuk penelitian masa depan di
daerah ini menarik di persimpangan media sosial dan mesin pencari.
Tidak ada komentar:
Posting Komentar