[Detecting Network Effects: Randomizing Over Randomized Experiments](http://www.youtube.com/watch?v=1v5_CzdRVAc)
Martin Saveski (MIT), Jean Pouget-Abadie (Harvard), Guillaume Saint-Jacques (MIT), Weitao Duan, Souvik Ghosh, Ya Xu (LinkedIn), Edo Airoldi (Harvard)
Randomized experiments, or A/B tests, are the standard approach for evaluating the causal effects of new product features, i.e., treatments. The validity of these tests rests on the “stable unit treatment value assumption” (SUTVA), which implies that the treatment only affects the behavior of treated users, and does not affect the behavior of others. Violations of SUTVA, common in features that exhibit network effects, result in inaccurate estimates of the causal effect of treatment. In this paper, we leverage a new experimental design for testing whether SUTVA holds, without making any assumptions on how treatment effects may spill over between the treatment and the control group. To achieve this, we simultaneously run both a completely randomized and a cluster-based randomized experiment, and then we compare the difference of the resulting estimates. We present a statistical test for measuring the significance of this difference and other theoretical bounds on the Type I error rate. We provide practical guidelines for implementing our methodology on large-scale experimentation platforms. Importantly, the proposed methodology can be applied to settings in which a network is not necessarily observed but, if available, can be used in the analysis. Finally, we deploy this design to LinkedIn’s experimentation platform and apply it to two online experiments, highlighting the presence of network effects and bias in standard A/B testing approaches in a real-world setting.
Martin Saveski (MIT), Jean Pouget-Abadie (Harvard), Guillaume Saint-Jacques (MIT), Weitao Duan, Souvik Ghosh, Ya Xu (LinkedIn), Edo Airoldi (Harvard)
Randomized experiments, or A/B tests, are the standard approach for evaluating the causal effects of new product features, i.e., treatments. The validity of these tests rests on the “stable unit treatment value assumption” (SUTVA), which implies that the treatment only affects the behavior of treated users, and does not affect the behavior of others. Violations of SUTVA, common in features that exhibit network effects, result in inaccurate estimates of the causal effect of treatment. In this paper, we leverage a new experimental design for testing whether SUTVA holds, without making any assumptions on how treatment effects may spill over between the treatment and the control group. To achieve this, we simultaneously run both a completely randomized and a cluster-based randomized experiment, and then we compare the difference of the resulting estimates. We present a statistical test for measuring the significance of this difference and other theoretical bounds on the Type I error rate. We provide practical guidelines for implementing our methodology on large-scale experimentation platforms. Importantly, the proposed methodology can be applied to settings in which a network is not necessarily observed but, if available, can be used in the analysis. Finally, we deploy this design to LinkedIn’s experimentation platform and apply it to two online experiments, highlighting the presence of network effects and bias in standard A/B testing approaches in a real-world setting.
YouTube
Detecting Network Effects: Randomizing Over Randomized Experiments
Detecting Network Effects: Randomizing Over Randomized Experiments
Martin Saveski (MIT)
Jean Pouget-Abadie (Harvard University)
Guillaume Saint-Jacques (MIT)
Weitao Duan (LinkedIn)
Souvik Ghosh (LinkedIn)
Ya Xu (LinkedIn)
Edo Airoldi (Harvard University)…
Martin Saveski (MIT)
Jean Pouget-Abadie (Harvard University)
Guillaume Saint-Jacques (MIT)
Weitao Duan (LinkedIn)
Souvik Ghosh (LinkedIn)
Ya Xu (LinkedIn)
Edo Airoldi (Harvard University)…
Визуализация больших графов для самых маленьких – Святослав Ковалев
https://www.youtube.com/watch?v=SjO_UyRgvlE
Визуализация и анализ комментариев на ютубе – Антон Костин
https://www.youtube.com/watch?v=wn9N82ut1ZAъ
Граф знаний для поиска: построение и использование – Дмитрий Ильвохин
https://www.youtube.com/watch?v=fgyw_j6qPSI
Ростислав Яворский, Высшая Школа Экономики, «Как использовать анализ сетевых данных для управленческих решений». Выступление Ростислава состоит из двух частей: «Визуализация реальной структуры организации» и «Анализ и визуализация профессиональных сообществ».
https://www.slideshare.net/MailRuGroup/ss-59828544
Анализ социальных сетей в телекоме — Александр Семёнов
https://www.youtube.com/watch?v=wuii1EOOhaY
Анализ данных в социальных сетях — Дмитрий Бугайченко
https://www.youtube.com/watch?v=FMoFg9pikWE
https://www.slideshare.net/MailRuGroup/ss-59828596
https://www.youtube.com/watch?v=SjO_UyRgvlE
Визуализация и анализ комментариев на ютубе – Антон Костин
https://www.youtube.com/watch?v=wn9N82ut1ZAъ
Граф знаний для поиска: построение и использование – Дмитрий Ильвохин
https://www.youtube.com/watch?v=fgyw_j6qPSI
Ростислав Яворский, Высшая Школа Экономики, «Как использовать анализ сетевых данных для управленческих решений». Выступление Ростислава состоит из двух частей: «Визуализация реальной структуры организации» и «Анализ и визуализация профессиональных сообществ».
https://www.slideshare.net/MailRuGroup/ss-59828544
Анализ социальных сетей в телекоме — Александр Семёнов
https://www.youtube.com/watch?v=wuii1EOOhaY
Анализ данных в социальных сетях — Дмитрий Бугайченко
https://www.youtube.com/watch?v=FMoFg9pikWE
https://www.slideshare.net/MailRuGroup/ss-59828596
YouTube
Визуализация больших графов для самых маленьких – Святослав Ковалев
Секция EDA & visualisation – Community lab stage, 11 мая 2019
Презентации с Data Fest 6 – https://drive.google.com/open?id=1LOmOoh1WLqmhSqTKjvdOQx-YOTyBgG-i
Презентации с Data Fest 6 – https://drive.google.com/open?id=1LOmOoh1WLqmhSqTKjvdOQx-YOTyBgG-i
Community Identity and User Engagement in a Multi-Community Landscape
https://cs.stanford.edu/people/jure/pubs/identity-icwsm17.pdf
https://cs.stanford.edu/people/jure/pubs/identity-icwsm17.pdf
Число Данбара — ограничение на количество постоянных социальных связей, которые человек может поддерживать.
Поддержание таких связей предполагает знание отличительных черт индивида, его характера, а также социального положения, что требует значительных интеллектуальных способностей. Лежит в диапазоне от 100 до 230, чаще всего считается равным 150.
https://en.wikipedia.org/wiki/Dunbar%27s_number
Поддержание таких связей предполагает знание отличительных черт индивида, его характера, а также социального положения, что требует значительных интеллектуальных способностей. Лежит в диапазоне от 100 до 230, чаще всего считается равным 150.
https://en.wikipedia.org/wiki/Dunbar%27s_number
Концепция силы слабых связей — это концепция американского социолога Марка Грановеттера, согласно которой в межличностной коммуникации слабые связи имеют большее значение, чем сильные. Отражена в статье Грановеттера «Сила слабых связей», наиболее известной его работе.
https://en.wikipedia.org/wiki/Interpersonal_ties
https://en.wikipedia.org/wiki/Interpersonal_ties
On the Structural Properties of Massive Telecom Call
Graphs: Findings and Implications
https://ebiquity.umbc.edu/_file_directory_/papers/452.pdf
Social Ties and their Relevance to Churn in Mobile
Telecom Networks
https://openproceedings.org/2008/conf/edbt/DasguptaSVCMNJ08.pdf
Calling patterns in human communication dynamics
https://arxiv.org/abs/1301.7173
Graphs: Findings and Implications
https://ebiquity.umbc.edu/_file_directory_/papers/452.pdf
Social Ties and their Relevance to Churn in Mobile
Telecom Networks
https://openproceedings.org/2008/conf/edbt/DasguptaSVCMNJ08.pdf
Calling patterns in human communication dynamics
https://arxiv.org/abs/1301.7173
Analysis of Network Clustering Algorithms and Cluster Quality Metrics at Scale
Louvain
Smart Local Moving (SLM)
Infomap
Label Propagation
Affinity Propagation
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0159161
Louvain
Smart Local Moving (SLM)
Infomap
Label Propagation
Affinity Propagation
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0159161
journals.plos.org
Analysis of Network Clustering Algorithms and Cluster Quality Metrics at Scale
Overview Notions of community quality underlie the clustering of networks. While studies surrounding network clustering are increasingly common, a precise understanding of the realtionship between different cluster quality metrics is unknown. In this paper…
The Matthew effect, Matthew principle, or Matthew effect of accumulated advantage can be observed in many aspects of life and fields of activity. It is sometimes summarized by the adage "the rich get richer and the poor get poorer".
https://en.wikipedia.org/wiki/Matthew_effect
https://en.wikipedia.org/wiki/Matthew_effect