Gerd Gigerenzer discusses how search engines use big data analytics to “diagnose” your state of health
Imagine this warning popping up on your search engine page: “Attention! There are signs that you might have pancreatic cancer. Please visit your doctor immediately.” Just as search engines use big data analytics to detect your book and music preferences, they may also “diagnose” your state of health.
Microsoft researchers have claimed that web search queries could predict pancreatic adenocarcinoma. [1] A retrospective study of 6.4 million users of Microsoft’s search engine Bing identified first-person queries suggestive of a recent diagnosis, such as “I was told I have pancreatic cancer, what to expect.” Then the researchers went back months before these queries were made and looked for earlier ones indicating symptoms or risk factors, such as blood clots and unexplained weight loss. They concluded that their statistical classifiers “can identify 5% to 15% of cases, while preserving extremely low false-positive rates (0.00001 to 0.0001)”, and that “this screening capability could increase 5-year survival.” The New York Times reported: “The study suggests that early screening can increase the five-year survival rate of pancreatic patients to 5 to 7 percent, from just 3 percent.” [2]
Thus, it appears that Microsoft researchers have found a low-cost, high-coverage surveillance system that produces almost no false alarms and saves lives—an improvement over previous diagnostic attempts using biomarkers or imaging. In this column, I do not deal with typical problems of Big Data, such as intransparent algorithms and the danger of overfitting noise, or with the ethics of not soliciting users’ consent to having their personal data (albeit anonymously) analyzed. Rather, I take the results as given and focus on how the presentation of these results invites several potential misunderstandings.
First, consider the prospect of increased survival rates, which suggests that surveillance saves lives. In fact, in the context of screening, the correlation between increases in survival rates and decreases in mortality rates is approximately zero for the 20 most common solid tumors over the past 50 years. [3] One reason for this is called “lead-time bias.” Early detection implies that diagnosis occurs at an earlier stage of a disease, which leads to higher 5-year survival rates from the time of diagnosis—even if the patients ultimately do not live any longer in terms of absolute age. Reporting survival instead of mortality rates misleads the reader about the benefits of cancer screening. [4]
Second, consider the extremely low false-positive rates. Does that mean that Bing users who get the news that they are positive are seldom falsely alarmed? To answer that, let us go through a simple example. Assume 100,000 users, 10 of whom have undetected pancreatic cancer. [5] Given a sensitivity of 10% (the average of 5% and 15%), we expect that one user correctly tests positive and the other nine are missed. Given a false-positive rate of 1 in 10,000 (or 0.0001), we expect that about 10 users test positive even though they do not have cancer. Thus, we expect a total of 11 users to test positive, of whom 10 do not have pancreatic cancer. The general point is that even with low false-positive rates, the proportion of false alarms among all users who test positive can be high if the disease is rare.
The authors note that surveillance by Bing does not replace a physician. Yet their presentation of the statistical results easily invites systematic misunderstandings by both patients and physicians. In the New York Times, Horvitz mentions his hope that the study will stimulate quite a bit of interesting conversation. My response is that in order to demonstrate the clinical usefulness of Big Data analytics, the first step would be toward more transparency and fewer misleading statistics. Not doing so recalls the rise and fall of Google Flu Trends, which in 2009 was trumpeted as being able to predict influenza but disappeared from sight after years of failing to meet its own projected rates of predictive accuracy. Big Data is known for its fanfare and hype. In this case, all has been quiet since the buzz last summer.
Gerd Gigerenzer, Director, Max Planck Institute for Human Development and Harding Center for Risk Literacy, Berlin.
Competing interests: None declared.
References:
[1] Paparrizos J, White RW & Horvitz E. Screening for pancreatic adenocarcinoma using signals from web search logs: Feasibility study and results J Oncol Pract 2016 Jun 7; pii: JOPR010504.
[2] Markoff J. Microsoft finds cancer clues in search queries. New York Times 2016 Jun 7.
[3] Welch HG, Schwartz LM, Woloshin S. Are increasing 5-year survival rates evidence of success against cancer? JAMA 2000;283:2975-8.
[4] Gigerenzer G. Breast cancer screening pamphlets mislead women. BMJ 2014 Apr 25;348:g2636.
[5] This is the estimate given by Paparrizos; see https://www.cs.columbia.edu/2016/web-searches-as-an-early-warning-system-for-pancreatic-cancer/