Tuesday, November 07, 2017

My Thoughts on a Japan Times Poll Cited by a Journalist

I believed that it was highly likely that The Japan Times’s survey was skewed against Abe because its readership, from which the sample was drawn, was skewed against. I thought that point was obvious when I used the Boston sports bar simile. After all, it’s the bar’s patrons that you interview, not the (inanimate) bar itself; The stand-in for The Japan Times was the location, the city of Boston. (A Boston sports bar could, of course, have plenty of Red Sox memorabilia and the like and the bartender would likely be a Bill Simmons type, which would sure have an effect on the tendencies of its tendencies—but I digress.)   And I think that disinterested parties will agree that David McNeill conceded as much as far as the past was concerned when he tweeted, “The JT is under new management, doncha know?”

Ironically, I’m the one who’s not so sure now. I had assumed that the responders to the online survey would reflect years of getting their news on Abe and other Japanese conservatives from the pages of JT. But nothing should have been further from the truth. After all, the poll was supposed to reflect the views of Japanese voters. If my casual observation on trains is any indication, Japanese readers of JT are usually students or salaryhumans intent on improving their English, not necessarily the most politically-motivated segment of the Japanese public.

But that’s where my doubts about the sampling process comes in. I did not write “300-plus” casually. I remembered the number from the unusually hard to find JT article as being in the high 300s. This is not insignificant. Reputable polls typically use samples of 300, 500, and, what, 2000? 3000? Anyway, a sample in the high 300s was anomalous, and statistically irrelevant—this layman’s understanding is that the gain from a randomized sample of 500 compared to 300 is fairly small. Thus, I suspected—still do—that the sampling process was self-selective, which suggested a sample skewed towards the highly-motivated with a cosmopolitan outlook.

Of course, there’s no way of knowing that unless we know how the survey was conducted. Heck, we don’t even know if the sample consisted largely of Japanese nationals. So, I went and looked, but to my consternation could not find the JT article. No, I could not find it again, but decided to tweet anyway, working from memory. Note, though, that raising the number to the 600s does not alleviate my doubts at all. If anything, it exacerbates it.

And here’s where the responsibility of the journalist comes in. When a journalist uses a poll number to illustrate a point, he/she’d better be sure of its provenance, particularly when it is conducted by a media outlet whose resources/capability to conduct a robust survey may be doubtful. It would dangerous to cite such a poll without confirming how it was conducted and preparing to answer questions about its veracity.

That is why I find the fact that the comment “Feel free to cite your own survey on high support for Abe-Trump bromance” is coming from a journalist rather disturbing. As people who know my thinking knows well, I believe that one difference between scientists and journalists (and pundits, if you insist) is that the former uses evidence to test hypotheses and the latter uses it to illustrate narratives. This is not a knock against journalists, investigative journalists included. It’s an acknowledgement of the multitude of constraints that journalists face when filing reports for the public. And it’s not as if there isn’t a huge gray area in between that is inhabited by practitioners of the soft sciences. But it does highlight the need for conscientious journalists to be meticulous in weighing their evidence from all feasible angles. An invitation to pick a survey of choice is an invitation of adopting standards befitting more of a, say, Fox and Friends or, if that example is not to your liking, SNL (I know, I know).

Finally, for what it’s worth, my interest is not in whether a given poll is to my liking—though it feels good when it does—but in questions like, “Why does the LDP and its administration do better on Yomiuri polls than on Asahi polls—except when it doesn’t?” The real money, by the way, is in the second part of the question. Now, let’s see if McNeill really meant it when he wrote, “Always pleasantly surprised by intelligent Twitter exchanges.”

Sidebar: I’ve noticed that people rarely acknowledge their mistakes on Twitter. Instead, they usually just go away, at which point I’ve decided I’ll just declare victory and move on. Even a “not necessarily” without commentary is better than par for the course, doncha think?

(Typos corrected, same day.)