Data Anxiety

Tempus fugit


A Letter to Journalists from a Former FCC Commissioner »

'“What a totally awesome job this is going to be,” I thought as I sat down at my desk. “I’ll be dealing with edge-of-the-envelope issues that are transforming the planet; I’ll meet the visionaries and innovators who are making it happen…And broadband, the savants told us, would bring the revolutionary wonders of the Internet to every home and hamlet. The new media of the Internet would complement the traditional media of newspapers, radio, TV and cable, ushering in a golden age of communications. News and information journalism would flourish, and America’s civic dialogue…would be nourished as never before.

My expectations were short-lived. It turned out that the FCC I was joining had an altogether different agenda. One of the first requests that I received from my new Chairman was to support a merger between two media companies…my waking hours would be spent listening to big media types tell me how their latest proposal to gobble up more properties would translate into enormous “efficiencies” and “economies of scale” to produce more and better news…everywhere I looked, I saw newsrooms being shuttered or drastically downsized, reporters getting the axe, and investigative journalism clinging to the slenderest of threads.’

In order to maximize profits and to finance their costly media transactions, the merged companies were under the financial gun to cut costs. The first place they looked to cut was, and is, the newsroom. Instead of expanding news and creating opportunities for journalists like you, they cut the muscle out of deep-dive reporting and disinvested in you and your future.

Then another light bulb went on: The public policy the FCC was making was a major force refashioning our media ecosystem. It wasn’t just the excesses of a Wall Street bazaar run wild. It wasn’t just private sector business plans wreaking all this havoc. It was proactive government policy-making. Government—my own agency—was the willing, indeed eager, accomplice in diminishing our news and disfiguring our journalism. The regulatory agency where I worked was actually making things worse. 

You need to know this story. 

To be continued…

“It should be illegal to publish poll numbers.”

So said Matt Taibbi in his assessment of how speculation about the outcome of the election eclipsed the event itself. His suggested remedy:

Banning poll numbers would force the media to actually cover the issues. As it stands now, the horse race is the entire story.

Obviously this was written awhile ago, on 9 October 2012. It doesn’t diminish the relevancy though. Taibbi continues by excoriating himself and his colleagues in journalism, but we can skip that. Journalists give people what they want, or what they are most interested in finding out.

I’ve noticed this behavior myself, regarding the election, and seeming obsession with forecasting the outcome while in progress. It definitely resembles a horse race, or football game, some sort of betting-related activity. There is competition at the center, with a vast audience of onlookers. Relatively few are direct stakeholders in the outcome, though. In some sense, of course, we all are, even external to the U.S.A.

The Future Journalism Project, from whom I reblogged the first portion of this post. continues on, and demonstrates quite thoroughly how Taibbi’s point is substantiated by findings from The Pew Research Center’s recent report, Winning the Media Campaign 2012. I suggest reading the original post by FJR for further details. I didn’t need much convincing about this. Personally, I found the election coverage very boring for the exact reasons highlighted, that there was an over-emphasis on the process rather than coverage of the underlying candidate stances and consequences.

The intense focus on Nate Silver and statistical forecasting using predictive analysis was not especially interesting to me either. It just seemed like more of the same.

Voting treemap courtesy of Wikipedia

Big Data Integrity?

There are numerous ethical questions pertaining to the use of behavioral data for predictive purposes, and certainly for influencing behavior. The over-emphasis on predicting the electoral results is problematic for the reason that polls might actually influence voters. Just as troubling or more so is the material in a new, three part series by MIT’s Technology Review, which covers the extreme granularity of the candidates’ efforts to influence voters:

the President’s team used Big Data and sophisticated analytics at an almost unprecedented scale to track voters, and nudge them in the direction that the Obama team wanted them to go… It essentially created a cohort-analysis system of data to judge every single voter it wanted to get to the polls. Obama’s team took the usual system of analytics and reduced it to the most granular level: the individual voter.

Remember, though, that while one can develop a system with this intended level of precision, it doesn’t necessarily mean that it is effective. Nevertheless, the intent was there. Given time and a few more elections, I suspect the process will be fine-tuned.

Predictive data and the 2012 U.S. Presidential election

To be fair, this effort was overwhelmingly implemented by the campaign for re-election of President Obama. As covered extensively elsewhere, the Romney campaign was not particularly successful in its efforts to use sophisticated analytic methods. According to yesterday’s Read Write Web article, about the use of text analysis and predictive models to influence voter behavior :
In contrast, the Romney campaign was still in an earlier mode of data analytics, focused around larger cohorts such as campaign topics (the fall of Obama-backed solar energy provider Solyndra, for instance) and how individual ads affected the voter mindset.

Ars Technica provided very detailed coverage of the shortcomings of the Romney campaign effort.

Troubling if wrong, troubling if right

I find this use of predictive analysis and statistical forecasting to be unsettling. First, I am not certain that it is as effective as some seem to believe:
The Obama team figured out what type of person a voter was and how that person would respond to certain types of stimuli - such as direct mail, person-to-person interviews, social media, advertising, and so on. Obama was then able to deploy his massive volunteer network (some 500,000 people) and other campaign resources as needed.

Second, I have reservations about using technology for these purposes. By way of analogy: Consider the effect that algorithmic and automated programs have had on financial markets, especially the effect of high frequency trading.

A deterministic statistical machine »


… the most recent batch of Y Combinator startups included a bunch of data-focused companies. One of these companies, StatWing, is a web-based tool for data analysis that looks like an improvement on SPSS with more plain text, more visualization, and…

SimplyStatistics conceives of a way to keep would-be data analysts honest, despite increasingly easy-to-use (and abuse) statistical software applications.

Soi-dissant tech journalists: Stop hyping unproven security tools »


“Tech journalists: Stop hyping unproven security tools”

Journalists love human interest stories

Kobeissi presents the kind of human interest story that journalists dream about: A Lebanese hacker who has lived through 4 wars in his 21 years…

This is about Cryptocat.

Cryptocat was here on tumblr, might still be. They were friendly, enjoyed math, probability, random number generator humor. I followed young (and cute) Kobeissi on Twitter.

Cryptocat is a proto-type. Read Mr. Kobeissi’s biography. He is bright. He is getting a fine education. I think he just completed a bachelor’s degree program in computer science at a good school in Canada, and has received funding from someone to do further work on Cryptocat. No problem there.

However, Wired Magazine, AND The New York Times, are not behaving responsibly. In my sometimes-not-so-humble opinion.

The New York Times really, REALLY needs to hire (or have on retainer) someone with credible training, education and experience in information security if they intend to cover  cryptography-related news.

As for Wired, well, I just don’t understand what they were thinking.

The Cryptocat media hyperbole reminds me of TechCrunch at its worst, e.g. when one of their staff attempted to do financial reporting or investigative journalism, but didn’t have the knowledge or time to do a good job. TechCrunch is a glorified blog though. They make mistakes, and that’s okay. They aren’t the Paper of Record. They aren’t Wired Magazine, which has been a decent specialty publication for over two decades.

What’s going on? A possibility: Business problems e.g. diminished advertising revenues and an increasingly desperate need for page views and circulation volume. I worry about that, for all mainstream news sources.

“FT says banking is so thoroughly corrupted that the only way to improve the ethics of banking may be to put the entire current generation out to pasture. I’d settle for stuffing them in jail.”

dafowc: Shaming the banks into better ways - Financial Times

The Financial Times are fine ones to talk, those handmaidens to capitalism! 

The best part was Gaius Marius’s comment. I decided to bequeath quote status upon it.

Using YNET as a source... »

Using YNET as a source… against Palestinians gives an impression that you don’t think. YNET is a ZIONIST owned news source, so they NEVER TELL THE TRUTH.

via the-gulf-is-persian

Zionist sources never tell the truth?

So any source that believes that Israel shouldn’t be eradicated is lying about everything?

Now I’m sure you believe that anti-Zionist sources provide the most honest and unbiased information possible. Essentially, anyone who says something that you don’t like must be lying. Truly, who could argue with that?

One quick note though: YNet actually leans a bit left, even if it’s not as far as Haaretz. Even so, it does a pretty good job (in my opinion) of providing accurate news, though others might disagree with me.

But anyway, if you would like to use some proof to back up your assertions, I’d be more inclined to take them more seriously.

via eretzyisrael

One way to prove whether YNet is deceptive would be to refute the news stories they report as factual. That’s the best approach, and won’t necessarily be perceived antagonistically either.

Everyone on the internet enjoys unearthing the REAL story!

  • Sometimes it is impossible, as there are insufficient facts available.
  • Sometimes the facts are not palatable to all. 

Generally, a challenge to a specific instance, backed with factual counter evidence is the best way of establishing a better rapport. This is the case, even if one party, either one, turns out to be incorrect.

Respect grows, for the future.

Otherwise known as plagiarism »

Simply Statistics:

I would like to define a new term: Reverse scooping is when someone publishes your idea after you, and doesn’t cite you. It has happened to me a few times. What does one do?

I usually send a polite message to the authors with a link to my related paper(s). These emails are usually ignored, but…

In my opinion, the trio of mild-mannered Ph.D.’s running the SimplyStatistics tumblr are far too easy-going about this. I am full of (vicarious) righteous indignation! But they may be more sensible than I realize, from a long-term point of view. Still… it seems so unfair.

“… during difficult economic times, many people misunderstand capitalism, and rationalize “a do what it takes” mindset that is ultimately self-defeating. We need the SEC, the EPA, transparent operations, a free press that cares about its mission and people willing and able to speak up [because] they make it expensive to choose the short-term option.”

Seth Godin, Short-term capitalism via wka

Emphasis on "the SEC, the EPA, transparent operations, a free press": Yes! Believe, or govern, as you wish. But as long as those basics are active, enforced consistently, without bias, much of the rest will fall into place.

* There is no shortage of people willing and able to speak up. Unfortunately, it is hard to tell what part to listen to sometimes… Be that as it may, it is preferable to the alternative.

Les livres du gouvernement des roys et des princes, W.144, folio 41v by Walters Art Museum Illuminated Manuscripts, on Flickr.

This English manuscript is an example of Henri de Gauchy’s French translation of De regimine principum. Giles of Rome first composed the text in Latin around 1277…The text is divided into three books, intended to instruct a prince on his ethical, economic and political responsibilities: the conduct of the individual, the rule of the family and household, and the governance of the kingdom.

Les livres du gouvernement des roys et des princes, W.144, folio 41v by Walters Art Museum Illuminated Manuscripts, on Flickr.

This English manuscript is an example of Henri de Gauchy’s French translation of De regimine principum. Giles of Rome first composed the text in Latin around 1277…

The text is divided into three books, intended to instruct a prince on his ethical, economic and political responsibilities: the conduct of the individual, the rule of the family and household, and the governance of the kingdom.