Data Anxiety

Tempus fugit

About the Hype Cycle »

nosql:

via Steve Francia on the NoSQL hype cycle[1]:

Technology trigger

As Facebook, Twitter and others saw them as solutions to their massive scalability problem (and because they were using relational databases for things they shouldn’t have) people began to see NoSQL as a panacea.

Peak of inflated expectations

Unfortunately knowing when to use the technology requires actual experience with it, which never seems to catch up to the hype engine quickly enough, so the technology transforms into a panacea, i.e. better at everything and ready to displace all predecessors.

Trough of Disillusionment

Current technologies exist because they do something well. When a new technology emerges it will likely be good at a different thing, meaning the two will co-exist.

hype cycle

Generalizable! This is a much-loved favorite.

Welcome to Tumblr, ODNI General Counsel Robert Litt! We are delighted that you have decided to join us. Your Tumblr HTML and CSS choices are very appropriate, although I would personally prefer a heavier typeface or more contrast, for the sake of readability.
On the off chance that you or the minions notice, I have a few suggestions. I have taken the liberty of excerpting from your speech, then sharing my thoughts. There’s a lot to be said for responsive reading, in the style of the Hebrew Union prayer book, like I used back home in New Mexico. Okay, here we go!
icontherecord:

Remarks of ODNI General Counsel Robert Litt at American University Washington College of Law Freedom of Information Day Celebration, March 17, 2004
President Obama, who has noted our “profound national commitment to ensuring an open government,” called upon the entire government to comply with both the letter and the spirit of the Freedom of Information Act.  Today I would like to talk to you about the challenges of reconciling that commitment with the secrecy necessary to conduct effective intelligence operations in defense of our national interests… We’ve set up a tumblr website – not that I really understand what a tumblr is – on which we post for the public what we have released. The President has directed that this transparency continue, and it will. This is something that I am personally involved in and personally committed to…

Stop reiterating your commitment to transparency. You are setting yourself up for failure and diminished public confidence. “Open” has become the subject of derision. 
There are many exemplary sources of government data, in human and machine digestible format. The Department of Defense candidly offers two decades of earmark history. The Federal Reserve maintains a host of numeric, geographical and historical data portals for public perusal and download: FRED, GeoFRED, ALFRED.
Open data market: Saturated with excess supply
I have an account on Socrata, the Gov 2.0 open data platform. Socrata is a CIA In-Q-Tel funded venture. It’s great, works really well! There’s fascinating data there, for our citizen journalists clamoring for transparency:
the data set of presidential appointments, linkable to historical political campaign contributions by party, candidate, donor and filing state. 
outlier searches, e.g. why does such-and-such bureau only employ attorneys, with average compensation of $185,000 per annum? What is all this foreign exchange activity going on at Dept. of the REDACTED? 
Wow, the annual budget for the National Endowment for the Arts was cut by nearly 40% in 1994, and again in 2014!
However, hardly a soul uses it. Data usage statistics are available. They are grim. A fine young data scientist, Thomas Levine, does periodic reports, and his findings confirm this. There are only 38 active citizen users of Socrata open data (Thomas’s post even includes bio’s with photographs of the top 3 most active users). All others are U.S. federal government and state agency staff, Socrata employees or bots. The bots all belong to recovery.gov or data.gov, with one exception, Kenya Open Data Bot.
You are TOO transparent!
Well, you’re transparent about the wrong things.

In particular, I think that greater transparency about our processes could help us cope with one of the principal failings I see in the current public discussion, which is the failure to distinguish among what the Intelligence Community can do technically, what it can do legally and what it actually does do.

You mentioned that the recent flood of FOIA requests were costing the NSA a small fortune. Maybe you could get some of the funding for Socrata reallocated to your FOIA group? It is atavistic, but it seems to be what the public truly wants.
Don’t be overwhelmed by technology
Consult with well-informed individuals, the sort that have training and experience. None of Condoleeza Rice’s and Hillary Clinton’s young Turks such as Jared Cohen or the insufferable ioerror. And stop listening to Harvard about the wonders of privatization! They’ll tell you to give everything to contractors like Booz Allen, USIS and Amazon.com.
Encryption

There have been press reports that NSA has engaged in a concerted effort to break encryption. Without commenting on the accuracy of any particular story, isn’t that exactly what intelligence agencies have historically done and what they are supposed to do?

Yes.

We know that our enemies use encryption and other techniques precisely to avoid surveillance; NSA’s job is to figure out how to break those techniques…But saying that NSA can break encryption is different from saying that they routinely spy on encrypted conversations of ordinary Americans or foreigners. They don’t…

The encryption breaking is problematic because it was in large part developed under the aegis of NIST. NIST is part of the U.S. Department of Commerce. That encryption is used globally, in government and commercial applications. Trust has been undermined. You must address this. It is vital.

And this leads to the other question I want to raise about transparency, which is, Who is responsible for deciding what should be secret and what should be public? In my opinion, we cannot survive as a nation if we let those decisions be made by individuals who do not have adequate insight into the implications of their actions…etcetera

You have many wise, compassionate and yes, friendly, experts available e.g. the Naval War College, FBI, Carnegie Mellon and Cornell Universities, IBM, RSA, Verisign and MITRE. Many of the recent Department of Homeland Security hires have distinctively political appointee-type curriculum vitae. I’d steer clear; same for technocrats. Technocrats are rarely technical, most are just good talkers!

Welcome to Tumblr, ODNI General Counsel Robert Litt! We are delighted that you have decided to join us. Your Tumblr HTML and CSS choices are very appropriate, although I would personally prefer a heavier typeface or more contrast, for the sake of readability.

On the off chance that you or the minions notice, I have a few suggestions. I have taken the liberty of excerpting from your speech, then sharing my thoughts. There’s a lot to be said for responsive reading, in the style of the Hebrew Union prayer book, like I used back home in New Mexico. Okay, here we go!

icontherecord:

Remarks of ODNI General Counsel Robert Litt at American University Washington College of Law Freedom of Information Day Celebration, March 17, 2004

President Obama, who has noted our “profound national commitment to ensuring an open government,” called upon the entire government to comply with both the letter and the spirit of the Freedom of Information Act.  Today I would like to talk to you about the challenges of reconciling that commitment with the secrecy necessary to conduct effective intelligence operations in defense of our national interests… We’ve set up a tumblr website – not that I really understand what a tumblr is – on which we post for the public what we have released. The President has directed that this transparency continue, and it will. This is something that I am personally involved in and personally committed to…

Stop reiterating your commitment to transparency. You are setting yourself up for failure and diminished public confidence. “Open” has become the subject of derision.

There are many exemplary sources of government data, in human and machine digestible format. The Department of Defense candidly offers two decades of earmark history. The Federal Reserve maintains a host of numeric, geographical and historical data portals for public perusal and download: FRED, GeoFRED, ALFRED.

Open data market: Saturated with excess supply

I have an account on Socrata, the Gov 2.0 open data platform. Socrata is a CIA In-Q-Tel funded venture. It’s great, works really well! There’s fascinating data there, for our citizen journalists clamoring for transparency:

  • the data set of presidential appointments, linkable to historical political campaign contributions by party, candidate, donor and filing state.
  • outlier searches, e.g. why does such-and-such bureau only employ attorneys, with average compensation of $185,000 per annum? What is all this foreign exchange activity going on at Dept. of the REDACTED?
  • Wow, the annual budget for the National Endowment for the Arts was cut by nearly 40% in 1994, and again in 2014!

However, hardly a soul uses it. Data usage statistics are available. They are grim. A fine young data scientist, Thomas Levine, does periodic reports, and his findings confirm this. There are only 38 active citizen users of Socrata open data (Thomas’s post even includes bio’s with photographs of the top 3 most active users). All others are U.S. federal government and state agency staff, Socrata employees or bots. The bots all belong to recovery.gov or data.gov, with one exception, Kenya Open Data Bot.

You are TOO transparent!

Well, you’re transparent about the wrong things.

In particular, I think that greater transparency about our processes could help us cope with one of the principal failings I see in the current public discussion, which is the failure to distinguish among what the Intelligence Community can do technically, what it can do legally and what it actually does do.

You mentioned that the recent flood of FOIA requests were costing the NSA a small fortune. Maybe you could get some of the funding for Socrata reallocated to your FOIA group? It is atavistic, but it seems to be what the public truly wants.

Don’t be overwhelmed by technology

Consult with well-informed individuals, the sort that have training and experience. None of Condoleeza Rice’s and Hillary Clinton’s young Turks such as Jared Cohen or the insufferable ioerror. And stop listening to Harvard about the wonders of privatization! They’ll tell you to give everything to contractors like Booz Allen, USIS and Amazon.com.

Encryption

There have been press reports that NSA has engaged in a concerted effort to break encryption. Without commenting on the accuracy of any particular story, isn’t that exactly what intelligence agencies have historically done and what they are supposed to do?

Yes.

We know that our enemies use encryption and other techniques precisely to avoid surveillance; NSA’s job is to figure out how to break those techniques…But saying that NSA can break encryption is different from saying that they routinely spy on encrypted conversations of ordinary Americans or foreigners. They don’t…

The encryption breaking is problematic because it was in large part developed under the aegis of NIST. NIST is part of the U.S. Department of Commerce. That encryption is used globally, in government and commercial applications. Trust has been undermined. You must address this. It is vital.

And this leads to the other question I want to raise about transparency, which is, Who is responsible for deciding what should be secret and what should be public? In my opinion, we cannot survive as a nation if we let those decisions be made by individuals who do not have adequate insight into the implications of their actions…etcetera

You have many wise, compassionate and yes, friendly, experts available e.g. the Naval War College, FBI, Carnegie Mellon and Cornell Universities, IBM, RSA, Verisign and MITRE. Many of the recent Department of Homeland Security hires have distinctively political appointee-type curriculum vitae. I’d steer clear; same for technocrats. Technocrats are rarely technical, most are just good talkers!

Consider this to be a very ad hoc comparative geography post. Above, based on population, we have seven Ukraines per USA. Since there is no legend, I would guess that we have a straightforward three color map. The common unit is population, not land area.
Source: thelandofmaps:

Population distribution of the United States, using the unit of “Ukraines”

I just happened to have a similar map* handy, that was based on Canada. Using the common metric of population, there are 10 Canada’s per USA.
Next, you could play around with number of Ukraines per Canada, or number of (Ukraines + Canada’s) per USA. 

* Original source: io9 
Without knowing more about the data sources, e.g. census year and survey methodology used for each of the three countries, it wouldn’t be too meaningful. If you have any insights, please feel free to share!
Sharing insights might be easier said than done. I seem to have lost my Disqus comments recently. If you happen to find them, might you tell me where they are, so I can get them back?

Consider this to be a very ad hoc comparative geography post. Above, based on population, we have seven Ukraines per USA. Since there is no legend, I would guess that we have a straightforward three color map. The common unit is population, not land area.

Source: thelandofmaps:

Population distribution of the United States, using the unit of “Ukraines”

I just happened to have a similar map* handy, that was based on Canada. Using the common metric of population, there are 10 Canada’s per USA.

Next, you could play around with number of Ukraines per Canada, or number of (Ukraines + Canada’s) per USA. 

Population distribution of the United States, measured in Canada's

* Original source: io9 

Without knowing more about the data sources, e.g. census year and survey methodology used for each of the three countries, it wouldn’t be too meaningful. If you have any insights, please feel free to share!

Sharing insights might be easier said than done. I seem to have lost my Disqus comments recently. If you happen to find them, might you tell me where they are, so I can get them back?

cityoflasvegas:

Yesterday, we celebrated the dedication of the latest addition to our public art collection, in the first-floor lobby… of the Development Services Center… The artwork, composed of fabric, thread and steel, measures 30 feet by 8.5 feet by 24 feet is entitled “Nephele”.  Las Vegas artist Cyd Bown [says]:
"I am aware of Nephele in Greek Mythology, and although the artwork wasn’t necessarily created or conceived with her story in mind I appreciate her mythos…"

Νεφελη 

When to use Greek entities

This entity set contains all the letters used in modern Greek. However, it does not include Greek punctuation, precomposed accented characters nor the non-spacing accents, tonos or dialytika… There are no archaic letters, Coptic-unique letters, or precomposed letters for Polytonic Greek.


The Characters

Typical invocation

cityoflasvegas:

Yesterday, we celebrated the dedication of the latest addition to our public art collection, in the first-floor lobby… of the Development Services Center… The artwork, composed of fabric, thread and steel, measures 30 feet by 8.5 feet by 24 feet is entitled “Nephele”.  Las Vegas artist Cyd Bown [says]:

"I am aware of Nephele in Greek Mythology, and although the artwork wasn’t necessarily created or conceived with her story in mind I appreciate her mythos…"

Νεφελη 

When to use Greek entities

This entity set contains all the letters used in modern Greek. However, it does not include Greek punctuation, precomposed accented characters nor the non-spacing accents, tonos or dialytika… There are no archaic letters, Coptic-unique letters, or precomposed letters for Polytonic Greek.

The Characters

Typical invocation