Nimsoft is the company who acquired Watchmouse, a long-time favorite of mine for performance monitoring, security, probably more. Nimsoft is probably best characterized as a comprehensive cloud computing provider. I read a post on their company blog, about the intense focus on big data:
This technology enables real time analysis of social network information… it is all about mining the trillions of trivial postings that we collectively make every second on the social networks of our choice. I know that “trivial” is harsh.
Why such interest in it then?
Mining and refining trivia can be used in innovative ways. Mostly it is about targeted advertising, but it is also about trend recognition and the analysis of collective thinking.
Sometimes the analysis of collective thinking is intriguing. Google’s Social Collider is one example. Another is Cultoromics, which was associated with development of the N-Gram Viewer. That was great. But such work also has a tendency to let questionable ideas roam farther than they would otherwise, particularly when they benefit from the veneer of faux-analytics.
Why else is big data so compelling? Well, there is technological challenge too.
What we have been using so far is inadequate for this job. With classical technology, and particularly SQL based databases, retrieval performance degrades exponentially with volume. Even the concept of “collect, store and analyze” has to be rethought. Now it is more like “collect, cache, analyze, store result”. To do that in real time with a variable and unpredictable arrival rate of data requires massive parallelism and efficiency of execution. It reminds me of the early days of computing when data storage structures were designed for performance and code execution times were measured and constantly optimized.
Technology innovators race to produce the fastest, most efficient, and most linear performance profile analytical tool. Are they doing this in order to accomplish anything productive? Sentiment analysis and the zeitgeist and the living pulse of our collective psyche, desires and dreams is cool to contemplate. Beyond that… I don’t have a clear vision beyond that point.
Post-script
Consider too this article about venture capital funding pouring into big data companies (ComputerWorld). Some of these companies are neither start-up’s, nor particularly innovative in storage or processing of enormous data sets:
Curt Monash warned investors to beware of the hype surrounding the technology. “A great example of hype is anybody calling Birst a ‘big data’ or ‘big data analytics’ company,” he said.
A prior Computerworld article described how Birst recently received $26 million in funding from Sequoia Capital and others, and has raised a rather hefty $46 million overall. Yet Birst went into business back in 2005, as a cloud-based business intelligence service. It has only recently begun presenting its products and software as a tool set for analyzing and deriving deeper meaning from petabyte-scale data sets.
As Curt Monash says:
“If anything, Birst is a ‘little data’ analytics company that claims, as a differentiating feature, that it can handle ordinary-sized data sets as well.”
