Has Big Data Reached Its Moment of Disillusionment?
One source of that disillusionment, she writes, is that companies are struggling with a basic problem: What questions do you attempt to answer with your data in the first place? “Several days ago, a financial industry client told me that framing a right question to express a game-changing idea is extremely challenging,” Sicular wrote. “First, selecting a question from multiple candidates; second, breaking it down to many sub-questions; and, third, answering even one of them reliably. It is hard.”
Hadoop doesn’t exactly make that process any easier. Once you’ve decided to use it, getting anything useful out of it requires some pretty specialized knowledge and training, and finding the right people to do that isn’t easy. But the industry is beginning to respond to that need: Startups like Mortar Data have sought to make Hadoop more readily accessible to mainstream programmers, while another called Trifacta makes the resulting data easier to manipulate.
And versions of Hadoop itself are getting incrementally easier to work with.