Pitfalls And Promises Of The Big Data Market For Solution Providers

Focus on big data
Focus on big data

Twenty-nine percent said they were still in the proof-of-concept stage. Nineteen percent had budgeted for a project and identified focus areas, but had yet to begin implementation. And 5 percent had no budget allocation for a big data project.

Those numbers, of course, mean there are lots of opportunities for solution providers to work with clients to get new big data projects off the ground or help with early-stage projects.

But there are some sobering numbers in the survey findings. Only 27 percent describe their big data initiatives as "successful" and only 8 percent described them as "very successful." Translation: Solution providers planning on getting into the big data business better know what they are doing.

The report delves into the reasons big data initiatives fail to meet expectations and offers solution providers guidance about what they can do to ensure that big data projects succeed.

The biggest challenge for big data projects is the lack of data integration across a business or organization. Seventy-nine percent of all organizations have not completely integrated their data sources, according to the report, and only 35 percent have robust processes for data capture, curation, validation and retention. So it's no surprise that 46 percent of survey respondents cited data scattered in silos across an organization as a hurdle for their big data projects.

The lack of a clear business case for funding and implementing a big data system was cited by 39 percent of those surveyed as a major hurdle to success. And 67 percent did not have well-defined criteria to measure the success of their big data initiatives.

Thirty-five percent said ineffective coordination of big data and analytics teams across an organization was a challenge. That's not surprising since 54 percent don't have joint project teams of business and IT executives working together on big data initiatives, and 53 percent don't follow a top-down approach for developing big data strategies.

Other challenges for big data projects include dependency on legacy systems for data processing and management, ineffective governance models for big data and analytics, and lack of sponsorship from top management.

Interestingly, while a great deal has been made in the last one-to-two years about the shortage of people with big data and data scientist skills, only 25 percent in the survey cited the lack of such skills as a problem.

Other findings of interest for solution providers: Only 36 percent of those surveyed were using cloud-based big data and analytic platforms and only 31 percent were using open-source big data and analytics tools.

The report lays out the components of a successful big data project, including having a strong leader/champion at the top driving big data initiatives, establishing well-defined organizational structures for big data analytics projects, developing a clear roadmap with timelines and milestones, having well-defined criteria for use-case selection, and having well-defined KPIs (key performance indicators) to measure project success.