For better value from big data improve models and algorithms. With all the hype around big data analytics, not enough attention is being given to the quality of data or the validation of models built on the data.
Coefficients of determination can easily be manipulated to fit the hypothesis behind the model. As such, doesn’t this also distort the analysis of the residuals? Models for spatial and temporal data would only appear to complicate validation even further. Data management tools have improved to significantly increase the reliability of the data inputs. Until machines devise the models, focus on the validity of the data would improve model validation and reduce, not eliminate inherent bias.
Algorithms are the fuel for a correlation engine to discover newer and more meaningful relationships in big data. The success of anticipatory analytics relies on the data set selection, the quality of the data being fed into the model, and the statistical models being used to analyze the data. Advanced statistical, data mining and machine learning algorithms can dig deeper to find these patterns faster and more cost-effectively than overmatched traditional BI tools. The winning formula lies in developing powerful new algorithms that leading companies use to strengthen competitiveness and brand perception, drive cost efficiencies in every facet of the enterprise, and manage risks more pro-actively all the while adhering to data governance and security mandates.
Over the next year, expect accelerated adoption of predictive analytics within and across business functions as organizations seek to find greater value from their data. Predictive analytics will help organizations uncover previously hidden patterns, identify classifications, associations, and segmentations, and make much more accurate predictions from structured and unstructured information. This will dramatically impact corporate strategy and planning processes as enterprises and service provider rely on real-time analysis of current activity and anticipate what will happen next. By identifying the key drivers of various business outcomes, organizations will be further enabled to deliver more personalized and contextual customer experiences.
However, there is a common misperception – fueled in part by companies with roots in complex event processing – that predictive analytics can predict the future. Perhaps “anticipatory analytics” is a better, albeit less marketable name. The statistics professor in my MBA program many years ago introduced himself to the class by saying that statistics can be used to prove any point of view. He is still right. However, better models and algorithms are the path to realizing the promise of big analytics.
