We’re witnessing the shift from big data collection to real-time data consumption, and 2019 will mark the tipping point as intelligent applications — not people — become the primary consumers of data. With that in mind, here’s a look at the trends in the data trenches that are facilitating this shift for companies worldwide and the predicted implications across multiple aspects of technology.
Onerous Mega Projects Become Iterative Micro Adjustments
I’ve seen firsthand how companies of all stripes are shifting from big data moonshots — such as all-encompassing data lakesor broad internet of things (IoT) platforms — to embrace the benefits of incremental improvements by leveraging “in the moment” data insights, and I believe this trend will accelerate in 2019. In part, this is a recognition of failure: These mega projects cost too much and took too long to implement (if they were even completed at all). Many were never delivered as promised, and those that were ended up being outdated by the time they were ready for production. It’s also a recognition of the larger shift from slow planning cycles to fast iteration. When getting data is slow and hard, lengthy planning is necessary and complexity is expected. When getting data is fast and easy, fast iteration and adjustment wins every time.
Streams Proliferate As Lakes Dry Up
A corollary to this is that 2019 will mark the decline of the data lake as organizations look to leverage real-time data streaming rather than dumping data into one big but slow and murky reservoir in hopes of deriving eventual value. Data lakes reflect the mega projects reminiscent of earlier big data initiatives, which generated a lot of revenue for consultants but few results for organizations. While there will certainly still be a need to store and archive data for long term needs, I think companies will focus more on extracting value and insight as data streams into the organization rather than on just collecting data for eventual refinement and use.
Data Becomes Part Of The Corporate Fabric
With the shift to real-time processing of fast data will come an emerging IT focus on providing a data fabricor backbone that applications and users tap for data as needed rather than hoarding their own copies. A 2017 surveyof corporate executives by NewVantage noted over 85% of firms have initiated programs to create a data-driven culture. New technologies — in particular, next-generation publish/subscribe and related messaging solutions — are making this possible, bringing multiple benefits.
One of these is simplicity, as companies move away from a patchwork of data systems and data stores to embrace a more integrated organization-wide data fabric. Secondly, we’re seeing the elimination of silos: When data becomes distributed and ubiquitous rather than trapped in isolated, discrete systems, silos disappear, and that data simultaneously becomes more accessible and delivers more value.
Data Processing Stretches To The Edge
Processing of and reaction to data is steadily moving closer to where the data originates, which in many cases is at the enterprise edge. Cloud vendors are certainly hoping to absorb this as part of the cloud, but I predict that won’t happen, at least not entirely. In large part, this is because of IoT, and I’m not just talking about your connected toaster. In industrial settings, there can be some serious muscle and significant sources of data in these devices. Think robotic manufacturing equipment, transportation equipment, medical devices and more. While IoT will certainly encompass vast numbers of simple sensors that “phone home” with the latest data for central or cloud processing, there will also be an ever-increasing focus on understanding and acting on that data closer to the edge, where the action is.
Serverless Finally Makes Multicloud A Reality
That may be an exaggeration, but it illuminates a point: Two of the hottest discussion topics in technology today — multicloud (using more than one cloud vendor platform) and serverless architectures (abstracting the infrastructure needed for code execution) — are just two different looks at the same realization that many people likely don’t care if they’re using Amazon Web Services, Azure or Google Cloud Platform. It seems lots of people don’t mindwhich service is hosting an app. Most users don’t care if they’re on a public cloud, private cloud, on-prem or all of the above. All they care about is where they submit their code or get their data. In that respect, distributed data will become the glue that holds everything together, while an app would provide the intelligence needed to act on that data.
Which brings us to the final prediction, the one we started with.
Intelligent Applications — Not People — Will Become The Primary Data Consumer
Even if it doesn’t happen all at once in 2019, I believe in the next few years we’ll be able to look back and see that this year marked a tipping point where artificial intelligence, data analytics and machine learning embedded in applications replaced reports, dashboards and other people-oriented output as the primary consumers of data. Software will be empowered to act on data for us — whether it’s machine-to-machine or machine-to-consumer — rather than simply surfacing the data for people to examine and use to make decisions.
This will have profound implications not only in terms of technology but in the fundamental nature of how people make decisions. I’ll also throw in a bonus prediction: The rise of the intelligent application will solve the data staffing crisis. There has been a long-standing disconnect between the vast and growing volume of job openings in data analytics and the dearth of trained applicants. This isn’t to say that data analytics will become any less valued — far from it. Someone still has to build the data models and analytics that tell the software what to do. But software will increasingly take on the burden of consuming and analyzing data.
Does that mean that we are moving inevitably to the day when we are replaced by our AI-enabled robot overlords? I’ll leave any prediction about that for another year.