Data Products are a bigger deal for you (and your company) than AI. You’ll see what I mean shortly.
When I first started writing about Data Products, I mentioned my excitement at how information management had finally become the next big thing. Even better, the business was demanding the metadata.
At the DGIQ Conference this past December, I taught a half-day workshop on Data Products. During the course of the discussion, I was asked an interesting question. I thought I’d share the question and the subsequent conversation here.
Let’s say we’ve just put in a lot of time and effort and energy into creating Data Products. And now by the time we’re done, there’s a new big thing. Why make the investment when there’s just going to be something else?
I understand the concern. After all, I’m of the generation that bought music on records, then bought that same music on cassette and 8-track tapes, then bought that same music on CDs, then ripped CDs to MP3s, then paid a subscription to listen to that same music through at least one streaming service. Some of the more aggressive technology adopters among us could even add digital audio tapes and mini-disks to the list. Data Products seem like a lot of work. Maybe it would be better just to wait and skip ahead to the next thing.
There will always be a next big thing.
The evolutionary path is well known. The idea of consolidating data to support analytics was formalized in the Data Warehouse. This was followed closely by Data Marts and Operational Data Stores. Remember the explosion of interest in Big Data with Hadoop? Then cloud and Data Lakes and Data Lakehouses.
But looking more closely, it becomes apparent that the advances were mainly in the supporting technology. More powerful processors, less expensive memory, larger and faster storage (disk then solid state), increased network bandwidth, and more sophisticated repositories and tools. Those platform innovations enabled more data–structured, unstructured, and streaming–more queries, and more complex workloads.
The underlying architectures, though, were pretty much the same.
Compare the original Data Warehouse architecture with a “modern” analytics architecture. Now, there’s a whole lot of great stuff in the modern architecture and capabilities that we didn’t have before, but when you step back from the technology details you see that it’s largely the same thing.
In this context, the question becomes even more relevant. Companies spent a lot of time and money migrating their Data Warehouses into Hadoop, and then again migrating their on-premise Data Lakes into cloud Data Lakes. They could have just skipped the former and gone directly to the latter without really missing out on much.
That brings us to Data Products and why it’s not only useful, but necessary to invest in them.
I don’t know what the next big thing will be, but I know that there will be one.
That seems to go without saying. You can consult the latest Gartner Hype Cycle for some good candidates. The difference with Data Products is that:
Implementing Data Products prepares you for that next big thing, whatever it is.
We’re already seeing this. Data Mesh and Data Fabric are pushing traditional analytics architectures in new directions. Both are built upon a foundation of data understanding.
AI may be the biggest next big thing we see in our careers. If it isn’t, it’ll certainly be toward the top of a short list (quantum computers, when they become available, will be up there, too).
Many companies are experiencing problems implementing AI today as the result of poorly understood poor quality data, and now find themselves having to repay decades of Data Debt.
When implemented properly, this is exactly the problem that Data Products directly address and resolve. I’ll go one step farther:
Future advances in data and analytics, and in AI, will depend upon well understood, accurate data.
Data Products, therefore, directly facilitate your success with not only the current next big thing but the next next big thing as well. It’s probably not a stretch to say that they will be a prerequisite. This is why Data Products are a bigger deal for you (and your company) than AI. Data Products support AI today, and whatever the next thing is tomorrow.
Companies that invested in understanding their data are now reaping the benefits through leadership in AI implementation. For those that haven’t yet, pursuing Data Products provides a framework for achieving that understanding.
The next major advances in data and analytics will leverage knowledge about and derived from the data, and are likely to be even greater competitive differentiators. The benefits to your company by these next big things, whatever they are, will be accelerated through an investment today in Data Products.
You know what they say: the best time to plant a tree is twenty years ago. The second best time is today. The same can be said for data curation and Data Product development.
Now is the time.
It will not be a waste of time.
In fact, it’s the opposite.
It’s necessary.
Sources
Vintage Analytics Architecture: Stanford. 2003. “Data Warehousing Concepts.”
Modern Analytics Architecture: Microsoft. 2023. “Designing and Implementing Modern Data Architecture on Azure Cloud.”
0 Comments