Great question. After all, a Data Product Provider is responsible for:
- producing the content, ensuring its availability, and validating its correctness
- collecting the metadata and publishing it through the Data Product Catalog
- responding to questions and enhancement requests
Or, more succinctly: content, curation, catalog, certification, and maintenance. Along the way there are review processes, move to production processes, and change control processes. And the more useful your Data Product, the more customers you will have, and the more time and effort it is likely to take to support them.
Creating this thing you call a Composed Data Product sure seems like a lot of work. I just want to make something for my team and me. We have an immediate business need. We know the business metrics that we use and how they should be defined. We are the ones that are going to the effort to create it. Tell you what. I’m just going to create some summary tables in a Data Mart. If you want to use them, I’ll give you access and you can use them. Seem fair?
The conversation doesn’t get any better with the source system teams on the topic of creating Foundational Data Products. My job is to support my business partner’s operational processes. Yes, data gets created as a byproduct and I’ll send you a feed if you want, but that’s the end of my involvement with it. After all, if it’s good enough to run the business, it’ll be good enough for you. I’m operations, not analytics.
We all know the arguments. We’ve all probably heard most of them. No time. No people. No interest. No tools / guidance / process / training / whatever. Not a priority. All cost, no benefit.
The primary challenge establishing Data Product creation as a standard operating procedure is the same as that which plagues Data Quality, Data Governance, and (the prevention of) Data Debt.
There is a fundamental misalignment between those that put in the initial effort and those that benefit from it.
It doesn’t matter that the benefit for the company as a whole far outweighs the investment by an individual team. I’m not going to risk my delivery date for something that’s not going to benefit me. Everybody else gets the benefit. I get the cost. And the headaches.
It’s hard to counter that argument. We first try appealing to altruism and corporate spirit, “You’re creating a Data Product for the betterment of the company. We’re all in this together.” I don’t see “betterment of the company” in my budget forecast or in my headcount.
But let’s say that they agree. Or they’re told to agree. Experience suggests that corporate resolve usually doesn’t remain steadfast for long. The slightest puff of resistance—think promised delivery date in jeopardy—and it gets thrown overboard. Like a New Year’s diet resolution at the first mid-afternoon hunger pang. Maybe you’ll get some grudging or aggressive compliance, but it’s not sustainable.
We need to figure out how to make this whole Data Product thing happen. The benefit to the company in accelerating and improving analytics (especially AI) is self-evident, at least to us, and the Data Product is the foundation upon which modern data architectures like Data Mesh and Data Fabric are built.
We have very few levers. We’ve already tried arguing that even the development groups will save time and people, and that all this stuff might, and I emphasize might, pay dividends in the future. Everybody’s got their numbers to make now. Information management has a long history of not being believed despite any demonstrated results (or in many cases because we have not been able to convincingly quantify those results).
One option is for the enterprise team to pay for the processing to calculate Data Products, the space to store it, and the resources to access it.
Now, that’s money in front of me right now. And if I have to pay for all non-Data Product activity myself, then maybe I’ll consider doing this Data Product thing.
There’s one other approach to consider. I’ve long said that: 1) The only weapon that an enterprise analytics team has is communication; and 2) Wherever two or more Vice Presidents are gathered in the presence of a bar chart with their names on it there will be competition. So, make a competition out of it.
Regularly publish bar charts showing which VPs are producing and consuming Data Products and which aren’t.
Don’t underestimate the power of executive peer pressure. It is an engine of corporate activity. Be aware that you might make some angry, and if your management isn’t committed to the objective they will crumple and make you take the chart down. But even if that happens, keep collecting the data.
The enterprise team has a role as well. I talked about this a couple of weeks ago. They need to develop the tools and processes that will enable the business process teams to do their job with as little friction as possible. Don’t be the excuse that they use to not create Data Products. The enterprise team can also offer operational support services like monitoring, mentoring, and training.
A multi-pronged approach to Data Products is required:
- Upper management must endorse and support an enterprise-wide data strategy and culture in which Data Products are a key component.
- Incentives must be created to encourage Data Product Providers, both business and development teams, to participate. These include enterprise funding for Data Product resources and operational support.
- Disincentives must be created to discourage one-off solutions. These include rejecting analytical artifacts that are not derived from Data Products. Don’t even look at them.
- Communicate the level of participation from each Vice President organization.
Finally, if you don’t have executive support but still want to do Data Products because you know it’s the right thing to do, I wish you luck. I’m a big fan of skunkworks projects. Find a group that’s willing to participate. Take control of the things that you can control. You can still streamline your processes. You can still prepare a Data Product Catalog, but maybe call it a Business Glossary or Metadata Repository. You can still identify semantic disconnects and duplicate queries. And you can still find a group that’s willing to participate, partner with them, and facilitate their work as a demonstration. Small wins go a long way toward convincing those that need convincing to start to support your efforts.
Modernization will happen eventually. You be ready when they’re ready.
