Over the past decade, Data Catalogs have emerged as important pillars in the landscape of data-driven initiatives. However, many vendors on the market fall short of expectations with lengthy timelines, complex and costly projects, bureaucratic Data Governance models, poor user adoption rates, and low-value creation. This discrepancy extends beyond metadata management projects, reflecting a broader failure at the data management level.
The present situation reveals a disconnection between technical proficiency and business knowledge, a lack of collaboration between data producers and consumers, persistent data latency and quality issues, and unmet scalability of data sources and use cases. Despite substantial investments in both personnel and technology, companies find themselves grappling with a stark reality – the failure to adequately address business needs.
The good news, however, is that this predicament can be reversed by embracing an Enterprise Data Marketplace (EDM) and leveraging existing investments.
Introducing the Enterprise Data Marketplace
An EDM is not a cure-all, but rather a transformative solution. It necessitates companies to reframe their approach to data, introducing a new entity – Data Products. A robust Data Mesh, as advocated by Zhamak Dehghani in her insightful blog post, becomes imperative, with the EDM serving as the experiential layer of the Data Mesh.
However, the landscape has evolved with a new breed of EDM – a Data Sharing Platform integrated with a robust federated Data Catalog:
EDM = Data Sharing Platform + Strong Data Catalog
This is precisely what Zeenea accomplishes, and plans to enhance further, with our definition of an EDM:
An Enterprise Data Marketplace is an e-commerce-like solution, where Data Producers publish their Data Products, and Data Consumers explore, understand, and acquire these published Data Products.
The Marketplace operates atop a Data Catalog, facilitating the sharing and exchange of the most valuable Domain Data packaged as Data Products.
Why complement your Data Catalog with an Enterprise Data Marketplace?
We’ve compiled 5 compelling reasons to enhance your Data Catalog with an Enterprise Data Marketplace.
Reason #1: Streamline the Value Creation Process
By entrusting domains with the responsibility of creating Data Products, you unlock the wealth of knowledge possessed by business professionals and foster a more seamless collaboration with Data Engineers, Data Scientists, and Infrastructure teams. Aligned with shared business objectives, the design, creation, and maintenance of valuable, ready-to-use Data Products will collectively adopt a Product Design Thinking mindset.
Within this framework, teams autonomously organize themselves, streamlining ceremonies for the incremental delivery of Data Products, bringing fluidity to the creation process. As Data Products incorporate fresh metadata to guide Data Consumers on their usage, an EDM assumes a pivotal role in shaping and exploring metadata related to Data Products – essentially serving as the Experience Plane within the Data Mesh framework.
By adhering to domain-specific nuances, there is a notable reduction in both the volume and type of metadata, alongside a more efficient curation process. In such instances, a robust EDM, anchored by a potent Data Catalog like Zeenea, emerges as the core engine. This EDM not only facilitates the design of domain-specific ontologies but also boasts automated harvesting capabilities from both on-premises and cloud-based data sources. Moreover, it empowers the federation of Data Catalogs to implement diverse Data Mesh topologies and grants end-users an effortlessly intuitive eCommerce-like Data Shopping experience.
Reason #2: Rationalize Existing Investments
By utilizing an EDM (alongside a powerful Data Catalog), existing investments in modern data platforms and people can be significantly enhanced. Eliminating intricate data pipelines, where data often doesn’t need to be moved, results in substantial cost savings. Similarly, cutting down on complex, numerous, and unnecessary synchronization meetings with cross-functional teams leads to considerable time savings.
Therefore, a focused approach is maintained by the federated governance body, concentrating solely on Data Mesh-related activities. This targeted strategy optimizes resource allocation and accelerates the creation of incremental, delegated Data Products, reducing the Time to Value.
To ensure measurable outcomes, closely monitoring the performance of Data Products with accurate KPIs becomes paramount – This proactive measure enhances decision-making and contributes to the delivery of tangible results.
Reason #3: Achieve Better Adoption Than with a Data Catalog Only
An EDM, coupled with a powerful Data Catalog, plays a pivotal role in facilitating adoption. At the domain level, it aids in designing and curating domain-specific metadata easily understood by Domain Business Users. This avoids the need for a “common layer”, a typical pitfall in Data Catalog adoption. At the Mesh Level, it offers means to consume Data Products effectively, providing information on location, version, quality, state, provenance, platform, schema, etc. A dynamic domain-specific metamodel, coupled with strong search and discovery capabilities, makes the EDM a game-changer.
The EDM’s added value lies in provisioning and access rights, integrating with ticketing systems, dedicated Data Policy Enforcement platforms, and features from Modern Data platform vendors – a concept termed Computational Data Governance.
Reason #4: Clarify Accountability and Monitor Value Creation Performance
Applying Product Management principles to Data Products and assigning ownership to domains brings clarity to responsibilities. Each domain becomes accountable for the design, production, and life cycle management of its Data Products. This focused approach ensures that roles and expectations are well-defined.
The EDM then opens up Data Products to the entire organization, setting standards that domains must adhere to. This exposure helps maintain consistency and ensures that Data Products align with organizational goals and quality benchmarks.
In the EDM framework, companies establish tangible KPIs to monitor the business performance of Data Products. This proactive approach enables organizations to assess the effectiveness of their data strategies. Additionally, it empowers Data Consumers to contribute to the evaluation process through crowd-sourced ratings, fostering a collaborative and inclusive environment for feedback and improvement.
Reason #5: Apply Proven Lean Software Development Principles to Data Strategy
The creation of Data Products follows a similar paradigm to the Lean Software Development principles that revolutionized digital transformation. Embracing principles like eliminating waste, amplifying learning, deciding late, delivering fast, and building quality is integral to the approach that a Data Mesh can enable.
In this context, the EDM acts as a collaborative platform for teams engaged in the creation of Data Products. It facilitates:
- Discovery Features: Offering automatic technical curation of data types, lineage information, and schemas, enabling the swift creation of ad hoc products.
- Data Mesh-Specific Metadata Curation: The EDM incorporates automatic metadata curation capabilities specifically tailored for Data Mesh, under the condition that the Data Catalog has federation capabilities.
- 360 Coverage of Data Products Information: Ensuring comprehensive coverage of information related to Data Products, encompassing their design and delivery aspects.
In essence, the collaboration between an Enterprise Data Marketplace and a powerful Data Catalog not only enhances the overall data ecosystem but also brings about tangible benefits by optimizing investments, reducing unnecessary complexities, and improving the efficiency of the data value creation process.
