Metadata management is an important component in a data management project and it requires more than just the data catalog solution, however connected it may be.

A data catalog tool will of course reduce the workload but won’t in and of itself guarantee the success of the project.

In this series of articles, discover the pitfalls and preconceived ideas that should be avoided when rolling out an enterprise-wide data catalog project. The traps described in this are articulated around 4 central themes that are crucial to the success of the initiative:

1. Data culture within the organization
2. Internal project sponsorship
3. Project leadership
4. Technical integration of the Data Catalog

—

Integrating the data catalog into the enterprise ecosystem will provide opportunities to create value. It is essential to consider these aspects and understand the potential rewards.

Not all metadata has to be entered manually

More and more systems produce, aggregate, and enable the entering of metadata for local value. This information has to be retrieved and consolidated in the catalog, without being entered twice, for obvious reasons (saving money, data reliability, and availability).

The data catalog, therefore, presents an opportunity to consolidate this information with the knowledge of the contributors in their respective fields. However, this consolidation has to be thought out through a technical integration rather than a manual effort. Even if it’s obvious that entering the same information twice isn’t efficient, nor is carrying out imports/exports between systems through human actions the way to go.

The strength of a data catalog remains its capacity to ingest metadata via technical integration chains and thus ensure a robust synchronization between systems.

The data catalog isn’t an “automagical” tool

On the flip side, thinking that a data catalog can extract all types of metadata regardless of its source or format, would be misleading.

The catalog should of course facilitate metadata retrieval, but some metadata won’t be retrievable automatically. There will therefore always be a cost linked to the intervention of the contributors.

The first reason for this resides in the origin of some metadata: some information may simply not be present in the systems because it originates solely from the knowledge of experts. The data catalog is therefore, in this case, a potential candidate for becoming the master system and eligible to receive this information.

And conversely, some information can be present in a system and be impossible to retrieve in an automated manner…for many reasons. For example, there could be an absence of
an interface that enables information to be accessed in a stable manner. The risk of producing noise around the information is therefore high and can lead to a degradation of the quality of the catalog content and ultimately turn the users off using it.

The data catalog must not be connected to a unique metadata source

Metadata stems from many varied layers. As a result, there are multiple and complementary sources involved for a global understanding. It is precisely the reconciliation of this information in a central solution, a data catalog, that will provide the necessary elements to the users.

Opting for a connected data catalog is a real asset, because asset discovery and the associated metadata retrieval are made considerably easier as a result of automation.

This connectivity can also extend to other complementary systems. These systems can potentially come before or after the first one, enabling, if needed, the materialization of the lineage and thus documenting the flows and transformations between systems.

The systems can also be independent of one another and simply allow for, by their addition to the catalog, an exhaustive cartography of the company’s patrimony.

Lastly, given the variety of the types of assets that can be documented in the catalog, the different connected sources can also contribute to the enrichment of a specific universe in the data catalog: semantic layers for some, physical layers for others, etc.

Always with an iterative approach in mind, the multiple sources that will feed the data catalog will be integrated progressively, in accordance with a strategy that seeks the production of value, under the global supervision of the Data Office.

Since the beginning of the 21st century, we’ve been experiencing a true digital revolution. The world is constantly being digitized and human activity is increasingly structured around data and network services. The manufacturing, leisure, administration, service, medical, and so many other industries are now organized around complex and interconnected information systems. As a result, more and more data is continuously collected by the devices and technologies present in our daily lives (Web, Smartphone, IoT) and transited from system to system. It has become central for any company that provides products or services to do everything possible to protect the data of their customers. The best approach to do so is through Privacy by Design. 

In this article, we explain what Privacy by Design is, how we applied this approach in the design of our data catalog, as well as how a data catalog can help companies implement Privacy by Design. 

Data protection: a key issue for enterprises

Among all the various data mentioned above, some allow the direct or indirect identification of physical persons. These are known as personal data, as defined by the CNIL. It is of paramount importance in the modern world because of its intrinsic value. 

On a daily basis, huge volumes of personal data pass between individuals, companies, and governments. There is a real risk of their misuse, as the Cambridge Analytica scandal in 2015 showed, for example. Cybercriminals can also make substantial gains from it, via account hacking, reselling data to other cybercriminals, identity theft, or attacking companies via phishing or president scams. For example, a real estate developer was recently robbed of several tens of millions of euros in France.

The need to protect data has never been so important.

States have quickly become aware of this issue to protect individuals from the abuses related to the exploitation of their data. In Europe, for example, the GDPR (the General Data Protection Regulation) has been in effect since 2016 and is already well established in the daily activities of companies. In the rest of the world, regulations are constantly evolving and are a concern for nearly every country. Recently, California passed a consumer data privacy law, a U.S. equivalent of the GDPR. Even China has just legislated on this topic.

Privacy by Design: defining a key concept for data protection

While many legislations rely heavily on the notion of Privacy by Design, it was conceptualized by Ann Cavoukian in the late 1990s when she was the Information and Privacy Commissioner of the Province of Ontario in Canada. The essence of this idea is to include the issue of personal data protection right from the design of a computer system. 

In this sense, Privacy by Design lists seven fundamental principles:

#1 – Proactivity: any company must put in place the necessary provisions for data protection upstream, and must not rely on a reactive policy;

#2 – Personal data protection as a default setting: any system must take as a default setting the highest possible level of protection for the sensitive data of its users;

#3 – Privacy by design: privacy should be a systematically studied and considered aspect of the design and implementation of new functionality;

#4 – Full functionality: no compromise should be made with security protocols or with the user experience;

#5 – End-to-end security: the system must ensure the security of data throughout its lifecycle, from collection to destruction (including if the data is outsourced);

#6 – Visibility and transparency: the system and the company must document and communicate personal data protection procedures and actions taken in a clear, consistent and transparent manner;

#7 – Respect for user privacy: every design and implementation decision must be made with the user’s interest at the center.

The application of Privacy by Design at Zeenea

At Zeenea, and particularly because the company was created in 2017, we’ve built our product on the foundations of Privacy by Design.

The treatment of users’ personal data

First of all, we have anchored data protection at the heart of our architecture. Each of our customer’s data is segregated into different tenants, each encrypted with their own key. User authentication is managed through a specialized third-party system. We encourage identity federation among our customers, which allows them to maintain control over the data needed for user identification and authentication.

We have also included the concept of Privacy by Design in the design of our application. For example, we collect only the bare minimum of information, all system outputs are anonymized (logs, application errors, APIs).

Processing customer business data

Our main mission being to document the data, our solution contains by essence metadata. By design, Zeenea does not extract any data from our customers’ systems. Indeed, the risk is intrinsically less on the metadata than on the data. 

Nevertheless, we offer within Zeenea, several features allowing us to provide information on the data present in the client systems (statistics, sampling, etc.). Because of our architecture, the calculations are always done on the client’s infrastructure, as close as possible to the data and its security. And in compliance with principle #2 of Privacy by Design, we have set the protection of personal data as a default setting. Thus, all these features are disabled by default and can only be activated by the customer.

How our data catalog helps companies implement Privacy by Design

Our data catalog can help your company implement Privacy by Design, especially on the control and verification aspects. Taking the 7 principles described earlier, the data catalog can effectively participate in two of them: the visibility and transparency principle, and the end-to-end security principle. The data catalog also enables the automation of the identification of sensitive data.

Visibility and transparency via the data catalog

The objective of a data catalog is to centralize a company’s data assets, document them, and share them with as many people as possible. This centralization allows each employee to know what data is collected by the CRM, and the marketing and customer success teams to process this information in the acquisition and churn tracking reports.

Once this inventory has been established, the catalog can be used to document certain additional information that is necessary for the company’s proper functioning. This is notably the case for the sensitive or non-sensitive nature of the documented information, the rules of governance, the processing, or the access procedures that must be applied. 

In the context of a Privacy by Design approach, the data catalog can be used to add a business term corresponding to sensitive data (a social security number, a telephone number, etc.). This business term can then be easily associated with the tables or physical fields that contain the data, thus allowing its easy identification. This initiative contributes to the principle of visibility and transparency of Privacy by Design.

End-to-end security via the data catalog

The data catalog also provides data lineage capabilities. Automatic data lineage ensures that the processes applied to data identified as sensitive comply with what is defined by the company’s data governance. It is then simple with the data catalog to fill in the governance rules to be applied to sensitive data. 

Moreover, the lineage allows us to follow the whole life cycle of the data, from its creation to its final use, including its transformations. This makes it easy to check that all the stages of this life cycle comply with the rules and correct any errors. 

The data catalog, via the data lineage, thus contributes to the principle of the end-to-end security of Privacy by Design.

With that said, at Zeenea we remain convinced that a data catalog is not a compliance solution, but rather a tool for the acculturation of teams to sensitive data and its particularities of use.

Identifying sensitive data via the data catalog

In a rapidly changing data environment, the data catalog must reflect reality as much as possible in order to maintain the trust of its users. Without this, the entire adoption of the data catalog project is put into question. 

At Zeenea, we are firmly convinced that the data catalog must be automated as much as possible to be scalable and efficient. This starts with the inventory of available data. In this sense, our inventory is automated and is responsible for passing on all modifications to the original system (source) of the data directly into the catalog. Thus, at any time, the customer has an exhaustive list of the data present in its systems. 

And to help our customers identify which of the inventoried data deserve special treatment because of their sensitive data status, the automation does not stop at the inventory. We now offer a system that suggests the tagging of new data inventoried with a sensitive data profile. This makes it easier to bring this data to the forefront and to spread the information faster and more easily throughout the company.

For more information on the technology used at Zeenea, download our eBook “The 5 technological breakthroughs of a Data Catalog“.

Conclusion

In the past few years, personal data has become a real concern for most consumers. More and more countries are setting up regulations to guarantee their citizens maximum protection. One of the major principles governing all these regulations is Privacy by Design.

At Zeenea, we have from the start included the reflection around personal data at the heart of our product. Both in our technical development and in the processing of our users’ data, as well as in our reflection on the data that our clients process via our catalog. 

We believe that a data catalog can be a significant asset in the implementation and monitoring of Privacy by Design policies. We also heavily rely on automation and AI to bring many more improvements in the upcoming months: automatic construction of technical data lineage, improved detection of sensitive data in the catalog objects to better document them, quality control of processes applied to sensitive data, etc. The possibilities are numerous. 

To learn more about the advantages of the catalog in the management of your sensitive and personal data, don’t hesitate to schedule a meeting with one of our experts:

To remain competitive, organizations must make decisions quickly, as the slightest mistake can lead to a waste of precious time in the race for success. Defining the company’s reason for being, its direction, and its strategy makes it possible to build a solid foundation for creating an alignment – subsequently facilitating decisions that impact product development. Aligning all stakeholders in product development is a real challenge for Product Managers. Yet, it is an essential mission to bring up a successful product and an obvious prerequisite to motivate teams who need to know why they get up each morning to go to work.

The foundations of a shared product vision within the company

Various frameworks (NorthStar, OKR, etc.) have been developed over the last few years to enable companies and their product teams to lay these foundations, disseminate them within the organization, and build a roadmap that creates cohesion. These frameworks generally define a few key artifacts and have already given rise to a large body of literature. Although versions may differ from one framework to another, the following concepts are generally found: 

• Vision: the dream, the true North of a team. The vision must be inspiring and create a common sense of purpose throughout the organization.
• The mission: it represents an organization’s primary objective and must be measurable and achievable.
• The objectives: these define measurable short and medium-term milestones to accomplish the mission.
• The roadmap: a source of shared truth – it describes the vision, direction, priorities, and progress of a product over time.

With a clear and shared definition of these concepts across the company, product teams have a solid foundation for identifying priority issues and effectively ordering product backlogs.

Product values: the key to team buy-in and alignment over time

Although well defined at the beginning, these concepts described above can nevertheless fall into oblivion after a while or become obsolete! Indeed, the company and the product evolve, teams change, and consequently the product can lose its direction… A work of reconsideration and acculturation must therefore be carried out continuously by the product teams in order for it to last.

Indeed, product development is both a sprint and a marathon! One of the main difficulties for product teams is to maintain this alignment over time. In this respect, another concept in these frameworks is often under-exploited when it is not completely forgotten by organizations: product values. 

Jeff Steiner, Executive Chairman at LinkedIn, particularly emphasized the importance of defining company values through the Vision to Values framework. LinkedIn defines values as “The principles that guide the organization’s day-to-day decisions; a defining element of your culture“. For example “be honest and constructive“, “demand excellence“, etc.

Defining product values in addition to corporate values can be a great way for product teams to create this alignment over time and this is exactly what we do at Zeenea.

From corporate vision to product values: a focus on Zeenea Data Catalog

Organization & product consistency at Zeenea

At Zeenea, we have a shared vision – “Be the first step of any data journey” – and a clear mission – “To help data teams accelerate their initiatives by creating a smart & reliable data asset landscape at the enterprise level“.  

We position ourselves as a data catalog pure-player and we share the responsibility of a single product between several Product Managers. This is why we have organized ourselves into feature teams. This way, each development team can take charge of any new feature or evolution according to the company’s priorities, and carry it out from start to finish.

If we prioritize the backlog and delivery by defining and adapting our strategy and organization according to the objectives, three problems remain: 

• How do we ensure that the product remains consistent over time when there are multiple pilots onboard the plane? 
• How do we favor one approach over another? 
• How do we ensure that a new feature is consistent with the rest of the application? 

Indeed, each product manager has his or her own sensitivity, his or her own background. And if the problems are clearly identified, there are usually several ways to solve them. This is where product values come into play…

Zeenea’s product values

If the vision and the mission help us to answer the “why?”, the product values allow us to remain aligned with the “how?”. It is a precious tool that challenges the different possible approaches to meet customer needs. And each Product Manager can refer to these common values to make decisions, prioritize a feature or reject it, and ensure a unified & unique user experience across the product.

Thus, each new feature is built with the following 5 product values as guides:

Simplicity

This value is at the heart of our convictions. The objective of a Data Catalog is to democratize data access. To achieve this, facilitating catalog adoption for end users is key. Simplicity is clearly reflected in the way each functionality is proposed. Many applications end up looking like Christmas trees with colored buttons all over the place that no one knows how to use; others require weeks of training before the first button is clicked. The use of the Data Catalog should not be reserved to experts and should therefore be obvious and fluid regardless of the user’s objective. This value was reflected in our decision to create two interfaces for our Data Catalog: one dedicated to search and exploration, and the other for the management and monitoring of the catalog’s documentation. 

Empowering

Documentation tasks are often time-consuming and it can be difficult to motivate knowledgeable people to share and formalize their knowledge. In the same way, the product must encourage data consumers to be autonomous in their use of data. This is why we have chosen not to offer rigid validation workflows, but rather a system of accountability. This allows Data Stewards to be aware of the impacts of their modifications. Coupled with an alerting and auditing system after the fact, it ensures better autonomy while maintaining traceability in the event of a problem.

Reassuring

It is essential to allow end-users to trust in the data they consume. The product must therefore reassure the user by the way it presents its information. Similarly, Data Stewards who maintain a large amount of data need to be reassured about the operations for which they are responsible: have I processed everything correctly? How can I be sure that there are no inconsistencies in the documentation? What will really happen if I click this button? What if it crashes? The product must create an environment where the user feels confident using the tool and its content. This value translates into preventive messages rather than error reports, a language type, idempotency of import operations, etc.

Flexibility  

Each client has their own business context, history, governance rules, needs, etc. The data catalog must be able to adapt to any context to facilitate its adoption. Flexibility is an essential value to enable the catalog to adapt to all current technological contexts and to be a true repository of data at enterprise level. The product must therefore adapt to the user’s context and be as close as possible to their uses. Our flat and incremental modeling is based on this value, as opposed to the more rigid hierarchical models offered on the market.

Deep Tech 

This value is also very important in our development decisions. Technology is at the heart of our product and must serve the other values (notably simplicity and flexibility). Documenting, maintaining, and exploiting the value of enterprise-wide data assets cannot be done without the help of intelligent technology (automation, AI, etc.). The choice to base our search engine on a knowledge graph or our positioning in terms of connectivity are illustrations of this “deep tech” value at Zeenea.

Take away

Creating alignment around a product is a long-term task. It requires Product Managers – in synergy with all stakeholders – to define from the very beginning: the vision, the mission, and the objectives of the company. This enables product management teams to effectively prioritize the work of their teams. However, to ensure the coherence of a product over time, the definition and use of product values  are essential. At Zeenea, our product values are simplicity, autonomy, trust, flexibility and deep-tech. They are reflected in the way we design and enhance our Data Catalog and allow us to ensure a better customer experience over time. 

If you would like to learn more about our product, or to get more information about Data Catalog:

A data catalog should also be smart in the experience it offers to its different pools of users. Indeed, one of the main challenges with the deployment of a data catalog is its level of adoption from those it is meant for: data consumers. And user experience plays a major role in this adoption.

User experience within the data catalog

The underlying purpose of user experience is the identification of personas whose behavior and objectives we are looking to model in order to provide them with a slick and efficient graphic interface. Pinning down personas in a data catalog is challenging – it is a universal tool that provides added value for any company regardless of its size, across all sectors of activity anywhere in the world.

Rather than attempting to model personas that are hard to define, it’s possible to handle the situation by focusing on the issue of data cataloging adoption. Here, there are two user populations that stand out:

• Metadata producers who feed the catalog and monitor the quality of its content – this population is generally referred to as Data Stewards;
• Metadata consumers who use the catalog to meet their business needs – well will call them Users.

These two groups are not totally unrelated to each other of course: some Data Stewards will also be Users.

The challenges of enterprise-wide catalog adoption

The real value of a data catalog resides in large-scale adoption by a substantial pool of (meta) data consumers, not just the data management specialists.

The pool of data consumers is very diverse. It includes data experts (engineers, architects, data analysts, data scientists, etc.), business people (project managers, business unit managers, product managers, etc.), compliance and risk managers. And more generally, all operational managers are likely to leverage data to improve their performances.

Data Catalog adoption by Users is often slowed down for the following reasons:

• Data catalog usage is sporadic: they will log on from time to time to obtain very specific answers to specific queries. They rarely have the time or patience to go through a learning curve on a tool they will only use periodically – weeks can go by between catalog usage.

• Not everyone has the same stance on metadata. Some will focus more on technical metadata, others will focus heavily on the semantic challenges, and others might be more interested in the organizational and governance aspects.

• Not everybody will understand the metamodel or the internal organization of the information within the catalog. They can quickly feel put off by an avalanche of concepts that feel irrelevant to their day-to-day needs.

The Smart Data Catalog attempts to jump these hurdles in order to accelerate catalog adoption. Here is how Zeenea meets these challenges.

How Zeenea facilitates catalog adoption

The first solution is the graphic interface. The Users’ learning curve needs to be as short as possible. Indeed, the User should be up and running without the need for any training. To make this possible, we made a number of choices.

The first choice was to provide two different interfaces, one for the Data Stewards and one for the Users:

Zeenea Studio: the management and monitoring tool for the catalog content – an expert tool solely for the Data Stewards.

Zeenea Explorer: for the Users –  it provides them with the simplest search and exploration experience possible.

Our approach is aligned with the user-friendly principles of marketplace solutions – the recognized specialists in catalog management (in the general sense). These solutions usually have two applications on offer. The first, a “back office” solution, which enables the staff of the marketplace (or its partners) to feed the catalog in the most automated manner possible and control its content to ensure its quality. The second application, for the consumers, usually takes the form of an e-commerce website and enables end-users to find articles or explore the catalog. Zeenea Studio and Zeena Explorer reflect these two roles.

The information is ranked in accordance with the role of the user within the organization

Our second choice is still at the experimental stage and consists in dynamically adapting the information hierarchy in the catalog according to User profiles.

This information hierarchy challenge is what differentiates a data catalog from a marketplace type catalog. Indeed, a data catalog’s information hierarchy depends on the operational role of the user. For some, the most relevant information in a dataset will be technical: location, security, formats, types, etc. Others will need to know the data semantics and their business lineage. Others still will want to know the processes and controls that drive data production – for compliance or operational considerations.

The Smart Data Catalog should be able to dynamically adjust the structure of the information to adapt to its different prisms. 

The last remaining challenge is the manner in which the information is organized in the catalog in the form of exploration paths by theme (something similar to shelving in a marketplace). It is difficult to find a structure that agrees with everybody. Some will explore the catalog along technical lines (systems, applications, technologies, etc.). Others will explore the catalog from a more functional perspective (business domains), others still from a semantic angle (through business glossaries, etc.).

The challenge of having everyone agree on a sole universal classification seems (to us) insurmountable. The Smart Data Catalog should be adaptable and should not ask Users to understand a classification that makes no sense to them. Ultimately, user experience is one of the most important success factors for a data catalog. 

For more information on how a Smart user experience enhances a Data Catalog, download our eBook:

“What is a Smart Data Catalog?”!

A powerful search engine for an efficient exploration

Given the enormous volumes of data involved in an enterprise catalog, we consider the search engine the principal mechanism through which users can explore the catalog. The search engine needs to be easy to use, powerful, and, most importantly, efficient – the results must meet user expectations. Google and Amazon have raised the bar very high in this respect and the search experience they offer has become a reference in the field. 

This second to none search experience can be summed up thus:

• I write a few words in the search bar, often with the help of a suggestion system that offers frequent associations of terms to help me narrow down my search.

• The near-instantaneous response provides results in a specific order and I fully expect to find the most relevant one on page one.

• Should this not be the case, I can simply add terms to narrow the search down even further or use the available filters to cancel out the non-relevant results.

Alas, the best currently on offer in the data cataloging market in terms of search capabilities seems to be limited to capable systems indexations, scoring, and filtering. This approach is satisfactory when the user has a specific idea of what they are looking for (high intent search) but can prove disappointing when the search is more exploratory (low intent search) or when the idea is simply to spontaneously suggest relevant results to a user (no intent).

In short, simple indexation is great for finding information whose characteristics are well known but falls short when the search is more exploratory. The results often include false positives and the order in which the search comes out is over-represented with exact matches.

A multidimensional search approach

We decided from the get-go that a simple indexation system would prove limited and would fall short of providing the most relevant results for the users. We, therefore, chose to isolate the search engine in a dedicated module on the platform and to turn it into a powerful innovation (and investment) zone.

We naturally took an interest in the work of the founders of Google on Page Rank, their algorithm. Page Rank takes into account several dozen aspects (called features), amongst which are the density of the relation between different graph objects (hypertext links in the case of internet pages), the linguistic treatment of search terms, or the semantic analysis of the knowledge graph.

Of course, we do not have the means Google has, nor its expertise in terms of search result optimization. But we have integrated into our search engine several features that provide a high level of relevant results, and those features are permanently evolving.

We have integrated the following core features:

• Standard, flat, indexation of all the attributes of an object (name, description, and properties) weighing it up in accordance with the type of property.
• An NLP layer (Natural Language Processing) that takes into account the near misses (typing or spelling errors).
• A semantic analysis layer that relies on the processing of the knowledge graph.
• A personalization layer that currently relies on a simple user classification according to their uses, and will in the future be enriched by individual profiling.

Smart filtering to contextualize and limit search results

To complete the search engine, we also provide what we call a smart filtering system. Smart filtering is something we often find on e-commerce websites (such as Amazon, booking.com, etc.) and it consists in providing contextual filters to limit the search result. These filters work in the following way:

• Only those properties that help reduce the list of results are offered in the list of filters – non-discriminating properties do not show up.
• Each filter shows its impact – meaning the number of residual results once the filter has been applied.
• Applying a filter refreshes the list of results instantaneously.

With this combination of multi-dimensional search and smart filtering, we feel that we offer a superior search experience to any of our competitors. And our decoupled architecture enables us to explore new approaches continuously, and rapidly integrate those that seem efficient.

For more information on how a Smart search engine enhances a Data Catalog, download our eBook:

“What is a Smart Data Catalog?”!

The second way to make a data catalog “smart“ is through its inventory. A data catalog is essentially a thorough inventory of information assets that include a bunch of metadata, which helps harness the information as efficiently as possible. Setting up a data catalog, therefore, depends first of all on an inventory of the assets from the different systems.

Automating the inventory: the challenges

A declarative approach to building the inventory doesn’t strike us as particularly smart, however well thought out it may be. It involves a lot of work at the launching and the up-keeping of the catalog – in a fast-changing digital landscape, the initial effort quickly becomes redundant.

The first step in creating a smart inventory is of course to automate it. With a few exceptions, enterprise datasets are managed by system specialists (involving distributed filing systems, ERPs, relational databases, software packages, data warehouses, etc.). They manage all these systems along with all the metadata required for them to work properly. There is no need to recreate this information manually: you just need to connect to the different registries and synchronize the catalog content with the source systems.

In theory, this should be straightforward but putting it into practice is actually rather difficult. The fact is, there is no universal standard to which the different technologies conform for a universal means of access to their metadata.

The essential role of connectivity to the system sources

A smart connectivity layer is a key part of the Smart Data Catalog. For a more detailed description of Zeenea’s connectivity technology, I recommend reading our previous ebook, the 5 technological breakthroughs of a next-generation catalog, but its main characteristics are:

• Proprietary – we do not rely on third parties so as to maintain a highly specialized extraction of the metadata.
• Distributed – in order to maximize the reach of the catalog.
• Open – anyone looking to enrich the catalog can develop their own
• connectors with ease.
• Universal – it can synchronize any source of metadata.

This connectivity can not only read and synchronize the metadata contained in the source registries, it can also produce metadata.

This production of metadata requires more than simple access to the source system registries. It also requires access to the data itself, which will be analyzed by our scanners in order to enrich the catalog automatically.

To date, we produce 2 types of metadata:

• Statistical analysis: to build a profile of the data – value distribution, rate of null values, top values, etc. (the nature of the metadata depends obviously on the native type of the data being analyzed);

• Structural analysis: to determine the operational type of specific textual data (email, postal address, social security number, client code, etc. – the system is scalable and customizable).

The inventory mechanism must also be smart

Our inventory mechanism is also smart in several ways:

• Dataset detection relies on extensive knowledge of the storage structures, particularly in a Big Data context. For example, an IoT dataset made up of thousands of files of time series measures can be identified as a unique dataset (the number of files and their location being only metadata).

• The inventory is not integrated into the catalog by default to prevent the import of technical or temporary datasets that would be of little use (either because the data is unexploitable, or because it is duplicated data).

• The selection process for the assets that should be imported into the catalog also benefits from some assistance – we strive to identify the most appropriate objects for integration in the catalog (with a variety of additional approaches to make this selection).

For more information on how Smart Data Inventorying enhances a Data Catalog, download our eBook:

“What is a Smart Data Catalog?”!

A universal and static metamodel cannot be smart

At an enterprise scale, the metadata required to harness in any meaningful way the informational assets can be considerable. And besides, metadata is specific to each organization, sometimes even amongst different populations within an organization. For example, a business analyst won’t necessarily seek the same information as an engineer or a product manager might.

Attempting to create a universal metamodel, therefore, does not seem very smart to us. Indeed, such a metamodel would have to adapt to a plethora of different situations, and will inevitably fall victim to one of the 3 pitfalls below:

• Excessive simplicity which won’t cover all the use cases needed;
• Excessive levels of abstraction with the potential to adapt to a number of contexts at the cost of arduous and time-consuming training – not an ideal situation for an enterprise-wide catalog deployment;
• Levels of abstraction lacking depth and ultimately leading to a multiplicity of concrete concepts bourn out on a combination of notions emanating from a variety of different contexts – many of which will be useless in any specific context, rendering the metamodel needlessly complicated and potentially incomprehensible.

In our view, smart metamodeling should ensure a metamodel that adapts to any context and can be enriched as use cases or maturity levels develop over time.

The organic approach to a metamodel

A metamodel is a field of knowledge and the formal structure of a knowledge model is referred to as an ontology.

An ontology defines a range of object classes, their attributes, and the relationships between them. In a universal model, the ontology is static – the classes, the attributes, and the relations are predefined, with varying levels of abstraction and complexity.

Zeenea chose not to rely on a static ontology but rather on a scalable knowledge graph.

The metamodel is therefore voluntarily simple at the start – there are only a handful of types, representing the different classes of information assets (data sources, datasets, fields, dashboards), each with a few essential attributes (name, description, contacts).

This metamodel is fed automatically by the technical metadata extracted from the datasources which vary depending on the technology in question (the technical metadata of a table in a data warehouse differs from the technical metadata of a file in a datalake).

For Zeenea, this organic metamodeling is the smartest way to handle the ontology issue in a data catalog. Indeed, it offers several advantages:

• The metamodel can adapt to each context, often relying on a pre-existing model, integrating the inhouse nomenclature and terminology without the need for a long and costly learning curve;
• The metamodel does not need to be fully defined before using the data catalog – you will only need to focus on a few classes of objects and the few necessary attributes to cover the initial use cases. You can then load the model as catalog adoption progresses over time;
• User feedback can be integrated progressively, improving catalog adoption, and as a result, ensuring return on investment for the metadata management.

Adding functional attributes to the metamodel in order to facilitate searching

There are considerable advantages to this metamodeling approach, but also one major inconvenience: since the metamodel is completely dynamic, it is difficult for the engine to understand the structure, and therefore difficult for it to help users feed the catalog and use the data (two core components of a Smart Data Catalog).

Part of the solution relates to the metamodel and the ontology attributes. Usually, metamodel attributes are defined by their technical types (date, number, chain of characters, list of values, etc.). With Zeenea, these library types do include these technical types of course.

But they also include functional types – quality levels, confidentiality levels, personal touch, etc. These functional types enable the Zeenea engine to better understand the ontology, refine the algorithms and adapt the representation of the information.

For more information on how Smart Metamodelling enhances a Data Catalog, download our eBook:

“What is a Smart Data Catalog?”!

An organization’s data catalog enhances all available data assets by relying on two types of information – on the one hand, purely technical information that is automatically synchronized from their sources; and on the other hand, business information that comes from the work of Data Stewards. The latter is updated manually and thus brings its share of risks to the entire organization.

A permission management system is therefore essential in order to define and control the access rights of catalog users. In this article, we detail the fundamental characteristics and the possible approaches to build an efficient permission management system, as well as the solution implemented by Zeenea Data Catalog.

Permission management system: an essential tool for the entire organization

For data catalog users to trust in the information they are viewing, it is essential that the documentation of cataloged objects is relevant, of high quality and, above all, reliable. Your users must be able to easily find, understand and use the data assets at their disposal. 

The origin of catalog information and automation 

A data catalog generally integrates two types of information. On the one hand, there is purely technical information that comes directly from the data source. At Zeenea, this information is synchronized in a completely automated and continuous way between the data catalog and each data source, to guarantee its veracity and freshness. On the other hand, the catalog contains all the business or organizational documentation, which comes from the work of the Data Stewards. This information cannot be automated; it is updated manually by the company’s data management teams.

A permission management system is a prerequisite for using a data catalog

To manage this second category of information, the catalog must include access and input control mechanisms. Indeed, it is not desirable that any user of your organization’s data catalog can create, edit, import, export or even delete information without having been given prior authorization. A user-based permission management system is therefore a prerequisite; it plays the role of a security guard for the access rights of users.

The 3 fundamental characteristics of a data catalog’s permission management system

The implementation of an enterprise-wide permission management system is subject to a number of expectations that must be taken into account in its design. Among them, we have chosen in this article to focus on three fundamental characteristics of a permission management system: its level of granularity and flexibility, its readability and auditability, and its ease of administration.

Granularity and flexibility

First of all, a permission management system must have the right level of granularity and flexibility. Some actions should be available to the entire catalog for ease of use. Other actions should be restricted to certain parts of the catalog only. Some users will have global rights related to all objects in the catalog, while others will be limited to editing only the perimeter that has been assigned to them. The permission management system must therefore allow for this range of possibilities, from global permission to the fineness of an object in the catalog. 

At Zeenea, for example, our clients are of all sizes, with very heterogeneous levels of maturity regarding data governance. Some are start-ups, others are large companies. Some have a data culture that is already well integrated into their processes, while others are only at the beginning of their data acculturation process. The permission management system must therefore be flexible enough to adapt to all types of organizations.

Readability and auditability

Second, a permission management system must be readable and easy to follow. During an audit, or a review of the system’s permission, an administrator who explores an object must be able to quickly determine who has the ability to modify it. Conversely, when an administrator looks at the details of a user’s permission set, they must quickly be able to determine the scope that is assigned to that user and their authorized actions on it. 

This simply ensures that the right people have access to the right perimeters, and have the right level of permission for their role in the company. 

Have you ever found yourself faced with a permission system that was so complex that it was impossible to understand why a user was allowed to access information? Or on the contrary was unable to do so?

Simplicity of administration

Finally, a permission management system must be resilient in facing the increasing catalog volume. We know today that we live in a world of data: 2.5 exabytes of data were generated per day in 2020 and it is estimated that 463 exabytes of data will be generated per day in 2025. New projects, new products, new uses: companies must deal with the explosion of their data assets on a daily basis.

To remain relevant, a data catalog must evolve with the company’s data. The permission management system must therefore be resilient to changes in content or even to the movement of employees within the organization.

Different approaches to designing a data catalog permission management system

There are different approaches to designing a data catalog permission management system, which more or less meet the main characteristics expected and mentioned above. We have chosen to detail three of them in this article.

Crowdsourcing

First, the crowdsourcing approach – where the collective is trusted to self-correct. A handful of administrators can moderate the content and all users can contribute to the documentation. An auditing system usually completes the system to make sure that no information is lost by mistake or malice. In this case, there is no control before documenting, but a collective correction afterwards. This is typically the system chosen by online encyclopedias such as Wikipedia. These systems depend on the number of contributors and their own knowledge to work well, as self-correction can only be effective through the collective. 

This system perfectly meets the need for readability – all users have the same level of rights, so there is no question about the access control of each user. It is also simple to administer – any new user has the same level of rights as everyone else, and any new object in the data catalog is accessible to everyone. On the other hand, there is no way to manage the granularity of rights. Everyone can do and see everything.

Permission attached to the user 

The second approach to designing the permission management system is using solutions where the scope is attached to the user’s profile. When a user is created in the data catalog, the administrators assign a perimeter that defines the resources that they will be able to see and modify. In this case, all controls are done upstream and a user cannot access a resource inadvertently. This is the type of system used by an OS such as Windows for example.

This system has the advantage of being very secure, there is no risk that a new resource will be visible or modifiable by people who do not have the right to do so. This approach also meets the need for readability: for each user, all the accessible resources are easy to find. The expected level of granularity is also good, since it is possible to allocate the data system resource by resource. 

On the other hand, administration is more complex – each time a new resource is added to the catalog, it must be added to the perimeters of the said users. It is possible to overcome this limitation by creating dynamic scopes. To do this, you can define rules that assign resources to users, for example all PDF files will be accessible to so-and-so. But contradictory rules can easily appear, complicating the readability of the system.

Permission attached to the resource 

The last major approach to designing a data catalog’s permission management system is to use solutions where the authorized actions are attached to the resource to be modified. For each resource, the possible permissions are defined user by user. Thus it is the resource that has its own permission set. By looking at the resource, it is then possible to know immediately who can view or edit it. This is for example the type of system of a UNIX-like OS.

The need for readability is perfectly fulfilled – an administrator can immediately see the permissions of different users when viewing the resource. The same goes for the need for granularity – this approach allows permissions to be given at the most macro level through an inheritance system, or at the most micro level directly on the resource. Finally, in terms of ease of administration, it is necessary to attach each new user to the various resources, which is potentially tedious. However, there are group systems that can mitigate this complexity.

The Zeenea Data Catalog permission management model: simple, readable and flexible

Among these approaches, let’s detail the one chosen by Zeenea and how it is applied.

The resource approach was preferred

Let’s summarize the various advantages and disadvantages of each of the approaches discussed above. In both resource and user-level permission management systems, the need for granularity is well addressed – these systems allow for resource-by-resource permission to be assigned. In contrast, in the case of crowdsourcing, the basic philosophy is that anyone can access anything. Readability is clearly better in crowdsourcing systems or in systems where permissions are attached to the resource. It remains adequate in systems where permissions are attached to the user, but often at the expense of simplicity of administration. Finally, the simplicity of administration is very much optimized for the crowdsourcing approach and depends on what you are going to modify the most – the resource or the users.

Since the need for granularity is not met in the crowdsourcing approach, we eliminated it. We were then left with two options: resource-based permission or user-based permission models. Since the readability is a bit better with resource-based permission, and since the content of the catalog will evolve faster than the number of users, the user-based permission option seemed the least relevant.

The option we have chosen at Zeenea was therefore the third one: user permissions are attached to the resource.

How the Zeenea Data Catalog permission management system works

In Zeenea Data Catalog, it is possible to define for each user if they have the right to manipulate the objects of the whole catalog, one or several types of objects, or only those of their perimeter. This allows for the finest granularity, but also for more global roles. For example, “super-stewards” could have permission to act on entire parts of the catalog, such as the glossary.

We then associate a list of Curators with each object in the catalog, i.e., those responsible for documenting that object. Thus, simply by exploring the details of the object, one can immediately know who to contact to correct or complete the documentation, or to answer a question about it. The system is therefore readable and easy to understand. The users’ scope of action is precisely determined through a granular system, right down to the object in the catalog.

When a new user is added to the catalog, it is then necessary to define its scope of actions. For the moment, this configuration is done through the bulk editing of objects. In order to simplify management even further, it will soon be possible to define specific groups of users, so that when a new collaborator arrives there is no longer any need to add them by name to each object in their scope. Instead, they simply need to be added to the group, and their scope will be automatically assigned to them.

Finally, we have voluntarily chosen not to implement a documentation validation workflow in the catalog. We believe that team accountability is one of the keys to the success of a data catalog adoption. This is why the only control we put in place is the one that determines the user’s rights and scope. Once these two elements have been determined, the people responsible for the documentation are free to act! The system is completed with an event log on modifications to allow complete auditability, as well as a discussion system on the objects. It allows everyone to suggest changes or report errors on the documentation.

If you would like to learn more about our permission management model, or get more information about our Data Catalog

Having large volumes of data isn’t enough: it’s what you make of it that counts! To make the most out of your data, you need to distill a real data culture within your company. The foundation of this culture is data curation.

0% of the world’s data has been created in the last two years. With the exponential growth of connected devices, companies will be confronted with the unfortunate reality that our ability to create data will far surpass our ability to manage and exploit it. 

And it’s not going to get any better! According to estimates published in Statista’s Digital Economy Compass 2020, the annual volume of digital data created globally has increased more than 20-fold over the past decade and will surpass the 50 zettabyte mark by 2021!

In this context, it is not surprising that most companies are currently only able to analyze 12% of the data they have at their disposal! Because, behind the collection, storage and security of data, there is above all the business value that can be derived from it.

This is the challenge addressed by the concept of Data Curation: the essential step to exploit the potential of an organization’s abundant data assets. 

The definition of Data Curation

According to the definition given by the INIST (Institut de l’Information Scientifique et Technique), which is attached to the CNRS, 

“Curation refers to all the activities and operations necessary for the active management of digital research data, throughout their life cycle. The objective is to make them accessible, shareable and reusable in a sustainable way. Three stakeholders can be identified in the data life cycle: the creators, most often researchers, the “curators” and the users.”

In other words, data curation is a task that consists of identifying in a data catalog those that can be valorized, exploited and in a second time, put them at the disposal of users likely to draw the best lessons from them.

To set up an efficient and relevant Data Curation, you need to start with a precise mapping of the available data. This initial mapping is the basis for pragmatic and operational data governance. 

Once the rules of governance have been established, it is towards the data user that all attention must be focused. Data is a mineral that is only worthwhile if it is properly valued. This valuation must be thought of as a response to the user’s needs. It is the latter who is at the origin of the data curation project. 

An iterative and continuous process for data exploitation, distinct from all the tasks essential to data governance (from quality management to data protection and even data life cycle management).

Data Curation: essential prerequisites, undeniable benefits

Data Curation is a perspective of rapid and massive development of a data culture within an organization.

The creation of a data management and curation strategy allows you to take stock of the data produced. It is then possible to select the most relevant data and enrich it with the metadata necessary to understand and reuse it, including by business users.

Everyone in the company can then base their choices, decisions, strategies and methods on the systematic use of data, without having to have specific skills.

The objective: creating the conditions for systematic use of data as a basis for any project or approach, and not to limit its use to Data Science or data expert teams.

To effectively deploy your data curation strategy, you must therefore rely on elements that are essential to the proper management of your data assets. The heart of the reactor is not limited to data catalogs! 

If they are essential and directly result from your data map, metadata governance plays an even more crucial role. Metadata makes it easier for users to interact with data portfolios in natural language. 

With data curation, get into a data-driven dynamic for good!

A Data Catalog is complex…but isn’t complicated

This closing statement seems to us, a fit ting summary for all of the above,

and it will be our conclusion.

We have seen too many Data Catalog initiatives morph into endless data governance projects that try to solve a laundry list of issues- ignoring those easily solved by a Data Catalog. Once you have removed the extra baggage.

The deployment of a Data Catalog only takes a few days, rather than months, to produce value.

The services rendered by a Data Catalog are simple. In its leanest form, a Data Catalog presents as a search bar, in which any user can type in a few keywords (or even pose a question in a natural language) and obtain a list of results with the first 5 elements being the most relevant, thus providing him with all the information he needs to use the data (just like a web search engine, or an online retailer).

This ease of use is crucial to guarantee adoption by the data teams. On the user front, the Data Catalog should be a simple affair with a clean design. Like any other search or recommendations engine, however, the underlying complexity is substantial.

The good news for the customer is that this complexity is nothing for you to worry about, it’s on us.

Zeenea has invested enormously on the structure of the information (building a knowledge graph), on automation and on the search and recommendations engine. This complexity isn’t visible but it is what constitutes the value of a Data Catalog.

The obsession for simplicity is at the heart of our values. Each functionality we choose to add to the product has to tick one of the two boxes below:

• Does this functionality help deploy the catalog faster in the organization?
• Does this functionality enable the data teams to find the information more quickly in order to get on with their projects?

If neither of the questions above are answered by yes, the functionality will be discarded.

The result is that you can connect Zeenea to your operational systems, configure and feed your first metamodel, and open the catalog to the end users within a matter of days.

Of course, going forward, you’ll need to complete the metamodel, integrate other sources, etc. But the value creation is instant.

Take Away

In line with the search for simplicity, the Data Catalog need not be an expensive solution.

This is true for the implementation costs – a Data Catalog deployment does not require thousands of work hours.

We offer a deployment program between 3 to 6 weeks, that includes the onboarding, integration, and the creation of a first metamodel for 3,000 Euros. This also applies for the software costs, forget 6 figure bills!

Our starting price is 18,000 Euros per year, for 5 data stewards, 50 data consumers, and 3 types of connectors.

Download our eBook: The 7 lies of Data Catalog Providers for more!

A Data Catalog must reply on automation!

Some Data Catalog vendors, who hail from the world of cartography, have developed the rhetoric that automation is a secondary topic, which can be addressed at a later stage.

They will tell you that a few manual file imports suffice, along with a generous user community collaborating on their tool to feed and use the catalog. A little arithmetic is enough to understand why this approach is doomed to failure in a data-centric organization.

An active Data Lake, even a modest one, quickly hoovers up, in its different layers, hundreds and even thousands of datasets. Along with these datasets, can be added those from other systems (database applications, various APIs, CRMs, ERPs, noSQL, etc) which we usually want to integrate in the catalog.

The orders of magnitude quickly go beyond thousands, sometimes tens of thousands of datasets. Each dataset contains dozens of fields. Datasets and fields alone represent several hundreds of thousands of objects (we could also include other assets: ML models, dashboards, reports, etc). In order for the catalog to be useful, inventorying those objects isn’t enough.

You also need to combine with them all the properties (metadata) which will enable end users to find, understand, and exploit these assets. There are several types of metadata: technical information, business classification, semantics, security, sensitivity, quality, norms, uses, popularity, contacts, etc. Here again, for each asset, there are dozens of properties.

Back to the arithmetics: Overall, we are dealing with millions of attributes needing to be maintained.

Such volumes alone should disqualify any temptation to choose the manual approach. But there is more. The stock of informational assets isn’t static. It is constantly growing. In a data-centric organization, datasets are created daily, others are moved or changed.

The Data Catalog needs to reflect these changes.

Otherwise, its content will be permanently obsolete and the end users will reject it. Who is going to trust a Data Catalog that is incomplete and wrong? If you feel that your organization can absorb the load and keep your catalog up to date, that’s wonderful. Otherwise, we would suggest you monitor as quickly as possible the level of automation provided by the different solutions you are looking at.

What can we automate in a Data catalog?  

In terms of automation, the most important capacity is the inventory.

A Data Catalog should be able to regularly scan all your data sources and automatically update the asset inventory (datasets, structures and technical metadata at a minimum) to reflect the day-to- day reality of the hosting systems.  

Believe us: a Data Catalog that cannot connect to your da ta sources will quickly become useless, because its content will always be in doubt.

Once the inventory is completed, the next challenge is to automate the metamodel feed.

Here, beyond the technical metadata, complete automation seems a little hard to imagine. It is still possible to significantly reduce the necessary workload for the maintenance of the metamodel. The value of certain properties can be determined by simply applying rules at the time of the integration of the objects in the catalog.

It is also possible to suggest property values using more or less sophisticated algorithms (semantic analysis, pattern matching, etc.).

Lastly, it’s often possible to feed part of the catalog by integrating the systems that produce or contain metadata. This can apply for instance for quality measurement, for lineage information, for business ontologies, etc.

For this approach to work, the Data Catalog must be open and offer a complete set of APIs that allow the metadata to be updated from other systems.

Take Away

A Data Catalog handles millions of information in a constantly shifting landscape.

Maintaining this information manually is virtually impossible, or extremely costly. Without automation, the content of the catalog will always be in doubt, and the data teams will not use it.

Download our eBook: The 7 lies of Data Catalog Providers for more!

For the last couple of years, the Zeenea international sales team has been in contact with prospective clients the world over, presenting Zeenea, running product demos, submitting RFPs (mostly against the big guns in the industry), advising CDOs on data management strategies, accompanying existing customers with their catalog deployment, helping address technical challenges when they arise and always discussing pricing…

Anyone who has been actively in the market for data catalogs, pricing will feel like a complex affair which varies widely depending on use cases, providers (SaaS, On-prem, annual subscription etc.), number of users, etc. 

This blog post seeks to demonstrate that putting a price tag on a data catalog should not really be a complex affair and while the technology that makes a catalog tick can be quite impressive (especially if its powered by a knowledge graph), its ultimate purpose is to help data users access, curate and leverage information that already exists. Nothing more nothing less.  

Below, we’ll relate a couple of fairly typical examples of Zeenea’s approach to pricing which resulted in customer wins for us, and money saved for the customer.

Avoid the quagmire of maximum cost/minimal adoption.

One telling example of the financial repercussions of mismanaged data catalog adoption occurred with a large French bank (Zeenea is a Paris-based startup).

This bank had initially chosen a well known data catalog provider to help with the management, governance and quality of their data. Like all financial institutions, this one was subject to BCBS 239 and compliance was therefore a key issue. The catalog provider, whose platform boasts a very sophisticated predefined metamodel with tons of features, therefore set out to help the customer organize its data governance around BCBS 239. Unfortunately, the project quickly began to turn sour for a couple of reasons:  

1-  The “out of the box” metamodel, it turned out, didn’t readily fit with the organization of the data teams. The predictable consequence of this mismatch was having to allocate far more internal resources than planned to handle their compliance initiative.

2- The focus on Data Governance didn’t really deal with the needs of business teams looking for easy access to their datasets. This unhappy situation had a negative impact on the overall adoption of the data catalog and end users simply stopped using it. 

By the time the client stopped the initiative, there were a very limited number of compliance experts actually using the catalog for a bill of a few million euros per year…Suffice it to say that the purse holders were not entirely satisfied with the cost-benefit ratio…

The lesson here is that in order to successfully implement a data catalog long term, one needs early and durable user adoption. A simple, well defined, use case with a manageable number of stakeholders will go a long way to achieving that whilst ensuring costs are kept within reasonable limits. This is precisely what Zeenea did with this customer.

The bill was reduced 10 fold and catalog adoption of our Explorer platform tripled.  

Don’t crack a walnut with a (costly) sledgehammer.

Another customer (an SME from the UK in the retail sector), whose data landscape consists of a BI tool, PostgresQL, an ETL platform and PowerBI, reached out to us having already received quotes from other well established data catalog providers. 

Their use case was straightforward: The data engineering team needed to clean up their data lake after years of neglect and the data users in the group, analytics folk mostly, wanted to access data assets more easily. 

I’m paraphrasing but the overriding sentiment was that the solutions they had looked at before consulting Zeenea, whilst undoubtedly useful for larger organisations with complex data landscapes, data governance and compliance imperatives, had too many features which were irrelevant to their actual requirements.

The customer was eventually sent a 6 figure quote which the CFO promptly discarded.

Today this mismatch is the most common one we come across, especially amongst small data teams with more straightforward use cases for a data catalog. Unlike the French bank mentioned above, most SMEse cannot afford to spend, let alone lose, vast amounts of money on a failed data experiment. 

So how much should a data catalog really cost…?

Early in 2020, we decided to take a different approach to pricing, one that was coherent with our “Start small, Scale fast” approach to data cataloging adoption. This pricing model, presented as a “Basic Data Discovery”, was designed to encourage data teams to start their data cataloging journey with a straightforward use case, roll out the catalog to other users/projects incrementally, thus ensuring maximum catalog adoption and, crucially, keep a handle on cost. 

Our basic Data Discovery conditions are simple (and easy to deliver on) – This offer includes the POC, 2 connectors, 2 data stewards, 20 data explorers and of course customer support.

Provide us with…

• A Use Case for the POC.
• The Data sources you need to connect to (up to 2 for the Basic Data Discovery).
• The number of Data Stewards needed (up to 2 for the Basic Data Discovery).
• The number of Data Explorers needed (up to 20 for the Basic Data Discovery). 

…and Zeenea will provide you with a quote your CFO can depend on. It really is that simple.*

Click here for more information, or to request a POC.

*These conditions were not chosen at random. In our experience, data teams looking to roll out a cataloging solution seldom choose a use case requiring more than 2 data connectors and a handful of data stewards, and we don’t necessarily recommend that they do.

A Data Catalog is NOT a Business Modeling Solution

Some organizations, usually large ones, have invested for years in the modeling of their business processes and information architecture.

They have developed several layers of models (conceptual, logical, physical) and have put in place an organization that helps the maintenance and sharing of these models with specific populations (business experts and IT people mostly).

We do not question the value of these models. They play a key role in the urbanization, the schema blueprints, the IS management, as well as regulatory compliance. But we seriously doubt that these modeling tools can provide a decent Data Catalog.

There is also a market phenomenon at play here: certain historical business modeling players are looking to widen the scope of their offer by positioning themselves on the Data Catalog market. After all, they do already manage a great deal of information on physical architecture, business classifications, glossaries, ontologies, information lineage, processes and roles, etc. But we can identify two major flaws in their approach.

The first is organic. By their nature, modeling tools produce top-down models to outline the information in an IS. However accurate it may be, a model remains a model: a simplified representation of reality.

They are very useful communication tools in a variety of domains, but they are not an exact reflection of the day-to-day operational reality which, for me, is crucial to keeping the promises of a Data Catalog (enabling teams to find data, understanding and knowing how to use the datasets).

The second flaw?: It is not user -friendly.

A modeling tool is complex and handles an important number of abstract concepts which require an important learning curve. It’s a tool for experts.

We could consider improving user friendliness of course to open it up to a wider audience. But the built-in complexity of the information won’t go away.

Understanding the information provided by these tools requires a solid understanding of modeling principles (object classes, logical levels, nomenclatures, etc). It is quite a challenge for data teams and a challenge that seems difficult to justify from an operational perspective.

The truth is, modeling tools that have been turned into Data Catalogs are faced with important adoption issues with the teams (they have to make huge efforts to learn how to use the tool, only to not find wha t they are looking for).  

A prospective client recently presented us with a metamodel they had built and asked us whether it was possible to implement it in Zeenea. Derived from their business models, the metamodel had several dozen classes of objects and thousands of attributes. To their question, the official answer was yes (the Zeenea metamodel is very flexible). But instead, we tried to dissuade them from taking that path: A metamodel that sophisticated ran the risk, in our opinion, of losing the end users, and turning the Data Catalog project into a failure…

Should we therefore abandon business models when putting a Data Catalog in place? Absolutely not.

It must, however, be remembered that business models are there to handle some issues, and the Data Catalog other issues. Some information contained within the models help structure the catalog and enrich its content in a very useful way (for instance responsibilities, classifications, and of course business glossaries).

The best approach is therefore, in our view, to conceive the catalog metamodel by focusing exclusively on the added value to the data teams (always with the same underlying question: does this information help find, localize, understand, and correctly use the data?), and then integrating the modeling tool and the Data Catalog in order to automate the supply of certain elements of the metamodel already present in the business model.

Take Away

 As useful and complete as they may be, business models are still just models: they are an imperfect reflection of the operational reality of the systems and therefore they struggle to provide a useful Data Catalog.

Modeling tools, as well as business models, are too complex and too abstract to be adopted by data teams. Our recommendation is that you define the metamodel of your catalog with a view to answering the questions of the data teams and supply some aspects of the metamodel with the business model.

Download our eBook: The 7 lies of Data Catalog Providers for more!

A Data Catalog is NOT a Query Solution

Here is another oddity of the Data Catalog market. Several vendors, whose initial aim was to allow users to query simultaneously several data sources, have “pivoted” towards a Data Catalog positioning on the market.

There is a reason for them to pivot.

The emergence of Data Lakes and Big Data have cornered them in a technological cul-de-sac that has weakened the market segment they were initially in.

A Data Lake is typically segmented into sever al layers. The “raw” layer integrates data without transformation, in formats that are more or less structured and in great quantities; A second layer, which we’ll call “clean”, will contain roughly the same data but in normalized formats, after a dust down. After that, there can be one or sever al “business” layers ready for use: A data warehouse and visualization tool for analytics, a Spark cluster for data science, a storage system for commercial distribution, etc. Within these layers, data is transformed, aggregated and optimized for use, along with the tools supporting this use (data visualization tools, notebooks, massive processing, etc).

In this landscape, a universal self-service query tool isn’t suitable.

It is of course possible to set up an SQL interpretation layer on top of the “clean” layer (like Hive) but query execution remains a domain for specialists. The volumes of data are huge and rarely indexed. 

Allowing users to define their own queries is very risky: On on-prem systems, they run the risk of collapsing the cluster by running a very expensive query. And on the Cloud, the bill could run very high indeed. Not to mention security and data sensitivity issues.

As for the “business” layers, they are generally coupled with more specialized solutions (such as a combination of Snowflake and Tableau for analytics) that offer very complete and secured tooling, offering great performance for self-service queries. With their market space shrinking like snow in the sun, some multi-source query vendors have pivoted towards Data Catalogs.

Their pitch is now to convince customers that the ability to execute queries makes their solution the Rolls-Royce of Data Catalogs (In order to justify their six-figure pricing). We would invite you to think twice about it…

Take Away

On a modern data architecture, the capacity to execute queries from a Data Catalog isn’t just unnecessary, it’s also very risky (performance, cost, security, etc.).

Data teams already have their own tools to execute queries on data, and if they haven’t, it may be a good idea to equip them. Integrating data access issues in the deployment of a catalog is the surest way to make it a long, costly, and disappointing project.

Download our eBook: The 7 lies of Data Catalog Providers for more!