Are you concerned about data quality? If so, you should be concerned about data normalization. Data normalization consists of transforming data without distorting it, so that it corresponds to a predefined and constrained set of values to improve its efficiency.

Discover the importance of this technique, which has become indispensable for data-driven companies.

As with any company that turns to data to improve its productivity and efficiency, or the relevance of its offer or its approach to its market, data representativeness is crucial. Your challenge is to maximize the intelligence derived from your data. To achieve this, you need to do everything in your power to limit the distortion of information. This is the vocation of data normalization.

Data normalization is commonly used in statistics, data science, and machine learning to scale the values of different variables within the same interval. The main company objectives in normalization are to make data comparable with each other and to make them more easily interpretable by analysis and modeling algorithms.

Why is data normalization important for companies?

In many cases, data can have very different scales, i.e. some variables may have much larger or smaller values than others. This can pose problems for certain statistical techniques or machine learning algorithms, as they can be sensitive to the scale of the data. Normalization solves this problem by adjusting variable values to lie within a specified interval, often between 0 and 1, or around the mean with a given standard deviation.

What are the benefits associated with data normalization?

Data normalization improves the quality, performance, and interpretability of statistical analyses and machine learning models by eliminating problems associated with variable scaling and enabling fairer comparisons between different data characteristics. In practice, this translates into concrete benefits:

Maximum comparability: Normalized data are scaled to the same level, enabling easier comparison and interpretation between different variables.

Optimized machine learning: Normalization facilitates faster convergence of machine learning algorithms by reducing the scale of variables, helping to achieve more reliable and consolidated results more quickly.

Enhanced model stability: Normalization reduces the impact of extreme values (outliers), making models more stable and resistant to data variations.

Improved interpretability: Data normalization facilitates the interpretation of coefficients, making analysis more comprehensible.

What methods are used to normalize data?

There are several methods of data normalization, but two stand out from the crowd starting with the Min-Max Scaling method. It is based on the principle of scaling the values of a variable so that they fall within a specified interval, usually between 0 and 1. This technique is particularly useful when you want to retain the linear relationship between the original values.

Another method, called Z-Score normalization, is a more standardization-oriented technique. It transforms the values of a variable so that they have a mean of 0 and a standard deviation of 1. Unlike Min-Max normalization, standardization does not impose a specific upper or lower limit on the transformed values. This technique is recommended when variables have very different scales, as it allows data to be centered around zero and scaled with respect to standard deviation.

Other methods may also be considered for data normalization, but these are more marginal. Decimal Scaling and Unit Vector Scaling are two examples.

Decimal normalization involves dividing each value of a variable by a power of 10, depending on the number of significant digits. This moves the decimal point to the left, placing the most significant digit to the left of the decimal. This technique adjusts the values to lie within a smaller interval, thus simplifying calculations.

Unit vector normalization is used in machine learning. It consists in dividing each value of a data vector by the Euclidean norm of the vector, thus transforming the vector into a unit vector (of length 1). This technique is often used in algorithms that calculate distances or similarities between vectors.

What’s the difference between data normalization and data standardization?

Data normalization and data standardization address the same issue of data representativeness but from different perspectives. Although they are both data scaling techniques, they differ in the way they transform variable values.

Data standardization transforms the values of a variable so that they have a mean of 0 and a standard deviation of 1. Unlike normalization, standardization does not set a specific range for the transformed values. Standardization is useful when variables have very different scales and allows data to be centered around zero and scaled with respect to standard deviation, which can facilitate the interpretation of coefficients in some models. Depending on the nature of your data and the lessons you wish to learn from it, you may need to use either data normalization or data standardization.

In order to stand out from your competitors, innovate, and offer personalized products and services, collecting data is essential. However, managing data isn’t a walk in the park: every day, small problems can affect their quality. Incomplete or inaccurate data, security problems, hidden data, duplicates, inconsistencies, or inaccuracies, and the list goes on.

Here is an overview of the most common data quality related issues and some best practices to use to curb them for good!

The risks associated with poor data quality

As it’s been said over and over again, when it comes to data, the real issue is not the quantity of data but its quality. Data Quality Management (DQM) is a demanding discipline that relies on the endless questioning of data processes and constant surveillance of the very nature of the information that constitutes your data assets. Poor data quality can directly translate into lower revenues and higher operational costs, potentially resulting in financial losses for your company.

When data quality is degraded, analyses, projections, forecasts, and even decisions can be distorted. And the greater the volume of degraded data is, the greater the gap between reality and your understanding of reality is. Ensuring data quality starts with a good understanding of the errors that can affect it.

The most common data quality issues

Ensuring data quality is a key topic for any company that bases its development strategy on data. To carry out targeted actions, you need to prioritize tasks and not spread yourself too thin. Data Quality Management consists in identifying all the erroneous information that could distort your decision-making. This erroneous data can be classified into four categories.

Duplicate data

When data is duplicated, it means that the same information is present multiple times in the same database or file. Data duplication is hence one of the most harmful issues because it is often difficult to detect. Beyond 5% of duplicated data, it is considered that the quality of the data starts to be degraded. For example, CRM tools often generate duplicate data, because their users sometimes add contacts without checking their presence in the database.

Hidden data

On a daily basis, your business generates an increasing amount of data. Very often, you only leverage a limited portion of the available information. The rest of the data produced by your business gets scattered and diluted in data silos. It then remains permanently untapped. For example, a customer’s purchase history is not always available to customer service teams. Yet, this information would allow them to better identify the customer’s profile and therefore, provide more relevant answers to their specific requests, or even upsell or cross-sell by making adapted suggestions.

Inconsistent data

Are John Smith and Jon Smith really two different customers? Inconsistent data significantly affects data quality. It can also be created by another well-known phenomenon: redundancy. This phenomenon occurs when you work with multiple sources (including third-party data) in addition to your own data. Discrepancies in data formats, units, or even spelling must be tracked in a data quality approach.

Inaccurate data

It may seem obvious, but inaccurate data is probably one of the worst issues that can undermine data quality. When customer data is inaccurate, any personalized experience will not be relevant. For example, if your data inventory is inaccurate, supply difficulties or storage costs can skyrocket. Whether it’s incorrect contact information or missing or empty fields, you need to do everything you can to eradicate inaccurate data.

How to solve data quality problems

While common sense often presides over good data quality management, they are not enough to ensure it.

To meet these challenges and solve your data quality issues, you’ll need a Data Quality Management tool. But in order to choose the right solution, you will need to start by mapping your data assets in order to identify and evaluate their actual quality. Deploying a Data Quality Management solution, data governance, training, and raising awareness of your teams to good data management… are all essential pillars to limit data quality-related issues!

To learn more about DQM, feel free to download our Guide to Data Quality Management 

Relying simply on intuition is no longer possible – to gain a competitive advantage, it is essential to elevate your data-driven strategy. With Data Ingestion, you can access information faster and more efficiently by centralizing it in a single location. Here is an overview.

In a hyper-competitive and ever-changing business world, companies are in a race against time. A race that does not necessarily oppose their direct competitors but rather their customers. The challenge is to identify consumer trends to anticipate their expectations. Being the first to satisfy a given need or to enter an emerging market… These strategic conditions can be met through Data Ingestion.

Data Ingestion allows you to have an even finer knowledge of your customers and your market through the exploitation of increasingly heterogeneous data, and identify weak signals in order to anticipate trends quickly, but above all efficiently.

Understanding Data Ingestion

The principle of Data Ingestion is based on the idea of centralizing different sources of data in one place. By nature, this heterogeneous data needs to be meticulously cleaned and deduplicated in order to be brought together in a target environment for processing and exploitation. Whether your data comes from a data lake, customer files, or SaaS applications, it can be aggregated within a target site in order to reconcile it with one goal: to improve the understanding of a market, an ecosystem, or a target.

The term reconciliation perfectly sums up the purpose of Data Ingestion. The aim is to combine the knowledge contained in different types of databases to maximize the lessons learned.

What are the main benefits of data ingestion?

If you decide to embark on a Data Ingestion project, you will be able to reap various benefits. First, you will inevitably gain responsiveness and flexibility. Indeed, Data Ingestion tools are able to manage and process not only very large volumes of data but also a wide range of data types, including unstructured data. Data Ingestion also promises simplicity. With its ability to reconcile disparate information sources, Data Ingestion makes it much easier to extract data and restructure it into predefined formats to make it more usable.

The information that Data Ingestion gives access to can then be leveraged within advanced analytical tools. Maximizing the benefits of this in-depth knowledge of your customers or market will feed BI tools and make it easier for you to take a step back and define new strategic directions. Indeed, Data Ingestion contributes to simplifying access to data for your employees.

A more developed data culture also means faster and more informed decision-making and, consequently, a competitive advantage in defining more effective tactical and strategic levers.

What are the challenges of successful data ingestion?

Data Ingestion remains demanding, and a certain number of conditions need to be met to deliver its full potential. For example, ingesting very large volumes of very different data can raise data quality issues that can not only degrade the relevance of analyses but also lengthen processing times. In addition, the diversity of data sources mechanically increases exposure to vulnerabilities.

More complexity and more exposure to risk mechanically lead to a risk of increased processing costs. To succeed in a data ingestion project, you need to be aware of these risks in order to know how to protect yourself against them…

How to succeed in a Data Ingestion project?

The first piece of advice for an effective Data Ingestion project is to anticipate. Your ability to anticipate risks and difficulties depends on the proper mapping of your data assets.

Another lever is to activate automation. The volumes of data processed by data management are so massive that manual operations must be limited to a minimum. Automating the processing of information also has the advantage of providing more consistency in the structure of your data.

Finally, to maximize the chances of success of your Data Ingestion project, you can also consider opting for real-time Data Ingestion.

Also known as Streaming Data Ingestion, it is particularly suitable when you are looking to constantly update the knowledge you have of a market. This real-time ingestion provides a key answer to real-time decision-making issues.

Data Quality refers to an organization’s ability to maintain the quality of its data in time. If we were to take some data professionals at their word, improving Data Quality is the panacea to all our business woes and should therefore be the top priority. 

At Zeenea, we believe this should be nuanced: Data Quality is a means amongst others to limit the uncertainties of meeting corporate objectives. 

In this series of articles, we will go over everything data professionals need to know about Data Quality Management (DQM):

1. The nine dimensions of Data Quality
2. The challenges and risks associated with Data Quality
3. The main features of Data Quality Management tools
4. The Data Catalog contribution to DQM

One way to better understand the challenges of Data Quality is to look at the existing Data Quality solutions on the market.

From an operational point of view, how do we identify and correct Data Quality issues? What features do Data Quality Management tools offer to improve Data Quality?

Without going into too much detail, let’s illustrate the pros of a Data Quality Management tool through the main evaluation criteria of Gartner’s Magic Quadrant for Data Quality Solutions.

Connectivity

A Data Quality Management tool has to be able to gather and apply quality rules on all enterprise data (internal, external, on-prem, cloud, relational, non-relational, etc.). The tool must be able to plug into all relevant data in order to apply quality rules.

Data profiling, data measuring, and data visualization

You cannot correct Data Quality issues if you cannot detect them first. Data profiling enables IT and business users to assess the quality of the data in order to identify and understand the Data Quality issues. 

The tool must be able to carry out what is outlined in The nine dimensions of data quality to identify quality issues throughout the key dimensions for the organization.

Monitoring

The tool must be able to monitor the evolution of the quality of the data and warn management at a certain point.

Data standardization and data cleaning

Then comes the data cleaning phase. The aim here is to provide data cleaning functionalities in order to enact norms or business rules to alter the data (format, values, page layout).

Data matching and merging

The aim is to identify and delete duplicates that can be present within or between datasets.

Address validation

The aim is to standardize addresses that could be incomplete or incorrect. 

Data curation and enrichment

The capabilities of a Data Quality Management tool are what enable the integration of data from external sources and improve completeness, thereby adding value to the data.

The development and putting in place of business rules

The capabilities of a Data Quality Management tool are what enable the creation, deployment, and management of business rules, which can then be used to validate the data.

Problem resolution

The quality management tool helps both IT and business users to assign, escalate, solve, and monitor Data Quality problems.

Metadata management

The tool should also be capable of capturing and reconciling all the metadata related to the Data Quality process.

User-friendliness

Lastly, a solution should be able to adapt to the different roles within the company, and specifically to non-technical business users.

Data Quality refers to an organization’s ability to maintain the quality of its data in time. If we were to take some data professionals at their word, improving Data Quality is the panacea to all our business woes and should therefore be the top priority. 

At Zeenea, we believe this should be nuanced: Data Quality is a means amongst others to limit the uncertainties of meeting corporate objectives. 

In this series of articles, we will go over everything data professionals need to know about Data Quality Management (DQM):

1. The nine dimensions of Data Quality
2. The challenges and risks associated with Data Quality
3. The main features of Data Quality Management tools
4. The Data Catalog contribution to DQM

The challenges of Data Quality for organizations

Initiatives for improving the quality of the data are usually put in place by organizations to meet the conformity requirements and risk reduction. They are indispensable for reliable decision-making. There are unfortunately many stumbling blocks that can hinder Data Quality improvement initiatives. Below are some examples:

• The exponential growth of the volume, speed, and variety of the data make the environment more complex and uncertain;
• Increasing pressure from conformity regulations such as GDPR, BCBS 239, or HIPAA;
• Teams are increasingly decentralized, and each have their own domain of expertise;
• IT and data teams are snowed under and don’t have time to solve Data Quality issues;
• The data aggregation processes are complex and long;
• It can be difficult to standardize data between different sources;
• Change audits among systems are complex;
• Governance policies are difficult to implement.

Having said that, there are also numerous opportunities to grab. High-quality data enables organizations to facilitate innovation with artificial intelligence and ensure a more personalized customer experience. Assuming there is enough quality data. 

Gartner has actually forecasted that until 2022, 85% of AI projects will produce erroneous data as a result of bias in the data, algorithms, or from teams in charge of data management.

Reducing the level of risk by improving the quality of the data

Poor Data Quality should be seen as a risk and quality improvement software as a possible solution to reduce this level of risk.

Processing a quality issue:

If we accept the notion above, any quality issue should be addressed in several phases:

1. Risk identification: this phase consists in seeking out, recognizing, and describing the risks that can help/prevent the organization from reaching its objectives – in part because of a lack of Data Quality.

2. Risk Analysis: the aim of this phase is to understand the nature of the risk and its characteristics. It includes factors for event similarities and their consequences, the nature, and importance of these consequences, etc. Here, we should seek to identify what has caused the poor quality of the marketing data. We could cite for example:

• A poor user experience of the source system leading to typing errors;
• A lack of verification of the completeness, accuracy, validity, uniqueness, consistency, or timeliness of the data;
• A lack of simple means to ensure the traceability, clarity, and availability of the data;
• The absence of a governance process and the implication for business teams.

3. Risk evaluation: the purpose of this phase is to compare the results of the risk analysis with the established risk criteria. It helps establish whether further action is needed for the decision-making – for instance keeping the current means in place, undertaking further analysis, etc.

Let’s focus on the nine dimensions of Data Quality and evaluate the impact of poor quality on each of them:

The values for the levels of probability and severity should be defined by the main stakeholders, who know the data in question best. 

4. Risk processing: this processing phase aims to set out the available options to reduce risk and roll them out. This processing also involves the ability to assess the usefulness of the actions taken, determining whether the residual risk is acceptable or not – and in this last case – consider further processing.

Therefore, improving the quality of the data is clearly not a goal in itself:

• Its cost must be evaluated based on company objectives;
• The treatments to be implemented must be evaluated through each dimension of quality.

Data Quality refers to an organization’s ability to maintain the quality of its data in time. If we were to take some data professionals at their word, improving Data Quality is the panacea to all our business woes and should therefore be the top priority. 

At Zeenea, we believe this should be nuanced: Data Quality is a means amongst others to limit the uncertainties of meeting corporate objectives. 

In this series of articles, we will go over everything data professionals need to know about Data Quality Management (DQM):

1. The nine dimensions of Data Quality
2. The challenges and risks associated with Data Quality
3. The main features of Data Quality Management tools
4. The Data Catalog contribution to DQM

Some definitions of Data Quality

Asking Data Analysts or Data Engineers for a definition of Data Quality will provide you with very different answers – even within the same company, amongst similar profiles. Some, for example, will focus on the unity of data, while others will prefer to reference standardization. You may yourself have your own interpretation.

The ISO 9000-2015 norm defines quality as “the capacity of an ensemble of intrinsic characteristics to satisfy requirements”. 

DAMA International (The Global Data Management Community) – a leading international association involving both business and technical data management professionals – adapts this definition to a data context: “Data Quality is the degree to which the data dimensions meet requirements.”

The dimensional approach to Data Quality

From an operational perspective, Data Quality translates into what we call Data Quality dimensions, in which each dimension relates to a specific aspect of quality. 

The 4 dimensions most often used are generally completeness, accuracy, validity, and availability. In literature, there are many dimensions and different criteria to describe Data Quality. There isn’t however any consensus on what these dimensions actually are.

For example, DAMA enumerates sixty dimensions – when most Data Quality Management (DQM) software vendors usually offer up five or six.

The nine dimensions of Data Quality

At Zeenea, we believe that the ideal compromise is to take into account nine Data Quality dimensions: completeness, accuracy, validity, uniqueness, consistency, timeliness, traceability, clarity, and availability.

We will illustrate these nine dimensions and the different concepts we refer to in this publication with a straightforward example:

Arthur is in charge of sending marketing campaigns to clients and prospects to present his company’s latest offers. He encounters, however, certain difficulties:

• Arthur sometimes sends communications to the same people several times,
• The emails provided in his CRM are often invalid,
• Prospects and clients do not always receive the right content,
• Some information pertaining to the prospects are obsolete,
• Some clients receive emails with erroneous gender qualifications,
• There are two addresses for clients/prospects but it’s difficult to understand what they relate to,
• He doesn’t know the origin of some of the data he is using or how he can access their source.

Below is the data Arthur has at hand for his sales efforts. We shall use them to illustrate each of the nine dimensions of Data Quality:

1. Completeness

Is the data complete? Is there information missing? The objective of this dimension is to identify the empty, null, or missing data. In this example, Arthur notices that there are missing email addresses:

To remedy this, he could try and identify whether other systems have the information needed. Arthur could also ask data specialists to manually insert the missing email addresses.

2. Accuracy

Are the existing values coherent with the actual data, i.e., the data we find in the real world?

Arthur noticed that some letters sent to important clients are returned because of incorrect postal addresses. Below, we can see that one of the addresses doesn’t match the standard address formats in the real world:

It could be helpful here for Arthur to use postal address verification services.

3. Validity

Does the data conform with the syntax of its definition? The purpose of this dimension is to ensure that the data conforms to a model of a particular rule.

Arthur noticed that he regularly gets bounced emails. Another problem is that certain prospects/clients do not receive the right content because they haven’t been accurately qualified. For example, the email address annalincoln@apple isn’t in the correct format and the Client Type Csutomer isn’t correct.

To solve this issue, he could for example make sure that the Client Type values are part of a list of reference values (Customer or Prospect) and that email addresses conform to a specific format.

4. Consistency

Are the different values of the same record in conformity with a given rule? The aim is to ensure the coherence of the data between several columns.

Arthur noticed that some of his male clients complain about receiving emails in which they are referred to as Miss. There does appear to be an incoherence between the Gender and Title columns for Lino Rodrigez.

To solve these types of problems, it is possible to create a logical rule that ensures that when the id Gender is Male, the title should be Mr.

5. Timeliness

Is the time lapse between the creation of the data and its availability appropriate? The aim is to ensure the data is accessible in as short a time as possible.

Arthur noticed that certain information on prospects is not always up to date because the data is too old. As a company rule, data on a prospect that is older than 6 months cannot be used.

He could solve this problem by creating a rule that identifies and excludes data that is too old. An alternative would be to harness this same information in another system that contains fresher data.

6. Uniqueness

Are there duplicate records? The aim is to ensure the data is not duplicated.

Arthur noticed he was sending the same communications several times to the same people. Lisa Smith, for instance, is duplicated in the folder:

In this simplified example, the duplicated data is identical. More advanced algorithms such as Jaro, Jaro-Winkler, or Levenshtein, for example, can regroup duplicated data more accurately.

7. Clarity

Is understanding the metadata easy for the data consumer? The aim here is to understand the significance of the data and avoid interpretations.

Arthur has doubts about the two addresses given as it is not easy to understand what they represent. The names Street Address 1 and Street Address 2 are subject to interpretation and should be modified, if possible.

Renaming within a database is often a complicated operation and should be correctly documented with at least one description.

8. Traceability

Is it possible to obtain traceability from data? The aim is to get to the origin of the data, along with any transformations it may have gone through.

Arthur doesn’t really know where the data comes from or where he can access the data sources. It would have been quite useful for him to know this as it would have ensured the problem was fixed at the source. He would have needed to know that the data he is using with his marketing tool originates from the data of the company data warehouse, itself sourced from the CRM tool.

9. Availability

How can the data be consulted or retrieved by the user? The aim is to facilitate access to the data.

Arthur doesn’t know how to easily access the source data. Staying with the previous schema, he wants to effortlessly access data from the data warehouse or the CRM tool. 

In some cases, Arthur will need to make a formal request to access this information directly.