Total, one of the 7 “SuperMajor” oil companies, has recently opened their Digital Factory earlier this year in Paris. The Digital Factory will bring together up to 300 different profiles such as developers, data scientists and other digital experts to accelerate the Group’s digital transformation.

More specifically, Total’s Digital Factory aims to develop the digital solutions Total needs to improve its availability and cost operations in order to offer new services to their customers. Their priorities are mainly centered around the management and control of energy consumption, the ability to extend their reach to new distributed energies, as well as provide more environmentally friendly solutions. Total’s ambition is to generate $1.5 billion in value per year for the company by 2025.

During France’s Best Developer 2019 contest, Patrick Pouyanné, Chairman and Chief Executive Officer of Total, stated:

 “I am convinced that digital technology is a critical driver for achieving our excellence objectives across all of Total’s business segments. Total’s Digital Factory will serve as an accelerator, allowing the Group to systematically deploy customized digital solutions. Artificial intelligence (AI), the Internet of Things (IoT) and 5G are revolutionizing our industrial practices, and we will have the know-how in Paris to integrate them in our businesses as early as possible. The Digital Factory will also attract the new talent essential to our company’s future.”

Who makes up the Digital Factory teams?

In an interview with Forbes this past October, Frédéric Gimenez, Chief Digital Officer and Head of the project described how the teams will be structured within Digital Factory. 

As mentioned above, the team will have around 300 different profiles, all working using agile methodologies: managerial lines will be flattened, teams will have great autonomy and development cycles will be short in order to “test & learn” quickly and efficiently. 

Gimenez explains that there will be multiple teams in his Digital Factory:

• Data Studio, which will consist of data scientists. Total’s CDO (Chief Data Officer) will be the one in charge of this team and their main missions will be to acculturate the enterprise to data and manage the data competences of the Digital Factory.

• A pool of developers and agile coaches. 

• Design Studio, that will regroup UX and UI professionals. They will help come up with various creative ideas and will interfere not only at the analysis stage of Total’s business projects but also during the customer journey stages.

• A “Tech Authority” team, in charge of the security and architecture of their data ecosystem, in order to effectively transform their legacy in a digital environment.

• A platform team, in charge of various data storages such as their Cloud environment, their data lake, etc.

• A Product & Value office in charge of managing the Digital Factory portfolio, assessing the value of projects with the business and analyzing all the use cases submitted to the Digital Factory.

• A HR & a general secretariat 

• Product Owners that come from all over the world. They are trained in agile methods on arrival and then immersed in their project for 4 to 6 months. They then accompany the transformation when they return to their jobs. 

These teams will soon be reunited in a 5,500m2 workspace in the heart of Paris in the 2nd arrondissement, an open-space favorising creativity and innovation. 

How governance works at Total’s Digital Factory

Gimenez explained that the business lines are responsible for their Digital Factory use cases. The Digital Factory analyzes the eligibility of their use cases through four criteria:

• Value brought during the 1st iteration and during its scaling up 
• Feasibility (technology / data)
• Customer Appetence / Internal Impact
• Scalability 

An internal committee at the Digital Factory then decides whether or not the use case is taken care of and the final decision is validated by Gimenez himself.  For good coordination with the business lines, the digital representatives in the branches are also located in the Digital Factory. They are responsible for acculturating the business lines and piloting the generation of ideas, but also for ensuring the consistency of their branch’s digital initiatives with the Group’s ambitions, Total calls them Digital Transformation Officers.

First success of Total’s Digital Factory

Digital Factory started this past March and deployed the first squads in April during the Corona virus lockdown in France. In the Forbes interview, Gimenez explained that 16 projects are in progress with a target of 25 squads in permanent operation.

The first two digital solutions will be delivered by the end of this year:

• A tool for Total Direct Energie to assist customers in finding the best payment schedule using algorithms and data

• A logistics optimization solution based on IoT trucks for the Marketing and Services branch, which will be deployed in 40 subsidiaries.

 In addition, Total managed to attract experts such as data scientists (despite a still very limited form of communication such as Welcome to the Jungle or Linkedin) and retain them by offering a diversity of projects.

“We are currently carrying out a first assessment of what has worked and what needs to be improved, we are in a permanent adaptation process.” stated Gimenez.

Digital Factory in the future?

Gimenez ended the Forbes interview by saying that the main reason for his project’s success is the general mobilization that everyone kept despite the sanitary context: “We received more use cases than we are able to deliver (50 projects per year to continuously feed our 25 squads)!”

Otherwise Total has two major KPI sets:

. measuring the proper functioning of the squads by examining the KPIs of their agile methodologies

. tracking the value generated

Are you interested in unlocking data access for your company?

Are you in the manufacturing industry? Get the keys to unlocking data access for your company by downloading our new white paper “Unlock data for the manufacturing industry” 

Digital transformation has become a priority in organizations’ business strategies and manufacturing industries are no exception to the rule! With stronger customer expectations, increased customization demands, and the complexity of the global supply chain, manufacturers are in need to find new, more innovative products and services. In response to these challenges, manufacturing companies are increasingly investing in IoT (Internet of Things). 

In fact, the IoT market has boosted exponentially over the past few years. IDC reports the IoT footprint is expected to grow up to $1.2 trillion in 2022, and Statista, by way of contrast, is confident its economic impact may be between $3.9 and $11.1 trillion by 2025. 

In this article, we define what IoT is and some manufacturing-specific use cases as well as explain why a Zeenea Data Catalog is an essential tool for manufacturers to advance in their IoT implementations.

What is IoT?

A quick definition 

According to Tech Target, the internet of things, (IoT), “is a system of interrelated computing devices, mechanical and digital machines, objects, or people that are provided with unique identifiers and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction.”

A “thing” in the IoT can therefore be a person with a heart monitor implant, an automobile that has built-in sensors to alert the driver when tire pressure is low or any other object that can be assigned an ID and is able to transfer data over a network.

From a manufacturing point of view, IoT is a way to digitize industry processes. Industrial IoT employs a network of sensors to collect critical production data and uses various software to turn this data into valuable insights about the efficiency of the manufacturing operations.

IoT use cases in manufacturing industries

Currently, many IoT projects deal with facility and asset management, security and operations, logistics, customer servicing, etc. Here is a list of examples of IoT use cases in manufacturing:

 Predictive maintenance

For industries, unexpected downtime and breakdowns are the biggest issues. Hence manufacturing companies realize the importance of identifying the potential failures, their occurrences and consequences. To overcome these potential issues, organizations now use machine learning for faster and smarter data-driven decisions.

With machine learning, it becomes easy to identify patterns in available data and predict machine outcomes. This works by identifying the correct data set, combining it with a machine to feed real-time data.This kind of information allows manufacturers to estimate the current condition of machinery, determine warning signs, transmit alerts and activate corresponding repair processes.

With Predictive maintenance through the use of IoT, manufacturers can lower the maintenance costs, lessen the downtime and extend equipment life, thereby enhancing quality of production by attending to problems before equipment fails. 

For instance, Medivators, one of the leading medical equipment manufacturers, successfully integrated IoT solutions across their service and experienced an impressive 78% boost of the service events that could be easily diagnosed and resolved without any additional human resources.

Asset tracking

IoT asset tracking is one of the fastest growing phenomena across manufacturing industries. It is expected that by 2027, there will be 267 million active asset trackers in use worldwide for agriculture, supply chain, construction, mining, and other markets. 

While in the past manufacturers would spend a lot of time manually tracking and checking their products, IoT uses sensors and asset management software to track things automatically. These sensors continuously or periodically broadcast their location information over the internet and the software then displays that information for you to see. This therefore allows manufacturing companies to reduce the amount of time they spend locating materials, tools, and equipment.

A striking example of this can be found in the automotive industry, where IoT has helped significantly in the tracking of data for individual vehicles. For example, Volvo Trucks introduced connected-fleet services that include smart navigation with real-time road conditions based on information from other local Volvo trucks. In the future, more real-time data from vehicles will help weather analytics work faster and more accurately; for example, windshield wiper and headlight use during the day indicate weather conditions. These updates can help maximize asset usage by rerouting vehicles in response to weather conditions.

Another tracking example is seen at Amazon. They are using WiFi robots to scan QR codes on its products to track and triage its orders. Imagine being able to track your inventory—including the supplies you have in stock for future manufacturing—at the click of a button. You’d never miss a deadline again! And again, all that data can be used to find trends to make manufacturing schedules even more efficient.

Driving innovation

By collecting and audit-trailing manufacturing data, companies can better track production processes and collect exponential amounts of data. That knowledge helps develop innovative products, services, and new business models. For example, JCDecaux Asia has developed their displaying strategy thanks to data and IoT. Their objective was to have a precise idea of the interest of the people for the campaigns they carried out, and to attract their attention more and more via animations. “On some screens, we have installed small cameras, which allow us to measure whether people slow down in front of the advertisement or not.”, explains Emmanuel Bastide, Managing Director for Asia at JCDecaux.

In the future, will displaying advertising be tailored to individual profiles? JCDecaux says that in airports, for example, it is possible to better target advertising according to the time of day or the landing of a plane coming from a particular country! By being connected to the airport’s arrival systems, the generated data can send the information to the displaying terminals, which can then display a specific advertisement for the arriving passengers. 

Data catalog: one way to rule data for any manufacturer

To enable advanced analytics, collect data from sensors, guarantee digital security and use machine learning and artificial intelligence, manufacturing industries need to “unlock data,” which means centralizing in a smart and easy-to-use corporate “Yellow Pages” of the data landscape. For industrial companies, extracting meaningful insights from data is made simpler and more accessible with a data catalog.

A data catalog is a central repository of metadata enabling anyone in the company to have access, understand and trust any necessary data to achieve a particular goal.

Zeenea data catalog x IoT: the perfect match

Zeenea helps industries build an end-to-end information value chain. Our data catalog allows you to manage a 360° knowledge base using the full potential of the metadata of your business assets.

Zeenea success story in the manufacturing industry

In 2017, Renault Digital was born with the aim of transforming the Renault Group into a data-driven company.  Today, this entity is made up of a community of experts in terms of digital practices, capable of innovating while delivering agile delivery and maximum value to the company’s business IT projects. In a conference in Zeenea’s Data Centric Exchange (French), Jean-Pierre Huchet, Head of Renault’s Data Lake, states that their main data challenges were: 

• Data was too siloed,
• Complicated data access,
• No clear and shared definitions of data terms,
• Lack of visibility on personal / sensitive data,
• Weak data literacy.

By choosing Zeenea Data Catalog as their data catalog software, they were able to overcome these challenges and more. Zeenea today has become an essential brick in Renault Digital’s data projects. Its success can be translated into :

• Its integration into Renault Digital’s onboarding: mastering the data catalog is part of their training program.
• Resilient documentation processes & rules implemented via Zeenea.
• Hundreds of active users. 

Now, Zeenea is their main data catalog, with Renault Digital’s objectives of having a clear vision of the data upstream and downstream of the hybrid data lake, a 360 degree view on the use of their data, as well as the creation of several thousands of Data Explorers. 

Zeenea’s unique features for manufacturing companies

At Zeenea, our data catalog has the following features to solve your problematics :

• Universal connectivity to all technologies used by leading manufacturers
• Flexible metamodel templates adapted to manufacturers’ use-cases
• Compliance to specific manufacturing standards through automatic data lineage
• A smooth transition in becoming data literate through compelling user experiences 
• An affordable platform with a fast return on investment (ROI) 

Are you interested in unlocking data access for your company?

Are you in the manufacturing industry? Get the keys to unlocking data access for your company by downloading our new white paper “Unlock data for the manufacturing industry” 

The benefits of implementing a Hybrid Cloud architecture

Air France’s initial questions regarding the move to the Cloud were :

• Air France KLM aims to standardize its calculation and storage services as much as possible.
• Not all data is eligible to leave Air France’s premises due to regulations or sensitive data.
• All the tools already used in UDA platforms are available both on-premise and in the public cloud.

Éric says that a hybrid Cloud architecture would allow the firm to have more flexibility to meet today’s challenges:

“Putting our UDA on the Public Cloud would give greater flexibility to the business and more options in terms of data deployment.”

According to Air France, here is the checklist of best practices before migrating to a Hybrid Cloud:

• check if the data has a good reason to be migrated to the Public Cloud
• check the level of sensitivity of the data (according to internal data management policies)
• verify compliance with the UDA implementation guidelines
• verify data stream designs
• configure the right network connection
• for each implementation tool, choose the right level of service management
• for each component, evaluate the locking level and exit conditions
• monitor and forecast possible costs
• Adopt a security model that allows Hybrid Cloud security to be as transparent as possible.
• Extend data governance in the Cloud

Where is Air France today? 

It’s clear that the COVID-19 crisis has completely changed the aviation sector. Every day, Air France has to take the time to understand new passenger behavior and adapt flight schedules in real time, in line with the travel restrictions put in place by various governments. By the end of summer 2020, Air France will have served nearly 170 destinations, or 85% of their regular network. 

Air France’s data architecture has therefore been a key catalyst for the recovery of their airlines:

“a huge thanks to our business users (data scientists) who every day try to optimize services in real time so that they can understand how passengers are behaving in the midst of a health crisis. Even if we are working on artificial intelligence, the human factor is still an essential resource in the success of a data strategy”. 

Conclusion: Monoprix is better in the Cloud

Since May 2020, Monoprix has been managing its data in the Cloud, and it’s been “life changing”. On the business side, there is less latency, queries that used to take hours now take minutes, (and employees are finally sleeping in the morning!). Business analyses are also much deeper, with the possibility of making analyses over five years, which was not possible with the old IT structure. But the most important point is the ability to easily share data with the firm’s partners and service providers.

Damien proudly explains.  “With the old structure, our marketing teams took 15 days to prepare the data and had to send thousands of files to our providers, today they connect in a few minutes and they fetch the data alone, without us having to intervene. That alone is a direct ROI.“ 

It is no secret that after various articles from Gartner, and other famous data and analytics consulting firms that data catalogs are an essential data management solution. Combining artificial intelligence and human skills,  Data Catalogs provide a next-generation workspace for data teams to find, understand and collaborate on their data assets and usages. 

In this article, we focus on the most-used data management solutions to which your enterprise can successfully collaborate with your data catalog software. These vendors have been repeatedly quoted by Gartner and used by many enterprises worldwide. We list the 5 main vendors in the following categories:

• Data Integration
• Data Preparation
• Data Visualization
• Data Governance

Let’s discover this list:

1. Data Integration Vendors

Data integration is the process of combining data from different sources, typically for analysis, business intelligence, reporting, or loading into an application. Data integration tools should be designed to transform, map, and clean data. They can also be integrated with data governance and data quality tools.

The top data integration vendors of 2020 include:

Informatica Data Integration Hub

Informatica’s data integration tools portfolio includes both on-prem and cloud deployments. The vendor combines advanced hybrid integration and governance functionalities with self-service business access for various analytic functions. Informatica touts strong interoperability between its growing list of data management software products.

IBM Infosphere Information Server

IBM offers several distinct data integration tools also both on-prem and cloud deployments, and for virtually every enterprise use case. Its on-prem data integration suite features tools for traditional and modern integration requirements. IBM also offers a variety of prebuilt functions and connectors. The mega-vendor’s cloud integration product is widely considered one of the best in the marketplace, and additional functionality is coming in the months ahead.

SAS Data Management

SAS is one of the largest independent vendor in the data integration tools market. The provider offers its core capabilities via SAS Data Management, where data integration and quality tools are interwoven. It includes query language support, metadata integration, push-down database processing, and various optimization capabilities. 

SAP Data Services

SAP provides on-prem and cloud integration functionality through two main channels. Traditional capabilities are offered through SAP Data Services, a data management platform that provides capabilities for data integration, quality, and cleansing. Integration Platform as a Service features are available through the SAP Cloud Platform.

Oracle Data Integration Cloud Service

Oracle offers a full spectrum of data integration tools for traditional use cases as well as modern ones, in both on-prem and cloud deployments. The company’s product portfolio features technologies and services that allow organizations to full lifecycle data movement and enrichment. 

2. Data Preparation Vendors

As defined in our last article on data preparation, is the process of gathering, combining, structuring and organizing data so it can be analyzed as part of data visualization, analytics and machine learning applications. In other words, it is the process of cleaning and transforming raw data prior to analysis.

The top data preparation vendors of 2020 include:

Alteryx Designer

Alteryx Designer features an intuitive user interface that enables users to connect and cleanse data from data warehouses, cloud applications, spreadsheets, and other sources. Users can leverage data quality, integration and transformation features as well. 

Talend Data Preparation

Talend Data Preparation utilizes machine learning algorithms for standardization, cleansing, and pattern recognition. The product also provides automated recommendations to guide users through the data preparation process. 

IBM Watson Analytics

Together with IBM Watson Machine Learning, IBM Watson Studio is a leading data science and machine learning platform built from the ground up for an AI-powered business. It helps enterprises scale data science operations across the lifecycle–simplifying the process of experimentation to deployment, speeding up data exploration and preparation, as well as model development and training.

Tableau Prep

Tableau Prep empowers more people to get to analysis faster by helping them quickly and confidently combine, shape, and clean their data. A direct and visual experience gives customers a deeper understanding of their data and smart features make data preparation simple.

Trifacta

Trifacta has been ranked as the top vendor in every analyst report published on data preparation to date. A self-service data preparation tool, Trifacta empowers all users, technical or non-technical, to clean & prepare their data efficiently. 

3. Data Visualization Vendors

Data visualization is defined as a graphical representation of data. It is used to help people understand the context and significance of their information by showing patterns, trends and correlations that may be difficult to interpret in plain text form.

The top data visualization vendors of 2020 include:

Tableau

Tableau is a data visualization tool that can be used by data analysts, scientists, statisticians, etc. to visualize the data and get a clear opinion based on the data analysis. Tableau is known for being able to take in data and produce the required data visualization output in a very short time. And it can do this while providing the highest level of security with a guarantee to handle security issues as soon as they arise or are found by users.

Looker

Looker data visualization can go in-depth in the data and analyze it to obtain useful insights. It provides real-time dashboards of the data for more in-depth analysis so that businesses can make instant decisions based on the data visualizations obtained. Looker also provides connections with Redshift, Snowflake, BigQuery, as well as more than 50 SQL supported dialects so you can connect to multiple databases without any issues.

Zoho Analytics

Zoho Analytics helps you create wonderful looking data visualizations based on your data in a few minutes. You can obtain data from multiple sources and mesh it together to create multidimensional data visualizations that allow you to view your business data across departments. In case you have any questions, you can use Zia which is a smart assistant created using artificial intelligence, machine learning, and natural language processing.

Sisense

Sisense provides various tools that allow data analysts to simplify complex data and obtain insights for their organization and outsiders. The solution tries its best to provide various data analytics tools to business teams and data analytics so that they can help make their companies the data-driven companies of the future.

IBM Cognos Analytics

IBM Cognos Analytics is an Artificial Intelligence-based business intelligence platform that supports data analytics. You can visualize as well as analyze your data and share actionable insights with anyone in your organization. Even if you have limited or no knowledge about data analytics, you can use IBM Cognos Analytics easily as it interprets the data for you and presents you with actionable insights in plain language.

4. Data Governance Vendors

We like to define data governance as an exercise of authority over decision-making power (planning, surveillance, and enforcement of rules) and the controls on data management.

In other words, it allows the clear documentation of the different roles and responsibilities around data as well as determining the procedures and the tools supporting data management within an organization.

Cloudera Data Platform

Cloudera Data Platform (CDP) combines the best of Hortonworks’ and Cloudera’s technologies to deliver the industry’s first enterprise data cloud. CDP delivers powerful self-service analytics across hybrid and multi-cloud environments, along with sophisticated and granular security and governance policies that IT and data leaders demand.

Stealthbits

Stealthbits’ Data Access Governance solution discovers where your data lives and then classifies, monitors, and remediates the conditions that make managing data access so difficult in the first place. The result is effective governance that promotes security, compliance, and operational efficiency.

Varonis

Varonis gives you the enterprise-wide visibility you need for effective discovery, auditing, and compliance reporting across a wide variety of regulatory standards. It quickly and accurately classifies sensitive, regulated information stored in on-premises and cloud data stores. Their classification engine prioritizes scans based on risk & exposure to give you actionable results quickly, no matter how much data you have.

Informatica

Informatica provides a quick fix for compliance and data governance which can be implemented on-premise or in the cloud. It offers strong visualization of data lineage and history, master data dashboards for proactive monitoring of data quality and dynamic masking for data security. It also provides the functionality to detect and protect sensitive customer data, managing GDPR data risks, and ensuring contact information is current, accurate and complete.

And finally…

Zeenea

Our data catalog centralizes all data knowledge in a single and easy-to-use interface. Automatically imported, generated, or added by the administrator, data specialists are able to enrich their data assets documentation directly within our tool.

Give meaning to your data thanks to metadata!

If you are interested in getting more information, getting a free personalized demo, or just want to say hi, do not hesitate to contact our team who will get back to you as soon as we’ve received your request 🙂

As it’s been repeatedly said, digital business can not happen without data and analytics at its core. Technology can be a point of failure if not handled properly, but it is often not the most important roadblock to progress. In Gartner’s annual Chief Data Officer survey, the top roadblocks for success were human factors – culture, resources, data literacy and skills. A similar pattern emerges from another study, Gartner’s CEO and Senior Business Executive Survey, where “Talent Management” was listed as the “number one organizational competency to be developed or improved.”

In this article, we would like to focus on the key data and analytics roles & leaders that are essential for enterprises seeking a data-driven organization.

Support roles

Chief Data Officer

The Chief Data Officer, or CDO, is a senior executive responsible for enhancing the quality, reliability and access of data. They are also in charge of creating value from their data assets and from their data ecosystem in general. Through data exploitation and by enabling all forms of business outcomes through analytics, the CDO can produce more value with their enterprise data. There are many variations of the title such as CAO (Chief Analytics Officer), CDAO (Chief Data & Analytics Officer), CDIO (Chief Digital Information Officer), etc.

See more in our article “What is a Chief Data Officer?”

Data & Analytics Manager

As the title implies, the Data & Analytics manager is responsible for managing the data & analytics center and is responsible for its delivery throughout the entire organization. They are a key contributor to the strategy and vision for the data & analytics department, they build the roadmap and are responsible for budget and resource planning. Besides measuring the performance of their analytics team, they are also responsible for tracking the contribution of data analytics in regards to business objectives.

Data Architect

The Data Architect, also referred as the Information Architect, strengthens the impact and proves recommendations on business information. They make the information available and shared across the company by presenting how information assets drive business outcomes. They “own” the data models. They understand the impact various data analytics scenarios on the overall IT architecture (such as data science or machine learning) and work closely with the business department.

Analysts

There isn’t a single type of analysts, but rather a spectrum of analysts. Their roles depend on their use cases and vary by responsibilities and skill requirements. There are data analysts who have a foundational understanding of statistical analytics. They are, or work closely with domain experts to support business areas, processes, or functions.

Project Manager

The project manager is responsible for the successful implementation of all projects in the enterprise portfolio. They plan, execute and deliver projects in accordance with business priorities. Throughout the project’s lifecycle, the project manager tracks their project’s status and manages their teams to limit any risks. They are the primary point of contact for data and analytics initiatives.

Data Roles

Data Engineer

A Data Engineer involves collaboration across business units and IT units and is the practice of making the appropriate data accessible and available to various data consumers (data scientists, data analysts, etc.). They are primarily responsible for building, managing and operationalizing data pipelines in support of data and analytics use cases. Also, they are responsible for leading tedious tasks such as curating datasets created by non-technical users (through self-service data preparation tools for example).

Without data engineers, data & analytics initiatives are more costly, take longer to deploy, and are prone to data quality and availability problems.

Data Steward

Data stewards are the first point reference for data in the enterprise and serve as the entry point to access data. They must ensure the proper documentation of data and facilitate their availability to their users, such as data scientists or project managers for example. Their communication skills enable them to identify the data managers and knowers, as well as to collect associated information in order to centralize them and perpetuate this knowledge within the enterprise. In short, data stewards provide metadata; a structured set of information describing datasets. They transform these abstract data into concrete assets for the profession.

See here for more information on Data Stewards

Analytics roles

Data Scientists

A data scientist is responsible for modeling business processes and discovering insights using statistical algorithms and visualization techniques. They typically have an advanced degree in computer science, statistics or other related fields. Data Scientists contribute to building and developing the enterprise’s data infrastructure and supports the organization with insights and analysis for better decision making. They predict or classify information to develop better action models.

Citizen Data Scientist

Contrary to data scientists, a “Citizen Data Scientist” is not a job title. They are “power business users” who can perform both simple and sophisticated analytical tasks. They can execute a variety of data science tasks, supported by augmented analytics tools for data discovery, data preparation, and model deployment. Potential citizen data scientists will vary based on their skills and interest in data science and machine learning.

See here for more information on citizen data scientists

AI / ML Developer

Artificial intelligence / Machine learning developers are increasingly responsible for enriching applications through the use of machine learning or other AI technologies such as natural language processing, optimization or image recognition. They embed, integrate and deploy AI models that are developed by data scientists or other AI experts either offered by service providers or developed by themselves. Other key skills include identifying and connecting potential data assets, data quality, data preparation and how these are used for model training execution.

Conclusion

The growing importance and strategic significance of data and analytics is creating new challenges for organizations and their data and analytics leaders. Some traditional IT roles are being disrupted by “citizen” roles performed by nontechnical business users. Other new hybrid roles are emerging that cut across functions and departments, and blend IT and business skills.

By putting together these must-have roles, your enterprise is a step closer to becoming data-driven.

Contemporary business initiatives such as digital transformation, are facing an explosion of data volume and diversity. In this context, organizations are looking for more flexibility and agility in their data management.

This is where Cloud strategies come in…

Data management definition

Before we begin, let’s define what data management is. Data Management, as described by TechTarget is “the process of ingesting, storing, organizing and maintaining the data created and collected by an organization”. Data management is a crucial part of an enterprise’s business and IT strategy, and provides analytical help that drives overall decision-making by executives.

As mentioned above, data is seen as a corporate asset that can be used to make better and faster decisions, improve marketing campaigns, increase overall revenue and profits, and above all: innovate. As a result, organizations are seeing cloud technologies as a way to improve their data initiatives.

Cloud strategies are the new black in data management disciplines

It is an undeniable fact that Cloud service providers are becoming the new default platform for database management. This phenomenon provides data management teams with great advantages:

• Cost-effective deployment: Greater flexibility and a more rapid configuration.
• Consumption-based spending: Pay what you use and do not over provision
• Easy maintenance: better control over the associated costs and investments 

By knowing this, there is no doubt that data leaders perceive cloud as a less expensive technology, driving this choice even more.

Data leaders will embrace the cloud as an integral part of their IT landscape in the coming years and months. However, we strongly believe that the rate at which organizations migrate  to the cloud will differ by organization size. Small or midsize organizations will migrate quicker , while larger organizations will take months, even years to migrate. 

Thus, the Cloud is going to become a default option for all data management technologies.  Many strategies appear including various deployment types or approaches. We have identified 3 main strategies:

• Hybrid Cloud: Made up of two or more separate Cloud infrastructures that may be private or public and that remain single entities
• Multicloud: Use more than one cloud service provider infrastructure as well as on-premises solutions.
• Intercloud: Where data is integrated or exchanged between cloud service providers as part of a logical application deployment.

The Cloud is also seen as an opportunity for data analytics leaders

The increased adoption of cloud strategy deployments regarding data management  has important implications for data and analytics strategies. As data is  moving to the cloud, the data and analytics applications they use must follow.

Indeed, the emphasis on the speed of value delivery has made cloud technologies the first choice for new data management solution development for vendors, and deployment for enterprises. Thus, enterprises and data leaders are choosing next-gen data management solutions! They will migrate their assets by selecting applications that connect to future cloud strategies and preparing their teams & budgets for the upcoming challenges they will overcome..

Those data leaders who use analytics, business intelligence (BI) and data science solutions are seeing Cloud solutions as greater opportunities to:

• Use a cloud sandbox environment for trial purposes in terms of onboarding, usages, connectivity and create a prototyping analytics environment before actually buying the solution.
• Facilitate application access wherever you are and improve collaboration between peers.
• Access to new emerging capabilities over time with ease, with continuous delivery approaches.
• Support heavy lifting with the cloud’s elasticity and scalability along the analytics process.

A data catalog, the new essential solution for cloud data management strategies

Data and analytics leaders will inevitably engage in more than one cloud where data management, governance and integration become more complex than ever before. Thus, data leaders must equip their organization to new metadata management solutions to assist in finding and inventorying data distributed across a hybrid and multi cloud ecosystem.  Failure to do so will result in a proliferation of data silos, leading to derailed data management, analytics and data science projects.

Data management teams will have to choose among the wide-range of data catalog in the market the most relevant one.

We like to define a data catalog as a way to create and maintain an inventory of data assets through the discovery, description and organization of distributed datasets.
If you are working on the data catalog project, you will find: 

• On the one hand by fairly old players, initially positioned on the Data Governance market.
  These players provide on premises solutions with rich but complex offers, which are expensive, difficult and time-consuming to deploy and maintain, and are designed for cross-functional governance teams. Their value proposition is focused on control, risk management and compliance.

• on the other hand by suppliers of data infrastructures (Amazon, Google, Microsoft, Cloudera, etc.) or data processing solutions (Tableau, Talend, Qlik, etc.), for which metadata management is an essential block to complete their offer. They offer much more pragmatic (and less costly) solutions, but are often highly technical and limited to their ecosystem.

 We consider those alternatives as not sufficient enough. Here are  some essential guidelines to find your future data catalog. It must:

– Be a cloud data catalog enabling competitive pricing and rapid ROI for your organization.
– Have universal connectivity, adapting to all systems and all data strategies (edge, cloud, multi-cloud, cross-cloud, hybrid).
– Have very advanced automation for the collection and enrichment of data assets as well as their attributes and links (augmented catalog). The automatic feeding mechanisms, as well as the suggestion and correction algorithms reduce the overall cost of the catalog and guarantees the quality of the information it contains
– Be strongly focused on user experience, especially for business users, to improve solution adoption.

To conclude, data management capabilities are becoming more and more cloud-first and in some cases cloud-only.

Data leaders who want to drive innovation in analytics will need to leverage cloud technologies from data assets. They will have to go from ingestion to transformation without forgetting to invest in an efficient data catalog in order to find their way in an ever more complex data world.

Data Ops is a new way to address the deployment of data and analytics solutions.

The success of this methodology is based on techniques that promote faster, more flexible and more reliable data delivery. To deliver on this promise, let’s take a moment and analyze this sentence: “the focus is not just on building the systems right, but also building the right systems”.

There are many different definitions, interpretations, and publications that address DataOps as a concept, but it is much more than just that. It is a way of understanding, discovering, producing analysis, and creating actionable intelligence with data. In a changing world that revolves around data, latencies in data products or their analysis are no longer acceptable.

The entire organization must be put to work to support the deployment and improvement of data and analysis projects!

Data Oops definition

The concept of DataOps emerged in response to the challenges of failing data systems, failing data project implementations, but also the fragility, friction or even fear, when it comes to the use of data. If you are experiencing this situation then don’t look too far… You are in the middle of a Data Oops!

In this context of Data Oops, you will agree that your data teams are struggling to achieve the speed and reliability of directed projects.

The main reasons are that companies have too many roles, are too complex and are constantly changing requirements or objectives making tasks difficult to frame and deliver.

This complexity is exacerbated by a lack of confidence in data, even to the point of “fearing” it. This occurs when we observe limited or inconsistent coordination between the different roles involved in the construction, deployment and maintenance of data flows. We are convinced that an organization that does not know their data are doomed to fail…

How to succeed in your DataOps?

Simply put, DataOps is a collaborative data management practice that aims to improve communication, integration and automation of data flows between data managers and data consumers within an organization. It is based on the alignment of objectives confronted by results. DataOps accepts failure and is built through continuous experimentation.

Here’s a list of principles for successful DataOps:

1. Learn from DevOps, through their techniques for developing and deploying agile applications in your data and analysis work.
2. Identify quantifiable, measurable and achievable business objectives. You will then be able to communicate more regularly, progress towards a common goal and adjust more easily.
3. Start by identifying and mapping your data (type, format, who, when, where, why, etc.) using data catalog solutions.
4. Encourage collaboration between different data stakeholders by providing communication channels and solutions for sharing metadata.
5. Take care of your data, as it may produce value at any given time. Clean it, catalog it, and make it a part of your enterprise’s key assets, whether it is valuable now or not.
6. A model may work well once, but not on the next batch of data. Over-specifying and over-engineering a model will likely not be applicable to previously unseen data or for new circumstances in which the model will be deployed.
7. Maximize your chances of success of introducing a DataOps approach by selecting data and analysis projects that are struggling due to a lack of collaboration or are struggling to keep pace. They will allow you to better demonstrate its value.
8. Keep it agile, short designed, develop, test, release, and repeat! Keep it lean and build on incremental changes. Continuous improvement is found when a culture of experimentation is encouraged and when people learn from their failures. Remember, data science is still science!

In summary, what are the benefits of DataOps?

DataOps helps your business move at the speed of data – keeping pace to deliver the right data. It focuses data activities to be aligned with business objectives, and not on the analytic inputs (big data hype). DataOps also focuses on delivering value from all your data activities, from even the smallest of these can inspire cultural changes needed for other implementations to come.

Adopting DataOps in a culture of experimentation is good data practice and empowers the innovators across the organization to smart small and scale fast.

It is the path to good business practices, and the path that steers you away from Data Oops!

“Within the next year, the number of data and analytics experts in business units will grow at three times the rate of experts in IT departments, which will force companies to rethink their organizational models and skill sets. – Gartner, 2020.

Data & Analytics teams are becoming more and more essential in supporting various complex business processes, and many are challenged with scaling the work they do in delivering data to support their use cases. The pressure to deliver faster and with higher quality is causing data & analytics leaders to rethink how their teams are organized…

Where traditional waterfall models were implemented and used in enterprises in the past, these methodologies are now proving to be too long, too siloed, and too overwhelming!

This is where Data Ops steps in: a more agile, collaborative and change-friendly approach for managing data pipelines.

Data Ops definition

Gartner defines Data Ops as being a “collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organization.”. Basically, making life easier for data users.

Similar to how DevOps, a set of practices that combines software development (Dev) and information-technology operations (Ops), changed the way we deliver software, DataOps uses the same methodologies for teams building data products.

While both agile frameworks, DataOps requires the coordination of data and anyone that works with data across the entire enterprise.

Specifically, data & analytics leaders should implement these key approaches that proved to deliver significant value for organizations:

• Deployment frequency increase: shifting towards a more rapid and continuous delivery methodology enables organizations to reduce the time to market.
• Automated testing: removing time-consuming, manual testing enables higher quality data deliveries.
• Metadata control: tracking and reporting metadata across all consumers in the data pipeline ensures better change management and avoids errors.
• Monitoring: tracking data behavior and the usage of the pipeline enables more rapid identification on both flawed – that needs to be corrected – and good quality data for new capabilities.
• Constant collaboration: communication between data stakeholders on data is essential for faster data delivery.

Who is involved in Data Ops?

Given the importance of data and analytics use cases today, the roles involved in successful data project delivery are more numerous and more distributed than ever before. Ranging from data science teams to people outside of IT, a large number of roles are involved:

• Business analysts,
• Data architects,
• Data engineers,
• Data stewards,
• Data scientists,
• Data product managers,
• Machine Learning developers,
• Database administrators,
• Etc.

As mentioned above, a Data Ops approach requires fluid communication and collaboration across these roles. Each collaborator needs to understand what others expect of them, what others produce, and must have a shared understanding of the goals of the data pipelines they are creating and evolving.

Creating channels through which these roles can work together, such as a collaboration tool, or metadata management solution, is the starting point!

Photo by Gavin Whitner

As the world leader in the music streaming market, it is without question that the huge firm is driven by data.

Spotify has access to the biggest collections of music in the world, along with podcasts and other audio content.

Whether they’re considering a shift in product strategy or deciding which tracks they should add, Spotify says that “data provides a foundation for sound decision making”.

Spotify in numbers

Founded in 2006 in Stockholm, Sweden, by Daniel Ek and Martin Lorentzon, the leading music app’s goal was to create a legal music platform in order to fight the challenge of online music piracy in the early 2000s.

Here are some statistics & facts about Spotify in 2020:

• 248 million active users worldwide,
• 20,000 songs are added per day on their platform,
• Spotify has 40% share of the global music streaming market,
• 20 billion hours of music were streamed in 2015

These numbers not only represent Spotify’s success, but also the colossal amounts of data that is generated each year, let alone each day! To enable their employees, or as they call them, Spotifiers, to make faster and smarter decisions, Spotify developed Lexikon.

Lexikon is a library of data and insights that helps employees find and understand their data and knowledge generated by their expert community.

What were the data issues at Spotify?

In their article How We Improved Data Discovery for Data Scientists at Spotify, Spotify explains that they started their data strategy by migrating data to the Google Cloud Platform, and saw an explosion of their datasets. They were also in the process of hiring many data specialists such as data scientists, analyst, etc. However, they explain that datasets lacked clear ownership and had little-to-no documentation, making it difficult for these experts to find them.

The next year, they released Lexikon, as a solution for this problem.

Their first release allowed their Spotifiers to search and browse through available BigQuery tables as well as discover past researches and analysis. However, months after the launch, their data scientists were still reporting data discovery as a major pain point, spending most of their time trying to find their datasets therefore delaying informed decision-making.

Spotify decided then to focus on this specific issue by iterating on Lexikon, with the unique goal to improve data discovery experience for data scientists.

How does Lexikon data discovery work?

In order for Lexikon to work, Spotify started out by conducting research on their users, their needs as well as their pain points. In doing so, the firm was able to gain a better understanding of their users intent and use this understanding to drive product development.

Low intent data discovery

For example, you’ve been in a foul mood so you’d like to listen to music to lift your spirits. So, you open Spotify, browse through different mood playlists and put on the “Mood Booster” playlist.

Tah-dah! This is an example of low-intent data discovery, meaning your goal was reached without extremely strict demands.

To put this into Spotify’s data scientists context, especially new ones, their low intent data discovery would be:

• find popular datasets used widely across the company,
• find datasets that are relevant to the work my team is doing, and/or
• find datasets that I might not be using, but I should know about.

So in order to satisfy these needs, Lexikon has a customizable homepage to serve personalized recommendations to users. The homepage recommends potentially relevant, automatically generated suggestions for datasets such as:

• popular datasets used within the company,
• dataset recently used by the user,
• datasets widely used by the team the user belongs to.

High intent data discovery

To explain this in simple terms, Spotify uses the example of hearing a song, and researching it over and over in the app until you finally find it, and listen to it on repeat. This is high intent data discovery!

A data scientist at Spotify with high intent has specific goals and is likely to know exactly what they are looking for. For example they might want to:

• find a dataset by its name,
• find a dataset that contains a specific schema field,
• find a dataset related to a particular topic,
• find a dataset that a colleague used of which they can’t remember the name,
• find the top datasets that a team has used for collaborative purposes.

To fulfill their data scientists needs, Spotify focused first on their search experience.

They built a search ranking algorithm based on popularity. By doing so, data scientists reported that their search results were more relevant, and had more confidence in the datasets they discovered because they were able to see which dataset was more widely-used by the company.

In addition to improving their search rank, they introduced new types of properties (schemas, fields, contact, team, etc.) to Lexikon to better represent their data landscape.

These properties are able to open up new pathways for data discovery. In the example down below, a data scientist is searching for a “track_uri”. They are able to navigate through the “track_uri” schema field page and see the top tables containing this information. Since adding this new feature, it has proven to be a critical pathway for data discovery, with 44% of Lexikon users visiting these types of pages.

’

Final thoughts on Lexikon

Since making these improvements, the use of Lexikon amongst data scientists has increased from 75% to 95%, putting it in the top 5 tools used by data scientists!

Data discovery is thus, no longer a major pain point for their Spotifiers.

Sources:

Spotify Usage and Revenue Statistics (2019): https://www.businessofapps.com/data/spotify-statistics/
How We Improved Data Discovery for Data Scientists at Spotify: https://labs.spotify.com/2020/02/27/how-we-improved-data-discovery-for-data-scientists-at-spotify/
75 amazing Spotify Statistics and Facts (2020): https://expandedramblings.com/index.php/spotify-statistics/

Learn more about data discovery solutions in our white paper: “Data Discovery through the eyes of Tech Giants”

Discover the various data discovery solutions developed by large Tech companies, some belonging to the famous “Big Five” or “GAFAM”, and how they helped them become data-driven.

Data life cycle analysis is an element in data management that enterprises are still struggling to implement.

Organizations at the forefront of data innovation such as Uber, LinkedIn, Netflix, Airbnb and Lyft have also seen the value of metadata in the magnitude of this challenge.

They thus developed a metadata management strategy using dedicated platforms. Frequently developed on a custom basis, they facilitate data ingestion, indexing, search, annotation and discovery in order to maintain high quality datasets.

The following examples highlight a shared constant: the difficulty, increased by volume and variety, of transforming business data into exploitable knowledge.

Let’s take a look at the analysis and context of these Web giants:

Uber

Every interaction on Uber’s platform, from their ride sharing services to their food deliveries, is data-driven. Through analysis, their data enables more reliable and relevant user experiences.

Uber’s key stats

• thousands of billions of Kafka messages a day,
• hundreds of petabytes of data in HDFS in data centers,
• millions of analytical queries weekly.

However, the volume of data generated alone is not sufficient to leverage the information it represents; to be used effectively and efficiently, data requires more context to make optimal business decisions.

To provide additional information, Uber therefore developed “Databook”, the company’s internal platform that collects and manages metadata on internal datasets in order to transform data into knowledge.

Databook is designed to enable Uber employees to effectively explore, discover and use Uber’s data. Databook gives context to their data (its meaning, quality, etc) and ensures that it is maintained in its platform for the thousands of employees who want to analyze the data. In short, Databook’s metadata enables data leaders to move from viewing raw data to actionable knowledge.

In the article Databook: Turning Big Data into Knowledge with Metadata at Uber, the article concludes that one of the biggest challenges for Databook was to move from manual metadata repository updates to automation.

Airbnb

At a conference in May 2017, John Bodley, Data Engineer at AirBnB, outlined new issues arising from the company’s growth: a confusing and non-unified landscape that wasn’t allowing access to increasingly important information.
What can we do with all this data collected on a daily basis? How do we turn them into assets for all Airbnb employees?

A dedicated team set out to develop a tool that would democratize access to data within the company. Their work was based both on the knowledge of the analysts and their ability to understand the critical points, and on that of the engineers, who were able to offer a more technical vision. At the heart of the project, interviews of employees concerning their issues were conducted.

What emerged from this survey was a difficulty in finding the information employees needed to work, and a still too tribal approach to sharing and holding information.

To meet these challenges, AirBnB created Data Portal, a metadata management platform. Data Portal centralizes and shares this information via this self-service platform.

Lyft

Lyft is a ride-sharing service and is Uber’s main competitor in the North American market.

The company found they were inefficiently providing data access for its analytical profiles. Its reflections focused on making data knowledge available to optimize its processes. In just a few months, their goal of creating an interface for researching data presented these two major challenges:

• Productivity – Whether it’s to create a new model, instrument a new metric, or perform an ad hoc analysis, how can Lyft use this data in the most productive and efficient way possible?
• Compliance – When collecting data about an organization’s users, how can Lyft comply with increasing regulatory requirements and maintain the trust of its users?

In their article Amundsen – Lyft’s data discovery & metadata engine, Lyft states that the key does not lie in the data, but in the metadata!

Netflix

As the world leader in video streaming, data exploitation at Netflix is, of course, a major strategic focus.

Given the diversity of their data sources, the video platform wanted to offer a way to federate and interact with these assets from a single tool. This search for a solution led to Metacat.

This tool acts as a layer of access to data and metadata from Netflix data sources. It allows its users to access data from any storage system through three different features:

1. Adding business metadata: By hand or user-defined, business metadata can be added via Metacat.
2. Data discovery: The tool publishes schema and business metadata defined by its users in Elasticsearch, facilitating full-text search of information in data sources.
3. Data Change Notification and Auditing: Metacat records and notifies all changes to metadata from storage systems.

In their blog article, “Metacat: Making Big Data Discoverable and Meaningful”, at Netflix, the firm confirms that they are far from finished working on their solution!

There are a few more features they have yet to work on to improve the data warehousing experience:

• Schema and metadata visioning to provide table history.
• Provide contextual information on arrays for better data lineage.
• Add support for datastores like Elasticsearch and Kafka.

Learn more about data discovery solutions in our white paper: “Data Discovery through the eyes of Tech Giants”

Discover the various data discovery solutions developed by large Tech companies, some belonging to the famous “Big Five” or “GAFAM”, and how they helped them become data-driven.

In our last article, we spoke of Uber’s Databook , an in-house platform designed by their very own engineers with the aim to turn data into contextualized assets. In this article, we will focus on Lyft’s very own data discovery and metadata platform: Amundsen.

In response to Uber’s success, the ride-sharing market saw a major wave of competitors arrive and among those, there is Lyft.

Lyft key figures & statistics

Founded in 2012 in San Francisco, Lyft operates in more than 300 cities across the United States and Canada. With over 29% of the US ride-sharing market*, Lyft has certainly secured the second position for itself, standing neck to neck with Uber. Some key statistics on Lyft include:

• 23 million Lyft users as of January 2018,
• More than a billion Lyft rides,
• 1,4 million drivers (Dec. 2017).

And of course, those numbers have transformed into colossal amounts of data to manage! In a modern data-driven company such as Lyft, it is evident that their platform is powered by their data. With the rapid increase of the data landscape, it becomes increasingly difficult to know what data exists, how to access them and what information is available.

This problem led to the creation of Amundsen, Lyft’s open source data discovery solution and metadata platform.

Let’s get to know Amundsen

Named after the Norwegian explorer Roald Amundsen, Lyft improves their data users productivity by providing an intuitive search interface for data, that looks like this:

While Lyft’s data scientists wanted to spend the majority of the time on model development and production, they realized that most of their time was being spent on data discovery. They would find themselves asking questions such as:

• Does this data exist? If it does, where can I find it? Can I access it?
• Who / which team is the owner? Who are the common users?
• Can I trust this data?

To answer these questions, Lyft was inspired by search engines like Google.

As shown above, their entry point is a simple search box where users can type any keyword such as “customers” “employees” or “price”. However, if the data user does not know what they are looking for, the platform presents the user with a list of the most popular tables, so they can browse through them freely.

Some key features:

The search results are shown in “list form” where the description about the table and the date when the table was last updated appears. The ranking used is similar to Google’s Page Rank, where the most popular and relevant tables show up in the first results.

When a data user at Lyft finds what they’re looking for and selects their choice, the user is directed to a detail page which shows the name of the table as well as its manually curated description. Users can also manually insert tags, the owners, and other descriptions. However, a lot of their metadata is automatically curated such as the table’s popularity or even its frequent users.

When in a table, users are able to explore the associated columns to further discover the table’s metadata.

For example, if you were to select the column “distance_travelled” as shown below, you will find a small definition of the field and its related stats such as the count record, the max count, min count, average count, etc, for data scientists to better understand the shape of their data.

Lastly, users can have access to view the data of the dataset by pressing the preview button of the page. Of course, this is only possible if the user has access to the underlying data in the first place.

How Amundsen democratizes data discovery

Showing the relevant data

Amundsen now empowers all employees at Lyft, from new employees to the most experienced, to become autonomous in their data discovery for their daily tasks.

Now let’s talk technical. Lyft’s data warehouse is on Hive and all physical partitions are stored in S3. Their data users rely on Presto, a live query engine, for their table’s discovery. In order for their search engine to show the most important or relevant tables for their users, Lyft uses the DataBuilder framework to build a query usage extractor that parses query logs to get table usage data. Then, they persist in this table usage as an Elasticsearch table document. And that’s how, in very short, they are able to retrieve the most relevant datasets for their data users.

Connecting data with people

As much as we like to claim how technical and digital we all are, processes for finding data consists mainly in interactions with people. And the notion of Data ownership is quite confusing; it is very time consuming unless you know exactly who to ask.

Amundsen addresses this issue by creating relationships between their users and their data thus, tribal knowledge is shared through exposing these relationships.

Lyft currently has three types of relationships between users and data: followed, owned and used. This information helps experienced employees become helpful resources for other employees with a similar job role. Amundsen also makes the tribal knowledge easier to find thanks to a link to each user profile on the internal employee directory.

They’ve also been working on implementing a notifications feature that would allow users to request more information from the data owners like for example, a missing description in a table.

If you’d like more information on Amundsen, please visit their website here.

What’s next for Lyft

Lyft is hoping to continue working with a growing community to enhance their data discovery experience and boost user productivity. Their roadmap currently includes email notifications system, data lineage, UI/UX redesign, and more!

The ride sharing company has not had its final word yet!

Sources:

Lyft – Statistics & Facts: https://www.statista.com/topics/4919/lyft/
Lyft And Its Drive Through To Success: https://www.startupstories.in/stories/lyft-and-its-drive-through-to-success
Lyft Revenue and Usage Statistics (2019): https://www.businessofapps.com/data/lyft-statistics/
Presto Infrastructure at Lyft: https://eng.lyft.com/presto-infrastructure-at-lyft-b10adb9db01?gi=f100fa852946
Open Sourcing Amundsen: A Data Discovery And Metadata Platform: https://eng.lyft.com/open-sourcing-amundsen-a-data-discovery-and-metadata-platform-2282bb436234
Amundsen — Lyft’s data discovery & metadata engine: https://eng.lyft.com/amundsen-lyfts-data-discovery-metadata-engine-62d27254fbb9

Learn more about data discovery solutions in our white paper: “Data Discovery through the eyes of Tech Giants”

Discover the various data discovery solutions developed by large Tech companies, some belonging to the famous “Big Five” or “GAFAM”, and how they helped them become data-driven.

Uber is one of the most fascinating companies to emerge over the past decade. Founded in 2009, Uber grew to become one of the highest valued startup companies in the world! In fact, there is even a term for their success: “uberization” which refers to changing the market for a service by introducing a different way of buying or using it, especially using mobile technology.

From peer-to-peer ride services to restaurant orders, it is clear Uber’s platform is data driven. Data is the center of the Uber’s global marketplace, creating better user experiences across their services for their customers, as well as empowering their employees to be more efficient at their jobs.

However, Big Data by itself wasn’t enough; the amount of data generated at Uber requires context to make business decisions. So as many other unicorn companies did such as Airbnb with Data Portal, Uber’s Engineering team built Databook. This internal platform aims to scan, collect and aggregate metadata in order to see more clearly on the location of data in Uber’s IS and their referents. In short, a platform that wants to transform raw data into contextualized data.

How Uber’s business (and data) grew

Since 2016, Uber has added new lines of businesses to its platform including Uber Eats and Jump Bikes. Some statistics on Uber include: 

• 15 million trips a day
• Over 75 million active riders
• 18,000 employees since its creation in 2009

As the firm grew, so did its data and metadata. To ensure that their data & analytics could keep up with their rapid pace of growth, they needed a more powerful system for discovering their relevant datasets. This led to the creation of Databook and its metadata curation.

The coming of Databook

The Databook platform manages rich metadata about Uber’s datasets and enables employees across the company to explore, discover, and efficiently use their data. The platform also ensures their data’s context isn’t lost among the hundreds of thousands of people trying to analyse it. All in all, Databook’s metadata empowers all engineers, data scientists and IT teams to go from just visualizing their data to turning it into exploitable knowledge

Databook enables employees to leverage automated metadata in order to collect a wide variety of frequently refreshed metadata. It provides a wide variety of metadata from Hive, MySQL, Cassandra and other internal storage systems.To make them accessible and searchable, Databook offers its consumers a user interface with a Google search engine or its RESTful API.

Databook’s architecture

Databook’s architecture is broken down into three parts: how the metadata is collected, stored, and how its data is surfaced.

Conceptually, the Databook architecture was designed to enable four key capabilities:

• Extensibility: New metadata, storage, and entities are easy to add.
• Accessibility: Services can access all metadata programmatically.
• Scalability: Support business user needs and technology novelty
• Power & speed of execution

To go further on Databook’s architecture, please read their article https://eng.uber.com/databook/

What’s next for Databook?

With Databook, metadata at Uber is now more useful than ever!

But they still hope to develop other functionalities such as the abilities to generate data insights with machine learning models and create advanced issue detection, prevention, and mitigation mechanisms.

Sources

• Databook: Turning Big Data into Knowledge with Metadata at Uber: https://eng.uber.com/databook/
• How LinkedIn, Uber, Lyft, Airbnb and Netflix are Solving Data Management and Discovery for Machine Learning Solutions: https://towardsdatascience.com/how-linkedin-uber-lyft-airbnb-and-netflix-are-solving-data-management-and-discovery-for-machine-9b79ee9184bb
• The Story of Uber https://www.investopedia.com/articles/personal-finance/111015/story-uber.asp
• The definition of uberization, Cambridge dictionary: https://dictionary.cambridge.org/dictionary/english/uberization

Learn more about data discovery solutions in our white paper: “Data Discovery through the eyes of Tech Giants”

Discover the various data discovery solutions developed by large Tech companies, some belonging to the famous “Big Five” or “GAFAM”, and how they helped them become data-driven.

With the emergence of Big Data, enterprises found themselves with a colossal amount of data. In order to understand and analyse their data, as well as meet the various regulatory requirements, it is vital for organizations to document their data assets. However, documenting and giving context to thousands of datasets is a very difficult, even impossible, task to do by hand.

Or, you can use Data Fingerprinting!

What is Data Fingerprinting?

In the data domain, a fingerprint represents a “signature”, or fingerprint, of a data column. The goal here is to give context to these columns.

Via this technology, a Data Fingerprint can automatically detect similar datasets in your databases and can document them more easily, making data steward’s tasks less fastidious and more efficient. For example, supervised by the data steward, data fingerprinting technologies allow us to understand that a column of data with the information “France”, “United States”, and “Australia” represents “Countries”.

Data Fingerprinting at Zeenea

In Zeenea’s case, our metadata management platform’s objective is to give meaning and context to your catalogued datasets in the most automatic way as possible. With our Machine Learning technologies, Zeenea identifies dataset schema columns, analyses them and gives them their own “signature”. In this way, if any of these fingerprints are similar, our Data Catalog will make suggestions as to whether the Data Steward should give the same information relative to another.

This technology also gives a means for DPOs to, among others, underline and point out personal or sensitive information that the organization possesses in its databases.

Data visualization definition

Data visualization is defined as a graphical representation of data. It is used to help people understand the context and significance of their information by showing patterns, trends and correlations that may be difficult to interpret in plain text form.

These visual representations can be in the form of graphs, pie charts, heat maps, sparklines, and much more.

What are the advantages of data visualization?

In BI, or Business Intelligence, data visualization is already a must have feature. With the emergence of Big Data, data visualization is becoming even more critical to help data citizens make sense of the millions of data being generated everyday. Not only does it help data citizens curate their data into an easy-to-understand visual representation, it also allows for employees to save time and work more efficiently.

In a way, data visualization also allows organizations to democratize data for everyone within an organization. With this, Data Leaders like Chief Data Officers see in this discipline a way to replace intuition decision-making with data analysis. Thus, be able to evangelize a data driven culture within their enterprises.

How can you get more value from modern data visualization platforms?

Most organizations that adopt data visualization tools struggle to visually represent their data in a way that maximizes data value. However, modern data visualization tools are expanding to include new use cases. These tools enable enterprises to find and communicate opportunities on important data analysis. Their strengths are:

Better communication and understanding of data

Data visualization allows employees, even those agnostic to data, to understand, analyse and communicate on data with new more interactive formats. This corporate will to become data-driven leads them to better inform and train their organizations to understand how to use data visualization tools and their relevant formats. These formats can be heat maps, bubble charts, tree maps, waterfall charts, etc.

Source: https://www.maptive.com/17-impressive-data-visualization-examples-need-see/

More interactions on data analysis

Reporting data is becoming more collaborative in organizations and presenting data a daily activity. Thus, data visualization is becoming more “responsive” allowing it to adapt to any device and any place the data is being shared. These tools open up to web and mobile techniques to share data stories and explore data collaboratively. Moreover, it’s large-format screens that create a more general understanding of the data in management meetings, for instance.

Supporting data storytelling

Data storytelling is about communicating findings rather than monitor or analyze their progress. Companies such as Data Telling and Nugit specialize in this. With the use of infographics, data visualization platforms can support data storytelling techniques in communicating the meaning of the data to the management teams. These kinds of representations grab people’s attention and better help them recall the information later.

An automatic data visualization

Data users are increasingly expecting their analytic software to do more for them. Augmented data visualization is very useful, where people are not sure which visual format is best-suited for the dataset they want to explore or analyze. These automatic features are best made for citizen data scientists, whose time will be spent on analyzing data and finding new use-cases rather than visualizing them.

Gartner’s top Analytics & BI platforms

According to Gartner, the analytics and business intelligence platform leaders are:

• Microsoft: Power BI by Microsoft is a customizable data visualization toolset that gives you a complete view of your business. It allows employees to collaborate and share reports inside and outside their organization and spot trends as they happen. Click for more information.

• Tableau: Tableau helps people transform data into actionable insights. They allow users to explore with limitless visual analytics, build dashboards, perform ad hoc analyses, and more. Find out more about Tableau.

• Qlik: With Qlik, users are able to create smart visualizations, and drag and drop to build rich analytics apps acceleratedby suggestions and automation from AI. Read more about Qlik.

• ThoughtSpot: ThoughtSpot allows user to get granular insights from billions of rows of data.With AI technology, uncover insights from questions you might not have thought to ask. Click for more information on ThoughtSpot

In conclusion: why should enterprises use data visualization?

The main reasons that data visualization is important to enterprises, among others, are:

• Data is easier to understand and remember
• Visualizing data trends and relationships is quicker
• Users are able to discovery data that they couldn’t have seen before
• Data leaders can make better, data-driven decisions