As the healthcare industry is transforming from a volume-based to a value-based model, it is unsurprising to see the need for more meaningful data to improve medical outcomes for patients, drive patient and member engagement, and bend the cost curve. These organizations are dependent upon the ability to rapidly ingest and analyze large volumes of data in batch or real-time from an extensive range of sources in a variety of formats.

There is an exponential increase in the volume of healthcare data due to information coming from medical imaging, clinical trials, biometric sensors, pharmaceuticals, patient records, wearables, and more. Gathering all of that data in a safe, secure environment and using it to make a quick and informed decision is challenging.

Today, a large part of the data is stored in the traditional Enterprise Data Warehouses. Data Warehouse, by definition, is a central repository of well-structured data gathered from diverse sources. In simpler terms, the data has already been cleansed and categorized and is stored in complex tables.

As the healthcare industry contends with greater volume, variety, and velocity of data, there has been some frustration with traditional data warehouse solutions wherein the rigidity of their structures is slowing down the process of achieving new insights. Also, an additional reason for the frustration of the healthcare organizations stems from the fact that Data warehouses do not allow the storage and analysis of unstructured data where the bulk of insight for proper healthcare management lies. This made way for a new concept of Data Lake to enter the healthcare arena, conceived as a flexible alternative to traditional data warehouses that keep data in a very structured format. The Data Lake concept allows storing unstructured data as well, with more flexibility in performing data analysis. The concept of Data Lake was introduced in 2010, and the idea has gained traction ever since. A data lake is a collection of various data assets that are stored within a Hadoop ecosystem with minimal change to the original format or content of the source data (or file). Essentially, a data lake is an architecture used to store high-volume, high-velocity, high-variety, as-is data in a centralized repository for Big Data and real-time analytics.

Comparison between Data lake and Data warehouse

• The data warehouse uses the ETL (Extract Transform Load) procedure where the data is transformed and then loaded into the data storage, which may take several months of modeling, mapping, ETL development, and testing to move source data into a dimensional schema. Data lake uses the ELT (Extract Load Transform) procedure, where the data gets processed after being loaded into a data lake.

Some of the healthcare Use cases that can benefit from Data lake architecture

• Healthcare organizations can collect and standardize a wide range of data, such as claims and Rx data, clinical information, health survey, administrative data, patient registries, data from EHRs, and EMRs to create a comprehensive view of patients, assisting in a variety of use cases comprising better outcomes, cost reduction programs, medical decision-making, and quality improvement initiatives.
• The analysis of a huge amount of unstructured data can offer a massive opportunity for payers to get quick & meaningful insights on unbilled texts from different sources such as transcripts of physician’s notes, labs, and related minor performed procedures. Based on data generated, payers can apply risk adjustments to segments of their patient population for compensating on a per-patient basis.
• Using information collected from bio-monitors, bedside sensors, and other IoT-enabled devices, healthcare providers can find valuable patterns using which changes can be brought in patient care delivery and care coordination.
• Combining data from hospital reports, transactional data from pharmacies, social media, and helpline data, healthcare organizations can evaluate them and can help to detect trends of disasters such as Covid or Ebola.
• Machine learning techniques used on top of high-resolution, global maps can identify the possibility of an epidemic’s rise in certain areas and estimate the number of people living in high-risk places by detecting likely locations for viruses to thrive.

Data Lakehouse-Evolution of a new analytic landscape

As we get more and more data on patients and members, there is a need to refine our analysis based on the heterogeneity of that data. It means a large amount of modeling on a growing number of cohorts with the ability to get deeper, more refined insights on segments of our populations. Neither the Data Warehouse nor the Data Lake is built to handle this.

Data warehouses have a long history in decision support and business intelligence applications. While warehouses were great for structured data, healthcare organizations, in particular, have to deal with unstructured data, semi-structured data, and data with a large variety, velocity, and volume. Data warehouses are not suited for many of these use cases, and they are certainly not the most cost-efficient. About a decade ago, organizations began building Data lake–repositories for raw data in various formats. While suitable for storing data, Data lake lack some critical features some of which are outlined as below:

For the above reasons, many of the promises of the data lakes have not materialized, and in many cases leading to a loss of many of the benefits of data warehouses.

Data lake due to its own sets of limitations, fostered the concept of a Data Lakehouse. A data lakehouse (or also known as lakehouse) is a relatively new term in the world of data science and engineering. Data Lakehouse is a combination of both a data lake and a data warehouse, hence the term lakehouse.

A lakehouse is essentially a new system design that allows for the convergence of multiple systems and brings in data lakes and data warehouses onto a single platform.

In a Data Lakehouse, the Data Warehouse and Data Lake will work in tandem. This means instead of processing the data in the Lake and then pushing it out for other add-on tools to deliver traditional BI and Data Science, the Data Lakehouse turns into a single data store that is capable of handling all queries.

Using Data Lakehouse, a data analyst gets the dual advantage of having the reliability and stricture of a data warehouse along with the agility and scalability of a Data Lake. Healthcare organizations engaged in value-based contracts cannot realize the full potential until they deploy a unified technology platform that can store and combine comprehensive data, analyze it, provide recommendations and interventions in real time, and integrate feedback. By enabling rapid ingestion from new data sources and ensuring governance with speed around the deployment of new data into existing analyzes and models, Data Lakehouse empowers teams engaged in delivering value based analytics.

With machine learning poised to disrupt every industry, the new data management architecture of the Lakehouse radically simplifies enterprise data infrastructure and helps accelerate innovation. This new architecture of Data Lakehouse is essential for the healthcare industry as it allows for easier integration of new strings of data. It overcomes the limitations of organizations forced to keep data storage, compute, analytics and governance siloed under the data lake and data warehouse models. It serves to bring complex data sources together centralizing sensitive data in a single, secure location enabling the healthcare providers and payers to improve decision-making, lower the risk of errors, achieve higher operational efficiency and lower costs, and ultimately drive better patient outcomes across the care continuum.

While Data Lakehouse is still at an early stage, we believe it’s a new data paradigm that healthcare organizations must recognize and build upon. With its mission to actualize data and drive improved outcomes, Resolve helps healthcare organizations build a strong data foundation and orchestrates the entire data lifecycle including data security, privacy and governance.