article banner

Lessons on big data from the healthcare sector

The healthcare sector is still experimenting with using big data, but it already has plenty to teach us about the benefits and challenges of big data analytics.

Healthcare providers have been collating big data long before we even knew it existed. Today, in certain controlled environments, technology allows for vast databases of detailed medical records to be digitised and analysed on a scale large enough to detect patterns invisible to the human eye.

At the same time, the healthcare sector is undergoing a fundamental cultural shift in which healthcare providers are paid for ensuring patients remain healthy rather than for a particular treatment or procedure. Big data analytics can help providers make this transition from a fee-for-service model to a value-based model.

However in making the transition, healthcare providers are encountering some important challenges: how do we organise the data we have, and how do we make impactful use of it? How do we collect and store highly sensitive data securely while making it available for data analytics? How do we make databases from different but related institutions talk to each other? And how is data in that federated environment managed and distributed effectively so that every data user has the same experience?

These are questions that the leaders of every business in every sector need to ask themselves before embarking on the big data journey.

A values-based approach

Healthcare systems around the world face multiple challenges: caring for aging populations; treating rising rates of serious conditions such as heart disease, diabetes and obesity; and increasing patient expectations and standards of accountability. This, twinned with dwindling public funds to pay for healthcare, is persuading policymakers to move towards a values-based approach to providing healthcare.

In the US, for example, where the federal government’s Medicare programme funds health insurance payments for the elderly and selected younger people, more than 50% of insurance payments for the over-65s must be paid this way by 2020. In 2009 the UK government introduced an initiative called Commissioning for Quality and Innovation, which allows 2.5% of the cost of hospital treatment to be held back, contingent on outcomes.

These types of reform are already driving efforts to use big data to understand the true health of patient populations and to make appropriate recommendations that keep people healthy.

However, big data analytics can play a much bigger part. For example, it can help detect early warning signs in a flu outbreak by understanding the underlying causes and therefore treat affected patients more efficiently. It can also help government healthcare systems and healthcare insurers discover insights that lead to more effective and financially sustainable treatments. For example, by understanding the underlying genetic causes of diseases, existing drugs can be repurposed to treat those illnesses.

Paying for outcomes

Alongside what providers are paid for, how they are paid also has an impact on the case for big data.

Many governments and insurers have moved towards a system where a single payment is offered for a full care cycle of acute medical conditions, or for a defined period of overall care for chronic conditions. In Sweden, the Stockholm authorities have been using this approach for hip and knee replacements since 2009; in Germany, such payments are offered for hospital inpatient care. It’s an approach that’s encouraging providers to be more efficient and collaborative in the way they deliver healthcare. Big data can help that collaboration.

For example, if healthcare providers can gather data about the treatment of a given condition across multiple institutions they can also identify common factors in efficient and cost effective instances. Sharing that data across the sector can help to reduce ‘clinical variation’ – instances of the same treatment leading to different outcomes.

Taking a longer-term view, big data can help providers understand if certain treatments or procedures really lead to improved patient welfare down the line. If they don’t, providers could save costs by not carrying out unnecessary procedures.  


A global push towards electronic medical records systems will create the big data that enables these types of collaborations to take place. But storing data electronically comes with risks. Cyber-crime and the threat to patient privacy is the top concern of hospital CEOs right now. In the US alone, 113 million medical records were hacked in 2015 and 3.5 million were breached in the first quarter of this year, according to government figures.

Data security isn’t the only issue that can undermine the public’s trust in having their personal data collected and stored. The lack of regulation governing how this data can be used is also a challenge and can have the add-on effect of people choosing not to share their data in the first place. This threatens the potential promise of data-derived innovation and makes large datasets more difficult to manage, increasing the chances of a data breach.

Electronic medical records are just one of hundreds of IT systems in the healthcare sector that will need to be interoperable – that is, be able to ‘talk’ to each other – if big data is to be used to its full potential. Data breaches could occur at every point of connectivity, which is why providers need to audit their systems to ensure they are secure and ensure there are safeguards at any points of weakness.

In addition, providers should have a well-thought-out plan they can implement in the event of a data breach. That’s tougher in healthcare than in other sectors because, as one hospital CEO told me: “You cannot shut down the clinical systems or people will die.”

Providers need to understand what types of breaches might occur, as well as the operational and IT implications of those breaches – principles that apply in other sectors. For example, which systems can and cannot be shut down? What are the legal implications to consider; not easy in a world where technological advances are outpacing the governance and legal frameworks, especially when it is unclear who 'owns' the data? Most healthcare providers haven’t thought through all these questions yet and so dealing with cyber-crime becomes  very reactive. Although some are starting to take a more planned and organised approach.

Private-sector opportunities

Despite the risks, the potential of big data to improve healthcare provision can’t be ignored. The challenge is finding the resources to exploit that potential. For public healthcare providers who are accountable for how they spend taxpayers’ money, that might not be an option, which leaves a space for private-sector providers to step in and offer this service.

Many big healthcare insurers are trialling big data, including Swiss Re, Zurich and several US-based insurers. They are using technology such as IBM's Watson which analyses vast amounts of unstructured data to answer particular questions, extract key information or reveal data insights to improve patient outcomes. Over the next few years, we are likely to see some private companies take a lead in this field, making a name for themselves by gathering and mining public/private healthcare data for beneficial patient and provider outcomes. Strong data governance systems in the private sector will also give rise to interoperability challenges that will need to be addressed.

Open data initiatives could provide another route to harnessing the benefits of big data. In the UK, the government-funded 100,000 Genomes Project has opened the genetic data of 100,000 patients up to public and private sectors organisations to help sequence 100,000 genomes by 2017. The project hopes to enable the UK to make strides in the genomics field, leading to a host of benefits such as earlier and more precise diagnosis, faster clinical trials for new drugs and treatments and, ultimately, new cures.

Embarking on the big data journey can be daunting but healthcare providers will need to get on board. As the sector moves towards a values-based rewards system, public and private providers that leverage big data and share it across the care-delivery ecosystem will be able to offer more efficient and effective treatment.

The healthcare industry can be a leader for what could happen in other sectors too. Yes, big data has its challenges: security, governance and interoperability, to name a few – but it also has multiple benefits that, ultimately, will lead to deeper insights and improved outcomes, performance and growth. Not to mention healthier communities.