“There is no problem that doesn’t have some underlying need for more optimism, stamina, resilience and collaboration.” This is a quote from the lips of the game designer Jane McGonigal and, while it was intended for an audience of enthusiastic electronic gamers, it applies more broadly to digital development – and big data.

Big data, hurdles to handling it and hopes to harness it, were the subjects of the plenary session of the DIA Euromeeting in Glasgow this year.

We know that digital technology is now a key driver of healthcare and is helping to evolve the way we work in this sector  – it is speeding things up. We know that digitalisation offers longitudinal capture of information across time, allowing us  a glimpse into how we can better treat patients and diseases. We know also that the smart phone has changed the way we interact with each other, health systems, physicians and nurses.

These are all huge positives in the healthcare space, but, after what seems to have been a flying start, we are entering something of a quagmire for a number of vitally important, but nevertheless  irritating reasons.

Firstly, there is the question of size: for example, there are now some 165,000 health apps on the market, so the volume of data we can access and the speed of accumulation are already presenting us with a challenge. Secondly, integrating data sources in the same area is challenging – and across different types of data. Thirdly, we need to tighten our collation – and certainly our analytical – processes to ensure that we have the right data in terms of quality and usefulness. Fourthly, unbelievable as it may seem, 80% of data is lost – because it is not findable. Finally, we are faced with  a dearth of  legislation in the area of biobank governance, and a lack of clarity as to who is using and managing our data: patients want to decide what is done with their data, rather than it simply disappearing into a black hole – patient trust is going to be key to developing the digital approach.

So we have entered a period of uncertainty in terms of where we are headed, but it is not all a picture off despair and woe.

In Europe we are making some headway towards addressing data privacy with the data privacy regulation, which will come into force  in 2018. This  introduces the concept of accountability for all organisations that use personal data.The idea is that data protection should be by design and not be default – it  should be built into everything we do. Patients understand the potential of big data in terms of treatment, but need to feel confident and trust the systems used to process this. Nevertheless, the DPR leaves a lot of room for implementation, with the risk that Member States might it differently from one another. A key point is to allow the re-use of data (for other purposes than the original collection), otherwise most applications (research, big data analytics, payment models…) will be impossible.

We need to understand the structure and provenance of data and, through, liaising with the data generators drive collection towards  areas of unmet need. This close contact will also prevent the duplication of efforts.

On data disparity, the answer is simple to state, but complex to implement: global convergence is key. We need to acknowledge that, while we have a plethora of registries, EHR systems and other data sources,they need to be useable from a clinical and regulatory perspective. We know that the financial industry has invested in tools to pull together all their data – we can learn from this.

To sift through and analyse the vast hoard of data on offer we need a three-pronged approach: it will be essential to train people with new and appropriate skill sets; we need to invest far more in artificial intelligence which can handle and break down huge data sets; and we’re we to achieve data interoperability, we actually would be able to combine several data sets into one big bundle.

As we travel along this route, we can already start plucking the low-hanging fruit. We can begin by reading and utilising our clinical trial data more effectively. This will aid in the development of biomarkers that can track disease progression.

As our understanding and abilities in the field mature, there are greater rewards to be reaped in the future. We should be looking, for example to integrate genomics and proteomics data sets with clinical data sets, making the analytics and development more granular. We should furthermore be seeking to ensure the interoperability of data, so that they can be shared effectively and made understandable for everyone – including  patients.

The final point is that we should strive to remain human in our analytical approach: we should control the data and not let the data control us.