How to support your business with big data without stifling innovation

There’s a paradox that often emerges when we try to get control of our data so that we can safely and effectively extract business value from it.

Putting all our data in one place should give it the availability, security and standardisation necessary to make it a highly valuable resource. Unfortunately, though, this often turns a data gold mine into a treasure-safe that only a select few can enter.

There’s a strong argument for centralising and controlling data

Data becomes locked for all the right reasons: recognising the friction, inertia, mistakes and lost opportunities that arise when our data infrastructure and governance is poor, we set about organising and curating. We want to be able to trust our data, so we try to bring it together in one place. We also demand that it meets high quality standards, is properly documented, and has controlled access. And we may look to cloud and big data technologies to let us do all of this in a consistent and economic way. Centralising data like this means you can adopt a financial model that makes sure the resources to store and process data and those needed to build systems that consume that data are sensibly allocated.

This all seems very reasonable – so what’s the issue? The problem is that while we might appear to have gained control, we’ve often shut the door on something else – innovation. Suddenly, the data scientists are struggling to bring forward new ideas and the pipeline of value is drying up. No doubt, it wasn’t intentional, so what’s happened?

Tightly controlled data can stifle innovation

The challenge is that the real world is dynamic and imperfect. Data doesn’t arrive fully formed, carefully structured and unambiguous. The uses that we might put a data asset to over its lifetime aren’t always obvious from the outset and they are likely to evolve and change. The same is true for the teams and people involved in its consumption.

So, we shouldn’t assume that a use case that turns up without a fully funded business case is unimportant. Nor should we assume that such cases will somehow scrape by with some minimal, default compute and storage resources, or that the people concerned will be able to navigate the processes that will allow them to discover and access the data they need.

The risk is that organisations create very safe and well organised places for their data and in doing so will lock it off from innovation. These places become like walled gardens - only the initiates can gain access and only the already well-understood business cases will be able to justify their seat at the table. Novel use cases will struggle to get going and soon enough, the big data dream will turn sour.

Spot the signs that data innovation is suffering

The clue that a walled garden has been constructed is that while ‘core’ applications seem to be well served, there’s growing frustration in the organisation about the ability to meet new challenges in a timely way. And this frustration shows up as the emergence of shadow IT, adopted by people in the business desperately trying to work around the barriers they experience.

Those who’ve created this data infrastructure and the ‘first class citizens’ who enjoy its benefits may not believe that these challenges are real. How do you change that?

Put yourself in the shoes of someone behind the ‘veil of ignorance’ and consider how easy or hard it might really be for a new team to bootstrap a novel use case that will depend heavily on access to your big data infrastructure. Do they have the tools to discover the data they need and to connect with the owners and the relevant domain knowledge – whether that’s written down or stored in people’s heads? Are the costs of bringing new data into this environment so high that they can’t even get started? How can they demonstrate the value of their application if doing so demands access to data and compute at scale, scale which is only available to first class citizens? And can they use the necessary tools, or do tight controls stop them from trying out new technologies?

Better still, find the people in your organisation who’re trying to use your big data to innovate. Ask them about their experiences – they’ll be grateful that someone is listening. And work with them to achieve a better balance between the here-and-now operational needs of the business and the ability to nurture and develop the new.

To find out more about how we can help you control your data, without losing the ability to innovate, please get in touch with your account manager. We’re here to help.

Discover how your business can exploit technology and innovation both now and in the future by downloading our ‘Winning the innovation race’ brochure.