How to bridge the gap between AI ambition and reality

AI often invokes the image of being the pinnacle of digital transformation, something every organisation should aspire to. Consequently, most of them do. But while 87% of organisations say AI is the future and vital to their survival, almost the same percentage (86) agree they wouldn’t trust the technology enough to forego human-driven decision-making. This was the result of a recent survey by Vanson Bourne, which polled over 500 senior IT and data science professionals across the UKI, US, France and Germany.

Across all the geographies surveyed, one contradiction was clear: although companies are laying the groundwork to develop general-purpose AI – collecting data, hiring data scientists and lining up investments – they are let down by poor data management, which hinders even basic areas of data analysis. When even the prevalent, hands-on approach to data analysis is plagued with inefficiencies, it’s not surprising that companies are reluctant to relinquish control and place their trust in the hands of AI. 

So, how can organisations overcome their distrust and embrace the promise of AI?

Dismantling the technical barriers to AI adoption 

“Garbage in, garbage out” is a popular way to explain why so many AI projects fail – the data feeding algorithms is often simply not good enough to return usable results. ‘Bad data’ has three faces: the incomplete, the inaccurate and the stale data. As the survey reveals, all of these play a part in organisations’ AI woes:

• 71% of organisations struggle to access all the data needed to run AI programmes, workloads and models
• 72% find it difficult to cleanse data into the right format to make it useable
• 73% find translating data into useful advice a challenge

Data engineers – who build and maintain the lines of codes known as data pipelines – may work tirelessly to make data accessible, but without automation, their cumbersome and time-consuming work still often results in stale data by the time it reaches data analysts or data scientists. Yet, nine in ten organisations still rely on manual processes.

Bad data is gambling with skills and money

Although the technical difficulties to data management – starting with simple access to data – may reasonably explain why AI ambitions are hindered, the study shows that organisations see the blame elsewhere. In fact, the most commonly-cited barrier to AI adoption was challenges with recruitment. This is despite all organisations surveyed having data scientists in place.

Hiring more data workers for a job that will leave them feeling frustrated is not the answer – empowering existing talent is. Highly-trained data specialists crave challenges and want to build interesting systems. Waiting on resources to become available is not only a waste of their talent – as indeed, 87% of respondents recognise data scientists in their organisations are not used to their full potential – it is also a waste of money.  

Organisations estimated that 5% of global revenues were being lost due to underperforming AI programmes, which are built on bad data. As AI investments look to continue to grow – participants expect to invest 13% of revenues into AI in the next three to five years, compared to 8% today – it is imperative that organisations first put in place solid data management foundations.

How to make data AI-ready 

The easiest way to visualise the building blocks of an AI-ready data strategy is to think of the ‘modern data stack’, or MDS. The MDS is a combination of three core technologies: automated data integration, a cloud data warehouse or data lake, and a business intelligence or data visualisation platform which a machine learning engine can further complement. 

With automated data integration, setting up data pipelines is a no-code process, which involves logging in, selecting fields and watching the ready-to-use data sets populate. The data is continuously updated, which not only eliminates issues around stale, inaccurate or incomplete data, but also saves valuable time for data engineers, analysts and scientists alike.

With data professionals in short supply and AI projects more in-demand than ever before, it’s key to use data scientist talent well. Alleviating the manual data preparation burden on data scientists will help organisations retain their top talent, while ensuring informed decision-making. 

Walk before you run

Organisations undoubtedly have a long way to go to reach AI maturity but they shouldn’t feel discouraged. Once they can trust the data, trusting AI should naturally follow. 

What’s more, by boosting the visibility, accuracy and reliability of their data, they are not only laying the foundations for a successful AI programme, they are also paving the way for more powerful analytics capabilities altogether. After all, accurately explaining what happened yesterday is just as, if not more important than trying to predict what will happen tomorrow. Those with the right foundations will always find a competitive edge in their data.