Stepping away from big data analysis to gain a broader view

‘Big data’ has become a bit of a buzzword for businesses as they look to strengthen their data and analytics capabilities, with the overall aim to drive growth and address business challenges. 

When correctly gathered and analysed, big data analysis can help to move businesses forward, allowing organisations to gather information from multiple data points to capture a variety of perspectives on customer, market and historical trends. 

By providing a clear overview on a range of trends within organisations, big data enables businesses to better understand the multiple market forces at play and ultimately make more informed decisions on the company's strategy.  

Despite the benefits of big data analysis, many technologists see a trend emerging that signifies companies are now moving to small and wide data to accelerate business decisions and growth.

Garnter has previously predicted that 70% of organisations will shift their focus from big data to small, wide data by 2025. Lending itself to that shift is the emergence of AI solutions for business, which require many different types of data to work effectively. 

Jennifer Belissent, Principal Data Strategist at Snowflake, believes that, despite this shift, data ‒ in all its different forms ‒ needs to be effectively analysed for businesses to gain real value: “Big data enables businesses to sort through massive amounts of data, such as transaction data or sensor data, and derive insights that improve understanding of customers or operations.” 

“Decision-makers are hungry for new insights. Marketing teams want to optimise campaigns and ensure an ROI on ad placements. Sales teams want to prioritise leads to not waste their time. Risk teams want to better predict fraud. Operations teams want to predict maintenance needs and avoid unscheduled downtimes…and the list goes on,” she continues.

Michael Gilfix, VP, Product Management, IBM Data & AI and Chief Product Officer, Cloud Paks & SaaS at IBM, while acknowledging the importance of data ‒ be it big, small or wide ‒ stresses the importance of analytics: “To really get the most value out of data, businesses need to adopt a data strategy that deploys a connective tissue between disparate data sources and storage repositories, what we call a data fabric architecture. It helps to foster broader information sharing and governance across different areas of the business, making it accessible to everyone, when and where they need it.”

Responding to regulatory setbacks with small and wide data

While businesses have been making sense of their data, there has also been a growing concern around privacy looking into the ways data is shared and handled. Often associated with invasive marketing techniques and disruptive advertising, big data creates a tricky regulatory environment for data engineers to navigate.

The tightening of regulations around this data and the way it is used for business growth has been a key driver of this data shift. Evidence of this includes the GDPR regulations enacted throughout Europe between 2016 and 2018, which places limitations on how data can be shared. As such, regulations are rendering big data less effective and efficient ‒ and, therefore, effectively obsolete, meaning that businesses are having to find alternative ways to gain value from data.

Shifting to small and wide data, Gilfix shares how small data can bridge this gap: “While big data can offer insights on big picture trends and patterns, small data can reveal what drives an individual customer or potential client. So instead of a company guessing that the broad trend it identified from its big data set will apply to its customers, it can use small data to meet their needs in a more personalised way.” In short, it prevents companies from assuming their customers are a monolith.

“Small data also offers the advantage of providing near real-time insights – such as a customer’s journey through a company’s digital ecosystem. Whereas by the time a big data set has been analysed, the insights are already old and remain broad in scope,” he continues.

Cloudera’s Field CTO for EMEA, Chris Royles, adds to this by stating that, despite an uptake in the analysis of small and wide data, this view should actually be added to big data analytics rather than replace it altogether: “Tapping into small, wide and big data, businesses are empowered to build more robust artificial intelligence and analytics to drive greater innovation and growth. Adopting this multi-data approach enables organisations to gather an even richer and comprehensive perspective of the market, all of which can be used to better inform decisions that will fuel the growth of the business.”

Combining types of data analysis for business intelligence

The shift in focus to alternative types of data doesn’t mean the abandonment of big data, particularly as many software providers have solutions that enable big data analytics with ease. It seems that instead of a focus on big data, organisations will reap more benefits from a diversity of data types and sources and analytic methods. 

Belissent adds that this does not end at big, small and wide data: “In addition to big data, small data, wide and diverse data, qualitative data will continue to have a place in generating insights ‒ particularly about customers. For example, recommendation engines trained on transaction data will be complemented with focus groups to test customer responses to specific offers. Ad placement algorithms will be combined with A/B testing to determine the impact of specific messages. Panel surveys will capture sentiments to complement textual sentiment analysis. Qualitative research enables the capture of feelings, and compliments research that predicts actions.”

There is no doubt that the amount and type of data businesses have access to will continue to grow as IoT devices and cloud data platforms become more readily available. 

With such a diverse range of data sources, it is now vital that businesses develop a data strategy that guarantees useful and actionable insights. “Whether it's big, small or wide data we’re talking about, adopting a data strategy centred on a data fabric architecture is essential to convert that data into business value,” notes Gilfix.

He concludes: “A data fabric connects different types of data across silos—be it on-premises or across multiple clouds—without ever having to copy or move any data. It can also democratise information while still helping to ensure an organisation can maintain the level of governance and privacy required.”