Big Data: Does size matter?

Analytics Big Data Marketing
September 8, 2016

The Big Data Pandemic by Ken Roberts, CEO of Forethought is an interesting and thought provoking read. The article sets the stage by predicting that big data will explode at an alarming rate (projected growth in 2013 was $18.1 Billion USD) and completely negate the need for Small Data such as hypothesis, surveys and sampling. Anyone working in Small Data, on reading that, would have pricked up their ears and exclaimed “hang on a minute”, which is precisely the path that Robert’s then takes by offering the other side to the argument and concluding that the jury is still out as to whether Big Data will outperform Small Data.

Although there are many advocates for big data and its contribution to the marketing environment, issues with stability and integrity of data as well as basing decisions purely on clusters of behaviors without understand what is driving things has left opinions somewhat divided. Couple that with the inability of business in general to actually implement Big Data programs, (many can’t even get to grips with Small Data so how can they be expected to embrace Big Data) we are probably still a long way from this making a significant impact, or change to, business as usual approaches.

140113_bigdata

As many of you know, I’m a huge advocate of Big Data. I’m totally fascinated by what information you can gain by using it and I think it can provide valuable insights in to identifying “what” is happening.

At the Bureau of Meteorology we use Big Data and Computer Models to provide weather forecasts. The Big Data generated by our observation networks coupled with the predictive analysis data from computer modeling can tell us with a high degree of certainty exactly what is happening, when it is happening and where it is happening. What it doesn’t necessarily do though is tell us “why” it is happening. For that we default back to our forecasters and observers on the ground to analyze what the models are outputting to help explain the “why”. In simple terms what I am talking about here is quantitative and qualitative data.

In the retail environment it is equally as complex. Big Data may well be able to provide valuable information on what people are buying and where they are buying it from, future trends etc. but is it really able to accurately predict how people feel about what they purchase, the decision making process they went through, whether they were happy with their purchase and would do it again? In an article by Chris Anderson in 2008 he argues that the why people do something or how they feel about it isn’t really that important as long as they do it, he is certainly more convinced with that than I am.

ckieagkwcaaj9gf

And that was my key take away from the Robert’s article. Big Data can be a great resource of qualitative data, but would you really want to leave it all in the hands of computer models and artificial intelligence and then base your most important business decisions on just that?

To obtain a really good understanding, I think it’s still important to qualify those results by understanding the “why” and for that, you need some human intervention. As Robert’s notes “perhaps it is not about big data versus small data but rather, big data and small data combining to produce synergistic insight.

My money is on Roberts. What do you think?

dashboard-snockered-624x418

You Might Also Like

4 Comments

  • Reply Smarketing September 10, 2016 at 3:24 pm

    Hi Keith,

    Great article. I’m really intrigued about the Bureau of Meteorology’s use of Big Data. While most people think about the commercial applications (which are obviously many and varied – and still evolving), it’s really interesting to consider applications for Big Data outside of consumer transaction habits or history.

    According to King, marketing applications are “a tiny fraction of what could be done” if it were possible for academic researchers to access the information held by companies. (Businesses now possess more social-science data than academics do, he notes—a shift from the recent past, when just the opposite was true.) (Roberts 2013).

    If social scientists could use that material, he says, “We could solve all kinds of problems.” But even in academia, King reports, data are not being shared in many fields. “There are even studies at this university in which you can’t analyze the data unless you make the original collectors of the data co-authors.” (Roberts 2013).

    The potential for doing good is perhaps nowhere greater than in public health and medicine, fields in which, King says, “People are literally dying every day” simply because data are not (Roberts 2013).

    In terms of the Bureau of Meteorology, do you feel there are other possibly applications for the Big Data collected that could contribute to social good?

    I suppose the obvious ones are predicting droughts or severe weather conditions such as cyclones or tsunamis. I’m wondering if there might be any other applications you know of or could think of?

    • Reply Keith Day September 10, 2016 at 4:39 pm

      Hi, there are a whole range of things that the data could, and is used for, around weather and climate. We offer the data to people to develop solutions, for example, this year we sponsored Gov Hack which is can annual event where we offer our data out and sponsor a prize pool to reward whoever can come up with the most innovative use of the data to solve a problem that would benefit society.

  • Reply Kelly McCarthy September 12, 2016 at 3:37 pm

    Hi Keith,
    Another interesting blog post, I agree with your conclusion that “To obtain a really good understanding, I think it’s still important to qualify those results by understanding the “why” and for that, you need some human intervention.”

    I recently did a group assignment in another subject on the product life cycle of an iPhone, specifically the impact of incorrect disposal and the huge number of people who do not recycle their phones. The big data told us that some 60% of Australians hold onto their phones, leading to 22.5 million mobile phones stowed at the back of drawers across the country. However, rather than making the assumption that Australians don’t recycle because there is no recycling service (because there is) or that they are not aware of them (mobile muster has good awareness), a class discussion revealed that a lot of people didn’t trust that their data would be safely destroyed in the recycling program and this was a key reason why they hold on to their phones.

    So by combining the best of big and small data, the mobile phone industry could achieve better results by resolving consumer fears about the safety of their data than throwing energy behind developing and promoting recycling programs.

    • Reply Keith Day September 12, 2016 at 4:08 pm

      Hi Kelly, that’s a great example and illustrates really well the point I was making so thank you for bringing that to the discussion

    Leave a Reply

    Top