I came across a presentation from a major BI vendor claiming that:
“Data science moves from the specialist to the everyman. Familiarity with data analysis becomes part of the skill set of ordinary business users, not just experts with "analyst" in their title.”
source: Top 10 Trends in Business Intelligence for 2014, by Ellie Fields
I was blown away by this statement. Maybe I live in a different BI universe, but where are these sophisticated "everymen"?
A friend of mine who happens to be a statistician became fascinated by a conversation he overheard while travelling on a train recently. He's a statistician but he looks like a tramp - maybe that's something to do with mathematics and a statistical analysis of the usefulness of clothing compared to its price, then again, it might just be the size and state of his beard. Anyway, his two co-travellers weren't tramps. They were clearly the opposite and after listening to their conversation for a while it became clear they worked in financial services in some capacity. They'd spent a whole day analysing their company's corporate data and as the train carried them homewards, they were coming up with a fiendish plan on how to use their analytical skills to get filthy rich in their spare time. My friend quietly sipped a can of beer and listened.
The guys' plan was simple. Google up a large quantity of data - because large data sources are where the gold is buried. Use Excel to analyse it - Excel can handle a million rows which should be big enough. Choose what to analyse - again, an easy answer, companies. Analyse the companies' attributes and the correlation between their attributes and their stock price. When they'd found the correlation, they would look for companies with the same attributes but whose share prices were still low, buy them and sit back and wait to get rich. Simple and devilishly clever, I'm sure you'll agree.
This was the moment when my friend finished his beer and while opening a new one, to the surprise of the two guys who hadn't expected a beer-drinking tramp to understand any of their sophisticated plan, let alone voice an opinion on it, added his two cents: "Guys there is definitely a correlation between an egg and a chicken, but still we don't know which was first!!!"
There was quiet in the compartment till the end of the journey.
For me, this story just reflects the state of current enterprise BI. There's no problem getting a substantial budget for Big Data, Predictive Analytics or any of the other of the current buzzword projects. However, it's really difficult to defend or find the budget for activities like the creation of a report catalogue or a data dictionary, maintaining data quality or data governance in general, because these are just boring activities without a clear-cut "business case". In a consumer-orientated society, housekeeping will never be cool. But shopping for new gadgets which we don't know how to use, definitely is. Even more so, when everyman has become a data scientist.