Data Science

Coaxing out the hidden gems from your data

Business Intelligence has traditionally focused on the collection of data and presenting it in dashboards to inform executives what has happened in the past, allowing them to use their intuition to wonder at what the future may hold.

Data science moves beyond this approach - it is about identifying new patterns and making predictions, through processes such as machine learning, to provide insights and give strategic direction.

Starting with data "munging" - the curation and manipulation of information - a data scientist will handle records from diverse sources, sometimes at large scale, and place it into a clean and tidy format ready for the next stage.

Data exploration then takes place - identifying any obvious patterns in the underlying information from which initial insights can be drawn, as well as issues that may require fixing before any models are built. Graphical visualisation plays a key part in conveying these ideas to end users of the information.

Next comes data mining; the application of machine learning techniques to deliver meaningful predictions and descriptions of the future. Once armed with these predictions, it is then possible to put the results into action to drive decisions.

What on earth is Data Science?

"The sexiest job of the 21st Century"*

R; founded in academia, now gaining traction in the commercial world.

Statistical programming tools such as SAS and SPSS have for many years been the industry leaders, but there is a new kid on the block - R. Although it is in fact over 20 years old, R has risen to prominence more recently thanks in no small part to the work of Kiwi statistician Hadley Wickham who has developed powerful packages with a focus on usability.

The best thing is, R is open source, and totally free to install and use. Perform tasks on your data that puts spreadsheets to shame and gives the commercial vendors a run for their money.

Read Article

Data Mining with visual workflows.

SAS Enterprise Miner has long been the industry leader in the visual machine learning space, with a highly polished visual interface for building advanced analytical models. As a long term user of Enterprise Miner, I can attest to its power and despite the cost, have seen it deliver multiple ROIs in quick time.

More recently I have begun to explore alternative visual modelling tools - including RapidMiner and KNIME - which are beginning to make inroads in this space and have been recognised by Gartner as being strong future contenders in this space.

Watch out for Microsoft too who are integrating advanced analytics into their Azure cloud platform.