The files are large. The complexity is great. Data processing takes time. So where is the money in it?
There is a big difference between the data miners that have a vision and those that do not. In this case bigger isn’t exactly better, it’s just bigger. The software will only pick up the relationships between the variables that you put in and provide you with a statistical output. If those variables have no relationship to your business goals, you’re just spinning your wheels.
It’s not just about collecting big data, at a cheap price. It’s not about having the latest version of SAS, R, SPSS to process the data and find relationships. It’s about collecting the information that matters to your business and aligning it with the other variables to find what really matters. This is where the money is, this is what will make you rich.
Time is money. Time to insight is your goal. Running algorithms on a large set of data for over a month to find nothing is a waste. However, running multiple algorithms around your business goals can make you rich.
Related posts: Big vs. Small Data, Building a decision tree model, First hands on experience with big data