Big Data Insights

I found this lengthy gem of an article ( https://motherduck.com/blog/big-data-is-dead/ ) about big data being dead written by JORDAN TIGANI. And I enjoyed every byte of it, especially because it reveals so much about HYPE vs reality, specifically about big data, and more concretely about Big Query.

My opinion is that now the big data hype has decreased considerably which is a great outcome but at its inception, like many other skeptics for me was a move to convince more institutions worldwide to get rid of on-premises facilities and instead use cloud services which it is merited but it really depends, after all, how you would ever handle the future Peta worth of data that you need in order to succeed?

In the article, we find a thing that I keep hammering: "that data so many times can be a very big liability" especially if you don't know what underlines the value of your data. And you just store stuff without thinking long-term and without having a process to maintain and provide a sustainable future and roadmap for data design. 

Now I don't tell many people but my master's specialization was in databases, and sometimes I keep hearing echoes of how important is to design smart relationships. And the fact that any App should start from data design and not the other way around. 

The article also highlights that many times up to 80-90% of traffic and value usage comes from recent data and that before starting to hoard data you should ask yourself if you can extract knowledge from old data and then just use that knowledge and dump or archive old data. 

Another interesting point is the difference between compute and storage, storing data will generally be cheap, but just because predominantly with column and docs databases you can query a huge chunk of data that does not mean you can do so cheaply. 

I tried to do a TLDR, but I really recommend checking the article, for me, it seems the author manages to tiptoe around NDAs and reveals a great deal of "insightful chunk of data" for lack of a better phrasing. You'll be hit with convincing graphs and arguments that will maybe help you to question your data storage and processing before you will need to run a migration every 3 days because you didn't consider your data sources, data design, or implications of storing things that you think are useful but you lack a demonstration of that.