BigData, Cloud, Analytics and the Magic of Doing a Lot More With a Lot Less

Big Data. Cloud. Analytics. All nice concepts that talk about progress. But are they?

There is a shift going on all around us and it is happening even now. Who haven't seen a YouTube video, Slideshare presentation or just an infographic with large number about consumption? Tweets, sharing on facebook, hours uploaded to YouTube every minute, the number of emails sent every second. Choose a type of an interaction (Instagram), an event (Sandy) and you will find the number for it. And it will be a BIG number.

The technologies that are used for such scale is often placed in one of these big marketing buckets:

• Cloud
• Virtualization
• Big Data
• Real-time analytics

And there's a lot of misunderstandings and misconceptions about them that I am not going to get into now – suffice to say that you need to tread carefully when selecting such technologies while at the same time make sure all of them are in your marketing collateral.

If that isn't enough, all of them also promise reduced TCO – a reduction in the hardware costs you have with previous generation solutions.

So we're now doing a lot more – with a lot less.

Everything has a cost. Even the reduction of a cost.

Big Data you say? Solving all of your scaling problems? No more sharding of databases you are promising me? And on commodity hardware no less. To get there, though, you usually need to sacrifice the ACID nature of your database and… settle for less. ACID means Atomicity, Consistency, Isolation and Durability – things I am sure you want in your database.

Virtualization it is then? Just check the answers to my question on a previous post on cloud vs. virtualization: When it comes to my readers – they will throw virtualization out the window and work harder to achieve better performance and lower latencies. I am sure some of the WebRTC guys out there are using virtualization properly even for the media processing itself.

It seems to me that all of these new technologies to deal with scale in a cost efficient way are doing so by changing the basic requirements from the system they come to replace – they question the need for certain features:

LEGACY: "Sure I want the system to run as fast as it can."

NEW TECH: "And if I give you something that… hick ups for a second once a day, but costs 10% of what you are paying now – will that… suffice?"

LEGACY: "When you put it that way…"

LEGACY: "And what about my database? I just have to have consistency in it. Can't do it otherwise."

NEW TECH: "Sure. Let's use an SQL database for the users table, but reduce consistency on the provides targeted ads to them. It will give the same results 99% of the time, and you will save a considerable sum of money on your storage costs."

LEGACY: "Where do I sign?"

-

New technology is great, but there's no magic in it. We are just learning to increase the number of options we put on the table and have developers pick and choose what fits their job a lot better.

Call it technology islands if you want. It is just more choice.