My 3 key learning in Data Science and BigData

Happy New Year 2016, folks!!
As 2016 sets in, the Analytics domain is in the limelight and Data Scientists are in huge demand. The maturity levels of the market has increased and the emergence of set processes and structured teams has become the norm.
In the Data Science world, the buzz word remains BigData and any Analytics professional worth his salt is keen to understand this domain. I am putting down 3 key points that I have understood.
1. BigData is all around us: – Earlier all documents, images and video / audio files generated in business were stored in physical form in storage units … boxes, vaults etc. Today, most of it is digitized and stored. All government records, business data, application forms with photo and customer / employee documentation, security camera footage … everything is being digitized for easier storage and fast access.

BigData universe
BigData universe

Source :-

2. BigData can be structured: – Somewhere the perception has built up that the BigData universe is unstructured. I guess that’s because each of us as individuals see Social Networks and their impact most closely .
However, with the emergence of IOT (Internet of things), sensors and machines are the biggest contributors of BigData . If you think about the additions of sources of data in the last 2 years , you will realize how much apps and sensors have come into our lives and are being used to book cabs , update news , shop etc.

Smart phone and growth of devices connected to the internet
Smart phone and growth of devices connected to the internet

Source :-

3. BigData and Smartphones :- This era belong to the smart phone . A relatively unknown device till 5 years back, it’s become the new ‘must have’ accessory. Research by various organizations tracking data on Smart phone usage predict that it will grow exponentially and its usage for various purposes will only multiply.
This is also changing the usage pattern in telecom . In 2015 the traffic of data has increased exponentially as compared to voice traffic in telecom networks.

Devices - growth


How is the world using BigData to create smart systems which make our lives easier?
The process generally works as follows:-
1. Network sensors are feeding in data related to ‘state of being of a person, place or thing’ (location, online status, timestamp of activity etc.) at regular intervals (per second, per minute, per status change etc.)
2. These data points (images/ text / other data formats) hit a Data store (generally a cloud server)
3. The Algorithm works to create recommendation/s which are fed back to the Network Sensor (point no. 1) / any other center which can execute the recommendation
Thus, the smart system is born.

Smart systems and how they work
Smart systems and how they work

Figure :- Smart systems and how they work

As a Data Scientist I find that skills built on structured data in organization systems and ERP software are very easily applicable to most of BigData which is coming in well thought-thru formats – via sensors and apps. That is indeed good news as this means that Hadoop and No SQL systems are of interest to me only in the context of volume and velocity of data. I can therefor upgrade my skills marginally and start using these as I use any other DBMS. All other statistical skills and business context can be applied with confidence. The use of BigData has the potential to uncover ‘hidden’ variables. Plugging in these ‘hidden’ variables can improve performance of models that already exist and are used for business decision making. The potential is limitless. Each variables has the probability to improve the existing business decision making systems as well as create new models to solve new business problems.

The potential of BigData and the confidence that skills acquired till date in Analytics are very relevant to this new type of data is a great take-away. Analytics and Data Science are here to stay and to make business decisions more effective and accurate.

Do you have a comment ? Please write in to me at

Leave a Reply

Your email address will not be published. Required fields are marked *