As monitoring and alerting matured, the focus was on gathering & logging health metrics to the smallest quantum of time or other units. This led to increased awareness of state of a system in real-time. However that became insufficient to deliver innovative or disruptive business ideas required to establish market leadership.
Much of an organizations efficiency lay in its ability to deliver operational excellence. This was typically supported by technologies which monitor, alert and report data on ‘state-of-the-company’. However those who analyzed and learned from these data points became more resilient and soon their ecosystems functioned at the ‘next level’. When their tools & technologies acquired this habit, it became a machine learning experience. When they added speed, they learned faster. When they added non-correlated data sources (e.g. social chatter), they learned newer ways of reinventing the way they did business. As a result almost anything on the planet became a data source. Slowly data had acquired volume, velocity and variety.
The goal-post was moving from capturing small data points to getting overwhelmed with big data sets. To embrace this reality and be the next leader in disruption, one has to have the right tools, references and processing stages working on data. Some in telecom arena may liken this deluge to extracting “signal” from “noise”, the difference being “signal” was unknown and waiting to be discovered.
One can help connect these dots of big data by getting smart on-
- Using right tools for data gathering. Different cloud providers provide services for ingestion of data whether it is transactional, real-time or batch.
- Using right storage to save data. Data may need to be stored structured/raw, transient/long-term, distributed/columnar.
- Using right processing tools. Data may come in batches or streaming fashion and each sample will provoke questions and produce answers (predictions).
