A common use case in big data is processing of logs and
getting some insight on how application is performing through visualization.
The problem however is that these logs are often plain text files
located on multiple servers in different formats. Consolidating all
logs, making them searchable and providing insights based on them is a challenge.
Log analysis software features
Any real time log processing software should support
at least the following features:
i)
Variety of different logs structures
ii)
Massive amount of data spread over multiple
nodes
iii)
Horizontal scalability as data grow
iv)
Fast searching
v)
Ad-hoc queries
vi)
Customizable visualization.
There are variety of log processing/analysis system
available in the market, one such popular log analysis system is Splunk. Splunk
has been facing more and tougher competition with each passing month and ELK is
the latest competitor of Splunk.
ELK
ELK is a term for integrated stack of Elasticsearch,
Logstash and Kibana. All three are available as separate components and have
different functionality. While Logstash is used for log processing and
ingestion, Kibana provides the visualization over the searchable content stored
in elasticsearch. Let see how the complete stack works together.
1.
Logstash has the capability of processing
different formats of log structure, further multiple logstash agents installed on
different nodes can ingest data into single elasticsearch cluster
2.
Once the data is structurally stored in
elasticsearch, it is searchable and Adhoc queries can be made on elasticsearch
using ‘curl’ or plugin such as ‘head’
3.
Kibana provides stunning visuals by reading the
data directly from elasticsearch,
further it provides real time dashboard updates as soon as the record is
available in elasticsearch
A Simple Architecture or data flow of ELK
As ELK consists of different components for processing,
storing and visualization, therefore, it is highly scalable and can be fit in
any customized requirement.
A scale out Architecture
A visual GUI interface such as Head can be used to make
adhoc queries and search on data stored on ElasticSearch.
Query browser in elasticsearch
Kibana
Kibana deserves a special mention as it has outperformed even
some of the paid tools with its stunning visuals and ease of configuration. Its
completely open source and currently only reads data from elasticsearch. Kibana
shows mostly time-series based graphs, it provides some basic analytical
functions such as sum, count, average, max, min, top n, etc. Here are some of
the Kibana dashboard visuals.
All of the three components are part of
elasticsearch.org. So far, the biggest
distinction between Splunk and its competition is how they're productized.
Elasticsearch also has been commercializing ELK by monetizing analytics
and is offering commercial support for all of its components.
