Log management is very important for monitoring server health, application health and security breaches. Log Management include below activity
– Receive logs and sanitize it if require.
– Store Logs .
– View logs, co-relate logs , analysis and reporting.
Each activity is important, and there is special tool for each activity
– Receive logs and sanitize it if require. – Logstash
– Store Logs . – ElasticSearch
– View logs, co-relate logs , analysis and reporting. – Kibana
Last thing first, Kibana does not interact with LogStash, it interact with ElasticSearch only. ElasticSearch support json queries over HTTP, We can query ElasticSearch directly and can get result in json format. Getting data in json format in not sufficient, many times we need to input these json data to graphs and Kibana do that exactly. Kibana send queries to ElasticSearch and get data in json format and then visualize that data for you as per requirement.
While Kibana takes input from ElasticSearch, ElasticSearch takes input from LogStash. LogStash act as listener for incoming logs, incoming logs can be in any format like syslog, netflow, Apache logs etc. LogStash accepts incoming logs and can filter that before dispatching that to ElasticSearch. Generally LogStash server considered as centralized log server.