ELK(Elastic, Logstash and Kibana) and Filebeat Stack with Dockers

In the Application Operations world, with huge infrastructure estate & multi tenant applications. Log analysis is not a easy task. During any major issues it is always important to review analyse and interpret the logs. It's never an easy task to perform with a ssh console.

Centralised logging will help to provide visibility of the health & performance of the application, enabling the operation team to easily drill down to issues.

What should we expect from a open search and analytics engine tool,

• Collection of data 
• Centralise and index data
• Search  
• Monitoring and alerting 
• Report and dashboard

Currently there are two big players in the log analytic world - Splunk & ELK stack.

Splunk has been a market leader till date while the ELK stack is an open source and a strong rival to the current one.

Comparing both the products give more or less the same features at some price point. The learning curve of Splunk is higher than that of ELK and the price tag is associated with Splunk.

We will explore on the ELK stack with setting up the stack and filebeat to push logs on a dockerized environment.

Elasticsearch :
Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured. Elasticsearch is built on Apache Lucene and was first released in 2010.

Raw data flows into Elasticsearch from a variety of sources, including logs, system metrics, and web applications. Data ingestion is the process by which this raw data is parsed, normalized, and enriched before it is indexed in Elasticsearch. Once indexed in Elasticsearch, users can run complex queries against their data and use aggregations to retrieve complex summaries of their data.

Logstash:

Logstash is used to aggregate and process data and send it to Elasticsearch. Logstash is an open source, server-side data processing pipeline that enables us to ingest data from multiple sources simultaneously and enrich, filter and transform it before it is indexed into Elasticsearch.

Elasticsearch:

Kibana is a data visualisation and management tool. It provides data from the Elasticsearch in the form of  real-time  histograms, line graphs, pie charts, and maps. Kibana also includes advanced applications such as Canvas, which allows users to create custom dynamic infographics based on their data.

Filebeat:

Filebeat is a lightweight shipper for forwarding and centralizing log data.Filebeat, looks at the locations you’ve specified for log data. For each log that Filebeat locates, Filebeat starts a harvester. Each harvester reads a single log for new content and sends the new log data to libbeat, which aggregates the events and sends the aggregated data to the output that you’ve configured for Filebeat.

Setup and initialise a ELK containers:

Setting up ELK stack on docker host is a quick task if the right configurations are placed and right ports have been mapped.

Below is the use-case where we have two containers with Weblogic, SOA & MFT installed on it, and the we would be pushing the server, diagnostic, access logs to elasticsearch so that those are visible in Kibana.

Below picture describes the flow and the architecture.

Starting the ELK containers:

Below is the docker compose file which can be executed to setup the ELK containers, the file is available at docker-compose.yml

Below updates are required,

1 -  Name of the container as per your choice.

2 - Specify the volume for Elasticsearch where the data can be persisted.

3 - Ports 9200, is the default port of Elasticsearch JSON interface, 9300 is additionally opened.

4 - Specify the volume for Logstash where the data can be persisted. Note the Logstash.conf file needs to                   be presenting the mounted path. Details of the logstash.conf file is below.
5 - Ports 9600 is used for the Logstash monitoring API, 5044 for Logstash filebeats interface.

6 - URL for the Kibana and Elasticsearch interface.

Logstash.conf : 

Below is the logstash.conf file,  the file is available at Logstash.conf

1 - Port on which the Logstash will read the inputs from filebeat.

2 - URL of the Elasticsearch to push the data.

After the configurations are in place, the containers can be started using the command.

docker-compose up

Once the docker containers are up the Kibana interface will be available on the 5601 port or the one mapped to as per the docker file.

Filebeat Container:

Starting up the filebeat container below is the docker command to execute, which can be found at readme.md

1 - name of the container and the network

2 - filebeat configuration file filebeat.docker.yml. remember the "ro" at the end as this needs root.

3 - Volume of the Weblogic containers are attached so that filebeat can read through the logs.

Below is the  filebeat.docker.yml which is configured to push log files from the loaded volumes of the Weblogic containers to Logstash, which can be found at  filebeat.docker.yml

1- id unique id for data, tags which helps for the queries in Kibana. 

2 - Path of the log file to be tracked.

3 - the Logstash configuration port to push the data.

Kibana : 

Once the stack is up, Kibana can be accessed on the configured port http://<IP>:7031 in my case.

you can check the ids and tags configured in the filebeat configurations.

Creating index :

Data won't be visible till a index is created,

As the data is delivered with Logstash an index is to be created, check the below.

Now the stack is ready, above all configuration can be changed as per the needs. Hope you had a great time reading and hope this aided you in configuring it, if you face any issues reach out to me.