Day 18: Centralized ELK Stack Setup

In this article, we will walk through setting up a centralized ELK (Elasticsearch, Logstash, Kibana) stack using Docker. This setup is crucial for monitoring and analyzing log data effectively.

Prerequisites

• Java: Ensure you have Java installed, as Elasticsearch requires it.

• Docker and Docker Compose: Install these tools for easier management of containerized applications.

Step 1: Clone the Repository

First, clone the repository where you want to set up your ELK stack:

git clone https://github.com/username/repo-name.git
cd repo-name

Step 2: Set Up Elasticsearch

1. Create a Docker Compose File: Create a file named docker-compose.yml in the root of your project.

   version: '3'
   services:
     elasticsearch:
       image: elasticsearch:7.10.0
       container_name: elasticsearch
       environment:
         - discovery.type=single-node
       ports:
         - "9200:9200"

1. Start Elasticsearch: Run the following command to start the Elasticsearch service:

   docker-compose up -d

Step 3: Set Up Logstash

1. Add Logstash to Docker Compose: Update your docker-compose.yml file to include Logstash.

     logstash:
       image: logstash:7.10.0
       container_name: logstash
       ports:
         - "5044:5044"
       volumes:
         - ./logstash/conf:/usr/share/logstash/pipeline

1. Create Logstash Configuration: Create a directory for Logstash configurations:

   mkdir -p logstash/conf

Then create a file named logstash.conf inside that directory:

   input {
     beats {
       port => 5044
     }
   }
   output {
     elasticsearch {
       hosts => ["elasticsearch:9200"]
       index => "logstash-%{+YYYY.MM.dd}"
     }
   }

Step 4: Set Up Kibana

1. Add Kibana to Docker Compose: Update your docker-compose.yml to add Kibana.

     kibana:
       image: kibana:7.10.0
       container_name: kibana
       ports:
         - "5601:5601"

Step 5: Run the ELK Stack

1. Start All Services: Use the following command to start all services defined in your docker-compose.yml:

   docker-compose up -d

1. Access Kibana: Open your web browser and navigate to http://localhost:5601 to access the Kibana dashboard.

Step 6: Configure GitHub Repository

Initialize Git (if not already initialized):

   git init

Add Your Files:

   git add docker-compose.yml
   git add logstash/conf/logstash.conf

Commit Changes:

   git commit -m "Initial commit of ELK stack setup"

Create a GitHub Repository: Go to GitHub and create a new repository.

Add Remote Origin:

   git remote add origin https://github.com/username/repo-name.git

Push to GitHub:

   git push -u origin master

Troubleshooting Guide

• Slow Docker Compose: Ensure Docker Desktop is running and restart if needed. Check system resources.

• Elasticsearch Issues: Check logs with docker-compose logs elasticsearch for error messages.

• Kibana Access Problems: Ensure port 5601 is free and not blocked by a firewall.

Conclusion

You have successfully set up a centralized ELK stack and pushed your configuration to GitHub. Use Kibana to monitor and analyze your log data effectively. If you encounter any issues, refer to the troubleshooting guide for assistance.