How to Run ELK on Windows

Required Windows executables:

• zookeeper-3.4.8
• kafka_2.11
• elasticsearch-2.3.3
• logstash-2.3.2
• kibana-4.5.1-windows

Introduction

This article helps kickoff beginner to start working on ELK with basic/default configurations, i.e., Elasticsearch 2.3.3 + Logstash 2.3.2 + Kibana 4.5. It is mainly for making above things run on Windows 7 + platform and some tricks to make it work. I am putting together ways to solve the common issues that I faced while configuring the above things.

Background

You can get more information on architecture and working of Kafka ELK on respective official ebsites. Google for it. :). For quickly knowing about these terms, you can treat each of them as:

• Zookeeper: Kind of server which keeps track of Topics, Clusters, Message indices
• Kafka: A log server
• Elasticsearch: Your fast searcher
• Logstash: Mediator between Kafka logs and Elasticsearch logs
• Kibana: Reporting on visualization server/tool

Using the code/batch scripts for configuring Kafka ELK on Windows 7 onward:

For editing files, I prefer Notepad++.

You need the below zip files extracted to get started configuring each of the below:

1. jre1.8.0_92 or jre1.8.0_92 depending on 32/64 bit architecture. I am using 64 bit machine.
2. Zookeeper 3.4.8
3. Kafka 2.11
4. Elasticsearch 2.3.3
5. Logstash 2.3.2
6. kibana-4.5.1-windows

I have extracted them on D:. You can choose one from your drive. But remember to change on every place of code where it appears to be D: with one that you have selected.

1) JRE

Install Java and set environment variable:

JAVA_HOME = C:\Program Files\Java\jre1.8.0_92

Those who are new to environment variables can Google how to set environment variables.

While setting environment variable under Path field, add the path to Java runtime. C:\Program Files\Java\jre1.8.0_92

2) Zookeeper

//Rename D:\zookeeper-3.4.8\conf\zookeeper.cfg file to D:\zookeeper-3.4.8\conf\zoo.cfg, edit the respective line to set data directory as:

dataDir=D:/zookeeper-3.4.8/data

To start zookeeper, open new Command window and execute:

D:\zookeeper-3.4.8>bin>zkserver

3) Kafka

To start Kafka server, open new Command window and execute:

D:\kafka_2.11-0.10.0.0>.\bin\windows\kafka-server-start.bat .\config\server.properties

To create a topic in Kafka, open new Command Window and execute the command below. You may create many topics, but go with the one below topic name 'test'.

D:\kafka_2.11-0.10.0.0>.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181
--replication-factor 1 --partitions 1 --topic test

To create a Producer, open new command window and execute:

D:\kafka_2.11-0.10.0.0>.\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test

In Producer window, you can type messages to be logged by Kafka. These messages are later used by Logstash to push them to Elasticsearch logs.

4) Elasticsearch

To configure elasticsearch with defaults, open new command window and execute:

D:\elasticsearch\bin>elasticsearch

To install elasticsearch service, open new command window and execute:

D:\elasticsearch\bin>service install

To start elasticsearch service, execute:

D:\elasticsearch\bin>service start

If you are unable to install elasticsearch service, you can try the trick below:

1. Extract old version of elasticsearch 1.4.4 on D:\
2. Delete Bin, Config, Lib folders from elasticsearch 1.4.4
3. Paste Bin, Config, Lib folders from elasticsearch 2.3..3 into elasticsearch 1.4.4 folder.
4. Run the below commands again. It should successfully install and start elasticsearch service.

D:\elasticsearch\bin>elasticsearch
D:\elasticsearch\bin>service install
D:\elasticsearch\bin>service start

Default Url for elasticsearch will be:

http://localhost:9200

Tip: For testing purposes, you can use Fiddler.exe to post json request to the above url and get your search result.

5) Logstash

For using Logstash to write to elasticsearch logs, open new command window again and execute:

D:\logstash-2.3.2\bin>logstash -e "input { kafka { topic_id => 'test' } }
output { stdout{ } elasticsearch { hosts => localhost } }"

If you get the error stating "The signal HUP is used by JVM...", there is a trick you can do.

Try installing logstash plugin by executing the command:

D:\logstash-2.3.2\bin>logstash-plugin.bat install logstash-output-kafka-master\logstash-output-kafka

Then, execute the original command:

D:\logstash-2.3.2\bin>logstash -e "input { kafka { topic_id => 'test' } }
output { stdout{ } elasticsearch { hosts => localhost } }"

It should start you logstash, that will get kafka messages posted to elasticsearch.

6) Kibana

To start Kibana server, open new command window and execute:

D:\kibana-4.5.1-windows\bin>kibana

You will get message kibana server running at http://0.0.0.0:5601.

Use actual url to access Kibana: http://localhost:5601

Use logstash-* in kibana to list all logs.

YOU ARE NOW SET TO CREATE YOUR FIRST KIBANA VISUALIZATION/REPORT.

Points of Interest

Configure all the applications to work together and learn reporting, analyzing, visualizing data using Kibana.