I have been using Graylog2 and VMware Log Insight for some time now and wanted to try out Logstash finally. So the first thing that I wanted to do was create an automated script to do most of the install and configuration to get everything running. I figured that as I am going through this I would share with everyone and start building on this script more based on feedback. I created a Graylog2 script (located here) that has proven to be of great help to the community and figured I might be able to do the same with the Logstash community, but even if it didn’t I would learn a great deal about Logstash in the meantime. There is a great community around Logstash so getting support should be very easy. As well as, I am just starting to learn Logstash now so this should be a lot of fun. Which also means that there will be a good amount of change around this post.
First off I will be keeping this script updated and available on Github located here. This will be the only location that I will be keeping up with it.
I would recommend using a clean install of Ubuntu 12.04 to install onto. However if you decided to install on an existing server I am not responsible for anything that may get broken.
So here is how we get started and get everything up and running. Open up a terminal session on your server that you will be installing to and run the following commands.
sudo apt-get update sudo apt-get -y install git cd ~ git clone https://github.com/mrlesmithjr/Logstash_Kibana3 chmod +x ./Logstash_Kibana3/install_logstash_kibana_ubuntu.sh sudo ./Logstash_Kibana3/install_logstash_kibana_ubuntu.sh
You will be prompted during the script to enter your domain name and ESXi naming convention. This will be used to configure logstash filtering for your ESXi hosts. If you do not monitor any ESXi just enter some random info into these. These are purely just collecting info to pass into a filtering rule for Logstash.
Once complete open your browser of choice and connect to http://logstashservername/kibana or http://ipaddress/kibana.
You will see the following screen once connected. Seeing as we are setting up Logstash with Kibana go ahead and select the link on the left.
Now here is a screenshot of some actual ESXi logging. Notice the tag called esxi, that is created by the filtering rule that we created with the installer which, is based off of the naming convention we passed to the installer.
Here is another screenshot of logging graphs by adding different search criteria items.
So what we have done with this script is installed Apache2, Elasticsearch, Logstash and Kibana3. Logstash has been configured to listen on UDP/514 (default syslog) and TCP/514 for ESX(i) logging.
If you want to purge and expire old logs have a look here. Jordan Sissel (creator of Logstash) has provided a python script to do this.
Here is how you setup the script. Open a terminal on your Logstash server and execute the following.
cd ~ sudo apt-get install python-pip sudo apt-get install git git clone https://github.com/logstash/expire-logs cd expire-logs sudo pip install -r requirements.txt
Now that you have this setup read the examples on the github link on different scenarios.
After you purge your logs using the above method you will need to restart elasticsearch.
sudo service elasticsearch restart
That should be it.
All comments and feedback are very much welcomed and encouraged.