Logstash install

Prerequisite

1. java 8

yum update

wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/8u45-b14/jdk-8u45-linux-x64.rpm"

rpm -ivh jdk-8u45-linux-x64.rpm

Install Log stash

1. rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

2. Add the following in your /etc/yum.repos.d/ directory in a file with a .repo suffix, for example logstash.repo

[logstash-5.x] name=Elastic repository for 5.x packages baseurl=https://artifacts.elastic.co/packages/5.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md

3. yum install logstash

4. info

 - /etc/logstash/logstash.yml

5. configurations for input (filebeat in my case)

vim /etc/logstash/conf.d/input-beat.conf

input { beats { port => 5044 } } filter { grok { match => {  "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:log_level}\] %{DATA:path} %{GREEDYDATA:tail}" } // this log format is for flask+ gevent wsgi server log } if "_grokparsefailure" not in [tags] { json { source => "tail" // json will decode target source and add all field to output json. so tail field is not needed anymore } mutate { remove_field => [ "tail", "message"] //will remove specific field gsub => ["timestamp", ",[0-9]+", ""] // replace (target field, regex, newStr)  } } }

6. configurations for output (s3, elasticsearch)

(if you are using aws-es, you should add your instance public ip on the es policy)

vim /etc/logstash/conf.d/output.conf

output { if "_grokparsefailure" not in [tags] { // well parsed    s3 {      access_key_id => "crazy_key"      secret_access_key => "monkey_access_key"      region => "us-east-1" (US Standard)      bucket => "your_bucket"      size_file => 2048 (optional)      time_file => 5    (optional) - Minutes      codec => "json" (or "plain") (optional. Options are "private", "public_read", "public_read_write", "authenticated_read". Defaults to "private" )    } } // all logs elasticsearch {     hosts => ["localhost:9200"]   } }

7. run

systemctl start logstash

8. check log

/var/log/logstash/

Reference

1. https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

2. https://alexandreesl.com/tag/logstash/