Handling Java Exceptions in ELK Stack

I use ELK (Elasticsearch, Logstash, Kibana) for visualizing and analyzing different things like Nginx logs, Application logs etc.
 
When ingesting application logs into ELK, one thing I wanted to achieve was how to handle the error or exceptions occurring in the logs. Surely Nginx logs can return 2xx but still, there can be some internal error in the application. Our application was written in Java, so I am only concentrating on it.

A simple java exception looks like :

Exception in thread "main" java.lang.NullPointerException
        at com.example.project.Book.getTitle(Book.java:16)
        at com.example.project.Author.getBookTitles(Author.java:25)
        at com.example.project.Bootstrap.main(Bootstrap.java:14)

Typically this will be parsed line by line in ELK, that means 4 different lines for the above single log.
 
So what we did was modified the

logstash pipeline in such a way that if the log contains a "tab" at the beginning, parse it as a single log till the tab is not found. 
This is because an exception occurred in java always start with a tab.

Here I'm using beats to send application logs to logstash server.
 
Logstash configuration :

input {
  beats{
    port => 5044
    codec => multiline {
      pattern => "^\t"
      what => "previous"
    }
  }
}

filter {
  if [message] =~ "^[^\t]" {
    grok {
      match => [ "message" , "%{JAVASTACKTRACEPART}"  ]
      add_tag => ["stacktrace"]
    }
  }
}

output{
  if "stacktrace" in [tags] {
    slack {
       url => "<your-slack-webhook-url"
       username => ["<your-username>"]  #optional
       icon_emoji => ["<your-emoji>"]   #optional
    }
    email {
       to => '<to-email-id>'
       codec => "plain"
       debug => 'true'
       via => 'smtp'
       address => 'smtp.gmail.com'
       username => '<from-email-id>'
       password => '<password>'
       subject => 'Alert - Code break in Production'
       body => '%{message}'
       port => 25
       use_tls => true
    }
  }
}

The above configuration will parse the exception as a single log using the multiline plugin in the input field.

The filter field in the logstash pipeline will tag that exception with "stack trace" field and extract useful information like Exception, class, line etc using the built-in logstash parser "JAVASTACKTRACEPART"

The last output field will send the exception as a message on your slack channel and Gmail account notifying you or your development team about the exception that occurred in the application which can be then corrected.

Conclusion :
In this blog, we learned how to handle the java exceptions in ELK stack and notify about the same on slack and Gmail.