ActiveMQ as a Message Broker for Logstash

ActiveMQ as a Message Broker for Logstash

Logstash, it is common to add a message broker that is used to temporarily buffer incoming messages before they are being processed by one or more Logstash nodes or writing output messages from one or more Logstash nodes. ActiveMQ is a widely used messaging and integration platform that supports different protocols and looks just perfect for the use as a message broker. Let's see the options to integrate it.

ActiveMQ supports the Stomp protocol and the Stomp - JMS mapping. This makes it easy to write a client in pure Ruby, Perl, Python or PHP for working with ActiveMQ.

Its very easy to enable ActiveMQ for Stomp. Just add a connector to the broker using the stomp URL.

<transportConnectors>

   <transportConnector name="stomp" uri="stomp://localhost:61613"/>

</transportConnectors>

For more details : http://activemq.apache.org/stomp.html

Broadcasting messages with STOMP

Simple Text Oriented Messaging Protocol is an option that is supported by ActiveMQ. Fortunately there is a dedicated input for it. It is not included in Logstash by default but can be installed easily.

 /bin/logstash-plugin install logstash-output-stomp 

Afterwards we can just use it in our Logstash config

output {

        stomp{

        debug => true

        host => "192.168.1.10"

        port => 61613

        user => "xxxx"

        password => "xxxx"

        destination => "vimal"

    }

}

Message at AMQ

Getting data in ActiveMQ

When using Java you can use something like a Log4j- or Logback-Appender that push the log events directly to the queue using JMS. When it comes to shipping data unfortunately none of the more popular solutions seems to be able to push data to ActiveMQ.