[jira] [Commented] (NIFI-856) Add Processor for Lumberjack protocol

["Andre (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15052305#comment-15052305 ] 

Andre commented on NIFI-856:
----------------------------

[~joewitt], [~bbende]

I've put together half of the processor.In [here|https://github.com/trixpan/nifi-lumberjack-bundle/] you can find an early prototype.

It is already working bi-directionally but it has some stuff missing (more on this later on).

in summary, once decoded, events are added to BlockingQueue (LumberjackLogEvents), from where they are consumed onTrigger and added to the session. 

So far decoding worked fine and I'm not facing issues with it (but I have a limited test case so time will tell).

Anyhow: 

Once the session part is completed, the value of the sequence number for each processed message is added to BlockingQueue(processedEvents). 

The decoder looks for these values to write acknowledgement to the channel. 

*now the rough edges:*

The solution current works on clear text, however, since logtash-forwarder does not allow me to disable TLS I use socat to bypass the currently broken SSL logic.

This means traffic is like:

{code}
logtash-forwarder  < ----- TLS ----- > socat < ------- CLEARTEXT ------- > ListenLumberjack
{code}

If I remove socat I can get the connection established, etc, but I am still figuring out what the best option of reading the data into the Lumberjack Decoder.

To this point, the depth of my JAVA skills is proving to be more challenging than the Lumberjack Protocol itself and I'm seriously puzzled on what is the best way of dealing with the TLS decoded channels. So I would truly appreciate some assistance on that.

You will also probably notice some weird issues around closing channels and some redundant codes and comments inherited from ListenSyslog. Don't worry too much about that, I will clean this up as I progress.

> Add Processor for Lumberjack protocol
> -------------------------------------
>
>                 Key: NIFI-856
>                 URL: https://issues.apache.org/jira/browse/NIFI-856
>             Project: Apache NiFi
>          Issue Type: New Feature
>            Reporter: Mike de Rhino 
>              Labels: features
>
> It would be great if NIFI could support the [lumberjack protocol|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md] so to enable the use of logstash forwarder as a source of data.
> A lot of non Java shops tend to avoid installing Java at data producing nodes and instead of Flume they end up using things like kafka, heka, fluentd or logstash-forwarded as data shipping mechanisms. 
> Kafka is great but its architecture seem to be better focused on multi-DC environments instead of multi-branch scenarios (imagine having to manager 80 Zookeeper quorum, one for each country where you operate?)
> [Heka|https://github.com/mozilla-services/heka] is fine, it has decent backpressure buffering but no concept of acknowledgement on the receiving side of a TCP stream. If the other end of a TCP stream is capable of listening but gets stuck with its messages it will keep spitting data through the pipe, oblivious to the woes at the other end.
> Logstash forwarder in the other hand, is a quite simple tool, with a reasonable implementation of acknowledgments on the receiving side but... it depends on Logstash(and logstash has its own issues).
> It would be great if NIFI could serve as a middle man, receiving lumberjack messages and offloading some of the hard work Logstash seems to struggle with (e.g. using NIFI to save to HDFS while a downstream Logstash writes into ES).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)