[jira] [Commented] (NIFI-856) Add Processor for Lumberjack protocol

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15223564#comment-15223564 ] 

ASF GitHub Bot commented on NIFI-856:
-------------------------------------

Github user trixpan commented on the pull request:

    https://github.com/apache/nifi/pull/290#issuecomment-205082226
  
    @bbende @joewitt can one of you guys bump travis-ci ? 
    
    this should pass build without dramas


> Add Processor for Lumberjack protocol
> -------------------------------------
>
>                 Key: NIFI-856
>                 URL: https://issues.apache.org/jira/browse/NIFI-856
>             Project: Apache NiFi
>          Issue Type: New Feature
>            Reporter: Mike de Rhino 
>              Labels: features
>
> It would be great if NIFI could support the [lumberjack protocol|https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md] so to enable the use of logstash forwarder as a source of data.
> A lot of non Java shops tend to avoid installing Java at data producing nodes and instead of Flume they end up using things like kafka, heka, fluentd or logstash-forwarded as data shipping mechanisms. 
> Kafka is great but its architecture seem to be better focused on multi-DC environments instead of multi-branch scenarios (imagine having to manager 80 Zookeeper quorum, one for each country where you operate?)
> [Heka|https://github.com/mozilla-services/heka] is fine, it has decent backpressure buffering but no concept of acknowledgement on the receiving side of a TCP stream. If the other end of a TCP stream is capable of listening but gets stuck with its messages it will keep spitting data through the pipe, oblivious to the woes at the other end.
> Logstash forwarder in the other hand, is a quite simple tool, with a reasonable implementation of acknowledgments on the receiving side but... it depends on Logstash(and logstash has its own issues).
> It would be great if NIFI could serve as a middle man, receiving lumberjack messages and offloading some of the hard work Logstash seems to struggle with (e.g. using NIFI to save to HDFS while a downstream Logstash writes into ES).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)