Beats processor?

["Gaspar, Carson" <ga...@deshaw.com>]

Is anyone working on a beats producer / consumer to integrate with the Elastic ecosystem?

Re: Beats processor?

[Andre <an...@fucs.org>]

Carson,

Sort of... there is a processor for the lumberjack protocol
(ListenLumberjack) that supports logstash-forwarder.

Because of the great similarity between the old Lumberjack and Beats,
extending ListenLumberjack to support "Lumberjack v2" (aka Beats) should be
reasonably trivial, however, elastic has yet to formalise what the protocol
is[1][2], leaving us with the choice of supporting what we understand it to
be or to wait.

I chose to wait but I suspect nobody will complain if someone decides to
integrate based on reversing the golang source code.

Cheers


[1] https://github.com/elastic/libbeat/issues/279#issuecomment-193373322
[2] https://github.com/elastic/go-lumber/issues/1#issuecomment-257786505



On Sat, Dec 10, 2016 at 2:55 AM, Gaspar, Carson <ga...@deshaw.com> wrote:

> Is anyone working on a beats producer / consumer to integrate with the
> Elastic ecosystem?
>
>
>

RE: Beats processor?

["Gaspar, Carson" <ga...@deshaw.com>]

That would be great if minify cpp were stable and had windows event log support, but sadly that isn't true, yet.

-----Original Message-----
From: Matt Burgess [mailto:mattyb149@apache.org] 
Sent: Friday, December 09, 2016 11:11 AM
To: users@nifi.apache.org
Subject: Re: Beats processor?

Not that I know of, I talked to a friend of mine who was interested but he's working on a different beat at the moment. In NiFi/MiNiFi the approach has been to get the data from the edge using MiNiFi agents (same place you might drop an Elasticsearch beat), sending back to a NiFi and using PutElasticsearch (using the transport protocol) to get the data into Elasticsearch.

How did you envision a beat being used? I think it would be great to have a Go library for site-to-site communications, then leverage that in a beat or anything else. The beat could ingest from a site-to-site connection and send to ES, for example. Then a MiNiFi agent could simply write to an output port (i.e. not needing to know there's a beat or anything else on the other side) and the beat could do the transport.  Having said that, a beat may be faster for this use case but you'd be losing other features/capabilities that NiFi offers such as provenance, replay, etc.

Regards,
Matt

On Fri, Dec 9, 2016 at 10:55 AM, Gaspar, Carson <ga...@deshaw.com> wrote:
> Is anyone working on a beats producer / consumer to integrate with the 
> Elastic ecosystem?
>
>

Re: Beats processor?

[Andre <an...@fucs.org>]

Matt,

> have a Go library for site-to-site communications, then leverage that
> in a beat or anything else. The beat could ingest from a site-to-site
> connection and send to ES, for example.

Because of their use of golang, extending beats requires recompilation

https://github.com/elastic/beats/pull/1525#issuecomment-217651768



On Sat, Dec 10, 2016 at 3:11 AM, Matt Burgess <ma...@apache.org> wrote:

> Not that I know of, I talked to a friend of mine who was interested
> but he's working on a different beat at the moment. In NiFi/MiNiFi the
> approach has been to get the data from the edge using MiNiFi agents
> (same place you might drop an Elasticsearch beat), sending back to a
> NiFi and using PutElasticsearch (using the transport protocol) to get
> the data into Elasticsearch.
>
> How did you envision a beat being used? I think it would be great to
> have a Go library for site-to-site communications, then leverage that
> in a beat or anything else. The beat could ingest from a site-to-site
> connection and send to ES, for example. Then a MiNiFi agent could
> simply write to an output port (i.e. not needing to know there's a
> beat or anything else on the other side) and the beat could do the
> transport.  Having said that, a beat may be faster for this use case
> but you'd be losing other features/capabilities that NiFi offers such
> as provenance, replay, etc.
>
> Regards,
> Matt
>
> On Fri, Dec 9, 2016 at 10:55 AM, Gaspar, Carson <ga...@deshaw.com> wrote:
> > Is anyone working on a beats producer / consumer to integrate with the
> > Elastic ecosystem?
> >
> >
>

Re: Beats processor?

[Matt Burgess <ma...@apache.org>]

Not that I know of, I talked to a friend of mine who was interested
but he's working on a different beat at the moment. In NiFi/MiNiFi the
approach has been to get the data from the edge using MiNiFi agents
(same place you might drop an Elasticsearch beat), sending back to a
NiFi and using PutElasticsearch (using the transport protocol) to get
the data into Elasticsearch.

How did you envision a beat being used? I think it would be great to
have a Go library for site-to-site communications, then leverage that
in a beat or anything else. The beat could ingest from a site-to-site
connection and send to ES, for example. Then a MiNiFi agent could
simply write to an output port (i.e. not needing to know there's a
beat or anything else on the other side) and the beat could do the
transport.  Having said that, a beat may be faster for this use case
but you'd be losing other features/capabilities that NiFi offers such
as provenance, replay, etc.

Regards,
Matt

On Fri, Dec 9, 2016 at 10:55 AM, Gaspar, Carson <ga...@deshaw.com> wrote:
> Is anyone working on a beats producer / consumer to integrate with the
> Elastic ecosystem?
>
>