Soliciting input: ElasticSearch receiver

[Scott Deboy <sc...@gmail.com>]

I'm looking at adding an ES receiver and was curious what folks would
like to see when it comes to configuration options/capabilities, other
than the ability to continually retrieve new events on some polling
interval, which I'll make sure to add.

Scott

Re: Soliciting input: ElasticSearch receiver

[Gary Gregory <ga...@gmail.com>]

And that will be in a new module ;-)

Gary

On Sat, Jan 27, 2018 at 6:47 PM, Matt Sicker <bo...@gmail.com> wrote:

> On the sending side, we'd want to be able to configure index name, mapping
> type name, override mapping settings, bulk request (i.e., batch) size,
> refresh interval overrides, something similar to the column mappings thing
> in the Cassandra and JDBC plugins, authentication, could be missing some
> ideas (which I'm sure I'll remember next week when I'm deep in ES again).
>
> On the receiving side, ability to specify an arbitrary query would be
> great. A minimal query feature could be to just specify the index name and
> do a match_all query on it. Add in a polling interval and JSON parsing
> (potentially with mappings from JSON output to whatever internal LogEvent
> type class is in use) along with authentication and that'd cover a lot of
> the basics. More advanced features are in Kibana <
> https://www.elastic.co/products/kibana>, so we could always take some
> ideas
> from there as well.
>
> For me, if I'm ever using ES for log data, I use it mostly for interactive
> queries, not for polling. For continual log ingestion, I'd go with Kafka or
> Flume depending on the infrastructure in place.
>
> On 27 January 2018 at 18:58, Remko Popma <re...@gmail.com> wrote:
>
> > Sorry I won’t be able to help you with that; no experience with
> > ElasticSearch.
> >
> > Remko
> >
> > (Shameless plug) Every java main() method deserves http://picocli.info
> >
> > > On Jan 28, 2018, at 9:48, Scott Deboy <sc...@gmail.com> wrote:
> > >
> > > I'm looking at adding an ES receiver and was curious what folks would
> > > like to see when it comes to configuration options/capabilities, other
> > > than the ability to continually retrieve new events on some polling
> > > interval, which I'll make sure to add.
> > >
> > > Scott
> >
>
>
>
> --
> Matt Sicker <bo...@gmail.com>
>

Re: Soliciting input: ElasticSearch receiver

[Matt Sicker <bo...@gmail.com>]

On the sending side, we'd want to be able to configure index name, mapping
type name, override mapping settings, bulk request (i.e., batch) size,
refresh interval overrides, something similar to the column mappings thing
in the Cassandra and JDBC plugins, authentication, could be missing some
ideas (which I'm sure I'll remember next week when I'm deep in ES again).

On the receiving side, ability to specify an arbitrary query would be
great. A minimal query feature could be to just specify the index name and
do a match_all query on it. Add in a polling interval and JSON parsing
(potentially with mappings from JSON output to whatever internal LogEvent
type class is in use) along with authentication and that'd cover a lot of
the basics. More advanced features are in Kibana <
https://www.elastic.co/products/kibana>, so we could always take some ideas
from there as well.

For me, if I'm ever using ES for log data, I use it mostly for interactive
queries, not for polling. For continual log ingestion, I'd go with Kafka or
Flume depending on the infrastructure in place.

On 27 January 2018 at 18:58, Remko Popma <re...@gmail.com> wrote:

> Sorry I won’t be able to help you with that; no experience with
> ElasticSearch.
>
> Remko
>
> (Shameless plug) Every java main() method deserves http://picocli.info
>
> > On Jan 28, 2018, at 9:48, Scott Deboy <sc...@gmail.com> wrote:
> >
> > I'm looking at adding an ES receiver and was curious what folks would
> > like to see when it comes to configuration options/capabilities, other
> > than the ability to continually retrieve new events on some polling
> > interval, which I'll make sure to add.
> >
> > Scott
>



-- 
Matt Sicker <bo...@gmail.com>

Re: Soliciting input: ElasticSearch receiver

[Remko Popma <re...@gmail.com>]

Sorry I won’t be able to help you with that; no experience with ElasticSearch. 

Remko 

(Shameless plug) Every java main() method deserves http://picocli.info

> On Jan 28, 2018, at 9:48, Scott Deboy <sc...@gmail.com> wrote:
> 
> I'm looking at adding an ES receiver and was curious what folks would
> like to see when it comes to configuration options/capabilities, other
> than the ability to continually retrieve new events on some polling
> interval, which I'll make sure to add.
> 
> Scott