Feature request: "outputIndexFunction" for Elasticsearch writer

[Vladimir Mikhailov <v....@content-media.ru>]

Hi

HDFS Writer has great functionality for defining the destination folder for indexing data:

{
  "index": "bro",
  "batchSize": 5,
  "outputPathFunction": "FORMAT('uid-%s', uid)"
}

https://github.com/apache/metron/blob/master/metron-platform/metron-writer/README.md#hdfs-writer

Is it possible to make similar functionality for Elasticsearch Writer?
(something like "outputIndexFunction")

Typical usage scenarios for this:

- different indices of the same data type for different clients,
- separate indices for pilot projects,
- different rollover policies for indices, depending on the condition.

It would also be convenient to have such setting at the global level so that it affects all active parsers.

Re: Feature request: "outputIndexFunction" for Elasticsearch writer

[Otto Fowler <ot...@gmail.com>]

What might even be more interesting would be to have stellar evaluate
conditions and set the index based on the evaluation:

pseudo:

IF ( parser == BRO ) THEN match(FIELD =x) index = y

or something




On December 19, 2019 at 05:14:01, Vladimir Mikhailov (
v.mikhailov@content-media.ru) wrote:

Hi

HDFS Writer has great functionality for defining the destination folder for
indexing data:

{
"index": "bro",
"batchSize": 5,
"outputPathFunction": "FORMAT('uid-%s', uid)"
}

https://github.com/apache/metron/blob/master/metron-platform/metron-writer/README.md#hdfs-writer

Is it possible to make similar functionality for Elasticsearch Writer?
(something like "outputIndexFunction")

Typical usage scenarios for this:

- different indices of the same data type for different clients,
- separate indices for pilot projects,
- different rollover policies for indices, depending on the condition.

It would also be convenient to have such setting at the global level so
that it affects all active parsers.