Flume ElasticSearchSink - how to customize index name

[Marina <pp...@yahoo.com>]

Hi,I'm researching am option of populating ElasticSearch cluster from Flume for Apache-like logs. One of the requirements is to add log events to a specific index - the index name has to be determined based on a 'customerId'-like property available in the log message, that can be parsed out.I see that I can potentially customize event parsing logic by writing a custom SErializer by implementing the ElasticSearchIndexRequestBuilderFactory interface.Any suggestion on how I could also customize index selection?
thanks!Marina
 

Re: Flume ElasticSearchSink - how to customize index name

[Marina <pp...@yahoo.com>]

Additions to my earlier question:
1. It looks like I can customize which index to write into in the prepareIndexREquest(...) method - when implementing my custom event serializer. I could probably do something like this (very pseudo-code-ish):-- get customerId from the Event [actually, where is the org.apache.flume.Event class? I could not find it in the  apache-flume-1.5.2-src.tar ...]-- create String myIndexName = 'indexPrefix-from-config' + "_" + customerId-- in [repareIndexRequest(....) do:      indexRequestBuilder.setIndex(myIndexName). ....would this work?
2. However, I am not sure how and when this index is created.... ? I understand that by default a new index with the configured indexPrefix will be created every day - is it possible to customize this behavior?
3. And one more question: I see that it is the IndexREquestBuilder class that is used for indexing, and not BulkRequestBuilder. Does it mean that Flume indexes one event at a time only? without any batching?
thanks!Marina
      From: Marina <pp...@yahoo.com>
 To: "user@flume.apache.org" <us...@flume.apache.org> 
 Sent: Wednesday, April 15, 2015 4:06 PM
 Subject: Flume ElasticSearchSink - how to customize index name
   
Hi,I'm researching am option of populating ElasticSearch cluster from Flume for Apache-like logs. One of the requirements is to add log events to a specific index - the index name has to be determined based on a 'customerId'-like property available in the log message, that can be parsed out.I see that I can potentially customize event parsing logic by writing a custom SErializer by implementing the ElasticSearchIndexRequestBuilderFactory interface.Any suggestion on how I could also customize index selection?
thanks!Marina