You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Tim Bacon (JIRA)" <ji...@apache.org> on 2013/04/29 11:04:16 UTC

[jira] [Comment Edited] (FLUME-2015) ElasticSearchSink: need access to IndexRequestBuilder instance during flume event processing

    [ https://issues.apache.org/jira/browse/FLUME-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644371#comment-13644371 ] 

Tim Bacon edited comment on FLUME-2015 at 4/29/13 9:03 AM:
-----------------------------------------------------------

Thanks for the positive feedback! I've attached a link to the [Code Review Board|https://reviews.apache.org/r/10835/] at the top of the page.

In reply to your comments:
* My (non-logging-related) use case has no need for the timestamp to be provided as a header on every event, which is why I didn't add the warnings. If this makes the patch unsuitable for your LogStash use case I can always add it back in such a way as to avoid the warnings in my own code path...
* There were 2 reasons why I wrapped the incoming event rather than adding the timestamp as another param
** I was striving to keep backwards-compatibility (speaking as someone who has been burned by the lack of it in the past!) and the contract with existing clients is based only on the event param
** The timestamp used to build the index name should match the timestamp value that goes into the index, and it's only at the level above {{ElasticSearchLogStashEventSerializer}} that consistency can be ensured
* Having said that I understand your concern about GC -- and, if it helps, I haven't noticed any performance hit while running the new code in my own flume test environment (with several million events per day)

Rgds,
Tim
                
      was (Author: prime8):
    Thanks for the positive feedback! I've attached a link to the [Code Review Board|https://reviews.apache.org/r/10835/] at the top of the page.

In reply to your comments:
* My (non-logging-related) use case has no need for the timestamp to be provided as a header on every event, which is why I didn't add it. If that makes it unusable for your LogStash use case I can always add it back in such a way as to avoid the warnings in my own code path...
* There were 2 reasons why I wrapped the incoming event rather than adding the timestamp as another param
** I was striving to keep backwards-compatibility (speaking as someone who has been burned by the lack of it in the past!) and the contract with existing clients is based only on the event param
** The timestamp used to build the index name should match the timestamp value that goes into the index, and it's only at the level above {{ElasticSearchLogStashEventSerializer}} that consistency can be ensured
* Having said that I understand your concern about GC -- and, if it helps, I haven't noticed any performance hit while running the new code in my own flume test environment (with several million events per day)

Rgds,
Tim
                  
> ElasticSearchSink: need access to IndexRequestBuilder instance during flume event processing
> --------------------------------------------------------------------------------------------
>
>                 Key: FLUME-2015
>                 URL: https://issues.apache.org/jira/browse/FLUME-2015
>             Project: Flume
>          Issue Type: Improvement
>          Components: Sinks+Sources
>    Affects Versions: v1.3.0, v1.3.1
>            Reporter: Tim Bacon
>         Attachments: FLUME-2015-1.4.0-0.patch, FLUME-2015-doc-1.4.0-0.patch
>
>   Original Estimate: 0h
>  Remaining Estimate: 0h
>
> I need more control over the indexing performed by the {{ElasticSearchSink}} -- in particular I need access to an {{IndexRequestBuilder}} instance during flume event processing. The interactions between {{ElasticSearchSink}} and {{ElasticSearchEventSerializer}} currently do not make this possible.
> I have authored a patch that meets my needs and maintains backwards-compatibility for existing users of the sink. It is available at [https://github.com/prime8/flume/commit/1056c129b10c95cee50e0c3f77e309668f82bfc6] -- please let me know if it can be pulled into the main ASF source tree!
> Thanks :-)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira