You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by "Tim Bacon (JIRA)" <ji...@apache.org> on 2013/04/29 11:04:16 UTC
[jira] [Comment Edited] (FLUME-2015) ElasticSearchSink: need access
to IndexRequestBuilder instance during flume event processing
[ https://issues.apache.org/jira/browse/FLUME-2015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13644371#comment-13644371 ]
Tim Bacon edited comment on FLUME-2015 at 4/29/13 9:03 AM:
-----------------------------------------------------------
Thanks for the positive feedback! I've attached a link to the [Code Review Board|https://reviews.apache.org/r/10835/] at the top of the page.
In reply to your comments:
* My (non-logging-related) use case has no need for the timestamp to be provided as a header on every event, which is why I didn't add the warnings. If this makes the patch unsuitable for your LogStash use case I can always add it back in such a way as to avoid the warnings in my own code path...
* There were 2 reasons why I wrapped the incoming event rather than adding the timestamp as another param
** I was striving to keep backwards-compatibility (speaking as someone who has been burned by the lack of it in the past!) and the contract with existing clients is based only on the event param
** The timestamp used to build the index name should match the timestamp value that goes into the index, and it's only at the level above {{ElasticSearchLogStashEventSerializer}} that consistency can be ensured
* Having said that I understand your concern about GC -- and, if it helps, I haven't noticed any performance hit while running the new code in my own flume test environment (with several million events per day)
Rgds,
Tim
was (Author: prime8):
Thanks for the positive feedback! I've attached a link to the [Code Review Board|https://reviews.apache.org/r/10835/] at the top of the page.
In reply to your comments:
* My (non-logging-related) use case has no need for the timestamp to be provided as a header on every event, which is why I didn't add it. If that makes it unusable for your LogStash use case I can always add it back in such a way as to avoid the warnings in my own code path...
* There were 2 reasons why I wrapped the incoming event rather than adding the timestamp as another param
** I was striving to keep backwards-compatibility (speaking as someone who has been burned by the lack of it in the past!) and the contract with existing clients is based only on the event param
** The timestamp used to build the index name should match the timestamp value that goes into the index, and it's only at the level above {{ElasticSearchLogStashEventSerializer}} that consistency can be ensured
* Having said that I understand your concern about GC -- and, if it helps, I haven't noticed any performance hit while running the new code in my own flume test environment (with several million events per day)
Rgds,
Tim
> ElasticSearchSink: need access to IndexRequestBuilder instance during flume event processing
> --------------------------------------------------------------------------------------------
>
> Key: FLUME-2015
> URL: https://issues.apache.org/jira/browse/FLUME-2015
> Project: Flume
> Issue Type: Improvement
> Components: Sinks+Sources
> Affects Versions: v1.3.0, v1.3.1
> Reporter: Tim Bacon
> Attachments: FLUME-2015-1.4.0-0.patch, FLUME-2015-doc-1.4.0-0.patch
>
> Original Estimate: 0h
> Remaining Estimate: 0h
>
> I need more control over the indexing performed by the {{ElasticSearchSink}} -- in particular I need access to an {{IndexRequestBuilder}} instance during flume event processing. The interactions between {{ElasticSearchSink}} and {{ElasticSearchEventSerializer}} currently do not make this possible.
> I have authored a patch that meets my needs and maintains backwards-compatibility for existing users of the sink. It is available at [https://github.com/prime8/flume/commit/1056c129b10c95cee50e0c3f77e309668f82bfc6] -- please let me know if it can be pulled into the main ASF source tree!
> Thanks :-)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira