[jira] [Commented] (FLUME-1782) Elastic Search sink does not use UTC to determine the correct index to write to.

["Mike Percy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLUME-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634455#comment-13634455 ] 

Mike Percy commented on FLUME-1782:
-----------------------------------

Hi Edward,
Thanks for the patch! I have one concern about the approach and a few minor requests/questions.

Concern: Since the sink is modifying the event itself, this can cause issues with memory channel rollback. i.e. if there is a channel rollback because ES is down, then any event processed when it comes back up will actually have the timestamp set by a previous run of the process() method. Depending on why you might want to have a local timestamp, that could be unexpected and/or desired.

Instead of ensuring that the timestamp is set in the event itself at the sink, we could provide some other method of falling back to a local time in the sink itself if the event doesn't have a timestamp set.

tl;dr: Don't modify the Event in the sink :)

Other minor feedback:
1. It would be great to post this onto Review Board @ https://reviews.apache.org/groups/Flume/ so that individual lines in the patch are easy to reference during review
2. In shouldSetAndWarnWhenNoHeaders() I don't think any Event should ever have null headers. Have you seen cases where that can happen?
3. Nit: can we use a constant for the values 1355364900011 and 1355364900079 that are used throughout this test? They are kinda magic numbers and it could use a little more explanation and/or ensuring they are used consistently by making them private static final variables. :)
4. Nit: consider using Charsets.UTF_8 from Guava rather than Charset.forName("UTF-8") ... this is optional and not a big deal.

Best,
Mike

                
> Elastic Search sink does not use UTC to determine the correct index to write to.
> --------------------------------------------------------------------------------
>
>                 Key: FLUME-1782
>                 URL: https://issues.apache.org/jira/browse/FLUME-1782
>             Project: Flume
>          Issue Type: Bug
>          Components: Sinks+Sources
>            Reporter: Edward Sargisson
>             Fix For: v1.4.0
>
>         Attachments: FLUME-1782-1.4.0-1.patch, FLUME-1782-1.4.0-2.patch, FLUME-1782-1.4.0.patch, flume-1782-2.patch, flume-1782-3.patch, flume-1782.patch
>
>
> The GUI for logs in ElasticSearch, Kibana, uses the utc date to determine which index to read for a search. The Flume ElasticSearch sink is using the local timezone to determine which index to write to. This means that events are being placed in the incorrect index and Kibana doesn't always find them.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira