[jira] [Created] (METRON-691) Elastic Writer index partitions on system time, not event time

["Simon Elliston Ball (JIRA)" <ji...@apache.org>]

Simon Elliston Ball created METRON-691:
------------------------------------------

             Summary: Elastic Writer index partitions on system time, not event time
                 Key: METRON-691
                 URL: https://issues.apache.org/jira/browse/METRON-691
             Project: Metron
          Issue Type: Bug
    Affects Versions: 0.3.0
            Reporter: Simon Elliston Ball


Currently the elastic writer determines the index destination for messages based on system time, rather than message time. As a consequence, around time boundaries, where there is more than a small lag in the topologies, an event can end up in the wrong index. 

This means the event is ignored in Kibana dashboard, which quite sensible limit the indices consulted, but filter on the exact timestamp. 

To reproduce this, index an older event, and note that a current time index is created. Searching within the actual event time period will not find the event, because it consults the wrong index. Searching within the index period will also not return the event due to the filtering on the actual event timestamp field.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)