[jira] [Updated] (YARN-4376) Memory Timeline Store return incorrect results on fromId paging

["Jonathan Eagles (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Eagles updated YARN-4376:
----------------------------------
    Attachment: YARN-4376.2.patch

Posting a patch that addresses the issues above. It's not easy to reproduce this error since the iteration order is implementation dependent. I went with adding the TreeSet as secondary view into the entities. As far as extra memory requirements needed, documentation states 40 bytes * capacity. For 1,000,000 entities the extra memory requirements will be 8MB. The insert time is increased for 1,000,000 entries from 3.5 seconds to 11 seconds. Looking into whether this is significant on amortization. Will test with a huge entity set to validate the performance before and after.

> Memory Timeline Store return incorrect results on fromId paging
> ---------------------------------------------------------------
>
>                 Key: YARN-4376
>                 URL: https://issues.apache.org/jira/browse/YARN-4376
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Jonathan Eagles
>            Assignee: Jonathan Eagles
>         Attachments: YARN-4376.2.patch
>
>
> As pointed out correctly by [~jlowe]. 
> https://issues.apache.org/jira/browse/TEZ-2628?focusedCommentId=14715831&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14715831
> The MemoryTimelineStore cannot page correctly when using fromId. This is due switching between data structures that apparently have different natural sorting. In addition, the approach of creating a new data structure every time from scratch is costly. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)