[jira] [Updated] (YARN-4265) Provide new timeline plugin storage to support fine-grained entity caching

["Li Lu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Li Lu updated YARN-4265:
------------------------
    Attachment: YARN-4265-trunk.002.patch

Thanks [~djp] for the review! In the 002 patch I addressed most of the checkstyle problems, as well as most existing comments. Please feel free to add more. Some comments:

bq. I noticed that we are setting 1 minutes as default scan interval but original patch in HDFS-3942 is 5 minutes. Why shall we do any update here? 
For now I increased the default frequency to scan HDFS and pull timeline data. Having a 5-minute time interval means users are less likely to see any running status for apps that finish within 5 minutes. Right now I'm setting this value to 1 minute to reduce reader react time. 

bq. The same question on "app-cache-size", the default value in HDFS-3942 is 5 but here is 10. Any reason to update the value?
In YARN-3942, caching is performed on application level. In this patch, caching is performed in entity groups. Each application may have a few to tens of entity groups. Normally, there are slightly more active entity groups than active applications in the system. For now, I'm increasing this default value to hold slightly more entity groups in cache. 

bq. Why we don't have any default value specified in property of "yarn.timeline-service.entity-group-fs-store.group-id-plugin-classes"?
Plugins are provided by third-party applications such as Tez. Right now we cannot assume which exact entity group plugin the user is using, therefore we have to conservatively leave this config as empty. 

bq. For EmptyTimelineEntityGroupPlugin.java, why we need this class? I didn't see any help provided even in tests. We should remove it if useless.
Ah, nice catch. Removed it. 

bq. Can we optimize the synchronization logic here? Like in synchronized method refreshCache, we are intialize/start/stop TimelineDataManager (and MemoryTimelineStore) which is quite expensive and unnecessary to block other synchronized operations. Shall we move these operations out of synchronized block?
It's certainly doable. Right now I have yet to optimize this part because it's a little bit tricky to fine tune synchronization performance before we have a relatively stable starting point. Also, we're using fine-grained locking for each cached item in the reader cache, and cache refresh only happens infrequently (~10 secs by default), so maybe we'd like to stabilize the whole synchronization story before fine tune this part? 

> Provide new timeline plugin storage to support fine-grained entity caching
> --------------------------------------------------------------------------
>
>                 Key: YARN-4265
>                 URL: https://issues.apache.org/jira/browse/YARN-4265
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: YARN-4265-trunk.001.patch, YARN-4265-trunk.002.patch, YARN-4265.YARN-4234.001.patch, YARN-4265.YARN-4234.002.patch
>
>
> To support the newly proposed APIs in YARN-4234, we need to create a new plugin timeline store. The store may have similar behavior as the EntityFileTimelineStore proposed in YARN-3942, but cache date in cache id granularity, instead of application id granularity. Let's have this storage as a standalone one, instead of updating EntityFileTimelineStore, to keep the existing store (EntityFileTimelineStore) stable. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)