[jira] [Comment Edited] (YARN-6357) Implement TimelineCollector#putEntitiesAsync

["Haibo Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-6357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930832#comment-15930832 ] 

Haibo Chen edited comment on YARN-6357 at 3/17/17 10:26 PM:
------------------------------------------------------------

Upload an initial patch for review. I did not call putEntitiesAsync() in putEntities because this will require putEntitiesAsync() to return a response, which I think is weird for an async method. Plus, the debug logging for the sync and async putEntities call can also be awkward. 


was (Author: haibochen):
Upload an initial patch for review

> Implement TimelineCollector#putEntitiesAsync
> --------------------------------------------
>
>                 Key: YARN-6357
>                 URL: https://issues.apache.org/jira/browse/YARN-6357
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: ATSv2, timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Joep Rottinghuis
>            Assignee: Haibo Chen
>              Labels: yarn-5355-merge-blocker
>         Attachments: YARN-6357.01.patch
>
>
> As discovered and discussed in YARN-5269 the TimelineCollector#putEntitiesAsync method is currently not implemented and TimelineCollector#putEntities is asynchronous.
> TimelineV2ClientImpl#putEntities vs TimelineV2ClientImpl#putEntitiesAsync correctly call TimelineEntityDispatcher#dispatchEntities(boolean sync,... with the correct argument. This argument does seem to make it into the params, and on the server side TimelineCollectorWebService#putEntities correctly pulls the async parameter from the rest call. See line 156:
> {code}
>     boolean isAsync = async != null && async.trim().equalsIgnoreCase("true");
> {code}
> However, this is where the problem starts. It simply calls TimelineCollector#putEntities and ignores the value of isAsync. It should instead have called TimelineCollector#putEntitiesAsync, which is currently not implemented.
> putEntities should call putEntitiesAsync and then after that call writer.flush()
> The fact that we flush on close and we flush periodically should be more of a concern of avoiding data loss; close in case sync is never called and the periodic flush to guard against having data from slow writers get buffered for a long time and expose us to risk of loss in case the collector crashes with data in its buffers. Size-based flush is a different concern to avoid blowing up memory footprint.
> The spooling behavior is also somewhat separate.
> We have two separate methods on our API putEntities and putEntitiesAsync and they should have different behavior beyond waiting for the request to be sent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org