[jira] [Commented] (YARN-4061) [Fault tolerance] Fault tolerant writer for timeline v2

["Sangjin Lee (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943999#comment-14943999 ] 

Sangjin Lee commented on YARN-4061:
-----------------------------------

Sorry it took me a while to get to this. Thanks for the proposal [~gtCarrera9]!

One potential area of concern is regarding flush and the associated contract with the client. If the client wanted to write critical data synchronously and specifically wanted to block until it receives a server (storage) response, how would that work in this scheme? The proposal has the following for the flush situation:
{quote}
When one log segment reaches a predefined size, or a time­ trigger, or an explicit flush call happens, it is published to a log queue.
{quote}

Since the actual storage writer (HBase) always acts on this queue asynchronously, it seems that the client cannot have a synchronous write semantics. Is that a correct reading? If so, how would we implement such a synchronous write?

> [Fault tolerance] Fault tolerant writer for timeline v2
> -------------------------------------------------------
>
>                 Key: YARN-4061
>                 URL: https://issues.apache.org/jira/browse/YARN-4061
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage down time and timeline collector failures. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)