[jira] [Commented] (HDFS-13150) Create fast path for SbNN tailing edits from JNs

["Konstantin Shvachko (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-13150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16397731#comment-16397731 ] 

Konstantin Shvachko commented on HDFS-13150:
--------------------------------------------

As a reminder, the two approaches for SBN / ObserverNode reading from Journal nodes are:
 # SBN *reads from a quorum* of JNs
 # SBN *reads from single* JN, while JNs guarantee serving only committed transactions

I am advocating that approach 2 is faster. Suppose we have 3 journal nodes.
 * When SBN reads from the quorum (approach 1) it updates its state as fast as the second slowest JN.
 * With approach 2 we can choose the fastest JN most of the time. By periodically polling JNs and switching to the one that has higher txId.

There is an issue of confirming committed transactions from ANN to JNs. But every next batch of edits sent by ANN to a JN essentially confirms that the previous batch is committed. This does not require extra dummy syncs. Under regular load ANN will be sending batches of edits continuously, so JNs will be up-to-date up to the last processed batch. ANN will occasionally need to send the extra “dummy” sync, but it is required only if ANN doesn't have load at all or no writes.
 _Having said that, I am fine with quorum reads as the initial implementation if it is simpler, as I was told._

> Create fast path for SbNN tailing edits from JNs
> ------------------------------------------------
>
>                 Key: HDFS-13150
>                 URL: https://issues.apache.org/jira/browse/HDFS-13150
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs, journal-node, namenode
>            Reporter: Erik Krogen
>            Assignee: Erik Krogen
>            Priority: Major
>         Attachments: edit-tailing-fast-path-design-v0.pdf, edit-tailing-fast-path-design-v1.pdf
>
>
> In the interest of making coordinated/consistent reads easier to complete with low latency, it is advantageous to reduce the time between when a transaction is applied on the ANN and when it is applied on the SbNN. We propose adding a new "fast path" which can be used to tail edits when low latency is desired. We leave the existing tailing logic in place, and fall back to this path on startup, recovery, and when the fast path encounters unrecoverable errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org