You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Erik Krogen (JIRA)" <ji...@apache.org> on 2018/09/10 15:28:01 UTC
[jira] [Comment Edited] (HDFS-13880) Add mechanism to allow certain
RPC calls to bypass sync
[ https://issues.apache.org/jira/browse/HDFS-13880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16609387#comment-16609387 ]
Erik Krogen edited comment on HDFS-13880 at 9/10/18 3:27 PM:
-------------------------------------------------------------
[~vagarychen] -
{quote}
The original thought was that, getContentSummary already yields locks so could already be returning results based on outdated info. Making it coordinated seems to have little gain.
{quote}
Yielding the locks results in a different type of inconsistency than lacking the use of the {{isCoordinated}} flag. Consider the following scenario:
* My home directory {{/user/xkrogen}} has a quota of 100GB and is currently empty.
* I move some 100GB file, {{/tmp/largeFile.avro}}, into my home directory.
* I check my quota via {{getContentSummary}}, it says I have 100GB remaining quota (because it is not coordinated and reads stale data)
* I try to move another file into my home directory and it fails because I've exceeded quota
* I check my quota again via {{getContentSummary}} and it still says I have 100GB remaining
While the lock yield means you could have misleading results for _concurrent_ modifications, the lack of {{isCoordinated}} means you can miss results for actions that occurred strictly before the content summary, losing RYOW semantics. I think this is much more confusing for a user.
Also, just for the sake of uniformity, it seems odd to leave as uncoordinated this one operation which does actually read from the global state - other uncoordinated ops deal with node-local state.
was (Author: xkrogen):
[~vagarychen] -
{quote}
The original thought was that, getContentSummary already yields locks so could already be returning results based on outdated info. Making it coordinated seems to have little gain.
{quote}
Yielding the locks results in a different type of inconsistency than lacking the use of the {{isCoordinated}} flag. Consider the following scenario:
* My home directory {{/user/xkrogen}} has a quota of 100GB and is currently empty.
* I move some 100GB file, {{/tmp/largeFile.avro}}, into my home directory.
* I check my quota via {{getContentSummary}, it says I have 100GB remaining quota (because it is not coordinated and reads stale data)
* I try to move another file into my home directory and it fails because I've exceeded quota
* I check my quota again via {{getContentSummary}} and it still says I have 100GB remaining
While the lock yield means you could have misleading results for _concurrent_ modifications, the lack of {{isCoordinated}} means you can miss results for actions that occurred strictly before the content summary, losing RYOW semantics. I think this is much more confusing for a user.
Also, just for the sake of uniformity, it seems odd to leave as uncoordinated this one operation which does actually read from the global state - other uncoordinated ops deal with node-local state.
> Add mechanism to allow certain RPC calls to bypass sync
> -------------------------------------------------------
>
> Key: HDFS-13880
> URL: https://issues.apache.org/jira/browse/HDFS-13880
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Chen Liang
> Assignee: Chen Liang
> Priority: Major
> Attachments: HDFS-13880-HDFS-12943.001.patch, HDFS-13880-HDFS-12943.002.patch, HDFS-13880-HDFS-12943.003.patch, HDFS-13880-HDFS-12943.004.patch
>
>
> Currently, every single call to NameNode will be synced, in the sense that NameNode will not process it until state id catches up. But in certain cases, we would like to bypass this check and allow the call to return immediately, even when the server id is not up to date. One case could be the to-be-added new API in HDFS-13749 that request for current state id. Others may include calls that do not promise real time responses such as {{getContentSummary}}. This Jira is to add the mechanism to allow certain calls to bypass sync.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org