You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Siyao Meng (Jira)" <ji...@apache.org> on 2019/10/22 18:57:00 UTC

[jira] [Comment Edited] (HDFS-14283) DFSInputStream to prefer cached replica

    [ https://issues.apache.org/jira/browse/HDFS-14283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16957276#comment-16957276 ] 

Siyao Meng edited comment on HDFS-14283 at 10/22/19 6:56 PM:
-------------------------------------------------------------

[~ayushtkn] I agree that we should add a new client-side config to prefer cached blocks, and default it to off (false). I see that rev 004 already added it. Thanks [~leosun08].

I'm okay with sorting cached blocks on server. It doesn't seem we can sort it on client side easily since the client might not have the distance information of each node (CMIIW).


was (Author: smeng):
[~ayushtkn] I agree that we should add a new client-side config to prefer cached blocks, and default it to off (false). I see that rev 004 already added it. Thanks [~leosun08].

But one more thing, would you also add it to hdfs-default.xml ?

I'm okay with sorting cached blocks on server. It doesn't seem we can sort it on client side easily since the client might not have the distance information of each node (CMIIW).

> DFSInputStream to prefer cached replica
> ---------------------------------------
>
>                 Key: HDFS-14283
>                 URL: https://issues.apache.org/jira/browse/HDFS-14283
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.6.0
>         Environment: HDFS Caching
>            Reporter: Wei-Chiu Chuang
>            Assignee: Lisheng Sun
>            Priority: Major
>         Attachments: HDFS-14283.001.patch, HDFS-14283.002.patch, HDFS-14283.003.patch, HDFS-14283.004.patch
>
>
> HDFS Caching offers performance benefits. However, currently NameNode does not treat cached replica with higher priority, so HDFS caching is only useful when cache replication = 3, that is to say, all replicas are cached in memory, so that a client doesn't randomly pick an uncached replica.
> HDFS-6846 proposed to let NameNode give higher priority to cached replica. Changing a logic in NameNode is always tricky so that didn't get much traction. Here I propose a different approach: let client (DFSInputStream) prefer cached replica.
> A {{LocatedBlock}} object already contains cached replica location so a client has the needed information. I think we can change {{DFSInputStream#getBestNodeDNAddrPair()}} for this purpose.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org