You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Lisheng Sun (Jira)" <ji...@apache.org> on 2020/03/08 06:49:00 UTC
[jira] [Comment Edited] (HDFS-14809) Reduce BlockReaderLocal RPC
calls
[ https://issues.apache.org/jira/browse/HDFS-14809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17054313#comment-17054313 ]
Lisheng Sun edited comment on HDFS-14809 at 3/8/20, 6:48 AM:
-------------------------------------------------------------
Thank [~ken_1994@163.com] for reporting this jira.
I don't quite understand some places in this jira description.
{quote}However, the slot and shared memory sturcture is only used by DataNode when uncaching replicas,
{quote}
you mean the client don't use the slot and shared memory sturcture when uncaching replicas.
How does the client know if the replica is invalid at this time?
The slot's effect is effectiveness of synchronizing replicas and number of replica ref on client and DN.
According to your implementation only requests the fds , request an rpc fds before each short circuit read.
Please correct me if i was wrong.
Regarding the performance of short-circuit read, you can pay attention to this jira HDFS-13639 and HDFS-13564
was (Author: leosun08):
Thank [~ken_1994@163.com] for reporting this jira.
I don't quite understand some places in this jira description.
{quote}
However, the slot and shared memory sturcture is only used by DataNode when uncaching replicas,
{quote}
you mean the client don't use the slot and shared memory sturcture when uncaching replicas.
How does the client know if the replica is invalid at this time?
The slot's effect is effectiveness of synchronizing replicas and number of replica ref on client and DN.
According to your implementation only requests the fds , request an rpc fds before each short circuit read.
Please correct me if i was wrong.
Regarding the performance of short-circuit read, you can pay attention to this jira HDFS-13639 and HDFS-13564
> Reduce BlockReaderLocal RPC calls
> ---------------------------------
>
> Key: HDFS-14809
> URL: https://issues.apache.org/jira/browse/HDFS-14809
> Project: Hadoop HDFS
> Issue Type: New Feature
> Affects Versions: 2.6.0
> Reporter: KenCao
> Assignee: KenCao
> Priority: Major
> Attachments: HADOOP-14809
>
>
> as we known, the hdfs client java lib uses BlockReaderLocal for short circuit read by default, which allocate shared memory first, and make a slot within it. After all these steps, it will request the fds from the DataNode. However, the slot and shared memory sturcture is only used by DataNode when uncaching replicas, the client process can work well just with the fds asked later and it is nearly impossible to cache replicas in product environment. The api to release fds is called by client only with the slot given, the fds is close in the client process finally.
> so i think we can make a new BlockReader implementation which just requests the fds, and it will reduce the rpc calls from 3(allocate shm, request fds, release fds) to 1(request fds).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org