[jira] [Commented] (HDFS-6268) Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found

["Andrew Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13987104#comment-13987104 ] 

Andrew Wang commented on HDFS-6268:
-----------------------------------

Thanks for the reviews everyone. Answers to your comments inline:

bq. Do we need to put another random one at the second position? 

Clients read from the nodes in order, so our readers will all be using the random first node. The second node only comes into play during a failure (or a hedged read I guess). Because we sometimes set the seed and this algo is deterministic, we could get stuck selecting the same random node for a block, but that's actually desirable. What we want to avoid is the same ingest node getting chosen for every block, which the randomness should prevent.

bq. It seems to me that the second sort (Array.sort) would change the result of the first (pesudoSortByDistance). Why the sorting is not consolidated into one to consider both distance and decommission/stale factors?

I suppose we could, I'll consider it if I rev the patch.

bq. This would remove one level of indentation, and make it easier to read.

The bit at the end about choosing a random node is not within this if block, so I don't think we can make this change.

ATM, I read this code and had the same thought. It would be cleaner and less corner-casey if we first binned by network distance, then randomized each bin. I didn't make this change since this is a hot code path and it'd be a bit slower, but since we're typically dealing with 3 replicas, I can't imagine it making a big difference. We could also potentially fold in the decom/stale state too, and get better locality for these edge cases. If you agree with this assessment, I'll redo this patch as per above.

> Better sorting in NetworkTopology#pseudoSortByDistance when no local node is found
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-6268
>                 URL: https://issues.apache.org/jira/browse/HDFS-6268
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.4.0
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>            Priority: Minor
>         Attachments: hdfs-6268-1.patch, hdfs-6268-2.patch
>
>
> In NetworkTopology#pseudoSortByDistance, if no local node is found, it will always place the first rack local node in the list in front.
> This became an issue when a dataset was loaded from a single datanode. This datanode ended up being the first replica for all the blocks in the dataset. When running an Impala query, the non-local reads when reading past a block boundary were all hitting this node, meaning massive load skew.



--
This message was sent by Atlassian JIRA
(v6.2#6252)