You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-dev@hadoop.apache.org by "Michael Stack (Jira)" <ji...@apache.org> on 2022/05/16 04:34:00 UTC

[jira] [Resolved] (HDFS-16540) Data locality is lost when DataNode pod restarts in kubernetes

     [ https://issues.apache.org/jira/browse/HDFS-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Michael Stack resolved HDFS-16540.
----------------------------------
    Hadoop Flags: Reviewed
      Resolution: Fixed

Merged to branch-3.3. and to trunk.

> Data locality is lost when DataNode pod restarts in kubernetes 
> ---------------------------------------------------------------
>
>                 Key: HDFS-16540
>                 URL: https://issues.apache.org/jira/browse/HDFS-16540
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 3.3.2
>            Reporter: Huaxiang Sun
>            Assignee: Huaxiang Sun
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 3.4.0, 3.3.4
>
>          Time Spent: 7h
>  Remaining Estimate: 0h
>
> We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod restarts, we found that data locality is lost after we do a major compaction of hbase regions. After some debugging, we found that upon pod restarts, its ip changes. In DatanodeManager, maps like networktopology are updated with the new info. host2DatanodeMap is not updated accordingly. When hdfs client with the new ip tries to find a local DataNode, it fails. 
>  



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org