You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Jinglun (Jira)" <ji...@apache.org> on 2020/01/02 13:37:00 UTC
[jira] [Comment Edited] (HDFS-15087) RBF: Balance/Rename across
federation namespaces
[ https://issues.apache.org/jira/browse/HDFS-15087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006657#comment-17006657 ]
Jinglun edited comment on HDFS-15087 at 1/2/20 1:36 PM:
--------------------------------------------------------
Hi [~elgoiri], thanks your nice comments !
{quote}Would it be possible to leverage HDFS snapshots instead of blocking writes and having the new tree related calls? Intuitively, I would expect for snapshots to cover 90% of the features described in the doc. I would try to improve snapshots to cover 100%.
{quote}
-I'm not familiar with the snapshot. In my rough thought as long as the snapshot meta could be transferred and rebuilt the HFR could support it.- -I'll try to write a demo to transfer and rebuild the snapshot across NameNodes-.
I have a quick look of snapshot and I'm not sure how to use it. Do you mean to use the diff of snapshots so we can do the balance in an incremental way ?
{quote}The approach described in the doc requires hard linking. I think this is a good idea for the start but I would push to make it pluggable/abstract so in the future we can have other implementations.
{quote}
Good idea. The design of HFR has considered it. The HFR is a combination of many tasks. Each task is plugable. For example if we want to use copy instead of hardlink, we can switch the HardLink task to a CopyReplica task.
{quote}Is hard linking available in Windows?
{quote}
After HADOOP-11483 we use jdk Files.createLink() to do the hardlinks. I test Files.createLink() on windows and it works.
See java doc [https://docs.oracle.com/javase/tutorial/essential/io/links.html]
was (Author: lijinglun):
Hi [~elgoiri], thanks your nice comments !
{quote}Would it be possible to leverage HDFS snapshots instead of blocking writes and having the new tree related calls? Intuitively, I would expect for snapshots to cover 90% of the features described in the doc. I would try to improve snapshots to cover 100%.
{quote}
I'm not familiar with the snapshot. In my rough thought as long as the snapshot meta could be transferred and rebuilt the HFR could support it. I'll try to write a demo to transfer and rebuild the snapshot across NameNodes.
{quote}The approach described in the doc requires hard linking. I think this is a good idea for the start but I would push to make it pluggable/abstract so in the future we can have other implementations.
{quote}
Good idea. The design of HFR has considered it. The HFR is a combination of many tasks. Each task is plugable. For example if we want to use copy instead of hardlink, we can switch the HardLink task to a CopyReplica task.
{quote}Is hard linking available in Windows?
{quote}
After HADOOP-11483 we use jdk Files.createLink() to do the hardlinks. I test Files.createLink() on windows and it works.
See java doc [https://docs.oracle.com/javase/tutorial/essential/io/links.html]
> RBF: Balance/Rename across federation namespaces
> ------------------------------------------------
>
> Key: HDFS-15087
> URL: https://issues.apache.org/jira/browse/HDFS-15087
> Project: Hadoop HDFS
> Issue Type: Improvement
> Reporter: Jinglun
> Priority: Major
> Attachments: HFR_Rename Across Federation Namespaces.pdf
>
>
> The Xiaomi storage team has developed a new feature called HFR(HDFS Federation Rename) that enables us to do balance/rename across federation namespaces. The idea is to first move the meta to the dst NameNode and then link all the replicas. It has been working in our largest production cluster for 2 months. We use it to balance the namespaces. It turns out HFR is fast and flexible. The detail could be found in the design doc.
> Looking forward to a lively discussion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org