You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Tom McCormick (Jira)" <ji...@apache.org> on 2022/09/27 00:11:00 UTC
[jira] [Updated] (HDFS-16782) RBF Migrate Sub cluster mount point
[ https://issues.apache.org/jira/browse/HDFS-16782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom McCormick updated HDFS-16782:
---------------------------------
Description:
At LinkedIn, we run very large HDFS volumes and many of them. We have a growing need to balance data across our hdfs volumes without downtime. (More info on our [setup|https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr]
The goal is to migrate data from one hdfs volume to another, with zero read downtime and zero (or near zero) write downtime.
This is a new mount point that will opportunistically write new files to the new volume with the goal being to reduce the amount of new data created on the original volume.
Similar to existing RBF subcluster mount points, it knows how to reconcile files and directories between hdfs volumes.
-----
*Related tickets*
https://issues.apache.org/jira/browse/HDFS-15294
* Currently this solution has consistency issues
* assumes some downtime
was:
The goal is to migrate data from one hdfs volume to another, with zero read downtime and zero (or near zero) write downtime.
This is a new mount point that will opportunistically write new files to the new volume with the goal being to reduce the amount of new data created on the original volume.
Similar to existing RBF subcluster mount points, it knows how to reconcile files and directories between hdfs volumes.
> RBF Migrate Sub cluster mount point
> -----------------------------------
>
> Key: HDFS-16782
> URL: https://issues.apache.org/jira/browse/HDFS-16782
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
> Reporter: Tom McCormick
> Priority: Major
>
> At LinkedIn, we run very large HDFS volumes and many of them. We have a growing need to balance data across our hdfs volumes without downtime. (More info on our [setup|https://engineering.linkedin.com/blog/2021/the-exabyte-club--linkedin-s-journey-of-scaling-the-hadoop-distr]
> The goal is to migrate data from one hdfs volume to another, with zero read downtime and zero (or near zero) write downtime.
> This is a new mount point that will opportunistically write new files to the new volume with the goal being to reduce the amount of new data created on the original volume.
> Similar to existing RBF subcluster mount points, it knows how to reconcile files and directories between hdfs volumes.
>
>
> -----
> *Related tickets*
> https://issues.apache.org/jira/browse/HDFS-15294
> * Currently this solution has consistency issues
> * assumes some downtime
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org