You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Stu Hood (JIRA)" <ji...@apache.org> on 2007/11/10 05:01:50 UTC
[jira] Created: (HADOOP-2186) DFS Upgrade should process
dfs.data.dirs in parallel
DFS Upgrade should process dfs.data.dirs in parallel
----------------------------------------------------
Key: HADOOP-2186
URL: https://issues.apache.org/jira/browse/HADOOP-2186
Project: Hadoop
Issue Type: Improvement
Reporter: Stu Hood
I just upgraded from 0.14.2 to 0.15.0, and things went very smoothly, if a little slowly.
The main reason the upgrade took so long was the block upgrades on the datanodes. Each of our datanodes has 3 drives listed for the dfs.data.dir parameter. From looking at the logs, it is fairly clear that the upgrade procedure does not attempt to upgrade all listed dfs.data.dir's in parallel.
I think even if all of your dfs.data.dir's are on the same physical device, there would still be an advantage to performing the upgrade process in parallel. The less downtime, the better: especially if it is potentially 20 minutes versus 60 minutes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2186) DFS Upgrade should process
dfs.data.dirs in parallel
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12542617 ]
Konstantin Shvachko commented on HADOOP-2186:
---------------------------------------------
I agree the upgrade should benefit from multi-threaded link creation.
Does anybody have any experimental data on how many threads would be optimal for that
kind of task taking into account both the number of hard drives and the number of cores?
> DFS Upgrade should process dfs.data.dirs in parallel
> ----------------------------------------------------
>
> Key: HADOOP-2186
> URL: https://issues.apache.org/jira/browse/HADOOP-2186
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.15.0
> Reporter: Stu Hood
> Priority: Minor
>
> I just upgraded from 0.14.2 to 0.15.0, and things went very smoothly, if a little slowly.
> The main reason the upgrade took so long was the block upgrades on the datanodes. Each of our datanodes has 3 drives listed for the dfs.data.dir parameter. From looking at the logs, it is fairly clear that the upgrade procedure does not attempt to upgrade all listed dfs.data.dir's in parallel.
> I think even if all of your dfs.data.dir's are on the same physical device, there would still be an advantage to performing the upgrade process in parallel. The less downtime, the better: especially if it is potentially 20 minutes versus 60 minutes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-2186) DFS Upgrade should process
dfs.data.dirs in parallel
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Stu Hood updated HADOOP-2186:
-----------------------------
Component/s: dfs
Priority: Minor (was: Major)
Affects Version/s: 0.15.0
> DFS Upgrade should process dfs.data.dirs in parallel
> ----------------------------------------------------
>
> Key: HADOOP-2186
> URL: https://issues.apache.org/jira/browse/HADOOP-2186
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.15.0
> Reporter: Stu Hood
> Priority: Minor
>
> I just upgraded from 0.14.2 to 0.15.0, and things went very smoothly, if a little slowly.
> The main reason the upgrade took so long was the block upgrades on the datanodes. Each of our datanodes has 3 drives listed for the dfs.data.dir parameter. From looking at the logs, it is fairly clear that the upgrade procedure does not attempt to upgrade all listed dfs.data.dir's in parallel.
> I think even if all of your dfs.data.dir's are on the same physical device, there would still be an advantage to performing the upgrade process in parallel. The less downtime, the better: especially if it is potentially 20 minutes versus 60 minutes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-2186) DFS Upgrade should process
dfs.data.dirs in parallel
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12542621 ]
Raghu Angadi commented on HADOOP-2186:
--------------------------------------
I don't have data but some time back for essentially random accesses, 5 x no of disks was pretty re good. In this case, these are hardlinks but treating it as random access would be conservative.
I think something like 10-15 threads will cover most common case of disks and cores.
> DFS Upgrade should process dfs.data.dirs in parallel
> ----------------------------------------------------
>
> Key: HADOOP-2186
> URL: https://issues.apache.org/jira/browse/HADOOP-2186
> Project: Hadoop
> Issue Type: Improvement
> Components: dfs
> Affects Versions: 0.15.0
> Reporter: Stu Hood
> Priority: Minor
>
> I just upgraded from 0.14.2 to 0.15.0, and things went very smoothly, if a little slowly.
> The main reason the upgrade took so long was the block upgrades on the datanodes. Each of our datanodes has 3 drives listed for the dfs.data.dir parameter. From looking at the logs, it is fairly clear that the upgrade procedure does not attempt to upgrade all listed dfs.data.dir's in parallel.
> I think even if all of your dfs.data.dir's are on the same physical device, there would still be an advantage to performing the upgrade process in parallel. The less downtime, the better: especially if it is potentially 20 minutes versus 60 minutes.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.