You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2008/11/06 01:16:44 UTC
[jira] Created: (HADOOP-4598) '-setrep' command skips
under-replicated blocks
'-setrep' command skips under-replicated blocks
-----------------------------------------------
Key: HADOOP-4598
URL: https://issues.apache.org/jira/browse/HADOOP-4598
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Reporter: Koji Noguchi
Priority: Minor
When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
hadoop dfs -setrep <higher_replication> <filename>
would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4598) '-setrep' command skips
under-replicated blocks
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649492#action_12649492 ]
Hairong Kuang commented on HADOOP-4598:
---------------------------------------
Ant test-core passed on my local machine.
BUILD SUCCESSFUL
Total time: 124 minutes 24 seconds
Ant test-patch result:
[exec] +1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] +1 tests included. The patch appears to include 4 new or modified tests.
[exec] +1 javadoc. The javadoc tool did not generate any warningmessages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
> '-setrep' command skips under-replicated blocks
> -----------------------------------------------
>
> Key: HADOOP-4598
> URL: https://issues.apache.org/jira/browse/HADOOP-4598
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Koji Noguchi
> Assignee: Hairong Kuang
> Priority: Minor
> Attachments: setRep.patch
>
>
> When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
> hadoop dfs -setrep <higher_replication> <filename>
> would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4598) '-setrep' command skips
under-replicated blocks
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4598:
----------------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> '-setrep' command skips under-replicated blocks
> -----------------------------------------------
>
> Key: HADOOP-4598
> URL: https://issues.apache.org/jira/browse/HADOOP-4598
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Koji Noguchi
> Assignee: Hairong Kuang
> Priority: Minor
> Attachments: setRep.patch
>
>
> When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
> hadoop dfs -setrep <higher_replication> <filename>
> would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4598) '-setrep' command skips
under-replicated blocks
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12647709#action_12647709 ]
Hairong Kuang commented on HADOOP-4598:
---------------------------------------
UnderReplicatedBlocks.update has an optimization that a under-replication block does not get added to the queue if a block was under-replicated and its previous priority is the same as the current priority. This optimization is based on the assumption that a previously under-replicated block should have already been in the queue.
For the case that Koji filed, due to for some bugs like HADOOP-4597, under-replicated blocks did not get to place in the under-replicated block queue at NameNode startup time. Later on changing a file's replication factor to be a bigger number does not trigger the move of the file's blocks into the queue if the replication factor change does not change their priority.
The fix for this problem is to remove the optimization.
> '-setrep' command skips under-replicated blocks
> -----------------------------------------------
>
> Key: HADOOP-4598
> URL: https://issues.apache.org/jira/browse/HADOOP-4598
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Koji Noguchi
> Assignee: Hairong Kuang
> Priority: Minor
>
> When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
> hadoop dfs -setrep <higher_replication> <filename>
> would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4598) '-setrep' command skips
under-replicated blocks
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12648836#action_12648836 ]
Konstantin Shvachko commented on HADOOP-4598:
---------------------------------------------
+1 This should work.
> '-setrep' command skips under-replicated blocks
> -----------------------------------------------
>
> Key: HADOOP-4598
> URL: https://issues.apache.org/jira/browse/HADOOP-4598
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Koji Noguchi
> Assignee: Hairong Kuang
> Priority: Minor
> Attachments: setRep.patch
>
>
> When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
> hadoop dfs -setrep <higher_replication> <filename>
> would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4598) '-setrep' command skips
under-replicated blocks
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang reassigned HADOOP-4598:
-------------------------------------
Assignee: Hairong Kuang
> '-setrep' command skips under-replicated blocks
> -----------------------------------------------
>
> Key: HADOOP-4598
> URL: https://issues.apache.org/jira/browse/HADOOP-4598
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Koji Noguchi
> Assignee: Hairong Kuang
> Priority: Minor
>
> When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
> hadoop dfs -setrep <higher_replication> <filename>
> would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4598) '-setrep' command skips
under-replicated blocks
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4598:
----------------------------------
Resolution: Fixed
Fix Version/s: 0.20.0
Status: Resolved (was: Patch Available)
I just committed this!
> '-setrep' command skips under-replicated blocks
> -----------------------------------------------
>
> Key: HADOOP-4598
> URL: https://issues.apache.org/jira/browse/HADOOP-4598
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Koji Noguchi
> Assignee: Hairong Kuang
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: setRep.patch
>
>
> When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
> hadoop dfs -setrep <higher_replication> <filename>
> would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4598) '-setrep' command skips
under-replicated blocks
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649697#action_12649697 ]
Hudson commented on HADOOP-4598:
--------------------------------
Integrated in Hadoop-trunk #667 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/667/])
. Setrep command skips under-replicated blocks. Contributed by Hairong Kuang.
> '-setrep' command skips under-replicated blocks
> -----------------------------------------------
>
> Key: HADOOP-4598
> URL: https://issues.apache.org/jira/browse/HADOOP-4598
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Koji Noguchi
> Assignee: Hairong Kuang
> Priority: Minor
> Fix For: 0.20.0
>
> Attachments: setRep.patch
>
>
> When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
> hadoop dfs -setrep <higher_replication> <filename>
> would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4598) '-setrep' command skips
under-replicated blocks
Posted by "Hairong Kuang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hairong Kuang updated HADOOP-4598:
----------------------------------
Attachment: setRep.patch
A patch for review.
> '-setrep' command skips under-replicated blocks
> -----------------------------------------------
>
> Key: HADOOP-4598
> URL: https://issues.apache.org/jira/browse/HADOOP-4598
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Koji Noguchi
> Assignee: Hairong Kuang
> Priority: Minor
> Attachments: setRep.patch
>
>
> When blocks are under-replicated due to some hadoop bugs, like HADOOP-4597, calling
> hadoop dfs -setrep <higher_replication> <filename>
> would not schedule the replication for the under-replicated blocks (unless it goes to a different priority queue).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.