You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Suresh Srinivas (JIRA)" <ji...@apache.org> on 2008/10/10 19:21:44 UTC
[jira] Created: (HADOOP-4388) Bug in Datanode transferBlocks code
Bug in Datanode transferBlocks code
-----------------------------------
Key: HADOOP-4388
URL: https://issues.apache.org/jira/browse/HADOOP-4388
Project: Hadoop Core
Issue Type: Bug
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
{noformat}
private void transferBlocks( Block blocks[],
DatanodeInfo xferTargets[][]
) throws IOException {
for (int i = 0; i < blocks.length; i++) {
if (!data.isValidBlock(blocks[i])) {
String errStr = "Can't send invalid block " + blocks[i];
LOG.info(errStr);
namenode.errorReport(dnRegistration,
DatanodeProtocol.INVALID_BLOCK,
errStr);
//
// ******** This should be continue instead of break?
//
break;
}
int numTargets = xferTargets[i].length;
if (numTargets > 0) {
if (LOG.isInfoEnabled()) {
StringBuilder xfersBuilder = new StringBuilder();
for (int j = 0; j < numTargets; j++) {
DatanodeInfo nodeInfo = xferTargets[i][j];
xfersBuilder.append(nodeInfo.getName());
if (j < (numTargets - 1)) {
xfersBuilder.append(", ");
}
}
String xfersTo = xfersBuilder.toString();
LOG.info(dnRegistration + " Starting thread to transfer block " +
blocks[i] + " to " + xfersTo);
}
new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
}
}
}
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4388:
------------------------------------
Fix Version/s: 0.20.0
Status: Patch Available (was: Open)
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645002#action_12645002 ]
Hadoop QA commented on HADOOP-4388:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12392619/HADOOP-4388.patch
against trunk revision 709609.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3526/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3526/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3526/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3526/console
This message is automatically generated.
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4388:
------------------------------------
Status: Patch Available (was: Open)
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4388:
------------------------------------
Attachment: HADOOP-4388.patch
Nice catch Nicholas. Here is the updated patch.
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12641649#action_12641649 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4388:
------------------------------------------------
For the same reason, should we do a try-catch in the second part of the loop? Otherwise, an IOException thrown within the loop leads to exiting the method.
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch, HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4388:
-------------------------------------------
Hadoop Flags: [Reviewed]
+1 patch looks good
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4388:
------------------------------------
Attachment: HADOOP-4388.patch
Updating the patch...
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch, HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4388:
------------------------------------
Attachment: HADOOP-4388.patch
Attached patch removes breaking out of "for" loop on encountering bad block. With this change, the bad block is ignored and the remaining blocks are transferred from one Datanode to another.
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12645029#action_12645029 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4388:
------------------------------------------------
No tests are added since the changes are obvious and there is no easy way to a new test for this bug.
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tsz Wo (Nicholas), SZE updated HADOOP-4388:
-------------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Suresh!
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HADOOP-4388:
-------------------------------------
Affects Version/s: 0.17.0
Good catch. The code has been like this for a long time, so it might be ok to schedule this for 0.20. Do you agree? Can you pl attach a patch?
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-4388:
------------------------------------
Component/s: dfs
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4388:
------------------------------------
Attachment: HADOOP-4388.patch
Another update to the patch to catch the IOExcpetions in the for loop that could interrupt the block transfer
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4388) Bug in Datanode transferBlocks code
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4388:
------------------------------------
Status: Open (was: Patch Available)
> Bug in Datanode transferBlocks code
> -----------------------------------
>
> Key: HADOOP-4388
> URL: https://issues.apache.org/jira/browse/HADOOP-4388
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.17.0
> Reporter: Suresh Srinivas
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch, HADOOP-4388.patch
>
>
> In the following code, when the blocks to be transferred has an invalid block, all the blocks that follow it are also not transferred. That might not be the intended behavior. Instead of breaking out of the loop, perhaps the right thing to do is to ignore the invalid block and continue with transferring the rest.
> {noformat}
> private void transferBlocks( Block blocks[],
> DatanodeInfo xferTargets[][]
> ) throws IOException {
> for (int i = 0; i < blocks.length; i++) {
> if (!data.isValidBlock(blocks[i])) {
> String errStr = "Can't send invalid block " + blocks[i];
> LOG.info(errStr);
> namenode.errorReport(dnRegistration,
> DatanodeProtocol.INVALID_BLOCK,
> errStr);
> //
> // ******** This should be continue instead of break?
> //
> break;
> }
> int numTargets = xferTargets[i].length;
> if (numTargets > 0) {
> if (LOG.isInfoEnabled()) {
> StringBuilder xfersBuilder = new StringBuilder();
> for (int j = 0; j < numTargets; j++) {
> DatanodeInfo nodeInfo = xferTargets[i][j];
> xfersBuilder.append(nodeInfo.getName());
> if (j < (numTargets - 1)) {
> xfersBuilder.append(", ");
> }
> }
> String xfersTo = xfersBuilder.toString();
> LOG.info(dnRegistration + " Starting thread to transfer block " +
> blocks[i] + " to " + xfersTo);
> }
> new Daemon(new DataTransfer(xferTargets[i], blocks[i], this)).start();
> }
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.