You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Ning Li (JIRA)" <ji...@apache.org> on 2008/08/11 20:24:44 UTC
[jira] Created: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
DataNode's BlockSender sends more data than necessary
-----------------------------------------------------
Key: HADOOP-3933
URL: https://issues.apache.org/jira/browse/HADOOP-3933
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Reporter: Ning Li
Priority: Minor
DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624806#action_12624806 ]
Hudson commented on HADOOP-3933:
--------------------------------
Integrated in Hadoop-trunk #581 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/581/])
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch, hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3933:
---------------------------------
Affects Version/s: 0.16.0
Fix Version/s: 0.19.0
Hadoop Flags: [Reviewed]
+1 for the patch. Marking this for 0.19. Not sure how to assign to Ning.
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3933:
---------------------------------
Status: Patch Available (was: Open)
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch, hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3933:
------------------------------------
Release Note: (was: DataNode sometimes sends up to io.bytes.per.checksum bytes more than required to client.)
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch, hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Ning Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Li updated HADOOP-3933:
----------------------------
Status: Patch Available (was: Open)
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Ning Li
> Priority: Minor
> Attachments: hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Ning Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621538#action_12621538 ]
Ning Li commented on HADOOP-3933:
---------------------------------
BlockSender's constructor contains the following:
offset = (startOffset - (startOffset % bytesPerChecksum));
if (length >= 0) {
// Make sure endOffset points to end of a checksumed chunk.
long tmpLen = startOffset + length + (startOffset - offset);
if (tmpLen % bytesPerChecksum != 0) {
tmpLen += (bytesPerChecksum - tmpLen % bytesPerChecksum);
}
if (tmpLen < endOffset) {
endOffset = tmpLen;
}
}
The line "long tmpLen = startOffset + length + (startOffset - offset);"
makes BlockSender send more data than necessary. It should be
replaced by "long tmpLen = startOffset + length;".
All unit tests pass. No additional test is needed. Do I need to submit
a separate patch file? It's a one-line fix.
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Ning Li
> Priority: Minor
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622942#action_12622942 ]
Raghu Angadi commented on HADOOP-3933:
--------------------------------------
+1. The build failures are known problems (e.g. HADOOP-3650). I don't think this needs a unit test.
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12623002#action_12623002 ]
Hadoop QA commented on HADOOP-3933:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12388326/hadoop-3933.patch
against trunk revision 686181.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3059/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3059/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3059/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3059/console
This message is automatically generated.
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch, hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Tom White (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tom White reassigned HADOOP-3933:
---------------------------------
Assignee: Ning Li
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3933:
---------------------------------
Attachment: hadoop-3933.patch
updated patch to remove conflict with HADOOP-3935.
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch, hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3933:
---------------------------------
Status: Open (was: Patch Available)
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch, hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12622095#action_12622095 ]
Hadoop QA commented on HADOOP-3933:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12387978/hadoop-3933.patch
against trunk revision 685406.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no tests are needed for this patch.
-1 javadoc. The javadoc tool appears to have generated 1 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3050/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3050/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3050/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3050/console
This message is automatically generated.
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12621545#action_12621545 ]
Raghu Angadi commented on HADOOP-3933:
--------------------------------------
+1. The existing code reads 0 to 511 bytes more depending on startOffset.
> Do I need to submit a separate patch file? It's a one-line fix.
yes. please.
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Ning Li
> Priority: Minor
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3933:
---------------------------------
Resolution: Fixed
Release Note: DataNode sometimes sends up to io.bytes.per.checksum bytes more than required to client.
Status: Resolved (was: Patch Available)
I just committed this. Thanks Ning.
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.16.0
> Reporter: Ning Li
> Assignee: Ning Li
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: hadoop-3933.patch, hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3933) DataNode's BlockSender sends more
data than necessary
Posted by "Ning Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ning Li updated HADOOP-3933:
----------------------------
Attachment: hadoop-3933.patch
The patch is attached.
> The existing code reads 0 to 511 bytes more depending on startOffset.
It could be more depending on the value of "io.bytes.per.checksum".
> DataNode's BlockSender sends more data than necessary
> -----------------------------------------------------
>
> Key: HADOOP-3933
> URL: https://issues.apache.org/jira/browse/HADOOP-3933
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Ning Li
> Priority: Minor
> Attachments: hadoop-3933.patch
>
>
> DataNode's BlockSender sends more data than necessary because of how it computes endOffset.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.