You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2012/09/25 02:04:07 UTC
[jira] [Created] (HDFS-3969) Small bug fixes and improvements for
disk locations API
Todd Lipcon created HDFS-3969:
---------------------------------
Summary: Small bug fixes and improvements for disk locations API
Key: HDFS-3969
URL: https://issues.apache.org/jira/browse/HDFS-3969
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 3.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
The new disk block locations API has a configurable timeout, but it's used inconsistently: the invokeAll() call to the thread pool assumes the timeout is in seconds, but the RPC timeout is set in milliseconds.
Also, we can improve the wire protocol for this API to be a lot more efficient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3969) Small bug fixes and improvements for
disk locations API
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462293#comment-13462293 ]
Todd Lipcon commented on HDFS-3969:
-----------------------------------
A few small improvements:
- I'm looking at a network trace of a usage of this API where the client is asking for locations of about 10,000 blocks. This resulted in a 750KB rpc call. Most of the data is spent in repeated serialization of the same block pool ID. We can change around the wire protocol so that the block pool is only passed once, and then use packed repeated fields for the block IDs. This should chop off about 40-45 bytes per block. We can also probably drop the genstamps, since blocks dont switch between drives when the gen stamp changes.
- It currently fills in an "empty" HdfsVolumeId which returns false for isValid() for any blocks that don't come back properly. Instead, I think the API would be clearer if it dropped those from the result rather than returning 'invalid' placeholder objects.
- We should improve the code so that the timeout parameter has a unit attached: eg 'timeoutMs' instead of 'timeout' to prevent bugs like this.
- When a DN does not respond correctly, we should include the exception text in the log message, even if not the full trace.
> Small bug fixes and improvements for disk locations API
> -------------------------------------------------------
>
> Key: HDFS-3969
> URL: https://issues.apache.org/jira/browse/HDFS-3969
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 3.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
>
> The new disk block locations API has a configurable timeout, but it's used inconsistently: the invokeAll() call to the thread pool assumes the timeout is in seconds, but the RPC timeout is set in milliseconds.
> Also, we can improve the wire protocol for this API to be a lot more efficient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3969) Small bug fixes and improvements for
disk locations API
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3969:
------------------------------
Status: Patch Available (was: Open)
> Small bug fixes and improvements for disk locations API
> -------------------------------------------------------
>
> Key: HDFS-3969
> URL: https://issues.apache.org/jira/browse/HDFS-3969
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 3.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-3969.txt
>
>
> The new disk block locations API has a configurable timeout, but it's used inconsistently: the invokeAll() call to the thread pool assumes the timeout is in seconds, but the RPC timeout is set in milliseconds.
> Also, we can improve the wire protocol for this API to be a lot more efficient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3969) Small bug fixes and improvements for
disk locations API
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HDFS-3969:
------------------------------
Attachment: hdfs-3969.txt
Here's an initial rev of this patch. Enumerating the changes:
- Got rid of the VolumeId.isValid() API. Instead, we just put a null in that slot of the locations array (a user of this API found the {{isValid()}} stuff confusing)
- Changed stringification to use hex encoding instead of Base64, since it's more readable by developers (eg "00000001" instead of "AAAAAQ==")
- Changed RPCs over to take a block pool ID and a packed array of longs, as described above. The RPCs should be substantially more efficient now.
- Fixed a bug where, if any of the blocks in the list were non-existent, it would previously throw an exception and return no data from the call. Now it just fills in a null. Added a new test case for this.
- Added some extra trace logs, etc, which I found handy in debugging here.
- Added exception text to the INFO log when a call fails
- Made the timeout default to 1 second instead of 60 seconds -- since this is a performance API, I expect a default timeout on this order is much more reasonable. Also renamed the parameter to make it a millisecond unit like most of our other timeouts, and fixed the timeout on the executor to match units.
This change is not wire-compatible, but since this API is marked experimental, I think it's OK. I used a new protobuf tag ID so that, if an old client talks to a new DN or vice versa, they'll get appropriate exceptions about missing fields rather than incorrect data.
> Small bug fixes and improvements for disk locations API
> -------------------------------------------------------
>
> Key: HDFS-3969
> URL: https://issues.apache.org/jira/browse/HDFS-3969
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 3.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-3969.txt
>
>
> The new disk block locations API has a configurable timeout, but it's used inconsistently: the invokeAll() call to the thread pool assumes the timeout is in seconds, but the RPC timeout is set in milliseconds.
> Also, we can improve the wire protocol for this API to be a lot more efficient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3969) Small bug fixes and improvements for
disk locations API
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13462496#comment-13462496 ]
Hadoop QA commented on HDFS-3969:
---------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12546427/hdfs-3969.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/3229//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/3229//console
This message is automatically generated.
> Small bug fixes and improvements for disk locations API
> -------------------------------------------------------
>
> Key: HDFS-3969
> URL: https://issues.apache.org/jira/browse/HDFS-3969
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
> Affects Versions: 3.0.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Attachments: hdfs-3969.txt
>
>
> The new disk block locations API has a configurable timeout, but it's used inconsistently: the invokeAll() call to the thread pool assumes the timeout is in seconds, but the RPC timeout is set in milliseconds.
> Also, we can improve the wire protocol for this API to be a lot more efficient.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira