You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2010/02/09 20:17:28 UTC
[jira] Created: (HDFS-962) Make DFSOutputStream MAX_PACKETS
configurable
Make DFSOutputStream MAX_PACKETS configurable
---------------------------------------------
Key: HDFS-962
URL: https://issues.apache.org/jira/browse/HDFS-962
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs client
Affects Versions: 0.22.0
Reporter: Todd Lipcon
Priority: Minor
HDFS-959 suggests that the MAX_PACKETS variable (which determines how many outstanding data packets the DFSOutputStream will permit) may have an impact on performance. If so, we should make it configurable to trade off between memory and performance. I think it ought to be a secret/undocumented config for now - this will make it easier to benchmark without confusing users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-962) Make DFSOutputStream MAX_PACKETS
configurable
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926436#action_12926436 ]
dhruba borthakur commented on HDFS-962:
---------------------------------------
The DFSClient has two parameters: the number of packets to buffer in the sliding window (default of 80) and the size of each packet. My first preference would be to make the number of packets a config parameter. Next, we can make the packet size (default of 64K) to be a config parameter too.
> Make DFSOutputStream MAX_PACKETS configurable
> ---------------------------------------------
>
> Key: HDFS-962
> URL: https://issues.apache.org/jira/browse/HDFS-962
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Priority: Minor
>
> HDFS-959 suggests that the MAX_PACKETS variable (which determines how many outstanding data packets the DFSOutputStream will permit) may have an impact on performance. If so, we should make it configurable to trade off between memory and performance. I think it ought to be a secret/undocumented config for now - this will make it easier to benchmark without confusing users.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.