You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Karthik Palanisamy (Jira)" <ji...@apache.org> on 2020/03/31 20:29:00 UTC
[jira] [Updated] (HDFS-15253) Set default throttle value on
dfs.image.transfer.bandwidthPerSec
[ https://issues.apache.org/jira/browse/HDFS-15253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthik Palanisamy updated HDFS-15253:
--------------------------------------
Description:
The default value dfs.image.transfer.bandwidthPerSec is set to 0 so it can use maximum available bandwidth for fsimage transfers during checkpoint. I think we should throttle this. Many users were experienced namenode failover when transferring large image size along with fsimage replication on dfs.namenode.name.dir. eg. >25Gb.
Thought to set,
dfs.image.transfer.bandwidthPerSec=52428800. (50 MB/s)
dfs.namenode.checkpoint.txns=2000000 (Default is 1M, good to avoid frequent checkpoint. However, the default checkpoint runs every 6 hours once)
was:
The default value dfs.image.transfer.bandwidthPerSec is set to 0 so it can use maximum available bandwidth for fsimage transfers during checkpoint. I think we should throttle this. Many users were experienced namenode failover when transferring large image size along with fsimage replication on dfs.namenode.name.dir. eg. >25Gb.
Thought to increase,
dfs.image.transfer.bandwidthPerSec=52428800. (50 MB/s)
dfs.namenode.checkpoint.txns=2000000 (Default is 1M, good to avoid frequent checkpoint. However, the default checkpoint runs every 6 hours once)
> Set default throttle value on dfs.image.transfer.bandwidthPerSec
> ----------------------------------------------------------------
>
> Key: HDFS-15253
> URL: https://issues.apache.org/jira/browse/HDFS-15253
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Reporter: Karthik Palanisamy
> Assignee: Karthik Palanisamy
> Priority: Major
>
> The default value dfs.image.transfer.bandwidthPerSec is set to 0 so it can use maximum available bandwidth for fsimage transfers during checkpoint. I think we should throttle this. Many users were experienced namenode failover when transferring large image size along with fsimage replication on dfs.namenode.name.dir. eg. >25Gb.
> Thought to set,
> dfs.image.transfer.bandwidthPerSec=52428800. (50 MB/s)
> dfs.namenode.checkpoint.txns=2000000 (Default is 1M, good to avoid frequent checkpoint. However, the default checkpoint runs every 6 hours once)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org