You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-user@hadoop.apache.org by 조주일 <tj...@kgrid.co.kr> on 2014/06/30 07:50:10 UTC

How quickly can I increase the number of replicas?

 
My cluser version hdfs 2.2 stable ( 2 ha namenodes, 10 datanodes)
 
I was command bin/hdfs dfs -setrep -R 2 / ( replication 1 to 2 )
 
I found that HDFS is actually replicating the under replicated blocks but it works very slowly. 
HDFS performs the replication about 1 block per second.
I have about 400000 under replicated blocks so it will take about 4 more days.
Is there any way to speed it up?

Re: How quickly can I increase the number of replicas?

Posted by Vikas Srivastava <vi...@apple.com>.

Hey,

You can set the below properties to speed up replication

<property>
  <!-- 100Mbit/s -->
  <name>dfs.balance.bandwidthPerSec</name>
  <value>131072000</value>
</property>
<property>
  <name>dfs.max-repl-streams</name>
  <value>50</value>
</property>

Thanks
Vikas

On Jun 29, 2014, at 10:50 PM, 조주일 <tj...@kgrid.co.kr> wrote:

>  
> My cluser version hdfs 2.2 stable ( 2 ha namenodes, 10 datanodes)
> 
>  
> I was command bin/hdfs dfs -setrep -R 2 / ( replication 1 to 2 )
> 
>  
> I found that HDFS is actually replicating the under replicated blocks but it works very slowly. 
> 
> HDFS performs the replication about 1 block per second.
> 
> I have about 400000 under replicated blocks so it will take about 4 more days.
> 
> Is there any way to speed it up?
>

Re: How quickly can I increase the number of replicas?

Posted by Vikas Srivastava <vi...@apple.com>.

Hey,

You can set the below properties to speed up replication

<property>
  <!-- 100Mbit/s -->
  <name>dfs.balance.bandwidthPerSec</name>
  <value>131072000</value>
</property>
<property>
  <name>dfs.max-repl-streams</name>
  <value>50</value>
</property>

Thanks
Vikas

On Jun 29, 2014, at 10:50 PM, 조주일 <tj...@kgrid.co.kr> wrote:

>  
> My cluser version hdfs 2.2 stable ( 2 ha namenodes, 10 datanodes)
> 
>  
> I was command bin/hdfs dfs -setrep -R 2 / ( replication 1 to 2 )
> 
>  
> I found that HDFS is actually replicating the under replicated blocks but it works very slowly. 
> 
> HDFS performs the replication about 1 block per second.
> 
> I have about 400000 under replicated blocks so it will take about 4 more days.
> 
> Is there any way to speed it up?
>

Re: How quickly can I increase the number of replicas?

Posted by Vikas Srivastava <vi...@apple.com>.

Hey,

You can set the below properties to speed up replication

<property>
  <!-- 100Mbit/s -->
  <name>dfs.balance.bandwidthPerSec</name>
  <value>131072000</value>
</property>
<property>
  <name>dfs.max-repl-streams</name>
  <value>50</value>
</property>

Thanks
Vikas

On Jun 29, 2014, at 10:50 PM, 조주일 <tj...@kgrid.co.kr> wrote:

>  
> My cluser version hdfs 2.2 stable ( 2 ha namenodes, 10 datanodes)
> 
>  
> I was command bin/hdfs dfs -setrep -R 2 / ( replication 1 to 2 )
> 
>  
> I found that HDFS is actually replicating the under replicated blocks but it works very slowly. 
> 
> HDFS performs the replication about 1 block per second.
> 
> I have about 400000 under replicated blocks so it will take about 4 more days.
> 
> Is there any way to speed it up?
>

Re: How quickly can I increase the number of replicas?

Posted by Vikas Srivastava <vi...@apple.com>.

Hey,

You can set the below properties to speed up replication

<property>
  <!-- 100Mbit/s -->
  <name>dfs.balance.bandwidthPerSec</name>
  <value>131072000</value>
</property>
<property>
  <name>dfs.max-repl-streams</name>
  <value>50</value>
</property>

Thanks
Vikas

On Jun 29, 2014, at 10:50 PM, 조주일 <tj...@kgrid.co.kr> wrote:

>  
> My cluser version hdfs 2.2 stable ( 2 ha namenodes, 10 datanodes)
> 
>  
> I was command bin/hdfs dfs -setrep -R 2 / ( replication 1 to 2 )
> 
>  
> I found that HDFS is actually replicating the under replicated blocks but it works very slowly. 
> 
> HDFS performs the replication about 1 block per second.
> 
> I have about 400000 under replicated blocks so it will take about 4 more days.
> 
> Is there any way to speed it up?
>