You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by ch huang <ju...@gmail.com> on 2014/04/10 03:46:50 UTC
use setrep change number of file replicas,but not work
hi,maillist:
i try modify replica number on some dir but it seems not work
,anyone know why?
# sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
Replication 2 set:
/user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
the file still store 3 replica ,but the echo number changed
# hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
Found 1 items
-rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
/user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
# sudo -u hdfs hdfs fsck
/user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
-blocks -locations
Connecting to namenode via http://ch11:50070
FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
/user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr 10
09:39:51 CST 2014
/user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660 bytes,
1 block(s): OK
0.
BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
192.168.11.14:50010]
i remove the file ,and upload new file ,as i understand ,the new file
should be stored in 2 replica,but it still store 3 replica ,why?
# sudo -u hdfs hadoop fs -rm -r -skipTrash
/user/hive/warehouse/mytest/dsp_request/2014-01-26/*
Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
# hadoop fs -put ./data_00000
/user/hive/warehouse/mytest/dsp_request/2014-01-26/
[root@ch12 ~]# hadoop fs -ls
/user/hive/warehouse/mytest/dsp_request/2014-01-26
Found 1 items
-rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
/user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
# sudo -u hdfs hdfs fsck
/user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
-blocks -locations
Connecting to namenode via http://ch11:50070
FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
/user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr 10
09:41:12 CST 2014
/user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660 bytes,
1 block(s): OK
0.
BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
192.168.11.13:50010]
Re: use setrep change number of file replicas,but not work
Posted by ch huang <ju...@gmail.com>.
i can use fsck to get Over-replicated blocks but how can i track pending
delete ?
On Thu, Apr 10, 2014 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> The replica deletion is asynchronous. You can track its deletions via
> the NameNode's over-replicated blocks and the pending delete metrics.
>
> On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> > hi,maillist:
> > i try modify replica number on some dir but it seems not work
> > ,anyone know why?
> >
> > # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> > Replication 2 set:
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > the file still store 3 replica ,but the echo number changed
> > # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:39:51 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> >
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> > len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> > 192.168.11.14:50010]
> >
> > i remove the file ,and upload new file ,as i understand ,the new file
> should
> > be stored in 2 replica,but it still store 3 replica ,why?
> > # sudo -u hdfs hadoop fs -rm -r -skipTrash
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> > Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # hadoop fs -put ./data_00000
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> > [root@ch12 ~]# hadoop fs -ls
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:41:12 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> > len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> > 192.168.11.13:50010]
>
>
>
> --
> Harsh J
>
Re: use setrep change number of file replicas,but not work
Posted by ch huang <ju...@gmail.com>.
i can use fsck to get Over-replicated blocks but how can i track pending
delete ?
On Thu, Apr 10, 2014 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> The replica deletion is asynchronous. You can track its deletions via
> the NameNode's over-replicated blocks and the pending delete metrics.
>
> On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> > hi,maillist:
> > i try modify replica number on some dir but it seems not work
> > ,anyone know why?
> >
> > # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> > Replication 2 set:
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > the file still store 3 replica ,but the echo number changed
> > # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:39:51 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> >
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> > len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> > 192.168.11.14:50010]
> >
> > i remove the file ,and upload new file ,as i understand ,the new file
> should
> > be stored in 2 replica,but it still store 3 replica ,why?
> > # sudo -u hdfs hadoop fs -rm -r -skipTrash
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> > Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # hadoop fs -put ./data_00000
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> > [root@ch12 ~]# hadoop fs -ls
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:41:12 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> > len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> > 192.168.11.13:50010]
>
>
>
> --
> Harsh J
>
Re: use setrep change number of file replicas,but not work
Posted by ch huang <ju...@gmail.com>.
i set replica number from 3 to 2,but i dump NN metrics ,the
PendingDeletionBlocks is zero ,why?
if the check thread will sleep a interval then do it's check work ,how long
the interval time is?
On Thu, Apr 10, 2014 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> The replica deletion is asynchronous. You can track its deletions via
> the NameNode's over-replicated blocks and the pending delete metrics.
>
> On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> > hi,maillist:
> > i try modify replica number on some dir but it seems not work
> > ,anyone know why?
> >
> > # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> > Replication 2 set:
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > the file still store 3 replica ,but the echo number changed
> > # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:39:51 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> >
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> > len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> > 192.168.11.14:50010]
> >
> > i remove the file ,and upload new file ,as i understand ,the new file
> should
> > be stored in 2 replica,but it still store 3 replica ,why?
> > # sudo -u hdfs hadoop fs -rm -r -skipTrash
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> > Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # hadoop fs -put ./data_00000
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> > [root@ch12 ~]# hadoop fs -ls
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:41:12 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> > len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> > 192.168.11.13:50010]
>
>
>
> --
> Harsh J
>
Re: use setrep change number of file replicas,but not work
Posted by ch huang <ju...@gmail.com>.
i can use fsck to get Over-replicated blocks but how can i track pending
delete ?
On Thu, Apr 10, 2014 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> The replica deletion is asynchronous. You can track its deletions via
> the NameNode's over-replicated blocks and the pending delete metrics.
>
> On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> > hi,maillist:
> > i try modify replica number on some dir but it seems not work
> > ,anyone know why?
> >
> > # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> > Replication 2 set:
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > the file still store 3 replica ,but the echo number changed
> > # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:39:51 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> >
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> > len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> > 192.168.11.14:50010]
> >
> > i remove the file ,and upload new file ,as i understand ,the new file
> should
> > be stored in 2 replica,but it still store 3 replica ,why?
> > # sudo -u hdfs hadoop fs -rm -r -skipTrash
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> > Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # hadoop fs -put ./data_00000
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> > [root@ch12 ~]# hadoop fs -ls
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:41:12 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> > len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> > 192.168.11.13:50010]
>
>
>
> --
> Harsh J
>
Re: use setrep change number of file replicas,but not work
Posted by ch huang <ju...@gmail.com>.
i can use fsck to get Over-replicated blocks but how can i track pending
delete ?
On Thu, Apr 10, 2014 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> The replica deletion is asynchronous. You can track its deletions via
> the NameNode's over-replicated blocks and the pending delete metrics.
>
> On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> > hi,maillist:
> > i try modify replica number on some dir but it seems not work
> > ,anyone know why?
> >
> > # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> > Replication 2 set:
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > the file still store 3 replica ,but the echo number changed
> > # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:39:51 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> >
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> > len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> > 192.168.11.14:50010]
> >
> > i remove the file ,and upload new file ,as i understand ,the new file
> should
> > be stored in 2 replica,but it still store 3 replica ,why?
> > # sudo -u hdfs hadoop fs -rm -r -skipTrash
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> > Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # hadoop fs -put ./data_00000
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> > [root@ch12 ~]# hadoop fs -ls
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:41:12 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> > len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> > 192.168.11.13:50010]
>
>
>
> --
> Harsh J
>
Re: use setrep change number of file replicas,but not work
Posted by ch huang <ju...@gmail.com>.
i set replica number from 3 to 2,but i dump NN metrics ,the
PendingDeletionBlocks is zero ,why?
if the check thread will sleep a interval then do it's check work ,how long
the interval time is?
On Thu, Apr 10, 2014 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> The replica deletion is asynchronous. You can track its deletions via
> the NameNode's over-replicated blocks and the pending delete metrics.
>
> On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> > hi,maillist:
> > i try modify replica number on some dir but it seems not work
> > ,anyone know why?
> >
> > # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> > Replication 2 set:
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > the file still store 3 replica ,but the echo number changed
> > # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:39:51 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> >
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> > len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> > 192.168.11.14:50010]
> >
> > i remove the file ,and upload new file ,as i understand ,the new file
> should
> > be stored in 2 replica,but it still store 3 replica ,why?
> > # sudo -u hdfs hadoop fs -rm -r -skipTrash
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> > Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # hadoop fs -put ./data_00000
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> > [root@ch12 ~]# hadoop fs -ls
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:41:12 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> > len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> > 192.168.11.13:50010]
>
>
>
> --
> Harsh J
>
Re: use setrep change number of file replicas,but not work
Posted by ch huang <ju...@gmail.com>.
i set replica number from 3 to 2,but i dump NN metrics ,the
PendingDeletionBlocks is zero ,why?
if the check thread will sleep a interval then do it's check work ,how long
the interval time is?
On Thu, Apr 10, 2014 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> The replica deletion is asynchronous. You can track its deletions via
> the NameNode's over-replicated blocks and the pending delete metrics.
>
> On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> > hi,maillist:
> > i try modify replica number on some dir but it seems not work
> > ,anyone know why?
> >
> > # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> > Replication 2 set:
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > the file still store 3 replica ,but the echo number changed
> > # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:39:51 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> >
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> > len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> > 192.168.11.14:50010]
> >
> > i remove the file ,and upload new file ,as i understand ,the new file
> should
> > be stored in 2 replica,but it still store 3 replica ,why?
> > # sudo -u hdfs hadoop fs -rm -r -skipTrash
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> > Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # hadoop fs -put ./data_00000
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> > [root@ch12 ~]# hadoop fs -ls
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:41:12 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> > len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> > 192.168.11.13:50010]
>
>
>
> --
> Harsh J
>
Re: use setrep change number of file replicas,but not work
Posted by ch huang <ju...@gmail.com>.
i set replica number from 3 to 2,but i dump NN metrics ,the
PendingDeletionBlocks is zero ,why?
if the check thread will sleep a interval then do it's check work ,how long
the interval time is?
On Thu, Apr 10, 2014 at 10:50 AM, Harsh J <ha...@cloudera.com> wrote:
> The replica deletion is asynchronous. You can track its deletions via
> the NameNode's over-replicated blocks and the pending delete metrics.
>
> On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> > hi,maillist:
> > i try modify replica number on some dir but it seems not work
> > ,anyone know why?
> >
> > # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> > Replication 2 set:
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > the file still store 3 replica ,but the echo number changed
> > # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> >
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:39:51 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> >
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> > len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> > 192.168.11.14:50010]
> >
> > i remove the file ,and upload new file ,as i understand ,the new file
> should
> > be stored in 2 replica,but it still store 3 replica ,why?
> > # sudo -u hdfs hadoop fs -rm -r -skipTrash
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> > Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # hadoop fs -put ./data_00000
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> > [root@ch12 ~]# hadoop fs -ls
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26
> > Found 1 items
> > -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> > # sudo -u hdfs hdfs fsck
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files
> -blocks
> > -locations
> > Connecting to namenode via http://ch11:50070
> > FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr
> 10
> > 09:41:12 CST 2014
> > /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660
> bytes, 1
> > block(s): OK
> > 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> > len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> > 192.168.11.13:50010]
>
>
>
> --
> Harsh J
>
Re: use setrep change number of file replicas,but not work
Posted by Harsh J <ha...@cloudera.com>.
The replica deletion is asynchronous. You can track its deletions via
the NameNode's over-replicated blocks and the pending delete metrics.
On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i try modify replica number on some dir but it seems not work
> ,anyone know why?
>
> # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> Replication 2 set:
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
>
> the file still store 3 replica ,but the echo number changed
> # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> Found 1 items
> -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
>
> # sudo -u hdfs hdfs fsck
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files -blocks
> -locations
> Connecting to namenode via http://ch11:50070
> FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr 10
> 09:39:51 CST 2014
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660 bytes, 1
> block(s): OK
> 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> 192.168.11.14:50010]
>
> i remove the file ,and upload new file ,as i understand ,the new file should
> be stored in 2 replica,but it still store 3 replica ,why?
> # sudo -u hdfs hadoop fs -rm -r -skipTrash
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> # hadoop fs -put ./data_00000
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> [root@ch12 ~]# hadoop fs -ls
> /user/hive/warehouse/mytest/dsp_request/2014-01-26
> Found 1 items
> -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> # sudo -u hdfs hdfs fsck
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files -blocks
> -locations
> Connecting to namenode via http://ch11:50070
> FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr 10
> 09:41:12 CST 2014
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660 bytes, 1
> block(s): OK
> 0. BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> 192.168.11.13:50010]
--
Harsh J
Re: use setrep change number of file replicas,but not work
Posted by Harsh J <ha...@cloudera.com>.
The replica deletion is asynchronous. You can track its deletions via
the NameNode's over-replicated blocks and the pending delete metrics.
On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i try modify replica number on some dir but it seems not work
> ,anyone know why?
>
> # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> Replication 2 set:
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
>
> the file still store 3 replica ,but the echo number changed
> # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> Found 1 items
> -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
>
> # sudo -u hdfs hdfs fsck
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files -blocks
> -locations
> Connecting to namenode via http://ch11:50070
> FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr 10
> 09:39:51 CST 2014
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660 bytes, 1
> block(s): OK
> 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> 192.168.11.14:50010]
>
> i remove the file ,and upload new file ,as i understand ,the new file should
> be stored in 2 replica,but it still store 3 replica ,why?
> # sudo -u hdfs hadoop fs -rm -r -skipTrash
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> # hadoop fs -put ./data_00000
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> [root@ch12 ~]# hadoop fs -ls
> /user/hive/warehouse/mytest/dsp_request/2014-01-26
> Found 1 items
> -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> # sudo -u hdfs hdfs fsck
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files -blocks
> -locations
> Connecting to namenode via http://ch11:50070
> FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr 10
> 09:41:12 CST 2014
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660 bytes, 1
> block(s): OK
> 0. BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> 192.168.11.13:50010]
--
Harsh J
Re: use setrep change number of file replicas,but not work
Posted by Harsh J <ha...@cloudera.com>.
The replica deletion is asynchronous. You can track its deletions via
the NameNode's over-replicated blocks and the pending delete metrics.
On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i try modify replica number on some dir but it seems not work
> ,anyone know why?
>
> # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> Replication 2 set:
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
>
> the file still store 3 replica ,but the echo number changed
> # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> Found 1 items
> -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
>
> # sudo -u hdfs hdfs fsck
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files -blocks
> -locations
> Connecting to namenode via http://ch11:50070
> FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr 10
> 09:39:51 CST 2014
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660 bytes, 1
> block(s): OK
> 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> 192.168.11.14:50010]
>
> i remove the file ,and upload new file ,as i understand ,the new file should
> be stored in 2 replica,but it still store 3 replica ,why?
> # sudo -u hdfs hadoop fs -rm -r -skipTrash
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> # hadoop fs -put ./data_00000
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> [root@ch12 ~]# hadoop fs -ls
> /user/hive/warehouse/mytest/dsp_request/2014-01-26
> Found 1 items
> -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> # sudo -u hdfs hdfs fsck
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files -blocks
> -locations
> Connecting to namenode via http://ch11:50070
> FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr 10
> 09:41:12 CST 2014
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660 bytes, 1
> block(s): OK
> 0. BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> 192.168.11.13:50010]
--
Harsh J
Re: use setrep change number of file replicas,but not work
Posted by Harsh J <ha...@cloudera.com>.
The replica deletion is asynchronous. You can track its deletions via
the NameNode's over-replicated blocks and the pending delete metrics.
On Thu, Apr 10, 2014 at 7:16 AM, ch huang <ju...@gmail.com> wrote:
> hi,maillist:
> i try modify replica number on some dir but it seems not work
> ,anyone know why?
>
> # sudo -u hdfs hadoop fs -setrep -R 2 /user/hive/warehouse/mytest
> Replication 2 set:
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
>
> the file still store 3 replica ,but the echo number changed
> # hadoop fs -ls /user/hive/warehouse/mytest/dsp_request/2014-01-26
> Found 1 items
> -rw-r--r-- 2 hdfs hdfs 17660 2014-01-26 18:34
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
>
> # sudo -u hdfs hdfs fsck
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files -blocks
> -locations
> Connecting to namenode via http://ch11:50070
> FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr 10
> 09:39:51 CST 2014
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660 bytes, 1
> block(s): OK
> 0.
> BP-1043055049-192.168.11.11-1382442676609:blk_-9219869107960013037_1976591
> len=17660 repl=3 [192.168.11.13:50010, 192.168.11.10:50010,
> 192.168.11.14:50010]
>
> i remove the file ,and upload new file ,as i understand ,the new file should
> be stored in 2 replica,but it still store 3 replica ,why?
> # sudo -u hdfs hadoop fs -rm -r -skipTrash
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/*
> Deleted /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> # hadoop fs -put ./data_00000
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/
> [root@ch12 ~]# hadoop fs -ls
> /user/hive/warehouse/mytest/dsp_request/2014-01-26
> Found 1 items
> -rw-r--r-- 3 root hdfs 17660 2014-04-10 09:40
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000
> # sudo -u hdfs hdfs fsck
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 -files -blocks
> -locations
> Connecting to namenode via http://ch11:50070
> FSCK started by hdfs (auth:SIMPLE) from /192.168.11.12 for path
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 at Thu Apr 10
> 09:41:12 CST 2014
> /user/hive/warehouse/mytest/dsp_request/2014-01-26/data_00000 17660 bytes, 1
> block(s): OK
> 0. BP-1043055049-192.168.11.11-1382442676609:blk_6517693524032437780_8889786
> len=17660 repl=3 [192.168.11.12:50010, 192.168.11.15:50010,
> 192.168.11.13:50010]
--
Harsh J