You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Vijay <vi...@gmail.com> on 2009/05/08 20:51:54 UTC
File system filling with Logs - Hadoop+ Chukwa
Hi,
I am trying out Chukwa on HDFS and the filesystem is been filled up because
of the amount of logs been generated by hadoop filesystem, Can anyone help
me understand why this is happening and how can i reduce the speed atleast?
awapisvr:qawapiall:root > df -m
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/sda2 19234 7987 10270 44% /
/dev/sda1 282 24 243 9% /boot
none 3983 0 3983 0% /dev/shm
/dev/sda3 38459 903 35604 3% /opt
/dev/sda8 338395 43645 277561 14% /spare
/dev/sda7 19234 77 18180 1% /tmp
/dev/sda5 38459 1701 34805 5% /var
64.68.126.215:/vol/vol2/logarchive
286720 52678 234043 19% /backup
awapisvr:qawapiall:root > df -m
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/sda2 19234 7987 10270 44% /
/dev/sda1 282 24 243 9% /boot
none 3983 0 3983 0% /dev/shm
/dev/sda3 38459 903 35604 3% /opt
/dev/sda8 338395 43648 277559 14% /spare
/dev/sda7 19234 77 18180 1% /tmp
/dev/sda5 38459 1701 34805 5% /var
64.68.126.215:/vol/vol2/logarchive
286720 52678 234043 19% /backup
in matter of millisec
i see the below in the logs...
2009-05-08 18:50:38,351 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel ip=/127.0.0.1
cmd=listStatus src=/chukwa/dataSinkArchives dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel ip=/127.0.0.1
cmd=listStatus src=/chukwa/dataSinkArchives dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel ip=/127.0.0.1
cmd=listStatus src=/chukwa/dataSinkArchives dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel ip=/127.0.0.1
cmd=listStatus src=/chukwa/dataSinkArchives dst=nullperm=null
2009-05-08 18:50:38,352 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel ip=/127.0.0.1
cmd=listStatus src=/chukwa/dataSinkArchives dst=nullperm=null
2009-05-08 18:50:38,353 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=root,root,bin,daemon,sys,adm,disk,wheel ip=/127.0.0.1
cmd=listStatus src=/chukwa/dataSinkArchives dst=nullperm=null
and
28527974-127.0.0.1-50010-1241721203027, blockid:
blk_-4466028835570444014_1290
2009-05-08 18:48:54,137 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
blk_-4466028835570444014_1290 terminating
2009-05-08 18:50:33,139 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-7272993078256778328_1291 src: /127.0.0.1:57067 dest: /127.0.0.1:50010
2009-05-08 18:50:34,222 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /
127.0.0.1:57067, dest: /127.0.0.1:50010, bytes: 69046, op: HDFS_WRITE,
cliID: DFSClient_-223712422, srvID:
DS-2128527974-127.0.0.1-50010-1241721203027, blockid:
blk_-7272993078256778328_1291
2009-05-08 18:50:34,223 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block
blk_-7272993078256778328_1291 terminating
2009-05-08 18:50:35,145 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block
blk_-2496183264937203840_1292 src: /127.0.0.1:57136 dest: /127.0.0.1:50010
2009-05-08 18:50:41,920 INFO
org.apache.hadoop.hdfs.server.datanode.DataBlockScanner: Verification
succeeded for blk_2625289997411230424_1240
Regards,
</VJ>