You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Bikash Singhal <bi...@cybage.com> on 2010/06/18 12:37:15 UTC
FW: hadoop-cluster-1.ic node errors and warnings
Hi folks ,
I have received this error in the hadoop cluster. Has anybody anybody
seen this . Any solution.
--------------------- namenode errors and warning ------------------
2010-06-17 18:35:19,970 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place
enough replicas, still in need of 1
2010-06-17 18:35:19,972 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
ask 10.13.53.127:50010 to delete blk_-5851901968674962584_28816
blk_-5704762921662804278_28806 blk_-3249260253840047858_28816
blk_-6785644112288085978_28816 blk_-5991019206688403229_28806
blk_3615269337590779570_28806 blk_-1181514666968321558_28806
blk_7264052281086809701_28816 blk_663323733333557946_28781
blk_5673553217372004676_28816 blk_3626732201125971756_28806
blk_8707314634323417208_28816 blk_-6887179921277000590_28812
blk_-3816018501532347017_28816 blk_-7238540159191278524_28816
blk_-7658691789280578071_28806 blk_3236948345360966700_28816
blk_-6254109926970173434_28816 blk_5195430663886472445_28816
blk_-412573738832546671_28816 blk_8109473696289375765_28806
blk_-5810373118380394822_28816 blk_2743979608816557043_28806
blk_5715402730928093013_28816 blk_8464503407273685120_28806
blk_-5062152410152898606_28816 blk_-3177617939821456482_28810
blk_-5555427680789719632_28816 blk_-5789474774332045366_28806
blk_-5623935836874134523_28806 blk_-4337702879554700741_28806
blk_-8461056881907762006_28806 blk_9026822631507898528_28816
blk_7156167055494887627_28816 blk_-8277210689227324100_28806
blk_-3751852098002911697_28806 blk_-5585636802791023762_28806
blk_7895962823848655899_28806 blk_-990310910545048635_28816
blk_-9154860186123815888_28806 blk_-4227488676014658152_28806
blk_8936924022792123139_28816 blk_-769244617229872944_28816
blk_2688806916705051324_28816 blk_-4338943142397506199_28816
blk_-509403461442878177_28806 blk_-4842888174619311915_28815
blk_-3952777283518699846_28806 blk_-5861717742640359549_28806
blk_-3952576171995960772_28816 blk_-163789650674468878_28816
blk_-4592795213928607535_28806 blk_2333271861949132522_28816
blk_4198208275476238889_28816 blk_360552057991500806_28816
blk_5491773302900671409_28806 blk_4047676415653175342_28806
blk_-1004260037826453175_28806 blk_2767440385264091225_28816
blk_5476421277228240141_28816 blk_4407005806101654351_28816
blk_4195254311
970318771_28816 blk_-737647080244712447_28806 blk_7438398943!
33029124
8_28806 blk_9065794226469565702_28806 blk_-909573278974779662_28816
blk_-720057463076992315_28806 blk_-3706189772409350949_28816
blk_-7669121304118161359_28816 blk_5626381233374602629_28816
blk_3226781108133048989_28806 blk_4534712638713702794_28816
blk_-2501367440326715009_28816 blk_1182598706752583066_28806
blk_2899109514732336036_28813 blk_-3583975497823582887_28806
blk_1637573118385058229_28816 blk_6123281255845765914_28816
blk_-7325625322659708014_28816 blk_5881568644646580533_28816
blk_-3117891195870623616_28816 blk_4293852484387807756_28806
blk_1053300130581981980_28781 blk_2196698001825902338_28809
2010-06-17 18:35:19,974 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
ask 10.13.53.128:50010 to delete blk_-5851901968674962584_28816
blk_-5991019206688403229_28806 blk_-3249260253840047858_28816
blk_-1074403248426101769_28816 blk_1102552384917878009_28816
blk_663323733333557946_28781 blk_7264052281086809701_28816
blk_-9186151702356219036_28806 blk_-6887179921277000590_28812
blk_-7403187390102989856_28814 blk_-3816018501532347017_28816
blk_-3256647808151302716_28806 blk_5195430663886472445_28816
blk_-6254109926970173434_28816 blk_-412573738832546671_28816
blk_8109473696289375765_28806 blk_2743979608816557043_28806
blk_8464503407273685120_28806 blk_-1927566815393964066_28813
blk_-910165942916728217_28816 blk_6991381516571090085_28816
blk_-5062152410152898606_28816 blk_-5555427680789719632_28816
blk_-347418184816360801_28806 blk_9026822631507898528_28816
blk_8645717205835711727_28816 blk_-8277210689227324100_28806
blk_-6820670802505184305_28809 blk_7156167055494887627_28816
blk_-3751852098002911697_28806 blk_2230966748060172045_28816
blk_-9154860186123815888_28806 blk_-4227488676014658152_28806
blk_8936924022792123139_28816 blk_-3952777283518699846_28806
blk_-163789650674468878_28816 blk_2333271861949132522_28816
blk_5491773302900671409_28806 blk_4047676415653175342_28806
blk_7147022655053153746_28806 blk_5476421277228240141_28816
blk_3601702425471675469_28806 blk_-1812525521195434130_28809
blk_6208014296194515686_28816 blk_-3588835264704726772_28816
blk_5487381930889488938_28806 blk_1786827488554715485_28806
blk_6381806163859245084_28806 blk_-909573278974779662_28816
blk_-720057463076992315_28806 blk_-3706189772409350949_28816
blk_5626381233374602629_28816 blk_8812361351759351875_28806
blk_3226781108133048989_28806 blk_2626414889098125951_28816
blk_5642048906322764709_28806 blk_-3583975497823582887_28806
blk_1637573118385058229_28816 blk_-7325625322659708014_28816
blk_4293852484387807756_28806 blk_7869940696281550339_28806
blk_2196698001825902
338_28809 blk_-6887326635813252129_28816 blk_-57047629216628!
04278_28
806 blk_3615269337590779570_28806 blk_-6785644112288085978_28816
blk_3880871563914097435_28806 blk_-1181514666968321558_28806
blk_3626732201125971756_28806 blk_-5111860011657356431_28812
blk_5673553217372004676_28816 blk_1144230701070834232_28816
blk_8707314634323417208_28816 blk_-2478324223605041774_28808
blk_-7238540159191278524_28816 blk_-7658691789280578071_28806
blk_3236948345360966700_28816 blk_744702532009597216_28816
blk_-5810373118380394822_28816 blk_2247163771290221731_28808
blk_5715402730928093013_28816 blk_-3177617939821456482_28810
blk_-5789474774332045366_28806 blk_-5623935836874134523_28806
blk_-4682289668534212518_28816 blk_-4337702879554700741_28806
blk_-8461056881907762006_28806 blk_3225744222537772912_28816
blk_-7377461388357398241_28816 blk_-5585636802791023762_28806
blk_7787185831361186700_28816 blk_7895962823848655899_28806
blk_-990310910545048635_28816 blk_-1876758814669931378_28816
blk_-769244617229872944_28816 blk_-497469852942635196_28806 blk_-433894
3142397506199_28816 blk_2688806916705051324_28816
blk_-6654089746586683607_28816 blk_-509403461442878177_28806
2010-06-17 18:35:19,975 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.127:50010 is added
to blk_151014847393206974_28824 size 41837
2010-06-17 18:35:19,978 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/var/lib/hadoop-0.20/cache/hadoop/mapred/system/job_201006090123_0068/jo
b.xml is closed by DFSClient_-1596407549
2010-06-17 18:35:19,986 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=hadoop,hadoop ip=/10.13.53.127 cmd=open
src=/var/lib/hadoop-0.20/cache/hadoop/mapred/system/job_201006090123_006
8/job.xml dst=null perm=null
2010-06-17 18:35:20,078 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=hadoop,hadoop ip=/10.13.53.127 cmd=open
src=/var/lib/hadoop-0.20/cache/hadoop/mapred/system/job_201006090123_006
8/job.jar dst=null perm=null
2010-06-17 18:35:21,581 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=hadoop,hadoop ip=/10.13.53.127 cmd=create
src=/i2a_logs/2010-06-16/output_i2a-coll-6.ic-/_logs/history/hadoop-mast
er_1276071808358_job_201006090123_0068_hadoop_I2ALogArchiver dst=null
perm=hadoop:supergroup:rw-r--r--
2010-06-17 18:35:21,710 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=hadoop,hadoop ip=/10.13.53.127 cmd=create
src=/i2a_logs/2010-06-16/output_i2a-coll-6.ic-/_logs/history/hadoop-mast
er_1276071808358_job_201006090123_0068_conf.xml dst=null
perm=hadoop:supergroup:rw-r--r--
2010-06-17 18:35:21,731 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/i2a_logs/2010-06-16/output_i2a-coll-6.ic-/_logs/history/hadoop-master_1
276071808358_job_201006090123_0068_conf.xml.
blk_1315602836759907786_28826
2010-06-17 18:35:21,748 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.128:50010 is added
to blk_1315602836759907786_28826 size 41823
--
2010-06-17 18:45:11,591 WARN
org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
PendingReplicationMonitor timed out block blk_-2915452315391280184_28822
2010-06-17 18:50:03,643 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
transactions: 676 Total time for transactions(ms): 11Number of
transactions batched in Syncs: 4 Number of syncs: 69 SyncTimes(ms): 3632
2010-06-17 18:50:03,645 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=hadoop,hadoop ip=/10.13.53.129 cmd=create
src=/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182031-access.log.0
dst=null perm=hadoop:supergroup:rw-r--r--
2010-06-17 18:50:03,659 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182031-access.log.0.
blk_-5308288487531094520_28832
2010-06-17 18:50:04,363 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.156:50010 is added
to blk_-5308288487531094520_28832 size 31804984
2010-06-17 18:50:04,363 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.128:50010 is added
to blk_-5308288487531094520_28832 size 31804984
2010-06-17 18:50:04,364 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.129:50010 is added
to blk_-5308288487531094520_28832 size 31804984
2010-06-17 18:50:04,366 INFO org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.completeFile: file
/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182031-access.log.0 is closed
by DFSClient_-1916945035
2010-06-17 18:50:06,172 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit:
ugi=hadoop,hadoop ip=/10.13.53.129 cmd=create
src=/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182531-access.log.0
dst=null perm=hadoop:supergroup:rw-r--r--
2010-06-17 18:50:06,187 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/i2a_logs/2010-06-17/i2a-coll-4.ic-100617-182531-access.log.0.
blk_1130168086083501329_28833
2010-06-17 18:50:07,024 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.13.53.127:50010 is added
to blk_1130168086083501329_28833 size 32278706
--------------------- jobtracker errors and warning ------------------
2010-06-17 17:00:06,382 WARN org.apache.hadoop.conf.Configuration:
/var/lib/hadoop-0.20/cache/hadoop/mapred/local/jobTracker/job_2010060901
23_0067.xml:a attempt to override final parameter: dfs.data.dir;
Ignoring.
2010-06-17 17:00:06,448 DEBUG org.apache.hadoop.mapred.JobTracker:
Per-Task memory configuration is not set on JT. Not checking the job for
invalid memory requirements.
2010-06-17 17:00:06,449 INFO org.apache.hadoop.mapred.JobTracker:
Initializing job_201006090123_0067
2010-06-17 17:00:06,451 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201006090123_0067
2010-06-17 17:00:06,698 INFO org.apache.hadoop.mapred.JobInProgress:
Input size for job job_201006090123_0067 = 8493462348. Number of splits
= 288
2010-06-17 17:00:06,698 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000000 has split on
node:/default-rack/10.13.53.127
2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000000 has split on
node:/default-rack/10.13.53.156
2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000000 has split on
node:/default-rack/10.13.53.128
2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000001 has split on
node:/default-rack/10.13.53.127
2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000001 has split on
node:/default-rack/10.13.53.129
2010-06-17 17:00:06,699 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0067_m_000001 has split on
node:/default-rack/10.13.53.128
--
2010-06-17 18:35:20,073 WARN org.apache.hadoop.conf.Configuration:
/var/lib/hadoop-0.20/cache/hadoop/mapred/local/jobTracker/job_2010060901
23_0068.xml:a attempt to override final parameter: dfs.data.dir;
Ignoring.
2010-06-17 18:35:20,132 DEBUG org.apache.hadoop.mapred.JobTracker:
Per-Task memory configuration is not set on JT. Not checking the job for
invalid memory requirements.
2010-06-17 18:35:20,133 INFO org.apache.hadoop.mapred.JobTracker:
Initializing job_201006090123_0068
2010-06-17 18:35:20,134 INFO org.apache.hadoop.mapred.JobInProgress:
Initializing job_201006090123_0068
2010-06-17 18:35:21,123 DEBUG org.apache.hadoop.mapred.JobTracker: Got
heartbeat from:
tracker_10.13.53.128:localhost.localdomain/127.0.0.1:35313 (restarted:
false initialContact: false acceptNewTasks: true) with responseId:
-32475
2010-06-17 18:35:21,186 DEBUG org.apache.hadoop.mapred.JobTracker: Got
heartbeat from:
tracker_10.13.53.127:localhost.localdomain/127.0.0.1:45581 (restarted:
false initialContact: false acceptNewTasks: true) with responseId:
-32074
2010-06-17 18:35:21,776 INFO org.apache.hadoop.mapred.JobInProgress:
Input size for job job_201006090123_0068 = 33327229. Number of splits =
288
2010-06-17 18:35:21,776 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0068_m_000000 has split on
node:/default-rack/10.13.53.127
2010-06-17 18:35:21,776 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0068_m_000000 has split on
node:/default-rack/10.13.53.156
2010-06-17 18:35:21,777 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0068_m_000000 has split on
node:/default-rack/10.13.53.129
2010-06-17 18:35:21,777 INFO org.apache.hadoop.mapred.JobInProgress:
tip:task_201006090123_0068_m_000001 has split on
node:/default-rack/10.13.53.127
--------------------- datanode errors and warning ------------------
Thanks & Regards
Bikash Singhal
"Legal Disclaimer: This electronic message and all contents contain information from Cybage Software Private Limited which may be privileged, confidential, or otherwise protected from disclosure. The information is intended to be for the addressee(s) only. If you are not an addressee, any disclosure, copy, distribution, or use of the contents of this message is strictly prohibited. If you have received this electronic message in error please notify the sender by reply e-mail to and destroy the original message and all copies. Cybage has taken every reasonable precaution to minimize the risk of malicious content in the mail, but is not liable for any damage you may sustain as a result of any malicious content in this e-mail. You should carry out your own malicious content checks before opening the e-mail or attachment."
www.cybage.com
Re: hadoop-cluster-1.ic node errors and warnings
Posted by Allen Wittenauer <aw...@linkedin.com>.
[Cutting the CC: line down to size ]
On Jun 18, 2010, at 3:37 AM, Bikash Singhal wrote:
> Hi folks ,
>
> I have received this error in the hadoop cluster. Has anybody anybody
> seen this . Any solution.
Since you aren't picking anything out and you've shared a bunch of messages, I'm going to go with the first one. If you have a specific question about a specific message, it would be helpful to point it out.
> --------------------- namenode errors and warning ------------------
>
> 2010-06-17 18:35:19,970 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place
> enough replicas, still in need of 1
This isn't an error, but a warning.
IIRC, you can get this either if the datanodes are out of space or there are not have enough racks defined to meet the defined replication policy. For example, requiring 10 replicas when only 3 racks are present will generate this warning.
Re: hadoop-cluster-1.ic node errors and warnings
Posted by Allen Wittenauer <aw...@linkedin.com>.
[Cutting the CC: line down to size ]
On Jun 18, 2010, at 3:37 AM, Bikash Singhal wrote:
> Hi folks ,
>
> I have received this error in the hadoop cluster. Has anybody anybody
> seen this . Any solution.
Since you aren't picking anything out and you've shared a bunch of messages, I'm going to go with the first one. If you have a specific question about a specific message, it would be helpful to point it out.
> --------------------- namenode errors and warning ------------------
>
> 2010-06-17 18:35:19,970 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Not able to place
> enough replicas, still in need of 1
This isn't an error, but a warning.
IIRC, you can get this either if the datanodes are out of space or there are not have enough racks defined to meet the defined replication policy. For example, requiring 10 replicas when only 3 racks are present will generate this warning.