You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by heyamin <he...@jiandan100.cn> on 2013/09/11 10:57:54 UTC
help!!!,what is happened with my project?
Hi:
Today when I run a task,I get some warnings,what is happened?
2013-09-11 16:45:17,486 INFO org.Apache.adopt.util.NativeCodeLoader: Loaded the native-hadoop library
2013-09-11 16:45:18,680 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-09-11 16:45:18,708 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ddc7394
2013-09-11 16:45:18,998 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://192.168.1.240:9000/user/hadoop/input/20130815-log.log:0+118673837
2013-09-11 16:45:19,027 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 40
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: data buffer = 31876710/39845888
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: record buffer = 104857/131072
2013-09-11 16:45:19,136 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 3082708; bufvoid = 39845888
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 104857; length = 131072
2013-09-11 16:45:23,661 INFO org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@36781b93
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:78)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2326)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.apache.hadoop.util.LineReader.close(LineReader.java:145)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:187)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:496)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1776)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:778)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-09-11 16:45:23,662 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
RE: help!!!,what is happened with my project?
Posted by Devaraj k <de...@huawei.com>.
It seems filesystem is closed in the Task. Do you see any error in Datanode/Namenode(HDFS) log files?
Thanks
Devaraj k
From: heyamin [mailto:heyamin@jiandan100.cn]
Sent: 11 September 2013 14:28
To: user@hadoop.apache.org
Cc: user-unsubscribe@hadoop.apache.org
Subject: help!!!,what is happened with my project?
Hi:
Today when I run a task,I get some warnings,what is happened?
2013-09-11 16:45:17,486 INFO org.Apache.adopt.util.NativeCodeLoader: Loaded the native-hadoop library
2013-09-11 16:45:18,680 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-09-11 16:45:18,708 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ddc7394<ma...@2ddc7394>
2013-09-11 16:45:18,998 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://192.168.1.240:9000/user/hadoop/input/20130815-log.log:0+118673837
2013-09-11 16:45:19,027 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 40
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: data buffer = 31876710/39845888
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: record buffer = 104857/131072
2013-09-11 16:45:19,136 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 3082708; bufvoid = 39845888
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 104857; length = 131072
2013-09-11 16:45:23,661 INFO org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@36781b93<ma...@36781b93>
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:78)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2326)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.apache.hadoop.util.LineReader.close(LineReader.java:145)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:187)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:496)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1776)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:778)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-09-11 16:45:23,662 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
RE: help!!!,what is happened with my project?
Posted by Devaraj k <de...@huawei.com>.
It seems filesystem is closed in the Task. Do you see any error in Datanode/Namenode(HDFS) log files?
Thanks
Devaraj k
From: heyamin [mailto:heyamin@jiandan100.cn]
Sent: 11 September 2013 14:28
To: user@hadoop.apache.org
Cc: user-unsubscribe@hadoop.apache.org
Subject: help!!!,what is happened with my project?
Hi:
Today when I run a task,I get some warnings,what is happened?
2013-09-11 16:45:17,486 INFO org.Apache.adopt.util.NativeCodeLoader: Loaded the native-hadoop library
2013-09-11 16:45:18,680 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-09-11 16:45:18,708 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ddc7394<ma...@2ddc7394>
2013-09-11 16:45:18,998 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://192.168.1.240:9000/user/hadoop/input/20130815-log.log:0+118673837
2013-09-11 16:45:19,027 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 40
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: data buffer = 31876710/39845888
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: record buffer = 104857/131072
2013-09-11 16:45:19,136 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 3082708; bufvoid = 39845888
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 104857; length = 131072
2013-09-11 16:45:23,661 INFO org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@36781b93<ma...@36781b93>
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:78)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2326)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.apache.hadoop.util.LineReader.close(LineReader.java:145)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:187)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:496)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1776)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:778)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-09-11 16:45:23,662 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
RE: help!!!,what is happened with my project?
Posted by java8964 java8964 <ja...@hotmail.com>.
Did you do a hadoop version upgrade before this error happened?
Yong
Date: Wed, 11 Sep 2013 16:57:54 +0800
From: heyamin@jiandan100.cn
To: user@hadoop.apache.org
CC: user-unsubscribe@hadoop.apache.org
Subject: help!!!,what is happened with my project?
Hi:
Today when I run a task,I get some warnings,what is happened?
2013-09-11 16:45:17,486 INFO org.Apache.adopt.util.NativeCodeLoader: Loaded the native-hadoop library
2013-09-11 16:45:18,680 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-09-11 16:45:18,708 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ddc7394
2013-09-11 16:45:18,998 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://192.168.1.240:9000/user/hadoop/input/20130815-log.log:0+118673837
2013-09-11 16:45:19,027 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 40
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: data buffer = 31876710/39845888
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: record buffer = 104857/131072
2013-09-11 16:45:19,136 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 3082708; bufvoid = 39845888
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 104857; length = 131072
2013-09-11 16:45:23,661 INFO org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@36781b93
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:78)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2326)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.apache.hadoop.util.LineReader.close(LineReader.java:145)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:187)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:496)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1776)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:778)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-09-11 16:45:23,662 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
RE: help!!!,what is happened with my project?
Posted by Devaraj k <de...@huawei.com>.
It seems filesystem is closed in the Task. Do you see any error in Datanode/Namenode(HDFS) log files?
Thanks
Devaraj k
From: heyamin [mailto:heyamin@jiandan100.cn]
Sent: 11 September 2013 14:28
To: user@hadoop.apache.org
Cc: user-unsubscribe@hadoop.apache.org
Subject: help!!!,what is happened with my project?
Hi:
Today when I run a task,I get some warnings,what is happened?
2013-09-11 16:45:17,486 INFO org.Apache.adopt.util.NativeCodeLoader: Loaded the native-hadoop library
2013-09-11 16:45:18,680 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-09-11 16:45:18,708 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ddc7394<ma...@2ddc7394>
2013-09-11 16:45:18,998 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://192.168.1.240:9000/user/hadoop/input/20130815-log.log:0+118673837
2013-09-11 16:45:19,027 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 40
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: data buffer = 31876710/39845888
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: record buffer = 104857/131072
2013-09-11 16:45:19,136 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 3082708; bufvoid = 39845888
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 104857; length = 131072
2013-09-11 16:45:23,661 INFO org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@36781b93<ma...@36781b93>
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:78)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2326)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.apache.hadoop.util.LineReader.close(LineReader.java:145)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:187)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:496)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1776)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:778)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-09-11 16:45:23,662 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
RE: help!!!,what is happened with my project?
Posted by java8964 java8964 <ja...@hotmail.com>.
Did you do a hadoop version upgrade before this error happened?
Yong
Date: Wed, 11 Sep 2013 16:57:54 +0800
From: heyamin@jiandan100.cn
To: user@hadoop.apache.org
CC: user-unsubscribe@hadoop.apache.org
Subject: help!!!,what is happened with my project?
Hi:
Today when I run a task,I get some warnings,what is happened?
2013-09-11 16:45:17,486 INFO org.Apache.adopt.util.NativeCodeLoader: Loaded the native-hadoop library
2013-09-11 16:45:18,680 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-09-11 16:45:18,708 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ddc7394
2013-09-11 16:45:18,998 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://192.168.1.240:9000/user/hadoop/input/20130815-log.log:0+118673837
2013-09-11 16:45:19,027 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 40
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: data buffer = 31876710/39845888
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: record buffer = 104857/131072
2013-09-11 16:45:19,136 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 3082708; bufvoid = 39845888
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 104857; length = 131072
2013-09-11 16:45:23,661 INFO org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@36781b93
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:78)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2326)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.apache.hadoop.util.LineReader.close(LineReader.java:145)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:187)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:496)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1776)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:778)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-09-11 16:45:23,662 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
RE: help!!!,what is happened with my project?
Posted by java8964 java8964 <ja...@hotmail.com>.
Did you do a hadoop version upgrade before this error happened?
Yong
Date: Wed, 11 Sep 2013 16:57:54 +0800
From: heyamin@jiandan100.cn
To: user@hadoop.apache.org
CC: user-unsubscribe@hadoop.apache.org
Subject: help!!!,what is happened with my project?
Hi:
Today when I run a task,I get some warnings,what is happened?
2013-09-11 16:45:17,486 INFO org.Apache.adopt.util.NativeCodeLoader: Loaded the native-hadoop library
2013-09-11 16:45:18,680 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-09-11 16:45:18,708 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ddc7394
2013-09-11 16:45:18,998 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://192.168.1.240:9000/user/hadoop/input/20130815-log.log:0+118673837
2013-09-11 16:45:19,027 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 40
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: data buffer = 31876710/39845888
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: record buffer = 104857/131072
2013-09-11 16:45:19,136 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 3082708; bufvoid = 39845888
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 104857; length = 131072
2013-09-11 16:45:23,661 INFO org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@36781b93
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:78)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2326)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.apache.hadoop.util.LineReader.close(LineReader.java:145)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:187)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:496)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1776)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:778)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-09-11 16:45:23,662 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
RE: help!!!,what is happened with my project?
Posted by java8964 java8964 <ja...@hotmail.com>.
Did you do a hadoop version upgrade before this error happened?
Yong
Date: Wed, 11 Sep 2013 16:57:54 +0800
From: heyamin@jiandan100.cn
To: user@hadoop.apache.org
CC: user-unsubscribe@hadoop.apache.org
Subject: help!!!,what is happened with my project?
Hi:
Today when I run a task,I get some warnings,what is happened?
2013-09-11 16:45:17,486 INFO org.Apache.adopt.util.NativeCodeLoader: Loaded the native-hadoop library
2013-09-11 16:45:18,680 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-09-11 16:45:18,708 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ddc7394
2013-09-11 16:45:18,998 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://192.168.1.240:9000/user/hadoop/input/20130815-log.log:0+118673837
2013-09-11 16:45:19,027 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 40
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: data buffer = 31876710/39845888
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: record buffer = 104857/131072
2013-09-11 16:45:19,136 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 3082708; bufvoid = 39845888
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 104857; length = 131072
2013-09-11 16:45:23,661 INFO org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@36781b93
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:78)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2326)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.apache.hadoop.util.LineReader.close(LineReader.java:145)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:187)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:496)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1776)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:778)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-09-11 16:45:23,662 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output
RE: help!!!,what is happened with my project?
Posted by Devaraj k <de...@huawei.com>.
It seems filesystem is closed in the Task. Do you see any error in Datanode/Namenode(HDFS) log files?
Thanks
Devaraj k
From: heyamin [mailto:heyamin@jiandan100.cn]
Sent: 11 September 2013 14:28
To: user@hadoop.apache.org
Cc: user-unsubscribe@hadoop.apache.org
Subject: help!!!,what is happened with my project?
Hi:
Today when I run a task,I get some warnings,what is happened?
2013-09-11 16:45:17,486 INFO org.Apache.adopt.util.NativeCodeLoader: Loaded the native-hadoop library
2013-09-11 16:45:18,680 INFO org.apache.hadoop.util.ProcessTree: setsid exited with exit code 0
2013-09-11 16:45:18,708 INFO org.apache.hadoop.mapred.Task: Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@2ddc7394<ma...@2ddc7394>
2013-09-11 16:45:18,998 INFO org.apache.hadoop.mapred.MapTask: Processing split: hdfs://192.168.1.240:9000/user/hadoop/input/20130815-log.log:0+118673837
2013-09-11 16:45:19,027 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb = 40
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: data buffer = 31876710/39845888
2013-09-11 16:45:19,113 INFO org.apache.hadoop.mapred.MapTask: record buffer = 104857/131072
2013-09-11 16:45:19,136 WARN org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library not loaded
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: Spilling map output: record full = true
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: bufstart = 0; bufend = 3082708; bufvoid = 39845888
2013-09-11 16:45:23,374 INFO org.apache.hadoop.mapred.MapTask: kvstart = 0; kvend = 104857; length = 131072
2013-09-11 16:45:23,661 INFO org.apache.hadoop.mapred.MapTask: Ignoring exception during close for org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader@36781b93<ma...@36781b93>
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:323)
at org.apache.hadoop.hdfs.DFSClient.access$1200(DFSClient.java:78)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.close(DFSClient.java:2326)
at java.io.FilterInputStream.close(FilterInputStream.java:181)
at org.apache.hadoop.util.LineReader.close(LineReader.java:145)
at org.apache.hadoop.mapreduce.lib.input.LineRecordReader.close(LineRecordReader.java:187)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.close(MapTask.java:496)
at org.apache.hadoop.mapred.MapTask.closeQuietly(MapTask.java:1776)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:778)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-09-11 16:45:23,662 INFO org.apache.hadoop.mapred.MapTask: Starting flush of map output