You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by renguihe <re...@ebupt.com> on 2013/12/25 12:29:20 UTC
Hive job leading Hadoop datanodes to dead
Hi,
I encountered a problem when I tried to run a Hive(version 0.11.0) job using dynamic partition on Hadoop(version 1.0.4 and 1.2.1,both) ,the job failed and made some datanodes lost heartbeats and turned to be dead gradually.
The JDK version is 1.6.0_45/1.6.0_30/1.6.0_43 ,the OS version is Red Hat Enterprise Linux Server release 6.4 (Santiago) .
I tried the versions all of above,but it still can not work.
Here is the Hive script:
set hive.exec.dynamic.partition.mode=nonstrict;
FROM (SELECT *
FROM SRC_IREAD.TEMP_IREAD_LOGON
) A
INSERT OVERWRITE TABLE src_iread.src_iread_logon_soft_normal partition(record_day)
SELECT A.* ,
substr(recordtime,1,8)
INSERT OVERWRITE TABLE src_iread.src_iread_logon_soft partition(record_day)
SELECT A.* ,
substr(recordtime,1,8)
;
Here is the Java error info from the datanode logs:
2013-12-23 00:52:35,561 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(192.168.20.15:50010, storageID=DS-795828338-192.168.20.15-50010-1387540641089, infoPort=50075, ipcPort=50020):DataXceiver
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:198)
at sun.nio.ch.IOUtil.read(IOUtil.java:171)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:243)
at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:55)
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:155)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:128)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readToBuf(BlockReceiver.java:265)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.readNextPacket(BlockReceiver.java:312)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:376)
at org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:532)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:398)
at org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:107)
at java.lang.Thread.run(Thread.java:662)
Finally,there were tons of logs like this,until manually shutdown the datanode:
2013-12-23 01:08:06,167 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:07,167 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:08,167 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:09,168 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:10,168 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:11,168 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:12,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:13,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:14,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:15,169 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:16,170 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:17,170 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:18,170 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:19,171 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:20,171 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:21,171 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:22,172 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:23,172 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
2013-12-23 01:08:24,172 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Waiting for threadgroup to exit, active threads is 1
PS:table DDL:
CREATE TABLE IF NOT EXISTS TEMP_IREAD_LOGON (
msisdn string COMMENT '$',
recordtime string COMMENT '$',
terminal string COMMENT '$',
platform string COMMENT '$',
channelcode string COMMENT '$',
wapip string COMMENT '$',
sessionid string COMMENT '$',
provinceid string COMMENT '$',
cityid string COMMENT '$',
clientchannelcode string COMMENT '$',
access_points string COMMENT '$',
logintype string COMMENT '$',
visittype string COMMENT '$',
sourcetype string COMMENT '$',
portalip string COMMENT '$' )
COMMENT 'TMP_IREAD_LOGON INFORMATION OF SRC_IREAD'
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE
CREATE TABLE IF NOT EXISTS SRC_IREAD_LOGON_SOFT_NORMAL (
msisdn string COMMENT '',
recordtime string COMMENT '',
terminal string COMMENT '$',
platform int COMMENT '',
channelcode string COMMENT '',
wapip string COMMENT '',
sessionid string COMMENT '',
provinceid string COMMENT '',
cityid string COMMENT '',
clientchannelcode string COMMENT '',
access_points int COMMENT '1?WLAN 2?CMWAP3?CMNET4?UNKNOWN',
logintype int COMMENT '???? 1 ??CMWAP???? 2 ?????????? 3 ???????????? 4 ????',
visittype int COMMENT '1? ???2? ?????????WAP??????????????',
sourcetype int COMMENT,
portalip string COMMENT '' )
COMMENT 'ENTERPRISE INFORMATION OF ODS_IREAD'
PARTITIONED BY (record_day string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
LINES TERMINATED BY '\n'
STORED AS TEXTFILE
CREATE TABLE IF NOT EXISTS SRC_IREAD_LOGON_SOFT (
msisdn string COMMENT '',
recordtime string COMMENT '',
terminal string COMMENT '$',
platform int COMMENT '',
channelcode string COMMENT '',
wapip string COMMENT '',
sessionid string COMMENT '',
provinceid string COMMENT '',
cityid string COMMENT '',
clientchannelcode string COMMENT '',
access_points int COMMENT '1?WLAN 2?CMWAP3?CMNET4?UNKNOWN',
logintype int COMMENT '???? 1 ??CMWAP???? 2 ?????????? 3 ???????????? 4 ????',
visittype int COMMENT '1? ???2? ?????????WAP??????????????',
sourcetype int ,
portalip string COMMENT '' )
COMMENT 'ENTERPRISE INFORMATION OF SRC_IREAD_LOGON_SOFT'
PARTITIONED BY (record_day string)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY '|'
STORED AS TEXTFILE
could anyone help me out of this problem?
THX!
renguihe