You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Iván de Prado <iv...@datasalt.com> on 2013/09/05 11:40:38 UTC
Infinite loop with ORC file and Hive 0.11
Hi,
We are using Hive 0.11 with ORC file format and we get some tasks blocked
in some kind of infinite loop. They keep working indefinitely when we set a
huge task expiry timeout. If we the expiry time to 600 second, the taks
fail because of not reporting progress, and finally, the Job fails.
That is not consistent, and some times between jobs executions the behavior
changes. It happen for different queries.
We are using Hive 0.11 with Hadoop hadoop-1.1.2.23 from Hortonworks. The
taks that is blocked keeps consuming 100% of CPU usage, and the stack trace
is always the same consistently. Everything points to some kind of infinite
loop. My guessing is that it has some relation to the ORC file. Maybe some
pointer is not right when writing generating some kind of infinite loop
when reading. Or maybe there is a bug in the reading stage. Any ideas?
Should I put that as a bug in Jira?
More information below. The stack trace:
------
"main" prio=10 tid=0x00007f20a000a800 nid=0x1ed2 runnable
[0x00007f20a8136000]
java.lang.Thread.State: RUNNABLE
at java.util.zip.Inflater.inflateBytes(Native Method)
at java.util.zip.Inflater.inflate(Inflater.java:256)
- locked <0x00000000f42a6ca0> (a java.util.zip.ZStreamRef)
at org.apache.hadoop.hive.ql.io.orc.ZlibCodec.decompress(ZlibCodec.java:64)
at
org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.readHeader(InStream.java:128)
at
org.apache.hadoop.hive.ql.io.orc.InStream$CompressedStream.read(InStream.java:143)
at
org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVulong(SerializationUtils.java:54)
at
org.apache.hadoop.hive.ql.io.orc.SerializationUtils.readVslong(SerializationUtils.java:65)
at
org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.readValues(RunLengthIntegerReader.java:66)
at
org.apache.hadoop.hive.ql.io.orc.RunLengthIntegerReader.next(RunLengthIntegerReader.java:81)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$IntTreeReader.next(RecordReaderImpl.java:332)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$StructTreeReader.next(RecordReaderImpl.java:802)
at
org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.next(RecordReaderImpl.java:1214)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:71)
at
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.next(OrcInputFormat.java:46)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:274)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:101)
at
org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
at
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:108)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:300)
at
org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:218)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:236)
- eliminated <0x00000000e1459700> (a
org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
at
org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:216)
- locked <0x00000000e1459700> (a
org.apache.hadoop.mapred.MapTask$TrackedRecordReader)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1178)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
--------
We have seen the same stack trace repeatedly for several executions of
jstack.
The log file for this kind of task is the following:
-------
2013-09-04 23:12:34,332 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2013-09-04 23:12:34,681 INFO org.apache.hadoop.mapred.TaskRunner: Creating
symlink:
/hd/hd6/mapred/local/taskTracker/distcache/1758137359311570022_-1138434677_1812098585/master/tmp/hive-datasalt/hive_2013-09-04_22-45-13_829_2202639092470021957/-mr-10004/22027cf8-f583-41d7-adb8-e7e74922d113
<-
/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/HIVE_PLAN22027cf8-f583-41d7-adb8-e7e74922d113
2013-09-04 23:12:34,718 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
symlink:
/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/com
<-
/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/com
2013-09-04 23:12:34,733 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
symlink:
/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javolution
<-
/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javolution
2013-09-04 23:12:34,743 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
symlink:
/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/org
<-
/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/org
2013-09-04 23:12:34,756 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
symlink:
/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/job.jar
<-
/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/job.jar
2013-09-04 23:12:34,768 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
symlink:
/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javax
<-
/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javax
2013-09-04 23:12:34,779 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
symlink:
/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/hive-exec-log4j.properties
<-
/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/hive-exec-log4j.properties
2013-09-04 23:12:34,790 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
symlink:
/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/META-INF
<-
/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/META-INF
2013-09-04 23:12:34,796 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
symlink:
/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/javaewah
<-
/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/javaewah
2013-09-04 23:12:34,813 INFO
org.apache.hadoop.filecache.TrackerDistributedCacheManager: Creating
symlink:
/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/.job.jar.crc
<-
/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/.job.jar.crc
2013-09-04 23:12:35,127 WARN
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already
exists!
2013-09-04 23:12:35,412 INFO org.apache.hadoop.util.ProcessTree: setsid
exited with exit code 0
2013-09-04 23:12:35,473 INFO org.apache.hadoop.mapred.Task: Using
ResourceCalculatorPlugin :
org.apache.hadoop.util.LinuxResourceCalculatorPlugin@5d0fa836
2013-09-04 23:12:36,927 INFO
org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file
hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28/000798_0
2013-09-04 23:12:36,927 INFO org.apache.hadoop.mapred.MapTask:
numReduceTasks: 72
2013-09-04 23:12:36,936 INFO org.apache.hadoop.mapred.MapTask: io.sort.mb =
100
2013-09-04 23:12:37,028 INFO org.apache.hadoop.mapred.MapTask: data buffer
= 79691776/99614720
2013-09-04 23:12:37,028 INFO org.apache.hadoop.mapred.MapTask: record
buffer = 262144/327680
2013-09-04 23:12:37,036 WARN
org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library is
available
2013-09-04 23:12:37,036 INFO
org.apache.hadoop.io.compress.snappy.LoadSnappy: Snappy native library
loaded
2013-09-04 23:12:37,046 INFO ExecMapper: maximum memory = 559284224
2013-09-04 23:12:37,046 INFO ExecMapper: conf classpath =
[file:/home/datasalt/sw/hadoop-1.1.2.23/conf/,
file:/opt/oracle-jdk-bin-1.7.0.17/lib/tools.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/,
file:/home/datasalt/sw/hadoop-1.1.2.23/hadoop-core-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/asm-3.2.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjrt-1.6.11.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjtools-1.6.11.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-1.7.0.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-core-1.8.0.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-cli-1.2.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-codec-1.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-collections-3.2.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-configuration-1.6.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-daemon-1.0.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-digester-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-el-1.0.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-httpclient-3.0.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-io-2.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-lang-2.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-1.1.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-api-1.0.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-math-2.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-net-3.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/core-3.1.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-capacity-scheduler-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-fairscheduler-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-thriftfs-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hsqldb-1.8.0.10.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-core-asl-1.8.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-mapper-asl-1.8.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-compiler-5.5.12.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-runtime-5.5.12.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jdeb-0.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-core-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-json-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-server-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jets3t-0.6.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-6.1.26.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-util-6.1.26.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsch-0.1.42.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/junit-4.5.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/kfs-0.2.2.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/log4j-1.2.15.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/mockito-all-1.8.5.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/oro-2.0.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/servlet-api-2.5-20081211.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-api-1.4.3.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-log4j12-1.4.3.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/xmlenc-0.52.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-2.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-api-2.1.jar,
file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/classes,
file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/,
file:/hd/hd1/mapred/local/taskTracker/datasalt/distcache/-1837517190973317952_-652944854_1812099894/master/tmp/hadoop-datasalt/mapred/staging/datasalt/.staging/job_201309040511_0003/libjars/derby-10.4.2.0.jar/,
file:/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/]
2013-09-04 23:12:37,047 INFO ExecMapper: thread classpath =
[file:/home/datasalt/sw/hadoop-1.1.2.23/conf/,
file:/opt/oracle-jdk-bin-1.7.0.17/lib/tools.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/,
file:/home/datasalt/sw/hadoop-1.1.2.23/hadoop-core-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/asm-3.2.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjrt-1.6.11.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/aspectjtools-1.6.11.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-1.7.0.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-beanutils-core-1.8.0.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-cli-1.2.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-codec-1.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-collections-3.2.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-configuration-1.6.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-daemon-1.0.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-digester-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-el-1.0.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-httpclient-3.0.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-io-2.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-lang-2.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-1.1.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-logging-api-1.0.4.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-math-2.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/commons-net-3.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/core-3.1.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-capacity-scheduler-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-fairscheduler-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hadoop-thriftfs-1.1.2.23.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/hsqldb-1.8.0.10.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-core-asl-1.8.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jackson-mapper-asl-1.8.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-compiler-5.5.12.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jasper-runtime-5.5.12.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jdeb-0.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-core-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-json-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jersey-server-1.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jets3t-0.6.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-6.1.26.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jetty-util-6.1.26.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsch-0.1.42.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/junit-4.5.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/kfs-0.2.2.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/log4j-1.2.15.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/mockito-all-1.8.5.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/oro-2.0.8.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/servlet-api-2.5-20081211.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-api-1.4.3.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/slf4j-log4j12-1.4.3.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/xmlenc-0.52.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-2.1.jar,
file:/home/datasalt/sw/hadoop-1.1.2.23/lib/jsp-2.1/jsp-api-2.1.jar,
file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/classes,
file:/hd/hd5/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/jars/,
file:/hd/hd1/mapred/local/taskTracker/datasalt/distcache/-1837517190973317952_-652944854_1812099894/master/tmp/hadoop-datasalt/mapred/staging/datasalt/.staging/job_201309040511_0003/libjars/derby-10.4.2.0.jar/,
file:/hd/hd1/mapred/local/taskTracker/datasalt/jobcache/job_201309040511_0003/attempt_201309040511_0003_m_000803_1/work/]
2013-09-04 23:12:37,085 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Adding alias hits to work list for file
hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28
2013-09-04 23:12:37,089 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
dump TS
struct<hour:int,user_ip:string,country_iso:string,user_language:string,num_hits:int,site_hostname:string,idsite:int,num_ads:int,num_ads_wanted:int,category:int,idcampaign:int,idvariation:int,sub_id:int,cpm_value:double,publisher_share:int,publisher_currency:string,advertiser_currency:string,idzone:int,idagent:int,idadvertiser:int,idpublisher:int,ddate:string>
2013-09-04 23:12:37,091 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Adding alias hits to work list for file
hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-29
2013-09-04 23:12:37,092 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Adding alias hits to work list for file
hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-30
2013-09-04 23:12:37,093 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Adding alias hits to work list for file
hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-31
2013-09-04 23:12:37,094 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Adding alias hits to work list for file
hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-01
2013-09-04 23:12:37,094 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Adding alias hits to work list for file
hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-02
2013-09-04 23:12:37,095 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Adding alias hits to work list for file
hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-09-03
2013-09-04 23:12:37,096 INFO ExecMapper:
<MAP>Id =7
<Children>
<TS>Id =3
<Children>
<SEL>Id =2
<Children>
<GBY>Id =1
<Children>
<RS>Id =0
<Parent>Id = 1 null<\Parent>
<\RS>
<\Children>
<Parent>Id = 2 null<\Parent>
<\GBY>
<\Children>
<Parent>Id = 3 null<\Parent>
<\SEL>
<\Children>
<Parent>Id = 7 null<\Parent>
<\TS>
<\Children>
<\MAP>
2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Initializing Self 7 MAP
2013-09-04 23:12:37,100 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing Self 3 TS
2013-09-04 23:12:37,100 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: Operator 3 TS initialized
2013-09-04 23:12:37,100 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: Initializing children of
3 TS
2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initializing child 2 SEL
2013-09-04 23:12:37,100 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initializing Self 2 SEL
2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
SELECT
struct<hour:int,user_ip:string,country_iso:string,user_language:string,num_hits:int,site_hostname:string,idsite:int,num_ads:int,num_ads_wanted:int,category:int,idcampaign:int,idvariation:int,sub_id:int,cpm_value:double,publisher_share:int,publisher_currency:string,advertiser_currency:string,idzone:int,idagent:int,idadvertiser:int,idpublisher:int,ddate:string>
2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Operator 2 SEL initialized
2013-09-04 23:12:37,105 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initializing children of 2 SEL
2013-09-04 23:12:37,105 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: Initializing child 1 GBY
2013-09-04 23:12:37,105 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: Initializing Self 1 GBY
2013-09-04 23:12:37,116 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: Operator 1 GBY initialized
2013-09-04 23:12:37,116 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: Initializing children of 1
GBY
2013-09-04 23:12:37,116 INFO
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing child 0 RS
2013-09-04 23:12:37,116 INFO
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initializing Self 0 RS
2013-09-04 23:12:37,116 INFO
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Using tag = -1
2013-09-04 23:12:37,139 INFO
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Operator 0 RS initialized
2013-09-04 23:12:37,139 INFO
org.apache.hadoop.hive.ql.exec.ReduceSinkOperator: Initialization Done 0 RS
2013-09-04 23:12:37,139 INFO
org.apache.hadoop.hive.ql.exec.GroupByOperator: Initialization Done 1 GBY
2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
Initialization Done 2 SEL
2013-09-04 23:12:37,139 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: Initialization Done 3 TS
2013-09-04 23:12:37,139 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Initialization Done 7 MAP
2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.MapOperator:
Processing alias hits for file
hdfs://master:54310/user/hive/warehouse/hits/ddate=2013-08-28
2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
forwarding 1 rows
2013-09-04 23:12:37,165 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1 rows
2013-09-04 23:12:37,165 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
2 forwarding 1 rows
2013-09-04 23:12:37,168 INFO ExecMapper: ExecMapper: processing 1 rows:
used memory = 151151936
2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
forwarding 10 rows
2013-09-04 23:12:37,169 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 10 rows
2013-09-04 23:12:37,169 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
2 forwarding 10 rows
2013-09-04 23:12:37,169 INFO ExecMapper: ExecMapper: processing 10 rows:
used memory = 151151936
2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
forwarding 100 rows
2013-09-04 23:12:37,175 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 100 rows
2013-09-04 23:12:37,175 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
2 forwarding 100 rows
2013-09-04 23:12:37,175 INFO ExecMapper: ExecMapper: processing 100 rows:
used memory = 151151936
2013-09-04 23:12:37,247 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
forwarding 1000 rows
2013-09-04 23:12:37,250 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1000 rows
2013-09-04 23:12:37,250 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
2 forwarding 1000 rows
2013-09-04 23:12:37,251 INFO ExecMapper: ExecMapper: processing 1000 rows:
used memory = 151151936
2013-09-04 23:12:37,642 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
forwarding 10000 rows
2013-09-04 23:12:37,642 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 10000 rows
2013-09-04 23:12:37,644 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
2 forwarding 10000 rows
2013-09-04 23:12:37,644 INFO ExecMapper: ExecMapper: processing 10000 rows:
used memory = 153032176
2013-09-04 23:12:38,467 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
forwarding 100000 rows
2013-09-04 23:12:38,468 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 100000 rows
2013-09-04 23:12:38,468 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
2 forwarding 100000 rows
2013-09-04 23:12:38,469 INFO ExecMapper: ExecMapper: processing 100000
rows: used memory = 125346272
2013-09-04 23:12:40,626 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 7
forwarding 1000000 rows
2013-09-04 23:12:40,627 INFO
org.apache.hadoop.hive.ql.exec.TableScanOperator: 3 forwarding 1000000 rows
2013-09-04 23:12:40,627 INFO org.apache.hadoop.hive.ql.exec.SelectOperator:
2 forwarding 1000000 rows
2013-09-04 23:12:40,627 INFO ExecMapper: ExecMapper: processing 1000000
rows: used memory = 123383952
-----------
For this particular example, the task was killed 6 hours after the last log
line.
The strace of the process when it is in the infinite loop has the following
looking:
-----------
Process 7887 attached with 24 threads
[pid 7930] futex(0x7f20a06df28c, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
...>
[pid 7922] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
[pid 7920] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
[pid 8333] accept(61, <unfinished ...>
[pid 7908] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
[pid 7907] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
[pid 7906] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
[pid 7905] futex(0x7f20a00fa554, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
...>
[pid 7903] futex(0x7f20a00f5254, FUTEX_WAIT_PRIVATE, 417, NULL <unfinished
...>
[pid 7904] futex(0x7f20a00f7c54, FUTEX_WAIT_PRIVATE, 387, NULL <unfinished
...>
[pid 7902] futex(0x7f20a7513dc0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished
...>
[pid 7901] futex(0x7f20a00a6f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid 7899] restart_syscall(<... resuming interrupted call ...> <unfinished
...>
[pid 7898] futex(0x7f20a0026554, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
...>
[pid 7900] futex(0x7f20a00a4b54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid 7897] futex(0x7f20a0024754, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
...>
[pid 7896] futex(0x7f20a0022a54, FUTEX_WAIT_PRIVATE, 41, NULL <unfinished
...>
[pid 7895] futex(0x7f20a0020c54, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
...>
[pid 7894] futex(0x7f20a001ef54, FUTEX_WAIT_PRIVATE, 37, NULL <unfinished
...>
[pid 7893] futex(0x7f20a001d154, FUTEX_WAIT_PRIVATE, 39, NULL <unfinished
...>
[pid 7887] futex(0x7f20a81389d0, FUTEX_WAIT, 7890, NULL <unfinished ...>
[pid 7892] futex(0x7f20a001b454, FUTEX_WAIT_PRIVATE, 35, NULL <unfinished
...>
[pid 7891] futex(0x7f20a0019654, FUTEX_WAIT_PRIVATE, 37, NULL <unfinished
...>
[pid 7906] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373836, 993325000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 43598000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 93858000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 144101000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 194344000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 244659000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 294924000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 345198000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 395462000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 445768000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 495983000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 546226000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 596489000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 646798000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 697063000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 747370000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 797688000},
ffffffff <unfinished ...>
[pid 7899] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7899] futex(0x7f20a009d054,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 782077000},
ffffffff <unfinished ...>
[pid 7906] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 848014000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 898283000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 948489000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373837, 998751000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 49086000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 99351000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 149589000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 199852000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 250180000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 300489000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 350762000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 401078000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 451258000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 501527000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 551767000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 601970000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 652161000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 702290000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 752523000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 802739000},
ffffffff <unfinished ...>
[pid 7899] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
out)
[pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7899] futex(0x7f20a009d054,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 782336000},
ffffffff <unfinished ...>
[pid 7922] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 7922] futex(0x7f20a06d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7922] futex(0x7f20a0550554, FUTEX_WAKE_OP_PRIVATE, 1, 1,
0x7f20a0550550, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
[pid 7907] <... restart_syscall resumed> ) = 0
[pid 7907] futex(0x7f20a0550528, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7922] write(56,
"\0\0\0\335\0\0\2.\0\4ping\0\0\0\2\0&org.apache.h"..., 225 <unfinished ...>
[pid 7907] read(56, <unfinished ...>
[pid 7922] <... write resumed> ) = 225
[pid 7907] <... read resumed> 0x7f2098000b70, 8192) = -1 EAGAIN (Resource
temporarily unavailable)
[pid 7922] futex(0x7f20a009d054, FUTEX_WAKE_OP_PRIVATE, 1, 1,
0x7f20a009d050, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
[pid 7899] <... futex resumed> ) = 0
[pid 7922] futex(0x7f20a06d3e54, FUTEX_WAIT_PRIVATE, 53, NULL <unfinished
...>
[pid 7907] epoll_ctl(59, EPOLL_CTL_ADD, 56, {...} <unfinished ...>
[pid 7899] futex(0x7f20a009d028, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 7907] <... epoll_ctl resumed> ) = 0
[pid 7899] <... futex resumed> ) = 0
[pid 7907] epoll_wait(59, {?} 0x7f20a0544000, 4096, 60000) = 1
[pid 7899] mprotect(0x7f20a8147000, 4096, PROT_READ <unfinished ...>
[pid 7907] futex(0x7f20a0551e54, FUTEX_WAIT_PRIVATE, 11, NULL <unfinished
...>
[pid 7899] <... mprotect resumed> ) = 0
[pid 7890] futex(0x7f20a000b854, FUTEX_WAIT_PRIVATE, 349, NULL <unfinished
...>
[pid 7899] mprotect(0x7f20a8147000, 4096, PROT_READ|PROT_WRITE) = 0
[pid 7899] mprotect(0x7f20a8148000, 4096, PROT_NONE) = 0
[pid 7899] mprotect(0x7f20a8148000, 4096, PROT_READ) = 0
[pid 7899] futex(0x7f20a0551e54, FUTEX_WAKE_OP_PRIVATE, 1, 1,
0x7f20a0551e50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
[pid 7899] futex(0x7f20a06d3e54, FUTEX_WAKE_OP_PRIVATE, 1, 1,
0x7f20a06d3e50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1} <unfinished ...>
[pid 7922] <... futex resumed> ) = 0
[pid 7907] <... futex resumed> ) = 0
[pid 7922] futex(0x7f20a06d3e28, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished
...>
[pid 7899] <... futex resumed> ) = 1
[pid 7907] futex(0x7f20a0551e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 7899] futex(0x7f20a06d3e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 7922] <... futex resumed> ) = 0
[pid 7907] <... futex resumed> ) = 0
[pid 7922] futex(0x7f20a06d3e28, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 7899] <... futex resumed> ) = 1
[pid 7922] <... futex resumed> ) = 0
[pid 7899] futex(0x7f20a009d054,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3, {1378373839, 794194000},
ffffffff <unfinished ...>
[pid 7922] futex(0x7f20a06d3a54, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
...>
[pid 7907] futex(0x7f20a000b854, FUTEX_WAKE_OP_PRIVATE, 1, 1,
0x7f20a000b850, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
[pid 7890] <... futex resumed> ) = 0
[pid 7890] futex(0x7f20a000b828, FUTEX_WAKE_PRIVATE, 1 <unfinished ...>
[pid 7907] epoll_ctl(59, EPOLL_CTL_DEL, 56, {...} <unfinished ...>
[pid 7890] <... futex resumed> ) = 0
[pid 7907] <... epoll_ctl resumed> ) = 0
[pid 7907] epoll_wait(59, {}, 4096, 0) = 0
[pid 7907] read(56, "\0\0\2.\0\0\0\0\0\7boolean\1", 8192) = 18
[pid 7907] futex(0x7f20a06d3a54, FUTEX_WAKE_OP_PRIVATE, 1, 1,
0x7f20a06d3a50, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
[pid 7922] <... futex resumed> ) = 0
[pid 7907] futex(0x7f20a0550554,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1107, {1378373848,
793674000}, ffffffff <unfinished ...>
[pid 7922] futex(0x7f20a06d3a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7922] futex(0x7f20a06d3a54,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3, {1378373841, 794758000},
ffffffff <unfinished ...>
[pid 7906] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 852867000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 903052000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373838, 953226000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 3436000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 53695000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 103966000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 154232000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 204466000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 254802000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 305067000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 355318000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 405623000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 455817000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 506130000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 556395000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 606661000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 656898000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
[pid 7906] futex(0x7f20a0104928, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 7906] futex(0x7f20a0104954,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1378373839, 707177000},
ffffffff) = -1 ETIMEDOUT (Connection timed out)
Regards,
--
Iván de Prado
CEO & Co-founder
www.datasalt.com