You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/10/29 06:45:56 UTC

[GitHub] [incubator-druid] liguifa opened a new issue #8772: The index Kafka task fails without any exception

liguifa opened a new issue #8772: The index Kafka task fails without any exception
URL: https://github.com/apache/incubator-druid/issues/8772
 
 
   The index Kafka task fails without any exception,Overlord log display 'Shutdown [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] because: [notified status change from task]'
   
   ### Affected Version
   0.15.0
   
   ### Description
   overlord log (id:index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd):
   
   > 2019-10-29T05:50:49,276 INFO [KafkaSupervisor-bi_imp_ad] org.apache.druid.indexing.overlord.MetadataTaskStorage - Inserting task index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd with status: TaskStatus{id=index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd, status=RUNNING, duration=-1, errorMsg=null}
   2019-10-29T05:50:49,285 INFO [KafkaSupervisor-bi_imp_ad] org.apache.druid.indexing.overlord.TaskLockbox - Adding task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] to activeTasks
   2019-10-29T05:50:49,285 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskQueue - Asking taskRunner to run: index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T05:50:49,285 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.RemoteTaskRunner - Added pending task index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T05:50:49,286 INFO [rtr-pending-tasks-runner-0] org.apache.druid.indexing.overlord.RemoteTaskRunner - Coordinator asking Worker[192.168.33.120:8091] to add task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd]
   2019-10-29T05:50:49,287 INFO [rtr-pending-tasks-runner-0] org.apache.druid.indexing.overlord.RemoteTaskRunner - Task index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd switched from pending to running (on [192.168.33.120:8091])
   2019-10-29T05:50:49,287 INFO [rtr-pending-tasks-runner-0] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] status changed to [RUNNING].
   2019-10-29T05:50:49,297 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[192.168.33.120:8091] wrote RUNNING status for task [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] on [TaskLocation{host='null', port=-1, tlsPort=-1}]
   2019-10-29T05:50:49,300 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[192.168.33.120:8091] wrote RUNNING status for task [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] on [TaskLocation{host='192.168.33.120', port=8106, tlsPort=-1}]
   2019-10-29T05:50:49,300 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] location changed to [TaskLocation{host='192.168.33.120', port=8106, tlsPort=-1}].
   2019-10-29T05:50:59,549 INFO [qtp1493789542-149] org.apache.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd]: SegmentAllocateAction{dataSource='bi_imp_ad', timestamp=2019-10-29T13:49:57.000Z, queryGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName='index_kafka_bi_imp_ad_7130098998f04ba_0', previousSegmentId='null', skipSegmentLineageCheck='true'}
   2019-10-29T05:51:00,061 INFO [qtp1493789542-149] org.apache.druid.indexing.overlord.TaskLockbox - Added task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] to TaskLock[index_kafka_bi_imp_ad]
   2019-10-29T05:51:00,061 INFO [qtp1493789542-149] org.apache.druid.indexing.overlord.MetadataTaskStorage - Adding lock on interval[2019-10-29T13:00:00.000Z/2019-10-29T14:00:00.000Z] version[2019-10-29T05:00:09.490Z] for task: index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T05:51:00,727 INFO [KafkaSupervisor-bi_imp_ad] org.apache.druid.indexing.seekablestream.supervisor.SeekableStreamSupervisor - {id='bi_imp_ad', generationTime=2019-10-29T05:51:00.727Z, payload=KafkaSupervisorReportPayload{dataSource='bi_imp_ad', topic='imp_grail', partitions=56, replicas=1, durationSeconds=3600, active=[{id='index_kafka_bi_imp_ad_3015cfeb8cbc386_ocoedlkc', startTime=2019-10-29T05:43:10.829Z, remainingSeconds=3130}, {id='index_kafka_bi_imp_ad_dacce03a98d6c54_keakkgmb', startTime=2019-10-29T05:43:28.833Z, remainingSeconds=3148}, {id='index_kafka_bi_imp_ad_ddff236726ee3c5_ojgcejjj', startTime=2019-10-29T05:43:28.558Z, remainingSeconds=3147}, {id='index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd', startTime=2019-10-29T05:50:59.957Z, remainingSeconds=3599}, {id='index_kafka_bi_imp_ad_7f55fc72ce1e23f_iiiofkmp', startTime=2019-10-29T05:43:10.105Z, remainingSeconds=3129}, {id='index_kafka_bi_imp_ad_dcfdd09dcbda25d_cmnpmacc', startTime=2019-10-29T05:43:29.209Z, remainingSeconds=3148}, {id='index_kafka_bi_imp_ad_f36bba4878f4769_ecckalkc', startTime=2019-10-29T05:43:28.630Z, remainingSeconds=3147}, {id='index_kafka_bi_imp_ad_a3d1ec6814e8946_jfmmgmjp', startTime=2019-10-29T05:43:29.130Z, remainingSeconds=3148}, {id='index_kafka_bi_imp_ad_2a66c4cb2dd3d1a_fpeclode', startTime=2019-10-29T05:43:10.786Z, remainingSeconds=3130}, {id='index_kafka_bi_imp_ad_817eec139354341_jmdmbdil', startTime=2019-10-29T05:43:28.704Z, remainingSeconds=3147}, {id='index_kafka_bi_imp_ad_bdba473f74a77c1_pjopjfbm', startTime=2019-10-29T05:43:28.725Z, remainingSeconds=3147}], publishing=[{id='index_kafka_bi_imp_ad_10980e7cdaf8f44_gbnkbaai', startTime=2019-10-29T04:42:59.263Z, remainingSeconds=1319}, {id='index_kafka_bi_imp_ad_275af480d9793d9_legcncch', startTime=2019-10-29T04:43:00.699Z, remainingSeconds=1335}, {id='index_kafka_bi_imp_ad_b5013e48d5bab6f_fmocbjbo', startTime=2019-10-29T04:43:01.176Z, remainingSeconds=1335}, {id='index_kafka_bi_imp_ad_06aad2175a5ca00_jgmlcnog', startTime=2019-10-29T04:50:04.605Z, remainingSeconds=1745}, {id='index_kafka_bi_imp_ad_a863afb3b8c8cb7_kgnfbnha', startTime=2019-10-29T04:43:00.846Z, remainingSeconds=1335}, {id='index_kafka_bi_imp_ad_16554e2a06d0d20_pioefden', startTime=2019-10-29T04:43:01.412Z, remainingSeconds=1335}, {id='index_kafka_bi_imp_ad_311b330da0934e8_elpbfidf', startTime=2019-10-29T04:43:00.823Z, remainingSeconds=1335}, {id='index_kafka_bi_imp_ad_8452cb360b4d29a_aifpnlal', startTime=2019-10-29T04:43:01.733Z, remainingSeconds=1335}], suspended=false, healthy=true, state=RUNNING, detailedState=RUNNING, recentErrors=[]}}
   2019-10-29T05:51:00,916 INFO [qtp1493789542-128] org.apache.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd]: SegmentAllocateAction{dataSource='bi_imp_ad', timestamp=2019-10-29T09:06:55.000Z, queryGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName='index_kafka_bi_imp_ad_7130098998f04ba_0', previousSegmentId='bi_imp_ad_2019-10-29T13:00:00.000Z_2019-10-29T14:00:00.000Z_2019-10-29T05:00:09.490Z_21', skipSegmentLineageCheck='true'}
   2019-10-29T05:51:01,214 INFO [qtp1493789542-128] org.apache.druid.indexing.overlord.TaskLockbox - Added task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] to TaskLock[index_kafka_bi_imp_ad]
   2019-10-29T05:51:01,214 INFO [qtp1493789542-128] org.apache.druid.indexing.overlord.MetadataTaskStorage - Adding lock on interval[2019-10-29T09:00:00.000Z/2019-10-29T10:00:00.000Z] version[2019-10-29T02:46:35.586Z] for task: index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T05:51:02,279 INFO [qtp1493789542-177] org.apache.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd]: SegmentAllocateAction{dataSource='bi_imp_ad', timestamp=2019-10-29T12:46:54.000Z, queryGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, preferredSegmentGranularity={type=period, period=PT1H, timeZone=UTC, origin=null}, sequenceName='index_kafka_bi_imp_ad_7130098998f04ba_0', previousSegmentId='bi_imp_ad_2019-10-29T09:00:00.000Z_2019-10-29T10:00:00.000Z_2019-10-29T02:46:35.586Z_37', skipSegmentLineageCheck='true'}
   2019-10-29T05:51:02,613 INFO [qtp1493789542-177] org.apache.druid.indexing.overlord.TaskLockbox - Added task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] to TaskLock[index_kafka_bi_imp_ad]
   2019-10-29T05:51:02,613 INFO [qtp1493789542-177] org.apache.druid.indexing.overlord.MetadataTaskStorage - Adding lock on interval[2019-10-29T12:00:00.000Z/2019-10-29T13:00:00.000Z] version[2019-10-29T04:00:09.473Z] for task: index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T05:51:04,474 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[192.168.33.120:8091] wrote FAILED status for task [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] on [TaskLocation{host='192.168.33.120', port=8106, tlsPort=-1}]
   2019-10-29T05:51:04,474 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Worker[192.168.33.120:8091] completed task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] with status[FAILED]
   2019-10-29T05:51:04,474 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Received FAILED status for task: index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T05:51:04,474 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Shutdown [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] because: [notified status change from task]
   2019-10-29T05:51:04,474 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Cleaning up task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] on worker[192.168.33.120:8091]
   2019-10-29T05:51:04,475 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] from activeTasks
   2019-10-29T05:51:04,475 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] from TaskLock[index_kafka_bi_imp_ad]
   2019-10-29T05:51:04,485 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] from TaskLock[index_kafka_bi_imp_ad]
   2019-10-29T05:51:04,494 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskLockbox - Removing task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] from TaskLock[index_kafka_bi_imp_ad]
   2019-10-29T05:51:04,508 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.MetadataTaskStorage - Updating task index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd to status: TaskStatus{id=index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd, status=FAILED, duration=15165, errorMsg=null}
   2019-10-29T05:51:04,514 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Task done: AbstractTask{id='index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd', groupId='index_kafka_bi_imp_ad', taskResource=TaskResource{availabilityGroup='index_kafka_bi_imp_ad_7130098998f04ba', requiredCapacity=1}, dataSource='bi_imp_ad', context={checkpoints={"0":{"3":5947398609,"36":209572764,"25":209572765,"14":209572760,"47":209572760}}, IS_INCREMENTAL_HANDOFF_SUPPORTED=true, forceTimeChunkLock=false, priority=100}}
   2019-10-29T05:51:04,514 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskQueue - Task FAILED: AbstractTask{id='index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd', groupId='index_kafka_bi_imp_ad', taskResource=TaskResource{availabilityGroup='index_kafka_bi_imp_ad_7130098998f04ba', requiredCapacity=1}, dataSource='bi_imp_ad', context={checkpoints={"0":{"3":5947398609,"36":209572764,"25":209572765,"14":209572760,"47":209572760}}, IS_INCREMENTAL_HANDOFF_SUPPORTED=true, forceTimeChunkLock=false, priority=100}} (15165 run duration)
   2019-10-29T05:51:04,514 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] status changed to [FAILED].
   2019-10-29T05:51:04,514 INFO [Curator-PathChildrenCache-1] org.apache.druid.indexing.overlord.RemoteTaskRunner - Task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] went bye bye.
   2019-10-29T06:27:53,157 INFO [qtp1493789542-117] org.apache.druid.storage.hdfs.tasklog.HdfsTaskLogs - Reading task log from: /common_druid/indexing-logs/index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T06:27:53,163 INFO [qtp1493789542-117] org.apache.druid.storage.hdfs.tasklog.HdfsTaskLogs - Read task log from: /common_druid/indexing-logs/index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd (seek = 400,689)
   
   > 
   
   middleManager log:
   > 2019-10-29T05:50:50,448 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] started.
   2019-10-29T05:50:50,448 INFO [forking-task-runner-2] org.apache.druid.indexing.overlord.ForkingTaskRunner - Running command: java -cp /home/iknow/apps/druid-0.16.0/conf/druid/cluster/_common:/home/iknow/apps/druid-0.16.0/conf/druid/cluster/middleManager:/home/iknow/apps/druid-0.16.0/lib/jackson-jq-0.0.7.jar:/home/iknow/apps/druid-0.16.0/lib/jetty-continuation-9.4.10.v20180503.jar:/home/iknow/apps/druid-0.16.0/lib/kylin-core-common-2.5.2.jar:/home/iknow/apps/druid-0.16.0/lib/netty-handler-proxy-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/httpclient-4.5.3.jar:/home/iknow/apps/druid-0.16.0/lib/protobuf-java-3.1.0.jar:/home/iknow/apps/druid-0.16.0/lib/zstd-jni-1.3.3-1.jar:/home/iknow/apps/druid-0.16.0/lib/httpcore-4.4.4.jar:/home/iknow/apps/druid-0.16.0/lib/calcite-core-1.17.0.jar:/home/iknow/apps/druid-0.16.0/lib/curator-recipes-4.1.0.jar:/home/iknow/apps/druid-0.16.0/lib/netty-codec-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-datatype-joda-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/jetty-client-9.4.10.v20180503.jar:/home/iknow/apps/druid-0.16.0/lib/commons-io-2.5.jar:/home/iknow/apps/druid-0.16.0/lib/aws-java-sdk-s3-1.11.199.jar:/home/iknow/apps/druid-0.16.0/lib/derby-10.11.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/jersey-guice-1.19.3.jar:/home/iknow/apps/druid-0.16.0/lib/aggdesigner-algorithm-6.0.jar:/home/iknow/apps/druid-0.16.0/lib/json-path-2.3.0.jar:/home/iknow/apps/druid-0.16.0/lib/commons-logging-1.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/commons-beanutils-1.9.3.jar:/home/iknow/apps/druid-0.16.0/lib/aws-java-sdk-core-1.11.199.jar:/home/iknow/apps/druid-0.16.0/lib/druid-gcp-common-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/jersey-servlet-1.19.3.jar:/home/iknow/apps/druid-0.16.0/lib/aether-impl-0.9.0.M2.jar:/home/iknow/apps/druid-0.16.0/lib/druid-services-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/slf4j-api-1.7.25.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-core-asl-1.9.13.jar:/home/iknow/apps/druid-0.16.0/lib/jsr311-api-1.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/commons-compiler-2.7.6.jar:/home/iknow/apps/druid-0.16.0/lib/mapdb-1.0.8.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-databind-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-datatype-guava-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/classmate-1.0.0.jar:/home/iknow/apps/druid-0.16.0/lib/commons-dbcp2-2.0.1.jar:/home/iknow/apps/druid-0.16.0/lib/jetty-proxy-9.4.10.v20180503.jar:/home/iknow/apps/druid-0.16.0/lib/asm-analysis-7.1.jar:/home/iknow/apps/druid-0.16.0/lib/xz-1.8.jar:/home/iknow/apps/druid-0.16.0/lib/maven-model-builder-3.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/jaxb-api-2.3.1.jar:/home/iknow/apps/druid-0.16.0/lib/aws-java-sdk-ec2-1.11.199.jar:/home/iknow/apps/druid-0.16.0/lib/log4j-jul-2.5.jar:/home/iknow/apps/druid-0.16.0/lib/maven-settings-builder-3.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/jersey-core-1.19.3.jar:/home/iknow/apps/druid-0.16.0/lib/ion-java-1.0.2.jar:/home/iknow/apps/druid-0.16.0/lib/zookeeper-3.4.11.jar:/home/iknow/apps/druid-0.16.0/lib/log4j-api-2.5.jar:/home/iknow/apps/druid-0.16.0/lib/jetty-util-9.4.10.v20180503.jar:/home/iknow/apps/druid-0.16.0/lib/druid-core-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/javax.inject-1.jar:/home/iknow/apps/druid-0.16.0/lib/javax.activation-1.2.0.jar:/home/iknow/apps/druid-0.16.0/lib/log4j-core-2.5.jar:/home/iknow/apps/druid-0.16.0/lib/netty-reactive-streams-2.0.0.jar:/home/iknow/apps/druid-0.16.0/lib/hibernate-validator-5.1.3.Final.jar:/home/iknow/apps/druid-0.16.0/lib/plexus-interpolation-1.19.jar:/home/iknow/apps/druid-0.16.0/lib/json-smart-2.3.jar:/home/iknow/apps/druid-0.16.0/lib/jmespath-java-1.11.199.jar:/home/iknow/apps/druid-0.16.0/lib/curator-client-4.1.0.jar:/home/iknow/apps/druid-0.16.0/lib/log4j-slf4j-impl-2.5.jar:/home/iknow/apps/druid-0.16.0/lib/janino-2.7.6.jar:/home/iknow/apps/druid-0.16.0/lib/guice-servlet-4.1.0.jar:/home/iknow/apps/druid-0.16.0/lib/google-http-client-jackson2-1.22.0.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-dataformat-smile-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/caffeine-2.5.5.jar:/home/iknow/apps/druid-0.16.0/lib/commons-math3-3.6.1.jar:/home/iknow/apps/druid-0.16.0/lib/maven-artifact-3.6.0.jar:/home/iknow/apps/druid-0.16.0/lib/netty-3.10.6.Final.jar:/home/iknow/apps/druid-0.16.0/lib/jboss-logging-3.1.3.GA.jar:/home/iknow/apps/druid-0.16.0/lib/commons-pool2-2.2.jar:/home/iknow/apps/druid-0.16.0/lib/maven-aether-provider-3.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/joni-2.1.11.jar:/home/iknow/apps/druid-0.16.0/lib/jetty-server-9.4.10.v20180503.jar:/home/iknow/apps/druid-0.16.0/lib/druid-console-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/javax.servlet-api-3.1.0.jar:/home/iknow/apps/druid-0.16.0/lib/druid-indexing-service-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/error_prone_annotations-2.3.2.jar:/home/iknow/apps/druid-0.16.0/lib/commons-collections4-4.1.jar:/home/iknow/apps/druid-0.16.0/lib/asm-7.1.jar:/home/iknow/apps/druid-0.16.0/lib/netty-resolver-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/log4j-1.2-api-2.5.jar:/home/iknow/apps/druid-0.16.0/lib/guice-4.1.0.jar:/home/iknow/apps/druid-0.16.0/lib/google-api-client-1.22.0.jar:/home/iknow/apps/druid-0.16.0/lib/aether-spi-0.9.0.M2.jar:/home/iknow/apps/druid-0.16.0/lib/avatica-metrics-1.12.0.jar:/home/iknow/apps/druid-0.16.0/lib/commons-cli-1.3.1.jar:/home/iknow/apps/druid-0.16.0/lib/antlr4-runtime-4.5.1.jar:/home/iknow/apps/druid-0.16.0/lib/avatica-core-1.12.0.jar:/home/iknow/apps/druid-0.16.0/lib/opencsv-4.2.jar:/home/iknow/apps/druid-0.16.0/lib/compress-lzf-1.0.4.jar:/home/iknow/apps/druid-0.16.0/lib/netty-transport-native-epoll-4.1.29.Final-linux-x86_64.jar:/home/iknow/apps/druid-0.16.0/lib/netty-buffer-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/javax.activation-api-1.2.0.jar:/home/iknow/apps/druid-0.16.0/lib/async-http-client-2.5.3.jar:/home/iknow/apps/druid-0.16.0/lib/netty-transport-native-unix-common-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/netty-transport-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/netty-codec-dns-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-mapper-asl-1.9.13.jar:/home/iknow/apps/druid-0.16.0/lib/validation-api-1.1.0.Final.jar:/home/iknow/apps/druid-0.16.0/lib/derbyclient-10.11.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/guice-multibindings-4.1.0.jar:/home/iknow/apps/druid-0.16.0/lib/maven-model-3.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/commons-text-1.3.jar:/home/iknow/apps/druid-0.16.0/lib/jna-4.5.1.jar:/home/iknow/apps/druid-0.16.0/lib/druid-server-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/jdbi-2.63.1.jar:/home/iknow/apps/druid-0.16.0/lib/commons-codec-1.12.jar:/home/iknow/apps/druid-0.16.0/lib/maven-settings-3.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/netty-all-4.1.30.Final.jar:/home/iknow/apps/druid-0.16.0/lib/google-http-client-1.22.0.jar:/home/iknow/apps/druid-0.16.0/lib/sigar-1.6.5.132.jar:/home/iknow/apps/druid-0.16.0/lib/icu4j-54.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/audience-annotations-0.5.0.jar:/home/iknow/apps/druid-0.16.0/lib/plexus-utils-3.0.15.jar:/home/iknow/apps/druid-0.16.0/lib/jsr305-2.0.1.jar:/home/iknow/apps/druid-0.16.0/lib/extendedset-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/jetty-servlets-9.4.10.v20180503.jar:/home/iknow/apps/druid-0.16.0/lib/netty-handler-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/kylin-core-metadata-2.5.2.jar:/home/iknow/apps/druid-0.16.0/lib/curator-x-discovery-4.1.0.jar:/home/iknow/apps/druid-0.16.0/lib/druid-indexing-hadoop-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/aether-util-0.9.0.M2.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-module-jaxb-annotations-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/shims-0.8.6.jar:/home/iknow/apps/druid-0.16.0/lib/guava-16.0.1.jar:/home/iknow/apps/druid-0.16.0/lib/netty-codec-socks-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-jaxrs-smile-provider-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/javax.el-api-3.0.0.jar:/home/iknow/apps/druid-0.16.0/lib/spymemcached-2.12.3.jar:/home/iknow/apps/druid-0.16.0/lib/jetty-servlet-9.4.10.v20180503.jar:/home/iknow/apps/druid-0.16.0/lib/druid-aws-common-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-core-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/maven-repository-metadata-3.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/rhino-1.7R5.jar:/home/iknow/apps/druid-0.16.0/lib/aether-api-0.9.0.M2.jar:/home/iknow/apps/druid-0.16.0/lib/aws-java-sdk-kms-1.11.199.jar:/home/iknow/apps/druid-0.16.0/lib/jline-0.9.94.jar:/home/iknow/apps/druid-0.16.0/lib/netty-resolver-dns-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/aopalliance-1.0.jar:/home/iknow/apps/druid-0.16.0/lib/jvm-attach-api-1.5.jar:/home/iknow/apps/druid-0.16.0/lib/druid-sql-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/jetty-http-9.4.10.v20180503.jar:/home/iknow/apps/druid-0.16.0/lib/jersey-server-1.19.3.jar:/home/iknow/apps/druid-0.16.0/lib/jcodings-1.0.13.jar:/home/iknow/apps/druid-0.16.0/lib/derbynet-10.11.1.1.jar:/home/iknow/apps/druid-0.16.0/lib/aether-connector-okhttp-0.0.9.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-module-guice-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-jaxrs-base-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/jetty-security-9.4.10.v20180503.jar:/home/iknow/apps/druid-0.16.0/lib/avatica-server-1.12.0.jar:/home/iknow/apps/druid-0.16.0/lib/commons-pool-1.6.jar:/home/iknow/apps/druid-0.16.0/lib/esri-geometry-api-2.0.0.jar:/home/iknow/apps/druid-0.16.0/lib/checker-qual-2.5.7.jar:/home/iknow/apps/druid-0.16.0/lib/kylin-core-dictionary-2.5.2.jar:/home/iknow/apps/druid-0.16.0/lib/aether-connector-file-0.9.0.M2.jar:/home/iknow/apps/druid-0.16.0/lib/commons-collections-3.2.2.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-annotations-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/druid-hll-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/airline-0.7.jar:/home/iknow/apps/druid-0.16.0/lib/commons-compress-1.18.jar:/home/iknow/apps/druid-0.16.0/lib/lz4-java-1.6.0.jar:/home/iknow/apps/druid-0.16.0/lib/disruptor-3.3.6.jar:/home/iknow/apps/druid-0.16.0/lib/reactive-streams-1.0.2.jar:/home/iknow/apps/druid-0.16.0/lib/druid-processing-0.16.0-incubating-SNAPSHOT.jar:/home/iknow/apps/druid-0.16.0/lib/jcl-over-slf4j-1.7.12.jar:/home/iknow/apps/druid-0.16.0/lib/RoaringBitmap-0.8.6.jar:/home/iknow/apps/druid-0.16.0/lib/commons-lang-2.6.jar:/home/iknow/apps/druid-0.16.0/lib/metrics-core-4.0.0.jar:/home/iknow/apps/druid-0.16.0/lib/asm-tree-7.1.jar:/home/iknow/apps/druid-0.16.0/lib/fastutil-8.1.0.jar:/home/iknow/apps/druid-0.16.0/lib/calcite-linq4j-1.17.0.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-dataformat-cbor-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/okhttp-1.0.2.jar:/home/iknow/apps/druid-0.16.0/lib/joda-time-2.10.2.jar:/home/iknow/apps/druid-0.16.0/lib/wagon-provider-api-2.4.jar:/home/iknow/apps/druid-0.16.0/lib/config-magic-0.9.jar:/home/iknow/apps/druid-0.16.0/lib/async-http-client-netty-utils-2.5.3.jar:/home/iknow/apps/druid-0.16.0/lib/commons-lang3-3.7.jar:/home/iknow/apps/druid-0.16.0/lib/javax.el-3.0.0.jar:/home/iknow/apps/druid-0.16.0/lib/jackson-jaxrs-json-provider-2.6.7.jar:/home/iknow/apps/druid-0.16.0/lib/accessors-smart-1.2.jar:/home/iknow/apps/druid-0.16.0/lib/jetty-io-9.4.10.v20180503.jar:/home/iknow/apps/druid-0.16.0/lib/curator-framework-4.1.0.jar:/home/iknow/apps/druid-0.16.0/lib/netty-common-4.1.29.Final.jar:/home/iknow/apps/druid-0.16.0/lib/google-oauth-client-1.22.0.jar:/home/iknow/apps/druid-0.16.0/lib/asm-commons-7.1.jar:/home/iknow/apps/druid-0.16.0/lib/tesla-aether-0.0.5.jar:/home/iknow/apps/druid-0.16.0/lib/netty-codec-http-4.1.29.Final.jar: -server -Xmx24g -XX:MaxDirectMemorySize=1024g -XX:+ExplicitGCInvokesConcurrent -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Ddruid.indexer.task.baseTaskDir=var/druid/task -Ddruid.host=192.168.33.120 -Ddruid.metadata.storage.connector.password=2ghlMcl_1hblSqt -Ddruid.indexer.fork.property.druid.processing.numThreads=3 -Ddruid.emitter.logging.logLevel=info -Ddruid.emitter=http -Ddruid.indexer.fork.property.druid.processing.buffer.sizeBytes=1073741824 -Ddruid.indexer.task.restoreTasksOnRestart=true -Duser.timezone=UTC -Ddruid.emitter.http.recipientBaseUrl=http://192.168.112.16:4311 -Dfile.encoding.pkg=sun.io -Ddruid.storage.storageDirectory=/common_druid/segments -Ddruid.selectors.coordinator.serviceName=druid/coordinator -Ddruid.selectors.indexing.serviceName=druid/overlord -Ddruid.indexing.doubleStorage=double -Ddruid.lookup.enableLookupSyncOnStartup=true -Ddruid.server.http.numThreads=60 -Ddruid.worker.capacity=10 -Ddruid.service=druid/middleManager -Ddruid.metadata.storage.connector.user=UserDataAnalyze -Ddruid.metadata.storage.type=mysql -Ddruid.metadata.storage.connector.connectURI=jdbc:mysql://192.168.0.152:3307/common_druid -Ddruid.plaintextPort=8091 -Djava.io.tmpdir=var/tmp -Ddruid.extensions.loadList=["druid-hdfs-storage", "mysql-metadata-storage", "druid-lookups-cached-global", "druid-kafka-indexing-service", "druid-datasketches", "druid-histogram", "druid-uniq"] -Ddruid.sql.enable=true -Ddruid.startup.logging.logProperties=true -Ddruid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"] -Ddruid.javascript.enabled=true -Ddruid.zk.service.host=192.168.32.62:2181,192.168.32.186:2181,192.168.33.143:2181,192.168.33.120:2181,192.168.33.121:2181 -Ddruid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"] -Ddruid.indexer.logs.directory=/common_druid/indexing-logs -Ddruid.zk.paths.base=/druid -Dfile.encoding=UTF-8 -Ddruid.storage.type=hdfs -Ddruid.indexer.task.hadoopWorkingPath=/druid/hadoop -Ddruid.indexer.fork.property.druid.processing.numMergeBuffers=2 -Ddruid.indexer.logs.type=hdfs -Ddruid.processing.numThreads=3 -Ddruid.processing.buffer.sizeBytes=1073741824 -Ddruid.processing.numMergeBuffers=2 -Ddruid.metrics.emitter.dimension.dataSource=bi_imp_ad -Ddruid.metrics.emitter.dimension.taskId=index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd -Ddruid.metrics.emitter.dimension.taskType=index_kafka -Ddruid.host=192.168.33.120 -Ddruid.plaintextPort=8106 -Ddruid.tlsPort=-1 org.apache.druid.cli.Main internal peon var/druid/task/index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd/task.json var/druid/task/index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd/c4407fd8-fa4c-42d8-a494-b68f9fbb1123/status.json var/druid/task/index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd/c4407fd8-fa4c-42d8-a494-b68f9fbb1123/report.json
   2019-10-29T05:50:50,450 INFO [forking-task-runner-2] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] location changed to [TaskLocation{host='192.168.33.120', port=8106, tlsPort=-1}].
   2019-10-29T05:50:50,450 INFO [forking-task-runner-2] org.apache.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] status changed to [RUNNING].
   2019-10-29T05:50:50,450 INFO [forking-task-runner-2] org.apache.druid.indexing.overlord.ForkingTaskRunner - Logging task index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd output to: var/druid/task/index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd/log
   2019-10-29T05:51:05,544 INFO [qtp1792015149-110] org.apache.druid.indexing.overlord.ForkingTaskRunner - Shutdown [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] because: [shut down request via HTTP endpoint]
   2019-10-29T05:51:05,544 INFO [qtp1792015149-110] org.apache.druid.indexing.overlord.ForkingTaskRunner - Killing process for task: index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T05:51:05,547 INFO [forking-task-runner-2] org.apache.druid.storage.hdfs.tasklog.HdfsTaskLogs - Writing task log to: /common_druid/indexing-logs/index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T05:51:05,607 INFO [forking-task-runner-2] org.apache.druid.storage.hdfs.tasklog.HdfsTaskLogs - Wrote task log to: /common_druid/indexing-logs/index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T05:51:05,608 INFO [forking-task-runner-2] org.apache.druid.indexing.overlord.ForkingTaskRunner - Removing task directory: var/druid/task/index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd
   2019-10-29T05:51:05,616 INFO [WorkerTaskManager-NoticeHandler] org.apache.druid.indexing.worker.WorkerTaskManager - Job's finished. Completed [index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] with status [FAILED]
   2019-10-29T05:55:58,244 INFO [WorkerTaskManager-CompletedTasksCleaner] org.apache.druid.indexing.worker.WorkerTaskManager - Deleting completed task[index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd] information, overlord task status[FAILED].
   
   > 
   
   index task log:
   
   > 2019-10-29T05:51:04,879 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed version.bin in 0 millis.
   2019-10-29T05:51:04,879 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed factory.json in 0 millis
   2019-10-29T05:51:04,880 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[customer] conversions with cardinality[13] in 1 millis.
   2019-10-29T05:51:04,880 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[grade] conversions with cardinality[13] in 0 millis.
   2019-10-29T05:51:04,880 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[courseType] conversions with cardinality[0] in 0 millis.
   2019-10-29T05:51:04,881 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[adpos] conversions with cardinality[60] in 1 millis.
   2019-10-29T05:51:04,881 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[os] conversions with cardinality[3] in 0 millis.
   2019-10-29T05:51:04,881 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[av] conversions with cardinality[52] in 0 millis.
   2019-10-29T05:51:04,882 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[campaign] conversions with cardinality[51] in 1 millis.
   2019-10-29T05:51:04,883 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[unit] conversions with cardinality[186] in 1 millis.
   2019-10-29T05:51:04,883 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[template] conversions with cardinality[31] in 0 millis.
   2019-10-29T05:51:04,884 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[province] conversions with cardinality[32] in 0 millis.
   2019-10-29T05:51:04,886 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[city] conversions with cardinality[295] in 2 millis.
   2019-10-29T05:51:04,886 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[conn] conversions with cardinality[5] in 0 millis.
   2019-10-29T05:51:04,886 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[role] conversions with cardinality[0] in 0 millis.
   2019-10-29T05:51:04,886 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[sex] conversions with cardinality[0] in 0 millis.
   2019-10-29T05:51:04,886 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[last7yike] conversions with cardinality[0] in 0 millis.
   2019-10-29T05:51:04,887 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[last30yike] conversions with cardinality[0] in 0 millis.
   2019-10-29T05:51:04,887 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[paycourse] conversions with cardinality[0] in 0 millis.
   2019-10-29T05:51:04,887 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[url] conversions with cardinality[0] in 0 millis.
   2019-10-29T05:51:04,887 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[rank_model] conversions with cardinality[0] in 0 millis.
   2019-10-29T05:51:04,888 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[adpos_is_bidding] conversions with cardinality[8] in 0 millis.
   2019-10-29T05:51:04,888 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[clickarea] conversions with cardinality[0] in 0 millis.
   2019-10-29T05:51:04,888 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[agent_id] conversions with cardinality[3] in 0 millis.
   2019-10-29T05:51:04,888 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[company_id] conversions with cardinality[0] in 0 millis.
   2019-10-29T05:51:04,890 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[creative] conversions with cardinality[329] in 2 millis.
   2019-10-29T05:51:04,890 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed dim conversions in 11 millis.
   2019-10-29T05:51:05,001 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - completed walk through of 2,338 rows in 108 millis.
   2019-10-29T05:51:05,001 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed time column in 0 millis.
   2019-10-29T05:51:05,001 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[bkorder_sum] in 0 millis.
   2019-10-29T05:51:05,001 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[buy_sum] in 0 millis.
   2019-10-29T05:51:05,002 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[buyuv] in 1 millis.
   2019-10-29T05:51:05,003 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[click_sum] in 1 millis.
   2019-10-29T05:51:05,003 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[click_uv] in 0 millis.
   2019-10-29T05:51:05,004 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[count] in 1 millis.
   2019-10-29T05:51:05,008 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[deliver_sum] in 4 millis.
   2019-10-29T05:51:05,008 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[dxborder_sum] in 0 millis.
   2019-10-29T05:51:05,009 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[land_sum] in 1 millis.
   2019-10-29T05:51:05,009 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[land_uv] in 0 millis.
   2019-10-29T05:51:05,009 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[order_sum] in 0 millis.
   2019-10-29T05:51:05,009 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[order_uv] in 0 millis.
   2019-10-29T05:51:05,009 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[pats_uv] in 0 millis.
   2019-10-29T05:51:05,010 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[pay2_sum] in 1 millis.
   2019-10-29T05:51:05,010 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[pay2_uv] in 0 millis.
   2019-10-29T05:51:05,010 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[pay_sum] in 0 millis.
   2019-10-29T05:51:05,010 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[pay_uv] in 0 millis.
   2019-10-29T05:51:05,010 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[pays_sum] in 0 millis.
   2019-10-29T05:51:05,010 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[presentation_sum] in 0 millis.
   2019-10-29T05:51:05,012 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[rdeliver_sum] in 2 millis.
   2019-10-29T05:51:05,012 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[rdeliver_uv] in 0 millis.
   2019-10-29T05:51:05,017 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[req_sum] in 4 millis.
   2019-10-29T05:51:05,022 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[resp_sum] in 4 millis.
   2019-10-29T05:51:05,022 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[stkorder_sum] in 0 millis.
   2019-10-29T05:51:05,022 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric column[ztkorder_sum] in 0 millis.
   2019-10-29T05:51:05,022 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed metric columns in 21 millis.
   2019-10-29T05:51:05,022 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[customer] inverted with cardinality[13] in 0 millis.
   2019-10-29T05:51:05,023 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[grade] inverted with cardinality[13] in 0 millis.
   2019-10-29T05:51:05,023 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[courseType] inverted with cardinality[0] in 0 millis.
   2019-10-29T05:51:05,024 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[adpos] inverted with cardinality[60] in 1 millis.
   2019-10-29T05:51:05,024 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[os] inverted with cardinality[3] in 0 millis.
   2019-10-29T05:51:05,026 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[av] inverted with cardinality[52] in 1 millis.
   2019-10-29T05:51:05,027 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[campaign] inverted with cardinality[51] in 1 millis.
   2019-10-29T05:51:05,028 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[unit] inverted with cardinality[186] in 1 millis.
   2019-10-29T05:51:05,029 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[template] inverted with cardinality[31] in 0 millis.
   2019-10-29T05:51:05,030 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[province] inverted with cardinality[32] in 1 millis.
   2019-10-29T05:51:05,032 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[city] inverted with cardinality[295] in 2 millis.
   2019-10-29T05:51:05,033 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[conn] inverted with cardinality[5] in 0 millis.
   2019-10-29T05:51:05,034 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[role] inverted with cardinality[0] in 0 millis.
   2019-10-29T05:51:05,034 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[sex] inverted with cardinality[0] in 0 millis.
   2019-10-29T05:51:05,034 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[last7yike] inverted with cardinality[0] in 0 millis.
   2019-10-29T05:51:05,034 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[last30yike] inverted with cardinality[0] in 0 millis.
   2019-10-29T05:51:05,034 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[paycourse] inverted with cardinality[0] in 0 millis.
   2019-10-29T05:51:05,035 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[url] inverted with cardinality[0] in 1 millis.
   2019-10-29T05:51:05,035 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[rank_model] inverted with cardinality[0] in 0 millis.
   2019-10-29T05:51:05,035 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[adpos_is_bidding] inverted with cardinality[8] in 0 millis.
   2019-10-29T05:51:05,036 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[clickarea] inverted with cardinality[0] in 0 millis.
   2019-10-29T05:51:05,036 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[agent_id] inverted with cardinality[3] in 0 millis.
   2019-10-29T05:51:05,037 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[company_id] inverted with cardinality[0] in 0 millis.
   2019-10-29T05:51:05,039 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.StringDimensionMergerV9 - Completed dim[creative] inverted with cardinality[329] in 2 millis.
   2019-10-29T05:51:05,040 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.IndexMergerV9 - Completed index.drd in 1 millis.
   2019-10-29T05:51:05,040 INFO [bi_imp_ad-incremental-persist] org.apache.druid.java.util.common.io.smoosh.FileSmoosher - Created smoosh file [/home/iknow/apps/druid-0.16.0/var/druid/task/index_kafka_bi_imp_ad_7130098998f04ba_hgcjbedd/work/persist/bi_imp_ad_2019-10-29T13:00:00.000Z_2019-10-29T14:00:00.000Z_2019-10-29T05:00:09.490Z_21/9/00000.smoosh] of size [234718] bytes.
   2019-10-29T05:51:05,048 INFO [bi_imp_ad-incremental-persist] org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Committing metadata[AppenderatorDriverMetadata{segments={index_kafka_bi_imp_ad_7130098998f04ba_0=[SegmentWithState{segmentIdentifier=bi_imp_ad_2019-10-29T09:00:00.000Z_2019-10-29T10:00:00.000Z_2019-10-29T02:46:35.586Z_37, state=APPENDING}, SegmentWithState{segmentIdentifier=bi_imp_ad_2019-10-29T12:00:00.000Z_2019-10-29T13:00:00.000Z_2019-10-29T04:00:09.473Z_32, state=APPENDING}, SegmentWithState{segmentIdentifier=bi_imp_ad_2019-10-29T13:00:00.000Z_2019-10-29T14:00:00.000Z_2019-10-29T05:00:09.490Z_21, state=APPENDING}]}, lastSegmentIds={index_kafka_bi_imp_ad_7130098998f04ba_0=bi_imp_ad_2019-10-29T12:00:00.000Z_2019-10-29T13:00:00.000Z_2019-10-29T04:00:09.473Z_32}, callerMetadata={nextPartitions=SeekableStreamEndSequenceNumbers{stream='imp_grail', partitionSequenceNumberMap={3=5947403642, 36=209577086, 25=209578334, 14=209578334, 47=209578210}}}}] for sinks[bi_imp_ad_2019-10-29T12:00:00.000Z_2019-10-29T13:00:00.000Z_2019-10-29T04:00:09.473Z_32:2, bi_imp_ad_2019-10-29T13:00:00.000Z_2019-10-29T14:00:00.000Z_2019-10-29T05:00:09.490Z_21:10, bi_imp_ad_2019-10-29T09:00:00.000Z_2019-10-29T10:00:00.000Z_2019-10-29T02:46:35.586Z_37:1].
   2019-10-29T05:51:05,050 INFO [appenderator_persist_0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Persist completed with metadata [AppenderatorDriverMetadata{segments={index_kafka_bi_imp_ad_7130098998f04ba_0=[SegmentWithState{segmentIdentifier=bi_imp_ad_2019-10-29T09:00:00.000Z_2019-10-29T10:00:00.000Z_2019-10-29T02:46:35.586Z_37, state=APPENDING}, SegmentWithState{segmentIdentifier=bi_imp_ad_2019-10-29T12:00:00.000Z_2019-10-29T13:00:00.000Z_2019-10-29T04:00:09.473Z_32, state=APPENDING}, SegmentWithState{segmentIdentifier=bi_imp_ad_2019-10-29T13:00:00.000Z_2019-10-29T14:00:00.000Z_2019-10-29T05:00:09.490Z_21, state=APPENDING}]}, lastSegmentIds={index_kafka_bi_imp_ad_7130098998f04ba_0=bi_imp_ad_2019-10-29T12:00:00.000Z_2019-10-29T13:00:00.000Z_2019-10-29T04:00:09.473Z_32}, callerMetadata={nextPartitions=SeekableStreamEndSequenceNumbers{stream='imp_grail', partitionSequenceNumberMap={3=5947403642, 36=209577086, 25=209578334, 14=209578334, 47=209578210}}}}]
   
   > 
   middleManager javaOpts:
   > druid.indexer.runner.javaOpts=-server -Xmx24g -XX:MaxDirectMemorySize=1024g -XX:+ExplicitGCInvokesConcurrent  -XX:+UseG1GC -XX:MaxGCPauseMillis=100 -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
   
   Not all tasks will fail, but for failed tasks, they always fail when running for more than 10 seconds, and there will be consecutive failures.
   
   Who can help me?
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org