[GitHub] [incubator-druid] alex790310 opened a new issue #7587: Kafka indexing service task of peon can't stop

[GitBox <gi...@apache.org>]

alex790310 opened a new issue #7587: Kafka  indexing service task of peon can't stop
URL: https://github.com/apache/incubator-druid/issues/7587
 
 
   The kafka indexing service task work exception, can't stop at all, then, the new task can't start cause of this.
   
   ### Affected Version
   
   druid-0.13.0-incubating
   
   ### Description
   
   I found  nio exception for kafka indexing service task in middleManager log :
   
   [2019-04-29 11:42:05:678] [WARN] - org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:361) - 
   java.nio.channels.CancelledKeyException
      at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) ~[?:1.8.0_152]
      at sun.nio.ch.SelectionKeyImpl.readyOps(SelectionKeyImpl.java:87) ~[?:1.8.0_152]
      at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.processSelected(ManagedSelector.java:443) ~[jetty-io-9.4.10.v20180503.jar:9.4.10.v20180503]
   
   Then,  overlord service couldn't send request to peon processor, I found exception:
   
   [2019-04-29 11:42:05:678] [WARN] - org.apache.druid.java.util.common.logger.Logger.warn(Logger.java:101) - Exception while sending request
   java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.jboss.netty.handler.timeout.ReadTimeoutException
     at org.apache.druid.indexing.common.IndexTaskClient.submitRequest(IndexTaskClient.java:306) ~[druid-indexing-service-0.13.0-incubating.jar]
   
   After that, overlord was found same exception over and over until the kafka indexing task duration is over, then overlord ask middleManager to stop the task, but task couldn't be stopped
   success.  
   The stop log show as :
   
   Triggering JVM shutdown.
   Running shutdown hook.
   unannouncing [/druid/listeners/lookups/__default/http:130.255.7.197:8100]
   
   But the peon task is not really stop, the task will run for ever. I can see log as 'Create smoosh file' in every 15 minutes and also can find task on zookeeper path /druid/indexer/status.
   
   Because of that, the new kafka indexing task can't run and it was be set to pending.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org