[GitHub] andreialecu commented on issue #435: Pods not starting properly after restarting Docker

[GitBox <gi...@apache.org>]

andreialecu commented on issue #435: Pods not starting properly after restarting Docker
URL: https://github.com/apache/incubator-openwhisk-deploy-kube/issues/435#issuecomment-467004615
 
 
   This is persistent and I can reproduce it every time. 
   
   Steps:
   1) Reset the kubernetes cluster 
   2) Install openwhisk chart via helm
   3) Verify everything runs properly (`helm test owdev --cleanup` succeeds)
   3) Restart docker
   
   Here's an alternative/similar log:
   
   ```
   [2019-02-25 13:05:35,506] INFO Loading logs. (kafka.log.LogManager)
   [2019-02-25 13:05:35,577] ERROR Could not find offset index file corresponding to log file /kafka/kafka-logs-owdev-kafka-0/cacheInvalidation-0/00000000000000000000.log, rebuilding index... (kafka.log.Log)
   [2019-02-25 13:05:35,660] INFO Recovering unflushed segment 0 in log cacheInvalidation-0. (kafka.log.Log)
   [2019-02-25 13:05:35,697] INFO Loading producer state from offset 84 for partition cacheInvalidation-0 with message format version 2 (kafka.log.Log)
   [2019-02-25 13:05:35,702] INFO Loading producer state from snapshot file '/kafka/kafka-logs-owdev-kafka-0/cacheInvalidation-0/00000000000000000084.snapshot' for partition cacheInvalidation-0 (kafka.log.ProducerStateManager)
   [2019-02-25 13:05:35,710] INFO Completed load of log cacheInvalidation-0 with 1 log segments, log start offset 0 and log end offset 84 in 162 ms (kafka.log.Log)
   [2019-02-25 13:05:35,724] ERROR Could not find offset index file corresponding to log file /kafka/kafka-logs-owdev-kafka-0/completed0-0/00000000000000000000.log, rebuilding index... (kafka.log.Log)
   [2019-02-25 13:05:35,758] INFO Recovering unflushed segment 0 in log completed0-0. (kafka.log.Log)
   [2019-02-25 13:05:35,788] INFO Loading producer state from offset 16 for partition completed0-0 with message format version 2 (kafka.log.Log)
   [2019-02-25 13:05:35,791] INFO Loading producer state from snapshot file '/kafka/kafka-logs-owdev-kafka-0/completed0-0/00000000000000000016.snapshot' for partition completed0-0 (kafka.log.ProducerStateManager)
   [2019-02-25 13:05:35,793] INFO Completed load of log completed0-0 with 1 log segments, log start offset 0 and log end offset 16 in 76 ms (kafka.log.Log)
   [2019-02-25 13:05:35,804] ERROR Could not find offset index file corresponding to log file /kafka/kafka-logs-owdev-kafka-0/events-0/00000000000000000000.log, rebuilding index... (kafka.log.Log)
   [2019-02-25 13:05:35,813] INFO Recovering unflushed segment 0 in log events-0. (kafka.log.Log)
   [2019-02-25 13:05:35,825] INFO Loading producer state from offset 0 for partition events-0 with message format version 2 (kafka.log.Log)
   [2019-02-25 13:05:35,829] INFO Completed load of log events-0 with 1 log segments, log start offset 0 and log end offset 0 in 33 ms (kafka.log.Log)
   [2019-02-25 13:05:35,848] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (/kafka/kafka-logs-owdev-kafka-0/health-0/00000000000000000000.index) has non-zero size but the last offset is 0 which is no larger than the base offset 0.}. deleting /kafka/kafka-logs-owdev-kafka-0/health-0/00000000000000000000.timeindex, /kafka/kafka-logs-owdev-kafka-0/health-0/00000000000000000000.index, and /kafka/kafka-logs-owdev-kafka-0/health-0/00000000000000000000.txnindex and rebuilding index... (kafka.log.Log)
   [2019-02-25 13:05:35,855] ERROR There was an error in one of the threads during logs loading: java.io.FileNotFoundException: /kafka/kafka-logs-owdev-kafka-0/health-0/00000000000000000000.index (No such file or directory) (kafka.log.LogManager)
   [2019-02-25 13:05:35,859] FATAL [Kafka Server 0], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
   java.io.FileNotFoundException: /kafka/kafka-logs-owdev-kafka-0/health-0/00000000000000000000.index (No such file or directory)
           at java.io.RandomAccessFile.open0(Native Method)
           at java.io.RandomAccessFile.open(RandomAccessFile.java:316)
           at java.io.RandomAccessFile.<init>(RandomAccessFile.java:243)
           at kafka.log.AbstractIndex.$anonfun$resize$1(AbstractIndex.scala:106)
           at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:213)
           at kafka.log.AbstractIndex.resize(AbstractIndex.scala:105)
           at kafka.log.LogSegment.recover(LogSegment.scala:256)
           at kafka.log.Log.recoverSegment(Log.scala:342)
           at kafka.log.Log.$anonfun$loadSegmentFiles$3(Log.scala:321)
           at scala.collection.TraversableLike$WithFilter.$anonfun$foreach$1(TraversableLike.scala:789)
           at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:32)
           at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:29)
           at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:191)
           at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:788)
           at kafka.log.Log.loadSegmentFiles(Log.scala:279)
           at kafka.log.Log.loadSegments(Log.scala:383)
           at kafka.log.Log.<init>(Log.scala:186)
           at kafka.log.Log$.apply(Log.scala:1610)
           at kafka.log.LogManager.$anonfun$loadLogs$12(LogManager.scala:172)
           at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:57)
           at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
           at java.util.concurrent.FutureTask.run(FutureTask.java:266)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
           at java.lang.Thread.run(Thread.java:748)
   [2019-02-25 13:05:35,867] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
   [2019-02-25 13:05:35,877] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
   [2019-02-25 13:05:35,878] WARN Found a corrupted index file due to requirement failed: Corrupt index found, index file (/kafka/kafka-logs-owdev-kafka-0/invoker0-0/00000000000000000000.index) has non-zero size but the last offset is 0 which is no larger than the base offset 0.}. deleting /kafka/kafka-logs-owdev-kafka-0/invoker0-0/00000000000000000000.timeindex, /kafka/kafka-logs-owdev-kafka-0/invoker0-0/00000000000000000000.index, and /kafka/kafka-logs-owdev-kafka-0/invoker0-0/00000000000000000000.txnindex and rebuilding index... (kafka.log.Log)
   [2019-02-25 13:05:35,887] INFO Session: 0x10000012f850001 closed (org.apache.zookeeper.ZooKeeper)
   [2019-02-25 13:05:35,888] INFO EventThread shut down for session: 0x10000012f850001 (org.apache.zookeeper.ClientCnxn)
   [2019-02-25 13:05:35,894] INFO [Kafka Server 0], shut down completed (kafka.server.KafkaServer)
   [2019-02-25 13:05:35,895] FATAL Exiting Kafka. (kafka.server.KafkaServerStartable)
   [2019-02-25 13:05:35,898] INFO [Kafka Server 0], shutting down (kafka.server.KafkaServer)
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services