You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "Fryuni (via GitHub)" <gi...@apache.org> on 2025/07/26 16:42:58 UTC
[I] Redundant tasks race each other causing error when publishing segments (druid)
Fryuni opened a new issue, #18330:
URL: https://github.com/apache/druid/issues/18330
### Affected Version
`33.0.0`
### Description
I'm seeing the error `Attempting to publish with empty segment set, but total row count was not 0`.
It is the same error as #9845, which was supposed to be fixed by #14995 that landed on v29. But this is happening on `v33`.
This only happens when task replicas are created to avoid problems when workers are terminated (by the K8s cluster auto-scaler, for example). Any task with `replicas` greater than 1 reaches the same error.
It doesn't happen to only one of the replicas. It seems to be a race, one task wins for a few segments and the other wins for other segments. Then both fail and nothing gets commited. Setting replicas back to one makes this issue go away, but then ingestion have downtimes whenever a worker is stopped.
<details>
<summary>Log of a failed task</summary>
```
2025-07-26T14:31:11+00:00 startup service overlord
2025-07-26 14:31:18.336 INFO net.spy.memcached.MemcachedConnection: Setting retryQueueSize to -1
2025-07-26 14:31:18.351 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/<redacted>, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2025-07-26 14:31:18.372 INFO net.spy.memcached.MemcachedConnection: Setting retryQueueSize to -1
2025-07-26 14:31:18.373 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/<redacted>, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
2025-07-26 14:31:18.377 INFO net.spy.memcached.MemcachedConnection: Setting retryQueueSize to -1
2025-07-26 14:31:18.378 INFO net.spy.memcached.MemcachedConnection: Added {QA sa=/<redacted>, #Rops=0, #Wops=0, #iq=0, topRop=null, topWop=null, toWrite=0, interested=0} to connect queue
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider as a provider class
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider as a provider class
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.druid.server.initialization.jetty.CustomExceptionMapper as a provider class
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper as a provider class
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper as a provider class
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.druid.server.initialization.jetty.ServiceUnavailableExceptionMapper as a provider class
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
INFO: Registering org.apache.druid.server.StatusResource as a root resource class
Jul 26, 2025 2:31:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.4 05/24/2017 03:20 PM'
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.server.initialization.jetty.CustomExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.server.initialization.jetty.ServiceUnavailableExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:20 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to GuiceManagedComponentProvider with the scope "Singleton"
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.security.basic.BasicSecurityResourceFilter to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.security.basic.authentication.endpoint.BasicAuthenticatorResource to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.security.basic.authorization.endpoint.BasicAuthorizerResource to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.security.pac4j.Pac4jCallbackResource to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.server.http.security.StateResourceFilter to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.server.http.SegmentListerResource to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.server.http.HistoricalResource to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.server.QueryResource to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.segment.realtime.ChatHandlerResource to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.server.http.security.ConfigResourceFilter to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.query.lookup.LookupListeningResource to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.query.lookup.LookupIntrospectionResource to GuiceInstantiatedComponentProvider
Jul 26, 2025 2:31:21 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
INFO: Binding org.apache.druid.server.StatusResource to GuiceManagedComponentProvider with the scope "Singleton"
14:36:50.981 [[index_kafka_goalByExperience_d9e37bb95ea1354_jocncobe]-publish] ERROR org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Error while publishing segments for sequenceNumber[SequenceMetadata{sequenceId=681, sequenceName='index_kafka_goalByExperience_d9e37bb95ea1354_681', assignments=[], startOffsets={KafkaTopicPartition{partition=23, topic='null', multiTopicPartition=false}=9469798, KafkaTopicPartition{partition=22, topic='null', multiTopicPartition=false}=9498999, KafkaTopicPartition{partition=25, topic='null', multiTopicPartition=false}=10047313, KafkaTopicPartition{partition=24, topic='null', multiTopicPartition=false}=9658869, KafkaTopicPartition{partition=27, topic='null', multiTopicPartition=false}=9504586, KafkaTopicPartition{partition=26, topic='null', multiTopicPartition=false}=9595552, KafkaTopicPartition{partition=29, topic='null', multiTopicPartition=false}=9606074, KafkaTopicPartition{partition=28, topic='null', multiTopicPartition=f
alse}=9340750, KafkaTopicPartition{partition=1, topic='null', multiTopicPartition=false}=9665107, KafkaTopicPartition{partition=0, topic='null', multiTopicPartition=false}=9467462, KafkaTopicPartition{partition=3, topic='null', multiTopicPartition=false}=9377332, KafkaTopicPartition{partition=2, topic='null', multiTopicPartition=false}=9570694, KafkaTopicPartition{partition=5, topic='null', multiTopicPartition=false}=9462158, KafkaTopicPartition{partition=4, topic='null', multiTopicPartition=false}=9457911, KafkaTopicPartition{partition=7, topic='null', multiTopicPartition=false}=9511729, KafkaTopicPartition{partition=6, topic='null', multiTopicPartition=false}=9447538, KafkaTopicPartition{partition=9, topic='null', multiTopicPartition=false}=9470660, KafkaTopicPartition{partition=8, topic='null', multiTopicPartition=false}=9381318, KafkaTopicPartition{partition=11, topic='null', multiTopicPartition=false}=9522858, KafkaTopicPartition{partition=10, topic='null', multiTopicPartition=
false}=9720465, KafkaTopicPartition{partition=13, topic='null', multiTopicPartition=false}=9639213, KafkaTopicPartition{partition=12, topic='null', multiTopicPartition=false}=9905181, KafkaTopicPartition{partition=15, topic='null', multiTopicPartition=false}=9435236, KafkaTopicPartition{partition=14, topic='null', multiTopicPartition=false}=10042299, KafkaTopicPartition{partition=17, topic='null', multiTopicPartition=false}=9405092, KafkaTopicPartition{partition=16, topic='null', multiTopicPartition=false}=9485933, KafkaTopicPartition{partition=19, topic='null', multiTopicPartition=false}=9355762, KafkaTopicPartition{partition=18, topic='null', multiTopicPartition=false}=9665043, KafkaTopicPartition{partition=21, topic='null', multiTopicPartition=false}=9593813, KafkaTopicPartition{partition=20, topic='null', multiTopicPartition=false}=9612303}, exclusiveStartPartitions=[], endOffsets={KafkaTopicPartition{partition=23, topic='null', multiTopicPartition=false}=9469798, KafkaTopicPart
ition{partition=22, topic='null', multiTopicPartition=false}=9497963, KafkaTopicPartition{partition=25, topic='null', multiTopicPartition=false}=10047313, KafkaTopicPartition{partition=24, topic='null', multiTopicPartition=false}=9657756, KafkaTopicPartition{partition=27, topic='null', multiTopicPartition=false}=9504586, KafkaTopicPartition{partition=26, topic='null', multiTopicPartition=false}=9595552, KafkaTopicPartition{partition=29, topic='null', multiTopicPartition=false}=9606074, KafkaTopicPartition{partition=28, topic='null', multiTopicPartition=false}=9340750, KafkaTopicPartition{partition=1, topic='null', multiTopicPartition=false}=9665107, KafkaTopicPartition{partition=0, topic='null', multiTopicPartition=false}=9467462, KafkaTopicPartition{partition=3, topic='null', multiTopicPartition=false}=9377332, KafkaTopicPartition{partition=2, topic='null', multiTopicPartition=false}=9570694, KafkaTopicPartition{partition=5, topic='null', multiTopicPartition=false}=9462158, KafkaTo
picPartition{partition=4, topic='null', multiTopicPartition=false}=9457911, KafkaTopicPartition{partition=7, topic='null', multiTopicPartition=false}=9511729, KafkaTopicPartition{partition=6, topic='null', multiTopicPartition=false}=9447538, KafkaTopicPartition{partition=9, topic='null', multiTopicPartition=false}=9470660, KafkaTopicPartition{partition=8, topic='null', multiTopicPartition=false}=9381318, KafkaTopicPartition{partition=11, topic='null', multiTopicPartition=false}=9521747, KafkaTopicPartition{partition=10, topic='null', multiTopicPartition=false}=9720465, KafkaTopicPartition{partition=13, topic='null', multiTopicPartition=false}=9638252, KafkaTopicPartition{partition=12, topic='null', multiTopicPartition=false}=9904681, KafkaTopicPartition{partition=15, topic='null', multiTopicPartition=false}=9435236, KafkaTopicPartition{partition=14, topic='null', multiTopicPartition=false}=10042299, KafkaTopicPartition{partition=17, topic='null', multiTopicPartition=false}=9405092,
KafkaTopicPartition{partition=16, topic='null', multiTopicPartition=false}=9485933, KafkaTopicPartition{partition=19, topic='null', multiTopicPartition=false}=9355762, KafkaTopicPartition{partition=18, topic='null', multiTopicPartition=false}=9665043, KafkaTopicPartition{partition=21, topic='null', multiTopicPartition=false}=9593813, KafkaTopicPartition{partition=20, topic='null', multiTopicPartition=false}=9612303}, sentinel=false, checkpointed=true}]
org.apache.druid.java.util.common.ISE: Attempting to publish with empty segment set, but total row count was not 0: [1565].
at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.publishInBackground(BaseAppenderatorDriver.java:610) ~[druid-server-33.0.0.jar:33.0.0]
at org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.lambda$publish$1(StreamAppenderatorDriver.java:289) ~[druid-server-33.0.0.jar:33.0.0]
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:223) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:210) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:123) [guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) [guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286) [guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055) [guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:782) [guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.setResult(AbstractTransformFuture.java:257) [guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:169) [guava-32.0.1-jre.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
14:37:15.007 [task-runner-0-priority-0] ERROR org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting.
java.util.concurrent.ExecutionException: org.apache.druid.java.util.common.ISE: Attempting to publish with empty segment set, but total row count was not 0: [1565].
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:592) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:551) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91) ~[guava-32.0.1-jre.jar:?]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.checkPublishAndHandoffFailure(SeekableStreamIndexTaskRunner.java:985) ~[druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:634) [druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:295) [druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.runTask(SeekableStreamIndexTask.java:152) [druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:179) [druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) [druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) [druid-indexing-service-33.0.0.jar:33.0.0]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) [guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75) [guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) [guava-32.0.1-jre.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.apache.druid.java.util.common.ISE: Attempting to publish with empty segment set, but total row count was not 0: [1565].
at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.publishInBackground(BaseAppenderatorDriver.java:610) ~[druid-server-33.0.0.jar:33.0.0]
at org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.lambda$publish$1(StreamAppenderatorDriver.java:289) ~[druid-server-33.0.0.jar:33.0.0]
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:223) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:210) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:123) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:782) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.setResult(AbstractTransformFuture.java:257) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:169) ~[guava-32.0.1-jre.jar:?]
... 3 more
14:37:15.463 [task-runner-0-priority-0] ERROR org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task.
java.util.concurrent.ExecutionException: org.apache.druid.java.util.common.ISE: Attempting to publish with empty segment set, but total row count was not 0: [1565].
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:592) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:551) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:91) ~[guava-32.0.1-jre.jar:?]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.checkPublishAndHandoffFailure(SeekableStreamIndexTaskRunner.java:985) ~[druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:634) ~[druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:295) [druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.runTask(SeekableStreamIndexTask.java:152) [druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.common.task.AbstractTask.run(AbstractTask.java:179) [druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) [druid-indexing-service-33.0.0.jar:33.0.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) [druid-indexing-service-33.0.0.jar:33.0.0]
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) [guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:75) [guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) [guava-32.0.1-jre.jar:?]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.apache.druid.java.util.common.ISE: Attempting to publish with empty segment set, but total row count was not 0: [1565].
at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.publishInBackground(BaseAppenderatorDriver.java:610) ~[druid-server-33.0.0.jar:33.0.0]
at org.apache.druid.segment.realtime.appenderator.StreamAppenderatorDriver.lambda$publish$1(StreamAppenderatorDriver.java:289) ~[druid-server-33.0.0.jar:33.0.0]
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:223) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture$AsyncTransformFuture.doTransform(AbstractTransformFuture.java:210) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:123) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:782) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture$TransformFuture.setResult(AbstractTransformFuture.java:257) ~[guava-32.0.1-jre.jar:?]
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:169) ~[guava-32.0.1-jre.jar:?]
... 3 more
Finished peon task
```
</details>
- Cluster size:
- Coordinator: 3
- Overlord: 2
- Brokers: 6
- Routers: 6
- Historicals: 15
- Peons: one per task (using Overlord Kubernetes Extensions)
- Configuration in use:
- <details>
<summary>Common config</summary>
```
# Disable property logging at startup
druid.startup.logging.logProperties=false
# Extensions
druid.extensions.loadList=["druid-basic-security","postgresql-metadata-storage","druid-google-extensions","druid-pac4j","druid-kafka-indexing-service","druid-lookups-cached-global","druid-multi-stage-query","druid-virtual-columns","druid-avro-extensions","druid-datasketches","druid-moving-average-query","druid-time-min-max","druid-stats","druid-kubernetes-overlord-extensions"]
# Service discovery
druid.selectors.indexing.serviceName=druid/overlord
druid.selectors.coordinator.serviceName=druid/coordinator
druid.indexer.runner.type=httpRemote
druid.indexer.runner.timeout=PT5H
druid.indexer.queue.startDelay=PT5S
druid.indexer.storage.type=metadata
# Zookeeper discovery
druid.zk.service.enabled=true
druid.zk.service.host=<redacted>
# Metadata Store
druid.metadata.storage.type=postgresql
druid.metadata.storage.connector.connectURI=jdbc:postgresql://<redacted>
druid.metadata.storage.connector.createTables=true
druid.metadata.storage.connector.user=druid
# Deep Storage
druid.storage.type=google
druid.google.bucket=<redacted>
# Indexer logs
druid.indexer.logs.type=google
druid.indexer.logs.bucket=<redacted>
druid.indexer.logs.prefix=taskLogs
druid.indexer.logs.kill.enabled=false
druid.indexer.logs.kill.durationToRetain=604800000
# Storage type of double columns
druid.indexing.doubleStorage=double
# Security
druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password", "password", "key", "token", "pwd"]
druid.server.http.enableRequestLimit=true
druid.server.http.defaultQueryTimeout=60000
druid.server.http.maxQueryTimeout=300000
# Authentication
druid.auth.authenticator.allowAll.type=allowAll
druid.auth.authorizer.allowAll.type=allowAll
druid.auth.authenticatorChain=["allowAll", "basic", "pac4j"]
druid.lookup.enableLookupSyncOnStartup=true
druid.lookup.lookupTier=broker
druid.javascript.enabled=false
druid.emmiter.logging.logLevel=warn
# Metrics
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor","org.apache.druid.java.util.metrics.JvmCpuMonitor","org.apache.druid.java.util.metrics.JvmThreadsMonitor"]
druid.query.groupBy.maxOnDiskStorage=1073741824
druid.query.groupBy.applyLimitPushDownToSegment=true
# SQL
druid.sql.enable=true
# Planning SQL query when there is aggregate distinct in the statement
druid.sql.planner.useGroupingSetForExactDistinct=true
# Expression processing config
druid.expressions.useStrictBooleans=true
# Http client
druid.global.http.eagerInitialization=false
# Cache
druid.cache.type=memcached
druid.cache.hosts<redacted>
druid.cache.numConnections=3
```
</details>
- <details>
<summary>Overlord config</summary>
```
druid.service=druid/overlord
druid.indexer.storage.type=metadata
druid.indexer.queue.startDelay=PT30S
druid.indexer.tasklock.batchSegmentAllocation=true
druid.indexer.task.encapsulatedTask=true
druid.indexer.fork.property.druid.processing.intermediaryData.storage.type=deepstore
druid.indexer.runner.type=k8s
druid.indexer.queue.maxSize=60
druid.indexer.runner.namespace=analytics
druid.processing.intermediaryData.storage.type=deepstore
druid.indexer.runner.k8s.adapter.type=customTemplateAdapter
druid.indexer.runner.k8s.podTemplate.base=/druid/tasktemplate/default/default-task-template.yaml
druid.indexer.runner.primaryContainerName=main
druid.indexer.runner.debugJobs=false
druid.indexer.runner.maxTaskDuration=PT4H
druid.indexer.runner.taskCleanupDelay=PT2H
druid.indexer.runner.taskCleanupInterval=PT10M
druid.indexer.runner.taskJoinTimeout=PT10M
druid.indexer.runner.K8sjobLaunchTimeout=PT1H
druid.indexer.runner.pendingTasksRunnerNumThreads=2
druid.indexer.runner.labels={"app.kubernetes.io/name":"Druid","app.kubernetes.io/component":"Peon"}
druid.indexer.runner.graceTerminationPeriodSeconds=PT1M
druid.manager.segments.useCache=ifSynced
druid.manager.segments.pollDuration=PT5M
```
</details>
- <details>
<summary>Peon config</summary>
```
druid.startup.logging.logProperties=true
druid.port=8100
druid.service=druid/peon
druid.server.http.numThreads=3
druid.processing.numMergeBuffers=2
druid.indexer.task.baseTaskDir=/druid/data
druid.peon.mode=remote
druid.indexer.task.encapsulatedTask=true
druid.indexer.fork.property.druid.processing.intermediaryData.storage.type=deepstore
druid.indexer.queue.startDelay=PT30S
druid.indexer.storage.type=metadata
druid.indexer.task.encapsulatedTask=true
druid.indexer.runner.type=k8s
druid.indexer.queue.maxSize=60
druid.indexer.runner.namespace=analytics
druid.processing.intermediaryData.storage.type=deepstore
druid.realtime.cache.useCache=true
druid.realtime.cache.populateCache=true
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Redundant tasks race each other causing error when publishing segments (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #18330: Redundant tasks race each other causing error when publishing segments
URL: https://github.com/apache/druid/issues/18330
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Redundant tasks race each other causing error when publishing segments (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #18330:
URL: https://github.com/apache/druid/issues/18330#issuecomment-4585318194
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Redundant tasks race each other causing error when publishing segments (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #18330:
URL: https://github.com/apache/druid/issues/18330#issuecomment-4365065439
This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org