You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "baaaaast (via GitHub)" <gi...@apache.org> on 2024/06/12 13:31:19 UTC
[I] Kafka indexing service duplicate key exception in druid_pendingSegments table (druid)
baaaaast opened a new issue, #16587:
URL: https://github.com/apache/druid/issues/16587
### Affected Version
29.0.1
### Description
#### KIS configuration :
- replica > 1
- lateMessageRejectionPeriod of several days
```
{
"type": "kafka",
"dataSchema":{
"dataSource":"my-datasource",
"parser":{
"type":"avro_stream",
"avroBytesDecoder" : {
"type" : "schema_registry",
"url" : "https://schema-registry.url"
},
"parseSpec": {
"format":"timeAndDims",
"timestampSpec": {
"column": "timestamp"
},
"dimensionsSpec":{
"dimensions":["my-dimension1","my-dimension2",...],
"spatialDimensions":[]
}
}
},
"metricsSpec":[
{
"type" : "longSum",
"name" : "my-name",
"fieldName": "my-name"
},...
],
"granularitySpec":{
"type":"uniform",
"segmentGranularity":"HOUR",
"queryGranularity":"HOUR"
}
},
"tuningConfig" : {
"type" : "kafka",
"maxRowsInMemory": 150000,
"maxBytesInMemory" : 0,
"maxRowsPerSegment": 5000000,
"workerThreads": 5,
"intermediatePersistPeriod" : "PT10M",
"maxPendingPersists" : 0,
"forceExtendableShardSpecs": true,
"indexSpec" : {
"bitmap" : {
"type" : "roaring"
},
"dimensionCompression" : "lz4",
"metricCompression" : "lz4",
"longEncoding" : "longs"
},
"resetOffsetAutomatically" : false
},
"ioConfig": {
"type" : "kafka",
"topic": "my-topic",
"taskCount": 2,
"replicas": 2,
"taskDuration": "PT1H",
"useEarliestOffset": false,
"lateMessageRejectionPeriod": "P8D",
"earlyMessageRejectionPeriod": "PT3H",
"consumerProperties": {
"bootstrap.servers": "broker-1:9092,broker-2:9092,broker-3:9092"
}
}
}
```
#### Particularity of the topic :
- consumed topic can have several million messages per minute
- messages with a timestamp to be dated several days in the past
=> KIS writes many segments in the past every hour.
#### Problem
After a while (after several intermediate persist periods), the overlord starts generating errors and returns a status 500 to the middlemanager. This causes the indexing task to fail.
I can't reproduce the problem if the kafka topic has a low rate of messages produced.
#### Overlord error
```
2024-06-12T08:46:42,877 WARN [qtp1884568750-98] org.apache.druid.indexing.overlord.http.OverlordResource - Failed to perform task action
java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: Batch entry 1 INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (('my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5'), ('my-datasource'), ('2024-06-12T08:46:42.875Z'), ('2024-06-12T01:00:00.000Z'), ('2024-06-12T02:00:00.000Z'), ('index_kafka_my-datasource_0f3e4661a8cd3ee_0'), ('my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6'), ('A10883453F9AA74CDD231465CFF98E73DE3A2864'), ?) was aborted: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists. Call getNextException to see other errors in the batch. [statement:"INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, \"end\", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (:id, :dataSource, :created_date, :start, :end, :sequence_name, :sequence_prev_id, :sequence_name_prev_id_sha1, :payload)", located:"null", rewritten:"null", arguments:{ positional:{}, named:{sequence_prev_id:'my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6',payload:[*, *...],start:'2024-06-12T01:00:00.000Z',end:'2024-06-12T02:00:00.000Z',id:'my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5',created_date:'2024-06-12T08:46:42.875Z',dataSource:'my-datasource',sequence_name:'index_kafka_my-datasource_0f3e4661a8cd3ee_0',sequence_name_prev_id_sha1:'A10883453F9AA74CDD231465CFF98E7
3DE3A2864'}, finder:[]}]
at org.apache.druid.indexing.common.actions.LocalTaskActionClient.performAction(LocalTaskActionClient.java:98) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.LocalTaskActionClient.submit(LocalTaskActionClient.java:80) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:627) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.http.OverlordResource$3.apply(OverlordResource.java:616) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.http.OverlordResource.asLeaderWith(OverlordResource.java:1108) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.http.OverlordResource.doAction(OverlordResource.java:613) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at jdk.internal.reflect.GeneratedMethodAccessor246.invoke(Unknown Source) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) ~[?:?]
at java.lang.reflect.Method.invoke(Unknown Source) ~[?:?]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.4.jar:1.19.4]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.4.jar:1.19.4]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.4.jar:1.19.4]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.4.jar:1.19.4]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:286) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:276) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:181) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) ~[guice-servlet-4.1.0.jar:?]
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:135) ~[guice-servlet-4.1.0.jar:?]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.apache.druid.server.http.RedirectFilter.doFilter(RedirectFilter.java:73) ~[druid-server-29.0.1.jar:29.0.1]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:84) ~[druid-server-29.0.1.jar:29.0.1]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.apache.druid.server.initialization.jetty.StandardResponseHeaderFilterHolder$StandardResponseHeaderFilter.doFilter(StandardResponseHeaderFilterHolder.java:164) ~[druid-server-29.0.1.jar:29.0.1]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78) ~[druid-server-29.0.1.jar:29.0.1]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:74) ~[druid-server-29.0.1.jar:29.0.1]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.apache.druid.server.security.AllowAllAuthenticator$1.doFilter(AllowAllAuthenticator.java:84) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) ~[druid-server-29.0.1.jar:29.0.1]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77) ~[druid-server-29.0.1.jar:29.0.1]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.53.v20231009.jar:9.4.53.v20231009]
at java.lang.Thread.run(Unknown Source) ~[?:?]
Caused by: java.util.concurrent.ExecutionException: org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: Batch entry 1 INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (('my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5'), ('my-datasource'), ('2024-06-12T08:46:42.875Z'), ('2024-06-12T01:00:00.000Z'), ('2024-06-12T02:00:00.000Z'), ('index_kafka_my-datasource_0f3e4661a8cd3ee_0'), ('my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6'), ('A10883453F9AA74CDD231465CFF98E73DE3A2864'), ?) was aborted: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists. Call getNextException to see other errors in the batch. [statement:"INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, \"end\", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (:id, :dataSource, :created_date, :start, :end, :sequence_name, :sequence_prev_id, :sequence_name_prev_id_sha1, :payload)", located:"null", rewritten:"null", arguments:{ positional:{}, named:{sequence_prev_id:'my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6',payload:[*, *...],start:'2024-06-12T01:00:00.000Z',end:'2024-06-12T02:00:00.000Z',id:'my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5',created_date:'2024-06-12T08:46:42.875Z',dataSource:'my-datasource',sequence_name:'index_kafka_my-datasource_0f3e4661a8cd3ee_0',sequence_name_prev_id_sha1:'A10883453F9AA74CDD231465CFF98E7
3DE3A2864'}, finder:[]}]
at java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]
at org.apache.druid.indexing.common.actions.LocalTaskActionClient.performAction(LocalTaskActionClient.java:90) ~[druid-indexing-service-29.0.1.jar:29.0.1]
... 85 more
Caused by: org.skife.jdbi.v2.exceptions.CallbackFailedException: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: Batch entry 1 INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (('my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5'), ('my-datasource'), ('2024-06-12T08:46:42.875Z'), ('2024-06-12T01:00:00.000Z'), ('2024-06-12T02:00:00.000Z'), ('index_kafka_my-datasource_0f3e4661a8cd3ee_0'), ('my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6'), ('A10883453F9AA74CDD231465CFF98E73DE3A2864'), ?) was aborted: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists. Call getNextException to see other errors in the batch. [statement:"INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, \"end\", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (:id, :dataSource, :created_date, :start, :end, :sequence_name, :sequence_prev_id, :sequence_name_prev_id_sha1, :payload)", located:"null", rewritten:"null", arguments:{ positional:{}, named:{sequence_prev_id:'my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6',payload:[*, *...],start:'2024-06-12T01:00:00.000Z',end:'2024-06-12T02:00:00.000Z',id:'my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5',created_date:'2024-06-12T08:46:42.875Z',dataSource:'my-datasource',sequence_name:'index_kafka_my-datasource_0f3e4661a8cd3ee_0',sequence_name_prev_id_sha1:'A10883453F9AA74CDD231465CFF98E7
3DE3A2864'}, finder:[]}]
at org.skife.jdbi.v2.DBI.withHandle(DBI.java:284) ~[jdbi-2.63.1.jar:2.63.1]
at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:161) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:645) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegmentIds(TaskLockbox.java:696) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegments(TaskLockbox.java:466) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForInterval(SegmentAllocationQueue.java:491) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForBatch(SegmentAllocationQueue.java:433) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatch(SegmentAllocationQueue.java:351) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatchesDue(SegmentAllocationQueue.java:266) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
... 1 more
Caused by: org.skife.jdbi.v2.exceptions.UnableToExecuteStatementException: java.sql.BatchUpdateException: Batch entry 1 INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (('my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5'), ('my-datasource'), ('2024-06-12T08:46:42.875Z'), ('2024-06-12T01:00:00.000Z'), ('2024-06-12T02:00:00.000Z'), ('index_kafka_my-datasource_0f3e4661a8cd3ee_0'), ('my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6'), ('A10883453F9AA74CDD231465CFF98E73DE3A2864'), ?) was aborted: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists. Call getNextException to see other errors in the batch. [statement:"INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, \"end\", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (:id, :dataSource, :created_date, :start, :end, :sequence_name, :sequence_prev_id, :sequence_name_prev_id_sha1, :payload)", located:"null", rewritten:"null", arguments:{ positional:{}, named:{sequence_prev_id:'my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6',payload:[*, *...],start:'2024-06-12T01:00:00.000Z',end:'2024-06-12T02:00:00.000Z',id:'my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5',created_date:'2024-06-12T08:46:42.875Z',dataSource:'my-datasource',sequence_name:'index_kafka_my-datasource_0f3e4661a8cd3ee_0',sequence_name_prev_id_sha1:'A10883453F9AA74CDD231465CFF98E7
3DE3A2864'}, finder:[]}]
at org.skife.jdbi.v2.PreparedBatch.execute(PreparedBatch.java:148) ~[jdbi-2.63.1.jar:2.63.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.insertPendingSegmentsIntoMetastore(IndexerSQLMetadataStorageCoordinator.java:1394) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:976) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.lambda$allocatePendingSegments$9(IndexerSQLMetadataStorageCoordinator.java:646) ~[druid-server-29.0.1.jar:29.0.1]
at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:161) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:645) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegmentIds(TaskLockbox.java:696) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegments(TaskLockbox.java:466) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForInterval(SegmentAllocationQueue.java:491) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForBatch(SegmentAllocationQueue.java:433) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatch(SegmentAllocationQueue.java:351) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatchesDue(SegmentAllocationQueue.java:266) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
... 1 more
Caused by: java.sql.BatchUpdateException: Batch entry 1 INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES (('my-datasource_2024-06-12T01:00:00.000Z_2024-06-12T02:00:00.000Z_2024-06-12T01:00:00.849Z_5'), ('my-datasource'), ('2024-06-12T08:46:42.875Z'), ('2024-06-12T01:00:00.000Z'), ('2024-06-12T02:00:00.000Z'), ('index_kafka_my-datasource_0f3e4661a8cd3ee_0'), ('my-datasource_2024-05-30T00:00:00.000Z_2024-05-31T00:00:00.000Z_2024-06-12T01:04:25.132Z_6'), ('A10883453F9AA74CDD231465CFF98E73DE3A2864'), ?) was aborted: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists. Call getNextException to see other errors in the batch.
at org.postgresql.jdbc.BatchResultHandler.handleCompletion(BatchResultHandler.java:186) ~[postgresql-42.7.2.jar:42.7.2]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:590) ~[postgresql-42.7.2.jar:42.7.2]
at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:912) ~[postgresql-42.7.2.jar:42.7.2]
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:936) ~[postgresql-42.7.2.jar:42.7.2]
at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1733) ~[postgresql-42.7.2.jar:42.7.2]
at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.skife.jdbi.v2.PreparedBatch.execute(PreparedBatch.java:138) ~[jdbi-2.63.1.jar:2.63.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.insertPendingSegmentsIntoMetastore(IndexerSQLMetadataStorageCoordinator.java:1394) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:976) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.lambda$allocatePendingSegments$9(IndexerSQLMetadataStorageCoordinator.java:646) ~[druid-server-29.0.1.jar:29.0.1]
at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:161) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:645) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegmentIds(TaskLockbox.java:696) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegments(TaskLockbox.java:466) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForInterval(SegmentAllocationQueue.java:491) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForBatch(SegmentAllocationQueue.java:433) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatch(SegmentAllocationQueue.java:351) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatchesDue(SegmentAllocationQueue.java:266) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
... 1 more
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
Detail: Key (sequence_name_prev_id_sha1)=(A10883453F9AA74CDD231465CFF98E73DE3A2864) already exists.
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2725) ~[postgresql-42.7.2.jar:42.7.2]
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2412) ~[postgresql-42.7.2.jar:42.7.2]
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:579) ~[postgresql-42.7.2.jar:42.7.2]
at org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:912) ~[postgresql-42.7.2.jar:42.7.2]
at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:936) ~[postgresql-42.7.2.jar:42.7.2]
at org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1733) ~[postgresql-42.7.2.jar:42.7.2]
at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.apache.commons.dbcp2.DelegatingStatement.executeBatch(DelegatingStatement.java:345) ~[commons-dbcp2-2.0.1.jar:2.0.1]
at org.skife.jdbi.v2.PreparedBatch.execute(PreparedBatch.java:138) ~[jdbi-2.63.1.jar:2.63.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.insertPendingSegmentsIntoMetastore(IndexerSQLMetadataStorageCoordinator.java:1394) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:976) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.lambda$allocatePendingSegments$9(IndexerSQLMetadataStorageCoordinator.java:646) ~[druid-server-29.0.1.jar:29.0.1]
at org.skife.jdbi.v2.DBI.withHandle(DBI.java:281) ~[jdbi-2.63.1.jar:2.63.1]
at org.apache.druid.metadata.SQLMetadataConnector.lambda$retryWithHandle$0(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:129) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:163) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:153) ~[druid-processing-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:151) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.SQLMetadataConnector.retryWithHandle(SQLMetadataConnector.java:161) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.metadata.IndexerSQLMetadataStorageCoordinator.allocatePendingSegments(IndexerSQLMetadataStorageCoordinator.java:645) ~[druid-server-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegmentIds(TaskLockbox.java:696) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.overlord.TaskLockbox.allocateSegments(TaskLockbox.java:466) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForInterval(SegmentAllocationQueue.java:491) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.allocateSegmentsForBatch(SegmentAllocationQueue.java:433) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatch(SegmentAllocationQueue.java:351) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at org.apache.druid.indexing.common.actions.SegmentAllocationQueue.processBatchesDue(SegmentAllocationQueue.java:266) ~[druid-indexing-service-29.0.1.jar:29.0.1]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
... 1 more
```
#### Postgresql error
```
2024-06-12 08:02:13.731 UTC [424455] druid_u@druid_db LOG: execute S_7: INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
2024-06-12 08:02:13.731 UTC [424455] druid_u@druid_db DETAIL: parameters: $1 = 'my-datasource_2024-06-12T07:00:00.000Z_2024-06-12T08:00:00.000Z_2024-06-12T07:03:58.908Z_4', $2 = 'my-datasource', $3 = '2024-06-12T08:02:13.731Z', $4 = '2024-06-12T07:00:00.000Z', $5 = '2024-06-12T08:00:00.000Z', $6 = 'index_kafka_my-datasource_658fb00f8020ee2_0', $7 = 'my-datasource_2024-06-09T03:00:00.000Z_2024-06-09T04:00:00.000Z_2024-06-09T03:00:00.440Z_35', $8 = '53051B2F1036FED82C2E66195320F7A1EC8609C1', $9 = '\x***'
2024-06-12 08:02:13.731 UTC [424455] druid_u@druid_db LOG: execute S_7: INSERT INTO druid_pendingSegments (id, dataSource, created_date, start, "end", sequence_name, sequence_prev_id, sequence_name_prev_id_sha1, payload) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
2024-06-12 08:02:13.731 UTC [424455] druid_u@druid_db DETAIL: parameters: $1 = 'my-datasource_2024-06-12T07:00:00.000Z_2024-06-12T08:00:00.000Z_2024-06-12T07:03:58.908Z_3', $2 = 'my-datasource', $3 = '2024-06-12T08:02:13.731Z', $4 = '2024-06-12T07:00:00.000Z', $5 = '2024-06-12T08:00:00.000Z', $6 = 'index_kafka_my-datasource_658fb00f8020ee2_0', $7 = '', $8 = '53051B2F1036FED82C2E66195320F7A1EC8609C1', $9 = '\x***'
2024-06-12 08:02:13.731 UTC [424455] druid_u@druid_db ERROR: duplicate key value violates unique constraint "druid_pendingsegments_sequence_name_prev_id_sha1_key"
```
=> We can see that the overlord tries to perform 2 consecutive INSERTs with the same sequence_name_prev_id_sha1.
### Temporary solution
If I disable the batching segments allocation option by setting the `druid.indexer.tasklock.batchSegmentAllocation` property to `false`, the problem disappears.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Kafka indexing service duplicate key exception in druid_pendingSegments table (druid)
Posted by "zargor (via GitHub)" <gi...@apache.org>.
zargor commented on issue #16587:
URL: https://github.com/apache/druid/issues/16587#issuecomment-2373324632
@kfaraz is there any news on this matter? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org