You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/11/11 02:47:50 UTC
[GitHub] [druid] Raven888888 opened a new issue #11906: Could not allocate segment to compacted segment
Raven888888 opened a new issue #11906:
URL: https://github.com/apache/druid/issues/11906
### Affected Version
Druid 0.21.0
### Description
I have a druid standalone setup using `small` `single-server` configs.
I have KIS running with segment granularity `hour` and query granularity `none`. So far so good, late data are ingested correctly into the corresponding segments. I then run manual compaction to older segments, which results in 1 big segment.
For instance, I have manually compacted segments for old data in the interval [2021-09-27/2021-10-04] to segment granularity `week`. Query granularity is not set, so it follows back the original one.
```
{
"type": "compact",
"dataSource": "device-health-check-log",
"ioConfig": {
"type": "compact",
"inputSpec": {
"type": "interval",
"interval": "2021-09-27/2021-10-04"
}
},
"segmentGranularity": "week"
}
```
KIS starts to fail (and continues to fail in a loop) when an old data in the compacted interval arrives. Below is the log:
```
2021-11-10T12:43:52,113 WARN [task-runner-0-priority-0] org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver - Cannot allocate segment for timestamp[2021-10-02T09:45:03.000Z], sequenceName[index_kafka_device-health-check-log_aed5bc375a9096d_0].
2021-11-10T12:43:52,113 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception in run() before persisting.
org.apache.druid.java.util.common.ISE: Could not allocate segment for row with timestamp[2021-10-02T09:45:03.000Z]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:670) [druid-indexing-service-0.21.0.jar:0.21.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:268) [druid-indexing-service-0.21.0.jar:0.21.0]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:146) [druid-indexing-service-0.21.0.jar:0.21.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:451) [druid-indexing-service-0.21.0.jar:0.21.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:423) [druid-indexing-service-0.21.0.jar:0.21.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292]
```
I was unable to reproduce the same issue on another druid server running `micro-quickstart` `single-server` configs, version 0.21.1.
On this server, behaviour is as expected, where (realtime) late data correctly creates a new segment accordingly, and KIS task succeed.
Both servers using the exact same KIS specs.
#### Similar issue
#9386
Tried re-compacting the segment, does not help. I had to hard reset supervisor to skip over problematic old data, which is undesired as late data is lost.
Any pointer is greatly appreciated! Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] mchades commented on issue #11906: Could not allocate segment to compacted segment
Posted by GitBox <gi...@apache.org>.
mchades commented on issue #11906:
URL: https://github.com/apache/druid/issues/11906#issuecomment-974746797
It's because another task(like hdoop task) has produced and publish segments of the intervals which within the failured KIS `lateMessageRejectionPeriod` specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] mchades commented on issue #11906: Could not allocate segment to compacted segment
Posted by GitBox <gi...@apache.org>.
mchades commented on issue #11906:
URL: https://github.com/apache/druid/issues/11906#issuecomment-968501962
I ran into the same problem, here is the relevant log of overlord:
```
2021-11-13T07:50:59,908 INFO [qtp870369046-115] io.druid.indexing.common.actions.LocalTaskActionClient - Performing action for task[index_kafka_xxx_online_detail_2a2c20bb975669d_bebmfiod]: SegmentAllocateAction{dataSource='xxx_online_detail', timestamp=2021-11-12T23:50:00.000Z, queryGranularity={type=period, period=PT1M, timeZone=UTC, origin=null}, preferredSegmentGranularity={type=period, period=P1D, timeZone=UTC, origin=null}, sequenceName='index_kafka_xxx_online_detail_2a2c20bb975669d_671', previousSegmentId='null', skipSegmentLineageCheck='true'}
2021-11-13T07:50:59,911 INFO [qtp870369046-115] io.druid.indexing.overlord.TaskLockbox - Task[index_kafka_xxx_online_detail_2a2c20bb975669d_bebmfiod] already present in TaskLock[index_kafka_xxx_online_detail]
2021-11-13T07:50:59,915 WARN [qtp870369046-115] io.druid.metadata.IndexerSQLMetadataStorageCoordinator - Cannot allocate new segment for dataSource[xxx_online_detail], interval[2021-11-12T00:00:00.000Z/2021-11-13T00:00:00.000Z], maxVersion[2021-11-12T00:00:09.097Z]: already have [17] chunks.
2021-11-13T07:50:59,915 ERROR [qtp870369046-115] io.druid.indexing.common.actions.SegmentAllocateAction - Could not allocate pending segment for rowInterval[2021-11-12T23:50:00.000Z/2021-11-12T23:51:00.000Z], segmentInterval[2021-11-12T00:00:00.000Z/2021-11-13T00:00:00.000Z].
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] mchades commented on issue #11906: Could not allocate segment to compacted segment
Posted by GitBox <gi...@apache.org>.
mchades commented on issue #11906:
URL: https://github.com/apache/druid/issues/11906#issuecomment-974746797
It's because another task(like hdoop task) has produced and publish segments of the intervals which within the failured KIS `lateMessageRejectionPeriod` specified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Raven888888 commented on issue #11906: Could not allocate segment to compacted segment
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #11906:
URL: https://github.com/apache/druid/issues/11906#issuecomment-974978584
@mchades I do not have hdoop task ongoing, only KIS.
Plus, my KIS is set `lateMessageRejectionPeriod` = `null`, which means it does not reject any late message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Raven888888 edited a comment on issue #11906: Could not allocate segment to compacted segment
Posted by GitBox <gi...@apache.org>.
Raven888888 edited a comment on issue #11906:
URL: https://github.com/apache/druid/issues/11906#issuecomment-978971215
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Raven888888 closed issue #11906: Could not allocate segment to compacted segment
Posted by GitBox <gi...@apache.org>.
Raven888888 closed issue #11906:
URL: https://github.com/apache/druid/issues/11906
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] Raven888888 commented on issue #11906: Could not allocate segment to compacted segment
Posted by GitBox <gi...@apache.org>.
Raven888888 commented on issue #11906:
URL: https://github.com/apache/druid/issues/11906#issuecomment-978971215
The root cause is having overlapping segments with different granularity. In my case I have:
Segment A: `2021-09-01/2021-10-01` [month]
Segment B: `2021-09-27/2021-10-04` [week]
When there is a late data `2021-09-15` in segment A, that it causes `Could not allocate segment` error as above, I could re-compact the segment A with interval `2021-09-01/2021-10-01` and granularity month. Then, the late data will ingest fine.
*However, this causes segment B to be corrupted.* If there is another late data `2021-10-02` in segment B, same `Could not allocate segment` error occurs. As you might have guessed, if I re-compact segment B, the second late data will work, but segment A will now be corrupted.
Takeaway: do not have 2 overlapping segments. Period.
Closing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org