You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "forzamehlano (via GitHub)" <gi...@apache.org> on 2023/08/07 10:14:29 UTC
[I] Query Failures (InvalidRoaringFormat) (druid)
forzamehlano opened a new issue, #14766:
URL: https://github.com/apache/druid/issues/14766
When running a query, we're seeing it fail due to an error on some of the historical nodes around the InvalidRoaringFormat
### Affected Version
26.0.0
### Description
Our historicals are running with the following config:
`
runtime.properties: |
druid.server.tier=hot
druid.service=druid/historical/hot
druid.plaintextPort=8083
druid.server.http.numThreads=100
druid.processing.buffer.sizeBytes=2000MiB
druid.processing.numMergeBuffers=5
druid.processing.numThreads=19
druid.processing.tmpDir=var/druid/processing
`
And we're seeing the following error on a handful of historicals:
`2023-08-07T09:07:31,673 ERROR [processing-35] org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2 - Exception with one of the sequences!
org.roaringbitmap.InvalidRoaringFormat: I failed to find one of the right cookies. 75124725
at org.roaringbitmap.buffer.ImmutableRoaringArray.<init>(ImmutableRoaringArray.java:48) ~[RoaringBitmap-0.9.0.jar:?]
at org.roaringbitmap.buffer.ImmutableRoaringBitmap.<init>(ImmutableRoaringBitmap.java:1057) ~[RoaringBitmap-0.9.0.jar:?]
at org.apache.druid.segment.data.RoaringBitmapSerdeFactory$ImmutableRoaringBitmapObjectStrategy.fromByteBuffer(RoaringBitmapSerdeFactory.java:80) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.data.RoaringBitmapSerdeFactory$ImmutableRoaringBitmapObjectStrategy.fromByteBuffer(RoaringBitmapSerdeFactory.java:64) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.data.GenericIndexed$BufferIndexed.get(GenericIndexed.java:497) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.column.IndexedUtf8ValueSetIndex.getBitmap(IndexedUtf8ValueSetIndex.java:128) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.column.IndexedUtf8ValueSetIndex.access$100(IndexedUtf8ValueSetIndex.java:40) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.column.IndexedUtf8ValueSetIndex$3$1.next(IndexedUtf8ValueSetIndex.java:230) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.column.IndexedUtf8ValueSetIndex$3$1.next(IndexedUtf8ValueSetIndex.java:205) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.collections.bitmap.RoaringBitmapFactory$1$1.next(RoaringBitmapFactory.java:84) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.collections.bitmap.RoaringBitmapFactory$1$1.next(RoaringBitmapFactory.java:68) ~[druid-processing-26.0.0.jar:26.0.0]
at org.roaringbitmap.buffer.BufferFastAggregation.naive_or(BufferFastAggregation.java:620) ~[RoaringBitmap-0.9.0.jar:?]
at org.roaringbitmap.buffer.BufferFastAggregation.or(BufferFastAggregation.java:713) ~[RoaringBitmap-0.9.0.jar:?]
at org.roaringbitmap.buffer.ImmutableRoaringBitmap.or(ImmutableRoaringBitmap.java:875) ~[RoaringBitmap-0.9.0.jar:?]
at org.apache.druid.collections.bitmap.RoaringBitmapFactory.union(RoaringBitmapFactory.java:142) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.DefaultBitmapResultFactory.unionDimensionValueBitmaps(DefaultBitmapResultFactory.java:73) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.DefaultBitmapResultFactory.unionDimensionValueBitmaps(DefaultBitmapResultFactory.java:25) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.column.SimpleImmutableBitmapIterableIndex.computeBitmapResult(SimpleImmutableBitmapIterableIndex.java:40) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.filter.AndFilter.getBitmapIndex(AndFilter.java:83) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.FilterAnalysis.analyzeFilter(FilterAnalysis.java:107) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.QueryableIndexCursorSequenceBuilder.buildVectorized(QueryableIndexCursorSequenceBuilder.java:213) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.segment.QueryableIndexStorageAdapter.makeVectorCursor(QueryableIndexStorageAdapter.java:236) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.groupby.epinephelinae.vector.VectorGroupByEngine$1.make(VectorGroupByEngine.java:148) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.groupby.epinephelinae.vector.VectorGroupByEngine$1.make(VectorGroupByEngine.java:144) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:39) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:98) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:185) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:44) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:165) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:252) ~[druid-processing-26.0.0.jar:26.0.0]
at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunnerV2$1$1$1.call(GroupByMergingQueryRunnerV2.java:239) ~[druid-processing-26.0.0.jar:26.0.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:251) ~[druid-processing-26.0.0.jar:26.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:829) ~[?:?]
`
Memory/CPU doesn't appear to be an issue. We have tried wiping all historicals and loading the data fresh from deep storage but the error persists.
Not really sure where to go from here...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Query Failures (InvalidRoaringFormat) (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #14766: Query Failures (InvalidRoaringFormat)
URL: https://github.com/apache/druid/issues/14766
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Query Failures (InvalidRoaringFormat) (druid)
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #14766:
URL: https://github.com/apache/druid/issues/14766#issuecomment-2234863879
This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
Re: [I] Query Failures (InvalidRoaringFormat) (druid)
Posted by "forzamehlano (via GitHub)" <gi...@apache.org>.
forzamehlano commented on issue #14766:
URL: https://github.com/apache/druid/issues/14766#issuecomment-1717517019
We're seeing this on 27.0.0 as well.
Any ideas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org