You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/06/18 10:07:13 UTC
[GitHub] [druid] yuanlihan edited a comment on issue #9967: Optimizing incremental-index ingestion using off-heap key/value map (OakMap)
yuanlihan edited a comment on issue #9967:
URL: https://github.com/apache/druid/issues/9967#issuecomment-645917973
Hi @liran-funaro, I would like to post feedback about performance improvement here later.
But currently before performing further performance testing, I suffer from NPE errors for `timeseries` queries which scan memory data of latest 10 minutes. Note that I applied the patch to branch `0.18.1` and built successfully except the benchmark module.
```
"timeseries_ds_name_[2020-06-18T09:07:46.039Z/2020-06-18T09:17:46.039Z]@12302" daemon prio=5 tid=0x152 nid=NA runnable
java.lang.Thread.State: RUNNABLE
at com.yahoo.oak.Slice.getAddress(Slice.java:172)
at com.yahoo.oak.KeyBuffer.getAddress(KeyBuffer.java:9)
at com.yahoo.oak.UnscopedBuffer.getAddress(UnscopedBuffer.java:132)
at org.apache.druid.segment.incremental.OakIncrementalIndexRow.<init>(OakIncrementalIndexRow.java:52)
at org.apache.druid.segment.incremental.OakIncrementalIndex$OakFactsHolder.lambda$transformStreamIterator$1(OakIncrementalIndex.java:482)
at org.apache.druid.segment.incremental.OakIncrementalIndex$OakFactsHolder$$Lambda$389.770801012.apply(Unknown Source:-1)
at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at org.apache.druid.segment.incremental.IncrementalIndexStorageAdapter$IncrementalIndexCursor.reset(IncrementalIndexStorageAdapter.java:360)
at org.apache.druid.segment.incremental.IncrementalIndexStorageAdapter$IncrementalIndexCursor.<init>(IncrementalIndexStorageAdapter.java:264)
at org.apache.druid.segment.incremental.IncrementalIndexStorageAdapter.lambda$makeCursors$0(IncrementalIndexStorageAdapter.java:213)
at org.apache.druid.segment.incremental.IncrementalIndexStorageAdapter$$Lambda$381.1128483784.apply(Unknown Source:-1)
at org.apache.druid.java.util.common.guava.MappingAccumulator.accumulate(MappingAccumulator.java:40)
at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44)
at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43)
at org.apache.druid.java.util.common.guava.FilteredSequence.accumulate(FilteredSequence.java:45)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.Sequence.toList(Sequence.java:85)
at org.apache.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:124)
at org.apache.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:114)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.druid.java.util.common.concurrent.DirectExecutorService.execute(DirectExecutorService.java:81)
at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134)
at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:58)
at org.apache.druid.query.ChainedExecutionQueryRunner$1.lambda$make$0(ChainedExecutionQueryRunner.java:112)
at org.apache.druid.query.ChainedExecutionQueryRunner$1$$Lambda$379.804059870.apply(Unknown Source:-1)
at com.google.common.collect.Iterators$8.transform(Iterators.java:794)
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
at com.google.common.collect.Iterators.addAll(Iterators.java:357)
at com.google.common.collect.Lists.newArrayList(Lists.java:147)
at com.google.common.collect.Lists.newArrayList(Lists.java:129)
at org.apache.druid.query.ChainedExecutionQueryRunner$1.make(ChainedExecutionQueryRunner.java:104)
at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:39)
at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.LazySequence.accumulate(LazySequence.java:40)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:74)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.query.spec.SpecificSegmentQueryRunner$1.accumulate(SpecificSegmentQueryRunner.java:87)
at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50)
at org.apache.druid.query.spec.SpecificSegmentQueryRunner.doNamed(SpecificSegmentQueryRunner.java:171)
at org.apache.druid.query.spec.SpecificSegmentQueryRunner.access$100(SpecificSegmentQueryRunner.java:44)
at org.apache.druid.query.spec.SpecificSegmentQueryRunner$2.wrap(SpecificSegmentQueryRunner.java:153)
at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45)
at org.apache.druid.java.util.common.guava.Sequence.toList(Sequence.java:85)
at org.apache.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:124)
at org.apache.druid.query.ChainedExecutionQueryRunner$1$1.call(ChainedExecutionQueryRunner.java:114)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:247)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
```
and log
```
2020-06-18T17:32:49,884 ERROR [timeseries_shopee_sto_apm__tracking_http_live_[2020-06-18T09:23:46.039Z/2020-06-18T09:33:46.039Z]] org.apache.druid.quer
y.ChainedExecutionQueryRunner - Exception with one of the sequences! (java.lang.NullPointerException)
```
The NPE error occurs when the `writeBuffer` is null. And I would like to provide more info to address this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org