You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@kylin.apache.org by Andras Nagy <an...@gmail.com> on 2019/09/03 14:37:41 UTC
Real-time OLAP - NPE in stream receiver when processing event
Dear All,
We seem to have run into an issue with real-time streaming (in the
stream-receiver). We get a NPE in the StreamingConsumerChannel thread when
an event is received from Kafka:
2019-09-03 13:50:59,341 ERROR [object_movements_channel]
consumer.StreamingConsumerChannel:119 : error happens when save
event:StreamingMessage{data=[1567492920948,
502a3dcb-547f-44bd-9f2b-e1e929ce42b0, d56ecaed-d5dd-460f-9e91-ea3fadf6f369,
e6a1760b-d023-4c92-aab5-e848067bca8c, 0f86f96e-2eb6-4271-b7a1-31a24a0b6e8e,
, , 40, 30, 6, 6, 0, 3, 9, 2019, e16a689e-f893-47a6-ab74-9881010333ec,
2019-09-03, device, c1332a9b-0160-456a-8e43-0a2319efcaa1,
F3DBDC26-B993-4470-9EB7-22E9961B2195, 11.143630707875111],
sourcePosition=org.apache.kylin.stream.source.kafka.KafkaPosition$KafkaPartitionPosition@26b00da,
timestamp=1567518120948, params={}}
java.lang.NullPointerException
at
org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:114)
at
org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:98)
at
org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValueOf(SegmentMemoryStore.java:166)
at
org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValue(SegmentMemoryStore.java:138)
at
org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.index(SegmentMemoryStore.java:101)
at
org.apache.kylin.stream.core.storage.columnar.ColumnarSegmentStore.addEvent(ColumnarSegmentStore.java:134)
at
org.apache.kylin.stream.core.storage.StreamingCubeSegment.addEvent(StreamingCubeSegment.java:152)
at
org.apache.kylin.stream.core.storage.StreamingSegmentManager.addEvent(StreamingSegmentManager.java:173)
at
org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:102)
at java.lang.Thread.run(Thread.java:748)
This is on our snapshot build from master (unchanged) but as I see the
https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java
class has last changed on 8th April so is still the same.
Does anyone have an idea what can be the issue or how we could debug it
further?
Thank you,
Andras
Re: Real-time OLAP - NPE in stream receiver when processing event
Posted by Xiaoxiang Yu <xi...@kyligence.io>.
Hi Andras,
Thank you for your collaboration and bravery. I will be happy if I could hear your suggestion in the future.
----------------
Best wishes,
Xiaoxiang Yu
发件人: Andras Nagy <an...@gmail.com>
日期: 2019年9月4日 星期三 17:04
收件人: Xiaoxiang Yu <xi...@kyligence.io>
抄送: "user@kylin.apache.org" <us...@kylin.apache.org>
主题: Re: Real-time OLAP - NPE in stream receiver when processing event
Hi Xiaoxiang & Chao!
I see, thanks a lot for the reply!
I think we will give Xiaoxiang's patch a test drive and compare the results with what we expect for the count distinct.
Best regards,
Andras
On Wed, Sep 4, 2019 at 4:43 AM Xiaoxiang Yu <xi...@kyligence.io>> wrote:
Hi Andras,
Count distinct(Bitmap) for String type is under development, we hope to release it at 3.0.0-beta. If you are interested, maybe you could help us review this PR (https://github.com/apache/kylin/pull/804).
----------------
Best wishes,
Xiaoxiang Yu
发件人: Chao Long <ch...@gmail.com>>
答复: "user@kylin.apache.org<ma...@kylin.apache.org>" <us...@kylin.apache.org>>
日期: 2019年9月4日 星期三 09:59
收件人: "user@kylin.apache.org<ma...@kylin.apache.org>" <us...@kylin.apache.org>>
主题: Re: Real-time OLAP - NPE in stream receiver when processing event
Hi Andras,
Real-time does not support count-distinct with non-integer datatype column yet, is there a count-distinct measure with non-integer datatype column?
On Tue, Sep 3, 2019 at 10:38 PM Andras Nagy <an...@gmail.com>> wrote:
Dear All,
We seem to have run into an issue with real-time streaming (in the stream-receiver). We get a NPE in the StreamingConsumerChannel thread when an event is received from Kafka:
2019-09-03 13:50:59,341 ERROR [object_movements_channel] consumer.StreamingConsumerChannel:119 : error happens when save event:StreamingMessage{data=[1567492920948, 502a3dcb-547f-44bd-9f2b-e1e929ce42b0, d56ecaed-d5dd-460f-9e91-ea3fadf6f369, e6a1760b-d023-4c92-aab5-e848067bca8c, 0f86f96e-2eb6-4271-b7a1-31a24a0b6e8e, , , 40, 30, 6, 6, 0, 3, 9, 2019, e16a689e-f893-47a6-ab74-9881010333ec, 2019-09-03, device, c1332a9b-0160-456a-8e43-0a2319efcaa1, F3DBDC26-B993-4470-9EB7-22E9961B2195, 11.143630707875111], sourcePosition=org.apache.kylin.stream.source.kafka.KafkaPosition$KafkaPartitionPosition@26b00da, timestamp=1567518120948, params={}}
java.lang.NullPointerException
at org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:114)
at org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:98)
at org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValueOf(SegmentMemoryStore.java:166)
at org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValue(SegmentMemoryStore.java:138)
at org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.index(SegmentMemoryStore.java:101)
at org.apache.kylin.stream.core.storage.columnar.ColumnarSegmentStore.addEvent(ColumnarSegmentStore.java:134)
at org.apache.kylin.stream.core.storage.StreamingCubeSegment.addEvent(StreamingCubeSegment.java:152)
at org.apache.kylin.stream.core.storage.StreamingSegmentManager.addEvent(StreamingSegmentManager.java:173)
at org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:102)
at java.lang.Thread.run(Thread.java:748)
This is on our snapshot build from master (unchanged) but as I see the https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java class has last changed on 8th April so is still the same.
Does anyone have an idea what can be the issue or how we could debug it further?
Thank you,
Andras
Re: Real-time OLAP - NPE in stream receiver when processing event
Posted by Andras Nagy <an...@gmail.com>.
Hi Xiaoxiang & Chao!
I see, thanks a lot for the reply!
I think we will give Xiaoxiang's patch a test drive and compare the results
with what we expect for the count distinct.
Best regards,
Andras
On Wed, Sep 4, 2019 at 4:43 AM Xiaoxiang Yu <xi...@kyligence.io>
wrote:
> Hi Andras,
>
> Count distinct(Bitmap) for String type is under development, we hope to
> release it at 3.0.0-beta. If you are interested, maybe you could help us
> review this PR (https://github.com/apache/kylin/pull/804).
>
>
>
> ----------------
>
> Best wishes,
>
> Xiaoxiang Yu
>
>
>
>
>
> *发件人**: *Chao Long <ch...@gmail.com>
> *答复**: *"user@kylin.apache.org" <us...@kylin.apache.org>
> *日期**: *2019年9月4日 星期三 09:59
> *收件人**: *"user@kylin.apache.org" <us...@kylin.apache.org>
> *主题**: *Re: Real-time OLAP - NPE in stream receiver when processing event
>
>
>
> Hi Andras,
>
> Real-time does not support count-distinct with non-integer datatype
> column yet, is there a count-distinct measure with non-integer datatype
> column?
>
>
>
> On Tue, Sep 3, 2019 at 10:38 PM Andras Nagy <an...@gmail.com>
> wrote:
>
> Dear All,
>
>
>
> We seem to have run into an issue with real-time streaming (in the
> stream-receiver). We get a NPE in the StreamingConsumerChannel thread when
> an event is received from Kafka:
>
> 2019-09-03 13:50:59,341 ERROR [object_movements_channel]
> consumer.StreamingConsumerChannel:119 : error happens when save
> event:StreamingMessage{data=[1567492920948,
> 502a3dcb-547f-44bd-9f2b-e1e929ce42b0, d56ecaed-d5dd-460f-9e91-ea3fadf6f369,
> e6a1760b-d023-4c92-aab5-e848067bca8c, 0f86f96e-2eb6-4271-b7a1-31a24a0b6e8e,
> , , 40, 30, 6, 6, 0, 3, 9, 2019, e16a689e-f893-47a6-ab74-9881010333ec,
> 2019-09-03, device, c1332a9b-0160-456a-8e43-0a2319efcaa1,
> F3DBDC26-B993-4470-9EB7-22E9961B2195, 11.143630707875111],
> sourcePosition=org.apache.kylin.stream.source.kafka.KafkaPosition$KafkaPartitionPosition@26b00da,
> timestamp=1567518120948, params={}}
> java.lang.NullPointerException
> at
> org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:114)
> at
> org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:98)
> at
> org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValueOf(SegmentMemoryStore.java:166)
> at
> org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValue(SegmentMemoryStore.java:138)
> at
> org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.index(SegmentMemoryStore.java:101)
> at
> org.apache.kylin.stream.core.storage.columnar.ColumnarSegmentStore.addEvent(ColumnarSegmentStore.java:134)
> at
> org.apache.kylin.stream.core.storage.StreamingCubeSegment.addEvent(StreamingCubeSegment.java:152)
> at
> org.apache.kylin.stream.core.storage.StreamingSegmentManager.addEvent(StreamingSegmentManager.java:173)
> at
> org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:102)
> at java.lang.Thread.run(Thread.java:748)
>
>
> This is on our snapshot build from master (unchanged) but as I see the
> https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java
> class has last changed on 8th April so is still the same.
>
> Does anyone have an idea what can be the issue or how we could debug it
> further?
>
>
>
> Thank you,
>
> Andras
>
>
Re: Real-time OLAP - NPE in stream receiver when processing event
Posted by Xiaoxiang Yu <xi...@kyligence.io>.
Hi Andras,
Count distinct(Bitmap) for String type is under development, we hope to release it at 3.0.0-beta. If you are interested, maybe you could help us review this PR (https://github.com/apache/kylin/pull/804).
----------------
Best wishes,
Xiaoxiang Yu
发件人: Chao Long <ch...@gmail.com>
答复: "user@kylin.apache.org" <us...@kylin.apache.org>
日期: 2019年9月4日 星期三 09:59
收件人: "user@kylin.apache.org" <us...@kylin.apache.org>
主题: Re: Real-time OLAP - NPE in stream receiver when processing event
Hi Andras,
Real-time does not support count-distinct with non-integer datatype column yet, is there a count-distinct measure with non-integer datatype column?
On Tue, Sep 3, 2019 at 10:38 PM Andras Nagy <an...@gmail.com>> wrote:
Dear All,
We seem to have run into an issue with real-time streaming (in the stream-receiver). We get a NPE in the StreamingConsumerChannel thread when an event is received from Kafka:
2019-09-03 13:50:59,341 ERROR [object_movements_channel] consumer.StreamingConsumerChannel:119 : error happens when save event:StreamingMessage{data=[1567492920948, 502a3dcb-547f-44bd-9f2b-e1e929ce42b0, d56ecaed-d5dd-460f-9e91-ea3fadf6f369, e6a1760b-d023-4c92-aab5-e848067bca8c, 0f86f96e-2eb6-4271-b7a1-31a24a0b6e8e, , , 40, 30, 6, 6, 0, 3, 9, 2019, e16a689e-f893-47a6-ab74-9881010333ec, 2019-09-03, device, c1332a9b-0160-456a-8e43-0a2319efcaa1, F3DBDC26-B993-4470-9EB7-22E9961B2195, 11.143630707875111], sourcePosition=org.apache.kylin.stream.source.kafka.KafkaPosition$KafkaPartitionPosition@26b00da, timestamp=1567518120948, params={}}
java.lang.NullPointerException
at org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:114)
at org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:98)
at org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValueOf(SegmentMemoryStore.java:166)
at org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValue(SegmentMemoryStore.java:138)
at org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.index(SegmentMemoryStore.java:101)
at org.apache.kylin.stream.core.storage.columnar.ColumnarSegmentStore.addEvent(ColumnarSegmentStore.java:134)
at org.apache.kylin.stream.core.storage.StreamingCubeSegment.addEvent(StreamingCubeSegment.java:152)
at org.apache.kylin.stream.core.storage.StreamingSegmentManager.addEvent(StreamingSegmentManager.java:173)
at org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:102)
at java.lang.Thread.run(Thread.java:748)
This is on our snapshot build from master (unchanged) but as I see the https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java class has last changed on 8th April so is still the same.
Does anyone have an idea what can be the issue or how we could debug it further?
Thank you,
Andras
Re: Real-time OLAP - NPE in stream receiver when processing event
Posted by Chao Long <ch...@gmail.com>.
Hi Andras,
Real-time does not support count-distinct with non-integer datatype
column yet, is there a count-distinct measure with non-integer datatype
column?
On Tue, Sep 3, 2019 at 10:38 PM Andras Nagy <an...@gmail.com>
wrote:
> Dear All,
>
> We seem to have run into an issue with real-time streaming (in the
> stream-receiver). We get a NPE in the StreamingConsumerChannel thread when
> an event is received from Kafka:
>
> 2019-09-03 13:50:59,341 ERROR [object_movements_channel]
> consumer.StreamingConsumerChannel:119 : error happens when save
> event:StreamingMessage{data=[1567492920948,
> 502a3dcb-547f-44bd-9f2b-e1e929ce42b0, d56ecaed-d5dd-460f-9e91-ea3fadf6f369,
> e6a1760b-d023-4c92-aab5-e848067bca8c, 0f86f96e-2eb6-4271-b7a1-31a24a0b6e8e,
> , , 40, 30, 6, 6, 0, 3, 9, 2019, e16a689e-f893-47a6-ab74-9881010333ec,
> 2019-09-03, device, c1332a9b-0160-456a-8e43-0a2319efcaa1,
> F3DBDC26-B993-4470-9EB7-22E9961B2195, 11.143630707875111],
> sourcePosition=org.apache.kylin.stream.source.kafka.KafkaPosition$KafkaPartitionPosition@26b00da,
> timestamp=1567518120948, params={}}
> java.lang.NullPointerException
> at
> org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:114)
> at
> org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:98)
> at
> org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValueOf(SegmentMemoryStore.java:166)
> at
> org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValue(SegmentMemoryStore.java:138)
> at
> org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.index(SegmentMemoryStore.java:101)
> at
> org.apache.kylin.stream.core.storage.columnar.ColumnarSegmentStore.addEvent(ColumnarSegmentStore.java:134)
> at
> org.apache.kylin.stream.core.storage.StreamingCubeSegment.addEvent(StreamingCubeSegment.java:152)
> at
> org.apache.kylin.stream.core.storage.StreamingSegmentManager.addEvent(StreamingSegmentManager.java:173)
> at
> org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:102)
> at java.lang.Thread.run(Thread.java:748)
>
>
> This is on our snapshot build from master (unchanged) but as I see the
> https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java
> class has last changed on 8th April so is still the same.
>
> Does anyone have an idea what can be the issue or how we could debug it
> further?
>
> Thank you,
> Andras
>