You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/02/10 08:43:43 UTC
[GitHub] quenlang opened a new issue #7047: ingestion data with quantiles
sketch module throw exception in 0.12.3
quenlang opened a new issue #7047: ingestion data with quantiles sketch module throw exception in 0.12.3
URL: https://github.com/apache/incubator-druid/issues/7047
@leventov @jon-wei @AlexanderSaydakov
I had met a exception when i use quantiles sketch aggregation in ```druid 0.12.3```.
The exception as below:
```
2019-02-10T03:46:18,582 WARN [appenderator_merge_0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Failed to push merged index for segment[BRS_WECHAT_APPLET_MIN_BAK_2019-02-04T12:00:00.000Z_2019-02-04T18:00:00.000Z_2019-02-10T01:46:35.227Z].
java.lang.NullPointerException
at io.druid.query.aggregation.datasketches.quantiles.DoublesSketchAggregatorFactory$1.compare(DoublesSketchAggregatorFactory.java:129) ~[?:?]
at io.druid.query.aggregation.datasketches.quantiles.DoublesSketchAggregatorFactory$1.compare(DoublesSketchAggregatorFactory.java:125) ~[?:?]
at io.druid.query.aggregation.datasketches.quantiles.DoublesSketchObjectStrategy.compare(DoublesSketchObjectStrategy.java:37) ~[?:?]
at io.druid.query.aggregation.datasketches.quantiles.DoublesSketchObjectStrategy.compare(DoublesSketchObjectStrategy.java:29) ~[?:?]
at io.druid.segment.data.GenericIndexedWriter.write(GenericIndexedWriter.java:212) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.serde.LargeColumnSupportedComplexColumnSerializer.serialize(LargeColumnSupportedComplexColumnSerializer.java:100) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.IndexMergerV9.mergeIndexesAndWriteColumns(IndexMergerV9.java:462) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:209) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:837) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:710) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:688) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:659) ~[druid-server-0.12.3.jar:0.12.3]
at io.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$0(AppenderatorImpl.java:563) ~[druid-server-0.12.3.jar:0.12.3]
at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) [guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
2019-02-10T03:46:18,613 ERROR [publish-0] io.druid.indexing.kafka.KafkaIndexTask - Error while publishing segments for sequence[SequenceMetadata{sequenceName='index_kafka_BRS_WECHAT_APPLET_MIN_BAK_305445d71cf9d8f_0', sequenceId=0, startOffsets={0=13195, 1=82987733, 2=13773273, 3=5077, 4=59584, 5=198, 6=12182050, 7=213853278, 8=606, 9=59971, 10=22860, 11=131652, 12=31679708, 13=156731, 14=22460033, 15=51988}, endOffsets={0=13255, 1=83041632, 2=20172715, 3=5119, 4=59584, 5=198, 6=14445404, 7=213853439, 8=606, 9=110911, 10=28108, 11=131652, 12=42833619, 13=174499, 14=26118960, 15=52000}, assignments=[], sentinel=false, checkpointed=true}]
java.lang.NullPointerException
at io.druid.query.aggregation.datasketches.quantiles.DoublesSketchAggregatorFactory$1.compare(DoublesSketchAggregatorFactory.java:129) ~[?:?]
at io.druid.query.aggregation.datasketches.quantiles.DoublesSketchAggregatorFactory$1.compare(DoublesSketchAggregatorFactory.java:125) ~[?:?]
at io.druid.query.aggregation.datasketches.quantiles.DoublesSketchObjectStrategy.compare(DoublesSketchObjectStrategy.java:37) ~[?:?]
at io.druid.query.aggregation.datasketches.quantiles.DoublesSketchObjectStrategy.compare(DoublesSketchObjectStrategy.java:29) ~[?:?]
at io.druid.segment.data.GenericIndexedWriter.write(GenericIndexedWriter.java:212) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.serde.LargeColumnSupportedComplexColumnSerializer.serialize(LargeColumnSupportedComplexColumnSerializer.java:100) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.IndexMergerV9.mergeIndexesAndWriteColumns(IndexMergerV9.java:462) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.IndexMergerV9.makeIndexFiles(IndexMergerV9.java:209) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.IndexMergerV9.merge(IndexMergerV9.java:837) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:710) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.IndexMergerV9.mergeQueryableIndex(IndexMergerV9.java:688) ~[druid-processing-0.12.3.jar:0.12.3]
at io.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:659) ~[druid-server-0.12.3.jar:0.12.3]
at io.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$0(AppenderatorImpl.java:563) ~[druid-server-0.12.3.jar:0.12.3]
at com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) ~[guava-16.0.1.jar:?]
at com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_60]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_60]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]
2019-02-10T03:46:18,618 INFO [task-runner-0-priority-0] io.druid.segment.realtime.appenderator.AppenderatorImpl - Shutting down immediately...
```
And my ingestion task spec file looks like:
```
{
"type": "kafka",
"dataSchema": {
"dataSource": "BRS_WECHAT_APPLET_MIN",
"parser": {
"type": "string",
"parseSpec": {
"format": "json",
"timestampSpec": {
"column": "timestamp",
"format": "millis"
},
"dimensionsSpec": {
"dimensions": [
"mp_id",
"application_id",
"instance_id",
"action_name",
"message_type",
"from_path",
"path",
"open_path",
"close_path",
"scene",
"country_id",
"region_id",
"city_id",
"carrier_id",
"error_message",
"error_filename",
"request_method",
"host",
"uri",
"network_type",
"wechat_version",
"route_chain",
"uid",
"http_code",
"system",
"ip",
"device_type",
"agreement_id",
"protocol"
]
}
}
},
"metricsSpec": [
{
"name": "server_count",
"fieldName": "server_count",
"type": "longSum"
},
{
"name": "quit_count",
"fieldName": "quit_count",
"type": "longSum"
},
{
"name": "on_ready",
"fieldName": "on_ready",
"type": "doubleSum"
},
{
"name": "custom_time",
"fieldName": "custom_time",
"type": "longSum"
},
{
"name": "first_response_time",
"fieldName": "first_response_time",
"type": "doubleSum"
},
{
"name": "response_time",
"fieldName": "response_time",
"type": "doubleSum"
},
{
"name": "application_server_time",
"fieldName": "application_server_time",
"type": "doubleSum"
},
{
"name": "network_time",
"fieldName": "network_time",
"type": "doubleSum"
},
{
"name": "callback_time",
"fieldName": "callback_time",
"type": "longSum"
},
{
"name": "bytes_sent",
"fieldName": "bytes_sent",
"type": "longSum"
},
{
"name": "bytes_received",
"fieldName": "bytes_received",
"type": "longSum"
},
{
"name": "msg_error_pv",
"fieldName": "msg_error_pv",
"type": "longSum"
},
{
"name": "file_error_pv",
"fieldName": "file_error_pv",
"type": "longSum"
},
{
"name": "count",
"fieldName": "count",
"type": "longSum"
},
{
"name": "on_ready_count",
"fieldName": "on_ready_count",
"type": "longSum"
},
{
"name": "open_count",
"fieldName": "open_count",
"type": "longSum"
},
{
"name": "net_count",
"fieldName": "net_count",
"type": "longSum"
},
{
"name": "net_error_count",
"fieldName": "net_error_count",
"type": "longSum"
},
{
"name": "js_error_count",
"fieldName": "js_error_count",
"type": "longSum"
},
{
"name": "slow_count",
"fieldName": "slow_count",
"type": "longSum"
},
{
"name": "net_slow_count",
"fieldName": "net_slow_count",
"type": "longSum"
},
{
"name": "uv",
"fieldName": "uid",
"type": "thetaSketch"
},
{
"name": "on_ready_histogram",
"fieldName": "on_ready",
"type": "approxHistogramFold",
"resolution": 50,
"numBuckets": 7,
"lowerLimit": 0
},
{
"name": "first_response_time_histogram",
"fieldName": "first_response_time",
"type": "approxHistogramFold",
"resolution": 50,
"numBuckets": 7,
"lowerLimit": 0
},
{
"name": "response_time_histogram",
"fieldName": "response_time",
"type": "approxHistogramFold",
"resolution": 50,
"numBuckets": 7,
"lowerLimit": 0
},
{
"name": "network_time_histogram",
"fieldName": "network_time",
"type": "approxHistogramFold",
"resolution": 50,
"numBuckets": 7,
"lowerLimit": 0
},
{
"name": "application_server_time_histogram",
"fieldName": "application_server_time",
"type": "approxHistogramFold",
"resolution": 50,
"numBuckets": 7,
"lowerLimit": 0
},
{
"type" : "quantilesDoublesSketch",
"name" : "on_ready_sketch",
"fieldName" : "on_ready",
"k": 256
},
{
"type" : "quantilesDoublesSketch",
"name" : "first_response_time_sketch",
"fieldName" : "first_response_time",
"k": 256
},
{
"type" : "quantilesDoublesSketch",
"name" : "response_time_sketch",
"fieldName" : "response_time",
"k": 256
},
{
"type" : "quantilesDoublesSketch",
"name" : "network_time_sketch",
"fieldName" : "network_time",
"k": 256
},
{
"type" : "quantilesDoublesSketch",
"name" : "application_server_time_sketch",
"fieldName" : "application_server_time",
"k": 256
}
],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "SIX_HOUR",
"queryGranularity": "MINUTE"
}
},
"tuningConfig": {
"type": "kafka",
"maxRowsPerSegment": 2500000
},
"ioConfig": {
"topic": "drd-mp-web-applet",
"consumerProperties": {
"bootstrap.servers": "kafka1:9092"
},
"taskCount": 1,
"taskDuration": "PT6H",
"replicas": 1,
"useEarliestOffset": true
}
}
```
I have no idea with this exception, can you give me some advices ? Thank you very much.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org