You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/07/23 21:49:06 UTC

[GitHub] [druid] druid-matt opened a new issue #10207: druid-time-min-max ingestion error

druid-matt opened a new issue #10207:
URL: https://github.com/apache/druid/issues/10207


   Reporting bug in the druid-time-min-max extenstion. It's returning TimestampAggregatorFactory from getCombiningFactory() of both TimestampMinAggregatorFactory and TimestampMaxAggregatorFactory. That's not proper, because TimestampAggregatorFactory is not JSON-round-trippable, and in general we expect the combining factories to be JSON-round-trippable.
    
   User is having trouble with druid-time-min-max extension
   He has metrics that looks like:
   ```         
   {
               "type": "timeMin",
               "name": "first_seen",
               "fieldName": "timestamp"
             },
             {
               "type": "timeMax",
               "name": "last_seen",
               "fieldName": "timestamp"
             }
   ```
   the extension seem to work generally but when he looks at the logs he sometimes see this during ingestion - not every time but just some time:
   ```
   20-07-22T18:30:12,176 WARN [[index_kafka_REDACTED_22eff7a102317ab_jipjfaed]-appenderator-persist] org.apache.druid.segment.IndexIO - Failed to load metadata for segment [var/druid/task/index_kafka_REDACTED_22eff7a102317ab_jipjfaed/work/persist/REDACTED_2020-07-22T00:00:00.000Z_2020-07-23T00:00:00.000Z_2020-07-22T17:54:42.013Z/6]
   com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Could not resolve type id 'TimestampAggregatorFactory' as a subtype of `org.apache.druid.query.aggregation.AggregatorFactory`: known type ids = [HLLSketchBuild, HLLSketchMerge, arrayOfDoublesSketch, cardinality, count, doubleFirst, doubleLast, doubleMax, doubleMean, doubleMin, doubleSum, filtered, floatFirst, floatLast, floatMax, floatMin, floatSum, histogram, hyperUnique, javascript, longFirst, longLast, longMax, longMin, longSum, quantilesDoublesSketch, quantilesDoublesSketchMerge, sketchBuild, sketchMerge, stringFirst, stringFirstFold, stringLast, stringLastFold, thetaSketch, timeMax, timeMin] (for POJO property 'aggregators')
    at [Source: (byte[])"{"container":{},"aggregators":[{"type":"longSum","name":"count","fieldName":"count","expression":null},{"type":"longSum","name":"data_bytes_from_dst_sum","fieldName":"data_bytes_from_dst_sum","expression":null},{"type":"longSum","name":"data_bytes_from_src_sum","fieldName":"data_bytes_from_src_sum","expression":null},{"type":"TimestampAggregatorFactory","name":"first_seen","fieldName":"first_seen","timeFormat":null},{"type":"TimestampAggregatorFactory","name":"last_seen","fieldName":"last_seen","[truncated 380 bytes]; line: 1, column: 328] (through reference chain: org.apache.druid.segment.Metadata["aggregators"]->java.lang.Object[][3])
   	at com.fasterxml.jackson.databind.exc.InvalidTypeIdException.from(InvalidTypeIdException.java:43) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.DeserializationContext.invalidTypeIdException(DeserializationContext.java:1758) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownTypeId(DeserializationContext.java:1265) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._handleUnknownTypeId(TypeDeserializerBase.java:290) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.jsontype.impl.TypeDeserializerBase._findDeserializer(TypeDeserializerBase.java:162) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:113) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:97) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:254) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:197) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.deser.std.ObjectArrayDeserializer.deserialize(ObjectArrayDeserializer.java:21) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:530) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:528) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:417) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1287) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:326) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:159) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4202) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3266) ~[jackson-databind-2.10.1.jar:2.10.1]
   	at org.apache.druid.segment.IndexIO$V9IndexLoader.load(IndexIO.java:570) [druid-processing-0.17.1.jar:0.17.1]
   	at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:194) [druid-processing-0.17.1.jar:0.17.1]
   	at org.apache.druid.segment.IndexIO.loadIndex(IndexIO.java:185) [druid-processing-0.17.1.jar:0.17.1]
   	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.persistHydrant(AppenderatorImpl.java:1348) [druid-server-0.17.1.jar:0.17.1]
   	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.access$100(AppenderatorImpl.java:103) [druid-server-0.17.1.jar:0.17.1]
   	at org.apache.druid.segment.realtime.appenderator.AppenderatorImpl$3.call(AppenderatorImpl.java:544) [druid-server-0.17.1.jar:0.17.1]
   	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_252]
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_252]
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_252]
   	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_252]
   ```
   
   reported via https://the-asf.slack.com/archives/CJ8D1JTB8/p1595442889073500
   see https://stackoverflow.com/c/imply/questions/1501/1502?noredirect=1#comment1456_1502


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org


[GitHub] [druid] clintropolis closed issue #10207: druid-time-min-max ingestion error

Posted by GitBox <gi...@apache.org>.
clintropolis closed issue #10207:
URL: https://github.com/apache/druid/issues/10207


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org