[GitHub] [incubator-druid] clintropolis opened a new issue #7252: Kafka Tasks unable restore persisted segments properly

[GitBox <gi...@apache.org>]

clintropolis opened a new issue #7252: Kafka Tasks unable restore persisted segments properly
URL: https://github.com/apache/incubator-druid/issues/7252
 
 
   ### Affected Version
   
   All versions after #6431
   
   ### Description
   
   Kakfa indexing tasks (likely kinesis as well) are unable to restore properly if segments have been incrementally persisted because they cannot deserialize `SequenceMetadata`, resulting in an error in the form of:
   
   ```
   2019-03-12T02:18:50,238 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner - Encountered exception while running task.
   com.fasterxml.jackson.databind.JsonMappingException: Unrecognized Type: [null]
       at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCache2(DeserializerCache.java:269) ~[jackson-databind-2.6.7.jar:2.6.7]
       at com.fasterxml.jackson.databind.deser.DeserializerCache._createAndCacheValueDeserializer(DeserializerCache.java:244) ~[jackson-databind-2.6.7.jar:2.6.7]
       at com.fasterxml.jackson.databind.deser.DeserializerCache.findValueDeserializer(DeserializerCache.java:142) ~[jackson-databind-2.6.7.jar:2.6.7]
       at com.fasterxml.jackson.databind.DeserializationContext.findContextualValueDeserializer(DeserializationContext.java:428) ~[jackson-databind-2.6.7.jar:2.6.7]
       at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:164) ~[jackson-databind-2.6.7.jar:2.6.7]
       at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.createContextual(CollectionDeserializer.java:25) ~[jackson-databind-2.6.7.jar:2.6.7]
       at com.fasterxml.jackson.databind.DeserializationContext.handleSecondaryContextualization(DeserializationContext.java:669) ~[jackson-databind-2.6.7.jar:2.6.7]
       at com.fasterxml.jackson.databind.DeserializationContext.findRootValueDeserializer(DeserializationContext.java:466) ~[jackson-databind-2.6.7.jar:2.6.7]
       at com.fasterxml.jackson.databind.ObjectMapper._findRootDeserializer(ObjectMapper.java:3838) ~[jackson-databind-2.6.7.jar:2.6.7]
       at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3732) ~[jackson-databind-2.6.7.jar:2.6.7]
       at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2639) ~[jackson-databind-2.6.7.jar:2.6.7]
       at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.restoreSequences(SeekableStreamIndexTaskRunner.java:942) ~[druid-indexing-service-0.14.0-iap-pre3.jar:0.14.0-iap-pre3]
   ...
   ```
   
   The issue is that `SequenceMetadata` is a private class using generic types defined in the outer class, `SeekableStreamIndexTaskRunner<PartitionIdType, SequenceOffsetType>`, which determine the map types which store offsets in the persisted sequence metadata json file. The file is written correctly, but the `TypeReference` given to the object mapper is not able to properly read it during deserialization, resulting in the above error. 
   
   Fix underway, but untangling this is sort of messy unfortunately.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org