[GitHub] [druid] gianm edited a comment on issue #10259: Kafka ingestion fails to parse multiple-line messages in 0.19

[GitBox <gi...@apache.org>]

gianm edited a comment on issue #10259:
URL: https://github.com/apache/druid/issues/10259#issuecomment-690521098


   We should be able to determine automatically if a Kafka message or a text file is line-delimited JSON or not. Check out the Jackson ObjectMapper.readValues method: it might already do what we need:
   
   ```
        * Sequence can be either wrapped or unwrapped root-level sequence:
        * wrapped means that the elements are enclosed in JSON Array;
        * and unwrapped that elements are directly accessed at main level.
        * Assumption is that iff the first token of the document is
        * <code>START_ARRAY</code>, we have a wrapped sequence; otherwise
        * unwrapped. For wrapped sequences, leading <code>START_ARRAY</code>
        * is skipped, so that for both cases, underlying {@link JsonParser}
        * will point to what is expected to be the first token of the first
        * element.
   ```
   
   It sounds like it could read a JSON array of objects ("wrapped"), newline delimited objects ("unwrapped"), a pretty-printed object, or even concatenated objects.
   
   To make this work, we might need to have JsonReader extend InputEntityReader directly, instead of TextReader.
   
   @FrankChen021 @jihoonson what do you think?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org