[GitHub] [druid] gianm edited a comment on pull request #10383: Fix ingestion failure of pretty-formatted JSON message

[GitBox <gi...@apache.org>]

gianm edited a comment on pull request #10383:
URL: https://github.com/apache/druid/pull/10383#issuecomment-691239531


   > I don't know whether jackson provides anyway to skip the ill-formed text, but I guess there's no such API for a parser.
   > So what's your suggestion on this problem ? @gianm @jihoonson
   
   Hmm, this is definitely a problem. When we're reading JSON from a file, we should be skipping the "lines" that aren't parseable, and marking them as unparseable. I guess that ObjectMapper API was too good to be true 🙂
   
   Maybe we can do one of the following two things.
   
   Option 1: After an error we can manually skip to the line that starts with `{` and begin parsing there. Because the JsonParser has a buffer, we need to make sure we don't miss buffered but unparsed content. We might be able to do that with the `JsonParser.releaseBuffered` method, assuming it works properly after an error.
   
   Option 2: Introduce a `lineSplittable` parameter for the JsonReader like you originally suggested, but it would behave differently in two ways. First: it wouldn't be available on the JsonInputFormat. Instead, it would be automatically set to `true` for batch ingestion and `false` for streaming ingestion. Second: `false` doesn't mean "not line splittable", it just means "we should auto detect". In the `true` case (batch ingestion), the JsonReader will split on lines and parse each line individually. Batch ingestion won't support pretty-printed input, but it never did in the past, and it doesn't seem likely that it's a common requirement anyway. In the `false` case (streaming ingestion), the JsonReader would use the ObjectMapper.readValues API, and it would support all these different kinds of payloads. But if there's an error, it would reject the entire payload. That all-or-nothing behavior seems OK for a streaming ingestion use case.
   
   IMO Option 2 sounds nicer, since it doesn't involve doing weird stuff with the ObjectReader APIs to try to skip unparseable data.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org