You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2021/01/20 19:39:15 UTC
[GitHub] [druid] gianm opened a new issue #10784: ParseExceptions from hasNext not caught and handled properly
gianm opened a new issue #10784:
URL: https://github.com/apache/druid/issues/10784
FilteringCloseableInputRowIterator is the chokepoint where all calls to `parseExceptionHandler.handle` get made when they come from issues in parsing the original input. It has logic to catch and handle parse exceptions from `next()` of the underlying Iterator, but it doesn't catch exceptions in `hasNext()`.
I noticed this when debugging the error below. Something in the way the iterator was constructed caused `hasNext()` to throw an exception. This is totally valid, and it's hard for FilteringCloseableInputRowIterator to handle properly, because what should it do if `hasNext()` throws an error? Just blindly call `next()`? That doesn't seem like a good idea, since it's outside of the general contract of Iterator.
IMO, the best solution is not to use an `Iterator<InputRow>` here. Maybe an `Iterator<Pair<ParseException, InputRow>>` so we don't need to mess about with catching exceptions during iteration. Or maybe a new interface.
```
org.apache.druid.java.util.common.parsers.ParseException: Unable to parse row [<deleted>]
at org.apache.druid.java.util.common.parsers.JSONPathParser.parseToMap(JSONPathParser.java:74) ~[druid-core-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.data.input.impl.StringInputRowParser.parseString(StringInputRowParser.java:155) ~[druid-core-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.data.input.impl.StringInputRowParser.buildStringKeyMap(StringInputRowParser.java:119) ~[druid-core-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.data.input.impl.StringInputRowParser.parseBatch(StringInputRowParser.java:80) ~[druid-core-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:50) ~[druid-processing-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:31) ~[druid-processing-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.indexing.seekablestream.StreamChunkParser.lambda$parseWithParser$0(StreamChunkParser.java:111) ~[druid-indexing-service-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at com.google.common.collect.Iterators$8.transform(Iterators.java:794) ~[guava-16.0.1.jar:?]
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) ~[guava-16.0.1.jar:?]
at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48) ~[guava-16.0.1.jar:?]
at com.google.common.collect.Iterators$5.hasNext(Iterators.java:543) ~[guava-16.0.1.jar:?]
at org.apache.druid.java.util.common.CloseableIterators$1.hasNext(CloseableIterators.java:71) ~[druid-core-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.indexing.common.task.FilteringCloseableInputRowIterator.hasNext(FilteringCloseableInputRowIterator.java:62) ~[druid-indexing-service-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at com.google.common.collect.Iterators.addAll(Iterators.java:356) ~[guava-16.0.1.jar:?]
at com.google.common.collect.Lists.newArrayList(Lists.java:147) ~[guava-16.0.1.jar:?]
at org.apache.druid.indexing.seekablestream.StreamChunkParser.parseWithParser(StreamChunkParser.java:119) ~[druid-indexing-service-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.indexing.seekablestream.StreamChunkParser.parse(StreamChunkParser.java:102) ~[druid-indexing-service-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.runInternal(SeekableStreamIndexTaskRunner.java:625) ~[druid-indexing-service-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTaskRunner.run(SeekableStreamIndexTaskRunner.java:268) [druid-indexing-service-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.indexing.seekablestream.SeekableStreamIndexTask.run(SeekableStreamIndexTask.java:146) [druid-indexing-service-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.indexing.overlord.ThreadingTaskRunner$1.call(ThreadingTaskRunner.java:211) [druid-indexing-service-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at org.apache.druid.indexing.overlord.ThreadingTaskRunner$1.call(ThreadingTaskRunner.java:151) [druid-indexing-service-0.21.0-SNAPSHOT.jar:0.21.0-SNAPSHOT]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10784: ParseExceptions from hasNext not caught and handled properly
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10784:
URL: https://github.com/apache/druid/issues/10784#issuecomment-763909460
Yeah, I'm curious if this problem happens only with legacy parsers. If so, an easy workaround would be using inputFormat.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on issue #10784: ParseExceptions from hasNext not caught and handled properly
Posted by GitBox <gi...@apache.org>.
gianm commented on issue #10784:
URL: https://github.com/apache/druid/issues/10784#issuecomment-763907331
I'm not sure since I haven't done a full analysis. But in this case, the task used a legacy `parser`, not an `inputFormat`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10784: ParseExceptions from hasNext not caught and handled properly
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10784:
URL: https://github.com/apache/druid/issues/10784#issuecomment-763906180
Does this happen with `InputFormat` as well? [`IntermediateRowParsingReader` should handle exceptions thrown in hasNext() via `ExceptionThrowingIterator`](https://github.com/apache/druid/blob/master/core/src/main/java/org/apache/druid/data/input/IntermediateRowParsingReader.java#L64-L72). I am just saying this is what it does with `InputFormat` and agree that there could be a better interface.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10784: ParseExceptions from hasNext not caught and handled properly
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10784:
URL: https://github.com/apache/druid/issues/10784#issuecomment-763906180
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] vikramarsid commented on issue #10784: ParseExceptions from hasNext not caught and handled properly
Posted by GitBox <gi...@apache.org>.
vikramarsid commented on issue #10784:
URL: https://github.com/apache/druid/issues/10784#issuecomment-777997975
@gianm @jihoonson I see a similar issue with #7690
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gianm commented on issue #10784: ParseExceptions from hasNext not caught and handled properly
Posted by GitBox <gi...@apache.org>.
gianm commented on issue #10784:
URL: https://github.com/apache/druid/issues/10784#issuecomment-763907331
I'm not sure since I haven't done a full analysis. But in this case, the task used a legacy `parser`, not an `inputFormat`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s closed issue #10784: ParseExceptions from hasNext not caught and handled properly
Posted by GitBox <gi...@apache.org>.
suneet-s closed issue #10784:
URL: https://github.com/apache/druid/issues/10784
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org