[jira] [Commented] (DRILL-4653) Malformed JSON should not stop the entire query from progressing

["subbu srinivasan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-4653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269945#comment-15269945 ] 

subbu srinivasan commented on DRILL-4653:
-----------------------------------------

Folks,
I went through the code for JsonParsing. The main call for JSON deserialization happens to be
in JSONReader which is called from JSONRecordParser. The issue is that a handleAndRaise call is made to all caught exceptions.
Would the proposal below be of acceptance to the community.
The proposal is to catch the IOException and not bail out.
try{
outside: while(recordCount < BaseValueVector.INITIAL_VALUE_ALLOCATION) {
try
{
writer.setPosition(recordCount);
write = jsonReader.write(writer);
if(write == ReadState.WRITE_SUCCEED)
{ // logger.debug("Wrote record."); recordCount++; }
else
{ // logger.debug("Exiting."); break outside; }
}
catch(IOException ex)
{ logger.error("Ignoring record. Error parsing JSON: ", ex); ++parseErrorCount; }
}

> Malformed JSON should not stop the entire query from progressing
> ----------------------------------------------------------------
>
>                 Key: DRILL-4653
>                 URL: https://issues.apache.org/jira/browse/DRILL-4653
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Storage - JSON
>    Affects Versions: 1.6.0
>            Reporter: subbu srinivasan
>
> Currently Drill query terminates upon first encounter of a invalid JSON line.
> Drill has to continue progressing after ignoring the bad records. Something 
> similar to a setting of (ignore.malformed.json) would help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)