Re: [I] Support Continue on Error in CSV Reader [arrow-rs]

["GCHQDeveloper61637 (via GitHub)" <gi...@apache.org>]

GCHQDeveloper61637 commented on issue #4809:
URL: https://github.com/apache/arrow-rs/issues/4809#issuecomment-1793464249

   Suggestion #1: if the reader encounters an unrecoverable error, it should be marked internally as being in an unrecoverable state and any attempts to continue reading should panic with an explicit message. Diagnosis of the panic would be easier than trying to work out where a silent out-of-memory error originates.  
   
   Suggestion #2: on encountering a schema mismatch, the reader should seek forward to the next unquoted delimiter and attempt to continue reading from there. If the delimiter is a newline character, the unquoted newline is only likely to be a problem if it occurs in a record which is already a schema mismatch; the reader would attempt to read forward from a non-terminal newline and the subsequent fields would also be mismatched to the schema.... at which point it's probably OK to fall back on the behaviour in suggestion #1. Would that be workable? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org