[GitHub] [arrow-datafusion] alamb opened a new issue #1651: Panic/dropped data when reading parquet files with incompatible shemas

[GitBox <gi...@apache.org>]

alamb opened a new issue #1651:
URL: https://github.com/apache/arrow-datafusion/issues/1651


   @thinkharderdev added the ability to read from multiple parquet files that have different schemas in https://github.com/apache/arrow-datafusion/pull/1622
   
   However, if some of the files have "incompatible" schemas (e.g. a different type), the data can be ignored from that file.
   
   Expected behavior would be to have the plan actually return an error rather than silently ignored. 
   
   > the issue is that the panic is on the reader thread. In effect it basically just causes the read on that partition to get dropped. We could test for a panic by calling `read_partition` directly from the test if you'd rather do it that way.
   
   _Originally posted by @thinkharderdev in https://github.com/apache/arrow-datafusion/pull/1622#discussion_r790151401_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] Dandandan closed issue #1651: Panic/dropped data when reading parquet files with incompatible shemas

[GitBox <gi...@apache.org>]

Dandandan closed issue #1651:
URL: https://github.com/apache/arrow-datafusion/issues/1651


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org