You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mbutrovich (via GitHub)" <gi...@apache.org> on 2025/02/12 19:16:56 UTC

Re: [PR] fix: Remove cast.rs logic from parquet_support.rs for experimental native readers [datafusion-comet]

mbutrovich commented on PR #1387:
URL: https://github.com/apache/datafusion-comet/pull/1387#issuecomment-2654633256

   For context: when we were working in the experimental branch and added the `SchemaAdapter`, we were calling into cast.rs for type conversions. However, we started making changes in cast.rs to get more tests to pass for the experimental native scans. Not wanting to commingle logic and accidentally break cast behavior, we duplicated cast.rs and created parquet_support.rs to start making changes in. I now realize that duplicating cast.rs might have been a mistake, and have removed most of that logic from parquet_support.rs. This PR now defaults to DataFusion's cast logic unless we need specialized logic as in `cast_struct_to_struct`.
   
   This change does not introduce any new failures for the experimental native scans, but that doesn't mean this approach is bug free. @andygrove is working on some fuzz testing that will help us have more confidence in the experimental native scans in the future. I propose we move forward with this design, and bring over (or call into) cast.rs logic as we discover issues with this design.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org