You are viewing a plain text version of this content. The canonical link for it is here.

Posted to github@arrow.apache.org by "jonded94 (via GitHub)" <gi...@apache.org> on 2025/07/26 21:51:26 UTC

Re: [I] Error when reading row group larger than 2GB (total string length per 8k row batch exceeds 2GB) [arrow-rs]

jonded94 commented on issue #7973:
URL: https://github.com/apache/arrow-rs/issues/7973#issuecomment-3123427817

   @alamb setting DataType::LargeUt8 (which is large_string() in pyarrow) does fix it, but this still is a arrow-rs bug AFAIS, since both pyarrow and duckdb can read the file just fine. I assume they do some internal splitting into smaller batches to not run into integer overflows.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org