[GitHub] [arrow] unReaLaneS commented on issue #37139: [Python] Read table stuck and hangs forever

["unReaLaneS (via GitHub)" <gi...@apache.org>]

unReaLaneS commented on issue #37139:
URL: https://github.com/apache/arrow/issues/37139#issuecomment-1677160681

   Hi, thanks for answering. I checked the memory usage, it seems pq.read_table().to_batches() loads entire file into memory, is it supposed to do so, cause it can be misunderstood that to_batches is going to take partial data. I also tracked locally and on the server, it is fetching entire file and then, until next file comes it is not calling s3 at all.
   
   Solution that I found for this issue, is to switch to the:
   `pq.ParquetFile("bucket_path", filesystem=self.s3_file_system).iter_batches(batch_size)`, which indeed loads data in batches into memory. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org