[GitHub] [arrow] zhihuiwan opened a new issue #12121: A MemoryError occurred when reading Arrow streams

[GitBox <gi...@apache.org>]

zhihuiwan opened a new issue #12121:
URL: https://github.com/apache/arrow/issues/12121


   The hdfs file size is 160G, and the machine memory is 64G. 
   pyarrow.NativeFile does not support readline and readlines methods, but I want to read the content by line, so I did the following test:
   
   ```python
   >>> from pyarrow import fs
   >>> path = 'hdfs://hadoop-cluster//wzh/test_160g'
   >>> hdfs_client = fs.HadoopFileSystem.from_uri(path)
   >>> buffer = hdfs_client.open_input_file(path)
   >>> reader = io.TextIOWrapper(buffer=buffer, encoding="utf-8")
   >>> reader.readlines(10)
   ```
   
   The result is the following error：
   ```python
   Traceback (most recent call last):
     File "<stdin>", line 1, in <module>
     File "pyarrow/io.pxi", line 376, in pyarrow.lib.NativeFile.read1
     File "pyarrow/io.pxi", line 323, in pyarrow.lib.NativeFile.read
   MemoryError
   ```
   Is there a good way to handle this?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org