[jira] [Created] (ARROW-9526) [Python] Memorymapped arrow file to parquet conversions loaded everything into memory

["Sep Dehpour (Jira)" <ji...@apache.org>]

Sep Dehpour created ARROW-9526:
----------------------------------

             Summary: [Python] Memorymapped arrow file to parquet conversions loaded everything into memory
                 Key: ARROW-9526
                 URL: https://issues.apache.org/jira/browse/ARROW-9526
             Project: Apache Arrow
          Issue Type: Bug
            Reporter: Sep Dehpour


When converting a memory mapped arrow file into parquet file, it loads the whole table into RAM. This effectively negates the point of memory mapping.

If this is not a bug,  perhaps there is a proper way of converting the memorymapped arrow file to parquet without using excessive memory?

 

Example code:
{code:java}
    source = pa.memory_map(path_to_arrow_file, 'r')
    table = pa.ipc.RecordBatchFileReader(source).read_all()
    # The followlng line will load the whole thing into RAM
    pq.write_table(table, path_to_parquet){code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)