[Java][Arrow IPC] Extreme memory usage when reading feather files

[Laura Doktorova <la...@layer6.ai>]

Use case:

   - process rows in Java from lots of feather files generated from pandas
   dataframes and convert to another format
   - files contain ~2000+ float8 columns and 200_000 rows each
   - do not need to have the whole file in memory
   - need to process files in parallel

Problem:

   - Reading files takes a lot of memory, even if we read just one file.
   - Reading one such file with ArrowFileReader fails if directMemorySize
   is set below 2.5gb (with swap set to 0).
   - Reading one such file with dataset api fails if system memory is below
   7gb (with swap set to 0).
   - Also noted that saving this file from pandas with a call to_feather
   fails when RAM is below 9GB of RAM (swap is 0). Dataframe itself is 3GB.


Is there a way to read feather files in a way that would not take so much
space? It seems that each file loads in batches of 4, but the memory taken
is much more than one batch.
I can provide the sample code if that helps.

Thank you

Re: [Java][Arrow IPC] Extreme memory usage when reading feather files

[Laura Doktorova <la...@layer6.ai>]

Each file has 4 batches, if each batch contains 200000 / 4 = 50000 rows by
2000 columns = 100_000_000 8byte floats - about 0.75G if the whole batch is
loaded into memory. So my expectation is for memory to be close to the
memory for one batch size or so. For this test, I am not creating extra
space when reading in my code, I only use one float array of 2000. No
Double objects are created.
With this, Dataset API throws OutOfMemory exception if there is less than
7GB of RAM. When reading with ArrowFileReader, it throws OutOfMemory
exception if there is less than 3GB of RAM.
Is there a way to read the file without having it all loaded into RAM?


On Mon, Jan 30, 2023 at 10:19 AM Larry White <lj...@gmail.com> wrote:

> If you bring the data into the java memory space, you will use a lot of
> memory even just for one file:  8 bytes * 200,000 rows * 2000 columns is
> 3.2 GB, even without the overhead of converting the values to Double
> objects (which could double the required memory).  The best approach would
> be to leave the data off-heap and access the values using
> DataHolders, which should let you access the values using one object per
> vector.
>
> On Mon, Jan 30, 2023 at 10:08 AM Chris Nuernberger <ch...@techascent.com>
> wrote:
>
>> TMD <https://github.com/techascent/tech.ml.dataset> supports memory
>> mapped arrow files.  We don't currently support float8 but I would be
>> interested in implementing that if you are interested in trying it out.
>> Its Clojure, not java, but is still on the JVM.
>>
>> This is likely to be your fastest option both in terms of raw performance
>> and time to final solution.
>>
>

Re: [Java][Arrow IPC] Extreme memory usage when reading feather files

[Larry White <lj...@gmail.com>]

If you bring the data into the java memory space, you will use a lot of
memory even just for one file:  8 bytes * 200,000 rows * 2000 columns is
3.2 GB, even without the overhead of converting the values to Double
objects (which could double the required memory).  The best approach would
be to leave the data off-heap and access the values using
DataHolders, which should let you access the values using one object per
vector.

On Mon, Jan 30, 2023 at 10:08 AM Chris Nuernberger <ch...@techascent.com>
wrote:

> TMD <https://github.com/techascent/tech.ml.dataset> supports memory
> mapped arrow files.  We don't currently support float8 but I would be
> interested in implementing that if you are interested in trying it out.
> Its Clojure, not java, but is still on the JVM.
>
> This is likely to be your fastest option both in terms of raw performance
> and time to final solution.
>

Re: [Java][Arrow IPC] Extreme memory usage when reading feather files

[Chris Nuernberger <ch...@techascent.com>]

TMD <https://github.com/techascent/tech.ml.dataset> supports memory mapped
arrow files.  We don't currently support float8 but I would be interested
in implementing that if you are interested in trying it out.  Its Clojure,
not java, but is still on the JVM.

This is likely to be your fastest option both in terms of raw performance
and time to final solution.