Petastorm: PyArrow based library for Tensorflow, PyTorch and others...

[Yevgeni Litvin <se...@gmail.com>]

Hi,

My name is Yevgeni Litvin. I am working on ML infra with a small team
within Uber ATG. Our team has recently open sourced Petastorm library. It
heavily relies on Apache Arrow so I wanted to share it with the community.

The goal of the project is to provide a convenient way for deep learning
community to use Apache Parquet store with sensor data from Tensorflow,
PyTorch or other Python based ML frameworks.

I believe our use of Parquet is different from mainstream applications as
our field sizes are asymetric (some are huge, such as images, and others
are small) and rowgroup sizes are relatively small (<100). That required
some adaptations.

We use PyArrow mostly for loading the data. We do see great potential for
further optimizations and speedups by relying more heavily on Arrow as
in-memory store.

You can find more information about our project here:

http://eng.uber.com/petastorm/
https://github.com/uber/petastorm

Would be more than happy to hear comments, feedback and suggestions!

Thank you,

- Yevgeni Litvin

Re: Petastorm: PyArrow based library for Tensorflow, PyTorch and others...

["Uwe L. Korn" <uw...@xhochy.com>]

Hello Yevgeni,

this looks interesting. Can you make a PR to https://github.com/apache/arrow so that  Petastorm is listed on https://arrow.apache.org/powered_by/ ? 

I browsed a bit through your code. As far as I can see your approach is store to have a set of Parquet files in a directory with a schema that can be translated for Spark, Tensorflow, Torch, … Is this schema persisted in the Parquet file metadata or as a separate file alongside the dataset? Could we extend Arrow's type system a bit to better suit all the frameworks you are targeting. As you had to build a more general schema class, I guess there are definitely things that could not be expressed in Arrow's schema definition. Not sure whether we could extend pyarrow's schema classes to fully support your use case but I would like to understand how to better support it.

Uwe

On Wed, Sep 26, 2018, at 8:59 PM, Yevgeni Litvin wrote:
> Hi,
> 
> My name is Yevgeni Litvin. I am working on ML infra with a small team
> within Uber ATG. Our team has recently open sourced Petastorm library. It
> heavily relies on Apache Arrow so I wanted to share it with the community.
> 
> The goal of the project is to provide a convenient way for deep learning
> community to use Apache Parquet store with sensor data from Tensorflow,
> PyTorch or other Python based ML frameworks.
> 
> I believe our use of Parquet is different from mainstream applications as
> our field sizes are asymetric (some are huge, such as images, and others
> are small) and rowgroup sizes are relatively small (<100). That required
> some adaptations.
> 
> We use PyArrow mostly for loading the data. We do see great potential for
> further optimizations and speedups by relying more heavily on Arrow as
> in-memory store.
> 
> You can find more information about our project here:
> 
> http://eng.uber.com/petastorm/
> https://github.com/uber/petastorm
> 
> Would be more than happy to hear comments, feedback and suggestions!
> 
> Thank you,
> 
> - Yevgeni Litvin