You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/08/22 15:00:40 UTC
[GitHub] [beam] esadler-hbo opened a new issue, #22813: [Feature Request]: Add a WriteBatchedParquet option for python parquet-io
esadler-hbo opened a new issue, #22813:
URL: https://github.com/apache/beam/issues/22813
### What would you like to happen?
Hello!
I have a version of a `WriteBatchedParquet`, which is like a complement to `ReadBatchedParquet` that I would like to contribute or something similar.
**Why?**
There is a pretty big performance gain converting batches of scores to pyarrow row-groups and then saving those to directly to parquet without having to split them into python dictionaries per row. This really helped with performance and memory usage for offline batched model inference.
I am very happy to work on this, so let me know if there is any interest.
.take-issue
### Issue Priority
Priority: 3
### Issue Component
Component: io-py-parquet
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] esadler-hbo commented on issue #22813: [Feature Request]: Add a WriteBatchedParquet option for python parquet-io
Posted by GitBox <gi...@apache.org>.
esadler-hbo commented on issue #22813:
URL: https://github.com/apache/beam/issues/22813#issuecomment-1234950729
Oh I didn’t know about the batched DoFns API. That looks incredible! So does the run inference api. I don't currently use beam at work so I am a bit out of loop.m :). Seems like y'all have added a lot of what we did at my last job and did a much better job haha.
I think that I just need to save the result from the run inference api to parquet without yielding individual rows. If that is possible with the batch api then my use-case is covered.
Would be happy to contribute and it would be serious honor, but it sounds like you might already have a good solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] esadler-hbo closed issue #22813: [Feature Request]: Add a WriteBatchedParquet option for python parquet-io
Posted by GitBox <gi...@apache.org>.
esadler-hbo closed issue #22813: [Feature Request]: Add a WriteBatchedParquet option for python parquet-io
URL: https://github.com/apache/beam/issues/22813
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on issue #22813: [Feature Request]: Add a WriteBatchedParquet option for python parquet-io
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on issue #22813:
URL: https://github.com/apache/beam/issues/22813#issuecomment-1234813284
Hi @esadler-hbo, this would be a very welcome contribution :) Please let me know if you need any advice on how to proceed.
I'm actually currently working on integrating ParquetIO with Batched DoFns (https://s.apache.org/batched-dofns) which will hopefully unify things a bit (https://github.com/apache/beam/pull/22947). But in the meantime adding a specifc "Batched" version would be great.
CC: @rezarokni (since you mentioned model inference, Reza may be interested in learning more about your use-case and whether the new RunInference API will work for you)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org