You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/03 08:55:26 UTC
[GitHub] [arrow] patcollis34 opened a new issue #12554: Plasma store as a production kv store for arrays
patcollis34 opened a new issue #12554:
URL: https://github.com/apache/arrow/issues/12554
Hey all,
90% sure this isn't the place for this question, but couldn't find a gitter or slack to post for this project, possibly JIRA but I've never used that before.
Wondering if the plasma datastore would/should be the recommended way to serve arrow(numpy arrays) in a distributed read-only environment(1 pod write, many pod read). In my first hack code I ran into ObjectID overwrite problems as well as the 20 bytes key limitation, which led me to think plasma wasn't fully suited for this workload.
Current setup: syncing local LMDBs per pod with a master efs, but has a few problems(serialiazation times, unnecessary syncs, etc..).
Any inputs or suggestions here is much appreciated
Also happy to delete this comment and move it to the proper forum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] patcollis34 commented on issue #12554: Plasma store as a production kv store for arrays
Posted by GitBox <gi...@apache.org>.
patcollis34 commented on issue #12554:
URL: https://github.com/apache/arrow/issues/12554#issuecomment-1059387343
great thank you for the info @westonpace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on issue #12554: Plasma store as a production kv store for arrays
Posted by GitBox <gi...@apache.org>.
westonpace commented on issue #12554:
URL: https://github.com/apache/arrow/issues/12554#issuecomment-1059373605
Plasma, at the moment, is not currently maintained by anyone on this project. The Ray project maintains a fork internally but (to the best of my knowledge) this isn't exposed as a standalone product. For more background you can check [this dicussion](https://lists.apache.org/thread/8zd8j8l2w0xlwznzjnzd2s228v31ld61) and [this older discussion](https://lists.apache.org/list?dev@arrow.apache.org:lte=2y:plasma) from the mailing list.
This place is probably an ok spot to ask a question but if you want a better chance at input into "what would be a good way to serve arrays in a distributed read-only environment?" then the mailing list is your best bet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] patcollis34 closed issue #12554: Plasma store as a production kv store for arrays
Posted by GitBox <gi...@apache.org>.
patcollis34 closed issue #12554:
URL: https://github.com/apache/arrow/issues/12554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org