You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/02/14 14:33:12 UTC
[GitHub] [arrow-datafusion] carols10cents opened a new issue #1832: Extract datafusion protobuf serialization into its own crate
carols10cents opened a new issue #1832:
URL: https://github.com/apache/arrow-datafusion/issues/1832
Hi, over in [influxdb_iox](https://github.com/influxdata/influxdb_iox), we have a need to serialize a data structure containing `datafusion::logical_plan::Expr`s using protobuf. [We have a solution](https://github.com/influxdata/influxdb_iox/pull/3707) inspired by the serialization in ballista.
We don't currently depend on ballista, though, and the datafusion-specific serialization might be useful to other projects besides IOx. Also, I see recent commits to the serialization here, and it makes more sense for this logic to live with datafusion rather than have us try to chase datafusion out of tree.
**Describe the solution you'd like**
- A crate named something like `datafusion-serialization` that implements serialization of `datafusion::logical_plan::Expr` data structures to and from protobuf, that both ballista and other projects could use.
**Describe alternatives you've considered**
- IOx and other projects maintain their own serialization logic
- Depend on all of ballista to only use the protobuf serialization implementations
**Additional context**
I think this would fit in nicely with the currently ongoing efforts to split datafusion into smaller crates!
I'm planning on working on a PR for this, but I wanted to open the issue in the meantime in case anyone has any concerns or discussion!
cc @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] realno commented on issue #1832: Extract datafusion protobuf serialization into its own crate
Posted by GitBox <gi...@apache.org>.
realno commented on issue #1832:
URL: https://github.com/apache/arrow-datafusion/issues/1832#issuecomment-1039659235
+1 It sounds reasonable to create a crate instead of depending on Ballista.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist closed issue #1832: Extract datafusion protobuf serialization into its own crate
Posted by GitBox <gi...@apache.org>.
Jimexist closed issue #1832:
URL: https://github.com/apache/arrow-datafusion/issues/1832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb edited a comment on issue #1832: Extract datafusion protobuf serialization into its own crate
Posted by GitBox <gi...@apache.org>.
alamb edited a comment on issue #1832:
URL: https://github.com/apache/arrow-datafusion/issues/1832#issuecomment-1039195572
I think this sounds like a good idea to me as it will allow other projects to serialize DataFusion structures if they desire. Thank you @carols10cents for opening the issue
I believe @thinkharderdev is also working on some changes related to serialization so perhaps they have some thoughts too.
One way to split up these crates might be to keep the datafusion and ballista bits separate:
```
# Datafusion specific serialization (like Expr, LogicalPlan)
# depends on datafusion
datafusion-serialization
# Ballista specific serialization (not sure exactly what this entails)
# depends on datafusion-serialization and ballista/rust/core
ballista/rust/serialization
```
cc @yahoNanJing @Ted-Jiang @realno @Jimexist @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1832: Extract datafusion protobuf serialization into its own crate
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1832:
URL: https://github.com/apache/arrow-datafusion/issues/1832#issuecomment-1039195572
I think this sounds like a good idea to me as it will allow other projects to serialize DataFusion structures if they desire. Thank you @carols10cents for opening the issue
I believe @thinkharderdev is also working on some changes related to serialization so perhaps they have some thoughts too.
One way to split up these crates might be to keep the datafusion and ballista bits separate:
```
# Datafusion specific serialization (like Expr, LogicalPlan)
# depends on datafusion
datafusion-serialization
# Ballista specific serialization (not sure exactly what this entails)
# depends on datafusion-serialization and ballista
ballista/rust/core
```
cc @yahoNanJing @Ted-Jiang @realno @Jimexist @andygrove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] realno commented on issue #1832: Extract datafusion protobuf serialization into its own crate
Posted by GitBox <gi...@apache.org>.
realno commented on issue #1832:
URL: https://github.com/apache/arrow-datafusion/issues/1832#issuecomment-1039659235
+1 It sounds reasonable to create a crate instead of depending on Ballista.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] thinkharderdev commented on issue #1832: Extract datafusion protobuf serialization into its own crate
Posted by GitBox <gi...@apache.org>.
thinkharderdev commented on issue #1832:
URL: https://github.com/apache/arrow-datafusion/issues/1832#issuecomment-1039385397
I think this makes sense. There are certainly other situations outside of Ballista where it may be useful to serialize plans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] thinkharderdev commented on issue #1832: Extract datafusion protobuf serialization into its own crate
Posted by GitBox <gi...@apache.org>.
thinkharderdev commented on issue #1832:
URL: https://github.com/apache/arrow-datafusion/issues/1832#issuecomment-1039385397
I think this makes sense. There are certainly other situations outside of Ballista where it may be useful to serialize plans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org