[GitHub] [arrow-datafusion] alamb opened a new issue, #2340: Implement `Serialization` API for DataFusion

[GitBox <gi...@apache.org>]

alamb opened a new issue, #2340:
URL: https://github.com/apache/arrow-datafusion/issues/2340

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   IOx has its own distribution mechanism that is different than Ballista where we need to send `Expr`s as part of a network request. We have a version that is a fork (ahem copy/paste) of an earlier version of ballista serialization, but we would like to use the code in `datafusion-proto` crate so
   
   1. We don't have to maintain our current fork (and deal with bugs like  https://github.com/influxdata/influxdb_iox/issues/4394)
   3. We can invest some of that effort back into the DataFusion implementation
   
   @carols10cents  started down the path a while ago by breaking out `datafusion-proto` into its own crate (https://github.com/apache/arrow-datafusion/issues/1832) but we hit some issues when actually trying to use datafusion-proto in IOx due to various fun with how `prost` is implemented and our use of `pbjson` (see https://github.com/influxdata/influxdb_iox/pull/3997) and the project has stalled. 
   
   However, it has recently become a priority again https://github.com/influxdata/influxdb_iox/issues/4394
   
   **Describe the solution you'd like**
   
   I would like a method that can serialize `Expr`s to/from bytes that I can embed in my messages without having to worry about the fact that they happen to be using binary protobuf format or making `prost` generated code play nicely across crates (😱 )
   
   Something like this:
   ```rust
   // Create a new `Expr` a < 32
   let expr = col("a").lt(lit(5i32));
   
   // Convert it to an opaque form
   let bytes = expr.serialize().unwrap();
   
   // Decode bytes from somewhere (over network, etc.
   let decoded_expr = Expr::deserialize(&bytes).unwrap();
   assert_eq!(expr, decoded_expr);
   ```
   
   **Describe alternatives you've considered**
   Implement more code in `datafusion-proto` to allow IOx to use it directly (so we could reuse the `prost` generated structures).
   
   **Additional context**
   
   See @tustvold  's comments https://github.com/influxdata/influxdb_iox/pull/3997#issuecomment-1077949298 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] alamb closed issue #2340: Implement `Serialization` API for DataFusion

[GitBox <gi...@apache.org>]

alamb closed issue #2340: Implement `Serialization` API for DataFusion
URL: https://github.com/apache/arrow-datafusion/issues/2340


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org