[GitHub] [arrow-datafusion] houqp opened a new issue #996: Optimize ScalarValue list variant using Array trait object

[GitBox <gi...@apache.org>]

houqp opened a new issue #996:
URL: https://github.com/apache/arrow-datafusion/issues/996


   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   This comes out from the arrow2 PR started by @jorgecarleitao. Here is a discussion on why it would be more efficient to model ScalarValue as `dyn Array`: https://github.com/apache/arrow-datafusion/pull/68#discussion_r706565686
   
   **Describe the solution you'd like**
   
   Change ScalarValue's List variant to:
   
   ```rust
   List(Option<Arc<dyn Array>>, DataType),
   ```
   
   **Describe alternatives you've considered**
   
   Current way of warping every list value with a ScalarValue enum has a lot of overhead on read/write.
   
   **Additional context**
   
   We should probably wait for the arrow2 PR merge before starting to work on this.
   
   https://github.com/apache/arrow-datafusion/pull/68#discussion_r707044869 has more discussion on how we can handle this in ballista's protobuf ser/de.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org