[GitHub] [arrow-datafusion] jorgecarleitao opened a new issue #351: Scalar::List does not encapsulate all information from ListArray

[GitBox <gi...@apache.org>]

jorgecarleitao opened a new issue #351:
URL: https://github.com/apache/arrow-datafusion/issues/351


   IMO the roundtrip `ListArray -> Scalar::List -> ListArray` is currently lossy. This happens because the `Scalar::List` does not encapsulate everything from the `ListArray`.
   
   Examples: 
   * it is not possible to recover the field name and nullability of the inner type.
   * nested types are hard to rebuild
   
   (new) suggested signature:
   
   ```
   Scalar::List(Option<Arc<dyn Array>>, DataType)
   ```
   
   With this, the second argument stores the original datatype, which allows to recover all the information from `ListArray`, and the first argument is easy to recover and build in both directions:
   
   * `ListArray -> Scalar`: use `array.value(index)`
   * `Scalar -> ListArray`: use `concatenate(repeat(inner).take(size))`
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow-datafusion] jorgecarleitao commented on issue #351: Scalar::List does not encapsulate all information from ListArray

[GitBox <gi...@apache.org>]

jorgecarleitao commented on issue #351:
URL: https://github.com/apache/arrow-datafusion/issues/351#issuecomment-855574159


   @andygrove , I was trying to address this in ballista, but I am struggling to encapsulate a ScalarValue that depends on an Array; how are we declaring arrow arrays in the protobuf? I can't find any reference to them.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org