[I] [Variant] Implement a `shred_variant` function [arrow-rs]

["scovich (via GitHub)" <gi...@apache.org>]

scovich opened a new issue, #8361:
URL: https://github.com/apache/arrow-rs/issues/8361

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   
   We currently have no way to convert normal (binary) variant data into shredded variant data, even though we have some support for reading shredded data.
   
   **Describe the solution you'd like**
   
   A new `shred_variant` function that takes a `VariantArray` and a `DataType` as input, and which shreds the variant values according to the data type.
   
   For example, requesting `DataType::Int64` would produce an output variant array with the schema:
   ```
   {
      metadata: BINARY,
      value: BINARY,
      typed_value: LONG,
   }
   ```
   
   And requesting `DataType::Struct` with two integer fields `a` and `b` would produce an output variant array with the schema:
   ```
   {
     metadata: BINARY,
     value: BINARY,
     typed_value: {
       a: {
         value: BINARY,
         typed_value: INT,
       },
       b: {
         value: BINARY,
         typed_value: INT,
       },
     }
   }
   ```
   
   It's an open question whether the function should support re-shredding operations, where the input is allowed to contain a `typed_value` column of the "wrong" type.
   
   **Describe alternatives you've considered**
   
   I originally thought `variant_get` would fill this role, but:
   1. We don't actually have any way to pass a specific variant shredding schema to `variant_get`
   2. Even if we did have a way to pass a shredding schema, the caller would have to provide a physical shredding schema instead of the more intuitive logical shredding schema.
   3. Even `variant_get` _could_ express it, and users didn't mind creating physical shredding schemas, it's still nice to have a dedicated name for that subset of functionality


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Variant] Implement a `shred_variant` function [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb closed issue #8361: [Variant] Implement a `shred_variant` function
URL: https://github.com/apache/arrow-rs/issues/8361


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [Variant] Implement a `shred_variant` function [arrow-rs]

["alamb (via GitHub)" <gi...@apache.org>]

alamb commented on issue #8361:
URL: https://github.com/apache/arrow-rs/issues/8361#issuecomment-3416674659

   `label_issue.py` automatically added labels {'parquet-variant'} from #8366


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org