You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/08/15 11:51:12 UTC

[I] [Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding) [arrow-rs]

alamb opened a new issue, #8153:
URL: https://github.com/apache/arrow-rs/issues/8153

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   - part of https://github.com/apache/arrow-rs/issues/6736
   
   **Note this is likely one of the most complex parts of implementing Shredded Variants , so it is not a good first task**
   
   We are trying to support the general case of the `variant_get` function,  which  allows runtime dynamic access to Variants (either shredded or unshredded). 
   
   - We found in  https://github.com/apache/arrow-rs/issues/8083 that supporting variant_get is quite complicated (see [here](https://github.com/apache/arrow-rs/pull/8122#discussion_r2277736911)), so we are proposing to brake it down into multiple piece.
   
   **This ticket tracks**
   Support `variant_get` for `Some(DataType::Struct)` (nested shredding)
   
   The idea here is that the user would specify a "shredding schema" (similar to what @friendlymatthew  is sketching out in https://github.com/apache/arrow-rs/pull/7921) and the variant_get kernel would produce a `VariantArray` with the defined schema, extracting fields as necessary
   
   Implementing this functionality will likely require the basic representation for shredded Variant arrays along with path traversal in `variant_get`. However, it does **NOT** cover the following (which are / will be broken into separate tickets)
   - Support for retrieving as a specific non Struct data type (e.g. `Some(DataType::Utf8)`)
   - Retrieving any arbitrary path and returning what is there (no type specified)
   - Retrieving any arbitrary path as a Variant (aka "unshredding")
   
   
   **Describe the solution you'd like**
   @scovich  sketched out a high level design for Shredded Objects (see [Representing Variant In Arrow Proposal: "Shredding an Object"](https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4/edit?tab=t.0#heading=h.wediefuitb91) and [Variant Shredding::Objects](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#objects)) in this PR
   - https://github.com/apache/arrow-rs/pull/8122
   
   This likely requires the ability to create variants without modifying the metadata
    - https://github.com/apache/arrow-rs/issues/8152
   
   So roughly that means supporting
   ```rust
   // get the named field of variant object as a typed field 
   variant_get(array, "$.field_name", DataType::Struct <....>)
   ```
   
   Where `$.field_name` represents some arbitrary `VariantPath` such as `a` for field "a", or `a.b` for field "b" of field "a"
   
   And DataType::Struct is a "shredding schema" that reflects both value and typed_value
   
   This should work for:
   1. Variants where the field_name is in a typed_value 
   2. Variants where the field_name is not in the typed value
   
   **Describe alternatives you've considered**
   1. Add a test that manually constructs a shredded variant array (follow the example in the arrow proposal)
   2. Add a test that calls variant_get appropriately
   3. Implement the code
   
   I suggest getting this working for non-nested obejcts first, and then working on nesting / pathing as a second pR
   
   
   **Additional context**
   
   
   
   Reference
   - [Variant Spec](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types)
   - [Variant Shredding Spec](https://github.com/apache/parquet-format/blob/master/VariantShredding.md#value-shredding)
   - [Representing Variant In Arrow Proposal](https://docs.google.com/document/d/1pw0AWoMQY3SjD7R4LgbPvMjG_xSCtXp3rZHkVp9jpZ4)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding) [arrow-rs]

Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #8153:
URL: https://github.com/apache/arrow-rs/issues/8153#issuecomment-4684507748

   https://github.com/apache/arrow-rs/pull/9681#discussion_r3397271791
   >
   >     2. Hmm... It's [[Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding) #8153](https://github.com/apache/arrow-rs/issues/8153).
   > 
   > 
   > I could do 2 but I'm not sure about `variant_get` custom shredding.
   > 
   > I checked Spark/duckdb/dremio `variant_get` implementations and it seems like no one does output shredding as a part of `variant_get` which is basically:
   > 
   > ```rust
   > variant_get(arr, path, Some(shredding_schema)) ≡ shred_variant(variant_get(arr, path), schema)
   > ```
   > 
   > There's a https://github.com/datafusion-contrib/datafusion-variant `Variant` integration and it strives for parity with spark/databricks.
   > 
   > Custom shredding will, however, skip unshred-reshred round trip that'd come from doing:
   > 
   > ```rust
   > shred_variant(variant_get(arr, path), schema)
   > ```
   > 
   > That's really a #8153 discussion
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding) [arrow-rs]

Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on issue #8153:
URL: https://github.com/apache/arrow-rs/issues/8153#issuecomment-4743974009

   > what do you think?
   
   About what, specifically?
   
   From a complexity perspective, it makes perfect sense to emulate shredded variant-get by shredding the result of a binary variant-get... reliable and can ship it quickly. But it definitely comes with a high runtime [opportunity] cost.
   
   But we already have the ability to do that by manually chaining the two operations. So is the question whether we should move that behind an API for convenience? Or whether somebody wants to tackle the complexity of directly reshredding as part of variant-get? Or something else?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding) [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8153:
URL: https://github.com/apache/arrow-rs/issues/8153#issuecomment-3193644756

   > I think two kinds of struct-based extractions are actually possible, and can actually both be used at the same time to produce a third flavor?
   
   sounds reasonable -- do you think we should file new tickets for each of the three kinds? Or maybe whoever works on this ticket can figure out the details and let us know if more tickets are needed


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding) [arrow-rs]

Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #8153:
URL: https://github.com/apache/arrow-rs/issues/8153#issuecomment-4092994173

   > I think two kinds of struct-based extractions are actually possible, and can actually both be used at the same time to produce a third flavor?
   > 
   > 1. User provides a plain old ordinary struct schema whose non-leaf fields are all normal struct/list and whose leaf fields are all strongly-typed.
   >    
   >    * The top-level does _not_ have a `metadata` column, for example. This really is just a plain ordinary struct.
   >    * A `variant_get` call should attempt to extract exactly that type, and either produce NULL or an error if any of the casting fails.
   >    * The result is a `StructArray` whose leaves are e.g. `PrimitiveArray`
   
   We do have support for shredded `Object`s in `variant_get` via _plain old ordinary struct schema_
   
   > 2. User provides an official shredding schema -- a variant-tagged struct schema where the top-level has a `metadata` column, and each field (both leaf and non-leaf) is a struct with value/typed_value pair that is characteristic of shredding.
   >    
   >    * A `variant_get` call should attempt to shred the data at each level into its respective `typed_value` field, falling back to `value` when necessary.
   >    * The result is a `VariantArray` that may have an arbitrarily complex shreding schema.
   > 3. And, just for fun, the user can do both at the same time -- providing a struct schema where some of the leaf fields are shredded variants.
   >    
   >    * The result is a `StructArray` where some of the leaves are `VariantArray`
   >    * Each of those variant leaves needs its own `metadata` column and has its own shredding schema, because it's now a stand-alone variant value (internal detail that they all can use a (shallow) copy of the original `metadata` column without having to re-encode anything).
   
   These two are not supported yet.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding) [arrow-rs]

Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on issue #8153:
URL: https://github.com/apache/arrow-rs/issues/8153#issuecomment-3192017919

   I think two kinds of struct-based extractions are actually possible, and can actually both be used at the same time to produce a third flavor?
   1. User provides a plain old ordinary struct schema whose non-leaf fields are all normal struct/list and whose leaf fields are all strongly-typed. 
      * The top-level does _not_ have a `metadata` column, for example. This really is just a plain ordinary struct.
      * A `variant_get` call should attempt to extract exactly that type, and either produce NULL or an error if any of the casting fails. 
      * The result is a `StructArray`.
   2. User provides an official shredding schema -- a variant-tagged struct schema where the top-level has a `metadata` column, and each field (both leaf and non-leaf) is a struct with value/typed_value pair that is characteristic of shredding.
      * A `variant_get` call should attempt to shred the data at each level into its respective `typed_value` field, falling back to `value` when necessary.
      * The result is a `VariantArray`.
   3. And, just for fun, the user can do both at the same time -- providing a struct schema where some of the leaf fields are shredded variants. 
      *The result is a `StructArray` where some of the leaves are `VariantArray`
      * Each of those variant leaves needs its own `metadata` column, because it's now a stand-alone variant value.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding) [arrow-rs]

Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #8153:
URL: https://github.com/apache/arrow-rs/issues/8153#issuecomment-4736860421

   @scovich what do you think?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding) [arrow-rs]

Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on issue #8153:
URL: https://github.com/apache/arrow-rs/issues/8153#issuecomment-4093498719

   @sdf-jkl your assessment makes sense to me. I think we even had conversations in the past about a utility method to expand a "logical" shredding schema into the equivalent physical shredding schema. As in, caller provides a struct that matches the desired shredding (what paths to shred as struct, int, string, etc vs leave as unshredded variant), and the utility spits out the corresponding arrangement of value and typed_value column pairs.
   
   We didn't pursue it back then because there was no way to know if we had a physical variant schema vs. a normal struct that happened to look variant-ish. Now that we have the variant extension type, a quick metadata check should distinguish the two and we could do what you say. Even better -- given the recursive nature of variant_get, we "should" get the mixed scenario "for free" once we have support for the two base cases.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Support Shredded Objects in `variant_get`: access as `Some(DataType::Struct)` (nested shredding) [arrow-rs]

Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #8153:
URL: https://github.com/apache/arrow-rs/issues/8153#issuecomment-4745810132

   @scovich sorry about that. My brain wasn't working when I typed that. I was thinking that we shouldn't support this since it's not something that other systems do. But it's an additive change and and a niccce optimization to be used downstream.
   
   > Or whether somebody wants to tackle the complexity of directly reshredding as part of variant-get?
   
   This is the path.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org