You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/04/18 18:53:49 UTC

[I] Variant: Rust API to Read Variant Values [arrow-rs]

alamb opened a new issue, #7423:
URL: https://github.com/apache/arrow-rs/issues/7423

   - Part of https://github.com/apache/arrow-rs/issues/6736
   
   
   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   The first part of supporting the Variant type in Parquet and Arrow is
   programmatic access to values encoded with the binary format described in
   [VariantEncoding.md]. This ticket covers the API to read such values, but not
   creating such values, or representing it using arrow or parquet which are
   covered in other tickets
   
   **Describe the solution you'd like**
   I would like a Rust API, similar to the Json::Value and similar APIs to dynamically access variant values.
   
   Here is some example binary data for testing:
   * https://github.com/apache/parquet-testing/pull/76
   
   
   **Describe alternatives you've considered**
   
   I think a Rust enum approach with references would be a good model.
   
   I suggest creating a  new crate, `arrow-variant`, and marking it as
   experimental, etc saying it will contain breaking changes for the next several
   releases (maybe we can even version it 0.1, etc)
   
   For example:
   
   ## Sketch of structures
   
   ```rust
   /// Variant value. May contain references to metadata and value
   /// 'a is lifetime for metadata
   /// 'b is lifetime for value
   pub enum Variant<'a, 'b> {
     Variant::Null,
     Variant::Int8
     ...
     // strings are stored in the value and thus have references to that value
     Variant::String(&'b str),
     Variant::ShortString(&'b str),
     // Objects and Arrays need the metadata and values, so store both.
     Variant::Object(VariantObject<'a, 'b>),
     VariantArray(VariantArray<'a, 'b>)
   }
   
   /// Wrapper over Variant Metadata
   pub struct VariantMetadata<'a> {
     metadata: &'a[u8],
     // perhaps access to header fields like dict length and is_sorted
   }
   
   /// Represents a Variant Object with references to the underlying metadata
   /// and value fields
   pub enum VariantObject<'a, 'b> {
     // pointer to metadata
     metadata: VariantMetadata<'a>,
     // pointer to value
     value: &'a [u8],
   }
   ```
   
   ## Creating `Variants` from buffers
   ```rust
   // Each variant has a metadata and value buffer:
   let metadata: &[u8] = ...;
   let value: &[u8] = ....;
   // The Rust API should NOT require allocations or copy the metadata/values
   let variant = Variant::try_new(metadata, value)?;
   ```
   
   ## Working with Primitive `Variants`
   
   ```rust
   // Act based on the type of variant
   match variant {
     Variant::Int8(val) => println!("The value was int8: {val}"),
     ...
     Variant::SmallString(val) => println!("The value was a small string: {val}"),
     ...
     Variant::Object(object) => {
       println("The variant was in object. The fields are:");
       for (field_name, field_value) in object.fields()? {
         // The inner field value is also a variant
         match field_value {
           Variant::...
         }
       }
     }
     // similarly for Variant::Array
   }
   ```
   
   I personally suggest doing this over a few PRs:
   1. Scaffolding: `Variant` struct/enum, support a few basic variant primtive types
   2. Basic nested type support: basic support for objects
   3. Array support: support for arrays
   4. Complete APIs, etc
   
   
   **Additional context**
   * @PinkCrow007s: https://github.com/apache/arrow-rs/pull/7404.  https://github.com/apache/arrow-rs/blob/0220e97b407cf6690ac8b51d53fa0e92273f1d8c/arrow-schema/src/extension/canonical/variant.rs#L38-L37
   * Spark implementation: https://github.com/apache/spark/blob/007c31df6da7b741f3a2a43c859ebed9f801dcfa/common/variant/src/main/java/org/apache/spark/types/variant/Variant.java#L43
   * Python implementation (also in spark): https://github.com/apache/spark/blob/master/python/pyspark/sql/variant_utils.py
   * Examples from @jonhoo and @wjones127  in https://github.com/datafusion-contrib/datafusion-functions-variant
   
   
   **Open Questions**:When should validation be done?
   
   I do think there should be an API like:
   ```rust
   /// ensure that metadata and value are valid according to the Variant spec, returns error if not.
   Variant::validate(&metadata, &value)?;
   ```
   
   However, the API sketched above proposes doing validation on access (when the
   values are accessed). An alternate approach would be to validate everything on
   creation and then use unchecked APIs during access.
   
   I think validating once upfront is better if most fields are accessed or certain
   fields are read multiple times. For the usecase where only some fields are read
   I think verifying on access would be faster.
   
   The spec also allows metadata to contain dictionary values that do not appear as
   struct names in the variant value itself, so eager validation would potentially
   verify string data uncessairly.
   
   
   
   I suggest starting with an API that is fallible (aka creating a Variant or
   accessing a field returns  `Result<Variant>`. We can always add unsafe versions
   of the APIs for usecases where validation overhead is significant (e.g. writing
   utf8 validation for field names when writing json), and justified with benchmarks
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #7423: Variant: Rust API to Read Variant Values
URL: https://github.com/apache/arrow-rs/issues/7423


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7423:
URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2894597621

   @scovich  had some great comments on https://github.com/apache/arrow-rs/pull/7452#discussion_r2095809708 that I wanted to copy/paste into this ticket perhaps for wider discussion:
   
   
   > I have not reviewed the code carefully at all yet, and what follows is a general observation based on the inherent nature of variant data and rust notions of safety:
   > 
   > It will be really tempting to have "efficient" code that e.g. uses [from_utf8_unchecked](https://doc.rust-lang.org/std/primitive.str.html#method.from_utf8_unchecked) to extract a `&str` from a `&[u8]`, or to use indexing operations like `v[10]` to extract bytes. But variant data is generally untrusted user input and whatever `Variant` struct/enum we define will become the first -- and often only -- line of defense against malicious or malformed input.
   > 
   > Hopefully we can code carefully, with the goal that sizes and/or contents of metadata and value slices will never cause a panic?
   > 
   > Additinoally, it seems like we have a few choices for values such as strings and decimals even a right-sized byte slice can contain invalid values:
   > 
   > 1. Return obviously unvalidated values, e.g. `&[u8]` instead of `&str` for strings, and `&[u8]` instead of whatever `VariantDecimal` struct we might otherwise define -- leaving the user responsible to finish the conversion as (un)safely as they deem prudent.
   > 2. Return ostensibly validated values, with (safe) checked and (unsafe) unchecked constructors and/or getters that let the user choose the one they deem appropriate.
   > 
   > I personally favor the latter approach (safe and easy to use, even if not always the absolutely max efficient), but the topic probably needs a wider discussion.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7423:
URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2897586489

   BTW I think @mapleFU has made a PR with a proposed C/C++ API for accessing variants which might have some additional inspiration:
   - https://github.com/apache/arrow/pull/46372


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7423:
URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2890601195

   FYI @PinkCrow007


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7423:
URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2890606191

   BTW there are a few known issues with the example variant values https://github.com/apache/parquet-testing/issues/75 in parquet-testing:
   
   Specifically
   - https://github.com/apache/parquet-testing/issues/81
   - https://github.com/apache/parquet-testing/issues/82
   
   I think they will be relatively easy to solve / workaround for the time being, but I wanted to bring them to your attention


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7423:
URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2890600766

   Thanks @mkarbo  !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7423:
URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2894610251

   > I personally favor the latter approach (safe and easy to use, even if not always the absolutely max efficient), but the topic probably needs a wider discussion.
   
   I likewise think the "safe / return error by default" is the right model. I also agree it should be an API goal that there should be no panics on invalid variant data (it should return an error instead)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "mkarbo (via GitHub)" <gi...@apache.org>.
mkarbo commented on issue #7423:
URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2890516466

   We'd like to pick this up, can you assign this one to me @alamb?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #7423: Variant: Rust API to Read Variant Values
URL: https://github.com/apache/arrow-rs/issues/7423


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #7423:
URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2992671689

   `label_issue.py` automatically added labels {'arrow'} from #7403


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Variant: Rust API to Read Variant Values [arrow-rs]

Posted by "mkarbo (via GitHub)" <gi...@apache.org>.
mkarbo commented on issue #7423:
URL: https://github.com/apache/arrow-rs/issues/7423#issuecomment-2894743866

   Yeah I agree, thanks for pointing it out


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org