You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2025/08/07 18:28:18 UTC
[I] [Variant] Full Casting support in `variant_get` [arrow-rs]
alamb opened a new issue, #8086:
URL: https://github.com/apache/arrow-rs/issues/8086
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
- part of https://github.com/apache/arrow-rs/issues/6736
We introduced the basic `variant_get` kernel in and now we need to expand its feature set
- https://github.com/apache/arrow-rs/pull/8021
However, it doesn't really handle the case when the variant contains a `int` column but the output calls for a float, for example
**Describe the solution you'd like**
I would like to handle this case within the variant_get kernel
**Describe alternatives you've considered**
I personally suggest we follow the rules in the existing arrow [cast kernel](https://docs.rs/arrow/latest/arrow/compute/kernels/cast/index.html), likely simply calling that kernel for shredded values, and implementing the equivalent logic for unshredded variants.
@scovich proposed https://github.com/apache/arrow-rs/pull/8021/files#r2257527874
> I guess we could pursue three potential levels of casting:
>
> Widening casts (based on type). E.g. if the type is int32, we can also handle int8, int16 and even decimal4 (if scale=0). The conversion is guaranteed to succeed losslessly.
> Lossless narrowing casts (based on value). E.g. if the type is int32, we can still handle a small int64 value like 42. TBD what should happen if any one value failed to cast?
> Converting casts. These are the lossy ones, e.g. converting 3.14f64 to 3i32, or 3i32 to string. I'm guessing we don't want to mess with these?
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] Casting errors behavior support in `variant_get` [arrow-rs]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on issue #8086:
URL: https://github.com/apache/arrow-rs/issues/8086#issuecomment-3652669270
This comment is a backup from #8873, as #8873 may close when the unification has been implemented.
1. We can unify the usage in `parquet-variant-compute` into `arrow-cast/CastOptions`(`arrow-cast/src/cast/mod.rs#CastOptions`).
2. We can add a `lossy` flag to `arrow-cast/CastOptions` to handle `Decimal` -> `Int` and `Timestamp(Micro)` -> `Timestamp(Second)` cases. The specific behavior is shown below:
> There are only 3 cases ` (lossy, *)`, `(lossless, safe)`, `(lossless, unsafe)` in the below table
| safe | lossy | 1234(Micro) -> Mills) |
| -- | -- | -- |
| true | true | 1 (= floor(1234 / 1_000)) |
| true | false | Null |
| false | true | 1 |
| false | false | Err |
| safe | lossy | 1000(Micro) -> Mills |
| -- | -- | -- |
| true | true | 1 |
| true | false | 1 |
| false | true | 1 |
| false | false | 1 |
The specific steps are as follows:
1. Add a lossy flag to `arrow-cast/CastOptions`.
2. Unify `CastOptions` usage in `parquet-variant-compute`.
2.1 Need to change the behavior for those that can't be cast safely. Currently, the default behavior of `arrow-cast/CastOptions` and `parquet-compute-variant/CastOptions` is different; the default value for [`arrow-cast/CastOptions` is `safe=true`](https://github.com/apache/arrow-rs/blob/c9fca0b1248911daeab5e92146d33a58f11e15d2/arrow-cast/src/cast/mod.rs#L83-L90), and [`parquet-variant-compute/CastOptions` is `strict=true`(which is that `safe=false`)](https://github.com/apache/arrow-rs/blob/c9fca0b1248911daeab5e92146d33a58f11e15d2/parquet-variant-compute/src/type_conversion.rs#L35-L39)
3. After adding `lossy` in `arrow-cast/CastOptions`, we can update parquet-variant-compute to adapt the corresponding logic.
Steps 1 and 2 do not depend on each other, and can push forward in parallel
Steps 1 and 2 will break the Public API; we need to do them in a major version
- Step 1 will add a new field in pub struct
- Step 2 will change the parameter and default behavior for some functions
Currently, the `CastOptions` usages in `parquet-variant-compute` are as follows:
- `shred_variant.rs/variant_to_arrow.rs/variant_get.rs` use `arrow-cast/CastOptions.`
- `arrow_to_variant.rs/cast_to_variant.rs/lib.rs(re-export)` use `parquet-variant-compute/CastOptions`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] Casting support in `variant_get` [arrow-rs]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on issue #8086:
URL: https://github.com/apache/arrow-rs/issues/8086#issuecomment-3165392839
> I personally suggest we follow the rules in the existing arrow [cast kernel](https://docs.rs/arrow/latest/arrow/compute/kernels/cast/index.html), likely simply calling that kernel for shredded values, and implementing the equivalent logic for unshredded variants.
Thinking out loud here --
The engine and/or user may have very different opinions than arrow about casting semantics, which is why I've been concerned about "just" doing a `cast` internally. Even with cast options I don't think it gives enough control.
However -- I _think_ the caller can just provide a schema where all leaf columns are VARIANT, if they want control over casting semantics? The resulting `VariantArray` would come back unchanged in its original form (shredded or unshredded). At worst, we might have to extract a subset of bytes if some requested leaf path penetrated an unshredded field.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] Casting support in `variant_get` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8086:
URL: https://github.com/apache/arrow-rs/issues/8086#issuecomment-3165521525
> The engine and/or user may have very different opinions than arrow about casting semantics, which is why I've been concerned about "just" doing a `cast` internally. Even with cast options I don't think it gives enough control.
In my opinion since this is all in the context of parquet-variant-compute -- namely manipulating variants using Arrow arrays -- I think it is actually fine to only provide arrow casting semantics.
> The resulting VariantArray would come back unchanged in its original form (shredded or unshredded). At worst, we might have to extract a subset of bytes if some requested leaf path penetrated an unshredded field.
Yes, this is my ideal outcome -- that the library is flexible enough people can implement whatever semantics if what is provided by variant_get is insufficient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] Casting errors behavior support in `variant_get` [arrow-rs]
Posted by "sdf-jkl (via GitHub)" <gi...@apache.org>.
sdf-jkl commented on issue #8086:
URL: https://github.com/apache/arrow-rs/issues/8086#issuecomment-4530928230
@klion26 Is there anything left here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] Casting errors behavior support in `variant_get` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8086:
URL: https://github.com/apache/arrow-rs/issues/8086#issuecomment-4533841325
Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] Casting errors behavior support in `variant_get` [arrow-rs]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on issue #8086:
URL: https://github.com/apache/arrow-rs/issues/8086#issuecomment-4533724259
@sdf-jkl thanks for tagging me. I think we can close this one now, as #8982 will track the progress of aligning cast logic between variant and `arrow-cast`, cc @alamb as I don't have the permission to close this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Variant] Casting errors behavior support in `variant_get` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8086: [Variant] Casting errors behavior support in `variant_get`
URL: https://github.com/apache/arrow-rs/issues/8086
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org