You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "klion26 (via GitHub)" <gi...@apache.org> on 2025/11/19 12:08:13 UTC

[I] [Variant] Unify the cast option usage in ParquentVariant [arrow-rs]

klion26 opened a new issue, #8873:
URL: https://github.com/apache/arrow-rs/issues/8873

   **Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
   <!--
   A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] 
   (This section helps Arrow developers understand the context and *why* for this feature, in addition to  the *what*)
   -->
   
   Currently, we use `arrow::compute::CastOptions` in `variant_get` and `crate::CastOptions` in `cast_to_variant_with_options`, maybe, we can unify these in the future.
   
   **Describe the solution you'd like**
   <!--
   A clear and concise description of what you want to happen.
   -->
   
   As the cast option here is for the variant, maybe we can unify this to use `carate::CastOption` and extend it if necessary (such as lossy, transform, etc.).
   
   **Describe alternatives you've considered**
   <!--
   A clear and concise description of any alternative solutions or features you've considered.
   -->
   
   **Additional context**
   <!--
   Add any other context or screenshots about the feature request here.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Unify the cast option usage in ParquentVariant [arrow-rs]

Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on issue #8873:
URL: https://github.com/apache/arrow-rs/issues/8873#issuecomment-3636085258

   
   > The two tables suggest there are only three actual values? (lossy, *), (lossless, safe), (lossless, unsafe)?这两张表是否表明实际上仅有三个取值?(有损,*)、(无损,安全)、(无损,不安全)? Or would (lossy, safe) and (lossy, unsafe) cases still matter when dealing with incompatible types?或者,在处理不兼容类型时,(有损、安全)和(有损、不安全)的情况是否仍然重要?
   
   Yes, there are only three actual values? (lossy, *), (lossless, safe), (lossless, unsafe), when dealing with incompatible types(e.g., `Duration` -> `Int`), (lossy, safe), and (lossy, unsafe) are the same
   
   > Variant is still an experimental feature, which I think means we don't have to wait for major version bumps to break compat?
   
   If we do not need to wait for a major version, then step 2 can be done for now, and steps 1 & 3 will be done in the next major version(it needs to modify the `arrow-cast` crate).
   
   I'll do step 2 first. If we don't need to wait for a major version, then we can merge it in now; otherwise, we can wait for the next major version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Unify the cast option usage in ParquentVariant [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #8873:
URL: https://github.com/apache/arrow-rs/issues/8873#issuecomment-3716624608

   `label_issue.py` automatically added labels {'parquet-variant'} from #8950


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Unify the cast option usage in ParquentVariant [arrow-rs]

Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on issue #8873:
URL: https://github.com/apache/arrow-rs/issues/8873#issuecomment-3630708708

   @alamb @scovich   After some analysis, my preliminary conclusions for this issue and #8086 are as follows:
   
   1. We can unify the usage in `parquet-variant-compute` into `arrow-cast/CastOptions`(`arrow-cast/src/cast/mod.rs#CastOptions`).
   2. We can add a `lossy` flag to `arrow-cast/CastOptions` to handle `Decimal` -> `Int` and `Timestamp(Micro)` -> `Timestamp(Second)` cases. The specific behavior is shown below:
   
   | safe | lossy | 1234(Micro) -> Mills) |
   | -- | -- | -- |
   | true | true  |  1  (= floor(1234 / 1_000)) |  
   | true | false |  Null |
   |  false | true | 1 |
   | false | false | Err |
   
   | safe | lossy | 1000(Micro) -> Mills |
   | -- | -- | -- |
   |  true | true | 1 |
   |  true | false | 1 |
   |  false | true | 1 |
   |  false | false | 1 |
   
   
   The specific steps are as follows:
   1. Add a lossy flag to `arrow-cast/CastOptions`.
   2. Unify `CastOptions` usage in `parquet-variant-compute`.
   2.1 Need to change the behavior for those that can't be cast safely. Currently, the default behavior of `arrow-cast/CastOptions` and `parquet-compute-variant/CastOptions` is different; the default value for [`arrow-cast/CastOptions` is `safe=true`](https://github.com/apache/arrow-rs/blob/c9fca0b1248911daeab5e92146d33a58f11e15d2/arrow-cast/src/cast/mod.rs#L83-L90), and [`parquet-variant-compute/CastOptions` is `strict=true`(which is that `safe=false`)](https://github.com/apache/arrow-rs/blob/c9fca0b1248911daeab5e92146d33a58f11e15d2/parquet-variant-compute/src/type_conversion.rs#L35-L39)
   3. After adding  `lossy` in `arrow-cast/CastOptions`, we can update parquet-variant-compute to adapt the corresponding logic.
   
   Steps 1 and 2 do not depend on each other, and can push forward in parallel
   
   Steps 1 and 2 will break the Public API; we need to do them in a major version
   - Step 1 will add a new field in pub struct
   - Step 2 will change the parameter and default behavior for some functions
   
   
   Currently, the `CastOptions` usages in `parquet-variant-compute` are as follows:
   - `shred_variant.rs/variant_to_arrow.rs/variant_get.rs` use `arrow-cast/CastOptions.`
   - `arrow_to_variant.rs/cast_to_variant.rs/lib.rs(re-export)` use `parquet-variant-compute/CastOptions`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Unify the cast option usage in ParquentVariant [arrow-rs]

Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on issue #8873:
URL: https://github.com/apache/arrow-rs/issues/8873#issuecomment-3634477628

   > Steps 1 and 2 will break the Public API; we need to do them in a major version
   > 
   >     * Step 1 will add a new field in pub struct
   > 
   >     * Step 2 will change the parameter and default behavior for some functions
   
   Variant is still an experimental feature, which I _think_ means we don't have to wait for major version bumps to break compat?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Unify the cast option usage in ParquentVariant [arrow-rs]

Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on issue #8873:
URL: https://github.com/apache/arrow-rs/issues/8873#issuecomment-3634486887

   > safe 	lossy 	1234(Micro) -> Mills)
   > true 	true 	1 (= floor(1234 / 1_000))
   > true 	false 	Null
   > false 	true 	1
   > false 	false 	Err
   > safe 	lossy 	1000(Micro) -> Mills
   > true 	true 	1
   > true 	false 	1
   > false 	true 	1
   > false 	false 	1
   
   The two tables suggest there are only three actual values? (lossy, *), (lossless, safe), (lossless, unsafe)?
   Or would (lossy, safe) and (lossy, unsafe) cases still matter when dealing with incompatible types?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [Variant] Unify the cast option usage in ParquentVariant [arrow-rs]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #8873: [Variant] Unify the cast option usage in ParquentVariant
URL: https://github.com/apache/arrow-rs/issues/8873


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org