You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2024/09/11 15:31:37 UTC
[PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
andygrove opened a new pull request, #6383:
URL: https://github.com/apache/arrow-rs/pull/6383
# Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes https://github.com/apache/arrow-rs/issues/6381
# Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
Fix a correct issue described in https://github.com/apache/arrow-rs/issues/6381
# What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
- Fix cast from decimal to dictionary so that the resulting array has the correct precision and scale
- Add new tests
# Are there any user-facing changes?
Yes, fixes a correctness issue
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!---
If there are any breaking changes to public APIs, please add the `breaking change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
Posted by "jhorstmann (via GitHub)" <gi...@apache.org>.
jhorstmann commented on code in PR #6383:
URL: https://github.com/apache/arrow-rs/pull/6383#discussion_r1758773588
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -203,10 +203,36 @@ pub(crate) fn cast_to_dictionary<K: ArrowDictionaryKeyType>(
UInt32 => pack_numeric_to_dictionary::<K, UInt32Type>(array, dict_value_type, cast_options),
UInt64 => pack_numeric_to_dictionary::<K, UInt64Type>(array, dict_value_type, cast_options),
Decimal128(_, _) => {
- pack_numeric_to_dictionary::<K, Decimal128Type>(array, dict_value_type, cast_options)
+ // pack_numeric_to_dictionary loses the precision and scale so we have to perform a
+ // second cast
+ let decimal_dict_max_precision_scale = pack_numeric_to_dictionary::<K, Decimal128Type>(
Review Comment:
The `PrimitiveDictionaryBuilder` used during casting seems to already loose the custom precision and scale. Its `new` or `with_capacity` methods use the default data type of the generic `ArrowPrimitiveType`. Maybe adding a `with_capacity_and_data_type` method that calls `values_builder.with_data_type` could be a more general fix.
https://github.com/apache/arrow-rs/blob/53.0.0/arrow-array/src/builder/primitive_builder.rs#L152
https://github.com/apache/arrow-rs/blob/53.0.0/arrow-cast/src/cast/dictionary.rs#L299
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6383:
URL: https://github.com/apache/arrow-rs/pull/6383#discussion_r1759200026
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -203,10 +203,36 @@ pub(crate) fn cast_to_dictionary<K: ArrowDictionaryKeyType>(
UInt32 => pack_numeric_to_dictionary::<K, UInt32Type>(array, dict_value_type, cast_options),
UInt64 => pack_numeric_to_dictionary::<K, UInt64Type>(array, dict_value_type, cast_options),
Decimal128(_, _) => {
- pack_numeric_to_dictionary::<K, Decimal128Type>(array, dict_value_type, cast_options)
+ // pack_numeric_to_dictionary loses the precision and scale so we have to perform a
+ // second cast
+ let decimal_dict_max_precision_scale = pack_numeric_to_dictionary::<K, Decimal128Type>(
Review Comment:
> I'm not sure how to do that. Could you give me some guidance?
```rust
Decimal128(p, s) => {
let dict = pack_numeric_to_dictionary::<K, Decimal128Type>(array, dict_value_type, cast_options)?
let dict = dict.as_dictionary::<K>().downcast_dict::<Decimal128Array>().unwrap();
let value = dict.values().clone();
// Set correct precision/scale
let value = value.with_precision_and_scale(p, s)?;
Ok(Arc::new(DictionaryArray::<K>::from((dict.keys().clone(), value)))
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6383:
URL: https://github.com/apache/arrow-rs/pull/6383#discussion_r1755426812
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -203,10 +203,36 @@ pub(crate) fn cast_to_dictionary<K: ArrowDictionaryKeyType>(
UInt32 => pack_numeric_to_dictionary::<K, UInt32Type>(array, dict_value_type, cast_options),
UInt64 => pack_numeric_to_dictionary::<K, UInt64Type>(array, dict_value_type, cast_options),
Decimal128(_, _) => {
- pack_numeric_to_dictionary::<K, Decimal128Type>(array, dict_value_type, cast_options)
+ // pack_numeric_to_dictionary loses the precision and scale so we have to perform a
+ // second cast
+ let decimal_dict_max_precision_scale = pack_numeric_to_dictionary::<K, Decimal128Type>(
Review Comment:
Hmm, looks like `pack_numeric_to_dictionary` did casting on dictionary values correctly. I think the issue you encountered is only the incorrect data type (i.e., it uses default precision/scale on dictionary type after casting).
It is because `pack_numeric_to_dictionary` takes second type parameter `Decimal128Type` and use it as dictionary value type in `PrimitiveDictionaryBuilder`. It will then use `Decimal128Type.DEFAULT_TYPE` as the value type.
We don't need to do second cast, but just need to change the dictionary value type after casting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6383:
URL: https://github.com/apache/arrow-rs/pull/6383#discussion_r1759200026
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -203,10 +203,36 @@ pub(crate) fn cast_to_dictionary<K: ArrowDictionaryKeyType>(
UInt32 => pack_numeric_to_dictionary::<K, UInt32Type>(array, dict_value_type, cast_options),
UInt64 => pack_numeric_to_dictionary::<K, UInt64Type>(array, dict_value_type, cast_options),
Decimal128(_, _) => {
- pack_numeric_to_dictionary::<K, Decimal128Type>(array, dict_value_type, cast_options)
+ // pack_numeric_to_dictionary loses the precision and scale so we have to perform a
+ // second cast
+ let decimal_dict_max_precision_scale = pack_numeric_to_dictionary::<K, Decimal128Type>(
Review Comment:
> I'm not sure how to do that. Could you give me some guidance?
```rust
Decimal128(p, s) => {
let dict = pack_numeric_to_dictionary::<K, Decimal128Type>(array, dict_value_type, cast_options)?
let dict = dict.as_dictionary::<K>().downcast_dict::<Decimal128Array>().unwrap();
let value = dict.values().clone();
// Set correct precision/scale
let value = value.with_precision_and_scale(p, s)?;
Ok(Arc::new(DictionaryArray::<K>::from((dict.keys().clone(), value)))
}
```
For `Decimal256`, we can do similarly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6383:
URL: https://github.com/apache/arrow-rs/pull/6383#discussion_r1765700144
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -203,10 +203,36 @@ pub(crate) fn cast_to_dictionary<K: ArrowDictionaryKeyType>(
UInt32 => pack_numeric_to_dictionary::<K, UInt32Type>(array, dict_value_type, cast_options),
UInt64 => pack_numeric_to_dictionary::<K, UInt64Type>(array, dict_value_type, cast_options),
Decimal128(_, _) => {
- pack_numeric_to_dictionary::<K, Decimal128Type>(array, dict_value_type, cast_options)
+ // pack_numeric_to_dictionary loses the precision and scale so we have to perform a
+ // second cast
+ let decimal_dict_max_precision_scale = pack_numeric_to_dictionary::<K, Decimal128Type>(
Review Comment:
Thanks, I have added this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on code in PR #6383:
URL: https://github.com/apache/arrow-rs/pull/6383#discussion_r1756050534
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -203,10 +203,36 @@ pub(crate) fn cast_to_dictionary<K: ArrowDictionaryKeyType>(
UInt32 => pack_numeric_to_dictionary::<K, UInt32Type>(array, dict_value_type, cast_options),
UInt64 => pack_numeric_to_dictionary::<K, UInt64Type>(array, dict_value_type, cast_options),
Decimal128(_, _) => {
- pack_numeric_to_dictionary::<K, Decimal128Type>(array, dict_value_type, cast_options)
+ // pack_numeric_to_dictionary loses the precision and scale so we have to perform a
+ // second cast
+ let decimal_dict_max_precision_scale = pack_numeric_to_dictionary::<K, Decimal128Type>(
Review Comment:
> We don't need to do second cast, but just need to change the dictionary value type after casting.
I'm not sure how to do that. Could you give me some guidance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6383:
URL: https://github.com/apache/arrow-rs/pull/6383#issuecomment-2344005310
@viirya @parthchandra this PR fixes a correctness issue that I ran into when working on dictionary support in Comet. I expect there is a more efficient way to do this though so could use some guidance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
Posted by "Dandandan (via GitHub)" <gi...@apache.org>.
Dandandan merged PR #6383:
URL: https://github.com/apache/arrow-rs/pull/6383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on code in PR #6383:
URL: https://github.com/apache/arrow-rs/pull/6383#discussion_r1759200026
##########
arrow-cast/src/cast/dictionary.rs:
##########
@@ -203,10 +203,36 @@ pub(crate) fn cast_to_dictionary<K: ArrowDictionaryKeyType>(
UInt32 => pack_numeric_to_dictionary::<K, UInt32Type>(array, dict_value_type, cast_options),
UInt64 => pack_numeric_to_dictionary::<K, UInt64Type>(array, dict_value_type, cast_options),
Decimal128(_, _) => {
- pack_numeric_to_dictionary::<K, Decimal128Type>(array, dict_value_type, cast_options)
+ // pack_numeric_to_dictionary loses the precision and scale so we have to perform a
+ // second cast
+ let decimal_dict_max_precision_scale = pack_numeric_to_dictionary::<K, Decimal128Type>(
Review Comment:
> I'm not sure how to do that. Could you give me some guidance?
```rust
Decimal128(p, s) => {
let dict = pack_numeric_to_dictionary::<K, Decimal128Type>(array, dict_value_type, cast_options)?
let dict = dict.as_dictionary::<K>().downcast_dict::<Decimal128Array>().unwrap();
let value = dict.values().clone();
let value = value.with_precision_and_scale(p, s)?;
Ok(Arc::new(DictionaryArray::<K>::from((dict.keys().clone(), value)))
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] fix: Stop losing precision and scale when casting decimal to dictionary [arrow-rs]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6383:
URL: https://github.com/apache/arrow-rs/pull/6383#issuecomment-2344509591
> Are parquet-testing and testing changes intentional?
oops, no .. reverted. Thanks for catching that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org