You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "aiguofer (via GitHub)" <gi...@apache.org> on 2024/01/06 00:03:10 UTC
[PR] GH-39484: [Java] Support 256 bit decimals in JdbcToArrowUtils [arrow]
aiguofer opened a new pull request, #39485:
URL: https://github.com/apache/arrow/pull/39485
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
This PR allows users of `JdbcToArrowUtils` to convert 256 bit decimals.
### What changes are included in this PR?
Add a `Decimal256Consumer` and logic to
### Are these changes tested?
No, at this point there are no good tests for JDBC consumers.
### Are there any user-facing changes?
Converting 256 bit decimals and ints bigger than 64 bit should now work
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39484: [Java] Support 256 bit decimals in JdbcToArrowUtils [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #39485:
URL: https://github.com/apache/arrow/pull/39485#discussion_r1444938080
##########
java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java:
##########
@@ -169,7 +171,11 @@ public static ArrowType getArrowTypeFromJdbcType(final JdbcFieldInfo fieldInfo,
case Types.DECIMAL:
int precision = fieldInfo.getPrecision();
int scale = fieldInfo.getScale();
- return new ArrowType.Decimal(precision, scale, 128);
+ if (precision > 38) {
Review Comment:
Should be right. Arrow Decimal128 maxes out at 38 as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39484: [Java] Support 256 bit decimals in JdbcToArrowUtils [arrow]
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on code in PR #39485:
URL: https://github.com/apache/arrow/pull/39485#discussion_r1443546283
##########
java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java:
##########
@@ -164,12 +167,21 @@ public static ArrowType getArrowTypeFromJdbcType(final JdbcFieldInfo fieldInfo,
case Types.INTEGER:
return new ArrowType.Int(32, true);
case Types.BIGINT:
- return new ArrowType.Int(64, true);
+ precision = fieldInfo.getPrecision();
+ if (precision > 18) {
+ return new ArrowType.Decimal(precision, 0);
Review Comment:
Nvm... I removed this since it broke some tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39484: [Java] Support 256 bit decimals in JdbcToArrowUtils [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm merged PR #39485:
URL: https://github.com/apache/arrow/pull/39485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39484: [Java] Support 256 bit decimals in JdbcToArrowUtils [arrow]
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on code in PR #39485:
URL: https://github.com/apache/arrow/pull/39485#discussion_r1443533290
##########
java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java:
##########
@@ -169,7 +171,11 @@ public static ArrowType getArrowTypeFromJdbcType(final JdbcFieldInfo fieldInfo,
case Types.DECIMAL:
int precision = fieldInfo.getPrecision();
int scale = fieldInfo.getScale();
- return new ArrowType.Decimal(precision, scale, 128);
+ if (precision > 38) {
Review Comment:
Is this correct? or should it be `precision + scale`? I'm not entirely sure how scale plays into how the values are stored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39484: [Java] Support 256 bit decimals in JdbcToArrowUtils [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39485:
URL: https://github.com/apache/arrow/pull/39485#issuecomment-1879420623
:warning: GitHub issue #39484 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39484: [Java] Support 256 bit decimals in JdbcToArrowUtils [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #39485:
URL: https://github.com/apache/arrow/pull/39485#issuecomment-1882920542
After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 1622a2e945d35fbb6848d0053b3ae30045d3fbda.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/20299192499) has more details. It also includes information about 1 possible false positive for unstable benchmarks that are known to sometimes produce them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39484: [Java] Support 256 bit decimals in JdbcToArrowUtils [arrow]
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on code in PR #39485:
URL: https://github.com/apache/arrow/pull/39485#discussion_r1443544489
##########
java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java:
##########
@@ -164,12 +167,21 @@ public static ArrowType getArrowTypeFromJdbcType(final JdbcFieldInfo fieldInfo,
case Types.INTEGER:
return new ArrowType.Int(32, true);
case Types.BIGINT:
- return new ArrowType.Int(64, true);
+ precision = fieldInfo.getPrecision();
+ if (precision > 18) {
+ return new ArrowType.Decimal(precision, 0);
Review Comment:
We ran into an issue with this on Snowflake, which can have precision up to 38 (128 bit). Although technically some 19 precision numbers can be expressed in 64 bit, the safest way to ensure the conversion always works seems to be setting it to Decimal if the column precision > 18.
I can remove this, or add it in a separate PR if ya'll think think that's better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39484: [Java] Support 256 bit decimals in JdbcToArrowUtils [arrow]
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on code in PR #39485:
URL: https://github.com/apache/arrow/pull/39485#discussion_r1443544489
##########
java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java:
##########
@@ -164,12 +167,21 @@ public static ArrowType getArrowTypeFromJdbcType(final JdbcFieldInfo fieldInfo,
case Types.INTEGER:
return new ArrowType.Int(32, true);
case Types.BIGINT:
- return new ArrowType.Int(64, true);
+ precision = fieldInfo.getPrecision();
+ if (precision > 18) {
+ return new ArrowType.Decimal(precision, 0);
Review Comment:
We ran into an issue with this on Snowflake. Although technically some 19 precision numbers can be expressed in 64 bit, the safest way to ensure the conversion always works seems to be doing this.
I can remove this, or add it in a separate PR if ya'll think think that's better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-39484: [Java] Support 256 bit decimals in JdbcToArrowUtils [arrow]
Posted by "aiguofer (via GitHub)" <gi...@apache.org>.
aiguofer commented on code in PR #39485:
URL: https://github.com/apache/arrow/pull/39485#discussion_r1443533290
##########
java/adapter/jdbc/src/main/java/org/apache/arrow/adapter/jdbc/JdbcToArrowUtils.java:
##########
@@ -169,7 +171,11 @@ public static ArrowType getArrowTypeFromJdbcType(final JdbcFieldInfo fieldInfo,
case Types.DECIMAL:
int precision = fieldInfo.getPrecision();
int scale = fieldInfo.getScale();
- return new ArrowType.Decimal(precision, scale, 128);
+ if (precision > 38) {
Review Comment:
Is this correct? this is how BigQuery seems to handle it, but not sure if it's any different on the Arrow side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org