You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/06/28 22:28:51 UTC
[GitHub] [arrow] tachyonwill opened a new pull request, #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays
tachyonwill opened a new pull request, #13456:
URL: https://github.com/apache/arrow/pull/13456
The precision calculation had been overflowing to infinity when the
length of the fixed_len_byte_array > 128, triggering an error when then
trying to convert infinity to an int32. We can actually simplify the
logic by noting that log_b(a^(x)) = log_b(a)*x. This avoids the
intermediate infinity. We also added a check for extremely large value
sizes implying a max precision that cannot fit in int32. Even 129 byte
decimal seems extreme.
The formula Parquet C++ was using is technically incorrect vs the
Parquet specification. The specification says that the max precision is
floor(log_10(2^(B*8 -1) - 1)), where the C++ implementation was omitting the
outer -1. However, this is okay as it is easy to prove that these values
will always be the same (ignoring the realities of FP arithmetic) & in
practice all three formulas agree through 128 when using FP.
Bug found through fuzzing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] tachyonwill commented on a diff in pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays
Posted by GitBox <gi...@apache.org>.
tachyonwill commented on code in PR #13456:
URL: https://github.com/apache/arrow/pull/13456#discussion_r910462791
##########
cpp/src/parquet/schema_test.cc:
##########
@@ -1691,6 +1691,18 @@ TEST(TestSchemaNodeCreation, FactoryExceptions) {
// Primitive too small for given precision ...
ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
DecimalLogicalType::Make(16, 6), Type::INT32));
+ ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
+ DecimalLogicalType::Make(10, 9), Type::INT32));
+ ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
+ DecimalLogicalType::Make(17, 18), Type::INT64));
Review Comment:
Yes, added a comment and a new test with precision=19.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield commented on a diff in pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays
Posted by GitBox <gi...@apache.org>.
emkornfield commented on code in PR #13456:
URL: https://github.com/apache/arrow/pull/13456#discussion_r910460401
##########
cpp/src/parquet/schema_test.cc:
##########
@@ -1691,6 +1691,18 @@ TEST(TestSchemaNodeCreation, FactoryExceptions) {
// Primitive too small for given precision ...
ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
DecimalLogicalType::Make(16, 6), Type::INT32));
+ ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
+ DecimalLogicalType::Make(10, 9), Type::INT32));
+ ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
+ DecimalLogicalType::Make(17, 18), Type::INT64));
Review Comment:
this looks like it should fail for other reasons (scale greater than precision).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13456:
URL: https://github.com/apache/arrow/pull/13456#issuecomment-1169353022
:warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13456:
URL: https://github.com/apache/arrow/pull/13456#issuecomment-1177634444
['Python', 'R'] benchmarks have high level of regressions.
[ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/779d141994d14242936455632e4daf6e...357b2aca986847279608608459eecea6/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] emkornfield closed pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays
Posted by GitBox <gi...@apache.org>.
emkornfield closed pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays
URL: https://github.com/apache/arrow/pull/13456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13456:
URL: https://github.com/apache/arrow/pull/13456#issuecomment-1169353012
https://issues.apache.org/jira/browse/PARQUET-2163
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays
Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13456:
URL: https://github.com/apache/arrow/pull/13456#issuecomment-1177634218
Benchmark runs are scheduled for baseline = 2c29205b32af2a65df3278333583103a17a99972 and contender = 423ca163a26781ef6a8229af22b7e6e2d7423a54. 423ca163a26781ef6a8229af22b7e6e2d7423a54 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/5406a3c4074e48e78bab5be2d4baae54...7175ffb8ce4f43da973085db914ecb71/)
[Failed :arrow_down:0.0% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/80608bc4a64245ebae8127f220363438...2db89db77bd545fca9825eca2ab47bd5/)
[Finished :arrow_down:0.27% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/779d141994d14242936455632e4daf6e...357b2aca986847279608608459eecea6/)
[Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/8d3b7ca591744b70837bf3368bfb57a4...a87c97669929479e9abbcbaaa04935f5/)
Buildkite builds:
[Finished] [`423ca163` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1058)
[Failed] [`423ca163` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1059)
[Finished] [`423ca163` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1049)
[Failed] [`423ca163` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1063)
[Finished] [`2c29205b` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1057)
[Failed] [`2c29205b` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1058)
[Finished] [`2c29205b` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1048)
[Failed] [`2c29205b` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1062)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org