You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/06/28 22:28:51 UTC

[GitHub] [arrow] tachyonwill opened a new pull request, #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays

tachyonwill opened a new pull request, #13456:
URL: https://github.com/apache/arrow/pull/13456

   The precision calculation had been overflowing to infinity when the
   length of the fixed_len_byte_array > 128, triggering an error when then
   trying to convert infinity to an int32. We can actually simplify the
   logic by noting that log_b(a^(x)) = log_b(a)*x. This avoids the
   intermediate infinity. We also added a check for extremely large value
   sizes implying a max precision that cannot fit in int32. Even 129 byte
   decimal seems extreme.
   
   The formula Parquet C++ was using is technically incorrect vs the
   Parquet specification. The specification says that the max precision is
   floor(log_10(2^(B*8 -1) - 1)), where the C++ implementation was omitting the
   outer -1. However, this is okay as it is easy to prove that these values
   will always be the same (ignoring the realities of FP arithmetic) & in
   practice all three formulas agree through 128 when using FP.
   
   Bug found through fuzzing.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] tachyonwill commented on a diff in pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays

Posted by GitBox <gi...@apache.org>.
tachyonwill commented on code in PR #13456:
URL: https://github.com/apache/arrow/pull/13456#discussion_r910462791


##########
cpp/src/parquet/schema_test.cc:
##########
@@ -1691,6 +1691,18 @@ TEST(TestSchemaNodeCreation, FactoryExceptions) {
   // Primitive too small for given precision ...
   ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
                                        DecimalLogicalType::Make(16, 6), Type::INT32));
+  ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
+                                       DecimalLogicalType::Make(10, 9), Type::INT32));
+  ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
+                                       DecimalLogicalType::Make(17, 18), Type::INT64));

Review Comment:
   Yes, added a comment and a new test with precision=19.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] emkornfield commented on a diff in pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays

Posted by GitBox <gi...@apache.org>.
emkornfield commented on code in PR #13456:
URL: https://github.com/apache/arrow/pull/13456#discussion_r910460401


##########
cpp/src/parquet/schema_test.cc:
##########
@@ -1691,6 +1691,18 @@ TEST(TestSchemaNodeCreation, FactoryExceptions) {
   // Primitive too small for given precision ...
   ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
                                        DecimalLogicalType::Make(16, 6), Type::INT32));
+  ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
+                                       DecimalLogicalType::Make(10, 9), Type::INT32));
+  ASSERT_ANY_THROW(PrimitiveNode::Make("decimal", Repetition::REQUIRED,
+                                       DecimalLogicalType::Make(17, 18), Type::INT64));

Review Comment:
   this looks like it should fail for other reasons (scale greater than precision).  



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13456:
URL: https://github.com/apache/arrow/pull/13456#issuecomment-1169353022

   :warning: Ticket **has not been started in JIRA**, please click 'Start Progress'.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] ursabot commented on pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13456:
URL: https://github.com/apache/arrow/pull/13456#issuecomment-1177634444

   ['Python', 'R'] benchmarks have high level of regressions.
   [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/779d141994d14242936455632e4daf6e...357b2aca986847279608608459eecea6/)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] emkornfield closed pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays

Posted by GitBox <gi...@apache.org>.
emkornfield closed pull request #13456: PARQUET-2163:  Handle decimal schemas with large fixed_len_byte_arrays
URL: https://github.com/apache/arrow/pull/13456


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] github-actions[bot] commented on pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays

Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #13456:
URL: https://github.com/apache/arrow/pull/13456#issuecomment-1169353012

   https://issues.apache.org/jira/browse/PARQUET-2163


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [arrow] ursabot commented on pull request #13456: PARQUET-2163: Handle decimal schemas with large fixed_len_byte_arrays

Posted by GitBox <gi...@apache.org>.
ursabot commented on PR #13456:
URL: https://github.com/apache/arrow/pull/13456#issuecomment-1177634218

   Benchmark runs are scheduled for baseline = 2c29205b32af2a65df3278333583103a17a99972 and contender = 423ca163a26781ef6a8229af22b7e6e2d7423a54. 423ca163a26781ef6a8229af22b7e6e2d7423a54 is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/5406a3c4074e48e78bab5be2d4baae54...7175ffb8ce4f43da973085db914ecb71/)
   [Failed :arrow_down:0.0% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/80608bc4a64245ebae8127f220363438...2db89db77bd545fca9825eca2ab47bd5/)
   [Finished :arrow_down:0.27% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/779d141994d14242936455632e4daf6e...357b2aca986847279608608459eecea6/)
   [Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/8d3b7ca591744b70837bf3368bfb57a4...a87c97669929479e9abbcbaaa04935f5/)
   Buildkite builds:
   [Finished] [`423ca163` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1058)
   [Failed] [`423ca163` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1059)
   [Finished] [`423ca163` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1049)
   [Failed] [`423ca163` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1063)
   [Finished] [`2c29205b` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/1057)
   [Failed] [`2c29205b` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/1058)
   [Finished] [`2c29205b` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/1048)
   [Failed] [`2c29205b` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/1062)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org