You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mapleFU (via GitHub)" <gi...@apache.org> on 2023/04/20 03:38:35 UTC

[GitHub] [arrow] mapleFU opened a new pull request, #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

mapleFU opened a new pull request, #35242:
URL: https://github.com/apache/arrow/pull/35242

   <!--
   Thanks for opening a pull request!
   If this is your first pull request you can find detailed information on how 
   to contribute here:
     * [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
     * [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
   
   
   If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
   
   Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
   
   Then could you also rename the pull request title in the following format?
   
       GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   or
   
       MINOR: [${COMPONENT}] ${SUMMARY}
   
   In the case of PARQUET issues on JIRA the title also supports:
   
       PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
   
   -->
   
   ### Rationale for this change
   
   <!--
    Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
   -->
   
   Implement crc for data page v2.
   
   ### What changes are included in this PR?
   
   <!--
   There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   1. When writing, able to write crc for data page v2
   2. When reading, able to read crc for data page v2
   3. Testing
   
   ### Are these changes tested?
   
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   4. Serve as another way to document the expected behavior of the code
   
   If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
   -->
   
   Yes, but parquet-testing don't have crc page v2 test. So no integration test
   
   ### Are there any user-facing changes?
   
   <!--
   If there are user-facing changes then we may require documentation to be updated before approving the PR.
   -->
   
   <!--
   If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
   -->
   <!-- **This PR includes breaking changes to public APIs.** -->
   
   <!--
   Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
   -->
   <!-- **This PR contains a "Critical Fix".** -->


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] ursabot commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1530649477

   Benchmark runs are scheduled for baseline = da6dbd48607089d716505054176e345b704570c5 and contender = 99ac74e6890767e1a3cf96d7aff0b96b9503153c. 99ac74e6890767e1a3cf96d7aff0b96b9503153c is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
   Conbench compare runs links:
   [Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/791e9d8a35684207938b385c938159f8...48452dd0b5a1435996191c282fe7a073/)
   [Failed :arrow_down:2.63% :arrow_up:0.15%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/4d474f672c074c97bffae72b2301f60e...6331adfa453343f1ab3db9d2d59c4f96/)
   [Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/2ec950b628834cf2bbd058a74bbedc9b...5378984264184d6eb9445a08ae5abd9d/)
   [Finished :arrow_down:0.51% :arrow_up:0.15%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/a60392c579404baf81f25d8575c07d33...ca813469982a4b6ea2c6a5f0c8f2aa62/)
   Buildkite builds:
   [Finished] [`99ac74e6` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2803)
   [Failed] [`99ac74e6` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2837)
   [Failed] [`99ac74e6` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2801)
   [Finished] [`99ac74e6` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2828)
   [Finished] [`da6dbd48` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2802)
   [Failed] [`da6dbd48` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2836)
   [Failed] [`da6dbd48` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2800)
   [Finished] [`da6dbd48` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2827)
   Supported benchmarks:
   ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
   test-mac-arm: Supported benchmark langs: C++, Python, R
   ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
   ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1515720236

   Oh, it's not that problem, damn...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1518558830

   The fuzzing find out that:
   
   ```
   /arrow/cpp/src/generated/parquet_types.cpp:4425:18: runtime error: load of value 16, which is not a valid value for type 'PageType::type'
   SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /arrow/cpp/src/generated/parquet_types.cpp:4425:18 in 
   1
   ```
   
   I guess previous it already has that problem...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on a diff in pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #35242:
URL: https://github.com/apache/arrow/pull/35242#discussion_r1176135459


##########
cpp/src/parquet/file_deserialize_test.cc:
##########
@@ -814,6 +834,30 @@ TEST_F(TestPageSerde, DictCrcCheckNonExistent) {
                          /* verification_checksum */ true, /* has_dictionary */ true);
 }
 
+TEST_F(TestPageSerde, DataPageV2CrcCheckSuccessful) {

Review Comment:
   > Should we also add test with `has_dictionary=true` for completeness?
   
   Sure, let me try it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1521315178

   @wjones127 Would you mind take a look?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1515667714

   * Closes: #35171


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1515667697

   @wjones127 @wgtmac PTAL :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on a diff in pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #35242:
URL: https://github.com/apache/arrow/pull/35242#discussion_r1180687834


##########
cpp/src/parquet/arrow/arrow_reader_writer_test.cc:
##########
@@ -4092,7 +4092,7 @@ TEST_P(TestArrowWriteDictionary, Statistics) {
       {{"b", "a"}, {"b", "a"}}, {{"c", "c"}, {"c", "c"}}, {{"d", "a"}, {"d", "a"}}};
   const std::vector<std::vector<std::vector<bool>>> expected_has_min_max_by_page = {
       {{true, true}, {true, true}},
-      {{true, true}, {true, true}},
+      {{true, false}, {true, false}},

Review Comment:
   https://github.com/apache/arrow/pull/35242#issuecomment-1517491968
   
   This is mentioned here. Let me explain it:
   1. Previously, `PageHeader` reuse same object, so `__isset` would not been cleared
   2. So, `has_min_max` will keep "true" if first page has min-max, and second page not has.
   3. In test case, the second page should not has min-max, but it has been set. I think previous test is a bug here. You can check it again



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on a diff in pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #35242:
URL: https://github.com/apache/arrow/pull/35242#discussion_r1176137136


##########
cpp/src/parquet/column_reader.cc:
##########
@@ -487,16 +489,15 @@ std::shared_ptr<Page> SerializedPageReader::NextPage() {
 
     const PageType::type page_type = LoadEnumSafe(&current_page_header_.type);
 
-    // TODO(PARQUET-594) crc checksum for DATA_PAGE_V2
-    if (properties_.page_checksum_verification() &&
-        (page_type == PageType::DATA_PAGE || page_type == PageType::DICTIONARY_PAGE) &&
-        current_page_header_.__isset.crc) {
+    if (properties_.page_checksum_verification() && current_page_header_.__isset.crc) {

Review Comment:
   Yes, so should I extract a pageTypeHasCrc ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] wjones127 commented on a diff in pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #35242:
URL: https://github.com/apache/arrow/pull/35242#discussion_r1180685776


##########
cpp/src/parquet/arrow/arrow_reader_writer_test.cc:
##########
@@ -4092,7 +4092,7 @@ TEST_P(TestArrowWriteDictionary, Statistics) {
       {{"b", "a"}, {"b", "a"}}, {{"c", "c"}, {"c", "c"}}, {{"d", "a"}, {"d", "a"}}};
   const std::vector<std::vector<std::vector<bool>>> expected_has_min_max_by_page = {
       {{true, true}, {true, true}},
-      {{true, true}, {true, true}},
+      {{true, false}, {true, false}},

Review Comment:
   Why did this change?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] wjones127 commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1527912958

   Most of the changes look good. But not sure why that test changed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1518544808

   I will find out why fuzzing failed..


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] wgtmac commented on a diff in pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #35242:
URL: https://github.com/apache/arrow/pull/35242#discussion_r1174492069


##########
cpp/src/parquet/file_deserialize_test.cc:
##########
@@ -814,6 +834,30 @@ TEST_F(TestPageSerde, DictCrcCheckNonExistent) {
                          /* verification_checksum */ true, /* has_dictionary */ true);
 }
 
+TEST_F(TestPageSerde, DataPageV2CrcCheckSuccessful) {

Review Comment:
   Should we also add test with `has_dictionary=true` for completeness?



##########
cpp/src/parquet/column_reader.cc:
##########
@@ -487,16 +489,15 @@ std::shared_ptr<Page> SerializedPageReader::NextPage() {
 
     const PageType::type page_type = LoadEnumSafe(&current_page_header_.type);
 
-    // TODO(PARQUET-594) crc checksum for DATA_PAGE_V2
-    if (properties_.page_checksum_verification() &&
-        (page_type == PageType::DATA_PAGE || page_type == PageType::DICTIONARY_PAGE) &&
-        current_page_header_.__isset.crc) {
+    if (properties_.page_checksum_verification() && current_page_header_.__isset.crc) {

Review Comment:
   This change works for standard parquet files. However, if user has customized parquet to support index page, this may break their workload.



##########
cpp/src/parquet/file_deserialize_test.cc:
##########
@@ -814,6 +834,30 @@ TEST_F(TestPageSerde, DictCrcCheckNonExistent) {
                          /* verification_checksum */ true, /* has_dictionary */ true);
 }
 
+TEST_F(TestPageSerde, DataPageV2CrcCheckSuccessful) {

Review Comment:
   nit: switching to parameterized test would be more favorable here.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1515745988

   After re-create page header, the test below failed:
   
   ```
   [  FAILED  ] 2 tests, listed below:
   [  FAILED  ] WriteDictionary/TestArrowWriteDictionary.Statistics/0, where GetParam() = 4-byte object <00-00 00-00>
   [  FAILED  ] WriteDictionary/TestArrowWriteDictionary.Statistics/1, where GetParam() = 4-byte object <01-00 00-00>
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1527870798

   Comments solved.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] github-actions[bot] commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1515667736

   :warning: GitHub issue #35171 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1524617294

   Gently ping @wjones127 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] wjones127 commented on a diff in pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #35242:
URL: https://github.com/apache/arrow/pull/35242#discussion_r1180697808


##########
cpp/src/parquet/arrow/arrow_reader_writer_test.cc:
##########
@@ -4092,7 +4092,7 @@ TEST_P(TestArrowWriteDictionary, Statistics) {
       {{"b", "a"}, {"b", "a"}}, {{"c", "c"}, {"c", "c"}}, {{"d", "a"}, {"d", "a"}}};
   const std::vector<std::vector<std::vector<bool>>> expected_has_min_max_by_page = {
       {{true, true}, {true, true}},
-      {{true, true}, {true, true}},
+      {{true, false}, {true, false}},

Review Comment:
   That makes sense. Thanks for explaining.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] wjones127 merged pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 merged PR #35242:
URL: https://github.com/apache/arrow/pull/35242


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on a diff in pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #35242:
URL: https://github.com/apache/arrow/pull/35242#discussion_r1176135459


##########
cpp/src/parquet/file_deserialize_test.cc:
##########
@@ -814,6 +834,30 @@ TEST_F(TestPageSerde, DictCrcCheckNonExistent) {
                          /* verification_checksum */ true, /* has_dictionary */ true);
 }
 
+TEST_F(TestPageSerde, DataPageV2CrcCheckSuccessful) {

Review Comment:
   > Should we also add test with `has_dictionary=true` for completeness?
   
   I don't think it will test more code. DICT_PAGE is different from data page. Testing for them is different



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1515708999

   Test failed because parquet-mr in test file write "incorrect" Data Page crc in v2. I will disable them first.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1517491968

   @westonpace Hi Pace, I find that `current_page_header_` message Deserialize will not mark `__isset` field to unset when meet a new message. When I change to use a new header, `TestArrowWriteDictionary.Statistics` failed. And when I go through this unittest, I found that I give a wrong stats.
   
   For test index:
   
   ```C++
   ArrayFromJSON(::arrow::int32(), R"([0, 1, null, 0, 1, null])"),
   ```
   
   It generate:
   
   ```
   RowGroup1 [ Page1 [0, 1] Page2 [ null ] ]
   RowGroup2 [ Page1 [0, 1] Page2 [ null ] ]
   ```
   
   So, I think page2 should not have min-max stats, and I fix it here. Would you mind take a look at this patch? Thanks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1527888938

   ( https://github.com/apache/arrow/pull/35242#issuecomment-1518558830 this happens because deserialized a bad enum value doesn't trigger ubsan, but assign it to another object may trigger...)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] mapleFU commented on pull request #35242: GH-35171: [C++][Parquet] Implement CRC for data page v2

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #35242:
URL: https://github.com/apache/arrow/pull/35242#issuecomment-1527936015

   I guess https://github.com/apache/arrow/actions/runs/4832925654/jobs/8612322729?pr=35242 and https://github.com/apache/arrow/actions/runs/4832925654/jobs/8612322729?pr=35242 are timeout, and https://github.com/apache/arrow/actions/runs/4832925659/jobs/8612318748?pr=35242 is pandas issue. But I'm not sure =_=


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org