You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2019/06/11 18:03:00 UTC
[jira] [Commented] (PARQUET-1580) Page-level CRC checksum
verification for DataPageV1
[ https://issues.apache.org/jira/browse/PARQUET-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16861280#comment-16861280 ]
ASF GitHub Bot commented on PARQUET-1580:
-----------------------------------------
bbraams commented on pull request #647: PARQUET-1580: Page-level CRC checksum verfication for DataPageV1
URL: https://github.com/apache/parquet-mr/pull/647
This PR implements page-level CRC checksums for DataPageV1. It is the implementation follow up to the clarification of the checksums in `parquet-format` ([Jira](https://jira.apache.org/jira/browse/PARQUET-1539), [PR](https://github.com/apache/parquet-format/pull/126)). Key points:
- Checksums are calculated using the CRC32 implementation in `java.util.zip`
- Disabled by default, writing out checksums on the write path and verification on the read path are feature flagged individually (see Documentation below)
- Backwards compatibility with Parquet files that do not have the CRC field set, even if checksum verification is turned on
Please excuse the size of the PR, the bulk of it is config propagation boilerplate.
### Jira
- [x] My PR addresses the following [Parquet Jira](https://jira.apache.org/jira/browse/PARQUET-1580) issue:
- https://jira.apache.org/jira/browse/PARQUET-1580
### Tests
This PR adds the following tests in a new test suite `TestDataPageV1Checksums`:
- Test writing out checksums without verification on read, verify calculated checksums are correct
- Test not writing out checksums without verification on read, tests feature can be turned off entirely
- Test not writing out checksums with verification on read, tests backward compatibility of read path with files without checksums
- Test writing out checksums with verification on read, tests that the checksum is correctly set and that we can read what we wrote
- Test corruption detection by corrupting bytes in data section of pages, tests that these corruptions go undetected with the feature disabled and are detected with the feature enabled
- Test checksums with compression enabled, tests that the checksum is calculated on the compressed version of the data (as per the specification)
- Test checksums with compression and nested schema, tests that the checksum is calculated on the compressed concatenation of the repetition levels, definition levels and the actual data (as per the specification)
### Documentation
The feature is feature flagged and is disabled by default. Both writing out checksums and verifying them on the read path can be turned on individually, via the following two new config flags:
* `parquet.page.write-checksum.enabled` (default: false)
* `parquet.page.verify-checksum.enabled` (default: false)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
> Page-level CRC checksum verification for DataPageV1
> ---------------------------------------------------
>
> Key: PARQUET-1580
> URL: https://issues.apache.org/jira/browse/PARQUET-1580
> Project: Parquet
> Issue Type: Improvement
> Components: parquet-mr
> Reporter: Boudewijn Braams
> Assignee: Boudewijn Braams
> Priority: Major
> Labels: pull-request-available
>
> Now that the specification with respect to page level checksums has been clarified (see https://jira.apache.org/jira/browse/PARQUET-1539), we can actually implement it. What needs to be done:
> * Implement calculating and writing checksums on write path
> * Implement checksum validation on read path
> * Make writing out checksums and validation on reads optional (default off)
> * Implement tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)