You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@parquet.apache.org by "Tongjie Chen (JIRA)" <ji...@apache.org> on 2014/12/10 20:25:13 UTC

[jira] [Created] (PARQUET-149) provide an option to skip a page in case corrupted bytes occur

Tongjie Chen created PARQUET-149:
------------------------------------

             Summary: provide an option to skip a page in case corrupted bytes occur
                 Key: PARQUET-149
                 URL: https://issues.apache.org/jira/browse/PARQUET-149
             Project: Parquet
          Issue Type: Improvement
            Reporter: Tongjie Chen


In case of hardware failure (disk, memory, etc), there might be corrupted bytes. That will result in ArrayIndexOutOfBoundException or/and data garbled.

Currently, jobs reading those Parquet files will fail unless the corrupted files are deleted/moved.

Currently page metadata has a CRC field (not used so far), which can be used to check integrity of the page. If page data is corrupted, skip the whole page.

related issue: PARQUET-148



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)