You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Micah Kornfield (Jira)" <ji...@apache.org> on 2021/10/08 05:35:00 UTC
[jira] [Commented] (SPARK-34276) Check the unreleased/unresolved
JIRAs/PRs of Parquet 1.11 and 1.12
[ https://issues.apache.org/jira/browse/SPARK-34276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17425959#comment-17425959 ]
Micah Kornfield commented on SPARK-34276:
-----------------------------------------
Sorry for the late reply. PARQUET-2089 has been a long standing bug in the C++ implementation where we were setting file_offset to the beginning of column_chunk metatadata and not the actual data page. It's not clear to me if this was a problem before parquet-mr 1.12 in practice. [~gershinsky] Would the fix in PARQUET-2078 make parquet-mr resilient to this bug?
> Check the unreleased/unresolved JIRAs/PRs of Parquet 1.11 and 1.12
> ------------------------------------------------------------------
>
> Key: SPARK-34276
> URL: https://issues.apache.org/jira/browse/SPARK-34276
> Project: Spark
> Issue Type: Task
> Components: Build, SQL
> Affects Versions: 3.2.0
> Reporter: Yuming Wang
> Assignee: Chao Sun
> Priority: Blocker
>
> Before the release, we need to double check the unreleased/unresolved JIRAs/PRs of Parquet 1.11/1.12 and then decide whether we should upgrade/revert Parquet. At the same time, we should encourage the whole community to do the compatibility and performance tests for their production workloads, including both read and write code paths.
> More details: [https://github.com/apache/spark/pull/26804#issuecomment-768790620]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org