You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dongjoon Hyun (JIRA)" <ji...@apache.org> on 2016/12/14 18:47:58 UTC
[jira] [Closed] (SPARK-18860) Update Parquet to 1.9.0
[ https://issues.apache.org/jira/browse/SPARK-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun closed SPARK-18860.
---------------------------------
Resolution: Fixed
> Update Parquet to 1.9.0
> -----------------------
>
> Key: SPARK-18860
> URL: https://issues.apache.org/jira/browse/SPARK-18860
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Reporter: Dongjoon Hyun
>
> This issue aims to update Parquet to 1.9.0 and remove the hacks due to Parquet 1.8.1 limitation.
> {code}
> - // !! HACK ALERT !!
> - //
> - // PARQUET-363 & PARQUET-278: parquet-mr 1.8.1 doesn't allow constructing empty GroupType,
> - // which prevents us to avoid selecting any columns for queries like `SELECT COUNT(*) FROM t`.
> - // This issue has been fixed in parquet-mr 1.8.2-SNAPSHOT.
> - //
> - // To workaround this problem, here we first construct a `MessageType` with a single dummy
> - // field, and then remove the field to obtain an empty `MessageType`.
> - //
> - // TODO Reverts this change after upgrading parquet-mr to 1.8.2+
> val EMPTY_MESSAGE = Types
> .buildMessage()
> - .required(PrimitiveType.PrimitiveTypeName.INT32).named("dummy")
> .named(ParquetSchemaConverter.SPARK_PARQUET_SCHEMA_NAME)
> - EMPTY_MESSAGE.getFields.clear()
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org