[jira] [Commented] (SPARK-18860) Update Parquet to 1.9.0

["Apache Spark (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-18860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15748281#comment-15748281 ] 

Apache Spark commented on SPARK-18860:
--------------------------------------

User 'dongjoon-hyun' has created a pull request for this issue:
https://github.com/apache/spark/pull/16281

> Update Parquet to 1.9.0
> -----------------------
>
>                 Key: SPARK-18860
>                 URL: https://issues.apache.org/jira/browse/SPARK-18860
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>            Reporter: Dongjoon Hyun
>
> This issue aims to update Parquet to 1.9.0 and remove the hacks due to Parquet 1.8.1 limitation.
> {code}
> -  // !! HACK ALERT !!
> -  //
> -  // PARQUET-363 & PARQUET-278: parquet-mr 1.8.1 doesn't allow constructing empty GroupType,
> -  // which prevents us to avoid selecting any columns for queries like `SELECT COUNT(*) FROM t`.
> -  // This issue has been fixed in parquet-mr 1.8.2-SNAPSHOT.
> -  //
> -  // To workaround this problem, here we first construct a `MessageType` with a single dummy
> -  // field, and then remove the field to obtain an empty `MessageType`.
> -  //
> -  // TODO Reverts this change after upgrading parquet-mr to 1.8.2+
>    val EMPTY_MESSAGE = Types
>        .buildMessage()
> -      .required(PrimitiveType.PrimitiveTypeName.INT32).named("dummy")
>        .named(ParquetSchemaConverter.SPARK_PARQUET_SCHEMA_NAME)
> -  EMPTY_MESSAGE.getFields.clear()
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org