[jira] [Commented] (SPARK-26345) Parquet support Column indexes

["Zoltan Ivanfi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-26345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16757209#comment-16757209 ] 

Zoltan Ivanfi commented on SPARK-26345:
---------------------------------------

Please note that column indexes will automatically get utilized if [spark.sql.parquet.enableVectorizedReader|https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-vectorized-parquet-reader.html#spark.sql.parquet.enableVectorizedReader] = false. If spark.sql.parquet.enableVectorizedReader = true, on the other hand (which is the default), then column indexes could only be utilized by duplicating the internal logic of parquet-mr, which would be disproportonate effort. We, the developers of the column index feature did not expect Spark to make this huge investment, and we would like to provide a vectorized API instead in a future release.

> Parquet support Column indexes
> ------------------------------
>
>                 Key: SPARK-26345
>                 URL: https://issues.apache.org/jira/browse/SPARK-26345
>             Project: Spark
>          Issue Type: Umbrella
>          Components: SQL
>    Affects Versions: 3.0.0
>            Reporter: Yuming Wang
>            Priority: Major
>
> Parquet 1.11.0 supports column indexing. Spark can supports this feature for good read performance.
> More details:
> https://issues.apache.org/jira/browse/PARQUET-1201



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org