[jira] [Assigned] (SPARK-25354) Parquet vectorized record reader has unneeded operation in several methods

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-25354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-25354:
------------------------------------

    Assignee:     (was: Apache Spark)

> Parquet vectorized record reader has unneeded operation in several methods
> --------------------------------------------------------------------------
>
>                 Key: SPARK-25354
>                 URL: https://issues.apache.org/jira/browse/SPARK-25354
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.4.0
>            Reporter: SongYadong
>            Priority: Major
>   Original Estimate: 6h
>  Remaining Estimate: 6h
>
> VectorizedParquetRecordReader class has unneeded operation in nextKeyValue method and other functions called from it:
> 1. In nextKeyValue() method, we call resultBatch() for only initializing a columnar batch if not initialized, not for a return of columnar batch. so we can move initBatch() operation to nextBatch();
> 2. In nextBatch() method, we need not reset columnVectors every time. When rowsReturned >= totalRowCount, function return, reset cost is vasted. so we can put "if (rowsReturned >= totalRowCount) return false;" before columnVectors reset for performance.
> 3. In nextBatch() method, we need not call checkEndOfRowGroup() every time. When rowsReturned != totalCountLoadedSoFar is true, checkEndOfRowGroup do nothing but just return, so we can call checkEndOfRowGroup only when rowsReturned == totalCountLoadedSoFar for reducing function calling.
> 4. In checkEndOfRowGroup() function, we need not get columns of requestedSchema every time. we can get columns only for the first time and save it for future use for performance.
> Accoring to analysis of spark application with JMC tool, we found parquet vectorized record reader call nextKeyValue() and subsequent function very very frequent, performance gains from optimizition of this process is worth to do.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org