You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Josh Rosen (Jira)" <ji...@apache.org> on 2019/12/24 01:42:00 UTC

[jira] [Created] (SPARK-30338) Avoid unnecessary InternalRow copies in ParquetRowConverter

Josh Rosen created SPARK-30338:
----------------------------------

             Summary: Avoid unnecessary InternalRow copies in ParquetRowConverter
                 Key: SPARK-30338
                 URL: https://issues.apache.org/jira/browse/SPARK-30338
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 3.0.0
            Reporter: Josh Rosen
            Assignee: Josh Rosen


ParquetRowConverter calls {{InternalRow.copy()}} in cases where the copy is unnecessary; this can severely harm performance when reading deeply-nested Parquet.

It looks like this copying was originally added to handle arrays and maps of structs (in which case we need to keep the copying), but we can omit it for the more common case of structs nested directly in structs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org