[jira] [Resolved] (ARROW-13965) [C++] dynamic_casts in parquet TypedColumnWriterImpl impacting performance

["Antoine Pitrou (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-13965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Antoine Pitrou resolved ARROW-13965.
------------------------------------
    Fix Version/s: 6.0.0
       Resolution: Fixed

Issue resolved by pull request 11131
[https://github.com/apache/arrow/pull/11131]

> [C++] dynamic_casts in parquet TypedColumnWriterImpl impacting performance
> --------------------------------------------------------------------------
>
>                 Key: ARROW-13965
>                 URL: https://issues.apache.org/jira/browse/ARROW-13965
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++
>         Environment: arrow 6.0.0-SNAPSHOT on both RHEL8 (gcc 8.4.1) and MacOS 11.5.2 (clang 11.0.0)
>            Reporter: Edward Seidl
>            Priority: Minor
>              Labels: pull-request-available
>             Fix For: 6.0.0
>
>         Attachments: arrow_downcast.patch
>
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> The methods WriteDictionaryPage(), CheckDictionarySizeLimit(), WriteValues(), and WriteValuesSpaced() in TypedColumnWriterImpl (cpp/src/parquet/column_writer.cc) perform dynamic_casts of the current_dict_ object to either DictEncoder or ValueEncoderType pointers.  When calling WriteBatch() with a large number of values this is ok, but when writing batches of 1 (as when using the stream api), these dynamic casts can consume a great deal of cpu.  Using gperftools against code I wrote to do a log structured merge of several parquet files, I measured the dynamic_casts taking as much as 25% of execution time.
> By modifying TypedColumnWriterImpl to save downcasted observer pointers of the appropriate types, I was able to cut my execution time from 32 to 24 seconds, validating the gpertools results.  I've attached a patch to show what I did.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)