[jira] [Commented] (AVRO-3156) Performance degradation in SpecificRecordBuilder introduced in 1.9.0

["Erik Krogen (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AVRO-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421809#comment-17421809 ] 

Erik Krogen commented on AVRO-3156:
-----------------------------------

[~steven.aerts] this should be fixed by [#1206|https://github.com/apache/avro/pull/1206] from AVRO-3048, can you take a look?

> Performance degradation in SpecificRecordBuilder introduced in 1.9.0
> --------------------------------------------------------------------
>
>                 Key: AVRO-3156
>                 URL: https://issues.apache.org/jira/browse/AVRO-3156
>             Project: Apache Avro
>          Issue Type: Improvement
>          Components: java
>    Affects Versions: 1.9.0, 1.10.0
>         Environment: Using SpecificData in environments with multiple classloaders.
>            Reporter: Steven Aerts
>            Priority: Major
>         Attachments: image-2021-06-11-14-27-16-689.png
>
>
> The change introduced in spark 1.9.0 which changed:
> {code:java}
>  SpecificData.get(){code}
> into:
> {code:java}
> SpecificData.getForSchema(schema){code}
> introduced a significant performance degradation in environments where the class of {{schema}} is provided by a different classloader then the classloader containing {{SpecificData}}. 
>  
>  A possible solution is to use the {{classCache}} of the default {{SpecificData}} so the sometimes expensive classloader codepath is cached. (PR coming up)
> We noticed this in after trying out a spark upstep from spark 3.1.0 (avro 1.8.2) to 3.2.0 (spark 1.10.2) where 74% of the time was spend in millions of times resolving the same class.
> With this patch this resolving time was brought back from 74% to 0.70%.
> JMC flamegraph showing this issue:
> !image-2021-06-11-14-27-16-689.png!
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)