You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Alexey Kudinkin (Jira)" <ji...@apache.org> on 2022/04/10 20:15:00 UTC

[jira] [Resolved] (HUDI-3834) Evaluate MT Column Stats Performance

     [ https://issues.apache.org/jira/browse/HUDI-3834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexey Kudinkin resolved HUDI-3834.
-----------------------------------

> Evaluate MT Column Stats Performance 
> -------------------------------------
>
>                 Key: HUDI-3834
>                 URL: https://issues.apache.org/jira/browse/HUDI-3834
>             Project: Apache Hudi
>          Issue Type: Bug
>            Reporter: Alexey Kudinkin
>            Assignee: Alexey Kudinkin
>            Priority: Blocker
>              Labels: pull-request-available
>             Fix For: 0.11.0
>
>         Attachments: Screen Shot 2022-04-07 at 7.46.01 PM.png, Screen Shot 2022-04-07 at 7.48.04 PM.png, Screen Shot 2022-04-08 at 9.12.48 AM.png
>
>
> h3. *UPDATE*
> --------
> *TL;DR* After identifying the bottlenecks as Avro 1.10 regression in generated Builder classes, relying on `SpecificData.getForSchema` call that loads corresponding model's class using reflection, we're able to bring down in a similar setting the runtime of reading for Column Stats Index from MT containing 800k records from *60s* to *3s* (20x)
>  
> Follow the comments for updates on that.
> --------
> Previously, while evaluating Data Skipping runtime in EMR setting, it was measured that reading of Column Stats Index of about ~800k records takes about *60s* (~10ms / record).
>  
> Given that the total size of the log-files involved is ~60Mb, seems like there are some performance bottlenecks that we should investigate before 0.11 release



--
This message was sent by Atlassian Jira
(v8.20.1#820001)