You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Kunal Khatua (JIRA)" <ji...@apache.org> on 2019/04/01 23:36:00 UTC
[jira] [Commented] (DRILL-7069) Poor performance of
transformBinaryInMetadataCache
[ https://issues.apache.org/jira/browse/DRILL-7069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16807286#comment-16807286 ]
Kunal Khatua commented on DRILL-7069:
-------------------------------------
[~ben-zvi] does this require any additional Documentation beyond a mention in the release notes? (cc: [~bbevens])
> Poor performance of transformBinaryInMetadataCache
> --------------------------------------------------
>
> Key: DRILL-7069
> URL: https://issues.apache.org/jira/browse/DRILL-7069
> Project: Apache Drill
> Issue Type: Improvement
> Components: Metadata
> Affects Versions: 1.15.0
> Reporter: Boaz Ben-Zvi
> Assignee: Boaz Ben-Zvi
> Priority: Major
> Labels: ready-to-commit
> Fix For: 1.16.0
>
>
> The performance of the method *transformBinaryInMetadataCache* scales poorly as the table's numbers of underlying files, row-groups and columns grow. This method is invoked during planning of every query using this table.
> A test on a table using 219 directories (each with 20 files), 1 row-group in each file, and 94 columns, measured about *1340 milliseconds*.
> The main culprit are the version checks, which take place in *every iteration* (i.e., about 400k times in the previous example) and involve construction of 6 MetadataVersion objects (and possibly garbage collections).
> Removing the version checks from the loops improved this method's performance on the above test down to about *250 milliseconds*.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)