You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/03/29 03:15:42 UTC
[jira] [Commented] (DRILL-5394) Optimize query planning for MapR-DB
tables by caching row counts
[ https://issues.apache.org/jira/browse/DRILL-5394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946459#comment-15946459 ]
ASF GitHub Bot commented on DRILL-5394:
---------------------------------------
GitHub user ppadma opened a pull request:
https://github.com/apache/drill/pull/802
DRILL-5394: Optimize query planning for MapR-DB tables by caching row…
… counts
During the planning phase, we fetch MapR DB table row count multiple times. This is an expensive operation and fetching it multiple times increases planning time considerably. Implemented per query cache. Fetch table rowCount only once, cache it in hbaseScanSpec and reuse it. Copy the rowCount from old to new hbaseScanSpec when a new groupScan is created i.e. when the filter is pushed down into scan.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ppadma/drill DRILL-5394
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/802.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #802
----
commit 8ff826c58f074ba58d1ee1b888b5da41eeb2d894
Author: Padma Penumarthy <pp...@yahoo.com>
Date: 2017-03-16T18:40:13Z
DRILL-5394: Optimize query planning for MapR-DB tables by caching row counts
----
> Optimize query planning for MapR-DB tables by caching row counts
> ----------------------------------------------------------------
>
> Key: DRILL-5394
> URL: https://issues.apache.org/jira/browse/DRILL-5394
> Project: Apache Drill
> Issue Type: Improvement
> Components: Query Planning & Optimization, Storage - MapRDB
> Affects Versions: 1.9.0, 1.10.0
> Reporter: Abhishek Girish
> Assignee: Padma Penumarthy
> Labels: MapR-DB-Binary
> Fix For: 1.11.0
>
>
> On large MapR-DB tables, it was observed that the query planning time was longer than expected. With DEBUG logs, it was understood that there were multiple calls being made to get MapR-DB region locations and to fetch total row count for tables.
> {code}
> 2017-02-23 13:59:55,246 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:05,006 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.planner.logical.DrillOptiq - Function
> ...
> 2017-02-23 14:00:05,031 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:16,438 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.planner.logical.DrillOptiq - Special
> ...
> 2017-02-23 14:00:16,439 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:28,479 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.planner.logical.DrillOptiq - Special
> ...
> 2017-02-23 14:00:28,480 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:42,396 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.planner.logical.DrillOptiq - Special
> ...
> 2017-02-23 14:00:42,397 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.s.m.d.b.BinaryTableGroupScan - Getting region locations
> 2017-02-23 14:00:54,609 [27513143-8718-7a47-a2d4-06850755568a:foreman] DEBUG o.a.d.e.p.s.h.DefaultSqlHandler - VOLCANO:Physical Planning (49588ms):
> {code}
> We should cache these stats and reuse them where all required during query planning. This should help reduce query planning time.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)