You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Kunal Khatua (JIRA)" <ji...@apache.org> on 2017/02/22 00:52:44 UTC
[jira] [Commented] (DRILL-5270) Improve loading of profiles listing
in the WebUI
[ https://issues.apache.org/jira/browse/DRILL-5270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877119#comment-15877119 ]
Kunal Khatua commented on DRILL-5270:
-------------------------------------
As a sample experiment, we ran a Drillbit to display the profile list for a directory containing 280K files.
We found that while the Drillbit took a long time to startup (DRILL-4990 will fix this), the load time improves as long as no new profiles are detected.
Using a PathFilter (HadoopAPI) implementation to find new profiles should have helped. However, it appears that the filter isn't pushed down to the file system, so we're not able to benefit from it, unless the HDFS API itself is improved. There is no regression, however, with this.
Using in conjunction with the patch for DRILL-5259 does show improved load times, since we don't go back to the DFS to re-read the profile list.
This is the content of the Drill log
{code}
2017-02-17 11:09:30,929 pssc-65.qa.lab [main] INFO o.apache.drill.exec.server.Drillbit - Startup completed (625436 ms).
2017-02-17 11:14:35,886 pssc-65.qa.lab [qtp2142187763-113] WARN o.a.d.e.s.s.s.LocalPersistentStore - Took 5876 ms to list+map from 281001 profiles
2017-02-17 11:14:35,893 pssc-65.qa.lab [qtp2142187763-113] DEBUG o.a.d.e.s.r.profile.ProfileResources - Time to load MRU 100 profiles: 5885 ms
2017-02-17 11:18:25,940 pssc-65.qa.lab [qtp2142187763-118] DEBUG o.a.d.e.s.r.profile.ProfileResources - Time to load MRU 100 profiles: 1 ms
2017-02-17 11:19:14,977 pssc-65.qa.lab [qtp2142187763-122] DEBUG o.a.d.e.s.r.profile.ProfileResources - Time to load MRU 75 profiles: 1 ms
2017-02-17 11:19:27,554 pssc-65.qa.lab [qtp2142187763-123] DEBUG o.a.d.e.s.r.profile.ProfileResources - Time to load MRU 150 profiles: 1 ms
2017-02-17 11:21:58,137 pssc-65.qa.lab [qtp2142187763-124] INFO o.a.drill.exec.client.DrillClient - Successfully connected to server pssc-65.qa.lab:31010
2017-02-17 11:21:58,409 pssc-65.qa.lab [2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query id 2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a: select * from sys.drillbits
2017-02-17 11:32:02,320 pssc-65.qa.lab [2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a:0:0: State change requested AWAITING_ALLOCATION --> RUNNING
2017-02-17 11:32:02,321 pssc-65.qa.lab [2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a:0:0: State to report: RUNNING
2017-02-17 11:32:02,506 pssc-65.qa.lab [2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a:frag:0:0] INFO o.a.d.e.c.ClassCompilerSelector - Java compiler policy: DEFAULT, Debug option: true
2017-02-17 11:32:02,792 pssc-65.qa.lab [2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a:frag:0:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a:0:0: State change requested RUNNING --> FINISHED
2017-02-17 11:32:02,793 pssc-65.qa.lab [2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a:frag:0:0] INFO o.a.d.e.w.f.FragmentStatusReporter - 2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a:0:0: State to report: FINISHED
2017-02-17 11:32:02,837 pssc-65.qa.lab [CONTROL-rpc-event-queue] INFO query.logger - {"queryId":"2758b2a9-6e44-9d79-df64-e6e3e6a9eb4a","schema":"","queryText":"select * from sys.drillbits","start":1487359318324,"finish":1487359922805,"outcome":"COMPLETED","username":"anonymous","remoteAddress":"10.10.103.65:56448"}
2017-02-17 11:34:49,443 pssc-65.qa.lab [qtp2142187763-129] WARN o.a.d.e.s.s.s.LocalPersistentStore - Took 5545 ms to list+map from 1 profiles
2017-02-17 11:34:49,448 pssc-65.qa.lab [qtp2142187763-129] DEBUG o.a.d.e.s.r.profile.ProfileResources - Time to load MRU 50 profiles: 5553 ms
2017-02-17 11:36:58,770 pssc-65.qa.lab [qtp2142187763-120] DEBUG o.a.d.e.s.r.profile.ProfileResources - Time to load MRU 60 profiles: 1 ms
{code}
> Improve loading of profiles listing in the WebUI
> ------------------------------------------------
>
> Key: DRILL-5270
> URL: https://issues.apache.org/jira/browse/DRILL-5270
> Project: Apache Drill
> Issue Type: Improvement
> Components: Web Server
> Affects Versions: 1.9.0
> Reporter: Kunal Khatua
> Assignee: Kunal Khatua
> Fix For: 1.10.0
>
>
> Currently, as the number of profiles increase, we reload the same list of profiles from the FS.
> An ideal improvement would be to detect if there are any new profiles and only reload from the disk then. Otherwise, a cached list is sufficient.
> For a directory of 280K profiles, the load time is close to 6 seconds on a 32 core server. With the caching, we can get it down to as much as a few milliseconds.
> To render the cache as invalid, we inspect the last modified time of the directory to confirm whether a reload is needed.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)