You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Rahul Raj <ra...@option3consulting.com> on 2017/08/29 14:46:15 UTC
Drill Profile page takes too much time to load
Hi,
The drill profile list page(<<ip>>:8047/profiles) takes few minutes to load
in one of the installation.
There was a considerable amount of processor(20%) and memory(15-20%) usage
during this time. Immediately after displaying the results, values return
to normal.
Could this be because of the values accumulated in profile storage? Is it
possible to purge some data?
Regards,
Rahul
--
**** This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom it is
addressed. If you are not the named addressee then you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately and delete this e-mail from your system.****
RE: Drill Profile page takes too much time to load
Posted by Kunal Khatua <kk...@mapr.com>.
This might be because that installation is possibly hosting a lot of Drill profiles. Do you know how many profiles you have residing in the underlying persistent store?
When a query is executed, the Foreman Drillbit (i.e. the node from where the query is submitted) writes the final profile to the file system. I believe, by default, this is the local file system, though in distributed env, it is recommended to use the DFS itself.
https://drill.apache.org/docs/persistent-configuration-storage/
For DFS, you can follow these steps: https://drill.apache.org/docs/persistent-configuration-storage/#how-to-configure-the-zookeeper-pstore
This will ensure that any profile written by DrillbitA is also viewable by DrillbitB (since they both read back the profiles from the DFS).
If this is the setup you have, you'd need to go to the DFS (say HDFS) and delete the old profiles (*.sys.drill files), while retaining only the most recent ones that you want.
Profile management is not prioritized high because of this work around.
If you don't care about the profiles persisting forever, but only for recent queries, there is an alternative which allows you to store them in memory within the Foreman Drillbit of each executed query. https://drill.apache.org/docs/persistent-configuration-storage/#storing-query-profiles-in-memory
You can force a cap on the number of profiles you wish to retain, and it will always keep that list trimmed. However, you can't see profiles on DrillbitA from another Drillbit as these are not shared.
In the meanwhile, if you think this has a lot of merit, it might be worth dropping a comment or at least voting up these related JIRAs:
https://issues.apache.org/jira/browse/DRILL-2362
https://issues.apache.org/jira/browse/DRILL-5270
-----Original Message-----
From: Rahul Raj [mailto:rahul.raj@option3consulting.com]
Sent: Tuesday, August 29, 2017 7:46 AM
To: user@drill.apache.org
Subject: Drill Profile page takes too much time to load
Hi,
The drill profile list page(<<ip>>:8047/profiles) takes few minutes to load in one of the installation.
There was a considerable amount of processor(20%) and memory(15-20%) usage during this time. Immediately after displaying the results, values return to normal.
Could this be because of the values accumulated in profile storage? Is it possible to purge some data?
Regards,
Rahul
--
**** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom it is addressed. If you are not the named addressee then you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and delete this e-mail from your system.****
Re: Drill Profile page takes too much time to load
Posted by Rahul Raj <ra...@option3consulting.com>.
As Kunal and Padma mentioned, there are two tickets and I have added my
comment there.
https://issues.apache.org/jira/browse/DRILL-2861
<https://issues.apache.org/jira/browse/DRILL-2362>
https://issues.apache.org/jira/browse/DRILL-2362
https://issues.apache.org/jira/browse/DRILL-5270
In https://drill.apache.org/docs/persistent-configuration-storage, under
ZooKeeper configuration it is mentioned that 'The ZooKeeper PStore provider
offloads query profile data to the Drill log directory on Drill nodes'. We
need to add a note on the performance impact of accumulating too many files
here and the work around of purging contents manually.
Regards,
Rahul
On Wed, Aug 30, 2017 at 10:17 PM, Paul Rogers <pr...@mapr.com> wrote:
> Is a JIRA filed for this case? Should we add a note to the documentation
> if we can’t fix the issue?
>
> Thanks,
>
> - Paul
>
> > On Aug 29, 2017, at 10:41 PM, Rahul Raj <ra...@option3consulting.com>
> wrote:
> >
> > We had more than 200,000 profiles stored :)
> >
> > Regards,
> > Rahul.
> >
> > On Wed, Aug 30, 2017 at 5:03 AM, Padma Penumarthy <pp...@mapr.com>
> > wrote:
> >
> >> If these channel closed exceptions happen when you try to list profiles
> >> using web UI, yes that could be related.
> >> one option is to change configuration to use a new directory for saving
> >> profiles.
> >> You can delete the old profiles (if you don’t need them) or save them.
> >>
> >> Thanks,
> >> Padma
> >>
> >>
> >>> On Aug 29, 2017, at 4:05 PM, Rahul Raj <rahul.raj@option3consulting.
> com>
> >> wrote:
> >>>
> >>> Thanks for the reply. I will try deleting the files manually.
> >>>
> >>> I have a web application that pools connections to drill. Once this
> >> happens
> >>> I can see channel closed exceptions and the entire pool stale. Could
> this
> >>> be due to a large GC causing ZK heartbeat miss?
> >>>
> >>> Regards,
> >>> Rahul
> >>>
> >>> On Tue, Aug 29, 2017 at 10:42 PM, Padma Penumarthy <
> ppenumarthy@mapr.com
> >>>
> >>> wrote:
> >>>
> >>>> yes, we save each query profile as file in a single directory.
> >>>> If there are large number of files in the directory, it can cause web
> UI
> >>>> to hang or slow down.
> >>>> That is because we try to list all the files in the directory when we
> >> want
> >>>> to view the profiles from web UI and
> >>>> that is an expensive operation.
> >>>> We have some open JIRAs for this issue (DRILL-2861, DRILL-2362)
> >>>> which we plan to address in the future.
> >>>> For now, you have to delete these files manually if it is an issue for
> >> you.
> >>>>
> >>>> Thanks,
> >>>> Padma
> >>>>
> >>>>> On Aug 29, 2017, at 7:46 AM, Rahul Raj <rahul.raj@option3consulting.
> >> com>
> >>>> wrote:
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> The drill profile list page(<<ip>>:8047/profiles) takes few minutes
> to
> >>>> load
> >>>>> in one of the installation.
> >>>>>
> >>>>> There was a considerable amount of processor(20%) and memory(15-20%)
> >>>> usage
> >>>>> during this time. Immediately after displaying the results, values
> >> return
> >>>>> to normal.
> >>>>>
> >>>>> Could this be because of the values accumulated in profile storage?
> Is
> >> it
> >>>>> possible to purge some data?
> >>>>>
> >>>>> Regards,
> >>>>> Rahul
> >>>>>
> >>>>> --
> >>>>> **** This email and any files transmitted with it are confidential
> and
> >>>>> intended solely for the use of the individual or entity to whom it is
> >>>>> addressed. If you are not the named addressee then you should not
> >>>>> disseminate, distribute or copy this e-mail. Please notify the sender
> >>>>> immediately and delete this e-mail from your system.****
> >>>>
> >>>>
> >>>
> >>> --
> >>> **** This email and any files transmitted with it are confidential and
> >>> intended solely for the use of the individual or entity to whom it is
> >>> addressed. If you are not the named addressee then you should not
> >>> disseminate, distribute or copy this e-mail. Please notify the sender
> >>> immediately and delete this e-mail from your system.****
> >>
> >>
> >
> > --
> > **** This email and any files transmitted with it are confidential and
> > intended solely for the use of the individual or entity to whom it is
> > addressed. If you are not the named addressee then you should not
> > disseminate, distribute or copy this e-mail. Please notify the sender
> > immediately and delete this e-mail from your system.****
>
>
--
**** This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom it is
addressed. If you are not the named addressee then you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately and delete this e-mail from your system.****
Re: Drill Profile page takes too much time to load
Posted by Paul Rogers <pr...@mapr.com>.
Is a JIRA filed for this case? Should we add a note to the documentation if we can’t fix the issue?
Thanks,
- Paul
> On Aug 29, 2017, at 10:41 PM, Rahul Raj <ra...@option3consulting.com> wrote:
>
> We had more than 200,000 profiles stored :)
>
> Regards,
> Rahul.
>
> On Wed, Aug 30, 2017 at 5:03 AM, Padma Penumarthy <pp...@mapr.com>
> wrote:
>
>> If these channel closed exceptions happen when you try to list profiles
>> using web UI, yes that could be related.
>> one option is to change configuration to use a new directory for saving
>> profiles.
>> You can delete the old profiles (if you don’t need them) or save them.
>>
>> Thanks,
>> Padma
>>
>>
>>> On Aug 29, 2017, at 4:05 PM, Rahul Raj <ra...@option3consulting.com>
>> wrote:
>>>
>>> Thanks for the reply. I will try deleting the files manually.
>>>
>>> I have a web application that pools connections to drill. Once this
>> happens
>>> I can see channel closed exceptions and the entire pool stale. Could this
>>> be due to a large GC causing ZK heartbeat miss?
>>>
>>> Regards,
>>> Rahul
>>>
>>> On Tue, Aug 29, 2017 at 10:42 PM, Padma Penumarthy <ppenumarthy@mapr.com
>>>
>>> wrote:
>>>
>>>> yes, we save each query profile as file in a single directory.
>>>> If there are large number of files in the directory, it can cause web UI
>>>> to hang or slow down.
>>>> That is because we try to list all the files in the directory when we
>> want
>>>> to view the profiles from web UI and
>>>> that is an expensive operation.
>>>> We have some open JIRAs for this issue (DRILL-2861, DRILL-2362)
>>>> which we plan to address in the future.
>>>> For now, you have to delete these files manually if it is an issue for
>> you.
>>>>
>>>> Thanks,
>>>> Padma
>>>>
>>>>> On Aug 29, 2017, at 7:46 AM, Rahul Raj <rahul.raj@option3consulting.
>> com>
>>>> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> The drill profile list page(<<ip>>:8047/profiles) takes few minutes to
>>>> load
>>>>> in one of the installation.
>>>>>
>>>>> There was a considerable amount of processor(20%) and memory(15-20%)
>>>> usage
>>>>> during this time. Immediately after displaying the results, values
>> return
>>>>> to normal.
>>>>>
>>>>> Could this be because of the values accumulated in profile storage? Is
>> it
>>>>> possible to purge some data?
>>>>>
>>>>> Regards,
>>>>> Rahul
>>>>>
>>>>> --
>>>>> **** This email and any files transmitted with it are confidential and
>>>>> intended solely for the use of the individual or entity to whom it is
>>>>> addressed. If you are not the named addressee then you should not
>>>>> disseminate, distribute or copy this e-mail. Please notify the sender
>>>>> immediately and delete this e-mail from your system.****
>>>>
>>>>
>>>
>>> --
>>> **** This email and any files transmitted with it are confidential and
>>> intended solely for the use of the individual or entity to whom it is
>>> addressed. If you are not the named addressee then you should not
>>> disseminate, distribute or copy this e-mail. Please notify the sender
>>> immediately and delete this e-mail from your system.****
>>
>>
>
> --
> **** This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom it is
> addressed. If you are not the named addressee then you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately and delete this e-mail from your system.****
Re: Drill Profile page takes too much time to load
Posted by Rahul Raj <ra...@option3consulting.com>.
We had more than 200,000 profiles stored :)
Regards,
Rahul.
On Wed, Aug 30, 2017 at 5:03 AM, Padma Penumarthy <pp...@mapr.com>
wrote:
> If these channel closed exceptions happen when you try to list profiles
> using web UI, yes that could be related.
> one option is to change configuration to use a new directory for saving
> profiles.
> You can delete the old profiles (if you don’t need them) or save them.
>
> Thanks,
> Padma
>
>
> > On Aug 29, 2017, at 4:05 PM, Rahul Raj <ra...@option3consulting.com>
> wrote:
> >
> > Thanks for the reply. I will try deleting the files manually.
> >
> > I have a web application that pools connections to drill. Once this
> happens
> > I can see channel closed exceptions and the entire pool stale. Could this
> > be due to a large GC causing ZK heartbeat miss?
> >
> > Regards,
> > Rahul
> >
> > On Tue, Aug 29, 2017 at 10:42 PM, Padma Penumarthy <ppenumarthy@mapr.com
> >
> > wrote:
> >
> >> yes, we save each query profile as file in a single directory.
> >> If there are large number of files in the directory, it can cause web UI
> >> to hang or slow down.
> >> That is because we try to list all the files in the directory when we
> want
> >> to view the profiles from web UI and
> >> that is an expensive operation.
> >> We have some open JIRAs for this issue (DRILL-2861, DRILL-2362)
> >> which we plan to address in the future.
> >> For now, you have to delete these files manually if it is an issue for
> you.
> >>
> >> Thanks,
> >> Padma
> >>
> >>> On Aug 29, 2017, at 7:46 AM, Rahul Raj <rahul.raj@option3consulting.
> com>
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> The drill profile list page(<<ip>>:8047/profiles) takes few minutes to
> >> load
> >>> in one of the installation.
> >>>
> >>> There was a considerable amount of processor(20%) and memory(15-20%)
> >> usage
> >>> during this time. Immediately after displaying the results, values
> return
> >>> to normal.
> >>>
> >>> Could this be because of the values accumulated in profile storage? Is
> it
> >>> possible to purge some data?
> >>>
> >>> Regards,
> >>> Rahul
> >>>
> >>> --
> >>> **** This email and any files transmitted with it are confidential and
> >>> intended solely for the use of the individual or entity to whom it is
> >>> addressed. If you are not the named addressee then you should not
> >>> disseminate, distribute or copy this e-mail. Please notify the sender
> >>> immediately and delete this e-mail from your system.****
> >>
> >>
> >
> > --
> > **** This email and any files transmitted with it are confidential and
> > intended solely for the use of the individual or entity to whom it is
> > addressed. If you are not the named addressee then you should not
> > disseminate, distribute or copy this e-mail. Please notify the sender
> > immediately and delete this e-mail from your system.****
>
>
--
**** This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom it is
addressed. If you are not the named addressee then you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately and delete this e-mail from your system.****
Re: Drill Profile page takes too much time to load
Posted by Padma Penumarthy <pp...@mapr.com>.
If these channel closed exceptions happen when you try to list profiles
using web UI, yes that could be related.
one option is to change configuration to use a new directory for saving profiles.
You can delete the old profiles (if you don’t need them) or save them.
Thanks,
Padma
> On Aug 29, 2017, at 4:05 PM, Rahul Raj <ra...@option3consulting.com> wrote:
>
> Thanks for the reply. I will try deleting the files manually.
>
> I have a web application that pools connections to drill. Once this happens
> I can see channel closed exceptions and the entire pool stale. Could this
> be due to a large GC causing ZK heartbeat miss?
>
> Regards,
> Rahul
>
> On Tue, Aug 29, 2017 at 10:42 PM, Padma Penumarthy <pp...@mapr.com>
> wrote:
>
>> yes, we save each query profile as file in a single directory.
>> If there are large number of files in the directory, it can cause web UI
>> to hang or slow down.
>> That is because we try to list all the files in the directory when we want
>> to view the profiles from web UI and
>> that is an expensive operation.
>> We have some open JIRAs for this issue (DRILL-2861, DRILL-2362)
>> which we plan to address in the future.
>> For now, you have to delete these files manually if it is an issue for you.
>>
>> Thanks,
>> Padma
>>
>>> On Aug 29, 2017, at 7:46 AM, Rahul Raj <ra...@option3consulting.com>
>> wrote:
>>>
>>> Hi,
>>>
>>> The drill profile list page(<<ip>>:8047/profiles) takes few minutes to
>> load
>>> in one of the installation.
>>>
>>> There was a considerable amount of processor(20%) and memory(15-20%)
>> usage
>>> during this time. Immediately after displaying the results, values return
>>> to normal.
>>>
>>> Could this be because of the values accumulated in profile storage? Is it
>>> possible to purge some data?
>>>
>>> Regards,
>>> Rahul
>>>
>>> --
>>> **** This email and any files transmitted with it are confidential and
>>> intended solely for the use of the individual or entity to whom it is
>>> addressed. If you are not the named addressee then you should not
>>> disseminate, distribute or copy this e-mail. Please notify the sender
>>> immediately and delete this e-mail from your system.****
>>
>>
>
> --
> **** This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom it is
> addressed. If you are not the named addressee then you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately and delete this e-mail from your system.****
Re: Drill Profile page takes too much time to load
Posted by Rahul Raj <ra...@option3consulting.com>.
Thanks for the reply. I will try deleting the files manually.
I have a web application that pools connections to drill. Once this happens
I can see channel closed exceptions and the entire pool stale. Could this
be due to a large GC causing ZK heartbeat miss?
Regards,
Rahul
On Tue, Aug 29, 2017 at 10:42 PM, Padma Penumarthy <pp...@mapr.com>
wrote:
> yes, we save each query profile as file in a single directory.
> If there are large number of files in the directory, it can cause web UI
> to hang or slow down.
> That is because we try to list all the files in the directory when we want
> to view the profiles from web UI and
> that is an expensive operation.
> We have some open JIRAs for this issue (DRILL-2861, DRILL-2362)
> which we plan to address in the future.
> For now, you have to delete these files manually if it is an issue for you.
>
> Thanks,
> Padma
>
> > On Aug 29, 2017, at 7:46 AM, Rahul Raj <ra...@option3consulting.com>
> wrote:
> >
> > Hi,
> >
> > The drill profile list page(<<ip>>:8047/profiles) takes few minutes to
> load
> > in one of the installation.
> >
> > There was a considerable amount of processor(20%) and memory(15-20%)
> usage
> > during this time. Immediately after displaying the results, values return
> > to normal.
> >
> > Could this be because of the values accumulated in profile storage? Is it
> > possible to purge some data?
> >
> > Regards,
> > Rahul
> >
> > --
> > **** This email and any files transmitted with it are confidential and
> > intended solely for the use of the individual or entity to whom it is
> > addressed. If you are not the named addressee then you should not
> > disseminate, distribute or copy this e-mail. Please notify the sender
> > immediately and delete this e-mail from your system.****
>
>
--
**** This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom it is
addressed. If you are not the named addressee then you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately and delete this e-mail from your system.****
Re: Drill Profile page takes too much time to load
Posted by Padma Penumarthy <pp...@mapr.com>.
yes, we save each query profile as file in a single directory.
If there are large number of files in the directory, it can cause web UI to hang or slow down.
That is because we try to list all the files in the directory when we want to view the profiles from web UI and
that is an expensive operation.
We have some open JIRAs for this issue (DRILL-2861, DRILL-2362)
which we plan to address in the future.
For now, you have to delete these files manually if it is an issue for you.
Thanks,
Padma
> On Aug 29, 2017, at 7:46 AM, Rahul Raj <ra...@option3consulting.com> wrote:
>
> Hi,
>
> The drill profile list page(<<ip>>:8047/profiles) takes few minutes to load
> in one of the installation.
>
> There was a considerable amount of processor(20%) and memory(15-20%) usage
> during this time. Immediately after displaying the results, values return
> to normal.
>
> Could this be because of the values accumulated in profile storage? Is it
> possible to purge some data?
>
> Regards,
> Rahul
>
> --
> **** This email and any files transmitted with it are confidential and
> intended solely for the use of the individual or entity to whom it is
> addressed. If you are not the named addressee then you should not
> disseminate, distribute or copy this e-mail. Please notify the sender
> immediately and delete this e-mail from your system.****