pending queries jamming the system

[Stefán Baxter <st...@activitystream.com>]

Hi,

I have a small cluster of 3 drillbits running. It's been working just fine
until it stopped working altogether. I notice a few "pending" queries and
when I try to cancel them, via the admin, they either report that they
don't know where they are running or the cancelling process freezes.

What is a) the easiest way to delete all pending tasks? and b) make sure
that this does not happen?

All the best,
 -Stefan

Re: pending queries jamming the system

[Jason Altekruse <al...@gmail.com>]

We are going to have a lot of users with less perseverance and black box
debugging skills than you have been showing in your evaluation of Drill. I
would not consider this a stupid user issue at all, we need to be clear
about the state of the system to users. If you have some time to record how
you got into this situation we can try to plan some work to make the user
experience better. In the meantime it will serve as nice documentaiton of
the current behavior if someone goes searching in the JIRAs for help with
this issue.

On Mon, Aug 3, 2015 at 9:56 AM, Stefán Baxter <st...@activitystream.com>
wrote:

> Hi Sudheesh,
>
> I have been looking at this and the root cause is that the file system,
> being queries, was not available.
> This was easy to fix but the reaction of Drill could perhaps have been
> better (maybe it was stuck waiting for the filesystem) or I could shorten
> the timeout.
>
> Anyways; I think this classifies as a "stupid user" case but I can mention
> it in Jira if you still think it's relevant.
>
> Thank you,
>  -Stefan
>
> On Mon, Aug 3, 2015 at 4:48 PM, Sudheesh Katkam <sk...@maprtech.com>
> wrote:
>
> > Hi Stefan,
> >
> > Can you create a JIRA for this? Please attach to the JIRA:
> > (1) thread dumps of the three Drillbits (you can get this using jstack),
> > and
> > (2) json query profiles of the PENDING queries (you can get this from the
> > "Full JSON Profile" at the bottom of the profile page).
> >
> > Thank you,
> > Sudheesh
> >
> > > On Aug 3, 2015, at 9:00 AM, Stefán Baxter <st...@activitystream.com>
> > wrote:
> > >
> > > Hi,
> > >
> > > I have a small cluster of 3 drillbits running. It's been working just
> > fine
> > > until it stopped working altogether. I notice a few "pending" queries
> and
> > > when I try to cancel them, via the admin, they either report that they
> > > don't know where they are running or the cancelling process freezes.
> > >
> > > What is a) the easiest way to delete all pending tasks? and b) make
> sure
> > > that this does not happen?
> > >
> > > All the best,
> > > -Stefan
> >
> >
>

Re: pending queries jamming the system

[Stefán Baxter <st...@activitystream.com>]

Hi Sudheesh,

I have been looking at this and the root cause is that the file system,
being queries, was not available.
This was easy to fix but the reaction of Drill could perhaps have been
better (maybe it was stuck waiting for the filesystem) or I could shorten
the timeout.

Anyways; I think this classifies as a "stupid user" case but I can mention
it in Jira if you still think it's relevant.

Thank you,
 -Stefan

On Mon, Aug 3, 2015 at 4:48 PM, Sudheesh Katkam <sk...@maprtech.com>
wrote:

> Hi Stefan,
>
> Can you create a JIRA for this? Please attach to the JIRA:
> (1) thread dumps of the three Drillbits (you can get this using jstack),
> and
> (2) json query profiles of the PENDING queries (you can get this from the
> "Full JSON Profile" at the bottom of the profile page).
>
> Thank you,
> Sudheesh
>
> > On Aug 3, 2015, at 9:00 AM, Stefán Baxter <st...@activitystream.com>
> wrote:
> >
> > Hi,
> >
> > I have a small cluster of 3 drillbits running. It's been working just
> fine
> > until it stopped working altogether. I notice a few "pending" queries and
> > when I try to cancel them, via the admin, they either report that they
> > don't know where they are running or the cancelling process freezes.
> >
> > What is a) the easiest way to delete all pending tasks? and b) make sure
> > that this does not happen?
> >
> > All the best,
> > -Stefan
>
>

Re: pending queries jamming the system

[Sudheesh Katkam <sk...@maprtech.com>]

Hi Stefan,

Can you create a JIRA for this? Please attach to the JIRA:
(1) thread dumps of the three Drillbits (you can get this using jstack), and 
(2) json query profiles of the PENDING queries (you can get this from the "Full JSON Profile" at the bottom of the profile page).

Thank you,
Sudheesh

> On Aug 3, 2015, at 9:00 AM, Stefán Baxter <st...@activitystream.com> wrote:
> 
> Hi,
> 
> I have a small cluster of 3 drillbits running. It's been working just fine
> until it stopped working altogether. I notice a few "pending" queries and
> when I try to cancel them, via the admin, they either report that they
> don't know where they are running or the cancelling process freezes.
> 
> What is a) the easiest way to delete all pending tasks? and b) make sure
> that this does not happen?
> 
> All the best,
> -Stefan