You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Tridib Samanta <tr...@live.com> on 2014/10/29 17:45:04 UTC
Apache Drill stuck for long time
Hello Experts,
I have recently started exploring drill. I was able to setup a drill cluster of two drillbits and query flat json structure of 50 records. When I started same query on 10 million json record, it never returned even after 30 mins.
What is the way to debug this scenario?
Also is there a way to check how much memory it's using and increase it?
Thanks & Regards
Tridib
RE: Apache Drill stuck for long time
Posted by Tridib Samanta <tr...@live.com>.
Hi Neeraja,
I tested this again. It worked with 1 drillbits as well as 4 drillbits.
Thanks
Tridib
> Date: Wed, 29 Oct 2014 17:10:36 -0700
> Subject: Re: Apache Drill stuck for long time
> From: nrentachintala@maprtech.com
> To: drill-user@incubator.apache.org
>
> Tridib
> Just going through this thread again. You mentioned Drill is setup in a
> private cloud.
> I am suspecting if it is due to UDP/multicast not enabled in your
> environment because hangs are typically the symptom there.
>
> Can you try this on a single node to see if it works.
> Work is in progress in the community to remove the UDP/Multicast dependency
> and it will be fixed in next release.
>
> -thanks
> Neeraja
>
>
>
>
> On Wed, Oct 29, 2014 at 4:53 PM, Ted Dunning <te...@gmail.com> wrote:
>
> > On Wed, Oct 29, 2014 at 1:57 PM, Tridib Samanta <tr...@live.com>
> > wrote:
> >
> > > Well, this seems to be not working. I ran the same query with increased
> > > memory. It got stuck again. I am getting very disappointed. :(. Is there
> > > anything I doing wrong?
> > >
> >
> > The only thing you are doing wrong is to get disappointed!
> >
> > The version you are using is very early and much progress has been made
> > since then. Drill is a new project and is moving and maturing quickly. We
> > have a strong community who will help you.
> >
Re: Apache Drill stuck for long time
Posted by Neeraja Rentachintala <nr...@maprtech.com>.
Tridib
Just going through this thread again. You mentioned Drill is setup in a
private cloud.
I am suspecting if it is due to UDP/multicast not enabled in your
environment because hangs are typically the symptom there.
Can you try this on a single node to see if it works.
Work is in progress in the community to remove the UDP/Multicast dependency
and it will be fixed in next release.
-thanks
Neeraja
On Wed, Oct 29, 2014 at 4:53 PM, Ted Dunning <te...@gmail.com> wrote:
> On Wed, Oct 29, 2014 at 1:57 PM, Tridib Samanta <tr...@live.com>
> wrote:
>
> > Well, this seems to be not working. I ran the same query with increased
> > memory. It got stuck again. I am getting very disappointed. :(. Is there
> > anything I doing wrong?
> >
>
> The only thing you are doing wrong is to get disappointed!
>
> The version you are using is very early and much progress has been made
> since then. Drill is a new project and is moving and maturing quickly. We
> have a strong community who will help you.
>
RE: Apache Drill stuck for long time
Posted by Tridib Samanta <tr...@live.com>.
I will be happy to see I am wrong here :) and want to see it as one of the top Apache Project.
BTW, I can only see version 0.5 while downloading. Which version should I take?
> From: ted.dunning@gmail.com
> Date: Wed, 29 Oct 2014 16:53:52 -0700
> Subject: Re: Apache Drill stuck for long time
> To: drill-user@incubator.apache.org
>
> On Wed, Oct 29, 2014 at 1:57 PM, Tridib Samanta <tr...@live.com>
> wrote:
>
> > Well, this seems to be not working. I ran the same query with increased
> > memory. It got stuck again. I am getting very disappointed. :(. Is there
> > anything I doing wrong?
> >
>
> The only thing you are doing wrong is to get disappointed!
>
> The version you are using is very early and much progress has been made
> since then. Drill is a new project and is moving and maturing quickly. We
> have a strong community who will help you.
Re: Apache Drill stuck for long time
Posted by Ted Dunning <te...@gmail.com>.
On Wed, Oct 29, 2014 at 1:57 PM, Tridib Samanta <tr...@live.com>
wrote:
> Well, this seems to be not working. I ran the same query with increased
> memory. It got stuck again. I am getting very disappointed. :(. Is there
> anything I doing wrong?
>
The only thing you are doing wrong is to get disappointed!
The version you are using is very early and much progress has been made
since then. Drill is a new project and is moving and maturing quickly. We
have a strong community who will help you.
RE: Apache Drill stuck for long time
Posted by Tridib Samanta <tr...@live.com>.
Well, this seems to be not working. I ran the same query with increased memory. It got stuck again. I am getting very disappointed. :(. Is there anything I doing wrong?
Thanks
Tridib
> From: tridib.samanta@live.com
> To: drill-user@incubator.apache.org
> Subject: RE: Apache Drill stuck for long time
> Date: Wed, 29 Oct 2014 10:50:28 -0700
>
> Hi Mufeed,
> I ran select count(*) from source;
>
> It came back with the count after 20 mins. There were 12 million json doc. I definitely got to improve the performance. But not sure where to look at. I am looking at the video https://www.youtube.com/watch?v=kG6vzsk8T7E&list=UUKmI-fBjvKexRlBBeVTwrrg . Do you have any other option documentation to speed up the query?
>
> Thanks
> Tridib
>
> From: tridib.samanta@live.com
> To: drill-user@incubator.apache.org
> Subject: RE: Apache Drill stuck for long time
> Date: Wed, 29 Oct 2014 10:24:11 -0700
>
>
>
>
> Hi Mufeed,
> I am using apache-drill-0.5.0-incubating. It's setup in our private cloud. Drillbit's log shows
>
> 2014-10-29 13:18:56,784 [f997d3d6-f163-4a64-a157-301a153f52f5:frag:0:0] DEBUG o.a.d.exec.compile.ClassTransformer - Done compiling (bytecode size=3.9 KiB, time:51 millis).
>
> After this I do not see anything.
>
> Thanks
> Tridib
>
> > Date: Wed, 29 Oct 2014 22:36:07 +0530
> > Subject: Re: Apache Drill stuck for long time
> > From: mufeed.usman@gmail.com
> > To: drill-user@incubator.apache.org
> >
> > Hello Tribid,
> >
> > What version of Drill are you using? Where is the cluster setup - physical
> > servers, cloud, etc? Do you see anything in the drillbit.log?
> >
> >
> > ---
> > Mufeed Usman
> > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> > Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> > <http://mufeed.livejournal.com>
> >
> >
> >
> >
> > On Wed, Oct 29, 2014 at 10:15 PM, Tridib Samanta <tr...@live.com>
> > wrote:
> >
> > >
> > >
> > >
> > > Hello Experts,
> > > I have recently started exploring drill. I was able to setup a drill
> > > cluster of two drillbits and query flat json structure of 50 records. When
> > > I started same query on 10 million json record, it never returned even
> > > after 30 mins.
> > >
> > > What is the way to debug this scenario?
> > > Also is there a way to check how much memory it's using and increase it?
> > >
> > >
> > > Thanks & Regards
> > > Tridib
> > >
> > >
>
RE: Apache Drill stuck for long time
Posted by Tridib Samanta <tr...@live.com>.
Hi Mufeed,
I ran select count(*) from source;
It came back with the count after 20 mins. There were 12 million json doc. I definitely got to improve the performance. But not sure where to look at. I am looking at the video https://www.youtube.com/watch?v=kG6vzsk8T7E&list=UUKmI-fBjvKexRlBBeVTwrrg . Do you have any other option documentation to speed up the query?
Thanks
Tridib
From: tridib.samanta@live.com
To: drill-user@incubator.apache.org
Subject: RE: Apache Drill stuck for long time
Date: Wed, 29 Oct 2014 10:24:11 -0700
Hi Mufeed,
I am using apache-drill-0.5.0-incubating. It's setup in our private cloud. Drillbit's log shows
2014-10-29 13:18:56,784 [f997d3d6-f163-4a64-a157-301a153f52f5:frag:0:0] DEBUG o.a.d.exec.compile.ClassTransformer - Done compiling (bytecode size=3.9 KiB, time:51 millis).
After this I do not see anything.
Thanks
Tridib
> Date: Wed, 29 Oct 2014 22:36:07 +0530
> Subject: Re: Apache Drill stuck for long time
> From: mufeed.usman@gmail.com
> To: drill-user@incubator.apache.org
>
> Hello Tribid,
>
> What version of Drill are you using? Where is the cluster setup - physical
> servers, cloud, etc? Do you see anything in the drillbit.log?
>
>
> ---
> Mufeed Usman
> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> <http://mufeed.livejournal.com>
>
>
>
>
> On Wed, Oct 29, 2014 at 10:15 PM, Tridib Samanta <tr...@live.com>
> wrote:
>
> >
> >
> >
> > Hello Experts,
> > I have recently started exploring drill. I was able to setup a drill
> > cluster of two drillbits and query flat json structure of 50 records. When
> > I started same query on 10 million json record, it never returned even
> > after 30 mins.
> >
> > What is the way to debug this scenario?
> > Also is there a way to check how much memory it's using and increase it?
> >
> >
> > Thanks & Regards
> > Tridib
> >
> >
RE: Apache Drill stuck for long time
Posted by Tridib Samanta <tr...@live.com>.
Hi Mufeed,
I am using apache-drill-0.5.0-incubating. It's setup in our private cloud. Drillbit's log shows
2014-10-29 13:18:56,784 [f997d3d6-f163-4a64-a157-301a153f52f5:frag:0:0] DEBUG o.a.d.exec.compile.ClassTransformer - Done compiling (bytecode size=3.9 KiB, time:51 millis).
After this I do not see anything.
Thanks
Tridib
> Date: Wed, 29 Oct 2014 22:36:07 +0530
> Subject: Re: Apache Drill stuck for long time
> From: mufeed.usman@gmail.com
> To: drill-user@incubator.apache.org
>
> Hello Tribid,
>
> What version of Drill are you using? Where is the cluster setup - physical
> servers, cloud, etc? Do you see anything in the drillbit.log?
>
>
> ---
> Mufeed Usman
> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> <http://mufeed.livejournal.com>
>
>
>
>
> On Wed, Oct 29, 2014 at 10:15 PM, Tridib Samanta <tr...@live.com>
> wrote:
>
> >
> >
> >
> > Hello Experts,
> > I have recently started exploring drill. I was able to setup a drill
> > cluster of two drillbits and query flat json structure of 50 records. When
> > I started same query on 10 million json record, it never returned even
> > after 30 mins.
> >
> > What is the way to debug this scenario?
> > Also is there a way to check how much memory it's using and increase it?
> >
> >
> > Thanks & Regards
> > Tridib
> >
> >
Re: Apache Drill stuck for long time
Posted by mufy <mu...@gmail.com>.
Hello Tribid,
What version of Drill are you using? Where is the cluster setup - physical
servers, cloud, etc? Do you see anything in the drillbit.log?
---
Mufeed Usman
My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
<http://mufeed.livejournal.com>
On Wed, Oct 29, 2014 at 10:15 PM, Tridib Samanta <tr...@live.com>
wrote:
>
>
>
> Hello Experts,
> I have recently started exploring drill. I was able to setup a drill
> cluster of two drillbits and query flat json structure of 50 records. When
> I started same query on 10 million json record, it never returned even
> after 30 mins.
>
> What is the way to debug this scenario?
> Also is there a way to check how much memory it's using and increase it?
>
>
> Thanks & Regards
> Tridib
>
>