You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Sudhir Rao <ys...@outlook.com> on 2015/04/18 03:13:39 UTC
Drill freezes and requires restart of drillbits
Hi there,
I am running a Hadoop cluster of 8 nodes.
4 Nodes : 23 GB RAM4 Nodes : 60 GB RAM
I have drillbits running on all the datanodes. when i login using sqlline and execute SQL queries.. things get stuck and i see the query in PENDING state for a really long time. It never ends up finishing ...
The only way to get my SQL queries working again is by restarting all my drillbits and things are happy. Then i leave the servers running for few hours and the freezing problem creeps again... What is going on here ?
Please help me resolve this issue.
Drillbit configuration : Direct Memory : 8G, Heap : 4G
Drill Version : 0.8
regards#sudhir
Re: Drill freezes and requires restart of drillbits
Posted by Chris Westin <ch...@gmail.com>.
Can you please open a JIRA with a reproducible test case?
On Fri, Apr 24, 2015 at 8:57 AM, Sudhir Rao <ys...@outlook.com> wrote:
> I don't know what else to check here.. drill continues to freeze and i
> don't see any other log messages
>
> > From: ysudhir@outlook.com
> > To: dev@drill.apache.org
> > Subject: RE: Drill freezes and requires restart of drillbits
> > Date: Sat, 18 Apr 2015 19:49:09 -0700
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > In the Running Queries section, I see the query as PENDING...
> > In the logs i only see a json dump and nothing else... i will look into
> config param that can dump additional debug messages....
> > 2015-04-18 19:45:31,093 [2acce934-b3c4-96b3-c2e1-557f7ef8e274:frag:0:0]
> INFO o.a.drill.exec.work.foreman.Foreman - foreman cleaning up - status:
> [0=>[0=>FragmentData [isLocal=true, status=profile {
> > state: FINISHED
> > minor_fragment_id: 0
> > operator_profile {
> > input_profile {
> > records: 0
> > batches: 0
> > schemas: 0
> > }
> > operator_id: 0
> > operator_type: 26
> > setup_nanos: 0
> > process_nanos: 4516562
> > peak_local_memory_allocated: 7209626
> > wait_nanos: 0
> > }
> > operator_profile {
> > input_profile {
> > records: 1
> > batches: 1
> > schemas: 1
> > } operator_id: 0 operator_type: 13 setup_nanos: 0
> process_nanos: 37325782 peak_local_memory_allocated: 0 metric {
> metric_id: 0 long_value: 49 } wait_nanos: 211427 } start_time:
> 1429411531036 end_time: 1429411531082 memory_used: 0 max_memory_used:
> 2000000 endpoint { address: "titan1.xxx.xxx.com" user_port: 31010
> control_port: 31011 data_port: 31012 }
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > }
> >
> >
> >
> >
> >
> >
> >
> > handle {
> > query_id {
> > part1: 3084096257405523635
> > part2: -4404144954512186764
> > }
> > major_fragment_id: 0
> > minor_fragment_id: 0
> > }
> > , lastStatusUpdate=1429411531082, endpoint=address: "titan1.xxx.xxx.com"
> > user_port: 31010
> > control_port: 31011
> > data_port: 31012
> > ]]]
> > regards#sudhir
> >
> >
> > > Date: Sat, 18 Apr 2015 09:53:52 +0530
> > > Subject: Re: Drill freezes and requires restart of drillbits
> > > From: mufeed.usman@gmail.com
> > > To: dev@drill.apache.org
> > >
> > > Hello Sudhir,
> > >
> > > Around the time of the freeze, did you get a chance to take a look at
> the
> > > drill logs?
> > >
> > >
> > > ---
> > > Mufeed Usman
> > > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> > > Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> > > <http://mufeed.livejournal.com>
> > >
> > >
> > >
> > >
> > > On Sat, Apr 18, 2015 at 6:43 AM, Sudhir Rao <ys...@outlook.com>
> wrote:
> > >
> > > > Hi there,
> > > > I am running a Hadoop cluster of 8 nodes.
> > > > 4 Nodes : 23 GB RAM4 Nodes : 60 GB RAM
> > > > I have drillbits running on all the datanodes. when i login using
> sqlline
> > > > and execute SQL queries.. things get stuck and i see the query in
> PENDING
> > > > state for a really long time. It never ends up finishing ...
> > > > The only way to get my SQL queries working again is by restarting
> all my
> > > > drillbits and things are happy. Then i leave the servers running for
> few
> > > > hours and the freezing problem creeps again... What is going on
> here ?
> > > > Please help me resolve this issue.
> > > > Drillbit configuration : Direct Memory : 8G, Heap : 4G
> > > > Drill Version : 0.8
> > > > regards#sudhir
> >
>
>
RE: Drill freezes and requires restart of drillbits
Posted by Sudhir Rao <ys...@outlook.com>.
I don't know what else to check here.. drill continues to freeze and i don't see any other log messages
> From: ysudhir@outlook.com
> To: dev@drill.apache.org
> Subject: RE: Drill freezes and requires restart of drillbits
> Date: Sat, 18 Apr 2015 19:49:09 -0700
>
>
>
>
>
>
>
>
>
> In the Running Queries section, I see the query as PENDING...
> In the logs i only see a json dump and nothing else... i will look into config param that can dump additional debug messages....
> 2015-04-18 19:45:31,093 [2acce934-b3c4-96b3-c2e1-557f7ef8e274:frag:0:0] INFO o.a.drill.exec.work.foreman.Foreman - foreman cleaning up - status: [0=>[0=>FragmentData [isLocal=true, status=profile {
> state: FINISHED
> minor_fragment_id: 0
> operator_profile {
> input_profile {
> records: 0
> batches: 0
> schemas: 0
> }
> operator_id: 0
> operator_type: 26
> setup_nanos: 0
> process_nanos: 4516562
> peak_local_memory_allocated: 7209626
> wait_nanos: 0
> }
> operator_profile {
> input_profile {
> records: 1
> batches: 1
> schemas: 1
> } operator_id: 0 operator_type: 13 setup_nanos: 0 process_nanos: 37325782 peak_local_memory_allocated: 0 metric { metric_id: 0 long_value: 49 } wait_nanos: 211427 } start_time: 1429411531036 end_time: 1429411531082 memory_used: 0 max_memory_used: 2000000 endpoint { address: "titan1.xxx.xxx.com" user_port: 31010 control_port: 31011 data_port: 31012 }
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> }
>
>
>
>
>
>
>
> handle {
> query_id {
> part1: 3084096257405523635
> part2: -4404144954512186764
> }
> major_fragment_id: 0
> minor_fragment_id: 0
> }
> , lastStatusUpdate=1429411531082, endpoint=address: "titan1.xxx.xxx.com"
> user_port: 31010
> control_port: 31011
> data_port: 31012
> ]]]
> regards#sudhir
>
>
> > Date: Sat, 18 Apr 2015 09:53:52 +0530
> > Subject: Re: Drill freezes and requires restart of drillbits
> > From: mufeed.usman@gmail.com
> > To: dev@drill.apache.org
> >
> > Hello Sudhir,
> >
> > Around the time of the freeze, did you get a chance to take a look at the
> > drill logs?
> >
> >
> > ---
> > Mufeed Usman
> > My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> > Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> > <http://mufeed.livejournal.com>
> >
> >
> >
> >
> > On Sat, Apr 18, 2015 at 6:43 AM, Sudhir Rao <ys...@outlook.com> wrote:
> >
> > > Hi there,
> > > I am running a Hadoop cluster of 8 nodes.
> > > 4 Nodes : 23 GB RAM4 Nodes : 60 GB RAM
> > > I have drillbits running on all the datanodes. when i login using sqlline
> > > and execute SQL queries.. things get stuck and i see the query in PENDING
> > > state for a really long time. It never ends up finishing ...
> > > The only way to get my SQL queries working again is by restarting all my
> > > drillbits and things are happy. Then i leave the servers running for few
> > > hours and the freezing problem creeps again... What is going on here ?
> > > Please help me resolve this issue.
> > > Drillbit configuration : Direct Memory : 8G, Heap : 4G
> > > Drill Version : 0.8
> > > regards#sudhir
>
RE: Drill freezes and requires restart of drillbits
Posted by Sudhir Rao <ys...@outlook.com>.
In the Running Queries section, I see the query as PENDING...
In the logs i only see a json dump and nothing else... i will look into config param that can dump additional debug messages....
2015-04-18 19:45:31,093 [2acce934-b3c4-96b3-c2e1-557f7ef8e274:frag:0:0] INFO o.a.drill.exec.work.foreman.Foreman - foreman cleaning up - status: [0=>[0=>FragmentData [isLocal=true, status=profile {
state: FINISHED
minor_fragment_id: 0
operator_profile {
input_profile {
records: 0
batches: 0
schemas: 0
}
operator_id: 0
operator_type: 26
setup_nanos: 0
process_nanos: 4516562
peak_local_memory_allocated: 7209626
wait_nanos: 0
}
operator_profile {
input_profile {
records: 1
batches: 1
schemas: 1
} operator_id: 0 operator_type: 13 setup_nanos: 0 process_nanos: 37325782 peak_local_memory_allocated: 0 metric { metric_id: 0 long_value: 49 } wait_nanos: 211427 } start_time: 1429411531036 end_time: 1429411531082 memory_used: 0 max_memory_used: 2000000 endpoint { address: "titan1.xxx.xxx.com" user_port: 31010 control_port: 31011 data_port: 31012 }
}
handle {
query_id {
part1: 3084096257405523635
part2: -4404144954512186764
}
major_fragment_id: 0
minor_fragment_id: 0
}
, lastStatusUpdate=1429411531082, endpoint=address: "titan1.xxx.xxx.com"
user_port: 31010
control_port: 31011
data_port: 31012
]]]
regards#sudhir
> Date: Sat, 18 Apr 2015 09:53:52 +0530
> Subject: Re: Drill freezes and requires restart of drillbits
> From: mufeed.usman@gmail.com
> To: dev@drill.apache.org
>
> Hello Sudhir,
>
> Around the time of the freeze, did you get a chance to take a look at the
> drill logs?
>
>
> ---
> Mufeed Usman
> My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
> Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
> <http://mufeed.livejournal.com>
>
>
>
>
> On Sat, Apr 18, 2015 at 6:43 AM, Sudhir Rao <ys...@outlook.com> wrote:
>
> > Hi there,
> > I am running a Hadoop cluster of 8 nodes.
> > 4 Nodes : 23 GB RAM4 Nodes : 60 GB RAM
> > I have drillbits running on all the datanodes. when i login using sqlline
> > and execute SQL queries.. things get stuck and i see the query in PENDING
> > state for a really long time. It never ends up finishing ...
> > The only way to get my SQL queries working again is by restarting all my
> > drillbits and things are happy. Then i leave the servers running for few
> > hours and the freezing problem creeps again... What is going on here ?
> > Please help me resolve this issue.
> > Drillbit configuration : Direct Memory : 8G, Heap : 4G
> > Drill Version : 0.8
> > regards#sudhir
Re: Drill freezes and requires restart of drillbits
Posted by mufy <mu...@gmail.com>.
Hello Sudhir,
Around the time of the freeze, did you get a chance to take a look at the
drill logs?
---
Mufeed Usman
My LinkedIn <http://www.linkedin.com/pub/mufeed-usman/28/254/400> | My
Social Cause <http://www.vision2016.org.in/> | My Blogs : LiveJournal
<http://mufeed.livejournal.com>
On Sat, Apr 18, 2015 at 6:43 AM, Sudhir Rao <ys...@outlook.com> wrote:
> Hi there,
> I am running a Hadoop cluster of 8 nodes.
> 4 Nodes : 23 GB RAM4 Nodes : 60 GB RAM
> I have drillbits running on all the datanodes. when i login using sqlline
> and execute SQL queries.. things get stuck and i see the query in PENDING
> state for a really long time. It never ends up finishing ...
> The only way to get my SQL queries working again is by restarting all my
> drillbits and things are happy. Then i leave the servers running for few
> hours and the freezing problem creeps again... What is going on here ?
> Please help me resolve this issue.
> Drillbit configuration : Direct Memory : 8G, Heap : 4G
> Drill Version : 0.8
> regards#sudhir