You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@drill.apache.org by Oscar Morante <sp...@gmail.com> on 2016/08/09 10:30:43 UTC

Query hangs on planning

Hi there,
I've been stuck with this for a while and I'm not sure if I'm running 
into a bug or I'm just doing something very wrong.

I have this stripped-down version of my query: 
https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c805b

The data is just a single file with one record (1.5K).

Without changing anything, explain takes ~1sec on my machine.  But if I 
uncomment the where clause then it runs for a couple of hours until it 
runs out of memory.

Also if I uncomment the where clause *and* take out the join, then it 
takes around 30s to plan.

Any ideas?
Thanks!


Re: Query hangs on planning

Posted by Oscar Morante <sp...@gmail.com>.
Hi Rahul,
I'm still very busy :(  But I haven't forgotten about this.  I'll open a 
JIRA with a proper test-case as soon as I get the chance.


On Thu, Sep 01, 2016 at 12:03:43PM -0700, Zelaine Fong wrote:
>Ah ... yes, you're right.  I forgot that was off heap.
>
>-- Zelaine
>
>On Thu, Sep 1, 2016 at 11:41 AM, Sudheesh Katkam <sk...@maprtech.com>
>wrote:
>
>> That setting is for off-heap memory. The earlier case hit heap memory
>> limit.
>>
>> > On Sep 1, 2016, at 11:36 AM, Zelaine Fong <zf...@maprtech.com> wrote:
>> >
>> > One other thing ... have you tried tuning the planner.memory_limit
>> > parameter?  Based on the earlier stack trace, you're hitting a memory
>> limit
>> > during query planning.  So, tuning this parameter should help that.  The
>> > default is 256 MB.
>> >
>> > -- Zelaine
>> >
>> > On Thu, Sep 1, 2016 at 11:21 AM, rahul challapalli <
>> > challapallirahul@gmail.com> wrote:
>> >
>> >> While planning we use heap memory. 2GB of heap should be sufficient for
>> >> what you mentioned. This looks like a bug to me. Can you raise a jira
>> for
>> >> the same? And it would be super helpful if you can also attach the data
>> set
>> >> used.
>> >>
>> >> Rahul
>> >>
>> >> On Wed, Aug 31, 2016 at 9:14 AM, Oscar Morante <sp...@gmail.com>
>> >> wrote:
>> >>
>> >>> Sure,
>> >>> This is what I remember:
>> >>>
>> >>> * Failure
>> >>>   - embedded mode on my laptop
>> >>>   - drill memory: 2Gb/4Gb (heap/direct)
>> >>>   - cpu: 4cores (+hyperthreading)
>> >>>   - `planner.width.max_per_node=6`
>> >>>
>> >>> * Success
>> >>>   - AWS Cluster 2x c3.8xlarge
>> >>>   - drill memory: 16Gb/32Gb
>> >>>   - cpu: limited by kubernetes to 24cores
>> >>>   - `planner.width.max_per_node=23`
>> >>>
>> >>> I'm very busy right now to test again, but I'll try to provide better
>> >> info
>> >>> as soon as I can.
>> >>>
>> >>>
>> >>>
>> >>> On Wed, Aug 31, 2016 at 05:38:53PM +0530, Khurram Faraaz wrote:
>> >>>
>> >>>> Can you please share the number of cores on the setup where the query
>> >> hung
>> >>>> as compared to the number of cores on the setup where the query went
>> >>>> through successfully.
>> >>>> And details of memory from the two scenarios.
>> >>>>
>> >>>> Thanks,
>> >>>> Khurram
>> >>>>
>> >>>> On Wed, Aug 31, 2016 at 4:50 PM, Oscar Morante <sp...@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>> For the record, I think this was just bad memory configuration after
>> >> all.
>> >>>>> I retested on bigger machines and everything seems to be working
>> fine.
>> >>>>>
>> >>>>>
>> >>>>> On Tue, Aug 09, 2016 at 10:46:33PM +0530, Khurram Faraaz wrote:
>> >>>>>
>> >>>>> Oscar, can you please report a JIRA with the required steps to
>> >> reproduce
>> >>>>>> the OOM error. That way someone from the Drill team will take a look
>> >> and
>> >>>>>> investigate.
>> >>>>>>
>> >>>>>> For others interested here is the stack trace.
>> >>>>>>
>> >>>>>> 2016-08-09 16:51:14,280 [285642de-ab37-de6e-a54c-
>> >> 378aaa4ce50e:foreman]
>> >>>>>> ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure
>> >>>>>> Occurred,
>> >>>>>> exiting. Information message: Unable to handle out of memory
>> condition
>> >>>>>> in
>> >>>>>> Foreman.
>> >>>>>> java.lang.OutOfMemoryError: Java heap space
>> >>>>>>       at java.util.Arrays.copyOfRange(Arrays.java:2694)
>> >>>>>> ~[na:1.7.0_111]
>> >>>>>>       at java.lang.String.<init>(String.java:203) ~[na:1.7.0_111]
>> >>>>>>       at java.lang.StringBuilder.toString(StringBuilder.java:405)
>> >>>>>> ~[na:1.7.0_111]
>> >>>>>>       at org.apache.calcite.util.Util.newInternal(Util.java:785)
>> >>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>> >>>>>>       at
>> >>>>>> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(
>> >>>>>> VolcanoRuleCall.java:251)
>> >>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>> >>>>>>       at
>> >>>>>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(
>> >>>>>> VolcanoPlanner.java:808)
>> >>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>> >>>>>>       at
>> >>>>>> org.apache.calcite.tools.Programs$RuleSetProgram.run(
>> >> Programs.java:303)
>> >>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>> >>>>>>       at
>> >>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>> >>>>>> .transform(DefaultSqlHandler.java:404)
>> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>> >>>>>>       at
>> >>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>> >>>>>> .transform(DefaultSqlHandler.java:343)
>> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>> >>>>>>       at
>> >>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>> >>>>>> .convertToDrel(DefaultSqlHandler.java:240)
>> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>> >>>>>>       at
>> >>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>> >>>>>> .convertToDrel(DefaultSqlHandler.java:290)
>> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>> >>>>>>       at
>> >>>>>> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.ge
>> >>>>>> tPlan(ExplainHandler.java:61)
>> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>> >>>>>>       at
>> >>>>>> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(Dri
>> >>>>>> llSqlWorker.java:94)
>> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>> >>>>>>       at
>> >>>>>> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978)
>> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>> >>>>>>       at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.
>> >> java:
>> >>>>>> 257)
>> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>> >>>>>>       at
>> >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> >>>>>> Executor.java:1145)
>> >>>>>> [na:1.7.0_111]
>> >>>>>>       at
>> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> >>>>>> lExecutor.java:615)
>> >>>>>> [na:1.7.0_111]
>> >>>>>>       at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
>> >>>>>>
>> >>>>>> Thanks,
>> >>>>>> Khurram
>> >>>>>>
>> >>>>>> On Tue, Aug 9, 2016 at 7:46 PM, Oscar Morante <sp...@gmail.com>
>> >>>>>> wrote:
>> >>>>>>
>> >>>>>> Yeah, when I uncomment only the `upload_date` lines (a dir0 alias),
>> >>>>>>
>> >>>>>>> explain succeeds within ~30s.  Enabling any of the other lines
>> >> triggers
>> >>>>>>> the
>> >>>>>>> failure.
>> >>>>>>>
>> >>>>>>> This is a log with the `upload_date` lines and `usage <> 'Test'`
>> >>>>>>> enabled:
>> >>>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e
>> >>>>>>>
>> >>>>>>> The client times out around here (~1.5hours):
>> >>>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>> >>>>>>> b3c55e#file-drillbit-log-L178
>> >>>>>>>
>> >>>>>>> And it still keeps running for a while until it dies (~2.5hours):
>> >>>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>> >>>>>>> b3c55e#file-drillbit-log-L178
>> >>>>>>>
>> >>>>>>> The memory settings for this test were:
>> >>>>>>>
>> >>>>>>>   DRILL_HEAP="4G"
>> >>>>>>>   DRILL_MAX_DIRECT_MEMORY="8G"
>> >>>>>>>
>> >>>>>>> This is on a laptop with 16G and I should probably lower it, but it
>> >>>>>>> seems
>> >>>>>>> a bit excessive for such a small query.  And I think I got the same
>> >>>>>>> results
>> >>>>>>> on a 2 node cluster with 8/16.  I'm gonna try again on the cluster
>> to
>> >>>>>>> make
>> >>>>>>> sure.
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> Oscar
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> On Tue, Aug 09, 2016 at 04:13:17PM +0530, Khurram Faraaz wrote:
>> >>>>>>>
>> >>>>>>> You mentioned "*But if I uncomment the where clause then it runs
>> for
>> >> a
>> >>>>>>>
>> >>>>>>>> couple of hours until it runs out of memory.*"
>> >>>>>>>>
>> >>>>>>>> Can you please share the OutOfMemory details from drillbit.log and
>> >> the
>> >>>>>>>> value of DRILL_MAX_DIRECT_MEMORY
>> >>>>>>>>
>> >>>>>>>> Can you also try to see what happens if you retain just this line
>> >>>>>>>> where
>> >>>>>>>> upload_date = '2016-08-01' in your where clause, can you check if
>> >> the
>> >>>>>>>> explain succeeds.
>> >>>>>>>>
>> >>>>>>>> Thanks,
>> >>>>>>>> Khurram
>> >>>>>>>>
>> >>>>>>>> On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <
>> spacepluk@gmail.com>
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>> Hi there,
>> >>>>>>>>
>> >>>>>>>> I've been stuck with this for a while and I'm not sure if I'm
>> >> running
>> >>>>>>>>> into
>> >>>>>>>>> a bug or I'm just doing something very wrong.
>> >>>>>>>>>
>> >>>>>>>>> I have this stripped-down version of my query:
>> >>>>>>>>> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c80
>> 5b
>> >>>>>>>>>
>> >>>>>>>>> The data is just a single file with one record (1.5K).
>> >>>>>>>>>
>> >>>>>>>>> Without changing anything, explain takes ~1sec on my machine.
>> But
>> >>>>>>>>> if I
>> >>>>>>>>> uncomment the where clause then it runs for a couple of hours
>> until
>> >>>>>>>>> it
>> >>>>>>>>> runs
>> >>>>>>>>> out of memory.
>> >>>>>>>>>
>> >>>>>>>>> Also if I uncomment the where clause *and* take out the join,
>> then
>> >> it
>> >>>>>>>>> takes around 30s to plan.
>> >>>>>>>>>
>> >>>>>>>>> Any ideas?
>> >>>>>>>>> Thanks!
>> >>>>>>>>>
>> >>>>>>>>>
>> >>
>>
>>

-- 
Oscar Morante
"Self-education is, I firmly believe, the only kind of education there is."
                                                          -- Isaac Asimov.

Re: Query hangs on planning

Posted by Zelaine Fong <zf...@maprtech.com>.
Ah ... yes, you're right.  I forgot that was off heap.

-- Zelaine

On Thu, Sep 1, 2016 at 11:41 AM, Sudheesh Katkam <sk...@maprtech.com>
wrote:

> That setting is for off-heap memory. The earlier case hit heap memory
> limit.
>
> > On Sep 1, 2016, at 11:36 AM, Zelaine Fong <zf...@maprtech.com> wrote:
> >
> > One other thing ... have you tried tuning the planner.memory_limit
> > parameter?  Based on the earlier stack trace, you're hitting a memory
> limit
> > during query planning.  So, tuning this parameter should help that.  The
> > default is 256 MB.
> >
> > -- Zelaine
> >
> > On Thu, Sep 1, 2016 at 11:21 AM, rahul challapalli <
> > challapallirahul@gmail.com> wrote:
> >
> >> While planning we use heap memory. 2GB of heap should be sufficient for
> >> what you mentioned. This looks like a bug to me. Can you raise a jira
> for
> >> the same? And it would be super helpful if you can also attach the data
> set
> >> used.
> >>
> >> Rahul
> >>
> >> On Wed, Aug 31, 2016 at 9:14 AM, Oscar Morante <sp...@gmail.com>
> >> wrote:
> >>
> >>> Sure,
> >>> This is what I remember:
> >>>
> >>> * Failure
> >>>   - embedded mode on my laptop
> >>>   - drill memory: 2Gb/4Gb (heap/direct)
> >>>   - cpu: 4cores (+hyperthreading)
> >>>   - `planner.width.max_per_node=6`
> >>>
> >>> * Success
> >>>   - AWS Cluster 2x c3.8xlarge
> >>>   - drill memory: 16Gb/32Gb
> >>>   - cpu: limited by kubernetes to 24cores
> >>>   - `planner.width.max_per_node=23`
> >>>
> >>> I'm very busy right now to test again, but I'll try to provide better
> >> info
> >>> as soon as I can.
> >>>
> >>>
> >>>
> >>> On Wed, Aug 31, 2016 at 05:38:53PM +0530, Khurram Faraaz wrote:
> >>>
> >>>> Can you please share the number of cores on the setup where the query
> >> hung
> >>>> as compared to the number of cores on the setup where the query went
> >>>> through successfully.
> >>>> And details of memory from the two scenarios.
> >>>>
> >>>> Thanks,
> >>>> Khurram
> >>>>
> >>>> On Wed, Aug 31, 2016 at 4:50 PM, Oscar Morante <sp...@gmail.com>
> >>>> wrote:
> >>>>
> >>>> For the record, I think this was just bad memory configuration after
> >> all.
> >>>>> I retested on bigger machines and everything seems to be working
> fine.
> >>>>>
> >>>>>
> >>>>> On Tue, Aug 09, 2016 at 10:46:33PM +0530, Khurram Faraaz wrote:
> >>>>>
> >>>>> Oscar, can you please report a JIRA with the required steps to
> >> reproduce
> >>>>>> the OOM error. That way someone from the Drill team will take a look
> >> and
> >>>>>> investigate.
> >>>>>>
> >>>>>> For others interested here is the stack trace.
> >>>>>>
> >>>>>> 2016-08-09 16:51:14,280 [285642de-ab37-de6e-a54c-
> >> 378aaa4ce50e:foreman]
> >>>>>> ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure
> >>>>>> Occurred,
> >>>>>> exiting. Information message: Unable to handle out of memory
> condition
> >>>>>> in
> >>>>>> Foreman.
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>       at java.util.Arrays.copyOfRange(Arrays.java:2694)
> >>>>>> ~[na:1.7.0_111]
> >>>>>>       at java.lang.String.<init>(String.java:203) ~[na:1.7.0_111]
> >>>>>>       at java.lang.StringBuilder.toString(StringBuilder.java:405)
> >>>>>> ~[na:1.7.0_111]
> >>>>>>       at org.apache.calcite.util.Util.newInternal(Util.java:785)
> >>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
> >>>>>>       at
> >>>>>> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(
> >>>>>> VolcanoRuleCall.java:251)
> >>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
> >>>>>>       at
> >>>>>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(
> >>>>>> VolcanoPlanner.java:808)
> >>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
> >>>>>>       at
> >>>>>> org.apache.calcite.tools.Programs$RuleSetProgram.run(
> >> Programs.java:303)
> >>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
> >>>>>>       at
> >>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
> >>>>>> .transform(DefaultSqlHandler.java:404)
> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>>>       at
> >>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
> >>>>>> .transform(DefaultSqlHandler.java:343)
> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>>>       at
> >>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
> >>>>>> .convertToDrel(DefaultSqlHandler.java:240)
> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>>>       at
> >>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
> >>>>>> .convertToDrel(DefaultSqlHandler.java:290)
> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>>>       at
> >>>>>> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.ge
> >>>>>> tPlan(ExplainHandler.java:61)
> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>>>       at
> >>>>>> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(Dri
> >>>>>> llSqlWorker.java:94)
> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>>>       at
> >>>>>> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978)
> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>>>       at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.
> >> java:
> >>>>>> 257)
> >>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>>>       at
> >>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> >>>>>> Executor.java:1145)
> >>>>>> [na:1.7.0_111]
> >>>>>>       at
> >>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> >>>>>> lExecutor.java:615)
> >>>>>> [na:1.7.0_111]
> >>>>>>       at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Khurram
> >>>>>>
> >>>>>> On Tue, Aug 9, 2016 at 7:46 PM, Oscar Morante <sp...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>> Yeah, when I uncomment only the `upload_date` lines (a dir0 alias),
> >>>>>>
> >>>>>>> explain succeeds within ~30s.  Enabling any of the other lines
> >> triggers
> >>>>>>> the
> >>>>>>> failure.
> >>>>>>>
> >>>>>>> This is a log with the `upload_date` lines and `usage <> 'Test'`
> >>>>>>> enabled:
> >>>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e
> >>>>>>>
> >>>>>>> The client times out around here (~1.5hours):
> >>>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
> >>>>>>> b3c55e#file-drillbit-log-L178
> >>>>>>>
> >>>>>>> And it still keeps running for a while until it dies (~2.5hours):
> >>>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
> >>>>>>> b3c55e#file-drillbit-log-L178
> >>>>>>>
> >>>>>>> The memory settings for this test were:
> >>>>>>>
> >>>>>>>   DRILL_HEAP="4G"
> >>>>>>>   DRILL_MAX_DIRECT_MEMORY="8G"
> >>>>>>>
> >>>>>>> This is on a laptop with 16G and I should probably lower it, but it
> >>>>>>> seems
> >>>>>>> a bit excessive for such a small query.  And I think I got the same
> >>>>>>> results
> >>>>>>> on a 2 node cluster with 8/16.  I'm gonna try again on the cluster
> to
> >>>>>>> make
> >>>>>>> sure.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Oscar
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Aug 09, 2016 at 04:13:17PM +0530, Khurram Faraaz wrote:
> >>>>>>>
> >>>>>>> You mentioned "*But if I uncomment the where clause then it runs
> for
> >> a
> >>>>>>>
> >>>>>>>> couple of hours until it runs out of memory.*"
> >>>>>>>>
> >>>>>>>> Can you please share the OutOfMemory details from drillbit.log and
> >> the
> >>>>>>>> value of DRILL_MAX_DIRECT_MEMORY
> >>>>>>>>
> >>>>>>>> Can you also try to see what happens if you retain just this line
> >>>>>>>> where
> >>>>>>>> upload_date = '2016-08-01' in your where clause, can you check if
> >> the
> >>>>>>>> explain succeeds.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Khurram
> >>>>>>>>
> >>>>>>>> On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <
> spacepluk@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi there,
> >>>>>>>>
> >>>>>>>> I've been stuck with this for a while and I'm not sure if I'm
> >> running
> >>>>>>>>> into
> >>>>>>>>> a bug or I'm just doing something very wrong.
> >>>>>>>>>
> >>>>>>>>> I have this stripped-down version of my query:
> >>>>>>>>> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c80
> 5b
> >>>>>>>>>
> >>>>>>>>> The data is just a single file with one record (1.5K).
> >>>>>>>>>
> >>>>>>>>> Without changing anything, explain takes ~1sec on my machine.
> But
> >>>>>>>>> if I
> >>>>>>>>> uncomment the where clause then it runs for a couple of hours
> until
> >>>>>>>>> it
> >>>>>>>>> runs
> >>>>>>>>> out of memory.
> >>>>>>>>>
> >>>>>>>>> Also if I uncomment the where clause *and* take out the join,
> then
> >> it
> >>>>>>>>> takes around 30s to plan.
> >>>>>>>>>
> >>>>>>>>> Any ideas?
> >>>>>>>>> Thanks!
> >>>>>>>>>
> >>>>>>>>>
> >>
>
>

Re: Query hangs on planning

Posted by Sudheesh Katkam <sk...@maprtech.com>.
That setting is for off-heap memory. The earlier case hit heap memory limit.

> On Sep 1, 2016, at 11:36 AM, Zelaine Fong <zf...@maprtech.com> wrote:
> 
> One other thing ... have you tried tuning the planner.memory_limit
> parameter?  Based on the earlier stack trace, you're hitting a memory limit
> during query planning.  So, tuning this parameter should help that.  The
> default is 256 MB.
> 
> -- Zelaine
> 
> On Thu, Sep 1, 2016 at 11:21 AM, rahul challapalli <
> challapallirahul@gmail.com> wrote:
> 
>> While planning we use heap memory. 2GB of heap should be sufficient for
>> what you mentioned. This looks like a bug to me. Can you raise a jira for
>> the same? And it would be super helpful if you can also attach the data set
>> used.
>> 
>> Rahul
>> 
>> On Wed, Aug 31, 2016 at 9:14 AM, Oscar Morante <sp...@gmail.com>
>> wrote:
>> 
>>> Sure,
>>> This is what I remember:
>>> 
>>> * Failure
>>>   - embedded mode on my laptop
>>>   - drill memory: 2Gb/4Gb (heap/direct)
>>>   - cpu: 4cores (+hyperthreading)
>>>   - `planner.width.max_per_node=6`
>>> 
>>> * Success
>>>   - AWS Cluster 2x c3.8xlarge
>>>   - drill memory: 16Gb/32Gb
>>>   - cpu: limited by kubernetes to 24cores
>>>   - `planner.width.max_per_node=23`
>>> 
>>> I'm very busy right now to test again, but I'll try to provide better
>> info
>>> as soon as I can.
>>> 
>>> 
>>> 
>>> On Wed, Aug 31, 2016 at 05:38:53PM +0530, Khurram Faraaz wrote:
>>> 
>>>> Can you please share the number of cores on the setup where the query
>> hung
>>>> as compared to the number of cores on the setup where the query went
>>>> through successfully.
>>>> And details of memory from the two scenarios.
>>>> 
>>>> Thanks,
>>>> Khurram
>>>> 
>>>> On Wed, Aug 31, 2016 at 4:50 PM, Oscar Morante <sp...@gmail.com>
>>>> wrote:
>>>> 
>>>> For the record, I think this was just bad memory configuration after
>> all.
>>>>> I retested on bigger machines and everything seems to be working fine.
>>>>> 
>>>>> 
>>>>> On Tue, Aug 09, 2016 at 10:46:33PM +0530, Khurram Faraaz wrote:
>>>>> 
>>>>> Oscar, can you please report a JIRA with the required steps to
>> reproduce
>>>>>> the OOM error. That way someone from the Drill team will take a look
>> and
>>>>>> investigate.
>>>>>> 
>>>>>> For others interested here is the stack trace.
>>>>>> 
>>>>>> 2016-08-09 16:51:14,280 [285642de-ab37-de6e-a54c-
>> 378aaa4ce50e:foreman]
>>>>>> ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure
>>>>>> Occurred,
>>>>>> exiting. Information message: Unable to handle out of memory condition
>>>>>> in
>>>>>> Foreman.
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>       at java.util.Arrays.copyOfRange(Arrays.java:2694)
>>>>>> ~[na:1.7.0_111]
>>>>>>       at java.lang.String.<init>(String.java:203) ~[na:1.7.0_111]
>>>>>>       at java.lang.StringBuilder.toString(StringBuilder.java:405)
>>>>>> ~[na:1.7.0_111]
>>>>>>       at org.apache.calcite.util.Util.newInternal(Util.java:785)
>>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>>>>       at
>>>>>> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(
>>>>>> VolcanoRuleCall.java:251)
>>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>>>>       at
>>>>>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(
>>>>>> VolcanoPlanner.java:808)
>>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>>>>       at
>>>>>> org.apache.calcite.tools.Programs$RuleSetProgram.run(
>> Programs.java:303)
>>>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>>>>       at
>>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>>>>> .transform(DefaultSqlHandler.java:404)
>>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>>>       at
>>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>>>>> .transform(DefaultSqlHandler.java:343)
>>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>>>       at
>>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>>>>> .convertToDrel(DefaultSqlHandler.java:240)
>>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>>>       at
>>>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>>>>> .convertToDrel(DefaultSqlHandler.java:290)
>>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>>>       at
>>>>>> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.ge
>>>>>> tPlan(ExplainHandler.java:61)
>>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>>>       at
>>>>>> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(Dri
>>>>>> llSqlWorker.java:94)
>>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>>>       at
>>>>>> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978)
>>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>>>       at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.
>> java:
>>>>>> 257)
>>>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>>>       at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>>>> Executor.java:1145)
>>>>>> [na:1.7.0_111]
>>>>>>       at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>>>> lExecutor.java:615)
>>>>>> [na:1.7.0_111]
>>>>>>       at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
>>>>>> 
>>>>>> Thanks,
>>>>>> Khurram
>>>>>> 
>>>>>> On Tue, Aug 9, 2016 at 7:46 PM, Oscar Morante <sp...@gmail.com>
>>>>>> wrote:
>>>>>> 
>>>>>> Yeah, when I uncomment only the `upload_date` lines (a dir0 alias),
>>>>>> 
>>>>>>> explain succeeds within ~30s.  Enabling any of the other lines
>> triggers
>>>>>>> the
>>>>>>> failure.
>>>>>>> 
>>>>>>> This is a log with the `upload_date` lines and `usage <> 'Test'`
>>>>>>> enabled:
>>>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e
>>>>>>> 
>>>>>>> The client times out around here (~1.5hours):
>>>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>>>>>>> b3c55e#file-drillbit-log-L178
>>>>>>> 
>>>>>>> And it still keeps running for a while until it dies (~2.5hours):
>>>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>>>>>>> b3c55e#file-drillbit-log-L178
>>>>>>> 
>>>>>>> The memory settings for this test were:
>>>>>>> 
>>>>>>>   DRILL_HEAP="4G"
>>>>>>>   DRILL_MAX_DIRECT_MEMORY="8G"
>>>>>>> 
>>>>>>> This is on a laptop with 16G and I should probably lower it, but it
>>>>>>> seems
>>>>>>> a bit excessive for such a small query.  And I think I got the same
>>>>>>> results
>>>>>>> on a 2 node cluster with 8/16.  I'm gonna try again on the cluster to
>>>>>>> make
>>>>>>> sure.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Oscar
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Aug 09, 2016 at 04:13:17PM +0530, Khurram Faraaz wrote:
>>>>>>> 
>>>>>>> You mentioned "*But if I uncomment the where clause then it runs for
>> a
>>>>>>> 
>>>>>>>> couple of hours until it runs out of memory.*"
>>>>>>>> 
>>>>>>>> Can you please share the OutOfMemory details from drillbit.log and
>> the
>>>>>>>> value of DRILL_MAX_DIRECT_MEMORY
>>>>>>>> 
>>>>>>>> Can you also try to see what happens if you retain just this line
>>>>>>>> where
>>>>>>>> upload_date = '2016-08-01' in your where clause, can you check if
>> the
>>>>>>>> explain succeeds.
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Khurram
>>>>>>>> 
>>>>>>>> On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <sp...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi there,
>>>>>>>> 
>>>>>>>> I've been stuck with this for a while and I'm not sure if I'm
>> running
>>>>>>>>> into
>>>>>>>>> a bug or I'm just doing something very wrong.
>>>>>>>>> 
>>>>>>>>> I have this stripped-down version of my query:
>>>>>>>>> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c805b
>>>>>>>>> 
>>>>>>>>> The data is just a single file with one record (1.5K).
>>>>>>>>> 
>>>>>>>>> Without changing anything, explain takes ~1sec on my machine.  But
>>>>>>>>> if I
>>>>>>>>> uncomment the where clause then it runs for a couple of hours until
>>>>>>>>> it
>>>>>>>>> runs
>>>>>>>>> out of memory.
>>>>>>>>> 
>>>>>>>>> Also if I uncomment the where clause *and* take out the join, then
>> it
>>>>>>>>> takes around 30s to plan.
>>>>>>>>> 
>>>>>>>>> Any ideas?
>>>>>>>>> Thanks!
>>>>>>>>> 
>>>>>>>>> 
>> 


Re: Query hangs on planning

Posted by Zelaine Fong <zf...@maprtech.com>.
One other thing ... have you tried tuning the planner.memory_limit
parameter?  Based on the earlier stack trace, you're hitting a memory limit
during query planning.  So, tuning this parameter should help that.  The
default is 256 MB.

-- Zelaine

On Thu, Sep 1, 2016 at 11:21 AM, rahul challapalli <
challapallirahul@gmail.com> wrote:

> While planning we use heap memory. 2GB of heap should be sufficient for
> what you mentioned. This looks like a bug to me. Can you raise a jira for
> the same? And it would be super helpful if you can also attach the data set
> used.
>
> Rahul
>
> On Wed, Aug 31, 2016 at 9:14 AM, Oscar Morante <sp...@gmail.com>
> wrote:
>
> > Sure,
> > This is what I remember:
> >
> > * Failure
> >    - embedded mode on my laptop
> >    - drill memory: 2Gb/4Gb (heap/direct)
> >    - cpu: 4cores (+hyperthreading)
> >    - `planner.width.max_per_node=6`
> >
> > * Success
> >    - AWS Cluster 2x c3.8xlarge
> >    - drill memory: 16Gb/32Gb
> >    - cpu: limited by kubernetes to 24cores
> >    - `planner.width.max_per_node=23`
> >
> > I'm very busy right now to test again, but I'll try to provide better
> info
> > as soon as I can.
> >
> >
> >
> > On Wed, Aug 31, 2016 at 05:38:53PM +0530, Khurram Faraaz wrote:
> >
> >> Can you please share the number of cores on the setup where the query
> hung
> >> as compared to the number of cores on the setup where the query went
> >> through successfully.
> >> And details of memory from the two scenarios.
> >>
> >> Thanks,
> >> Khurram
> >>
> >> On Wed, Aug 31, 2016 at 4:50 PM, Oscar Morante <sp...@gmail.com>
> >> wrote:
> >>
> >> For the record, I think this was just bad memory configuration after
> all.
> >>> I retested on bigger machines and everything seems to be working fine.
> >>>
> >>>
> >>> On Tue, Aug 09, 2016 at 10:46:33PM +0530, Khurram Faraaz wrote:
> >>>
> >>> Oscar, can you please report a JIRA with the required steps to
> reproduce
> >>>> the OOM error. That way someone from the Drill team will take a look
> and
> >>>> investigate.
> >>>>
> >>>> For others interested here is the stack trace.
> >>>>
> >>>> 2016-08-09 16:51:14,280 [285642de-ab37-de6e-a54c-
> 378aaa4ce50e:foreman]
> >>>> ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure
> >>>> Occurred,
> >>>> exiting. Information message: Unable to handle out of memory condition
> >>>> in
> >>>> Foreman.
> >>>> java.lang.OutOfMemoryError: Java heap space
> >>>>        at java.util.Arrays.copyOfRange(Arrays.java:2694)
> >>>> ~[na:1.7.0_111]
> >>>>        at java.lang.String.<init>(String.java:203) ~[na:1.7.0_111]
> >>>>        at java.lang.StringBuilder.toString(StringBuilder.java:405)
> >>>> ~[na:1.7.0_111]
> >>>>        at org.apache.calcite.util.Util.newInternal(Util.java:785)
> >>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
> >>>>        at
> >>>> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(
> >>>> VolcanoRuleCall.java:251)
> >>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
> >>>>        at
> >>>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(
> >>>> VolcanoPlanner.java:808)
> >>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
> >>>>        at
> >>>> org.apache.calcite.tools.Programs$RuleSetProgram.run(
> Programs.java:303)
> >>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
> >>>>        at
> >>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
> >>>> .transform(DefaultSqlHandler.java:404)
> >>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>        at
> >>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
> >>>> .transform(DefaultSqlHandler.java:343)
> >>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>        at
> >>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
> >>>> .convertToDrel(DefaultSqlHandler.java:240)
> >>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>        at
> >>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
> >>>> .convertToDrel(DefaultSqlHandler.java:290)
> >>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>        at
> >>>> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.ge
> >>>> tPlan(ExplainHandler.java:61)
> >>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>        at
> >>>> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(Dri
> >>>> llSqlWorker.java:94)
> >>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>        at
> >>>> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978)
> >>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.
> java:
> >>>> 257)
> >>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
> >>>>        at
> >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> >>>> Executor.java:1145)
> >>>> [na:1.7.0_111]
> >>>>        at
> >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> >>>> lExecutor.java:615)
> >>>> [na:1.7.0_111]
> >>>>        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
> >>>>
> >>>> Thanks,
> >>>> Khurram
> >>>>
> >>>> On Tue, Aug 9, 2016 at 7:46 PM, Oscar Morante <sp...@gmail.com>
> >>>> wrote:
> >>>>
> >>>> Yeah, when I uncomment only the `upload_date` lines (a dir0 alias),
> >>>>
> >>>>> explain succeeds within ~30s.  Enabling any of the other lines
> triggers
> >>>>> the
> >>>>> failure.
> >>>>>
> >>>>> This is a log with the `upload_date` lines and `usage <> 'Test'`
> >>>>> enabled:
> >>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e
> >>>>>
> >>>>> The client times out around here (~1.5hours):
> >>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
> >>>>> b3c55e#file-drillbit-log-L178
> >>>>>
> >>>>> And it still keeps running for a while until it dies (~2.5hours):
> >>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
> >>>>> b3c55e#file-drillbit-log-L178
> >>>>>
> >>>>> The memory settings for this test were:
> >>>>>
> >>>>>    DRILL_HEAP="4G"
> >>>>>    DRILL_MAX_DIRECT_MEMORY="8G"
> >>>>>
> >>>>> This is on a laptop with 16G and I should probably lower it, but it
> >>>>> seems
> >>>>> a bit excessive for such a small query.  And I think I got the same
> >>>>> results
> >>>>> on a 2 node cluster with 8/16.  I'm gonna try again on the cluster to
> >>>>> make
> >>>>> sure.
> >>>>>
> >>>>> Thanks,
> >>>>> Oscar
> >>>>>
> >>>>>
> >>>>> On Tue, Aug 09, 2016 at 04:13:17PM +0530, Khurram Faraaz wrote:
> >>>>>
> >>>>> You mentioned "*But if I uncomment the where clause then it runs for
> a
> >>>>>
> >>>>>> couple of hours until it runs out of memory.*"
> >>>>>>
> >>>>>> Can you please share the OutOfMemory details from drillbit.log and
> the
> >>>>>> value of DRILL_MAX_DIRECT_MEMORY
> >>>>>>
> >>>>>> Can you also try to see what happens if you retain just this line
> >>>>>> where
> >>>>>> upload_date = '2016-08-01' in your where clause, can you check if
> the
> >>>>>> explain succeeds.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Khurram
> >>>>>>
> >>>>>> On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <sp...@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>> Hi there,
> >>>>>>
> >>>>>> I've been stuck with this for a while and I'm not sure if I'm
> running
> >>>>>>> into
> >>>>>>> a bug or I'm just doing something very wrong.
> >>>>>>>
> >>>>>>> I have this stripped-down version of my query:
> >>>>>>> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c805b
> >>>>>>>
> >>>>>>> The data is just a single file with one record (1.5K).
> >>>>>>>
> >>>>>>> Without changing anything, explain takes ~1sec on my machine.  But
> >>>>>>> if I
> >>>>>>> uncomment the where clause then it runs for a couple of hours until
> >>>>>>> it
> >>>>>>> runs
> >>>>>>> out of memory.
> >>>>>>>
> >>>>>>> Also if I uncomment the where clause *and* take out the join, then
> it
> >>>>>>> takes around 30s to plan.
> >>>>>>>
> >>>>>>> Any ideas?
> >>>>>>> Thanks!
> >>>>>>>
> >>>>>>>
>

Re: Query hangs on planning

Posted by rahul challapalli <ch...@gmail.com>.
While planning we use heap memory. 2GB of heap should be sufficient for
what you mentioned. This looks like a bug to me. Can you raise a jira for
the same? And it would be super helpful if you can also attach the data set
used.

Rahul

On Wed, Aug 31, 2016 at 9:14 AM, Oscar Morante <sp...@gmail.com> wrote:

> Sure,
> This is what I remember:
>
> * Failure
>    - embedded mode on my laptop
>    - drill memory: 2Gb/4Gb (heap/direct)
>    - cpu: 4cores (+hyperthreading)
>    - `planner.width.max_per_node=6`
>
> * Success
>    - AWS Cluster 2x c3.8xlarge
>    - drill memory: 16Gb/32Gb
>    - cpu: limited by kubernetes to 24cores
>    - `planner.width.max_per_node=23`
>
> I'm very busy right now to test again, but I'll try to provide better info
> as soon as I can.
>
>
>
> On Wed, Aug 31, 2016 at 05:38:53PM +0530, Khurram Faraaz wrote:
>
>> Can you please share the number of cores on the setup where the query hung
>> as compared to the number of cores on the setup where the query went
>> through successfully.
>> And details of memory from the two scenarios.
>>
>> Thanks,
>> Khurram
>>
>> On Wed, Aug 31, 2016 at 4:50 PM, Oscar Morante <sp...@gmail.com>
>> wrote:
>>
>> For the record, I think this was just bad memory configuration after all.
>>> I retested on bigger machines and everything seems to be working fine.
>>>
>>>
>>> On Tue, Aug 09, 2016 at 10:46:33PM +0530, Khurram Faraaz wrote:
>>>
>>> Oscar, can you please report a JIRA with the required steps to reproduce
>>>> the OOM error. That way someone from the Drill team will take a look and
>>>> investigate.
>>>>
>>>> For others interested here is the stack trace.
>>>>
>>>> 2016-08-09 16:51:14,280 [285642de-ab37-de6e-a54c-378aaa4ce50e:foreman]
>>>> ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure
>>>> Occurred,
>>>> exiting. Information message: Unable to handle out of memory condition
>>>> in
>>>> Foreman.
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>        at java.util.Arrays.copyOfRange(Arrays.java:2694)
>>>> ~[na:1.7.0_111]
>>>>        at java.lang.String.<init>(String.java:203) ~[na:1.7.0_111]
>>>>        at java.lang.StringBuilder.toString(StringBuilder.java:405)
>>>> ~[na:1.7.0_111]
>>>>        at org.apache.calcite.util.Util.newInternal(Util.java:785)
>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>>        at
>>>> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(
>>>> VolcanoRuleCall.java:251)
>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>>        at
>>>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(
>>>> VolcanoPlanner.java:808)
>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>>        at
>>>> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
>>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>>        at
>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>>> .transform(DefaultSqlHandler.java:404)
>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>        at
>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>>> .transform(DefaultSqlHandler.java:343)
>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>        at
>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>>> .convertToDrel(DefaultSqlHandler.java:240)
>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>        at
>>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>>> .convertToDrel(DefaultSqlHandler.java:290)
>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>        at
>>>> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.ge
>>>> tPlan(ExplainHandler.java:61)
>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>        at
>>>> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(Dri
>>>> llSqlWorker.java:94)
>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>        at
>>>> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978)
>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:
>>>> 257)
>>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>>        at
>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>>> Executor.java:1145)
>>>> [na:1.7.0_111]
>>>>        at
>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>>> lExecutor.java:615)
>>>> [na:1.7.0_111]
>>>>        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
>>>>
>>>> Thanks,
>>>> Khurram
>>>>
>>>> On Tue, Aug 9, 2016 at 7:46 PM, Oscar Morante <sp...@gmail.com>
>>>> wrote:
>>>>
>>>> Yeah, when I uncomment only the `upload_date` lines (a dir0 alias),
>>>>
>>>>> explain succeeds within ~30s.  Enabling any of the other lines triggers
>>>>> the
>>>>> failure.
>>>>>
>>>>> This is a log with the `upload_date` lines and `usage <> 'Test'`
>>>>> enabled:
>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e
>>>>>
>>>>> The client times out around here (~1.5hours):
>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>>>>> b3c55e#file-drillbit-log-L178
>>>>>
>>>>> And it still keeps running for a while until it dies (~2.5hours):
>>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>>>>> b3c55e#file-drillbit-log-L178
>>>>>
>>>>> The memory settings for this test were:
>>>>>
>>>>>    DRILL_HEAP="4G"
>>>>>    DRILL_MAX_DIRECT_MEMORY="8G"
>>>>>
>>>>> This is on a laptop with 16G and I should probably lower it, but it
>>>>> seems
>>>>> a bit excessive for such a small query.  And I think I got the same
>>>>> results
>>>>> on a 2 node cluster with 8/16.  I'm gonna try again on the cluster to
>>>>> make
>>>>> sure.
>>>>>
>>>>> Thanks,
>>>>> Oscar
>>>>>
>>>>>
>>>>> On Tue, Aug 09, 2016 at 04:13:17PM +0530, Khurram Faraaz wrote:
>>>>>
>>>>> You mentioned "*But if I uncomment the where clause then it runs for a
>>>>>
>>>>>> couple of hours until it runs out of memory.*"
>>>>>>
>>>>>> Can you please share the OutOfMemory details from drillbit.log and the
>>>>>> value of DRILL_MAX_DIRECT_MEMORY
>>>>>>
>>>>>> Can you also try to see what happens if you retain just this line
>>>>>> where
>>>>>> upload_date = '2016-08-01' in your where clause, can you check if the
>>>>>> explain succeeds.
>>>>>>
>>>>>> Thanks,
>>>>>> Khurram
>>>>>>
>>>>>> On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <sp...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Hi there,
>>>>>>
>>>>>> I've been stuck with this for a while and I'm not sure if I'm running
>>>>>>> into
>>>>>>> a bug or I'm just doing something very wrong.
>>>>>>>
>>>>>>> I have this stripped-down version of my query:
>>>>>>> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c805b
>>>>>>>
>>>>>>> The data is just a single file with one record (1.5K).
>>>>>>>
>>>>>>> Without changing anything, explain takes ~1sec on my machine.  But
>>>>>>> if I
>>>>>>> uncomment the where clause then it runs for a couple of hours until
>>>>>>> it
>>>>>>> runs
>>>>>>> out of memory.
>>>>>>>
>>>>>>> Also if I uncomment the where clause *and* take out the join, then it
>>>>>>> takes around 30s to plan.
>>>>>>>
>>>>>>> Any ideas?
>>>>>>> Thanks!
>>>>>>>
>>>>>>>

Re: Query hangs on planning

Posted by Oscar Morante <sp...@gmail.com>.
Sure,
This is what I remember:

* Failure
    - embedded mode on my laptop
    - drill memory: 2Gb/4Gb (heap/direct)
    - cpu: 4cores (+hyperthreading)
    - `planner.width.max_per_node=6`

* Success
    - AWS Cluster 2x c3.8xlarge
    - drill memory: 16Gb/32Gb
    - cpu: limited by kubernetes to 24cores
    - `planner.width.max_per_node=23`

I'm very busy right now to test again, but I'll try to provide better 
info as soon as I can.


On Wed, Aug 31, 2016 at 05:38:53PM +0530, Khurram Faraaz wrote:
>Can you please share the number of cores on the setup where the query hung
>as compared to the number of cores on the setup where the query went
>through successfully.
>And details of memory from the two scenarios.
>
>Thanks,
>Khurram
>
>On Wed, Aug 31, 2016 at 4:50 PM, Oscar Morante <sp...@gmail.com> wrote:
>
>> For the record, I think this was just bad memory configuration after all.
>> I retested on bigger machines and everything seems to be working fine.
>>
>>
>> On Tue, Aug 09, 2016 at 10:46:33PM +0530, Khurram Faraaz wrote:
>>
>>> Oscar, can you please report a JIRA with the required steps to reproduce
>>> the OOM error. That way someone from the Drill team will take a look and
>>> investigate.
>>>
>>> For others interested here is the stack trace.
>>>
>>> 2016-08-09 16:51:14,280 [285642de-ab37-de6e-a54c-378aaa4ce50e:foreman]
>>> ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure
>>> Occurred,
>>> exiting. Information message: Unable to handle out of memory condition in
>>> Foreman.
>>> java.lang.OutOfMemoryError: Java heap space
>>>        at java.util.Arrays.copyOfRange(Arrays.java:2694) ~[na:1.7.0_111]
>>>        at java.lang.String.<init>(String.java:203) ~[na:1.7.0_111]
>>>        at java.lang.StringBuilder.toString(StringBuilder.java:405)
>>> ~[na:1.7.0_111]
>>>        at org.apache.calcite.util.Util.newInternal(Util.java:785)
>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>        at
>>> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(
>>> VolcanoRuleCall.java:251)
>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>        at
>>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(
>>> VolcanoPlanner.java:808)
>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>        at
>>> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
>>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>>        at
>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>> .transform(DefaultSqlHandler.java:404)
>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>        at
>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>> .transform(DefaultSqlHandler.java:343)
>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>        at
>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>> .convertToDrel(DefaultSqlHandler.java:240)
>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>        at
>>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>>> .convertToDrel(DefaultSqlHandler.java:290)
>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>        at
>>> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.ge
>>> tPlan(ExplainHandler.java:61)
>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>        at
>>> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(Dri
>>> llSqlWorker.java:94)
>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>        at
>>> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978)
>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:
>>> 257)
>>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>>        at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>>> Executor.java:1145)
>>> [na:1.7.0_111]
>>>        at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>>> lExecutor.java:615)
>>> [na:1.7.0_111]
>>>        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
>>>
>>> Thanks,
>>> Khurram
>>>
>>> On Tue, Aug 9, 2016 at 7:46 PM, Oscar Morante <sp...@gmail.com>
>>> wrote:
>>>
>>> Yeah, when I uncomment only the `upload_date` lines (a dir0 alias),
>>>> explain succeeds within ~30s.  Enabling any of the other lines triggers
>>>> the
>>>> failure.
>>>>
>>>> This is a log with the `upload_date` lines and `usage <> 'Test'` enabled:
>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e
>>>>
>>>> The client times out around here (~1.5hours):
>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>>>> b3c55e#file-drillbit-log-L178
>>>>
>>>> And it still keeps running for a while until it dies (~2.5hours):
>>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>>>> b3c55e#file-drillbit-log-L178
>>>>
>>>> The memory settings for this test were:
>>>>
>>>>    DRILL_HEAP="4G"
>>>>    DRILL_MAX_DIRECT_MEMORY="8G"
>>>>
>>>> This is on a laptop with 16G and I should probably lower it, but it seems
>>>> a bit excessive for such a small query.  And I think I got the same
>>>> results
>>>> on a 2 node cluster with 8/16.  I'm gonna try again on the cluster to
>>>> make
>>>> sure.
>>>>
>>>> Thanks,
>>>> Oscar
>>>>
>>>>
>>>> On Tue, Aug 09, 2016 at 04:13:17PM +0530, Khurram Faraaz wrote:
>>>>
>>>> You mentioned "*But if I uncomment the where clause then it runs for a
>>>>> couple of hours until it runs out of memory.*"
>>>>>
>>>>> Can you please share the OutOfMemory details from drillbit.log and the
>>>>> value of DRILL_MAX_DIRECT_MEMORY
>>>>>
>>>>> Can you also try to see what happens if you retain just this line where
>>>>> upload_date = '2016-08-01' in your where clause, can you check if the
>>>>> explain succeeds.
>>>>>
>>>>> Thanks,
>>>>> Khurram
>>>>>
>>>>> On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <sp...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Hi there,
>>>>>
>>>>>> I've been stuck with this for a while and I'm not sure if I'm running
>>>>>> into
>>>>>> a bug or I'm just doing something very wrong.
>>>>>>
>>>>>> I have this stripped-down version of my query:
>>>>>> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c805b
>>>>>>
>>>>>> The data is just a single file with one record (1.5K).
>>>>>>
>>>>>> Without changing anything, explain takes ~1sec on my machine.  But if I
>>>>>> uncomment the where clause then it runs for a couple of hours until it
>>>>>> runs
>>>>>> out of memory.
>>>>>>
>>>>>> Also if I uncomment the where clause *and* take out the join, then it
>>>>>> takes around 30s to plan.
>>>>>>
>>>>>> Any ideas?
>>>>>> Thanks!
>>>>>>

Re: Query hangs on planning

Posted by Khurram Faraaz <kf...@maprtech.com>.
Can you please share the number of cores on the setup where the query hung
as compared to the number of cores on the setup where the query went
through successfully.
And details of memory from the two scenarios.

Thanks,
Khurram

On Wed, Aug 31, 2016 at 4:50 PM, Oscar Morante <sp...@gmail.com> wrote:

> For the record, I think this was just bad memory configuration after all.
> I retested on bigger machines and everything seems to be working fine.
>
>
> On Tue, Aug 09, 2016 at 10:46:33PM +0530, Khurram Faraaz wrote:
>
>> Oscar, can you please report a JIRA with the required steps to reproduce
>> the OOM error. That way someone from the Drill team will take a look and
>> investigate.
>>
>> For others interested here is the stack trace.
>>
>> 2016-08-09 16:51:14,280 [285642de-ab37-de6e-a54c-378aaa4ce50e:foreman]
>> ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure
>> Occurred,
>> exiting. Information message: Unable to handle out of memory condition in
>> Foreman.
>> java.lang.OutOfMemoryError: Java heap space
>>        at java.util.Arrays.copyOfRange(Arrays.java:2694) ~[na:1.7.0_111]
>>        at java.lang.String.<init>(String.java:203) ~[na:1.7.0_111]
>>        at java.lang.StringBuilder.toString(StringBuilder.java:405)
>> ~[na:1.7.0_111]
>>        at org.apache.calcite.util.Util.newInternal(Util.java:785)
>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>        at
>> org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(
>> VolcanoRuleCall.java:251)
>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>        at
>> org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(
>> VolcanoPlanner.java:808)
>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>        at
>> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
>> ~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>>        at
>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>> .transform(DefaultSqlHandler.java:404)
>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>        at
>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>> .transform(DefaultSqlHandler.java:343)
>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>        at
>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>> .convertToDrel(DefaultSqlHandler.java:240)
>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>        at
>> org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler
>> .convertToDrel(DefaultSqlHandler.java:290)
>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>        at
>> org.apache.drill.exec.planner.sql.handlers.ExplainHandler.ge
>> tPlan(ExplainHandler.java:61)
>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>        at
>> org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(Dri
>> llSqlWorker.java:94)
>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>        at
>> org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978)
>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:
>> 257)
>> ~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>>        at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1145)
>> [na:1.7.0_111]
>>        at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:615)
>> [na:1.7.0_111]
>>        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
>>
>> Thanks,
>> Khurram
>>
>> On Tue, Aug 9, 2016 at 7:46 PM, Oscar Morante <sp...@gmail.com>
>> wrote:
>>
>> Yeah, when I uncomment only the `upload_date` lines (a dir0 alias),
>>> explain succeeds within ~30s.  Enabling any of the other lines triggers
>>> the
>>> failure.
>>>
>>> This is a log with the `upload_date` lines and `usage <> 'Test'` enabled:
>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e
>>>
>>> The client times out around here (~1.5hours):
>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>>> b3c55e#file-drillbit-log-L178
>>>
>>> And it still keeps running for a while until it dies (~2.5hours):
>>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>>> b3c55e#file-drillbit-log-L178
>>>
>>> The memory settings for this test were:
>>>
>>>    DRILL_HEAP="4G"
>>>    DRILL_MAX_DIRECT_MEMORY="8G"
>>>
>>> This is on a laptop with 16G and I should probably lower it, but it seems
>>> a bit excessive for such a small query.  And I think I got the same
>>> results
>>> on a 2 node cluster with 8/16.  I'm gonna try again on the cluster to
>>> make
>>> sure.
>>>
>>> Thanks,
>>> Oscar
>>>
>>>
>>> On Tue, Aug 09, 2016 at 04:13:17PM +0530, Khurram Faraaz wrote:
>>>
>>> You mentioned "*But if I uncomment the where clause then it runs for a
>>>> couple of hours until it runs out of memory.*"
>>>>
>>>> Can you please share the OutOfMemory details from drillbit.log and the
>>>> value of DRILL_MAX_DIRECT_MEMORY
>>>>
>>>> Can you also try to see what happens if you retain just this line where
>>>> upload_date = '2016-08-01' in your where clause, can you check if the
>>>> explain succeeds.
>>>>
>>>> Thanks,
>>>> Khurram
>>>>
>>>> On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <sp...@gmail.com>
>>>> wrote:
>>>>
>>>> Hi there,
>>>>
>>>>> I've been stuck with this for a while and I'm not sure if I'm running
>>>>> into
>>>>> a bug or I'm just doing something very wrong.
>>>>>
>>>>> I have this stripped-down version of my query:
>>>>> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c805b
>>>>>
>>>>> The data is just a single file with one record (1.5K).
>>>>>
>>>>> Without changing anything, explain takes ~1sec on my machine.  But if I
>>>>> uncomment the where clause then it runs for a couple of hours until it
>>>>> runs
>>>>> out of memory.
>>>>>
>>>>> Also if I uncomment the where clause *and* take out the join, then it
>>>>> takes around 30s to plan.
>>>>>
>>>>> Any ideas?
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>>
> --
> Oscar Morante
> "Self-education is, I firmly believe, the only kind of education there is."
>                                                          -- Isaac Asimov.
>

Re: Query hangs on planning

Posted by Oscar Morante <sp...@gmail.com>.
For the record, I think this was just bad memory configuration after 
all.  I retested on bigger machines and everything seems to be working 
fine.

On Tue, Aug 09, 2016 at 10:46:33PM +0530, Khurram Faraaz wrote:
>Oscar, can you please report a JIRA with the required steps to reproduce
>the OOM error. That way someone from the Drill team will take a look and
>investigate.
>
>For others interested here is the stack trace.
>
>2016-08-09 16:51:14,280 [285642de-ab37-de6e-a54c-378aaa4ce50e:foreman]
>ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred,
>exiting. Information message: Unable to handle out of memory condition in
>Foreman.
>java.lang.OutOfMemoryError: Java heap space
>        at java.util.Arrays.copyOfRange(Arrays.java:2694) ~[na:1.7.0_111]
>        at java.lang.String.<init>(String.java:203) ~[na:1.7.0_111]
>        at java.lang.StringBuilder.toString(StringBuilder.java:405)
>~[na:1.7.0_111]
>        at org.apache.calcite.util.Util.newInternal(Util.java:785)
>~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>        at
>org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
>~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>        at
>org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
>~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>        at
>org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
>~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
>        at
>org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:404)
>~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>        at
>org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:343)
>~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>        at
>org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:240)
>~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>        at
>org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:290)
>~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>        at
>org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
>~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>        at
>org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:94)
>~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>        at
>org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978)
>~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:257)
>~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
>        at
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>[na:1.7.0_111]
>        at
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>[na:1.7.0_111]
>        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]
>
>Thanks,
>Khurram
>
>On Tue, Aug 9, 2016 at 7:46 PM, Oscar Morante <sp...@gmail.com> wrote:
>
>> Yeah, when I uncomment only the `upload_date` lines (a dir0 alias),
>> explain succeeds within ~30s.  Enabling any of the other lines triggers the
>> failure.
>>
>> This is a log with the `upload_date` lines and `usage <> 'Test'` enabled:
>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e
>>
>> The client times out around here (~1.5hours):
>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>> b3c55e#file-drillbit-log-L178
>>
>> And it still keeps running for a while until it dies (~2.5hours):
>> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
>> b3c55e#file-drillbit-log-L178
>>
>> The memory settings for this test were:
>>
>>    DRILL_HEAP="4G"
>>    DRILL_MAX_DIRECT_MEMORY="8G"
>>
>> This is on a laptop with 16G and I should probably lower it, but it seems
>> a bit excessive for such a small query.  And I think I got the same results
>> on a 2 node cluster with 8/16.  I'm gonna try again on the cluster to make
>> sure.
>>
>> Thanks,
>> Oscar
>>
>>
>> On Tue, Aug 09, 2016 at 04:13:17PM +0530, Khurram Faraaz wrote:
>>
>>> You mentioned "*But if I uncomment the where clause then it runs for a
>>> couple of hours until it runs out of memory.*"
>>>
>>> Can you please share the OutOfMemory details from drillbit.log and the
>>> value of DRILL_MAX_DIRECT_MEMORY
>>>
>>> Can you also try to see what happens if you retain just this line where
>>> upload_date = '2016-08-01' in your where clause, can you check if the
>>> explain succeeds.
>>>
>>> Thanks,
>>> Khurram
>>>
>>> On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <sp...@gmail.com>
>>> wrote:
>>>
>>> Hi there,
>>>> I've been stuck with this for a while and I'm not sure if I'm running
>>>> into
>>>> a bug or I'm just doing something very wrong.
>>>>
>>>> I have this stripped-down version of my query:
>>>> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c805b
>>>>
>>>> The data is just a single file with one record (1.5K).
>>>>
>>>> Without changing anything, explain takes ~1sec on my machine.  But if I
>>>> uncomment the where clause then it runs for a couple of hours until it
>>>> runs
>>>> out of memory.
>>>>
>>>> Also if I uncomment the where clause *and* take out the join, then it
>>>> takes around 30s to plan.
>>>>
>>>> Any ideas?
>>>> Thanks!
>>>>
>>>>
>>>>

-- 
Oscar Morante
"Self-education is, I firmly believe, the only kind of education there is."
                                                          -- Isaac Asimov.

Re: Query hangs on planning

Posted by Khurram Faraaz <kf...@maprtech.com>.
Oscar, can you please report a JIRA with the required steps to reproduce
the OOM error. That way someone from the Drill team will take a look and
investigate.

For others interested here is the stack trace.

2016-08-09 16:51:14,280 [285642de-ab37-de6e-a54c-378aaa4ce50e:foreman]
ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred,
exiting. Information message: Unable to handle out of memory condition in
Foreman.
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOfRange(Arrays.java:2694) ~[na:1.7.0_111]
        at java.lang.String.<init>(String.java:203) ~[na:1.7.0_111]
        at java.lang.StringBuilder.toString(StringBuilder.java:405)
~[na:1.7.0_111]
        at org.apache.calcite.util.Util.newInternal(Util.java:785)
~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
        at
org.apache.calcite.plan.volcano.VolcanoRuleCall.onMatch(VolcanoRuleCall.java:251)
~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
        at
org.apache.calcite.plan.volcano.VolcanoPlanner.findBestExp(VolcanoPlanner.java:808)
~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
        at
org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:303)
~[calcite-core-1.4.0-drill-r16-PATCHED.jar:1.4.0-drill-r16-PATCHED]
        at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:404)
~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
        at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.transform(DefaultSqlHandler.java:343)
~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
        at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:240)
~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
        at
org.apache.drill.exec.planner.sql.handlers.DefaultSqlHandler.convertToDrel(DefaultSqlHandler.java:290)
~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
        at
org.apache.drill.exec.planner.sql.handlers.ExplainHandler.getPlan(ExplainHandler.java:61)
~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
        at
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:94)
~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
        at
org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:978)
~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
        at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:257)
~[drill-java-exec-1.8.0-SNAPSHOT.jar:1.8.0-SNAPSHOT]
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_111]
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_111]
        at java.lang.Thread.run(Thread.java:745) [na:1.7.0_111]

Thanks,
Khurram

On Tue, Aug 9, 2016 at 7:46 PM, Oscar Morante <sp...@gmail.com> wrote:

> Yeah, when I uncomment only the `upload_date` lines (a dir0 alias),
> explain succeeds within ~30s.  Enabling any of the other lines triggers the
> failure.
>
> This is a log with the `upload_date` lines and `usage <> 'Test'` enabled:
> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e
>
> The client times out around here (~1.5hours):
> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
> b3c55e#file-drillbit-log-L178
>
> And it still keeps running for a while until it dies (~2.5hours):
> https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022
> b3c55e#file-drillbit-log-L178
>
> The memory settings for this test were:
>
>    DRILL_HEAP="4G"
>    DRILL_MAX_DIRECT_MEMORY="8G"
>
> This is on a laptop with 16G and I should probably lower it, but it seems
> a bit excessive for such a small query.  And I think I got the same results
> on a 2 node cluster with 8/16.  I'm gonna try again on the cluster to make
> sure.
>
> Thanks,
> Oscar
>
>
> On Tue, Aug 09, 2016 at 04:13:17PM +0530, Khurram Faraaz wrote:
>
>> You mentioned "*But if I uncomment the where clause then it runs for a
>> couple of hours until it runs out of memory.*"
>>
>> Can you please share the OutOfMemory details from drillbit.log and the
>> value of DRILL_MAX_DIRECT_MEMORY
>>
>> Can you also try to see what happens if you retain just this line where
>> upload_date = '2016-08-01' in your where clause, can you check if the
>> explain succeeds.
>>
>> Thanks,
>> Khurram
>>
>> On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <sp...@gmail.com>
>> wrote:
>>
>> Hi there,
>>> I've been stuck with this for a while and I'm not sure if I'm running
>>> into
>>> a bug or I'm just doing something very wrong.
>>>
>>> I have this stripped-down version of my query:
>>> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c805b
>>>
>>> The data is just a single file with one record (1.5K).
>>>
>>> Without changing anything, explain takes ~1sec on my machine.  But if I
>>> uncomment the where clause then it runs for a couple of hours until it
>>> runs
>>> out of memory.
>>>
>>> Also if I uncomment the where clause *and* take out the join, then it
>>> takes around 30s to plan.
>>>
>>> Any ideas?
>>> Thanks!
>>>
>>>
>>>

Re: Query hangs on planning

Posted by Oscar Morante <sp...@gmail.com>.
Yeah, when I uncomment only the `upload_date` lines (a dir0 alias), 
explain succeeds within ~30s.  Enabling any of the other lines triggers 
the failure.

This is a log with the `upload_date` lines and `usage <> 'Test'` 
enabled: 
https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e

The client times out around here (~1.5hours): 
https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e#file-drillbit-log-L178

And it still keeps running for a while until it dies (~2.5hours): 
https://gist.github.com/spacepluk/d7ac11c0de6859e4bd003d2022b3c55e#file-drillbit-log-L178

The memory settings for this test were:

    DRILL_HEAP="4G"
    DRILL_MAX_DIRECT_MEMORY="8G"

This is on a laptop with 16G and I should probably lower it, but it 
seems a bit excessive for such a small query.  And I think I got the 
same results on a 2 node cluster with 8/16.  I'm gonna try again on the 
cluster to make sure.

Thanks,
Oscar


On Tue, Aug 09, 2016 at 04:13:17PM +0530, Khurram Faraaz wrote:
>You mentioned "*But if I uncomment the where clause then it runs for a
>couple of hours until it runs out of memory.*"
>
>Can you please share the OutOfMemory details from drillbit.log and the
>value of DRILL_MAX_DIRECT_MEMORY
>
>Can you also try to see what happens if you retain just this line where
>upload_date = '2016-08-01' in your where clause, can you check if the
>explain succeeds.
>
>Thanks,
>Khurram
>
>On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <sp...@gmail.com> wrote:
>
>> Hi there,
>> I've been stuck with this for a while and I'm not sure if I'm running into
>> a bug or I'm just doing something very wrong.
>>
>> I have this stripped-down version of my query:
>> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c805b
>>
>> The data is just a single file with one record (1.5K).
>>
>> Without changing anything, explain takes ~1sec on my machine.  But if I
>> uncomment the where clause then it runs for a couple of hours until it runs
>> out of memory.
>>
>> Also if I uncomment the where clause *and* take out the join, then it
>> takes around 30s to plan.
>>
>> Any ideas?
>> Thanks!
>>
>>

Re: Query hangs on planning

Posted by Khurram Faraaz <kf...@maprtech.com>.
You mentioned "*But if I uncomment the where clause then it runs for a
couple of hours until it runs out of memory.*"

Can you please share the OutOfMemory details from drillbit.log and the
value of DRILL_MAX_DIRECT_MEMORY

Can you also try to see what happens if you retain just this line where
upload_date = '2016-08-01' in your where clause, can you check if the
explain succeeds.

Thanks,
Khurram

On Tue, Aug 9, 2016 at 4:00 PM, Oscar Morante <sp...@gmail.com> wrote:

> Hi there,
> I've been stuck with this for a while and I'm not sure if I'm running into
> a bug or I'm just doing something very wrong.
>
> I have this stripped-down version of my query:
> https://gist.github.com/spacepluk/9ab1e1a0cfec6f0efb298f023f4c805b
>
> The data is just a single file with one record (1.5K).
>
> Without changing anything, explain takes ~1sec on my machine.  But if I
> uncomment the where clause then it runs for a couple of hours until it runs
> out of memory.
>
> Also if I uncomment the where clause *and* take out the join, then it
> takes around 30s to plan.
>
> Any ideas?
> Thanks!
>
>