You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by Anup Tiwari <an...@games24x7.com> on 2017/04/18 13:14:04 UTC
[Drill 1.10.0] : Memory was leaked by query
Hi Team,
Please find following information :
*Cluster configuration :*
Number of Nodes : 5
Cores/Node : 8
RAM : 32
*Variable values :*
planner.width.max_per_node = 5
planner.width.max_per_query = 30
planner.memory.max_query_memory_per_node = 4294967296
I am getting following error on simple select statement which is coming 6
times out of 10 times, let me know if i am missing anything:
*Query :*
select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl
like '%/j_check%' and ajaxResponse like '%success%true%') limit 5;
*Error :*
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING
--> CANCELLATION_REQUESTED
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.f.FragmentStatusReporter -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report:
CANCELLATION_REQUESTED
2017-04-18 18:21:54,173 [BitServer-4] WARN
o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist.
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested
CANCELLATION_REQUESTED --> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FINISHED
2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
Regards,
*Anup Tiwari*
Re: [Drill 1.10.0] : Memory was leaked by query
Posted by Anup Tiwari <an...@games24x7.com>.
Thanks Padma, it worked.
Regards,
*Anup Tiwari*
On Wed, Apr 19, 2017 at 1:13 AM, Kunal Khatua <kk...@mapr.com> wrote:
> Could you also share the profiles for the failed queries as well?
>
>
> Thanks
>
> Kunal
>
> ________________________________
> From: Padma Penumarthy <pp...@mapr.com>
> Sent: Tuesday, April 18, 2017 7:18:08 AM
> To: user@drill.apache.org
> Cc: dev@drill.apache.org
> Subject: Re: [Drill 1.10.0] : Memory was leaked by query
>
> Seems like you are running into DRILL-5435<https://issues.apac
> he.org/jira/browse/DRILL-5435>.
> Try turning off async parquet reader and see if that helps.
> alter session set `store.parquet.reader.pagereader.async`=false;
>
> Thanks,
> Padma
>
>
> On Apr 18, 2017, at 6:14 AM, Anup Tiwari <anup.tiwari@games24x7.com<mai
> lto:anup.tiwari@games24x7.com>> wrote:
>
> Hi Team,
>
> Please find following information :
>
> *Cluster configuration :*
> Number of Nodes : 5
> Cores/Node : 8
> RAM : 32
>
> *Variable values :*
> planner.width.max_per_node = 5
> planner.width.max_per_query = 30
> planner.memory.max_query_memory_per_node = 4294967296
>
> I am getting following error on simple select statement which is coming 6
> times out of 10 times, let me know if i am missing anything:
>
> *Query :*
> select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl
> like '%/j_check%' and ajaxResponse like '%success%true%') limit 5;
>
> *Error :*
>
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
> Allocator(op:1:24:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
>
> Fragment 1:24
>
> [Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
> prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo
> p-101.bom-prod.aws.games24x7.com>:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
> Allocator(op:1:24:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
>
> Fragment 1:24
>
> [Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
> prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo
> p-101.bom-prod.aws.games24x7.com>:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.
> build(UserException.java:544)
> ~[drill-common-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFin
> alState(FragmentExecutor.java:293)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup
> (FragmentExecutor.java:160)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(Fra
> gmentExecutor.java:262)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleanin
> gRunnable.java:38)
> [drill-common-1.10.0.jar:1.10.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> [na:1.8.0_72]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> [na:1.8.0_72]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(op:1:24:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[drill-memory-base-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.OperatorContextImpl.close(Operator
> ContextImpl.java:149)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.FragmentContext.suppressingClose(F
> ragmentContext.java:422)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOu
> tResources(FragmentExecutor.java:318)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup
> (FragmentExecutor.java:155)
> [drill-java-exec-1.10.0.jar:1.10.0]
> ... 5 common frames omitted
> 2017-04-18 18:21:54,172 [BitServer-4] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING
> --> CANCELLATION_REQUESTED
> 2017-04-18 18:21:54,172 [BitServer-4] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report:
> CANCELLATION_REQUESTED
> 2017-04-18 18:21:54,173 [BitServer-4] WARN
> o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist.
> 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
> INFO o.a.d.e.w.fragment.FragmentExecutor -
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested
> CANCELLATION_REQUESTED --> FAILED
> 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
> INFO o.a.d.e.w.fragment.FragmentExecutor -
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
> --> FAILED
> 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
> INFO o.a.d.e.w.fragment.FragmentExecutor -
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
> --> FINISHED
> 2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
> Allocator(op:1:21:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
>
> Fragment 1:21
>
> [Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
> prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo
> p-101.bom-prod.aws.games24x7.com>:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
> Allocator(op:1:21:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
>
> Fragment 1:21
>
> [Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
> prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo
> p-101.bom-prod.aws.games24x7.com>:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.
> build(UserException.java:544)
> ~[drill-common-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFin
> alState(FragmentExecutor.java:293)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup
> (FragmentExecutor.java:160)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(Fra
> gmentExecutor.java:262)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleanin
> gRunnable.java:38)
> [drill-common-1.10.0.jar:1.10.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> [na:1.8.0_72]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> [na:1.8.0_72]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(op:1:21:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[drill-memory-base-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.OperatorContextImpl.close(Operator
> ContextImpl.java:149)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.FragmentContext.suppressingClose(F
> ragmentContext.java:422)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOu
> tResources(FragmentExecutor.java:318)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup
> (FragmentExecutor.java:155)
> [drill-java-exec-1.10.0.jar:1.10.0]
> ... 5 common frames omitted
>
>
> Regards,
> *Anup Tiwari*
>
>
Re: [Drill 1.10.0] : Memory was leaked by query
Posted by Anup Tiwari <an...@games24x7.com>.
Thanks Padma, it worked.
Regards,
*Anup Tiwari*
On Wed, Apr 19, 2017 at 1:13 AM, Kunal Khatua <kk...@mapr.com> wrote:
> Could you also share the profiles for the failed queries as well?
>
>
> Thanks
>
> Kunal
>
> ________________________________
> From: Padma Penumarthy <pp...@mapr.com>
> Sent: Tuesday, April 18, 2017 7:18:08 AM
> To: user@drill.apache.org
> Cc: dev@drill.apache.org
> Subject: Re: [Drill 1.10.0] : Memory was leaked by query
>
> Seems like you are running into DRILL-5435<https://issues.apac
> he.org/jira/browse/DRILL-5435>.
> Try turning off async parquet reader and see if that helps.
> alter session set `store.parquet.reader.pagereader.async`=false;
>
> Thanks,
> Padma
>
>
> On Apr 18, 2017, at 6:14 AM, Anup Tiwari <anup.tiwari@games24x7.com<mai
> lto:anup.tiwari@games24x7.com>> wrote:
>
> Hi Team,
>
> Please find following information :
>
> *Cluster configuration :*
> Number of Nodes : 5
> Cores/Node : 8
> RAM : 32
>
> *Variable values :*
> planner.width.max_per_node = 5
> planner.width.max_per_query = 30
> planner.memory.max_query_memory_per_node = 4294967296
>
> I am getting following error on simple select statement which is coming 6
> times out of 10 times, let me know if i am missing anything:
>
> *Query :*
> select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl
> like '%/j_check%' and ajaxResponse like '%success%true%') limit 5;
>
> *Error :*
>
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
> Allocator(op:1:24:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
>
> Fragment 1:24
>
> [Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
> prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo
> p-101.bom-prod.aws.games24x7.com>:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
> Allocator(op:1:24:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
>
> Fragment 1:24
>
> [Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
> prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo
> p-101.bom-prod.aws.games24x7.com>:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.
> build(UserException.java:544)
> ~[drill-common-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFin
> alState(FragmentExecutor.java:293)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup
> (FragmentExecutor.java:160)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(Fra
> gmentExecutor.java:262)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleanin
> gRunnable.java:38)
> [drill-common-1.10.0.jar:1.10.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> [na:1.8.0_72]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> [na:1.8.0_72]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(op:1:24:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[drill-memory-base-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.OperatorContextImpl.close(Operator
> ContextImpl.java:149)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.FragmentContext.suppressingClose(F
> ragmentContext.java:422)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOu
> tResources(FragmentExecutor.java:318)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup
> (FragmentExecutor.java:155)
> [drill-java-exec-1.10.0.jar:1.10.0]
> ... 5 common frames omitted
> 2017-04-18 18:21:54,172 [BitServer-4] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING
> --> CANCELLATION_REQUESTED
> 2017-04-18 18:21:54,172 [BitServer-4] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report:
> CANCELLATION_REQUESTED
> 2017-04-18 18:21:54,173 [BitServer-4] WARN
> o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist.
> 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
> INFO o.a.d.e.w.fragment.FragmentExecutor -
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested
> CANCELLATION_REQUESTED --> FAILED
> 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
> INFO o.a.d.e.w.fragment.FragmentExecutor -
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
> --> FAILED
> 2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
> INFO o.a.d.e.w.fragment.FragmentExecutor -
> 2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
> --> FINISHED
> 2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
> ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
> Allocator(op:1:21:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
>
> Fragment 1:21
>
> [Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
> prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo
> p-101.bom-prod.aws.games24x7.com>:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
> IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
> Allocator(op:1:21:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
>
> Fragment 1:21
>
> [Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
> prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoo
> p-101.bom-prod.aws.games24x7.com>:31010]
> at
> org.apache.drill.common.exceptions.UserException$Builder.
> build(UserException.java:544)
> ~[drill-common-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.sendFin
> alState(FragmentExecutor.java:293)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup
> (FragmentExecutor.java:160)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.run(Fra
> gmentExecutor.java:262)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.common.SelfCleaningRunnable.run(SelfCleanin
> gRunnable.java:38)
> [drill-common-1.10.0.jar:1.10.0]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> [na:1.8.0_72]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> [na:1.8.0_72]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
> Caused by: java.lang.IllegalStateException: Memory was leaked by query.
> Memory leaked: (1048576)
> Allocator(op:1:21:6:ParquetRowGroupScan)
> 1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
>
> at
> org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
> ~[drill-memory-base-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.OperatorContextImpl.close(Operator
> ContextImpl.java:149)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.FragmentContext.suppressingClose(F
> ragmentContext.java:422)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
> ~[drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.closeOu
> tResources(FragmentExecutor.java:318)
> [drill-java-exec-1.10.0.jar:1.10.0]
> at
> org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup
> (FragmentExecutor.java:155)
> [drill-java-exec-1.10.0.jar:1.10.0]
> ... 5 common frames omitted
>
>
> Regards,
> *Anup Tiwari*
>
>
Re: [Drill 1.10.0] : Memory was leaked by query
Posted by Kunal Khatua <kk...@mapr.com>.
Could you also share the profiles for the failed queries as well?
Thanks
Kunal
________________________________
From: Padma Penumarthy <pp...@mapr.com>
Sent: Tuesday, April 18, 2017 7:18:08 AM
To: user@drill.apache.org
Cc: dev@drill.apache.org
Subject: Re: [Drill 1.10.0] : Memory was leaked by query
Seems like you are running into DRILL-5435<https://issues.apache.org/jira/browse/DRILL-5435>.
Try turning off async parquet reader and see if that helps.
alter session set `store.parquet.reader.pagereader.async`=false;
Thanks,
Padma
On Apr 18, 2017, at 6:14 AM, Anup Tiwari <an...@games24x7.com>> wrote:
Hi Team,
Please find following information :
*Cluster configuration :*
Number of Nodes : 5
Cores/Node : 8
RAM : 32
*Variable values :*
planner.width.max_per_node = 5
planner.width.max_per_query = 30
planner.memory.max_query_memory_per_node = 4294967296
I am getting following error on simple select statement which is coming 6
times out of 10 times, let me know if i am missing anything:
*Query :*
select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl
like '%/j_check%' and ajaxResponse like '%success%true%') limit 5;
*Error :*
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING
--> CANCELLATION_REQUESTED
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.f.FragmentStatusReporter -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report:
CANCELLATION_REQUESTED
2017-04-18 18:21:54,173 [BitServer-4] WARN
o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist.
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested
CANCELLATION_REQUESTED --> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FINISHED
2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
Regards,
*Anup Tiwari*
Re: [Drill 1.10.0] : Memory was leaked by query
Posted by Kunal Khatua <kk...@mapr.com>.
Could you also share the profiles for the failed queries as well?
Thanks
Kunal
________________________________
From: Padma Penumarthy <pp...@mapr.com>
Sent: Tuesday, April 18, 2017 7:18:08 AM
To: user@drill.apache.org
Cc: dev@drill.apache.org
Subject: Re: [Drill 1.10.0] : Memory was leaked by query
Seems like you are running into DRILL-5435<https://issues.apache.org/jira/browse/DRILL-5435>.
Try turning off async parquet reader and see if that helps.
alter session set `store.parquet.reader.pagereader.async`=false;
Thanks,
Padma
On Apr 18, 2017, at 6:14 AM, Anup Tiwari <an...@games24x7.com>> wrote:
Hi Team,
Please find following information :
*Cluster configuration :*
Number of Nodes : 5
Cores/Node : 8
RAM : 32
*Variable values :*
planner.width.max_per_node = 5
planner.width.max_per_query = 30
planner.memory.max_query_memory_per_node = 4294967296
I am getting following error on simple select statement which is coming 6
times out of 10 times, let me know if i am missing anything:
*Query :*
select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl
like '%/j_check%' and ajaxResponse like '%success%true%') limit 5;
*Error :*
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING
--> CANCELLATION_REQUESTED
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.f.FragmentStatusReporter -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report:
CANCELLATION_REQUESTED
2017-04-18 18:21:54,173 [BitServer-4] WARN
o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist.
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested
CANCELLATION_REQUESTED --> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FINISHED
2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
Regards,
*Anup Tiwari*
Re: [Drill 1.10.0] : Memory was leaked by query
Posted by Padma Penumarthy <pp...@mapr.com>.
Seems like you are running into DRILL-5435<https://issues.apache.org/jira/browse/DRILL-5435>.
Try turning off async parquet reader and see if that helps.
alter session set `store.parquet.reader.pagereader.async`=false;
Thanks,
Padma
On Apr 18, 2017, at 6:14 AM, Anup Tiwari <an...@games24x7.com>> wrote:
Hi Team,
Please find following information :
*Cluster configuration :*
Number of Nodes : 5
Cores/Node : 8
RAM : 32
*Variable values :*
planner.width.max_per_node = 5
planner.width.max_per_query = 30
planner.memory.max_query_memory_per_node = 4294967296
I am getting following error on simple select statement which is coming 6
times out of 10 times, let me know if i am missing anything:
*Query :*
select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl
like '%/j_check%' and ajaxResponse like '%success%true%') limit 5;
*Error :*
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING
--> CANCELLATION_REQUESTED
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.f.FragmentStatusReporter -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report:
CANCELLATION_REQUESTED
2017-04-18 18:21:54,173 [BitServer-4] WARN
o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist.
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested
CANCELLATION_REQUESTED --> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FINISHED
2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
Regards,
*Anup Tiwari*
Re: [Drill 1.10.0] : Memory was leaked by query
Posted by Padma Penumarthy <pp...@mapr.com>.
Seems like you are running into DRILL-5435<https://issues.apache.org/jira/browse/DRILL-5435>.
Try turning off async parquet reader and see if that helps.
alter session set `store.parquet.reader.pagereader.async`=false;
Thanks,
Padma
On Apr 18, 2017, at 6:14 AM, Anup Tiwari <an...@games24x7.com>> wrote:
Hi Team,
Please find following information :
*Cluster configuration :*
Number of Nodes : 5
Cores/Node : 8
RAM : 32
*Variable values :*
planner.width.max_per_node = 5
planner.width.max_per_query = 30
planner.memory.max_query_memory_per_node = 4294967296
I am getting following error on simple select statement which is coming 6
times out of 10 times, let me know if i am missing anything:
*Query :*
select udf_channel,uid from dfs.tmp.tt1 where (event = 'ajax' and ajaxurl
like '%/j_check%' and ajaxResponse like '%success%true%') limit 5;
*Error :*
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:24
[Error Id: a54cc1bf-794a-4143-bd82-0dd5fa3c8f52 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:24:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested RUNNING
--> CANCELLATION_REQUESTED
2017-04-18 18:21:54,172 [BitServer-4] INFO
o.a.d.e.w.f.FragmentStatusReporter -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State to report:
CANCELLATION_REQUESTED
2017-04-18 18:21:54,173 [BitServer-4] WARN
o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment.
2709f415-c08a-13b9-9f05-fcf9008c484f:1:24 does not exist.
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested
CANCELLATION_REQUESTED --> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FAILED
2017-04-18 18:21:54,229 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
INFO o.a.d.e.w.fragment.FragmentExecutor -
2709f415-c08a-13b9-9f05-fcf9008c484f:1:21: State change requested FAILED
--> FINISHED
2017-04-18 18:21:54,230 [2709f415-c08a-13b9-9f05-fcf9008c484f:frag:1:21]
ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
org.apache.drill.common.exceptions.UserException: SYSTEM ERROR:
IllegalStateException: Memory was leaked by query. Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
Fragment 1:21
[Error Id: 8b3bb6e8-77a0-4747-8602-43b40b349354 on
prod-hadoop-101.bom-prod.aws.games24x7.com<http://prod-hadoop-101.bom-prod.aws.games24x7.com>:31010]
at
org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:544)
~[drill-common-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:293)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:262)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
[drill-common-1.10.0.jar:1.10.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_72]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_72]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
Caused by: java.lang.IllegalStateException: Memory was leaked by query.
Memory leaked: (1048576)
Allocator(op:1:21:6:ParquetRowGroupScan)
1000000/1048576/27140096/10000000000 (res/actual/peak/limit)
at
org.apache.drill.exec.memory.BaseAllocator.close(BaseAllocator.java:502)
~[drill-memory-base-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.OperatorContextImpl.close(OperatorContextImpl.java:149)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.suppressingClose(FragmentContext.java:422)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.ops.FragmentContext.close(FragmentContext.java:411)
~[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.closeOutResources(FragmentExecutor.java:318)
[drill-java-exec-1.10.0.jar:1.10.0]
at
org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:155)
[drill-java-exec-1.10.0.jar:1.10.0]
... 5 common frames omitted
Regards,
*Anup Tiwari*