You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Abhishek Girish (JIRA)" <ji...@apache.org> on 2015/07/27 09:10:05 UTC
[jira] [Comment Edited] (DRILL-3555) Changing defaults for
planner.memory.max_query_memory_per_node causes queries with window
function to fail
[ https://issues.apache.org/jira/browse/DRILL-3555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14642375#comment-14642375 ]
Abhishek Girish edited comment on DRILL-3555 at 7/27/15 7:09 AM:
-----------------------------------------------------------------
TPC-DS SF100 - Parquet.
was (Author: agirish):
Parquet.
> Changing defaults for planner.memory.max_query_memory_per_node causes queries with window function to fail
> ----------------------------------------------------------------------------------------------------------
>
> Key: DRILL-3555
> URL: https://issues.apache.org/jira/browse/DRILL-3555
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Affects Versions: 1.1.0, 1.2.0
> Environment: 4 Nodes. Direct Memory= 48 GB each
> Reporter: Abhishek Girish
> Assignee: Jinfeng Ni
> Priority: Critical
>
> Changing the default value for planner.memory.max_query_memory_per_node from 2 GB to anything higher causes queries with window functions to fail.
> Changed system options
> {code:sql}
> > select * from sys.options where status like '%CHANGE%';
> +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> | name | kind | type | status | num_val | string_val | bool_val | float_val |
> +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> | planner.enable_decimal_data_type | BOOLEAN | SYSTEM | CHANGED | null | null | true | null |
> | planner.memory.max_query_memory_per_node | LONG | SYSTEM | CHANGED | 8589934592 | null | null | null |
> +-------------------------------------------+----------+---------+----------+-------------+-------------+-----------+------------+
> 2 rows selected (0.249 seconds)
> {code}
> Query
> {code:sql}
> > SELECT SUM(ss.ss_net_paid_inc_tax) OVER (PARTITION BY ss.ss_store_sk) FROM store_sales ss LIMIT 20;
> java.lang.RuntimeException: java.sql.SQLException: SYSTEM ERROR: DrillRuntimeException: Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> at sqlline.IncrementalRows.hasNext(IncrementalRows.java:73)
> at sqlline.TableOutputFormat$ResizingRowsProvider.next(TableOutputFormat.java:87)
> at sqlline.TableOutputFormat.print(TableOutputFormat.java:118)
> at sqlline.SqlLine.print(SqlLine.java:1583)
> at sqlline.Commands.execute(Commands.java:852)
> at sqlline.Commands.sql(Commands.java:751)
> at sqlline.SqlLine.dispatch(SqlLine.java:738)
> at sqlline.SqlLine.begin(SqlLine.java:612)
> at sqlline.SqlLine.start(SqlLine.java:366)
> at sqlline.SqlLine.main(SqlLine.java:259)
> {code}
> Log:
> {code}
> 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO o.a.d.e.w.fragment.FragmentExecutor - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State change requested RUNNING --> FINISHED
> 2015-07-23 18:16:52,292 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:2:2] INFO o.a.d.e.w.f.FragmentStatusReporter - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:2:2: State to report: FINISHED
> 2015-07-23 18:17:05,485 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR o.a.d.e.p.i.s.SortRecordBatchBuilder - Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824
> 2015-07-23 18:17:05,486 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested RUNNING --> FAILED
> ...
> 2015-07-23 18:17:05,990 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] INFO o.a.d.e.w.fragment.FragmentExecutor - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0: State change requested FAILED --> FINISHED
> 2015-07-23 18:17:05,999 [2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:frag:1:0] ERROR o.a.d.e.w.fragment.FragmentExecutor - SYSTEM ERROR: DrillRuntimeException: Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: DrillRuntimeException: Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824
> Fragment 1:0
> [Error Id: 9c2ec9cf-21c6-4d5e-b0d6-7cd59e32c49d on abhi1:31010]
> at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:523) ~[drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:323) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:178) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:292) [drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_65]
> at java.lang.Thread.run(Thread.java:745) [na:1.7.0_65]
> Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Adding this batch causes the total size to exceed max allowed size. Current runningBytes 1073638500, Incoming batchBytes 127875. maxBytes 1073741824
> at org.apache.drill.exec.physical.impl.sort.SortRecordBatchBuilder.add(SortRecordBatchBuilder.java:115) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.innerNext(ExternalSortBatch.java:380) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:147) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:105) ~[drill-java-exec-1.2.0-SNAPSHOT.jar:1.2.0-SNAPSHOT]
> ...
> 2015-07-23 18:17:06,021 [BitServer-3] WARN o.a.d.e.w.b.ControlMessageHandler - Dropping request to cancel fragment. 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:0 does not exist.
> 2015-07-23 18:17:06,022 [BitServer-3] INFO o.a.d.e.w.fragment.FragmentExecutor - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4: State change requested RUNNING --> CANCELLATION_REQUESTED
> 2015-07-23 18:17:06,023 [BitServer-3] INFO o.a.d.e.w.f.FragmentStatusReporter - 2a4e6e2e-8cfa-ed8f-de56-e6c5517b5da6:1:4: State to report: CANCELLATION_REQUESTED
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)