You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "salim achouche (JIRA)" <ji...@apache.org> on 2018/07/23 20:47:00 UTC
[jira] [Assigned] (DRILL-6626) Hash Aggregate: Index out of bounds
with small output batch size and spilling
[ https://issues.apache.org/jira/browse/DRILL-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
salim achouche reassigned DRILL-6626:
-------------------------------------
Assignee: salim achouche
> Hash Aggregate: Index out of bounds with small output batch size and spilling
> -----------------------------------------------------------------------------
>
> Key: DRILL-6626
> URL: https://issues.apache.org/jira/browse/DRILL-6626
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 1.14.0
> Reporter: Boaz Ben-Zvi
> Assignee: salim achouche
> Priority: Major
>
> This new IOOB failure was seen while trying to recreate the NPE failure in DRILL-6622 (over TPC-DS SF1). The proposed fix for the latter (PR #1391) does not seem to make a difference.
> This IOOB can easily be created with other large Hash-Agg queries that need to spill.
> The IOOB was caused after restricting the output batch size (to force many), and the Hash Aggr memory (to force a spill):
> {code}
> 0: jdbc:drill:zk=local> alter system set `drill.exec.memory.operator.output_batch_size` = 262144;
> +-------+--------------------------------------------------------+
> | ok | summary |
> +-------+--------------------------------------------------------+
> | true | drill.exec.memory.operator.output_batch_size updated. |
> +-------+--------------------------------------------------------+
> 1 row selected (0.106 seconds)
> 0: jdbc:drill:zk=local>
> 0: jdbc:drill:zk=local> alter session set `exec.errors.verbose` = true;
> +-------+-------------------------------+
> | ok | summary |
> +-------+-------------------------------+
> | true | exec.errors.verbose updated. |
> +-------+-------------------------------+
> 1 row selected (0.081 seconds)
> 0: jdbc:drill:zk=local>
> 0: jdbc:drill:zk=local> alter session set `exec.hashagg.mem_limit` = 16777216;
> +-------+----------------------------------+
> | ok | summary |
> +-------+----------------------------------+
> | true | exec.hashagg.mem_limit updated. |
> +-------+----------------------------------+
> 1 row selected (0.089 seconds)
> 0: jdbc:drill:zk=local>
> 0: jdbc:drill:zk=local> SELECT c_customer_id FROM dfs.`/data/tpcds/sf1/parquet/customer`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT ca_address_id FROM dfs.`/data/tpcds/sf1/parquet/customer_address`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT cd_credit_rating FROM dfs.`/data/tpcds/sf1/parquet/customer_demographics`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT hd_buy_potential FROM dfs.`/data/tpcds/sf1/parquet/household_demographics`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT i_item_id FROM dfs.`/data/tpcds/sf1/parquet/item`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT p_promo_id FROM dfs.`/data/tpcds/sf1/parquet/promotion`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT t_time_id FROM dfs.`/data/tpcds/sf1/parquet/time_dim`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT d_date_id FROM dfs.`/data/tpcds/sf1/parquet/date_dim`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT s_store_id FROM dfs.`/data/tpcds/sf1/parquet/store`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT w_warehouse_id FROM dfs.`/data/tpcds/sf1/parquet/warehouse`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT sm_ship_mode_id FROM dfs.`/data/tpcds/sf1/parquet/ship_mode`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT r_reason_id FROM dfs.`/data/tpcds/sf1/parquet/reason`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT cc_call_center_id FROM dfs.`/data/tpcds/sf1/parquet/call_center`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT web_site_id FROM dfs.`/data/tpcds/sf1/parquet/web_site`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT wp_web_page_id FROM dfs.`/data/tpcds/sf1/parquet/web_page`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT cp_catalog_page_id FROM dfs.`/data/tpcds/sf1/parquet/catalog_page`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: Index: 26474, Size: 7
> Fragment 4:0
> [Error Id: d44e64ea-f474-436e-94b0-61c61eec2227 on 172.30.8.176:31020]
> (java.lang.IndexOutOfBoundsException) Index: 26474, Size: 7
> java.util.ArrayList.rangeCheck():653
> java.util.ArrayList.get():429
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.rehash():293
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.access$1300():120
> org.apache.drill.exec.physical.impl.common.HashTableTemplate.resizeAndRehashIfNeeded():805
> org.apache.drill.exec.physical.impl.common.HashTableTemplate.put():682
> org.apache.drill.exec.physical.impl.aggregate.HashAggTemplate.checkGroupAndAggrValues():1379
> org.apache.drill.exec.physical.impl.aggregate.HashAggTemplate.doWork():604
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():273
> org.apache.drill.exec.record.AbstractRecordBatch.next():172
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():229
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142
> org.apache.drill.exec.record.AbstractRecordBatch.next():172
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():229
> org.apache.drill.exec.physical.impl.BaseRootExec.next():103
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
> org.apache.drill.exec.physical.impl.BaseRootExec.next():93
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():422
> org.apache.hadoop.security.UserGroupInformation.doAs():1657
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():281
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745 (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)