You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Aman Sinha (JIRA)" <ji...@apache.org> on 2014/06/06 01:09:02 UTC
[jira] [Comment Edited] (DRILL-918) Merge join results in a
java.lang.IllegalArgumentException during sort
[ https://issues.apache.org/jira/browse/DRILL-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14019391#comment-14019391 ]
Aman Sinha edited comment on DRILL-918 at 6/5/14 11:08 PM:
-----------------------------------------------------------
The IllegalArgumentException is actually preceded by an AssertionError (this can be see if assertions are enabled in the JVM arguments).
Exception in thread "WorkManager-6" Exception in thread "WorkManager-5" java.lang.AssertionError
at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.next(ExternalSortBatch.java:221)
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111)
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63)
at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47)
at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:91)
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111)
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
at org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft(JoinStatus.java:70)
at org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:79)
at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.next(MergeJoinBatch.java:138)
The underlying assert is line 221 in ExternalSortBatch.java: assert sv2.getCount() > 0;
So we are getting an empty batch as input to the sort. I believe this started happening after we enabled the broadcast join. The left child of the join is not distributed (previously it was getting hash distributed) and one of the fragments encounters an empty batch.
was (Author: amansinha100):
The IllegalArgumentException is actually preceded by an AssertionError (this can be see if assertions are enabled in the jam arguments).
Exception in thread "WorkManager-6" Exception in thread "WorkManager-5" java.lang.AssertionError
at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.next(ExternalSortBatch.java:221)
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111)
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63)
at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47)
at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:91)
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:111)
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69)
at org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft(JoinStatus.java:70)
at org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:79)
at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.next(MergeJoinBatch.java:138)
The underlying assert is line 221 in ExternalSortBatch.java: assert sv2.getCount() > 0;
So we are getting an empty batch as input to the sort. I believe this started happening after we enabled the broadcast join. The left child of the join is not distributed (previously it was getting hash distributed) and one of the fragments encounters an empty batch.
> Merge join results in a java.lang.IllegalArgumentException during sort
> ----------------------------------------------------------------------
>
> Key: DRILL-918
> URL: https://issues.apache.org/jira/browse/DRILL-918
> Project: Apache Drill
> Issue Type: Bug
> Components: Query Planning & Optimization
> Reporter: Ramana Inukonda Nagaraj
> Assignee: Aman Sinha
>
> This is one of the smoke tests which was passing earlier.
> The testcase looks like this
> alter session set `planner.enable_hashjoin` = false;
> select count(*)
> from (select l.l_orderkey as x, c.c_custkey as y
> from lineitem l
> left outer join customer c
> on l.l_orderkey = c.c_custkey) as foo
> where x < 10000;
> And there is an error during the sort process(presumably because merge join is used)
> java.lang.IllegalArgumentException: Undefined for 0
> at org.apache.hadoop.util.QuickSort.getMaxDepth(QuickSort.java:41) ~[hadoop-0.20.2-dev-core.jar:na]
> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59) ~[hadoop-0.20.2-dev-core.jar:na]
> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:51) ~[hadoop-0.20.2-dev-core.jar:na]
> at org.apache.drill.exec.test.generated.SingleBatchSorterGen50.sort(SingleBatchSorterTemplate.java:51) ~[na:na]
> at org.apache.drill.exec.physical.impl.xsort.ExternalSortBatch.next(ExternalSortBatch.java:225) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.next(RemovingRecordBatch.java:91) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.join.JoinStatus.nextLeft(JoinStatus.java:70) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.join.JoinStatus.ensureInitial(JoinStatus.java:79) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.join.MergeJoinBatch.next(MergeJoinBatch.java:138) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractSingleRecordBatch.next(AbstractSingleRecordBatch.java:47) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.next(ProjectRecordBatch.java:83) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:69) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:63) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.aggregate.StreamingAggBatch.next(StreamingAggBatch.java:86) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.next(SingleSenderCreator.java:74) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:98) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_4
--
This message was sent by Atlassian JIRA
(v6.2#6252)