You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by zbdzzg <gi...@git.apache.org> on 2016/09/09 08:29:21 UTC
[GitHub] drill pull request #584: DRILL-4884: Fix bug that drill sometimes produced I...
GitHub user zbdzzg opened a pull request:
https://github.com/apache/drill/pull/584
DRILL-4884: Fix bug that drill sometimes produced IOB exception while querying data of 65536 limitation
Drill produces IOB while using a non batched scanner and limiting SQL by 65536.
SQL:
```
select id from isearch.tmall_auction_cluster limit 1 offset 65535
```
Result:
```
at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:534) ~[classes/:na]
at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:324) [classes/:na]
at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:184) [classes/:na]
at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:290) [classes/:na]
at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [classes/:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
Caused by: java.lang.IndexOutOfBoundsException: index: 131072, length: 2 (expected: range(0, 131072))
at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:175) ~[classes/:4.0.27.Final]
at io.netty.buffer.DrillBuf.chk(DrillBuf.java:197) ~[classes/:4.0.27.Final]
at io.netty.buffer.DrillBuf.setChar(DrillBuf.java:517) ~[classes/:4.0.27.Final]
at org.apache.drill.exec.record.selection.SelectionVector2.setIndex(SelectionVector2.java:79) ~[classes/:na]
at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.limitWithNoSV(LimitRecordBatch.java:167) ~[classes/:na]
at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.doWork(LimitRecordBatch.java:145) ~[classes/:na]
at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93) ~[classes/:na]
at org.apache.drill.exec.physical.impl.limit.LimitRecordBatch.innerNext(LimitRecordBatch.java:115) ~[classes/:na]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[classes/:na]
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215) ~[classes/:na]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[classes/:na]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[classes/:na]
at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[classes/:na]
at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:94) ~[classes/:na]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[classes/:na]
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215) ~[classes/:na]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[classes/:na]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[classes/:na]
at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[classes/:na]
at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132) ~[classes/:na]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[classes/:na]
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:215) ~[classes/:na]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) ~[classes/:na]
at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) ~[classes/:na]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) ~[classes/:na]
at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:256) ~[classes/:na]
at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:250) ~[classes/:na]
at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_101]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_101]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[hadoop-common-2.7.1.jar:na]
at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:250) [classes/:na]
... 4 common frames omitted
```
Code from IteratorValidatorBatchIterator.java said that it is OK incoming returning 65536 records:
```
if (incoming.getRecordCount() > MAX_BATCH_SIZE) { // MAX_BATCH_SIZE == 65536
throw new IllegalStateException(
String.format(
"Incoming batch [#%d, %s] has size %d, which is beyond the"
+ " limit of %d",
instNum, batchTypeName, incoming.getRecordCount(), MAX_BATCH_SIZE
));
}
```
Code from LimitRecordBatch.java said that a loop will not break as expected when the incoming returns 65536 records:
```
private void limitWithNoSV(int recordCount) {
final int offset = Math.max(0, Math.min(recordCount - 1, recordsToSkip));
recordsToSkip -= offset;
int fetch;
if(noEndLimit) {
fetch = recordCount;
} else {
fetch = Math.min(recordCount, offset + recordsLeft);
recordsLeft -= Math.max(0, fetch - offset);
}
int svIndex = 0;
for(char i = (char) offset; i < fetch; svIndex++, i++) { // since fetch==recordCount==65536, param i can be increased from 65535 to 65536, then be casted to char type of value 0, the loop abnormally continues.
outgoingSv.setIndex(svIndex, i);
}
outgoingSv.setRecordCount(svIndex);
}
```
The IllegalStateException should be thrown when incoming returns 65535 records rather than 65536.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zbdzzg/drill master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/584.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #584
----
commit c01f326860fe4d9bed21c3807153d94dc75a7794
Author: hongze.zhz <ho...@alibaba-inc.com>
Date: 2016-09-09T08:19:16Z
DRILL-4884: Fix bug that drill sometimes produced IOB exception while querying data of 65536 limitation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #584: DRILL-4884: Fix bug that drill sometimes produced IOB exce...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on the issue:
https://github.com/apache/drill/pull/584
The fix looks fine to me. But do you plan to add a unit test case? thx.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #584: DRILL-4884: Fix bug that drill sometimes produced IOB exce...
Posted by zbdzzg <gi...@git.apache.org>.
Github user zbdzzg commented on the issue:
https://github.com/apache/drill/pull/584
@dsbos
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #584: DRILL-4884: Fix bug that drill sometimes produced I...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/584
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #584: DRILL-4884: Fix bug that drill sometimes produced IOB exce...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on the issue:
https://github.com/apache/drill/pull/584
+1
@zbdzzg , thanks for the patch!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #584: DRILL-4884: Fix bug that drill sometimes produced I...
Posted by zbdzzg <gi...@git.apache.org>.
Github user zbdzzg commented on a diff in the pull request:
https://github.com/apache/drill/pull/584#discussion_r84828174
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java ---
@@ -301,7 +301,7 @@ public IterOutcome next() {
"Incoming batch [#%d, %s] has an empty schema. This is not allowed.",
instNum, batchTypeName));
}
- if (incoming.getRecordCount() > MAX_BATCH_SIZE) {
+ if (incoming.getRecordCount() >= MAX_BATCH_SIZE) {
--- End diff --
Thanks for reply.
I tried turn assertion off and rerun, the IOB is produced again. The IllegalStateException is thrown when turning it on.
The built-in storage plug-ins always have smaller batch limitation (smaller than 65536) in reader implementations, so this problem is supposed to be avoided by design. The scene to us running into this problem is that we have a customized storage plug-in with a large batch-size set in the reader part. I personally guess that the assertion is in technically enabled in testing / debugging for a new written storage plug-in, so the IllegalStateException should be thrown in which the assertion is on, then we know that our batch-size is too large to execute all stuffs. As a comparison, a wired IOB exception makes us confused.
If we consider the scene disabling assertion, another bound check should be inserted to LimitRecordBatch.java, but what I am seeing is that most check logics is written at IteratorValidator, I suppose that author wants storage developers make all things work in the early debugging, the assertion can be off after the exceptions like this IOB are resolved.
So I don t think that making any changes to LimitRecordBatch.java is reasonable rather than Doing same thing to validator part. This change benefits the plugin development. Do you agree?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill issue #584: DRILL-4884: Fix bug that drill sometimes produced IOB exce...
Posted by zbdzzg <gi...@git.apache.org>.
Github user zbdzzg commented on the issue:
https://github.com/apache/drill/pull/584
@jinfengni A test case has been added, See 2c67dd78e463514393fd33779b64fa88c7c22ebd
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #584: DRILL-4884: Fix bug that drill sometimes produced I...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/584#discussion_r84801937
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java ---
@@ -301,7 +301,7 @@ public IterOutcome next() {
"Incoming batch [#%d, %s] has an empty schema. This is not allowed.",
instNum, batchTypeName));
}
- if (incoming.getRecordCount() > MAX_BATCH_SIZE) {
+ if (incoming.getRecordCount() >= MAX_BATCH_SIZE) {
--- End diff --
I'm not sure if this is the right fix for this IOB problem.
1. IteratorValidator is only inserted when assertion is enabled [1]. Fixing only in IteratorValidatorBatchIterator will mean that the issue will be still there if assertion is disabled.
2. Even we do turn on assertion, will the query hit IllegalStateException, in stead of IOB?
Can you try run the query when assertion is off / on, and see if the query is successful in both cases?
[1] https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/ImplCreator.java#L72-L74
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #584: DRILL-4884: Fix bug that drill sometimes produced I...
Posted by zbdzzg <gi...@git.apache.org>.
Github user zbdzzg commented on a diff in the pull request:
https://github.com/apache/drill/pull/584#discussion_r84835453
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java ---
@@ -301,7 +301,7 @@ public IterOutcome next() {
"Incoming batch [#%d, %s] has an empty schema. This is not allowed.",
instNum, batchTypeName));
}
- if (incoming.getRecordCount() > MAX_BATCH_SIZE) {
+ if (incoming.getRecordCount() >= MAX_BATCH_SIZE) {
--- End diff --
Pretty clear, see commit ccab954.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request #584: DRILL-4884: Fix bug that drill sometimes produced I...
Posted by jinfengni <gi...@git.apache.org>.
Github user jinfengni commented on a diff in the pull request:
https://github.com/apache/drill/pull/584#discussion_r84831079
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/validate/IteratorValidatorBatchIterator.java ---
@@ -301,7 +301,7 @@ public IterOutcome next() {
"Incoming batch [#%d, %s] has an empty schema. This is not allowed.",
instNum, batchTypeName));
}
- if (incoming.getRecordCount() > MAX_BATCH_SIZE) {
+ if (incoming.getRecordCount() >= MAX_BATCH_SIZE) {
--- End diff --
Drill requires that batch with no selection vector(SV), and batch with SV2 is bounded by 65536. This requirement is valid across the entire Drill code base. What this IteratorVAlidator tries to enforce is to make sure every incoming batch meet this requirement, when assertion is enabled. However, it's each operator's responsibility to enforce this. For instance, as you saw, each reader in Drill should produce a batch no larger than 65536. If you develop a new storage plugin with a new reader, then the new reader should enforce this rule as well.
Therefore, in your situation where you develop a new reader, the right approach is that you need make sure reader produces batch no larger than this threshold.
The original code IteratorValidatorBatchIterator.java should be fine. For the repo I tried, I feel the fix should be in LimitRecordBatch.java. As you indicated earlier, the index "i" is defined as char, which is not right.
Would you like to modify your patch ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---