You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Parth Chandra (JIRA)" <ji...@apache.org> on 2015/04/01 20:45:52 UTC
[jira] [Resolved] (DRILL-1962) accessing nested array from multiple
files causing IndexOutOfBoundException
[ https://issues.apache.org/jira/browse/DRILL-1962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Parth Chandra resolved DRILL-1962.
----------------------------------
Resolution: Fixed
fixed by DRILL-2375. Unit tests added in fb4533f
> accessing nested array from multiple files causing IndexOutOfBoundException
> ---------------------------------------------------------------------------
>
> Key: DRILL-1962
> URL: https://issues.apache.org/jira/browse/DRILL-1962
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Data Types
> Affects Versions: 0.8.0
> Reporter: Chun Chang
> Assignee: Parth Chandra
> Priority: Critical
> Fix For: 0.9.0
>
>
> #Wed Jan 07 18:54:07 EST 2015
> git.commit.id.abbrev=35a350f
> If the dataset contains nested array of array, and the data is contained in more than one file, accessing the second nested array cause IndexOutOfBoundsException. For example, with the following dataset:
> {code}
> {
> "id": 2,
> "oooa": {
> "oa": {
> "oab": {
> "oabc": [
> {
> "rowId": 2
> },
> {
> "rowValue1": [{"rv1":1, "rv2":2}, {"rva1":3, "rva2":4}],
> "rowValue2": [{"rw1":1, "rw2":2}, {"rwa1":3, "rwa2":4}]
> }
> ]
> }
> }
> }
> }
> {code}
> If you put it in two separate files in the same directory, query using wild card to accessing the two files at the second array level will cause the exception.
> {code}
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select t.oooa.oa.oab.oabc[1].rowValue1 from `jira2file/jira*.json` t;
> +------------+
> | EXPR$0 |
> +------------+
> | [{"rv1":1,"rv2":2},{"rva1":3,"rva2":4}] |
> Query failed: Query failed: Failure while running fragment., index: -4, length: 4 (expected: range(0, 16384)) [ 78235243-4f01-4ee3-9675-fc18bd1e66e3 on qa-node120.qa.lab:31010 ]
> [ 78235243-4f01-4ee3-9675-fc18bd1e66e3 on qa-node120.qa.lab:31010 ]
> java.lang.RuntimeException: java.sql.SQLException: Failure while executing query.
> at sqlline.SqlLine$IncrementalRows.hasNext(SqlLine.java:2514)
> at sqlline.SqlLine$TableOutputFormat.print(SqlLine.java:2148)
> at sqlline.SqlLine.print(SqlLine.java:1809)
> at sqlline.SqlLine$Commands.execute(SqlLine.java:3766)
> at sqlline.SqlLine$Commands.sql(SqlLine.java:3663)
> at sqlline.SqlLine.dispatch(SqlLine.java:889)
> at sqlline.SqlLine.begin(SqlLine.java:763)
> at sqlline.SqlLine.start(SqlLine.java:498)
> at sqlline.SqlLine.main(SqlLine.java:460)
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ>
> {code}
> Same query on single file works.
> {code}
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> select t.oooa.oa.oab.oabc[1].rowValue1 from `jira2file/jira1.json` t;
> +------------+
> | EXPR$0 |
> +------------+
> | [{"rv1":1,"rv2":2},{"rva1":3,"rva2":4}] |
> +------------+
> {code}
> stack trace:
> {code}
> 2015-01-08 14:00:16,127 [2b51020e-daab-a903-ef7c-ef6f9bd606c7:frag:0:0] WARN o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing fragment
> java.lang.IndexOutOfBoundsException: index: -4, length: 4 (expected: range(0, 16384))
> at io.netty.buffer.DrillBuf.checkIndexD(DrillBuf.java:156) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
> at io.netty.buffer.DrillBuf.chk(DrillBuf.java:178) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
> at io.netty.buffer.DrillBuf.getInt(DrillBuf.java:447) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:4.0.24.Final]
> at org.apache.drill.exec.vector.UInt4Vector$Accessor.get(UInt4Vector.java:297) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapAccessor.get(RepeatedMapVector.java:542) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:90) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setChildrenPosition(RepeatedMapReaderImpl.java:45) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:96) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.test.generated.ProjectorGen4732.doEval(ProjectorTemplate.java:30) ~[na:na]
> at org.apache.drill.exec.test.generated.ProjectorGen4732.projectRecords(ProjectorTemplate.java:64) ~[na:na]
> at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.doWork(ProjectRecordBatch.java:172) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:93) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:97) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:57) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:114) ~[drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:254) [drill-java-exec-0.8.0-SNAPSHOT-rebuffed.jar:0.8.0-SNAPSHOT]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
> at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
> {code}
> physical plan:
> {code}
> 0: jdbc:drill:schema=dfs.drillTestDirComplexJ> explain plan for select t.oooa.oa.oab.oabc[1].rowValue1 from `jira2file/jira*.json` t;
> +------------+------------+
> | text | json |
> +------------+------------+
> | 00-00 Screen
> 00-01 Project(EXPR$0=[ITEM(ITEM(ITEM(ITEM(ITEM($0, 'oa'), 'oab'), 'oabc'), 1), 'rowValue1')])
> 00-02 Scan(groupscan=[EasyGroupScan [selectionRoot=/drill/testdata/complex_type/json/jira2file, numFiles=2, columns=[`oooa`.`oa`.`oab`.`oabc`[1].`rowValue1`], files=[maprfs:/drill/testdata/complex_type/json/jira2file/jira1.json, maprfs:/drill/testdata/complex_type/json/jira2file/jira2.json]]])
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)