You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Ramana Inukonda Nagaraj (JIRA)" <ji...@apache.org> on 2015/01/13 23:51:34 UTC
[jira] [Commented] (DRILL-2005) Create table fails to write out a
parquet file created from hive- read works fine
[ https://issues.apache.org/jira/browse/DRILL-2005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14276124#comment-14276124 ]
Ramana Inukonda Nagaraj commented on DRILL-2005:
------------------------------------------------
{code}
2015-01-13 14:45:24,129 [2b4a5ffb-ef9c-f7a0-3186-cefd4744ea9a:frag:0:0] WARN o.a.d.e.w.fragment.FragmentExecutor - Error while initializing or executing fragment
java.lang.AssertionError: Attempted to access index 2147483646 when value capacity is 4095
at org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapAccessor.get(RepeatedMapVector.java:541) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:90) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl.reader(SingleMapReaderImpl.java:61) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter$MapParquetConverter.<init>(ParquetRecordWriter.java:220) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter.getNewMapConverter(ParquetRecordWriter.java:210) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.EventBasedRecordWriter.getConverter(EventBasedRecordWriter.java:114) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter$RepeatedMapParquetConverter.<init>(ParquetRecordWriter.java:249) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter.getNewRepeatedMapConverter(ParquetRecordWriter.java:239) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.EventBasedRecordWriter.getConverter(EventBasedRecordWriter.java:116) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter$MapParquetConverter.<init>(ParquetRecordWriter.java:220) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter.getNewMapConverter(ParquetRecordWriter.java:210) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.EventBasedRecordWriter.getConverter(EventBasedRecordWriter.java:114) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.EventBasedRecordWriter.initFieldWriters(EventBasedRecordWriter.java:73) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.EventBasedRecordWriter.<init>(EventBasedRecordWriter.java:46) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.WriterRecordBatch.setupNewSchema(WriterRecordBatch.java:176) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:113) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:97) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:57) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT] at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:114) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:254) [drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
2015-01-13 14:45:24,129 [2b4a5ffb-ef9c-f7a0-3186-cefd4744ea9a:frag:0:0] ERROR o.a.drill.exec.ops.FragmentContext - Fragment Context received failure.
java.lang.AssertionError: Attempted to access index 2147483646 when value capacity is 4095
at org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapAccessor.get(RepeatedMapVector.java:541) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:90) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl.reader(SingleMapReaderImpl.java:61) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter$MapParquetConverter.<init>(ParquetRecordWriter.java:220) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter.getNewMapConverter(ParquetRecordWriter.java:210) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.EventBasedRecordWriter.getConverter(EventBasedRecordWriter.java:114) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter$RepeatedMapParquetConverter.<init>(ParquetRecordWriter.java:249) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter.getNewRepeatedMapConverter(ParquetRecordWriter.java:239) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.EventBasedRecordWriter.getConverter(EventBasedRecordWriter.java:116) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter$MapParquetConverter.<init>(ParquetRecordWriter.java:220) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter.getNewMapConverter(ParquetRecordWriter.java:210) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.EventBasedRecordWriter.getConverter(EventBasedRecordWriter.java:114) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.EventBasedRecordWriter.initFieldWriters(EventBasedRecordWriter.java:73) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.EventBasedRecordWriter.<init>(EventBasedRecordWriter.java:46) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.WriterRecordBatch.setupNewSchema(WriterRecordBatch.java:176) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.WriterRecordBatch.innerNext(WriterRecordBatch.java:113) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:99) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:89) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:132) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:142) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next(IteratorValidatorBatchIterator.java:118) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:67) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:97) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:57) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:114) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.work.WorkManager$RunnableWrapper.run(WorkManager.java:254) [drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]
2015-01-13 14:45:24,130 [2b4a5ffb-ef9c-f7a0-3186-cefd4744ea9a:frag:0:0] ERROR o.a.d.e.w.f.AbstractStatusReporter - Error 7c030695-9dee-4e15-b9a7-7c807dbd3e5f: Failure while running fragment.
java.lang.AssertionError: Attempted to access index 2147483646 when value capacity is 4095
at org.apache.drill.exec.vector.complex.RepeatedMapVector$RepeatedMapAccessor.get(RepeatedMapVector.java:541) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.vector.complex.impl.RepeatedMapReaderImpl.setPosition(RepeatedMapReaderImpl.java:90) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.vector.complex.impl.SingleMapReaderImpl.reader(SingleMapReaderImpl.java:61) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetRecordWriter$MapParquetConverter.<init>(ParquetRecordWriter.java:220) ~[drill-java-exec-0.7.0-SNAPSHOT-rebuffed.jar:0.7.0-SNAPSHOT]
{code}
> Create table fails to write out a parquet file created from hive- read works fine
> ---------------------------------------------------------------------------------
>
> Key: DRILL-2005
> URL: https://issues.apache.org/jira/browse/DRILL-2005
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Writer
> Affects Versions: 0.7.0
> Reporter: Ramana Inukonda Nagaraj
> Assignee: Parth Chandra
> Priority: Critical
> Attachments: hive_alltypes.parquet
>
>
> Created a parquet file in hive having the following DDL
> hive> desc alltypesparquet;
> OK
> c1 int
> c2 boolean
> c3 double
> c4 string
> c5 array<int>
> c6 map<int,string>
> c7 map<string,string>
> c8 struct<r:string,s:int,t:double>
> c9 tinyint
> c10 smallint
> c11 float
> c12 bigint
> c13 array<array<string>>
> c15 struct<r:int,s:struct<a:int,b:string>>
> c16 array<struct<m:map<string,string>,n:int>>
> Time taken: 0.076 seconds, Fetched: 15 row(s)
> Now tried to write the file out using drill
> 0: jdbc:drill:> create table hive_alltypesparquet as select * from `/user/hive/warehouse/alltypesparquet`;
> Query failed: Query failed: Failure while running fragment., Attempted to access index 2147483646 when value capacity is 4095 [ 7c030695-9dee-4e15-b9a7-7c807dbd3e5f on 10.10.30.167:31010 ]
> [ 7c030695-9dee-4e15-b9a7-7c807dbd3e5f on 10.10.30.167:31010 ]
> Drill is able to read the file (reasonably well- see 1997,1999,2000)
> 0: jdbc:drill:> select * from `/user/hive/warehouse/alltypesparquet`;
> +------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
> | c1 | c2 | c3 | c4 | c5 | c6 | c7 | c8 | c9 | c10 | c11 | c12 | c13 | c15 | c16 |
> +------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
> | null | null | null | null | {"bag":[]} | {"map":[]} | {"map":[]} | {} | null | null | null | null | {"bag":[]} | {"s":{}} | {"bag":[]} |
> | -1 | false | -1.1 | [B@62369833 | {"bag":[]} | {"map":[]} | {"map":[]} | {} | -1 | -1 | -1.0 | -1 | {"bag":[]} | {"s":{}} | {"bag":[]} |
> | 1 | true | 1.1 | [B@6e426ea4 | {"bag":[{"array_element":1},{"array_element":2}]} | {"map":[{"key":1,"value":"eA=="},{"key":2,"value":"eQ=="}]} | {"map":[{"key":"aw==","value":"dg=="}]} | {"r":"YQ==","s":9,"t":2.2} | 1 | 1 | 1.0 | 1 | {"bag":[{"array_element":{"bag":[{"array_element":"YQ=="},{"array_element":"Yg=="}]}},{"array_element":{"bag":[{"array_element":"Yw=="},{"array_element":"ZA=="}]}}]} | {"r":1,"s":{"a":2,"b":"eA=="}} | {"bag":[{"array_element":{"m":{"map":[]},"n":1}},{"array_element":{"m":{"map":[{"key":"YQ==","value":"Yg=="},{"key":"Yw==","value":"ZA=="}]},"n":2}}]} |
> +------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+------------+
> 3 rows selected (0.101 seconds)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)