You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@drill.apache.org by "Robert Hou (JIRA)" <ji...@apache.org> on 2018/08/13 23:05:00 UTC
[jira] [Updated] (DRILL-6685) Error in parquet record reader
[ https://issues.apache.org/jira/browse/DRILL-6685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Hou updated DRILL-6685:
------------------------------
Description:
This is the query:
alter session set `drill.exec.memory.operator.project.output_batch_size` = 131072;
alter session set `planner.width.max_per_node` = 1;
alter session set `planner.width.max_per_query` = 1;
select * from (
select BigIntValue, BooleanValue, DateValue, FloatValue, DoubleValue, IntegerValue, TimeValue, TimestampValue, IntervalYearValue, IntervalDayValue, IntervalSecondValue, VarbinaryValue1, VarcharValue1, VarbinaryValue2, VarcharValue2 from (select * from dfs.`/drill/testdata/batch_memory/fourvarchar_asc_nulls_16MB_1GB.parquet` order by IntegerValue)) d where d.VarcharValue1 = 'Fq';
It appears to be caused by this commit:
DRILL-6570: Fixed IndexOutofBoundException in Parquet Reader
aee899c1b26ebb9a5781d280d5a73b42c273d4d5
This is the stack trace:
{noformat}
oadd.org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR: Error in parquet record reader.^M
Message: ^M
Hadoop path: /drill/testdata/batch_memory/fourvarchar_asc_nulls_16MB_1GB.parquet/0_0_0.parquet^M
Total records read: 0^M
Row group index: 0^M
Records in row group: 14565^M
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {^M
optional int64 Index;^M
optional binary VarbinaryValue1;^M
optional int64 BigIntValue;^M
optional boolean BooleanValue;^M
optional int32 DateValue (DATE);^M
optional float FloatValue;^M
optional binary VarcharValue1 (UTF8);^M
optional double DoubleValue;^M
optional int32 IntegerValue;^M
optional int32 TimeValue (TIME_MILLIS);^M
optional int64 TimestampValue (TIMESTAMP_MILLIS);^M
optional binary VarbinaryValue2;^M
optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL);^M
optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL);^M
optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL);^M
optional binary VarcharValue2 (UTF8);^M
}^M
, metadata: {drill-writer.version=2, drill.version=1.14.0-SNAPSHOT}}, blocks: [BlockMetaData{14565, 268477520 [ColumnMetaData{UNCOMPRESSED [Index] optional int64 Index [PLAIN, RLE, BIT_PACKED], 4}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue1] optional binary VarbinaryValue1 [PLAIN, RLE, BIT_PACKED], 116579}, ColumnMetaData{UNCOMPRESSED [BigIntValue] optional int64 BigIntValue [PLAIN, RLE, BIT_PACKED], 91098467}, ColumnMetaData{UNCOMPRESSED [BooleanValue] optional boolean BooleanValue [PLAIN, RLE, BIT_PACKED], 91155431}, ColumnMetaData{UNCOMPRESSED [DateValue] optional int32 DateValue (DATE) [PLAIN, RLE, BIT_PACKED], 91157291}, ColumnMetaData{UNCOMPRESSED [FloatValue] optional float FloatValue [PLAIN, RLE, BIT_PACKED], 91215598}, ColumnMetaData{UNCOMPRESSED [VarcharValue1] optional binary VarcharValue1 (UTF8) [PLAIN, RLE, BIT_PACKED], 91273905}, ColumnMetaData{UNCOMPRESSED [DoubleValue] optional double DoubleValue [PLAIN, RLE, BIT_PACKED], 114039039}, ColumnMetaData{UNCOMPRESSED [IntegerValue] optional int32 IntegerValue [PLAIN, RLE, BIT_PACKED], 114155614}, ColumnMetaData{UNCOMPRESSED [TimeValue] optional int32 TimeValue (TIME_MILLIS) [PLAIN, RLE, BIT_PACKED], 114213921}, ColumnMetaData{UNCOMPRESSED [TimestampValue] optional int64 TimestampValue (TIMESTAMP_MILLIS) [PLAIN, RLE, BIT_PACKED], 114272228}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue2] optional binary VarbinaryValue2 [PLAIN, RLE, BIT_PACKED], 114388803}, ColumnMetaData{UNCOMPRESSED [IntervalYearValue] optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222455665}, ColumnMetaData{UNCOMPRESSED [IntervalDayValue] optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222630508}, ColumnMetaData{UNCOMPRESSED [IntervalSecondValue] optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222805351}, ColumnMetaData{UNCOMPRESSED [VarcharValue2] optional binary VarcharValue2 (UTF8) [PLAIN, RLE, BIT_PACKED], 222980194}]}]}^M
^M
Fragment 0:0^M
^M
[Error Id: c6690ea1-2f28-4fbe-969f-d8b90da488fb on qa-node186.qa.lab:31010]^M
^M
(org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet record reader.^M
Message: ^M
Hadoop path: /drill/testdata/batch_memory/fourvarchar_asc_nulls_16MB_1GB.parquet/0_0_0.parquet^M
Total records read: 0^M
Row group index: 0^M
Records in row group: 14565^M
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {^M
optional int64 Index;^M
optional binary VarbinaryValue1;^M
optional int64 BigIntValue;^M
optional boolean BooleanValue;^M
optional int32 DateValue (DATE);^M
optional float FloatValue;^M
optional binary VarcharValue1 (UTF8);^M
optional double DoubleValue;^M
optional int32 IntegerValue;^M
optional int32 TimeValue (TIME_MILLIS);^M
optional int64 TimestampValue (TIMESTAMP_MILLIS);^M
optional binary VarbinaryValue2;^M
optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL);^M
optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL);^M
optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL);^M
optional binary VarcharValue2 (UTF8);^M
}^M
, metadata: {drill-writer.version=2, drill.version=1.14.0-SNAPSHOT}}, blocks: [BlockMetaData{14565, 268477520 [ColumnMetaData{UNCOMPRESSED [Index] optional int64 Index [PLAIN, RLE, BIT_PACKED], 4}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue1] optional binary VarbinaryValue1 [PLAIN, RLE, BIT_PACKED], 116579}, ColumnMetaData{UNCOMPRESSED [BigIntValue] optional int64 BigIntValue [PLAIN, RLE, BIT_PACKED], 91098467}, ColumnMetaData{UNCOMPRESSED [BooleanValue] optional boolean BooleanValue [PLAIN, RLE, BIT_PACKED], 91155431}, ColumnMetaData{UNCOMPRESSED [DateValue] optional int32 DateValue (DATE) [PLAIN, RLE, BIT_PACKED], 91157291}, ColumnMetaData{UNCOMPRESSED [FloatValue] optional float FloatValue [PLAIN, RLE, BIT_PACKED], 91215598}, ColumnMetaData{UNCOMPRESSED [VarcharValue1] optional binary VarcharValue1 (UTF8) [PLAIN, RLE, BIT_PACKED], 91273905}, ColumnMetaData{UNCOMPRESSED [DoubleValue] optional double DoubleValue [PLAIN, RLE, BIT_PACKED], 114039039}, ColumnMetaData{UNCOMPRESSED [IntegerValue] optional int32 IntegerValue [PLAIN, RLE, BIT_PACKED], 114155614}, ColumnMetaData{UNCOMPRESSED [TimeValue] optional int32 TimeValue (TIME_MILLIS) [PLAIN, RLE, BIT_PACKED], 114213921}, ColumnMetaData{UNCOMPRESSED [TimestampValue] optional int64 TimestampValue (TIMESTAMP_MILLIS) [PLAIN, RLE, BIT_PACKED], 114272228}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue2] optional binary VarbinaryValue2 [PLAIN, RLE, BIT_PACKED], 114388803}, ColumnMetaData{UNCOMPRESSED [IntervalYearValue] optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222455665}, ColumnMetaData{UNCOMPRESSED [IntervalDayValue] optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222630508}, ColumnMetaData{UNCOMPRESSED [IntervalSecondValue] optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222805351}, ColumnMetaData{UNCOMPRESSED [VarcharValue2] optional binary VarcharValue2 (UTF8) [PLAIN, RLE, BIT_PACKED], 222980194}]}]}^M
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleException():271^M
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next():290^M
org.apache.drill.exec.physical.impl.ScanBatch.next():177^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():411^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():361^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():316^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.physical.impl.BaseRootExec.next():103^M
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83^M
org.apache.drill.exec.physical.impl.BaseRootExec.next():93^M
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294^M
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281^M
java.security.AccessController.doPrivileged():-2^M
javax.security.auth.Subject.doAs():422^M
org.apache.hadoop.security.UserGroupInformation.doAs():1595^M
org.apache.drill.exec.work.fragment.FragmentExecutor.run():281^M
org.apache.drill.common.SelfCleaningRunnable.run():38^M
java.util.concurrent.ThreadPoolExecutor.runWorker():1149^M
java.util.concurrent.ThreadPoolExecutor$Worker.run():624^M
java.lang.Thread.run():748^M
Caused By (java.lang.ArrayIndexOutOfBoundsException) null^M
java.lang.System.arraycopy():-2^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenAbstractEntryReader.vlCopyGTLongWithPadding():118^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenAbstractEntryReader.vlCopy():90^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenNullableFixedEntryReader.getEntry():75^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getFixedEntry():167^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getEntry():132^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next():154^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next():38^M
org.apache.drill.exec.vector.VarBinaryVector$Mutator.setSafe():622^M
org.apache.drill.exec.vector.NullableVarBinaryVector$Mutator.setSafe():716^M
org.apache.drill.exec.store.parquet.columnreaders.VarLengthColumnReaders$NullableVarBinaryColumn.setSafe():311^M
org.apache.drill.exec.store.parquet.columnreaders.VarLengthValuesColumn.readRecordsInBulk():98^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readRecordsInBulk():114^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readFields():92^M
org.apache.drill.exec.store.parquet.columnreaders.BatchReader$VariableWidthReader.readRecords():156^M
org.apache.drill.exec.store.parquet.columnreaders.BatchReader.readBatch():43^M
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next():288^M
org.apache.drill.exec.physical.impl.ScanBatch.next():177^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():411^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():361^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():316^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.physical.impl.BaseRootExec.next():103^M
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83^M
org.apache.drill.exec.physical.impl.BaseRootExec.next():93^M
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294^M
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281^M
java.security.AccessController.doPrivileged():-2^M
javax.security.auth.Subject.doAs():422^M
org.apache.hadoop.security.UserGroupInformation.doAs():1595^M
org.apache.drill.exec.work.fragment.FragmentExecutor.run():281^M
org.apache.drill.common.SelfCleaningRunnable.run():38^M
java.util.concurrent.ThreadPoolExecutor.runWorker():1149^M
java.util.concurrent.ThreadPoolExecutor$Worker.run():624^M
java.lang.Thread.run():748^M
^M
at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)^M
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)^M
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)^M
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)^M
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)^M
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
at oadd.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)^M
at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)^M
at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)^M
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)^M
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)^M
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)^M
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)^M
at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)^M
at java.lang.Thread.run(Thread.java:748)^M
{noformat}
Table can be found in 10.10.100.186:/tmp/fourvarchar_asc_nulls_16MB_1GB.parquet
sys.version is:
1.15.0-SNAPSHOT a05f17d6fcd80f0d21260d3b1074ab895f457bac Changed PROJECT_OUTPUT_BATCH_SIZE to System + Session 30.07.2018 @ 17:12:53 PDT rhou@mapr.com 30.07.2018 @ 17:25:21 PDT^M
fourvarchar_asc_nulls70.q
was:
This is the query:
alter session set `drill.exec.memory.operator.project.output_batch_size` = 131072;
alter session set `planner.width.max_per_node` = 1;
alter session set `planner.width.max_per_query` = 1;
select * from (
select BigIntValue, BooleanValue, DateValue, FloatValue, DoubleValue, IntegerValue, TimeValue, TimestampValue, IntervalYearValue, IntervalDayValue, IntervalSecondValue, VarbinaryValue1, VarcharValue1, VarbinaryValue2, VarcharValue2 from (select * from dfs.`/drill/testdata/batch_memory/fourvarchar_asc_nulls_16MB_1GB.parquet` order by IntegerValue)) d where d.VarcharValue1 = 'Fq';
It appears to be caused by this commit:
DRILL-6570: Fixed IndexOutofBoundException in Parquet Reader
aee899c1b26ebb9a5781d280d5a73b42c273d4d5
This is the stack trace:
{noformat}
oadd.org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR: Error in parquet record reader.^M
Message: ^M
Hadoop path: /drill/testdata/batch_memory/fourvarchar_asc_nulls_16MB_1GB.parquet/0_0_0.parquet^M
Total records read: 0^M
Row group index: 0^M
Records in row group: 14565^M
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {^M
optional int64 Index;^M
optional binary VarbinaryValue1;^M
optional int64 BigIntValue;^M
optional boolean BooleanValue;^M
optional int32 DateValue (DATE);^M
optional float FloatValue;^M
optional binary VarcharValue1 (UTF8);^M
optional double DoubleValue;^M
optional int32 IntegerValue;^M
optional int32 TimeValue (TIME_MILLIS);^M
optional int64 TimestampValue (TIMESTAMP_MILLIS);^M
optional binary VarbinaryValue2;^M
optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL);^M
optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL);^M
optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL);^M
optional binary VarcharValue2 (UTF8);^M
}^M
, metadata: {drill-writer.version=2, drill.version=1.14.0-SNAPSHOT}}, blocks: [BlockMetaData{14565, 268477520 [ColumnMetaData{UNCOMPRESSED [Index] optional int64 Index [PLAIN, RLE, BIT_PACKED], 4}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue1] optional binary VarbinaryValue1 [PLAIN, RLE, BIT_PACKED], 116579}, ColumnMetaData{UNCOMPRESSED [BigIntValue] optional int64 BigIntValue [PLAIN, RLE, BIT_PACKED], 91098467}, ColumnMetaData{UNCOMPRESSED [BooleanValue] optional boolean BooleanValue [PLAIN, RLE, BIT_PACKED], 91155431}, ColumnMetaData{UNCOMPRESSED [DateValue] optional int32 DateValue (DATE) [PLAIN, RLE, BIT_PACKED], 91157291}, ColumnMetaData{UNCOMPRESSED [FloatValue] optional float FloatValue [PLAIN, RLE, BIT_PACKED], 91215598}, ColumnMetaData{UNCOMPRESSED [VarcharValue1] optional binary VarcharValue1 (UTF8) [PLAIN, RLE, BIT_PACKED], 91273905}, ColumnMetaData{UNCOMPRESSED [DoubleValue] optional double DoubleValue [PLAIN, RLE, BIT_PACKED], 114039039}, ColumnMetaData{UNCOMPRESSED [IntegerValue] optional int32 IntegerValue [PLAIN, RLE, BIT_PACKED], 114155614}, ColumnMetaData{UNCOMPRESSED [TimeValue] optional int32 TimeValue (TIME_MILLIS) [PLAIN, RLE, BIT_PACKED], 114213921}, ColumnMetaData{UNCOMPRESSED [TimestampValue] optional int64 TimestampValue (TIMESTAMP_MILLIS) [PLAIN, RLE, BIT_PACKED], 114272228}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue2] optional binary VarbinaryValue2 [PLAIN, RLE, BIT_PACKED], 114388803}, ColumnMetaData{UNCOMPRESSED [IntervalYearValue] optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222455665}, ColumnMetaData{UNCOMPRESSED [IntervalDayValue] optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222630508}, ColumnMetaData{UNCOMPRESSED [IntervalSecondValue] optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222805351}, ColumnMetaData{UNCOMPRESSED [VarcharValue2] optional binary VarcharValue2 (UTF8) [PLAIN, RLE, BIT_PACKED], 222980194}]}]}^M
^M
Fragment 0:0^M
^M
[Error Id: c6690ea1-2f28-4fbe-969f-d8b90da488fb on qa-node186.qa.lab:31010]^M
^M
(org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet record reader.^M
Message: ^M
Hadoop path: /drill/testdata/batch_memory/fourvarchar_asc_nulls_16MB_1GB.parquet/0_0_0.parquet^M
Total records read: 0^M
Row group index: 0^M
Records in row group: 14565^M
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {^M
optional int64 Index;^M
optional binary VarbinaryValue1;^M
optional int64 BigIntValue;^M
optional boolean BooleanValue;^M
optional int32 DateValue (DATE);^M
optional float FloatValue;^M
optional binary VarcharValue1 (UTF8);^M
optional double DoubleValue;^M
optional int32 IntegerValue;^M
optional int32 TimeValue (TIME_MILLIS);^M
optional int64 TimestampValue (TIMESTAMP_MILLIS);^M
optional binary VarbinaryValue2;^M
optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL);^M
optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL);^M
optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL);^M
optional binary VarcharValue2 (UTF8);^M
}^M
, metadata: {drill-writer.version=2, drill.version=1.14.0-SNAPSHOT}}, blocks: [BlockMetaData{14565, 268477520 [ColumnMetaData{UNCOMPRESSED [Index] optional int64 Index [PLAIN, RLE, BIT_PACKED], 4}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue1] optional binary VarbinaryValue1 [PLAIN, RLE, BIT_PACKED], 116579}, ColumnMetaData{UNCOMPRESSED [BigIntValue] optional int64 BigIntValue [PLAIN, RLE, BIT_PACKED], 91098467}, ColumnMetaData{UNCOMPRESSED [BooleanValue] optional boolean BooleanValue [PLAIN, RLE, BIT_PACKED], 91155431}, ColumnMetaData{UNCOMPRESSED [DateValue] optional int32 DateValue (DATE) [PLAIN, RLE, BIT_PACKED], 91157291}, ColumnMetaData{UNCOMPRESSED [FloatValue] optional float FloatValue [PLAIN, RLE, BIT_PACKED], 91215598}, ColumnMetaData{UNCOMPRESSED [VarcharValue1] optional binary VarcharValue1 (UTF8) [PLAIN, RLE, BIT_PACKED], 91273905}, ColumnMetaData{UNCOMPRESSED [DoubleValue] optional double DoubleValue [PLAIN, RLE, BIT_PACKED], 114039039}, ColumnMetaData{UNCOMPRESSED [IntegerValue] optional int32 IntegerValue [PLAIN, RLE, BIT_PACKED], 114155614}, ColumnMetaData{UNCOMPRESSED [TimeValue] optional int32 TimeValue (TIME_MILLIS) [PLAIN, RLE, BIT_PACKED], 114213921}, ColumnMetaData{UNCOMPRESSED [TimestampValue] optional int64 TimestampValue (TIMESTAMP_MILLIS) [PLAIN, RLE, BIT_PACKED], 114272228}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue2] optional binary VarbinaryValue2 [PLAIN, RLE, BIT_PACKED], 114388803}, ColumnMetaData{UNCOMPRESSED [IntervalYearValue] optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222455665}, ColumnMetaData{UNCOMPRESSED [IntervalDayValue] optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222630508}, ColumnMetaData{UNCOMPRESSED [IntervalSecondValue] optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222805351}, ColumnMetaData{UNCOMPRESSED [VarcharValue2] optional binary VarcharValue2 (UTF8) [PLAIN, RLE, BIT_PACKED], 222980194}]}]}^M
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleException():271^M
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next():290^M
org.apache.drill.exec.physical.impl.ScanBatch.next():177^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():411^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():361^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():316^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.physical.impl.BaseRootExec.next():103^M
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83^M
org.apache.drill.exec.physical.impl.BaseRootExec.next():93^M
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294^M
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281^M
java.security.AccessController.doPrivileged():-2^M
javax.security.auth.Subject.doAs():422^M
org.apache.hadoop.security.UserGroupInformation.doAs():1595^M
org.apache.drill.exec.work.fragment.FragmentExecutor.run():281^M
org.apache.drill.common.SelfCleaningRunnable.run():38^M
java.util.concurrent.ThreadPoolExecutor.runWorker():1149^M
java.util.concurrent.ThreadPoolExecutor$Worker.run():624^M
java.lang.Thread.run():748^M
Caused By (java.lang.ArrayIndexOutOfBoundsException) null^M
java.lang.System.arraycopy():-2^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenAbstractEntryReader.vlCopyGTLongWithPadding():118^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenAbstractEntryReader.vlCopy():90^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenNullableFixedEntryReader.getEntry():75^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getFixedEntry():167^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getEntry():132^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next():154^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next():38^M
org.apache.drill.exec.vector.VarBinaryVector$Mutator.setSafe():622^M
org.apache.drill.exec.vector.NullableVarBinaryVector$Mutator.setSafe():716^M
org.apache.drill.exec.store.parquet.columnreaders.VarLengthColumnReaders$NullableVarBinaryColumn.setSafe():311^M
org.apache.drill.exec.store.parquet.columnreaders.VarLengthValuesColumn.readRecordsInBulk():98^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readRecordsInBulk():114^M
org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readFields():92^M
org.apache.drill.exec.store.parquet.columnreaders.BatchReader$VariableWidthReader.readRecords():156^M
org.apache.drill.exec.store.parquet.columnreaders.BatchReader.readBatch():43^M
org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next():288^M
org.apache.drill.exec.physical.impl.ScanBatch.next():177^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():411^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():361^M
org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():316^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
org.apache.drill.exec.physical.impl.BaseRootExec.next():103^M
org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83^M
org.apache.drill.exec.physical.impl.BaseRootExec.next():93^M
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294^M
org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281^M
java.security.AccessController.doPrivileged():-2^M
javax.security.auth.Subject.doAs():422^M
org.apache.hadoop.security.UserGroupInformation.doAs():1595^M
org.apache.drill.exec.work.fragment.FragmentExecutor.run():281^M
org.apache.drill.common.SelfCleaningRunnable.run():38^M
java.util.concurrent.ThreadPoolExecutor.runWorker():1149^M
java.util.concurrent.ThreadPoolExecutor$Worker.run():624^M
java.lang.Thread.run():748^M
^M
at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)^M
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)^M
at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)^M
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)^M
at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)^M
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
at oadd.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)^M
at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)^M
at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)^M
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)^M
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)^M
at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)^M
at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)^M
at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)^M
at java.lang.Thread.run(Thread.java:748)^M
Table can be found in 10.10.100.186:/tmp/fourvarchar_asc_nulls_16MB_1GB.parquet
sys.version is:
1.15.0-SNAPSHOT a05f17d6fcd80f0d21260d3b1074ab895f457bac Changed PROJECT_OUTPUT_BATCH_SIZE to System + Session 30.07.2018 @ 17:12:53 PDT rhou@mapr.com 30.07.2018 @ 17:25:21 PDT^M
fourvarchar_asc_nulls70.q
> Error in parquet record reader
> ------------------------------
>
> Key: DRILL-6685
> URL: https://issues.apache.org/jira/browse/DRILL-6685
> Project: Apache Drill
> Issue Type: Bug
> Components: Storage - Parquet
> Affects Versions: 1.14.0
> Reporter: Robert Hou
> Assignee: salim achouche
> Priority: Major
> Fix For: 1.15.0
>
>
> This is the query:
> alter session set `drill.exec.memory.operator.project.output_batch_size` = 131072;
> alter session set `planner.width.max_per_node` = 1;
> alter session set `planner.width.max_per_query` = 1;
> select * from (
> select BigIntValue, BooleanValue, DateValue, FloatValue, DoubleValue, IntegerValue, TimeValue, TimestampValue, IntervalYearValue, IntervalDayValue, IntervalSecondValue, VarbinaryValue1, VarcharValue1, VarbinaryValue2, VarcharValue2 from (select * from dfs.`/drill/testdata/batch_memory/fourvarchar_asc_nulls_16MB_1GB.parquet` order by IntegerValue)) d where d.VarcharValue1 = 'Fq';
> It appears to be caused by this commit:
> DRILL-6570: Fixed IndexOutofBoundException in Parquet Reader
> aee899c1b26ebb9a5781d280d5a73b42c273d4d5
> This is the stack trace:
> {noformat}
> oadd.org.apache.drill.common.exceptions.UserRemoteException: INTERNAL_ERROR ERROR: Error in parquet record reader.^M
> Message: ^M
> Hadoop path: /drill/testdata/batch_memory/fourvarchar_asc_nulls_16MB_1GB.parquet/0_0_0.parquet^M
> Total records read: 0^M
> Row group index: 0^M
> Records in row group: 14565^M
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {^M
> optional int64 Index;^M
> optional binary VarbinaryValue1;^M
> optional int64 BigIntValue;^M
> optional boolean BooleanValue;^M
> optional int32 DateValue (DATE);^M
> optional float FloatValue;^M
> optional binary VarcharValue1 (UTF8);^M
> optional double DoubleValue;^M
> optional int32 IntegerValue;^M
> optional int32 TimeValue (TIME_MILLIS);^M
> optional int64 TimestampValue (TIMESTAMP_MILLIS);^M
> optional binary VarbinaryValue2;^M
> optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL);^M
> optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL);^M
> optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL);^M
> optional binary VarcharValue2 (UTF8);^M
> }^M
> , metadata: {drill-writer.version=2, drill.version=1.14.0-SNAPSHOT}}, blocks: [BlockMetaData{14565, 268477520 [ColumnMetaData{UNCOMPRESSED [Index] optional int64 Index [PLAIN, RLE, BIT_PACKED], 4}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue1] optional binary VarbinaryValue1 [PLAIN, RLE, BIT_PACKED], 116579}, ColumnMetaData{UNCOMPRESSED [BigIntValue] optional int64 BigIntValue [PLAIN, RLE, BIT_PACKED], 91098467}, ColumnMetaData{UNCOMPRESSED [BooleanValue] optional boolean BooleanValue [PLAIN, RLE, BIT_PACKED], 91155431}, ColumnMetaData{UNCOMPRESSED [DateValue] optional int32 DateValue (DATE) [PLAIN, RLE, BIT_PACKED], 91157291}, ColumnMetaData{UNCOMPRESSED [FloatValue] optional float FloatValue [PLAIN, RLE, BIT_PACKED], 91215598}, ColumnMetaData{UNCOMPRESSED [VarcharValue1] optional binary VarcharValue1 (UTF8) [PLAIN, RLE, BIT_PACKED], 91273905}, ColumnMetaData{UNCOMPRESSED [DoubleValue] optional double DoubleValue [PLAIN, RLE, BIT_PACKED], 114039039}, ColumnMetaData{UNCOMPRESSED [IntegerValue] optional int32 IntegerValue [PLAIN, RLE, BIT_PACKED], 114155614}, ColumnMetaData{UNCOMPRESSED [TimeValue] optional int32 TimeValue (TIME_MILLIS) [PLAIN, RLE, BIT_PACKED], 114213921}, ColumnMetaData{UNCOMPRESSED [TimestampValue] optional int64 TimestampValue (TIMESTAMP_MILLIS) [PLAIN, RLE, BIT_PACKED], 114272228}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue2] optional binary VarbinaryValue2 [PLAIN, RLE, BIT_PACKED], 114388803}, ColumnMetaData{UNCOMPRESSED [IntervalYearValue] optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222455665}, ColumnMetaData{UNCOMPRESSED [IntervalDayValue] optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222630508}, ColumnMetaData{UNCOMPRESSED [IntervalSecondValue] optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222805351}, ColumnMetaData{UNCOMPRESSED [VarcharValue2] optional binary VarcharValue2 (UTF8) [PLAIN, RLE, BIT_PACKED], 222980194}]}]}^M
> ^M
> Fragment 0:0^M
> ^M
> [Error Id: c6690ea1-2f28-4fbe-969f-d8b90da488fb on qa-node186.qa.lab:31010]^M
> ^M
> (org.apache.drill.common.exceptions.DrillRuntimeException) Error in parquet record reader.^M
> Message: ^M
> Hadoop path: /drill/testdata/batch_memory/fourvarchar_asc_nulls_16MB_1GB.parquet/0_0_0.parquet^M
> Total records read: 0^M
> Row group index: 0^M
> Records in row group: 14565^M
> Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {^M
> optional int64 Index;^M
> optional binary VarbinaryValue1;^M
> optional int64 BigIntValue;^M
> optional boolean BooleanValue;^M
> optional int32 DateValue (DATE);^M
> optional float FloatValue;^M
> optional binary VarcharValue1 (UTF8);^M
> optional double DoubleValue;^M
> optional int32 IntegerValue;^M
> optional int32 TimeValue (TIME_MILLIS);^M
> optional int64 TimestampValue (TIMESTAMP_MILLIS);^M
> optional binary VarbinaryValue2;^M
> optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL);^M
> optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL);^M
> optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL);^M
> optional binary VarcharValue2 (UTF8);^M
> }^M
> , metadata: {drill-writer.version=2, drill.version=1.14.0-SNAPSHOT}}, blocks: [BlockMetaData{14565, 268477520 [ColumnMetaData{UNCOMPRESSED [Index] optional int64 Index [PLAIN, RLE, BIT_PACKED], 4}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue1] optional binary VarbinaryValue1 [PLAIN, RLE, BIT_PACKED], 116579}, ColumnMetaData{UNCOMPRESSED [BigIntValue] optional int64 BigIntValue [PLAIN, RLE, BIT_PACKED], 91098467}, ColumnMetaData{UNCOMPRESSED [BooleanValue] optional boolean BooleanValue [PLAIN, RLE, BIT_PACKED], 91155431}, ColumnMetaData{UNCOMPRESSED [DateValue] optional int32 DateValue (DATE) [PLAIN, RLE, BIT_PACKED], 91157291}, ColumnMetaData{UNCOMPRESSED [FloatValue] optional float FloatValue [PLAIN, RLE, BIT_PACKED], 91215598}, ColumnMetaData{UNCOMPRESSED [VarcharValue1] optional binary VarcharValue1 (UTF8) [PLAIN, RLE, BIT_PACKED], 91273905}, ColumnMetaData{UNCOMPRESSED [DoubleValue] optional double DoubleValue [PLAIN, RLE, BIT_PACKED], 114039039}, ColumnMetaData{UNCOMPRESSED [IntegerValue] optional int32 IntegerValue [PLAIN, RLE, BIT_PACKED], 114155614}, ColumnMetaData{UNCOMPRESSED [TimeValue] optional int32 TimeValue (TIME_MILLIS) [PLAIN, RLE, BIT_PACKED], 114213921}, ColumnMetaData{UNCOMPRESSED [TimestampValue] optional int64 TimestampValue (TIMESTAMP_MILLIS) [PLAIN, RLE, BIT_PACKED], 114272228}, ColumnMetaData{UNCOMPRESSED [VarbinaryValue2] optional binary VarbinaryValue2 [PLAIN, RLE, BIT_PACKED], 114388803}, ColumnMetaData{UNCOMPRESSED [IntervalYearValue] optional fixed_len_byte_array(12) IntervalYearValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222455665}, ColumnMetaData{UNCOMPRESSED [IntervalDayValue] optional fixed_len_byte_array(12) IntervalDayValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222630508}, ColumnMetaData{UNCOMPRESSED [IntervalSecondValue] optional fixed_len_byte_array(12) IntervalSecondValue (INTERVAL) [PLAIN, RLE, BIT_PACKED], 222805351}, ColumnMetaData{UNCOMPRESSED [VarcharValue2] optional binary VarcharValue2 (UTF8) [PLAIN, RLE, BIT_PACKED], 222980194}]}]}^M
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleException():271^M
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next():290^M
> org.apache.drill.exec.physical.impl.ScanBatch.next():177^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():411^M
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():361^M
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():316^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.physical.impl.BaseRootExec.next():103^M
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83^M
> org.apache.drill.exec.physical.impl.BaseRootExec.next():93^M
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294^M
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281^M
> java.security.AccessController.doPrivileged():-2^M
> javax.security.auth.Subject.doAs():422^M
> org.apache.hadoop.security.UserGroupInformation.doAs():1595^M
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():281^M
> org.apache.drill.common.SelfCleaningRunnable.run():38^M
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149^M
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624^M
> java.lang.Thread.run():748^M
> Caused By (java.lang.ArrayIndexOutOfBoundsException) null^M
> java.lang.System.arraycopy():-2^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLenAbstractEntryReader.vlCopyGTLongWithPadding():118^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLenAbstractEntryReader.vlCopy():90^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLenNullableFixedEntryReader.getEntry():75^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getFixedEntry():167^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getEntry():132^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next():154^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next():38^M
> org.apache.drill.exec.vector.VarBinaryVector$Mutator.setSafe():622^M
> org.apache.drill.exec.vector.NullableVarBinaryVector$Mutator.setSafe():716^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLengthColumnReaders$NullableVarBinaryColumn.setSafe():311^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLengthValuesColumn.readRecordsInBulk():98^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readRecordsInBulk():114^M
> org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readFields():92^M
> org.apache.drill.exec.store.parquet.columnreaders.BatchReader$VariableWidthReader.readRecords():156^M
> org.apache.drill.exec.store.parquet.columnreaders.BatchReader.readBatch():43^M
> org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next():288^M
> org.apache.drill.exec.physical.impl.ScanBatch.next():177^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.loadBatch():411^M
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.load():361^M
> org.apache.drill.exec.physical.impl.xsort.managed.ExternalSortBatch.innerNext():316^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():119^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():109^M
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63^M
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142^M
> org.apache.drill.exec.record.AbstractRecordBatch.next():172^M
> org.apache.drill.exec.physical.impl.BaseRootExec.next():103^M
> org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext():83^M
> org.apache.drill.exec.physical.impl.BaseRootExec.next():93^M
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294^M
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281^M
> java.security.AccessController.doPrivileged():-2^M
> javax.security.auth.Subject.doAs():422^M
> org.apache.hadoop.security.UserGroupInformation.doAs():1595^M
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():281^M
> org.apache.drill.common.SelfCleaningRunnable.run():38^M
> java.util.concurrent.ThreadPoolExecutor.runWorker():1149^M
> java.util.concurrent.ThreadPoolExecutor$Worker.run():624^M
> java.lang.Thread.run():748^M
> ^M
> at oadd.org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:123)^M
> at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)^M
> at oadd.org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)^M
> at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:274)^M
> at oadd.org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:244)^M
> at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
> at oadd.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
> at oadd.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
> at oadd.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)^M
> at oadd.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
> at oadd.io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)^M
> at oadd.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)^M
> at oadd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)^M
> at oadd.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)^M
> at oadd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)^M
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)^M
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)^M
> at oadd.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)^M
> at oadd.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)^M
> at oadd.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)^M
> at java.lang.Thread.run(Thread.java:748)^M
> {noformat}
> Table can be found in 10.10.100.186:/tmp/fourvarchar_asc_nulls_16MB_1GB.parquet
> sys.version is:
> 1.15.0-SNAPSHOT a05f17d6fcd80f0d21260d3b1074ab895f457bac Changed PROJECT_OUTPUT_BATCH_SIZE to System + Session 30.07.2018 @ 17:12:53 PDT rhou@mapr.com 30.07.2018 @ 17:25:21 PDT^M
> fourvarchar_asc_nulls70.q
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)