You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by "Khurram Faraaz (JIRA)" <ji...@apache.org> on 2019/05/21 23:02:00 UTC
[jira] [Created] (DRILL-7275) CTAS + CTE query fails with
IllegalStateException: Read batch count [%d] should be greater than zero
[0]
Khurram Faraaz created DRILL-7275:
-------------------------------------
Summary: CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0]
Key: DRILL-7275
URL: https://issues.apache.org/jira/browse/DRILL-7275
Project: Apache Drill
Issue Type: Bug
Components: Storage - Parquet
Affects Versions: 1.15.0
Reporter: Khurram Faraaz
CTAS + CTE query fails with IllegalStateException: Read batch count [%d] should be greater than zero [0]
Precondition check fails on line 47 in VarLenFixedEntryReader.java
{noformat}
44 final int expectedDataLen = columnPrecInfo.precision;
45 final int entrySz = 4 + columnPrecInfo.precision;
46 final int readBatch = getFixedLengthMaxRecordsToRead(valuesToRead, entrySz);
47 Preconditions.checkState(readBatch > 0, "Read batch count [%d] should be greater than zero", readBatch);
{noformat}
Stack trace from drillbit.log, also has the failing query.
{noformat}
2019-05-13 14:40:14,090 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO o.a.drill.exec.work.foreman.Foreman - Query text for query with id 23268c40-ef3a-6349-5901-5762f6888971 issued by scoop_stc: CREATE TABLE TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityProxyDailyReturn AS
WITH
che AS (
SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_c_CompositeHierarchyEntry_TimeVarying
WHERE CompositeHierarchyName = 'AxiomaRegion/AxiomaSector/VectorUniverse'
AND state = 'DupesRemoved'
AND CompositeLevel = 'AxiomaRegion_1/AxiomaSector_1/VectorUniverse_0'
),
ef AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_c_EquityDailyReturn_FXAdjusted WHERE Status = 'PresentInRawData'),
d AS (SELECT * FROM TEST_TEMPLATE_SCHEMA_creid.tbl_r_BusinessDate WHERE IsWeekday),
x AS
(
SELECT
che.CompositeHierarchyName,
che.State,
che.CompositeNodeName,
d.`Date` AS RecordDate,
COUNT(che.CompositeNodeName) AS countDistinctConstituents,
COUNT(ef.VectorListingId) AS countDataPoints,
AVG(ef.DailyReturn) AS AvgReturn,
AVG(ef.DailyReturnUSD) AS AvgReturnUSD,
AVG(ef.NotionalReturnUSD) AS AvgNotionalReturnUSD
FROM d
INNER JOIN che ON d.`Date` BETWEEN che.CompositeUltimateChildStartDate AND che.CompositeUltimateChildEndDate
LEFT OUTER JOIN ef ON d.`Date` = ef.RecordDate AND 'VectorListingId_' || CAST(ef.VectorListingId AS VARCHAR(100)) = che.UltimateChild
GROUP BY che.CompositeHierarchyName, che.State, che.CompositeNodeName, d.`Date`, d.IsWeekday, d.IsHoliday
)
SELECT * FROM x
2019-05-13 14:40:16,971 [23268c40-ef3a-6349-5901-5762f6888971:foreman] INFO o.a.d.e.p.s.h.CreateTableHandler - Creating persistent table [tbl_c_EquityProxyDailyReturn].
...
...
2019-05-13 14:40:20,036 [23268c40-ef3a-6349-5901-5762f6888971:frag:6:10] INFO o.a.d.exec.physical.impl.ScanBatch - User Error Occurred: Error in parquet record reader.
Message:
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
optional int64 VectorListingId;
optional int32 RecordDate (DATE);
required binary Status (UTF8);
required binary CurrencyISO (UTF8);
optional double DailyReturn;
optional double DailyReturnUSD;
optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks: [BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId] optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE) [RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status] required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050}, ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8) [BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn] optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715}, ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD [RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED [NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED, PLAIN], 178749}]}]} (Error in parquet record reader.
...
...
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
optional int64 VectorListingId;
optional int32 RecordDate (DATE);
required binary Status (UTF8);
required binary CurrencyISO (UTF8);
optional double DailyReturn;
optional double DailyReturnUSD;
optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks: [BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId] optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE) [RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status] required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050}, ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8) [BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn] optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715}, ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD [RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED [NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED, PLAIN], 178749}]}]})
org.apache.drill.common.exceptions.UserException: INTERNAL_ERROR ERROR: Error in parquet record reader.
Message:
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
optional int64 VectorListingId;
optional int32 RecordDate (DATE);
required binary Status (UTF8);
required binary CurrencyISO (UTF8);
optional double DailyReturn;
optional double DailyReturnUSD;
optional double NotionalReturnUSD;
}
...
...
Caused by: org.apache.drill.common.exceptions.DrillRuntimeException: Error in parquet record reader.
Message:
Hadoop path: /DEV/tbl_c_EquityDailyReturn_FXAdjusted/1_32_32.parquet
Total records read: 0
Row group index: 0
Records in row group: 3243
Parquet Metadata: ParquetMetaData{FileMetaData{schema: message root {
optional int64 VectorListingId;
optional int32 RecordDate (DATE);
required binary Status (UTF8);
required binary CurrencyISO (UTF8);
optional double DailyReturn;
optional double DailyReturnUSD;
optional double NotionalReturnUSD;
}
, metadata: \{drill-writer.version=2, drill.version=1.15.0.0-mapr}}, blocks: [BlockMetaData\{3243, 204762 [ColumnMetaData{UNCOMPRESSED [VectorListingId] optional int64 VectorListingId [RLE, BIT_PACKED, PLAIN], 4}, ColumnMetaData\{UNCOMPRESSED [RecordDate] optional int32 RecordDate (DATE) [RLE, BIT_PACKED, PLAIN], 26021}, ColumnMetaData\{UNCOMPRESSED [Status] required binary Status (UTF8) [BIT_PACKED, PLAIN], 39050}, ColumnMetaData\{UNCOMPRESSED [CurrencyISO] required binary CurrencyISO (UTF8) [BIT_PACKED, PLAIN], 103968}, ColumnMetaData\{UNCOMPRESSED [DailyReturn] optional double DailyReturn [RLE, BIT_PACKED, PLAIN], 126715}, ColumnMetaData\{UNCOMPRESSED [DailyReturnUSD] optional double DailyReturnUSD [RLE, BIT_PACKED, PLAIN], 152732}, ColumnMetaData\{UNCOMPRESSED [NotionalReturnUSD] optional double NotionalReturnUSD [RLE, BIT_PACKED, PLAIN], 178749}]}]}
at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.handleException(ParquetRecordReader.java:271) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:290) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.ScanBatch.internalNext(ScanBatch.java:223) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.physical.impl.ScanBatch.next(ScanBatch.java:271) [drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
... 44 common frames omitted
Caused by: java.lang.IllegalStateException: Read batch count [%d] should be greater than zero [0]
at org.apache.drill.shaded.guava.com.google.common.base.Preconditions.checkState(Preconditions.java:509) ~[drill-shaded-guava-23.0.jar:23.0]
at org.apache.drill.exec.store.parquet.columnreaders.VarLenFixedEntryReader.getEntry(VarLenFixedEntryReader.java:47) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getFixedEntry(VarLenBulkPageReader.java:169) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.VarLenBulkPageReader.getEntry(VarLenBulkPageReader.java:132) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next(VarLenColumnBulkInput.java:154) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.VarLenColumnBulkInput.next(VarLenColumnBulkInput.java:38) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe(VarCharVector.java:624) ~[vector-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.vector.VarCharVector$Mutator.setSafe(VarCharVector.java:608) ~[vector-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.VarLengthColumnReaders$VarCharColumn.setSafe(VarLengthColumnReaders.java:168) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.VarLengthValuesColumn.readRecordsInBulk(VarLengthValuesColumn.java:98) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readRecordsInBulk(VarLenBinaryReader.java:114) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.VarLenBinaryReader.readFields(VarLenBinaryReader.java:92) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.BatchReader$VariableWidthReader.readRecords(BatchReader.java:156) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.BatchReader.readBatch(BatchReader.java:43) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
at org.apache.drill.exec.store.parquet.columnreaders.ParquetRecordReader.next(ParquetRecordReader.java:288) ~[drill-java-exec-1.15.0.0-mapr.jar:1.15.0.0-mapr]
... 46 common frames omitted
{noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)