You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "dzcxzl (Jira)" <ji...@apache.org> on 2022/07/21 12:21:00 UTC
[jira] [Commented] (SPARK-39830) Reading ORC table that requires type promotion may throw AIOOBE
[ https://issues.apache.org/jira/browse/SPARK-39830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17569385#comment-17569385 ]
dzcxzl commented on SPARK-39830:
--------------------------------
cc @[~dongjoon]
> Reading ORC table that requires type promotion may throw AIOOBE
> ---------------------------------------------------------------
>
> Key: SPARK-39830
> URL: https://issues.apache.org/jira/browse/SPARK-39830
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.3.0
> Reporter: dzcxzl
> Priority: Trivial
>
> We can add a UT to test the scenario after the ORC-1205 release.
>
> bin/spark-shell
> {code:java}
> spark.sql("set orc.stripe.size=10240")
> spark.sql("set orc.rows.between.memory.checks=1")
> spark.sql("set spark.sql.orc.columnarWriterBatchSize=1")
> val df = spark.range(1, 1+512, 1, 1).map { i =>
> if( i == 1 ){
> (i, Array.fill[Byte](5 * 1024 * 1024)('X'))
> } else {
> (i,Array.fill[Byte](1)('X'))
> }
> }.toDF("c1","c2")
> df.write.format("orc").save("file:///tmp/test_table_orc_t1")
> spark.sql("create external table test_table_orc_t1 (c1 string ,c2 binary) location 'file:///tmp/test_table_orc_t1' stored as orc ")
> spark.sql("select * from test_table_orc_t1").show() {code}
> Querying this table will get the following exception
> {code:java}
> java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.orc.impl.TreeReaderFactory$TreeReader.nextVector(TreeReaderFactory.java:387)
> at org.apache.orc.impl.TreeReaderFactory$LongTreeReader.nextVector(TreeReaderFactory.java:740)
> at org.apache.orc.impl.ConvertTreeReaderFactory$StringGroupFromAnyIntegerTreeReader.nextVector(ConvertTreeReaderFactory.java:1069)
> at org.apache.orc.impl.reader.tree.StructBatchReader.readBatchColumn(StructBatchReader.java:65)
> at org.apache.orc.impl.reader.tree.StructBatchReader.nextBatchForLevel(StructBatchReader.java:100)
> at org.apache.orc.impl.reader.tree.StructBatchReader.nextBatch(StructBatchReader.java:77)
> at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1371)
> at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:84)
> at org.apache.orc.mapreduce.OrcMapreduceRecordReader.nextKeyValue(OrcMapreduceRecordReader.java:102)
> at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39)
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org