You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@orc.apache.org by "Dongjoon Hyun (Jira)" <ji...@apache.org> on 2022/05/07 23:45:00 UTC
[jira] [Commented] (ORC-1167) Support orc.row.batch.size configuration
[ https://issues.apache.org/jira/browse/ORC-1167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17533363#comment-17533363 ]
Dongjoon Hyun commented on ORC-1167:
------------------------------------
Thank you for filing a JIRA and making a PR, [~dzcxzl]
> Support orc.row.batch.size configuration
> ----------------------------------------
>
> Key: ORC-1167
> URL: https://issues.apache.org/jira/browse/ORC-1167
> Project: ORC
> Issue Type: Improvement
> Reporter: dzcxzl
> Priority: Minor
>
> Now create OrcMapreduceRecordReader, the default value of batch size is 1024, we can support the configuration in Reader.Options.
>
> If we read 1024 relatively large strings, we might get NegativeArraySizeException, but no configuration to reduce batch size.
>
> {code:java}
> java.lang.NegativeArraySizeException
> at org.apache.orc.impl.TreeReaderFactory$BytesColumnVectorUtil.commonReadByteArrays(TreeReaderFactory.java:1544)
> at org.apache.orc.impl.TreeReaderFactory$BytesColumnVectorUtil.readOrcByteArrays(TreeReaderFactory.java:1566)
> at org.apache.orc.impl.TreeReaderFactory$StringDirectTreeReader.nextVector(TreeReaderFactory.java:1662)
> at org.apache.orc.impl.TreeReaderFactory$StringTreeReader.nextVector(TreeReaderFactory.java:1508)
> at org.apache.orc.impl.TreeReaderFactory$StructTreeReader.nextBatch(TreeReaderFactory.java:2047)
> at org.apache.orc.impl.RecordReaderImpl.nextBatch(RecordReaderImpl.java:1219)
> at org.apache.orc.mapreduce.OrcMapreduceRecordReader.ensureBatch(OrcMapreduceRecordReader.java:84)
> at org.apache.orc.mapreduce.OrcMapreduceRecordReader.nextKeyValue(OrcMapreduceRecordReader.java:102)
> at org.apache.spark.sql.execution.datasources.RecordReaderIterator.hasNext(RecordReaderIterator.scala:39) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)