You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2025/09/23 20:44:01 UTC
[I] HFile writer throws BufferOverflowException exception [hudi]
linliu-code opened a new issue, #13978:
URL: https://github.com/apache/hudi/issues/13978
### Bug Description
**What happened:**
A spark job failed with the following exception:
> Caused by: org.apache.hudi.exception.HoodieException: Error writing record HoodieRecord{key=HoodieKey { recordKey=__all_partitions__ partitionPath=files}, currentLocation='HoodieRecordLocation {instantTime=20250923172019848, fileId=files-0000-0, position=-1}', newLocation='HoodieRecordLocation {instantTime=20250923172019848, fileId=files-0000-0, position=-1}'}
at org.apache.hudi.client.FailOnFirstErrorWriteStatus.markFailure(FailOnFirstErrorWriteStatus.java:49)
at org.apache.hudi.io.FileGroupReaderBasedMergeHandle.doMerge(FileGroupReaderBasedMergeHandle.java:304)
at org.apache.hudi.table.action.compact.HoodieCompactor.compact(HoodieCompactor.java:157)
at org.apache.hudi.table.action.compact.HoodieCompactor.lambda$compact$da19f3d2$1(HoodieCompactor.java:140)
at org.apache.spark.api.java.JavaPairRDD$.$anonfun$toScalaFunction$1(JavaPairRDD.scala:1070)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:486)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:492)
at org.apache.spark.storage.memory.MemoryStore.putIterator(MemoryStore.scala:223)
at org.apache.spark.storage.memory.MemoryStore.putIteratorAsBytes(MemoryStore.scala:352)
at org.apache.spark.storage.BlockManager.$anonfun$doPutIterator$1(BlockManager.scala:1614)
at org.apache.spark.storage.BlockManager.org$apache$spark$storage$BlockManager$$doPut(BlockManager.scala:1524)
at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1588)
at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:1389)
at org.apache.spark.storage.BlockManager.getOrElseUpdateRDDBlock(BlockManager.scala:1343)
at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:379)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:329)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:367)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:331)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:93)
at org.apache.spark.TaskContext.runTaskWithListeners(TaskContext.scala:166)
at org.apache.spark.scheduler.Task.run(Task.scala:141)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$4(Executor.scala:620)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally(SparkErrorUtils.scala:64)
at org.apache.spark.util.SparkErrorUtils.tryWithSafeFinally$(SparkErrorUtils.scala:61)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:94)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:623)
... 3 more
Caused by: java.nio.BufferOverflowException
at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:194)
at java.nio.ByteBuffer.put(ByteBuffer.java:867)
at org.apache.hudi.io.hfile.HFileDataBlock.getUncompressedBlockDataToWrite(HFileDataBlock.java:235)
at org.apache.hudi.io.hfile.HFileBlock.serialize(HFileBlock.java:269)
at org.apache.hudi.io.hfile.HFileWriterImpl.flushCurrentDataBlock(HFileWriterImpl.java:139)
at org.apache.hudi.io.hfile.HFileWriterImpl.append(HFileWriterImpl.java:98)
at org.apache.hudi.io.hadoop.HoodieAvroHFileWriter.writeAvro(HoodieAvroHFileWriter.java:151)
at org.apache.hudi.io.storage.HoodieAvroFileWriter.write(HoodieAvroFileWriter.java:51)
at org.apache.hudi.io.storage.HoodieFileWriter.write(HoodieFileWriter.java:43)
at org.apache.hudi.io.HoodieWriteMergeHandle.writeToFile(HoodieWriteMergeHandle.java:415)
at org.apache.hudi.io.FileGroupReaderBasedMergeHandle.doMerge(FileGroupReaderBasedMergeHandle.java:299)
... 29 more
**What you expected:**
This is a bug due to fix buffer capacity. When a key value is larger than the given block size, this error happens.
**Steps to reproduce:**
1. Modify `TestHFileWriter`, and set their block size = 1 for any tests.
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HFile writer throws BufferOverflowException exception [hudi]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code closed issue #13978: HFile writer throws BufferOverflowException exception
URL: https://github.com/apache/hudi/issues/13978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] HFile writer throws BufferOverflowException exception [hudi]
Posted by "DanielFrantes (via GitHub)" <gi...@apache.org>.
DanielFrantes commented on issue #13978:
URL: https://github.com/apache/hudi/issues/13978#issuecomment-3341799376
@linliu-code hey this is resolved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org