You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 03:28:44 UTC
[I] IndexOutOfBoundsException in MessageColumnIORecordConsumer.addBinary when writing parquet [hudi]
hudi-bot opened a new issue, #14552:
URL: https://github.com/apache/hudi/issues/14552
Some writes fail with java.lang.IndexOutOfBoundsException : Invalid array range: X to X inside MessageColumnIORecordConsumer.addBinary call.
Specifically: getColumnWriter().write(value, r[currentLevel], currentColumnIO.getDefinitionLevel());
fails as size of r is the same as current level. What can be causing it?
It gets executed via: ParquetWriter.write(IndexedRecord) Library version: 1.10.1 Avro is a very complex object (~2.5k columns, highly nested, arrays of unions present).
But what is surprising is that it fails to write top level field: PrimitiveColumnIO _hoodie_commit_time r:0 d:1 [_hoodie_commit_time] which is the first top level field in Avro: {"_hoodie_commit_time": "20200317215711", "_hoodie_commit_seqno": "20200317215711_0_650",
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-722
- Type: Bug
- Affects version(s):
- 0.9.0
---
## Comments
22/Mar/20 17:34;vinoth;cc
@lamber-ken interested? :);;;
---
28/Mar/20 01:17;lamber-ken;Sure. Hi [~afilipchik], if you have time, can you share more context about it? e.g demo code, thanks.;;;
---
28/Mar/20 18:30;afilipchik;Sure. The context -> he have a very schema heavy stream, means schema has multiple levels, arrays of structs which have arrays of structs. I saw 2500 columns on parquet level.
We caught bunch of issues with avro aonversions with that stream. It works fine on 0.5, but when I tried to upgrade to 0.6 I got this error. The table type is MOR and INSERT.
If you want I we can do a f2f session (Zoom, Hangouts) as it will be easier to explain or even debug.;;;
---
01/Apr/20 17:24;lamber-ken;IMO, it's hard to debug this issue, because writes are not always fails. I am sorry that I can not do a f2f session because of the limit of network. :(;;;
---
08/Apr/20 16:12;vinoth;recently, we landed some fixes around nested data structures.. https://github.com/apache/incubator-hudi/pull/1406
[~afilipchik] it would be immensely helpful, if you could replicate the shape of the schema and give us a small failing test, given this is dependent on the shape of the data. ;;;
---
16/May/20 13:10;shivnarayan;[~afilipchik]: If you get a chance, would be of great help to give us a reproducible code. ;;;
---
02/Feb/21 16:09;shivnarayan;[~afilipchik]: just going through all tickets. Do you happened to have a reproducible code by any chance. ;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] IndexOutOfBoundsException in MessageColumnIORecordConsumer.addBinary when writing parquet [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14552:
URL: https://github.com/apache/hudi/issues/14552#issuecomment-4795188089
Closing for inactivity. JIRA: [HUDI-722](https://issues.apache.org/jira/browse/HUDI-722). This issue has had no activity for over two years and the GitHub thread has no replies. If this is still relevant, please open a fresh issue with reproduction details on a recent release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] IndexOutOfBoundsException in MessageColumnIORecordConsumer.addBinary when writing parquet [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14552: IndexOutOfBoundsException in MessageColumnIORecordConsumer.addBinary when writing parquet
URL: https://github.com/apache/hudi/issues/14552
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org