You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "DuanRuixiao (via GitHub)" <gi...@apache.org> on 2026/08/03 20:23:09 UTC
[I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
DuanRuixiao opened a new issue, #17501:
URL: https://github.com/apache/iceberg/issues/17501
### Apache Iceberg version
1.11.0 (latest release)
### Query engine
Snowflake
### Please describe the bug 🐞
## Bug description
`ValueWriters.FixedByteBufferWriter.write(ByteBuffer, Encoder)` calls `encoder.writeBytes(bytes)`
instead of `encoder.writeFixed(arr, 0, length)`. For Avro `fixed[N]` fields this is wrong:
`writeBytes` prepends a zigzag-encoded length (e.g. `0x06` for a 3-byte value) before the data,
but the Avro reader consumes exactly N bytes for a `fixed` field and has no length prefix to skip.
The stray prefix byte spills into the next field in the record.
**Affected version:** 1.11.0 (also present in prior versions)
## Where the bug is
`core/src/main/java/org/apache/iceberg/avro/ValueWriters.java`, inner class `FixedByteBufferWriter`:
```java
private static class FixedByteBufferWriter implements ValueWriter<ByteBuffer> {
private final int length;
@Override
public void write(ByteBuffer bytes, Encoder encoder) throws IOException {
Preconditions.checkArgument(bytes.remaining() == length, ...);
encoder.writeBytes(bytes); // ← BUG: should be writeFixed
}
}
```
The sibling class `FixedWriter` (which handles `byte[]`) already does it correctly:
```java
private static class FixedWriter implements ValueWriter<byte[]> {
@Override
public void write(byte[] bytes, Encoder encoder) throws IOException {
Preconditions.checkArgument(bytes.length == length, ...);
encoder.writeFixed(bytes); // ← correct
}
}
```
## Concrete corruption example
Writing a `FIXED(3)` partition value `[AB CD EF]` via `FixedByteBufferWriter`:
| Encoding | Output bytes | What the reader sees |
|---|---|---|
| `writeBytes` (current) | `06 AB CD EF` | `fixed[3]` consumes `06 AB CD`; `EF` spills into the next field |
| `writeFixed` (correct) | `AB CD EF` | `fixed[3]` consumes `AB CD EF` ✓ |
When the spill byte `EF` prefixes the subsequent `record_count` varint `08` (zigzag for 4 rows),
the combined varint `EF 08` decodes to raw value `1135`, which zigzag-decodes to `-568`. Every
reader (Java, native, Spark) trusts this value from the manifest and sees `record_count = -568`
for a file that has 4 rows.
## How to reproduce
1. Create a partitioned Iceberg table with a `FIXED(3)` identity-partition column.
2. Write a data file via the Iceberg Java writer (triggers `InternalWriter` → `FixedByteBufferWriter`).
3. Read the written manifest Avro directly and inspect the `data_file.record_count` field for the
partition entry — it will be a large negative number instead of the actual row count.
4. Alternatively: observe that `data_file.partition.val8` in the manifest contains `06abcd` instead
of `abcdef` for a `[AB CD EF]` partition value.
## Proposed fix
In `FixedByteBufferWriter.write()`, extract the bytes from the `ByteBuffer` and call
`encoder.writeFixed()`:
```java
@Override
public void write(ByteBuffer bytes, Encoder encoder) throws IOException {
Preconditions.checkArgument(bytes.remaining() == length,
"Cannot write byte buffer of length %s as fixed[%s]", bytes.remaining(), length);
byte[] arr = new byte[length];
bytes.duplicate().get(arr);
encoder.writeFixed(arr);
}
```
A regression test should assert that a round-trip write/read of a manifest entry with a `FIXED(N)`
identity-partition column preserves the exact partition value bytes and the correct `record_count`.
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "DuanRuixiao (via GitHub)" <gi...@apache.org>.
DuanRuixiao commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5546484363
As this is definitely a breaking change, I talked with @huaxingao offline, and we decided to merge this fix to Iceberg V4.
We'll need to wait for Iceberg V4 branch cut and merge the fix. For now, there's no action item.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "DuanRuixiao (via GitHub)" <gi...@apache.org>.
DuanRuixiao commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5171450809
Proposed a PR for fix: https://github.com/apache/iceberg/pull/17502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "DuanRuixiao (via GitHub)" <gi...@apache.org>.
DuanRuixiao commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5206018798
@amitpoorab Thanks for that! I believe since this is not a backward compatible change and will break future read on existing data. We'll need to figure out a smooth way before merging this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "amitpoorab (via GitHub)" <gi...@apache.org>.
amitpoorab commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5199132459
I'd like to help move this forward, should I test it locally or help with any remaining review feedback?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "DuanRuixiao (via GitHub)" <gi...@apache.org>.
DuanRuixiao commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5273778761
I have added backward compatibility support in https://github.com/apache/iceberg/pull/17502
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "DuanRuixiao (via GitHub)" <gi...@apache.org>.
DuanRuixiao commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5185993926
Thanks @waterWang! Feel free to continue with your PR but we probably need to think about release. This is a breaking change so it might impact any new read on old write. We probably need to figure out a way before merging this fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "amitpoorab (via GitHub)" <gi...@apache.org>.
amitpoorab commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5233293714
Thanks @DuanRuixiao for the feedback! You're right about backward compatibility.
What approach do you think we need, do we need brainstorming? ?
Options I can think of:
1. Detect and support both formats (auto-detect on read) - could be harder and would not be clean.
2. Document as a breaking change requiring data re-write - simple, not clean as users need to read the docs.
3. Add a migration tool to convert old files to new format .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "DuanRuixiao (via GitHub)" <gi...@apache.org>.
DuanRuixiao commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5349341457
Now the blocker is the backward compatibility for both read and write. My proposed solution works fine in the case where new reader reads either old writer or new writer. However, I can't figure out a clean way for old reader reading new writer. It has to be a breaking change release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "DuanRuixiao (via GitHub)" <gi...@apache.org>.
DuanRuixiao commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5618834786
> Thank you. Is there a list we are tracking? How do we make sure this
> doesn’t slips through the cracks.
>
>
> Regards,
> Amit
> […](#)
I will work closely with @huaxingao to make sure this goes into Iceberg V4. Once the V4 release branch is cut or finalized, I'll port over the fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "DuanRuixiao (via GitHub)" <gi...@apache.org>.
DuanRuixiao commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5258873781
@amitpoorab Thanks for the brainstorm. I believe we could probably do following things:
1. Version-aware reader (must ship first). Before the corrected writer is enabled by default, the reader needs to handle both encodings. A table property (e.g., `write.avro.fixed-encoding`) or manifest-level flag can serve as the discriminator.
2. Tie the corrected writer to new format version. This ensures the discriminator is always present and inspectable by the reader. Old writers produce old manifests; new writers produce new manifests. Both coexist safely during a rolling upgrade.
3. Provide a manifest rewrite migration procedure. Long-lived tables accumulate manifests that won't naturally be replaced. A `rewriteManifests()`-style procedure lets users actively migrate existing data to the corrected encoding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "waterWang (via GitHub)" <gi...@apache.org>.
waterWang commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5174860659
Submitted a fix in PR #17505 — changed `encoder.writeBytes(bytes)` to convert the ByteBuffer to a byte[] and call `encoder.writeFixed(arr)` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Bug: `ValueWriters.FixedByteBufferWriter` uses `writeBytes()` instead of `writeFixed()` for Avro `fixed[N]` fields [iceberg]
Posted by "amitpoorab (via GitHub)" <gi...@apache.org>.
amitpoorab commented on issue #17501:
URL: https://github.com/apache/iceberg/issues/17501#issuecomment-5548448942
Thank you. Is there a list we are tracking? How do we make sure this
doesn’t slips through the cracks.
Regards,
Amit
On Friday, September 4, 2026, Ruixiao Duan ***@***.***> wrote:
> *DuanRuixiao* left a comment (apache/iceberg#17501)
> <https://github.com/apache/iceberg/issues/17501#issuecomment-5546484363>
>
> As this is definitely a breaking change, I talked with @huaxingao
> <https://github.com/huaxingao> offline, and we decided to merge this fix
> to Iceberg V4.
> We'll need to wait for Iceberg V4 branch cut and merge the fix. For now,
> there's no action item.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/iceberg/issues/17501?email_source=notifications&email_token=AEOX5VQEJJO5NG37BXRTVW35NMVURA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJUGY2DQNBTGYZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-5546484363>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AEOX5VWZ7JAL2QJZKZV7GLL5NMVURAVCNFSNUABFKJSXA33TNF2G64TZHMYTKOBSGU3DINZZHNEXG43VMU5TKMBVGM4DSMZWGEZ2C5QC>
> .
> Triage notifications, keep track of coding agent tasks and review pull
> requests on the go with GitHub Mobile for iOS
> <https://github.com/notifications/mobile/ios/AEOX5VRXPOVMSAR6YKYRFW35NMVURA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJUGY2DQNBTGYZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJKTGN5XXIZLSL5UW64Y>
> and Android
> <https://github.com/notifications/mobile/android/AEOX5VQOWHNM5PQKLAMUHC35NMVURA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKNJUGY2DQNBTGYZ2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLTGN5XXIZLSL5QW4ZDSN5UWI>.
> Download it today!
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org