You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Ruiqi Dong (Jira)" <ji...@apache.org> on 2026/06/05 05:47:00 UTC

[jira] [Created] (AVRO-4268) BytesWritableConverter serializes unused capacity bytes

Ruiqi Dong created AVRO-4268:
--------------------------------

             Summary: BytesWritableConverter serializes unused capacity bytes
                 Key: AVRO-4268
                 URL: https://issues.apache.org/jira/browse/AVRO-4268
             Project: Apache Avro
          Issue Type: Bug
          Components: java
            Reporter: Ruiqi Dong


*Summary*
`AvroDatumConverterFactory.BytesWritableConverter` converts Hadoop `BytesWritable` to Avro `bytes` using `ByteBuffer.wrap(input.getBytes())`. `BytesWritable.getBytes()` returns the backing array, whose length can be larger than the logical value length reported by `getLength()`. The converter therefore serializes stale or unused capacity bytes.
 
*Affected code*
File: `lang/java/mapred/src/main/java/org/apache/avro/hadoop/io/AvroDatumConverterFactory.java`
{code:java}
@Override
public ByteBuffer convert(BytesWritable input) {
  return ByteBuffer.wrap(input.getBytes());
} {code}
*Reproducer*
Add this test to `lang/java/mapred/src/test/java/org/apache/avro/hadoop/io/TestAvroDatumConverterFactory.java`
{code:java}
@Test
void convertBytesWritableRespectsLogicalLength() {
  AvroDatumConverter<BytesWritable, ByteBuffer> converter = mFactory.create(BytesWritable.class);
  BytesWritable writable = new BytesWritable(new byte[] { 1, 2, 3, 4, 5 });
  writable.setSize(3);

  ByteBuffer bytes = converter.convert(writable);

  assertEquals(3, bytes.remaining());
  assertEquals(1, bytes.get(0));
  assertEquals(2, bytes.get(1));
  assertEquals(3, bytes.get(2));
} {code}
Run:
{code:java}
MAVEN_SKIP_RC=true JAVA_HOME=/opt/homebrew/Cellar/openjdk@21/21.0.6/libexec/openjdk.jdk/Contents/Home \
PATH=/opt/homebrew/Cellar/openjdk@21/21.0.6/libexec/openjdk.jdk/Contents/Home/bin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin \
/opt/homebrew/bin/mvn -q -t toolchains-local.xml -pl lang/java/mapred -am \
  -Dtest=org.apache.avro.hadoop.io.TestAvroDatumConverterFactory#convertBytesWritableRespectsLogicalLength \
  -DfailIfNoTests=false -Dsurefire.failIfNoSpecifiedTests=false -Dinvoker.skip=true -Drat.skip=true test
 {code}
*Observed behavior*
The converted `ByteBuffer` has `remaining() == 5`.

*Expected behavior*
The converted `ByteBuffer` should have `remaining() == input.getLength()`, which is `3` in the reproducer.


Hadoop `BytesWritable` separates capacity from logical length. Avro `bytes` should encode only the logical value, not unused backing-array capacity. The likely fix is `ByteBuffer.wrap(input.copyBytes())` or `ByteBuffer.wrap(input.getBytes(), 0, input.getLength())`.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)