You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@avro.apache.org by "Sandeep Rathinam (Jira)" <ji...@apache.org> on 2023/03/02 10:16:00 UTC
[jira] [Created] (AVRO-3719) OutOfMemoryError issue in BinaryEncoder - EncoderFactory
Sandeep Rathinam created AVRO-3719:
--------------------------------------
Summary: OutOfMemoryError issue in BinaryEncoder - EncoderFactory
Key: AVRO-3719
URL: https://issues.apache.org/jira/browse/AVRO-3719
Project: Apache Avro
Issue Type: Bug
Components: java
Affects Versions: 1.11.1
Reporter: Sandeep Rathinam
Fix For: 1.12.0
The issue occurred during a performance test of an application which reads the message send from the kafka topic, batches and compress in to avro message. Below is the snippet of code used to do the process mentioned above.
public <T> byte[] encode(T map) {
DatumWriter<T> mapWriter = new SpecificDatumWriter<>(schema);
BinaryEncoder binaryEncoder;
try (var out = new ByteArrayOutputStream()) {
binaryEncoder = EncoderFactory.get().binaryEncoder(out, null);
mapWriter.write(map, binaryEncoder);
binaryEncoder.flush();
return out.toByteArray();
} catch (IOException e) {
throw new GenericException(e.getMessage(), e);
}
}
Exception message:
java.lang.OutOfMemoryError: Java heap space
at java.base/java.util.Arrays.copyOf(Arrays.java:3537) ~[na:na]
at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) ~[na:na]
at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) ~[na:na]
at org.apache.avro.io.BufferedBinaryEncoder$OutputStreamSink.innerWrite(BufferedBinaryEncoder.java:227) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.BufferedBinaryEncoder.flushBuffer(BufferedBinaryEncoder.java:96) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.BufferedBinaryEncoder.ensureBounds(BufferedBinaryEncoder.java:115) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.BufferedBinaryEncoder.writeInt(BufferedBinaryEncoder.java:130) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:77) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.BinaryEncoder.writeString(BinaryEncoder.java:46) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.io.Encoder.writeString(Encoder.java:128) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:346) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.specific.SpecificDatumWriter.writeString(SpecificDatumWriter.java:72) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:151) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:110) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:84) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:257) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:137) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:110) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:84) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) ~[avro-1.11.0.jar!/:1.11.0]
at com.vi.edge.avro.AvroEncoder.encode(AvroEncoder.java:71) ~[shared-library-0.0.1-SNAPSHOT.jar!/:0.0.1-SNAPSHOT]
at com.vi.edge.avro.AvroEncoder.encodeAndCompress(AvroEncoder.java:83) ~[shared-library-0.0.1-SNAPSHOT.jar!/:0.0.1-SNAPSHOT]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)