You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@avro.apache.org by "Martin Tzvetanov Grigorov (Jira)" <ji...@apache.org> on 2023/03/02 13:27:00 UTC
[jira] [Commented] (AVRO-3719) OutOfMemoryError issue in BinaryEncoder - EncoderFactory
[ https://issues.apache.org/jira/browse/AVRO-3719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17695716#comment-17695716 ]
Martin Tzvetanov Grigorov commented on AVRO-3719:
-------------------------------------------------
What exactly is `map` ?
How big is it ?
How much memory did you give to your java process ?
Did you try with 1.11.1 ?
> OutOfMemoryError issue in BinaryEncoder - EncoderFactory
> --------------------------------------------------------
>
> Key: AVRO-3719
> URL: https://issues.apache.org/jira/browse/AVRO-3719
> Project: Apache Avro
> Issue Type: Bug
> Components: java
> Affects Versions: 1.11.1
> Reporter: Sandeep Rathinam
> Priority: Major
> Fix For: 1.12.0
>
>
> The issue occurred during a performance test of an application which reads the message send from the kafka topic, batches and compress in to avro message. Below is the snippet of code used to do the process mentioned above.
> public <T> byte[] encode(T map) {
> DatumWriter<T> mapWriter = new SpecificDatumWriter<>(schema);
> BinaryEncoder binaryEncoder;
> try (var out = new ByteArrayOutputStream()) {
> binaryEncoder = EncoderFactory.get().binaryEncoder(out, null);
> mapWriter.write(map, binaryEncoder);
> binaryEncoder.flush();
> return out.toByteArray();
> } catch (IOException e) {
> throw new GenericException(e.getMessage(), e);
> }
> }
> Exception message:
> java.lang.OutOfMemoryError: Java heap space
> at java.base/java.util.Arrays.copyOf(Arrays.java:3537) ~[na:na]
> at java.base/java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:100) ~[na:na]
> at java.base/java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:130) ~[na:na]
> at org.apache.avro.io.BufferedBinaryEncoder$OutputStreamSink.innerWrite(BufferedBinaryEncoder.java:227) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.io.BufferedBinaryEncoder.flushBuffer(BufferedBinaryEncoder.java:96) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.io.BufferedBinaryEncoder.ensureBounds(BufferedBinaryEncoder.java:115) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.io.BufferedBinaryEncoder.writeInt(BufferedBinaryEncoder.java:130) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.io.BinaryEncoder.writeBytes(BinaryEncoder.java:77) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.io.BinaryEncoder.writeString(BinaryEncoder.java:46) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.io.Encoder.writeString(Encoder.java:128) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.writeString(GenericDatumWriter.java:346) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.specific.SpecificDatumWriter.writeString(SpecificDatumWriter.java:72) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:151) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:110) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:84) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.writeArray(GenericDatumWriter.java:257) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:137) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:110) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:84) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83) ~[avro-1.11.0.jar!/:1.11.0]
> at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73) ~[avro-1.11.0.jar!/:1.11.0]
> at com.vi.edge.avro.AvroEncoder.encode(AvroEncoder.java:71) ~[shared-library-0.0.1-SNAPSHOT.jar!/:0.0.1-SNAPSHOT]
> at com.vi.edge.avro.AvroEncoder.encodeAndCompress(AvroEncoder.java:83) ~[shared-library-0.0.1-SNAPSHOT.jar!/:0.0.1-SNAPSHOT]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)