You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@tez.apache.org by "Jonathan Eagles (JIRA)" <ji...@apache.org> on 2016/05/11 21:43:13 UTC
[jira] [Resolved] (TEZ-3242) Reduce bytearray copy with TezEvent
Serialization and deserialization
[ https://issues.apache.org/jira/browse/TEZ-3242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles resolved TEZ-3242.
----------------------------------
Resolution: Fixed
Fix Version/s: 0.9.0
Thanks [~rohini]. Committed this to master, branch-0.8, and branch-0.7.
> Reduce bytearray copy with TezEvent Serialization and deserialization
> ---------------------------------------------------------------------
>
> Key: TEZ-3242
> URL: https://issues.apache.org/jira/browse/TEZ-3242
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.7.2, 0.9.0, 0.8.4
>
> Attachments: TEZ-3242-1.patch, TEZ-3242.1-branch-0.7.patch
>
>
> Byte arrays are created for serializing protobuf messages and parsing them which creates lot of garbage when we have lot of events.
> {code}
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:3236)
> at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
> at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
> at org.apache.tez.runtime.api.impl.TezEvent.serializeEvent(TezEvent.java:197)
> at org.apache.tez.runtime.api.impl.TezEvent.write(TezEvent.java:268)
> at org.apache.tez.runtime.api.impl.TezHeartbeatResponse.write(TezHeartbeatResponse.java:95)
> at org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:202)
> at org.apache.hadoop.io.ObjectWritable.writeObject(ObjectWritable.java:128)
> at org.apache.hadoop.io.ObjectWritable.write(ObjectWritable.java:82)
> at org.apache.hadoop.ipc.Server.setupResponse(Server.java:2496)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)