[jira] [Commented] (FLINK-7512) avoid unnecessary buffer copies during network serialization

["Bowen Li (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-7512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16142007#comment-16142007 ] 

Bowen Li commented on FLINK-7512:
---------------------------------

Hi [~NicoK],

I'm seeing the follow errors in our jobmanager.log


{code:java}
2017-08-25 17:26:32,015 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph        - Job com.offerup.stream_processing.item_view_stats.ItemViewStatsStreamingApp (edb5deab43903b425aba6537e752b700) switched from state RUNNING to FAILING.
java.lang.OutOfMemoryError: Java heap space
	at java.util.HashMap.resize(HashMap.java:703)
	at java.util.HashMap.putVal(HashMap.java:662)
	at java.util.HashMap.put(HashMap.java:611)
	at java.util.HashSet.add(HashSet.java:219)
	at org.apache.flink.streaming.api.operators.HeapInternalTimerService.restoreTimersForKeyGroup(HeapInternalTimerService.java:346)
	at org.apache.flink.streaming.api.operators.InternalTimeServiceManager.restoreStateForKeyGroup(InternalTimeServiceManager.java:141)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:496)
	at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.initializeState(AbstractUdfStreamOperator.java:104)
	at org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:251)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeOperators(StreamTask.java:675)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:662)
	at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:251)
	at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702)
	at java.lang.Thread.run(Thread.java:748)
{code}

It seems like the program was trying to do some buffer copy-over, and ran out of memory. Is the error related to this ticket?  


> avoid unnecessary buffer copies during network serialization
> ------------------------------------------------------------
>
>                 Key: FLINK-7512
>                 URL: https://issues.apache.org/jira/browse/FLINK-7512
>             Project: Flink
>          Issue Type: Improvement
>          Components: Network
>    Affects Versions: 1.4.0
>            Reporter: Nico Kruber
>            Assignee: Nico Kruber
>
> Currently, we have our own {{Buffer}} class backed by a {{MemorySegment}} and whenever we try to write to or read from Netty, we need to copy to / from Netty's {{ByteBuf}} instances.
> This is am umbrella task for avoiding these buffer copies and some related code changes



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)