You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "Qi (JIRA)" <ji...@apache.org> on 2018/11/20 05:05:00 UTC

[jira] [Created] (FLINK-10941) Slots prematurely released which still contain unconsumed data

Qi created FLINK-10941:
--------------------------

             Summary: Slots prematurely released which still contain unconsumed data 
                 Key: FLINK-10941
                 URL: https://issues.apache.org/jira/browse/FLINK-10941
             Project: Flink
          Issue Type: Bug
          Components: ResourceManager
    Affects Versions: 1.5.5
            Reporter: Qi


Our case is: Flink 1.5 batch mode, 32 parallelism to read data source and 4 parallelism to write data sink.
 
The read task worked perfectly with 32 TMs. However when the job was executing the write task, since only 4 TMs were needed, other 28 TMs were released. This caused RemoteTransportException in the write task:
 
org.apache.flink.runtime.io.network.netty.exception.RemoteTransportException: Connection unexpectedly closed by remote task manager ’the_previous_TM_used_by_read_task'. This might indicate that the remote task manager was lost.
	at org.apache.flink.runtime.io.network.netty.PartitionRequestClientHandler.channelInactive(PartitionRequestClientHandler.java:133)
	at org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:237)
	...
 
After skimming YarnFlinkResourceManager related code, it seems to me that Flink is releasing TMs when they’re idle, regardless of whether working TMs need them.
 
Put in another way, Flink seems to prematurely release slots which contain unconsumed data and, thus, eventually release a TM which then fails a consuming task.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)