[jira] [Commented] (FLINK-9300) Improve error message when in-memory state is too large

["Stephan Ewen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-9300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16466227#comment-16466227 ] 

Stephan Ewen commented on FLINK-9300:
-------------------------------------

This should actually not happen with the FsStateBackend. The message occurs when "in-line" state (i.e. state stored in the state handled, persisted with the metadata) grows too large.

The FsStateBackend writes state on checkpoints directly to the file system and communicate only the paths in the state handles.

This looks like there is a case where either the configuration is not properly propagated (and the MemoryStateBackend is used after all), or some code path uses the wrong type of state handle.

Can you share the stack trace of the original problem?

> Improve error message when in-memory state is too large
> -------------------------------------------------------
>
>                 Key: FLINK-9300
>                 URL: https://issues.apache.org/jira/browse/FLINK-9300
>             Project: Flink
>          Issue Type: Improvement
>    Affects Versions: 1.4.2
>            Reporter: Ken Krugler
>            Assignee: vinoyang
>            Priority: Minor
>
> Currently in the {{MemCheckpointStreamFactory.checkSize()}} method, it can throw an {{IOException}} via:
> {code:java}
> throw new IOException(
> "Size of the state is larger than the maximum permitted memory-backed state. Size="
> + size + " , maxSize=" + maxSize
> + " . Consider using a different state backend, like the File System State backend.");{code}
> But this will happen even if you’re using the File System State backend.
> This came up here: [https://stackoverflow.com/questions/50149005/ioexception-size-of-the-state-is-larger-than-the-maximum-permitted-memory-backe]
> We could change the message to be:
> {quote}Please consider increasing the maximum permitted memory size, increasing the task manager parallelism, or using a non-memory-based state backend such as RocksDB.
> {quote}
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)