You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Yun Tang (JIRA)" <ji...@apache.org> on 2018/12/09 17:01:00 UTC
[jira] [Updated] (FLINK-11107) [state] Avoid memory stateBackend to
create arbitrary folders under HA path when no checkpoint path configured
[ https://issues.apache.org/jira/browse/FLINK-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yun Tang updated FLINK-11107:
-----------------------------
Description:
Currently, memory state-backend would create a folder named with random UUID under HA directory if no checkpoint path ever configured. (the code logic locates within {{StateBackendLoader#fromApplicationOrConfigOrDefault}}) However, the default memory state-backend would not only be created on JM side, but also on each task manager's side, which means many folders with random UUID would be created under HA directory. It would result in exception like:
{noformat}
The directory item limit of /tmp/flink/ha is exceeded: limit=1048576 items=1048576{noformat}
If this happens, no new jobs could be submitted only if we clean up those directories manually.
was:
Currently, memory state-backend would create a folder named with random UUID under HA directory if no checkpoint path ever configured. (the code logic locates within {{StateBackendLoader#fromApplicationOrConfigOrDefault}}) However, the default memory state-backend would not only be created on JM side, but also on each task manager's side, which means many folders with random UUID would be created under HA directory. It would result in exception like:
{noformat}
The directory item limit of /tmp/flink/ha is exceeded: limit=1048576 items=1048576{noformat}
If this happens, no new jobs could be submitted only if we do not clean up those directories manually.
> [state] Avoid memory stateBackend to create arbitrary folders under HA path when no checkpoint path configured
> --------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-11107
> URL: https://issues.apache.org/jira/browse/FLINK-11107
> Project: Flink
> Issue Type: Bug
> Components: State Backends, Checkpointing
> Affects Versions: 1.6.2, 1.7.0
> Reporter: Yun Tang
> Assignee: Yun Tang
> Priority: Major
> Fix For: 1.6.3, 1.7.1
>
>
> Currently, memory state-backend would create a folder named with random UUID under HA directory if no checkpoint path ever configured. (the code logic locates within {{StateBackendLoader#fromApplicationOrConfigOrDefault}}) However, the default memory state-backend would not only be created on JM side, but also on each task manager's side, which means many folders with random UUID would be created under HA directory. It would result in exception like:
> {noformat}
> The directory item limit of /tmp/flink/ha is exceeded: limit=1048576 items=1048576{noformat}
> If this happens, no new jobs could be submitted only if we clean up those directories manually.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)