[jira] [Created] (FLINK-11695) [checkpoint] Make sharedStateDir could create sub-directories to avoid MaxDirectoryItemsExceededException

["Yun Tang (JIRA)" <ji...@apache.org>]

Yun Tang created FLINK-11695:
--------------------------------

             Summary: [checkpoint] Make sharedStateDir could create sub-directories to avoid MaxDirectoryItemsExceededException
                 Key: FLINK-11695
                 URL: https://issues.apache.org/jira/browse/FLINK-11695
             Project: Flink
          Issue Type: Improvement
          Components: State Backends, Checkpointing
            Reporter: Yun Tang
            Assignee: Yun Tang


We meet this annoying problem many times when the {{sharedStateDir}} in checkpoint path exceeded the directory items limit due to large checkpoint:
{code:java}
org.apache.hadoop.hdfs.protocol.FSLimitException$MaxDirectoryItemsExceededException: The directory item limit of xxx is exceeded: limit=1048576 items=1048576 
{code}
Currently, our solution is to let {{FsCheckpointStorage}} could create sub-dirs when calling {{resolveCheckpointStorageLocation}}. The default value for the number of sub-dirs is zero, just keep backward compatibility as current situation. The created sub-dirs have the name as integer value of \{{0 ~ num-of-sub-dirs }}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)