[jira] [Commented] (FLINK-6647) Fail-fast on invalid RocksDBStateBackend configuration

["Helder Pereira (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-6647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017687#comment-16017687 ] 

Helder Pereira commented on FLINK-6647:
---------------------------------------

The {{File.pathSeparator}} seems a bit dangerous here, as it will strip out the schema and consider it an alternative path:
https://github.com/apache/flink/blob/master/flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackendFactory.java#L64

And there seems to be a mismatch with what is expected here, which is prepared to get paths with schema:
https://github.com/apache/flink/blob/master/flink-contrib/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java#L367

> Fail-fast on invalid RocksDBStateBackend configuration
> ------------------------------------------------------
>
>                 Key: FLINK-6647
>                 URL: https://issues.apache.org/jira/browse/FLINK-6647
>             Project: Flink
>          Issue Type: Bug
>            Reporter: Andrey
>
> Currently:
> * setup "state.backend.rocksdb.checkpointdir=hdfs:///some/base/path/hdfs"
> * setup backend: state.backend: "org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory"
> * rocksdb doesn't support hdfs backend so in logs:
> {code}
> 2017-05-19 15:42:33,737 ERROR org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Local DB files directory '/some/base/path/hdfs' does not exist and cannot be created.
> {code}
> * however job continue execution and IOManager temp directory will be picked up for rocksdb files.
> There are several issues with such approach:
> * after "ERROR" message printed and before developer fixes configuration, /tmp directory/partition might run out of disk space.
> * if hdfs base path is the same as local path, then no errors in logs and rocksdb files will be written into an incorrect location. For example: "hdfs:///home/flink/data" will cause an issue.
> Expected:
> * validate URI and throw IllegalArgumentException like already implemented in "RocksDBStateBackend.setDbStoragePaths" method.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)