You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2016/06/23 23:04:16 UTC
[jira] [Commented] (FLINK-4115) FsStateBackend filesystem
verification can cause classpath exceptions
[ https://issues.apache.org/jira/browse/FLINK-4115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15347390#comment-15347390 ]
ASF GitHub Bot commented on FLINK-4115:
---------------------------------------
GitHub user jfg9 opened a pull request:
https://github.com/apache/flink/pull/2157
[FLINK-4115] FsStateBackend filesystem verification can cause classpath exceptions
There are two changes to FsStateBackend to avoid classpath exceptions when submitting a job from a Flink client which does not have the necessary file system classes on its classpath:
- The filesystem is no longer initialised in the FsStateBackend constructor
- The verification checks for the checkpoint directory in the FsStateBackend constructor are skipped if Flink does not have built-in support for the URI scheme
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jfg9/flink master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/2157.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2157
----
commit e8557895b4ff8e40a0705152ec65d355fe7d5f65
Author: Josh <jo...@permutive.com>
Date: 2016-06-23T21:18:02Z
[FLINK-4115] Skip filesystem checks for filesystems with no built-in support
commit 94ec8dca6c016c1c500ade90b4c823e5faaecaf3
Author: Josh <jo...@permutive.com>
Date: 2016-06-23T21:22:40Z
[FLINK-4115] Remove filesystem initialisation from FsStateBackend constructor
----
> FsStateBackend filesystem verification can cause classpath exceptions
> ---------------------------------------------------------------------
>
> Key: FLINK-4115
> URL: https://issues.apache.org/jira/browse/FLINK-4115
> Project: Flink
> Issue Type: Bug
> Components: Local Runtime
> Affects Versions: 1.1.0
> Reporter: Josh Forman-Gornall
> Priority: Minor
>
> In the constructor of FsStateBackend, the FileSystem for the checkpoint directory is initialised and it is verified that the checkpoint path exists. This verification happens in the Flink client program when submitting a job and can cause classpath issues if classes required to access the file system are not available in the client's classpath.
> For example, if we run Flink on YARN over AWS EMR using RocksDBStateBackend and an s3:// checkpoint directory, we get the below ClassNotFoundException. This is because the jars needed to use the EMR file system are available only in the YARN context and not when submitting the job via the Flink client.
> {noformat}
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found
> at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2227)
> at org.apache.flink.runtime.fs.hdfs.HadoopFileSystem.getHadoopWrapperClassNameForFileSystem(HadoopFileSystem.java:460)
> at org.apache.flink.core.fs.FileSystem.getHadoopWrapperClassNameForFileSystem(FileSystem.java:352)
> at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:280)
> at org.apache.flink.runtime.state.filesystem.FsStateBackend.validateAndNormalizeUri(FsStateBackend.java:383)
> at org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:175)
> at org.apache.flink.runtime.state.filesystem.FsStateBackend.<init>(FsStateBackend.java:144)
> at org.apache.flink.contrib.streaming.state.RocksDBStateBackend.<init>(RocksDBStateBackend.java:205)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)