You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by zhangminglei <gi...@git.apache.org> on 2017/06/29 05:58:05 UTC
[GitHub] flink pull request #4219: [FLINK-6842] [runtime] Uncomment and activate code...
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/4219
[FLINK-6842] [runtime] Uncomment and activate code in HadoopFileSystem
Uncomment and activate ```getFileSystemClass``` method in HadoopFileSystem.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink-6842
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4219.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4219
----
commit 91a7e6a14e827fe1c071fedd6ba3a37c7402b1f4
Author: zhangminglei <zm...@163.com>
Date: 2017-06-29T05:50:51Z
[FLINK-6842] [runtime] Uncomment and activate code in HadoopFileSystem
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4219: [FLINK-6842] [runtime] Uncomment and activate code in Had...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/4219
+1
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4219: [FLINK-6842] [runtime] Uncomment and activate code...
Posted by zhangminglei <gi...@git.apache.org>.
Github user zhangminglei commented on a diff in the pull request:
https://github.com/apache/flink/pull/4219#discussion_r125896588
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFileSystem.java ---
@@ -459,17 +459,14 @@ public boolean isDistributedFS() {
@Override
public Class<?> getHadoopWrapperClassNameForFileSystem(String scheme) {
Configuration hadoopConf = getHadoopConfiguration();
- Class<? extends org.apache.hadoop.fs.FileSystem> clazz;
- // We can activate this block once we drop Hadoop1 support (only hd2 has the getFileSystemClass-method)
-// try {
-// clazz = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConf);
-// } catch (IOException e) {
-// LOG.info("Flink could not load the Hadoop File system implementation for scheme "+scheme);
-// return null;
-// }
- clazz = hadoopConf.getClass("fs." + scheme + ".impl", null, org.apache.hadoop.fs.FileSystem.class);
-
- if (clazz != null && LOG.isDebugEnabled()) {
+ Class<? extends org.apache.hadoop.fs.FileSystem> clazz = null;
+ try {
+ clazz = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConf);
+ } catch (IOException e) {
+ LOG.info("Flink could not load the Hadoop File system implementation for scheme " + scheme);
+ }
+
+ if(LOG.isDebugEnabled()) {
--- End diff --
Since clazz is null here is unreachable. ```getFileSystemClass``` will check null condition. And if ```clazz``` is null, ```getFileSystemClass``` will give a ```throw new IOException("No FileSystem for scheme: " + scheme); ``` instead. Another question, you said this style ? like ```if( LOG.isDebugEnabled() )``` ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4219: [FLINK-6842] [runtime] Uncomment and activate code...
Posted by zhangminglei <gi...@git.apache.org>.
Github user zhangminglei commented on a diff in the pull request:
https://github.com/apache/flink/pull/4219#discussion_r125926605
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFileSystem.java ---
@@ -459,17 +459,14 @@ public boolean isDistributedFS() {
@Override
public Class<?> getHadoopWrapperClassNameForFileSystem(String scheme) {
Configuration hadoopConf = getHadoopConfiguration();
- Class<? extends org.apache.hadoop.fs.FileSystem> clazz;
- // We can activate this block once we drop Hadoop1 support (only hd2 has the getFileSystemClass-method)
-// try {
-// clazz = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConf);
-// } catch (IOException e) {
-// LOG.info("Flink could not load the Hadoop File system implementation for scheme "+scheme);
-// return null;
-// }
- clazz = hadoopConf.getClass("fs." + scheme + ".impl", null, org.apache.hadoop.fs.FileSystem.class);
-
- if (clazz != null && LOG.isDebugEnabled()) {
+ Class<? extends org.apache.hadoop.fs.FileSystem> clazz = null;
+ try {
+ clazz = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConf);
+ } catch (IOException e) {
+ LOG.info("Flink could not load the Hadoop File system implementation for scheme " + scheme);
+ }
+
+ if(LOG.isDebugEnabled()) {
--- End diff --
Had updated code. Well fine ~ :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4219: [FLINK-6842] [runtime] Uncomment and activate code in Had...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/4219
The call to `hadoopConf.getClass` is no longer needed and the early `return null` could be removed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4219: [FLINK-6842] [runtime] Uncomment and activate code...
Posted by zhangminglei <gi...@git.apache.org>.
Github user zhangminglei commented on a diff in the pull request:
https://github.com/apache/flink/pull/4219#discussion_r125925167
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFileSystem.java ---
@@ -459,17 +459,14 @@ public boolean isDistributedFS() {
@Override
public Class<?> getHadoopWrapperClassNameForFileSystem(String scheme) {
Configuration hadoopConf = getHadoopConfiguration();
- Class<? extends org.apache.hadoop.fs.FileSystem> clazz;
- // We can activate this block once we drop Hadoop1 support (only hd2 has the getFileSystemClass-method)
-// try {
-// clazz = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConf);
-// } catch (IOException e) {
-// LOG.info("Flink could not load the Hadoop File system implementation for scheme "+scheme);
-// return null;
-// }
- clazz = hadoopConf.getClass("fs." + scheme + ".impl", null, org.apache.hadoop.fs.FileSystem.class);
-
- if (clazz != null && LOG.isDebugEnabled()) {
+ Class<? extends org.apache.hadoop.fs.FileSystem> clazz = null;
+ try {
+ clazz = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConf);
+ } catch (IOException e) {
+ LOG.info("Flink could not load the Hadoop File system implementation for scheme " + scheme);
+ }
+
+ if(LOG.isDebugEnabled()) {
--- End diff --
Thank you so much. Greg is correct! Update code soon enough.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4219: [FLINK-6842] [runtime] Uncomment and activate code...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/4219#discussion_r125902757
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFileSystem.java ---
@@ -459,17 +459,14 @@ public boolean isDistributedFS() {
@Override
public Class<?> getHadoopWrapperClassNameForFileSystem(String scheme) {
Configuration hadoopConf = getHadoopConfiguration();
- Class<? extends org.apache.hadoop.fs.FileSystem> clazz;
- // We can activate this block once we drop Hadoop1 support (only hd2 has the getFileSystemClass-method)
-// try {
-// clazz = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConf);
-// } catch (IOException e) {
-// LOG.info("Flink could not load the Hadoop File system implementation for scheme "+scheme);
-// return null;
-// }
- clazz = hadoopConf.getClass("fs." + scheme + ".impl", null, org.apache.hadoop.fs.FileSystem.class);
-
- if (clazz != null && LOG.isDebugEnabled()) {
+ Class<? extends org.apache.hadoop.fs.FileSystem> clazz = null;
+ try {
+ clazz = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConf);
+ } catch (IOException e) {
+ LOG.info("Flink could not load the Hadoop File system implementation for scheme " + scheme);
+ }
+
+ if(LOG.isDebugEnabled()) {
--- End diff --
Checkstyle is `if (...`. When the `IOException` is caught `clazz` will still be `null`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4219: [FLINK-6842] [runtime] Uncomment and activate code in Had...
Posted by zhangminglei <gi...@git.apache.org>.
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/4219
Thanks, Greg.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4219: [FLINK-6842] [runtime] Uncomment and activate code in Had...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on the issue:
https://github.com/apache/flink/pull/4219
Thanks for the contribution @zhangminglei. Merging ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4219: [FLINK-6842] [runtime] Uncomment and activate code...
Posted by greghogan <gi...@git.apache.org>.
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/4219#discussion_r125893611
--- Diff: flink-runtime/src/main/java/org/apache/flink/runtime/fs/hdfs/HadoopFileSystem.java ---
@@ -459,17 +459,14 @@ public boolean isDistributedFS() {
@Override
public Class<?> getHadoopWrapperClassNameForFileSystem(String scheme) {
Configuration hadoopConf = getHadoopConfiguration();
- Class<? extends org.apache.hadoop.fs.FileSystem> clazz;
- // We can activate this block once we drop Hadoop1 support (only hd2 has the getFileSystemClass-method)
-// try {
-// clazz = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConf);
-// } catch (IOException e) {
-// LOG.info("Flink could not load the Hadoop File system implementation for scheme "+scheme);
-// return null;
-// }
- clazz = hadoopConf.getClass("fs." + scheme + ".impl", null, org.apache.hadoop.fs.FileSystem.class);
-
- if (clazz != null && LOG.isDebugEnabled()) {
+ Class<? extends org.apache.hadoop.fs.FileSystem> clazz = null;
+ try {
+ clazz = org.apache.hadoop.fs.FileSystem.getFileSystemClass(scheme, hadoopConf);
+ } catch (IOException e) {
+ LOG.info("Flink could not load the Hadoop File system implementation for scheme " + scheme);
+ }
+
+ if(LOG.isDebugEnabled()) {
--- End diff --
Why was the null check on `clazz` removed? And for the new checkstyle (which is not yet enforced in flink-runtime) we'll want a space separating `if` and the opening parenthesis.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4219: [FLINK-6842] [runtime] Uncomment and activate code in Had...
Posted by zhangminglei <gi...@git.apache.org>.
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/4219
Thanks @greghogan, and PR have been updated, please check again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4219: [FLINK-6842] [runtime] Uncomment and activate code...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4219
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---