You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by zhangminglei <gi...@git.apache.org> on 2017/07/07 14:41:59 UTC
[GitHub] flink pull request #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in Had...
GitHub user zhangminglei opened a pull request:
https://github.com/apache/flink/pull/4285
[FLINK-7118] [hadoop] Remove hadoop1.x code in HadoopUtils
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/zhangminglei/flink flink-7118
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4285.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4285
----
commit cf9e267c99361fe4c69fff824e3a0eb11b62d32a
Author: zhangminglei <zm...@163.com>
Date: 2017-07-07T14:40:09Z
[FLINK-7118] [hadoop] Remove hadoop1.x code in HadoopUtils
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in Had...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/4285
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in HadoopUtil...
Posted by zhangminglei <gi...@git.apache.org>.
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/4285
@StephanEwen PR have been updated. Please check it out again ~
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in Had...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/4285#discussion_r127992635
--- Diff: flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java ---
@@ -126,18 +128,18 @@ public void open(int taskNumber, int numTasks) throws IOException {
this.jobConf.setInt("mapreduce.task.partition", taskNumber + 1);
try {
- this.context = HadoopUtils.instantiateTaskAttemptContext(this.jobConf, taskAttemptID);
+ this.context = new TaskAttemptContextImpl(this.jobConf, taskAttemptID);
} catch (Exception e) {
- throw new RuntimeException(e);
+ throw new IOException("Could not create instance of TaskAttemptContext.", e);
}
this.outputCommitter = this.jobConf.getOutputCommitter();
JobContext jobContext;
try {
- jobContext = HadoopUtils.instantiateJobContext(this.jobConf, new JobID());
+ jobContext = new JobContextImpl(this.jobConf, new JobID());
} catch (Exception e) {
- throw new RuntimeException(e);
+ throw new IOException("Could not create instance of JobContext.", e);
--- End diff --
Same as above...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in HadoopUtil...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4285
Since these utility methods are now so simple, I think it makes sense to inline them in the two places where they are called. Then we could also get rid of the extra exception catch blocks and avoid the extra wrapping into RuntimeExceptions.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in Had...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on a diff in the pull request:
https://github.com/apache/flink/pull/4285#discussion_r127992595
--- Diff: flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java ---
@@ -126,18 +128,18 @@ public void open(int taskNumber, int numTasks) throws IOException {
this.jobConf.setInt("mapreduce.task.partition", taskNumber + 1);
try {
- this.context = HadoopUtils.instantiateTaskAttemptContext(this.jobConf, taskAttemptID);
+ this.context = new TaskAttemptContextImpl(this.jobConf, taskAttemptID);
} catch (Exception e) {
- throw new RuntimeException(e);
+ throw new IOException("Could not create instance of TaskAttemptContext.", e);
--- End diff --
I think the `TaskAttemptContextImpl` construction does not throw any Exception, so we can drop the try/catch block around it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in HadoopUtil...
Posted by zhangminglei <gi...@git.apache.org>.
Github user zhangminglei commented on the issue:
https://github.com/apache/flink/pull/4285
Thanks, Stephan. I think it is a good idea. PR has been updated. :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in HadoopUtil...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4285
Actually, sorry, is it possible to replace the code with just a proper `new TaskAttemptContextImpl(...)` call, rather than using reflection?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in Had...
Posted by zhangminglei <gi...@git.apache.org>.
Github user zhangminglei commented on a diff in the pull request:
https://github.com/apache/flink/pull/4285#discussion_r128009928
--- Diff: flink-java/src/main/java/org/apache/flink/api/java/hadoop/mapred/HadoopOutputFormatBase.java ---
@@ -126,18 +128,18 @@ public void open(int taskNumber, int numTasks) throws IOException {
this.jobConf.setInt("mapreduce.task.partition", taskNumber + 1);
try {
- this.context = HadoopUtils.instantiateTaskAttemptContext(this.jobConf, taskAttemptID);
+ this.context = new TaskAttemptContextImpl(this.jobConf, taskAttemptID);
} catch (Exception e) {
- throw new RuntimeException(e);
+ throw new IOException("Could not create instance of TaskAttemptContext.", e);
--- End diff --
Yes. Stephan is correct!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in HadoopUtil...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/4285
Looks good, thanks, merging this...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink issue #4285: [FLINK-7118] [hadoop] Remove hadoop1.x code in HadoopUtil...
Posted by tillrohrmann <gi...@git.apache.org>.
Github user tillrohrmann commented on the issue:
https://github.com/apache/flink/pull/4285
Thanks for your contribution @zhangminglei. Changes look good to me. Merging this PR together with #4362.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---