You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by rmetzger <gi...@git.apache.org> on 2014/12/05 20:01:08 UTC
[GitHub] incubator-flink pull request: [FLINK-1305] [FLINK-1304] Test for H...
GitHub user rmetzger opened a pull request:
https://github.com/apache/incubator-flink/pull/252
[FLINK-1305] [FLINK-1304] Test for HadoopInputWrapper and NullWritable support
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rmetzger/incubator-flink flink1304
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/252.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #252
----
commit 48778e83d2c04e2f65649b7c96f1aa92d4e1d350
Author: Robert Metzger <rm...@apache.org>
Date: 2014-12-05T18:19:29Z
[FLINK-1305] [FLINK-1304] Test for HadoopInputWrapper and NullWritable support
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1305] [FLINK-1304] Test for H...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/incubator-flink/pull/252#issuecomment-65959720
This change adds hadoop as a hard dependency to the `flink-java` project. Per the discussion on the mailing list, concerning support for Hadoop Writables, we voted to not do that and instead add an "mimick interface" to the Java API.
For a big change like reversing that, it would be good to have some reasons...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1305] [FLINK-1304] Test for H...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/252#issuecomment-66112318
The reason why I added the dependency to `hadoop-common` is that I need it to be able to instantiate the `NullWritable`.
Before that, we only needed the `Writable` interface which didn't require any other classes from Hadoop. However the `NullWritable` is depending on a lot of classes. I first tried it by copy-pasting classes from Hadoop, but after the 5th file or so I gave up. I don't know how many files it will be in the end.
But it quickly becomes dangerous shipping Hadoop code due to incompatible versions.
Also, at the end of the day, people are going to have the Hadoop jars in their classpath anyways, because flink-runtime is depending on it.
The only argument left is probably the collection based execution. I think that one only requires flink-core and flink-java. But if somebody has an issue with the hadoop dependency, they can exclude it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1305] [FLINK-1304] Test for H...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-flink/pull/252
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1305] [FLINK-1304] Test for H...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/incubator-flink/pull/252#issuecomment-66916880
Okay, let's add it as a dependency library. Might also make the "Hadoop compatibility" implementation easier, if we can assume it as a dependency.
For the collection execution, it should be fine. One can strip the Hadoop libraries if no Hadoop-related feature is used.
+1
Will merge this...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---