You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by warneke <gi...@git.apache.org> on 2014/07/17 00:28:55 UTC
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
GitHub user warneke opened a pull request:
https://github.com/apache/incubator-flink/pull/71
[FLINK-1006] Added support for the MapR file system
[FLINK-1006] Added support for the MapR file system.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/warneke/incubator-flink mapr
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/71.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #71
----
commit d88f24a443a6ca6fec53b79d679fe96d32c4f07c
Author: Daniel Warneke <wa...@apache.org>
Date: 2014-07-16T19:55:51Z
[FLINK-1006] Added support for the MapR file system
commit 860c24cf07b4297fefc78c0098c8c867786a5268
Author: Daniel Warneke <wa...@apache.org>
Date: 2014-07-16T22:20:47Z
Merge branch 'master' into mapr
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by warneke <gi...@git.apache.org>.
Github user warneke closed the pull request at:
https://github.com/apache/incubator-flink/pull/71
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-53278710
This pull request has been merged, without closing the issue: https://github.com/apache/incubator-flink/commit/bbde5faef4bbf73c53c33f50276e19d69dd72a9a#diff-d41d8cd98f00b204e9800998ecf8427e
@warneke: Could you close the pull request for us? (We don't have any additional permissions in this repo, as it belongs to the ASF).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-52477522
Looks good to me, I'll merge it...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by StephanEwen <gi...@git.apache.org>.
Github user StephanEwen commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-52232837
Looks good to me.
@warneke Have you tested it with a MapR file system? If yes, I would say it is good to merge.
BTW: Does MapR create its own version of zookeeper?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by warneke <gi...@git.apache.org>.
Github user warneke commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-54086339
Thank you for merging!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-52236780
I also got an offer from MapR to give us access to a testing system, but I would prefer if Daniel can test it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by warneke <gi...@git.apache.org>.
Github user warneke commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-50386469
When you build Flink with the hadoop2 version set to 2.3.0-mapr-4.0.0-FCS, all dependencies are included in the library folder automatically through the regular Maven dependency resolution. There are no special configuration settings necessary. To be honest, I never tried to use the vanilla build and add the dependencies for MapR later on.
If people don't want to compile Flink themselves, I assume they are also not too keen on manually fixing Hadoop dependencies. How about we provide precompiled versions of Flink releases for the three major Hadoop players? I think you guys started it for Cloudera anyway, right? I would volunteer to cover the MapR part.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-50352447
Since you did not add an additional dependency, I assume users have to put a maprfs jar in their `lib/` folder?
Can you add a little bit of documentation to the `docs/` directory on how to use the maprfs ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-52280957
Your build is good. One of the 6 tasks failed with the following message: "I'm sorry but your test run exceeded 50.0 minutes. ". Our tests often run out of the 50 minutes limit.
But @aljoscha has worked on the maven configuration to parallelize the build execution, so we are going to resolve the issue soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by warneke <gi...@git.apache.org>.
Github user warneke commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-52276826
Yes, I tested the code on a MapR system. Both file system discovery and file system functions work fine, so it should be good to merge (the code is basically a class loader and a wrapper).
StephanEwen: What do you mean by "own" version of Zookeeper?
BTW: Why did the build fail on Travis CI. Was that related to my code changes?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-50442883
Ah, good.
Yes, I'm going to create binaries for major Hadoop distributions with the next release.
If its just setting this MapR version for hadoop2, I can also create a package for MapR (as long as we can do it, license wise)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-flink pull request: [FLINK-1006] Added support for the M...
Posted by rmetzger <gi...@git.apache.org>.
Github user rmetzger commented on the pull request:
https://github.com/apache/incubator-flink/pull/71#issuecomment-50811704
So when adding the mapr dependency, it pulls the following MapR specific files:
```
./hadoop-hdfs-2.3.0-mapr-4.0.0-FCS.jar
./zookeeper-3.4.5-mapr-1401.jar
./hadoop-annotations-2.3.0-mapr-4.0.0-FCS.jar
./hadoop-common-2.3.0-mapr-4.0.0-FCS.jar
./hadoop-auth-2.3.0-mapr-4.0.0-FCS.jar
./maprfs-2.3.0-mapr-4.0.0-FCS.jar
```
I think the regular hadoop dependencies with `mapr` versions are fine, but I'm not quite sure about the `maprfs` package. The contained namespaces are not `org.apache.` but `com.mapr`.
Sadly, non of the license tags inside the `pom.xml` is stating anything about the license.
http://repository.mapr.com/nexus/content/repositories/mapr-public/com/mapr/hadoop/maprfs/2.3.0-mapr-4.0.0-FCS/
So I will not add a MapR distribution for the 0.6 release to our download page. The licensing situation is just too uncertain. I'll ask on their user's forum.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---