You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by jbonofre <gi...@git.apache.org> on 2015/09/03 16:26:14 UTC
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
GitHub user jbonofre opened a pull request:
https://github.com/apache/nifi/pull/83
NIFI-925: Upgrade to Hadoop 2.7.1
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jbonofre/nifi NIFI-925
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/83.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #83
----
commit 9f0ea08b370e07a14416898b8e1f5cd74f8b8b27
Author: Jean-Baptiste Onofré <jb...@apache.org>
Date: 2015-09-03T14:25:25Z
NIFI-925: Upgrade to Hadoop 2.7.1
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by busbey <gi...@git.apache.org>.
Github user busbey commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-150367325
The problem with keeping client libraries up to date is that often times they don't work with older servers. YCSB hits this pretty hard for Cassandra, ElasticSearch, and HBase. The only real solution is something like NIFI-710.
For the specific case of Hadoop, I'd look to see if the project makes any promises about newer clients talking to older servers. Absent that, I would presume we shouldn't update the client without a big notice to downstream users. Despite that, the Hadoop 2.6.0 client library is particularly bad so if we could update to 2.6.1 in the short term, that'd be good.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
Re: [GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by Tony Kurc <tr...@gmail.com>.
If i recall correctly, 2.6.2 is in in release candidate stage. For 0.4
would that be a better target? I haven't had a chance to look at client
code changes
On Oct 23, 2015 9:32 AM, "rickysaltzer" <gi...@git.apache.org> wrote:
> Github user rickysaltzer commented on the pull request:
>
> https://github.com/apache/nifi/pull/83#issuecomment-150573514
>
> I'm also a **+1** for upgrading to 2.6.1. I was also thinking that
> this would need to play into the [template registry](
> https://cwiki.apache.org/confluence/display/NIFI/Extension+and+Template+Registry)
> proposal. We should definitely continue to iron out the details so we can
> get that feature into a not too distant release.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastructure@apache.org or file a JIRA ticket
> with INFRA.
> ---
>
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by rickysaltzer <gi...@git.apache.org>.
Github user rickysaltzer commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-150573514
I'm also a **+1** for upgrading to 2.6.1. I was also thinking that this would need to play into the [template registry](https://cwiki.apache.org/confluence/display/NIFI/Extension+and+Template+Registry) proposal. We should definitely continue to iron out the details so we can get that feature into a not too distant release.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-150340867
Totally with you on this Ricky. As far as the contrib goes that is legit. We should keep it up to date. However, you're right that we need a better model for things like Hadoop. It should be braindead easy for folks to plugin different Hadoop versions/providers and the user just get to select from a list.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by trkurc <gi...@git.apache.org>.
Github user trkurc commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-152341713
My preference would be to shoot for 2.6.2 in 0.4. If it isn't done, go with 2.6.1. I do not believe 0.4 should be released with hadoop 2.6.0 client libs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by rickysaltzer <gi...@git.apache.org>.
Github user rickysaltzer commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-150322439
Is there a specific reason for depending on Hadoop 2.7? I'm not entirely certain how we should approach upgrading Hadoop packages (or other dependencies for that matter). Take other projects for instance (i.e. Spark, Flink), they have separate builds for major Hadoop version (e.g. 2.4, 2.6). I wonder if this is an option we should consider...
Preferably, I would like some sort of class loader isolation in place where you can decide which version of the library to use on a processor by processor basis.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-150368632
i am comfortable with the idea of 2.6.1 with our current approach. Once we have NIFI-710 in play then we should support 2.6, 2.7, X.Y, whatever. We need to do that in conjunction with having a registry model. So many good plays to work with here. Mark Payne was talking about some cool ideas he had to pull this off recently. We need to get him to write those down!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by jbonofre <gi...@git.apache.org>.
Github user jbonofre commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-152998024
Allright, I will update this PR accordingly. Thanks guys !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by busbey <gi...@git.apache.org>.
Github user busbey commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-150578536
On the one hand, I prefer to engage hadoop-as-it-is rather than waiting for changes.
On the other hand, 2.6.2 RC0 closes in 5 days, so figure about a week for there to be artifacts presuming it passes?
2.6.1 has the critical problems fixed (like corrupting data when writing to HDFS encrypted directories), so I'd rather upgrade to it now and then go to 2.6.2 if/when it gets released.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/83
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by jbonofre <gi...@git.apache.org>.
Github user jbonofre commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-154731244
PR rebased and updated.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by busbey <gi...@git.apache.org>.
Github user busbey commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-152348193
:+1: the 2.6.2 vote just passed last night, so I'd expect artifacts to be published with plenty of time for 0.4.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-925: Upgrade to Hadoop 2.7.1
Posted by rickysaltzer <gi...@git.apache.org>.
Github user rickysaltzer commented on the pull request:
https://github.com/apache/nifi/pull/83#issuecomment-150361061
Yeah I agree, keeping libraries up to date is a good thing. We should definitely discuss strategies on how we can achieve multi-version support for libraries, eventually. There's currently a JIRA ([NIFI-710](https://issues.apache.org/jira/browse/NIFI-710)) which talks about this in the context of Hadoop.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---