You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by rdblue <gi...@git.apache.org> on 2015/11/19 01:24:07 UTC
[GitHub] nifi pull request: NIFI-1193: Add Hive support to Kite storage pro...
GitHub user rdblue opened a pull request:
https://github.com/apache/nifi/pull/128
NIFI-1193: Add Hive support to Kite storage processor.
This adds a Hive dataset implementation that shades and minimizes the
classes needed to connect to the Hive MetaStore.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/incubator-nifi NIFI-1193-kite-hive-support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/128.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #128
----
commit dcf19984ec755e9c706cb3c4a12b7d0d025b89a9
Author: Ryan Blue <bl...@apache.org>
Date: 2015-11-01T00:23:08Z
NIFI-1193: Add Hive support to Kite storage processor.
This adds a Hive dataset implementation that shades and minimizes the
classes needed to connect to the Hive MetaStore.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add Hive support to Kite storage pro...
Posted by joey <gi...@git.apache.org>.
Github user joey commented on the pull request:
https://github.com/apache/nifi/pull/128#issuecomment-158084660
@rdblue I'm guessing that you used the shade plugin due to the hive-exec jar embedding so many libraries in unshaded package names?
If so, I've got a pom that pulls in direct dependencies that can talk to the Hive metastore without the hive-exec jar. Let me know if that would help here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add Hive support to Kite storage pro...
Posted by joey <gi...@git.apache.org>.
Github user joey commented on the pull request:
https://github.com/apache/nifi/pull/128#issuecomment-158172948
Here's what we've been using to avoid `hive-exec`:
```
<dependency>
<groupId>org.apache.hive.hcatalog</groupId>
<artifactId>hive-hcatalog-core</artifactId>
<exclusions>
<exclusion>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
</exclusion>
<exclusion>
<artifactId>jersey-servlet</artifactId>
<groupId>com.sun.jersey</groupId>
</exclusion>
<exclusion>
<artifactId>jersey-core</artifactId>
<groupId>com.sun.jersey</groupId>
</exclusion>
<exclusion>
<artifactId>jersey-server</artifactId>
<groupId>com.sun.jersey</groupId>
</exclusion>
<exclusion>
<artifactId>servlet-api</artifactId>
<groupId>javax.servlet</groupId>
</exclusion>
<exclusion>
<artifactId>jetty-all</artifactId>
<groupId>org.eclipse.jetty.aggregate</groupId>
</exclusion>
<exclusion>
<groupId>org.apache.hive</groupId>
<artifactId>hive-exec</artifactId>
</exclusion>
<exclusion>
<artifactId>parquet-hadoop-bundle</artifactId>
<groupId>com.twitter</groupId>
</exclusion>
</exclusions>
</dependency>
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add Hive support to Kite storage pro...
Posted by busbey <gi...@git.apache.org>.
Github user busbey commented on the pull request:
https://github.com/apache/nifi/pull/128#issuecomment-158108067
yes pleas @joey
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add Hive support to Kite storage pro...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue closed the pull request at:
https://github.com/apache/nifi/pull/128
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add Hive support to Kite storage pro...
Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on the pull request:
https://github.com/apache/nifi/pull/128#issuecomment-158137742
@rdblue @joey @busbey maybe we just don't add the nar to the assembly itself but put this in the source tree. Once we get a template/extension registry built then this is ok anyway. What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add Hive support to Kite storage pro...
Posted by joewitt <gi...@git.apache.org>.
Github user joewitt commented on the pull request:
https://github.com/apache/nifi/pull/128#issuecomment-158074311
Ryan,
Definitely appreciate you trying to make it less size prohibitive. I think we'll want to avoid having shaded jars and such being utilized for this. It complicates the licensing and related concerns and we have done an extremely good job getting those clean even to the point of every binary artifact we produce (nars) having embedded license/notice data correct to all sub-dependencies.
This is an area which really highlights our need to tackle the extension registry. We need to, on the other side of this release, figure out how we as a community can get more agility for releasing extensions like this versus the core framework.
OlegZ: We do need to tackle any copyright assertions on contributed source and ensure all proper license and notice adherence occurs. I'm overly generalizing here but there are like maybe five people on earth (I'm looking at you Sean Busbey) that care about following the strict guidance of licensing and notices at the level we do. In basically every contrib that brings in dependencies we'll have to help others most likely. Feels like a fine trade in exchange for contributions of helpful things the community will benefit from.
We also need to ensure that there is appropriate testing. However, above all else we need to keep in mind this community is powered by contributions. So in every exchange let's make sure our discussions stay focused on helping folks bring contribs along. As specific example consider the lack of unit tests. We could as part of the review build them. Or as part of the feedback ask if there are ideas on how to include some. Some extensions and contributions are inherently really hard to unit test. I don't know if this one is or isn't. Adding unit tests or asking if unit tests can be included is more powerful than saying we can't accept the contrib without them. The difference can at times be subtle but the effect on the community and tenor of discussion can be dramatic.
So far everyone in the community has done an awesome job of helping each other find the middle ground on contributions so that we can be inclusive and encouraging while increasing quality as well.
Thanks
Joe
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add Hive support to Kite storage pro...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/nifi/pull/128#issuecomment-158137120
@joey, that's exactly the problem. If you have a solution that avoids hive-exec, that would be great!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---