You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by rdblue <gi...@git.apache.org> on 2015/12/23 20:06:56 UTC
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
GitHub user rdblue opened a pull request:
https://github.com/apache/nifi/pull/147
NIFI-1193: Add support for storing data in Hive tables.
This adds support for storing data in Hive tables to the Kite processor using @joey's suggestion on #128. I've excluded as many dependencies as I could find to get the final size down to something reasonable, both by excluding the ones not used by the metastore (which is what Joey's suggestion helped with) and excluding the ones that are already included in NiFi or the Hadoop dependencies nar.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/incubator-nifi NIFI-1193-add-kite-hive-support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/147.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #147
----
commit c244a97069a4bae6065157c2b3552d2fafaa058f
Author: Ryan Blue <bl...@apache.org>
Date: 2015-12-23T19:02:51Z
NIFI-1193: Add support for storing data in Hive tables.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by jskora <gi...@git.apache.org>.
Github user jskora commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-176331346
I can also build master successfully now.
Interestingly, it downloaded the hive-serde jar file from the Cloudera repository. Not sure why it didn't do that yesterday.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-175186662
Now that this is in master, I'm going to close this PR. Thanks @rickysaltzer!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-176603003
@rdblue Thanks! Travis is giving the green thumb up.
@rickysaltzer Would you be able to give this the same inspection/evaluation as before now that the build is good to go?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-176866130
Thanks, @rickysaltzer! I'll close this again.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by mattyb149 <gi...@git.apache.org>.
Github user mattyb149 commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-175207659
It's in the Cloudera repo but not sure we can resolve from there: https://maven-repository.com/artifact/org.apache.hive/hive-serde/0.12.0-cdh5.0.0
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-175205635
I'm getting this as well. a little more of the output to provide additional context:
[ERROR] ----------
[ERROR] 1) org.apache.hive:hive-serde:jar:0.12.0-cdh5.0.0
[ERROR]
[ERROR] Try downloading the file manually from the project website.
[ERROR]
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=org.apache.hive -DartifactId=hive-serde -Dversion=0.12.0-cdh5.0.0 -Dpackaging=jar -Dfile=/path/to/file
[ERROR]
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=org.apache.hive -DartifactId=hive-serde -Dversion=0.12.0-cdh5.0.0 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR]
[ERROR] Path to dependency:
[ERROR] 1) org.apache.nifi:nifi-kite-processors:jar:0.4.2-SNAPSHOT
[ERROR] 2) org.apache.hive.hcatalog:hive-hcatalog-core:jar:1.2.0
[ERROR] 3) org.apache.hive:hive-metastore:jar:1.2.0
[ERROR] 4) org.apache.hive:hive-serde:jar:0.12.0-cdh5.0.0
[ERROR]
[ERROR] ----------
[ERROR] 1 required artifact is missing.
[ERROR]
[ERROR] for artifact:
[ERROR] org.apache.nifi:nifi-kite-processors:jar:0.4.2-SNAPSHOT
[ERROR]
[ERROR] from the specified remote repositories:
[ERROR] central (https://repo1.maven.org/maven2, releases=true, snapshots=false),
[ERROR] apache-repo (https://repository.apache.org/content/repositories/releases, releases=true, snapshots=false),
[ERROR] jcenter (http://jcenter.bintray.com, releases=true, snapshots=false),
[ERROR] apache.snapshots (http://repository.apache.org/snapshots, releases=false, snapshots=true)
[INFO] Copying httpclient-4.4.1.jar to /devel/nifi/nifi-nar-bundles/nifi-standard-services/nifi-hbase_1_1_2-client-service-bundle/nifi-hbase_1_1_2-client-service-nar/target/classes/META-INF/bundled-dependencies/httpclient-4.4.1.jar
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :nifi-kite-processors
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-177054637
@rdblue There was a comment @joewitt had left on the associated ticket.
https://issues.apache.org/jira/browse/NIFI-1193?focusedCommentId=15123007&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15123007
Was there to be more associated with tihs?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rickysaltzer <gi...@git.apache.org>.
Github user rickysaltzer commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-176801515
will do - checking on it now
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-174752878
Yep!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-176297371
Looks like the build is working after the new commit. Thanks Ricky!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by jskora <gi...@git.apache.org>.
Github user jskora commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-175199900
I cannot build a clean checkout of master from the Apache git repo after this merge. I get
> `[ERROR] Failed to execute goal org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process (default) on project nifi-kite-processors: Failed to resolve dependencies for one or more projects in the reactor. Reason: Missing:`
> `[ERROR] ----------`
> `[ERROR] 1) org.apache.hive:hive-serde:jar:0.12.0-cdh5.0.0`
If I roll back to the previous commit on master it builds fine.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rickysaltzer <gi...@git.apache.org>.
Github user rickysaltzer commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-174751825
If I understand this patch correctly, you're just including the necessary dependencies for dataset URIs like `dataset:hive:...` to work?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rickysaltzer <gi...@git.apache.org>.
Github user rickysaltzer commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-176806382
Build and functional testing are a +1 from me. I'll push this out
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-175765398
@apiri I've reopened this and will fix it in this PR. Thanks for rolling back the changes in master.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-176281907
I just pushed a fix that @rickysaltzer came up with. The jars being pulled in aren't actually needed for the tests so we can exclude them.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-175230521
I'll find where that leak is coming from, sorry about this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rickysaltzer <gi...@git.apache.org>.
Github user rickysaltzer commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-175174510
I re-built the kite nar with this patch and was able to insert data into a `dataset:hive` dataset.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue closed the pull request at:
https://github.com/apache/nifi/pull/147
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rickysaltzer <gi...@git.apache.org>.
Github user rickysaltzer commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-175207550
Good catch, guys. This must have built fine for me because I already had that jar in my local repo. I'm working with @rdblue to address this.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-176373718
Squashed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
Github user rdblue closed the pull request at:
https://github.com/apache/nifi/pull/147
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rickysaltzer <gi...@git.apache.org>.
Github user rickysaltzer commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-174753214
cool! I'll pull this down and test it out this week. Thanks, Ryan!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rdblue <gi...@git.apache.org>.
GitHub user rdblue reopened a pull request:
https://github.com/apache/nifi/pull/147
NIFI-1193: Add support for storing data in Hive tables.
This adds support for storing data in Hive tables to the Kite processor using @joey's suggestion on #128. I've excluded as many dependencies as I could find to get the final size down to something reasonable, both by excluding the ones not used by the metastore (which is what Joey's suggestion helped with) and excluding the ones that are already included in NiFi or the Hadoop dependencies nar.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/rdblue/incubator-nifi NIFI-1193-add-kite-hive-support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/147.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #147
----
commit c244a97069a4bae6065157c2b3552d2fafaa058f
Author: Ryan Blue <bl...@apache.org>
Date: 2015-12-23T19:02:51Z
NIFI-1193: Add support for storing data in Hive tables.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-176354248
@rdblue Could you please squash the commits? I don't carry a squash all mentality, but would prefer to avoid reintroducing the troublesome commit, even if it is immediately adjusted by the one following it.
Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by rickysaltzer <gi...@git.apache.org>.
Github user rickysaltzer commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-176808901
patch checked into `master` https://git-wip-us.apache.org/repos/asf?p=nifi.git;h=73c0637
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request: NIFI-1193: Add support for storing data in Hive...
Posted by apiri <gi...@git.apache.org>.
Github user apiri commented on the pull request:
https://github.com/apache/nifi/pull/147#issuecomment-175401857
@rdblue Can we reopen this just for the sake of continuing tracking this given the current state of things? I have reopened the associated JIRA issue.
For future reference, while the Travis CI build can be flaky at times, it has greatly aided in my reviews. Typically, the majority of the builds in the matrix work appropriately and can serve as a pretty decent cleanroom build process and can help catch things like these. By default this does `a mvn clean install -Pcontrib-check` and allows any reviewer to just do a multithreaded build skipping tests and contrib to generate an assembly for functional testing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---