You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by mosermw <gi...@git.apache.org> on 2016/08/03 20:02:33 UTC
[GitHub] nifi pull request #779: NIFI-2471 fix Hadoop configuration resources when ta...
GitHub user mosermw opened a pull request:
https://github.com/apache/nifi/pull/779
NIFI-2471 fix Hadoop configuration resources when talking to ...
\u2026multiple Hadoop clusters
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mosermw/nifi NIFI-2471
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/779.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #779
----
commit 873a7a747070284dbd72eaba49b5bc388c1c7a77
Author: Mike Moser <mo...@apache.org>
Date: 2016-08-03T19:58:57Z
NIFIDEV-2471 fix Hadoop configuration resources when talking to multiple Hadoop clusters
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #779: NIFI-2471 fix Hadoop configuration resources when ta...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/779
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi issue #779: NIFI-2471 fix Hadoop configuration resources when talking t...
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/779
OK thanks @mosermw. Looks good. Merged to master & 0.x branches.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #779: NIFI-2471 fix Hadoop configuration resources when ta...
Posted by mosermw <gi...@git.apache.org>.
Github user mosermw commented on a diff in the pull request:
https://github.com/apache/nifi/pull/779#discussion_r73542639
--- Diff: nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java ---
@@ -177,9 +178,17 @@ protected KerberosProperties getKerberosProperties() {
return results;
}
+ @Override
+ public void onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue) {
--- End diff --
Thanks for looking at this @markap14. I originally thought the same way you describe. During testing, though, the AbstractHadoopProcessor's OnScheduled method works but the GetHDFS OnScheduled method throws a "Failed to invoke OnScheduled method due to java.lang.RuntimeException: Caused by java.io.IOException: PropertyDescriptor[Directory] has invalid value /some/directory. The directory does not exist." When stopping the invalid processor, the abstractOnStopped() method is not called. Perhaps that's an issue unto itself, but adding an onPropertyModified method here fixed the problem that HdfsResources was not null the next time the processor was started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #779: NIFI-2471 fix Hadoop configuration resources when ta...
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/779#discussion_r73546113
--- Diff: nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java ---
@@ -177,9 +178,17 @@ protected KerberosProperties getKerberosProperties() {
return results;
}
+ @Override
+ public void onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue) {
--- End diff --
Looks like you're right - it is not calling @OnStopped methods when @OnScheduled throws an Exception. I created a ticket for that - https://issues.apache.org/jira/browse/NIFI-2484. I think we should avoid working around the bug in this PR though and just address it in the framework.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #779: NIFI-2471 fix Hadoop configuration resources when ta...
Posted by markap14 <gi...@git.apache.org>.
Github user markap14 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/779#discussion_r73518484
--- Diff: nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java ---
@@ -177,9 +178,17 @@ protected KerberosProperties getKerberosProperties() {
return results;
}
+ @Override
+ public void onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue) {
--- End diff --
@mosermw I'm not sure that this is really necessary. The abstractOnStopped() method resets hdfsResources to new HdfsResources(null, null, null) - so every time that it's stopped, it gets reset. I think maybe the only change needed is the change below to config.set(disableCacheName, "true") ??
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] nifi pull request #779: NIFI-2471 fix Hadoop configuration resources when ta...
Posted by mosermw <gi...@git.apache.org>.
Github user mosermw commented on a diff in the pull request:
https://github.com/apache/nifi/pull/779#discussion_r73552111
--- Diff: nifi-nar-bundles/nifi-hadoop-bundle/nifi-hdfs-processors/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java ---
@@ -177,9 +178,17 @@ protected KerberosProperties getKerberosProperties() {
return results;
}
+ @Override
+ public void onPropertyModified(PropertyDescriptor descriptor, String oldValue, String newValue) {
--- End diff --
Alrighty, no problem. I've added a commit to this PR to remove this onPropertyModified method. Hopefully the two commits can be squashed to one. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---