You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by pvillard31 <gi...@git.apache.org> on 2018/08/29 12:58:56 UTC
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
GitHub user pvillard31 opened a pull request:
https://github.com/apache/nifi/pull/2973
NIFI-5561 - Add component name filtering to S2S Provenance Reporting …
…Task
Thank you for submitting a contribution to Apache NiFi.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with NIFI-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] Have you ensured that the full suite of tests is executed via mvn -Pcontrib-check clean install at the root nifi folder?
- [ ] Have you written or updated unit tests to verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file, including the main LICENSE file under nifi-assembly?
- [ ] If applicable, have you updated the NOTICE file, including the main NOTICE file found under nifi-assembly?
- [ ] If adding new Properties, have you added .displayName in addition to .name (programmatic access) for each of the new properties?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pvillard31/nifi NIFI-5561
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/nifi/pull/2973.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2973
----
commit 6a8f6caa9d9d31fc23e1f1868f37bf73c9d6d346
Author: Pierre Villard <pi...@...>
Date: 2018-08-29T12:58:15Z
NIFI-5561 - Add component name filtering to S2S Provenance Reporting Task
----
---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2973#discussion_r213990560
--- Diff: nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java ---
@@ -151,6 +151,25 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
+ static final PropertyDescriptor FILTER_COMPONENT_NAME = new PropertyDescriptor.Builder()
+ .name("s2s-prov-task-name-filter")
+ .displayName("Component Name to Include")
+ .description("Regular expression to filter the provenance events based on the component name. Only the events matching the regular "
+ + "expression will be sent. If no filter is set, all the events are sent. If multiple filters are set, the filters are cumulative.")
+ .required(false)
+ .addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
--- End diff --
Fair enough, I'll add it on the properties. On a related note, I hope to find time to start exploring NIFI-5367 to enable VR scope by default in NiFi.
---
[GitHub] nifi issue #2973: NIFI-5561 - Add component name filtering to S2S Provenance...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2973
Thanks Pierre. Running a final check and will merge.
---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Posted by MikeThomsen <gi...@git.apache.org>.
Github user MikeThomsen commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2973#discussion_r213980541
--- Diff: nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java ---
@@ -151,6 +151,25 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
+ static final PropertyDescriptor FILTER_COMPONENT_NAME = new PropertyDescriptor.Builder()
+ .name("s2s-prov-task-name-filter")
+ .displayName("Component Name to Include")
+ .description("Regular expression to filter the provenance events based on the component name. Only the events matching the regular "
+ + "expression will be sent. If no filter is set, all the events are sent. If multiple filters are set, the filters are cumulative.")
+ .required(false)
+ .addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
--- End diff --
Our CM story is still really developing at this point, so I would say whatever could be done to help facilitate making it easier for admins to configure NiFi in a standardized way is going to make the a little happier even if it's just "copy and paste this EL string."
---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2973#discussion_r213905074
--- Diff: nifi-nar-bundles/nifi-extension-utils/nifi-reporting-utils/src/main/java/org/apache/nifi/reporting/util/provenance/ProvenanceEventConsumer.java ---
@@ -241,20 +255,24 @@ private long updateLastEventId(final List<ProvenanceEventRecord> events, final S
private boolean isFilteringEnabled() {
return componentTypeRegex != null || !eventTypes.isEmpty() || !componentIds.isEmpty()
- || componentTypeRegexExclude != null || !eventTypesExclude.isEmpty() || !componentIdsExclude.isEmpty();
+ || componentTypeRegexExclude != null || !eventTypesExclude.isEmpty() || !componentIdsExclude.isEmpty()
+ || componentNameRegex != null || componentNameRegexExclude != null;
--- End diff --
In general, I don't push changing "traditional" code blocks to "new" style just for the sake of it, but in this case, I think a Java 8-style `.stream()` construction will make this clearer (some of these fields are `List` and some are `Pattern`, and the `Pattern` can be empty/blank (which is not checked here), which would not (logically) enable filtering.
I made two commits ([a regression test on the current logic](https://github.com/alopresto/nifi/commit/0e717001a6124d6863f21efd18130ba996513e77) and [a new implementation](https://github.com/alopresto/nifi/commit/f586dd4f7cf1b0ab33c77e7053028a758ca057d1)) and it still works. Personally, I think this form is clearer and will scale more gracefully if new properties are added. Let me know your thoughts.
---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/nifi/pull/2973
---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2973#discussion_r214106940
--- Diff: nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java ---
@@ -151,6 +151,25 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
+ static final PropertyDescriptor FILTER_COMPONENT_NAME = new PropertyDescriptor.Builder()
+ .name("s2s-prov-task-name-filter")
+ .displayName("Component Name to Include")
+ .description("Regular expression to filter the provenance events based on the component name. Only the events matching the regular "
+ + "expression will be sent. If no filter is set, all the events are sent. If multiple filters are set, the filters are cumulative.")
+ .required(false)
+ .addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
--- End diff --
That's an excellent point about the VR scoping that I forgot. I think this is nominally more "consistent" across the app, but you're right that it's not as valuable as I expected. Thanks.
---
[GitHub] nifi issue #2973: NIFI-5561 - Add component name filtering to S2S Provenance...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on the issue:
https://github.com/apache/nifi/pull/2973
I created a flow which uses 3 `GenerateFlowFile` processors (named _Component A_, _Component A_, and _Component A (B)_) to evaluate the new filtering behavior. Each generates a flowfile with the content and _source_ attribute indicating the source (for the 2nd processor, the message & attribute refer to _Component A (2)_). I exercised the `SiteToSiteProvenanceReportingTask` with no accept/deny list, then only an accept value, then both. The functionality worked as expected; explicit deny overrode allow (see screenshot).
The template is [available here](https://gist.github.com/alopresto/b78ec5da5482f4af7f88faf7eeefd3b7).
---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Posted by alopresto <gi...@git.apache.org>.
Github user alopresto commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2973#discussion_r213905518
--- Diff: nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java ---
@@ -151,6 +151,25 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
+ static final PropertyDescriptor FILTER_COMPONENT_NAME = new PropertyDescriptor.Builder()
+ .name("s2s-prov-task-name-filter")
+ .displayName("Component Name to Include")
+ .description("Regular expression to filter the provenance events based on the component name. Only the events matching the regular "
+ + "expression will be sent. If no filter is set, all the events are sent. If multiple filters are set, the filters are cumulative.")
+ .required(false)
+ .addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
--- End diff --
Should these properties support Expression Language (Variables Only)? I can imagine a variable being set with a "component name filter" specific to a process group. Thoughts?
---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2973#discussion_r213933052
--- Diff: nifi-nar-bundles/nifi-extension-utils/nifi-reporting-utils/src/main/java/org/apache/nifi/reporting/util/provenance/ProvenanceEventConsumer.java ---
@@ -241,20 +255,24 @@ private long updateLastEventId(final List<ProvenanceEventRecord> events, final S
private boolean isFilteringEnabled() {
return componentTypeRegex != null || !eventTypes.isEmpty() || !componentIds.isEmpty()
- || componentTypeRegexExclude != null || !eventTypesExclude.isEmpty() || !componentIdsExclude.isEmpty();
+ || componentTypeRegexExclude != null || !eventTypesExclude.isEmpty() || !componentIdsExclude.isEmpty()
+ || componentNameRegex != null || componentNameRegexExclude != null;
--- End diff --
Yeah agree, I added your two commits on this pull request. Thanks!
---
[GitHub] nifi pull request #2973: NIFI-5561 - Add component name filtering to S2S Pro...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2973#discussion_r213933585
--- Diff: nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-site-to-site-reporting-task/src/main/java/org/apache/nifi/reporting/SiteToSiteProvenanceReportingTask.java ---
@@ -151,6 +151,25 @@
.addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
.build();
+ static final PropertyDescriptor FILTER_COMPONENT_NAME = new PropertyDescriptor.Builder()
+ .name("s2s-prov-task-name-filter")
+ .displayName("Component Name to Include")
+ .description("Regular expression to filter the provenance events based on the component name. Only the events matching the regular "
+ + "expression will be sent. If no filter is set, all the events are sent. If multiple filters are set, the filters are cumulative.")
+ .required(false)
+ .addValidator(StandardValidators.REGULAR_EXPRESSION_VALIDATOR)
--- End diff --
Since it's a reporting task, it won't access the variable registries defined at UI-level. I believe this will only access the global file-based variable registry (defined in nifi.properties) and the environment/jvm variables. Not sure this provides much value but on the other hand it's easy to add. Thoughts?
---
[GitHub] nifi issue #2973: NIFI-5561 - Add component name filtering to S2S Provenance...
Posted by pvillard31 <gi...@git.apache.org>.
Github user pvillard31 commented on the issue:
https://github.com/apache/nifi/pull/2973
Added VR scope on properties and fixed indentation to have the same everywhere. To see the changes and masking the indentation changes: https://github.com/apache/nifi/pull/2973/files?w=1
---