You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "bbende (via GitHub)" <gi...@apache.org> on 2023/05/01 15:42:29 UTC
[GitHub] [nifi] bbende opened a new pull request, #7218: NIFI-11464 Improvements for importing nested versioned flows
bbende opened a new pull request, #7218:
URL: https://github.com/apache/nifi/pull/7218
- Introduce FlowSnapshotContainer to return root snapshot + children
- Introduce ControllerServiceResolver to extract logic from service facade
- Update resolution logic to correctly consider all services in the hierarchy
- Merge additional parameter contexts and parameter providers from child to parent
- Add unit test for controller service resolver
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-11464](https://issues.apache.org/jira/browse/NIFI-11464)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [X] Pull Request based on current revision of the `main` branch
- [X] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [X] Build completed using `mvn clean install -P contrib-check`
- [X] JDK 11
- [ ] JDK 17
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] gresockj commented on a diff in pull request #7218: NIFI-11464 Improvements for importing nested versioned flows
Posted by "gresockj (via GitHub)" <gi...@apache.org>.
gresockj commented on code in PR #7218:
URL: https://github.com/apache/nifi/pull/7218#discussion_r1184867897
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceResolver.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.service;
+
+import org.apache.nifi.authorization.Authorizer;
+import org.apache.nifi.authorization.RequestAction;
+import org.apache.nifi.authorization.user.NiFiUser;
+import org.apache.nifi.controller.flow.FlowManager;
+import org.apache.nifi.flow.ControllerServiceAPI;
+import org.apache.nifi.flow.ExternalControllerServiceReference;
+import org.apache.nifi.flow.VersionedConfigurableExtension;
+import org.apache.nifi.flow.VersionedControllerService;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.flow.VersionedProcessor;
+import org.apache.nifi.flow.VersionedPropertyDescriptor;
+import org.apache.nifi.groups.ProcessGroup;
+import org.apache.nifi.registry.flow.FlowSnapshotContainer;
+import org.apache.nifi.registry.flow.RegisteredFlowSnapshot;
+import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Stack;
+import java.util.stream.Collectors;
+
+public class StandardControllerServiceResolver implements ControllerServiceResolver {
+
+ private final Authorizer authorizer;
+ private final FlowManager flowManager;
+ private final NiFiRegistryFlowMapper flowMapper;
+ private final ControllerServiceProvider controllerServiceProvider;
+ private final ControllerServiceApiLookup controllerServiceApiLookup;
+
+ public StandardControllerServiceResolver(final Authorizer authorizer,
+ final FlowManager flowManager,
+ final NiFiRegistryFlowMapper flowMapper,
+ final ControllerServiceProvider controllerServiceProvider,
+ final ControllerServiceApiLookup controllerServiceApiLookup) {
+ this.authorizer = authorizer;
+ this.flowManager = flowManager;
+ this.flowMapper = flowMapper;
+ this.controllerServiceProvider = controllerServiceProvider;
+ this.controllerServiceApiLookup = controllerServiceApiLookup;
+ }
+
+ @Override
+ public void resolveInheritedControllerServices(final FlowSnapshotContainer flowSnapshotContainer, final String parentGroupId, final NiFiUser user) {
+ final RegisteredFlowSnapshot topLevelSnapshot = flowSnapshotContainer.getFlowSnapshot();
+ final VersionedProcessGroup versionedGroup = topLevelSnapshot.getFlowContents();
+ final Map<String, ExternalControllerServiceReference> externalControllerServiceReferences = topLevelSnapshot.getExternalControllerServices();
+
+ final ProcessGroup parentGroup = flowManager.getGroup(parentGroupId);
+
+ final Set<VersionedControllerService> ancestorServices = parentGroup.getControllerServices(true).stream()
+ .filter(serviceNode -> serviceNode.isAuthorized(authorizer, RequestAction.READ, user))
+ .map(serviceNode -> flowMapper.mapControllerService(serviceNode, controllerServiceProvider, Collections.emptySet(), Collections.emptyMap()))
Review Comment:
When importing a versioned group with a child versioned group, I get the following exception:
```
2023-05-04 07:03:09,458 ERROR [NiFi Web Server-2125] o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: java.lang.UnsupportedOperationException. Returning Internal Server Error response.
java.lang.UnsupportedOperationException: null
at java.base/java.util.AbstractMap.put(AbstractMap.java:209)
at org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper.mapPropertyDescriptors(NiFiRegistryFlowMapper.java:638)
at org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper.mapControllerService(NiFiRegistryFlowMapper.java:532)
at org.apache.nifi.controller.service.StandardControllerServiceResolver.lambda$resolveInheritedControllerServices$1(StandardControllerServiceResolver.java:72)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
at java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1621)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
at org.apache.nifi.controller.service.StandardControllerServiceResolver.resolveInheritedControllerServices(StandardControllerServiceResolver.java:73)
at org.apache.nifi.web.StandardNiFiServiceFacade.resolveInheritedControllerServices(StandardNiFiServiceFacade.java:4062)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] bbende commented on a diff in pull request #7218: NIFI-11464 Improvements for importing nested versioned flows
Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #7218:
URL: https://github.com/apache/nifi/pull/7218#discussion_r1185257232
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/controller/service/StandardControllerServiceResolver.java:
##########
@@ -0,0 +1,195 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.controller.service;
+
+import org.apache.nifi.authorization.Authorizer;
+import org.apache.nifi.authorization.RequestAction;
+import org.apache.nifi.authorization.user.NiFiUser;
+import org.apache.nifi.controller.flow.FlowManager;
+import org.apache.nifi.flow.ControllerServiceAPI;
+import org.apache.nifi.flow.ExternalControllerServiceReference;
+import org.apache.nifi.flow.VersionedConfigurableExtension;
+import org.apache.nifi.flow.VersionedControllerService;
+import org.apache.nifi.flow.VersionedProcessGroup;
+import org.apache.nifi.flow.VersionedProcessor;
+import org.apache.nifi.flow.VersionedPropertyDescriptor;
+import org.apache.nifi.groups.ProcessGroup;
+import org.apache.nifi.registry.flow.FlowSnapshotContainer;
+import org.apache.nifi.registry.flow.RegisteredFlowSnapshot;
+import org.apache.nifi.registry.flow.mapping.NiFiRegistryFlowMapper;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.Stack;
+import java.util.stream.Collectors;
+
+public class StandardControllerServiceResolver implements ControllerServiceResolver {
+
+ private final Authorizer authorizer;
+ private final FlowManager flowManager;
+ private final NiFiRegistryFlowMapper flowMapper;
+ private final ControllerServiceProvider controllerServiceProvider;
+ private final ControllerServiceApiLookup controllerServiceApiLookup;
+
+ public StandardControllerServiceResolver(final Authorizer authorizer,
+ final FlowManager flowManager,
+ final NiFiRegistryFlowMapper flowMapper,
+ final ControllerServiceProvider controllerServiceProvider,
+ final ControllerServiceApiLookup controllerServiceApiLookup) {
+ this.authorizer = authorizer;
+ this.flowManager = flowManager;
+ this.flowMapper = flowMapper;
+ this.controllerServiceProvider = controllerServiceProvider;
+ this.controllerServiceApiLookup = controllerServiceApiLookup;
+ }
+
+ @Override
+ public void resolveInheritedControllerServices(final FlowSnapshotContainer flowSnapshotContainer, final String parentGroupId, final NiFiUser user) {
+ final RegisteredFlowSnapshot topLevelSnapshot = flowSnapshotContainer.getFlowSnapshot();
+ final VersionedProcessGroup versionedGroup = topLevelSnapshot.getFlowContents();
+ final Map<String, ExternalControllerServiceReference> externalControllerServiceReferences = topLevelSnapshot.getExternalControllerServices();
+
+ final ProcessGroup parentGroup = flowManager.getGroup(parentGroupId);
+
+ final Set<VersionedControllerService> ancestorServices = parentGroup.getControllerServices(true).stream()
+ .filter(serviceNode -> serviceNode.isAuthorized(authorizer, RequestAction.READ, user))
+ .map(serviceNode -> flowMapper.mapControllerService(serviceNode, controllerServiceProvider, Collections.emptySet(), Collections.emptyMap()))
Review Comment:
@gresockj thanks for the review, not sure why I never ran into this, but I replaced the the use of emptyMap and emptySet there with new instances which should solve the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] bbende commented on pull request #7218: NIFI-11464 Improvements for importing nested versioned flows
Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on PR #7218:
URL: https://github.com/apache/nifi/pull/7218#issuecomment-1576884863
I'm going to merge based on the two approvals, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] simonbence commented on a diff in pull request #7218: NIFI-11464 Improvements for importing nested versioned flows
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7218:
URL: https://github.com/apache/nifi/pull/7218#discussion_r1218085936
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryClientNode.java:
##########
@@ -44,7 +44,7 @@ public interface FlowRegistryClientNode extends ComponentNode {
RegisteredFlow getFlow(FlowRegistryClientUserContext context, String bucketId, String flowId) throws FlowRegistryException, IOException;
Set<RegisteredFlow> getFlows(FlowRegistryClientUserContext context, String bucketId) throws FlowRegistryException, IOException;
- RegisteredFlowSnapshot getFlowContents(FlowRegistryClientUserContext context, String bucketId, String flowId, int version, boolean fetchRemoteFlows) throws FlowRegistryException, IOException;
+ FlowSnapshotContainer getFlowContents(FlowRegistryClientUserContext context, String bucketId, String flowId, int version, boolean fetchRemoteFlows) throws FlowRegistryException, IOException;
Review Comment:
This looks to be a breaking change on the API. I think, this can be okay for a 2.X change, but the Apache ticket is marked as an 1.21.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] simonbence commented on a diff in pull request #7218: NIFI-11464 Improvements for importing nested versioned flows
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7218:
URL: https://github.com/apache/nifi/pull/7218#discussion_r1218085936
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryClientNode.java:
##########
@@ -44,7 +44,7 @@ public interface FlowRegistryClientNode extends ComponentNode {
RegisteredFlow getFlow(FlowRegistryClientUserContext context, String bucketId, String flowId) throws FlowRegistryException, IOException;
Set<RegisteredFlow> getFlows(FlowRegistryClientUserContext context, String bucketId) throws FlowRegistryException, IOException;
- RegisteredFlowSnapshot getFlowContents(FlowRegistryClientUserContext context, String bucketId, String flowId, int version, boolean fetchRemoteFlows) throws FlowRegistryException, IOException;
+ FlowSnapshotContainer getFlowContents(FlowRegistryClientUserContext context, String bucketId, String flowId, int version, boolean fetchRemoteFlows) throws FlowRegistryException, IOException;
Review Comment:
This looks to be a breaking change on the API. I think, this can be okay for a 2.X change, but the Apache ticket is marked as an 1.21.0
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core-api/src/main/java/org/apache/nifi/registry/flow/FlowSnapshotContainer.java:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.registry.flow;
+
+import org.apache.nifi.flow.ParameterProviderReference;
+import org.apache.nifi.flow.VersionedParameter;
+import org.apache.nifi.flow.VersionedParameterContext;
+import org.apache.nifi.flow.VersionedProcessGroup;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Objects;
+import java.util.Set;
+
+/**
+ * Holds the results of recursively fetching the contents of a registered flow snapshot where some child groups may
+ * also be under version control. The top level flow snapshot will have the contents of all the child groups populated,
+ * and the child snapshots here provide a mechanism to obtain the corresponding metadata for a versioned controlled
+ * child group, such as the external service references, etc.
+ */
+public class FlowSnapshotContainer {
Review Comment:
Minor: I would find it beneficial to mark this as final
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] bbende merged pull request #7218: NIFI-11464 Improvements for importing nested versioned flows
Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende merged PR #7218:
URL: https://github.com/apache/nifi/pull/7218
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org