You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "bbende (via GitHub)" <gi...@apache.org> on 2024/05/01 20:20:32 UTC

[PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

bbende opened a new pull request, #8726:
URL: https://github.com/apache/nifi/pull/8726

   - Methods for getBranches, getDefaultBranch, and createBranch in flow registry client
   - Add branch as a field to all relevant objects/dtos/etc
   - Add optional branch argument to REST API for retrieving buckets
   - Add new REST API for retrieving branches for a given registry client
   
   <!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
   <!-- contributor license agreements.  See the NOTICE file distributed with -->
   <!-- this work for additional information regarding copyright ownership. -->
   <!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
   <!-- (the "License"); you may not use this file except in compliance with -->
   <!-- the License.  You may obtain a copy of the License at -->
   <!--     http://www.apache.org/licenses/LICENSE-2.0 -->
   <!-- Unless required by applicable law or agreed to in writing, software -->
   <!-- distributed under the License is distributed on an "AS IS" BASIS, -->
   <!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
   <!-- See the License for the specific language governing permissions and -->
   <!-- limitations under the License. -->
   
   # Summary
   
   [NIFI-13120](https://issues.apache.org/jira/browse/NIFI-13120)
   
   This PR introduces further refactoring to the `FlowRegistryClient` API. It adds new methods related to branches which have default implementations, and changes the parameters of other methods to use "location" objects to consolidate the combos of branch/bucketId/flowId/version that need to be passed around.
   
   The NiFi Registry client will ignore the incoming branch on all methods and continue to function as it always has.
   
   This PR also adds a REST API for retrieving branches:
   `nifi-api/flow/registries/{registryId}/branches`
   
   Optional query param on the REST API for retrieving buckets:
   `/nifi-api/flow/registries/{registryId}/buckets?branch=main`
   
   
   # Tracking
   
   Please complete the following tracking steps prior to pull request creation.
   
   ### Issue Tracking
   
   - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
   
   ### Pull Request Tracking
   
   - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
   - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
   
   ### Pull Request Formatting
   
   - [X] Pull Request based on current revision of the `main` branch
   - [X] Pull Request refers to a feature branch with one commit containing changes
   
   # Verification
   
   Please indicate the verification steps performed prior to pull request creation.
   
   ### Build
   
   - [X] Build completed using `mvn clean install -P contrib-check`
     - [X] JDK 21
   
   ### UI Contributions
   
   - [ ] NiFi is modernizing its UI. Any contributions that update the [current UI](https://github.com/apache/nifi/tree/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-ui) also need to be implemented in the [new UI](https://github.com/apache/nifi/tree/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-frontend/src/main/nifi).  
   
   ### Licensing
   
   - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
   - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
   
   ### Documentation
   
   - [ ] Documentation formatting appears as expected in rendered files
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory closed pull request #8726: NIFI-13120 Introduce branch as a concept in registry client and VCI
URL: https://github.com/apache/nifi/pull/8726


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1587893025


##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java:
##########
@@ -4022,7 +4027,7 @@ private VersionedFlowSynchronizationContext createGroupSynchronizationContext(fi
     }
 
     @Override
-    public void verifyCanSaveToFlowRegistry(final String registryId, final String bucketId, final String flowId, final String saveAction) {
+    public void verifyCanSaveToFlowRegistry(final String registryId, final String branch, final String bucketId, final String flowId, final String saveAction) {

Review Comment:
   You are probably right, I'll take a look at this. It was hard to determine at which points were the correct place to use the objects outside of the FlowRegistryClient API.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on PR #8726:
URL: https://github.com/apache/nifi/pull/8726#issuecomment-2090466214

   Hi @simonbence ! I have been working on a GitHub based registry client and this evolved from that. 
   
   In my first iteration I had a property called "Branch" on the GH Registry Client, which does work fine, but it means that all operations on the client are tied to that one branch, which means just to import or save to a different branch you need multiple registry clients per branch which didn't feel correct. 
   
   Branch really needs to be another parameter to all the methods the way that we send in some combination of bucketId, flowId, version. So I refactored some of the methods to consolidate all the String arguments into "location" objects to reduce the number of arguments per method, and also to hopefully avoid needing to change method signatures in the future if a new argument is needed. The new branch methods all provide default implementations so any existing registry clients can ignore those if they don't care about branching.
   
   I'm not sure when the UI can be updated, but the end goal would be for the screens where you save/import to/from a registry client, that you would see:
   - Registry Client
   - Branch
   - Bucket
   - Flow
   - Versions list
   
   For now branch is an optional field when saving/importing, and if not specified then the backend will call `getDefaultBranch` from the client and use that.
   
   For NiFi Registry, I doubt that we will implement branching there, although its possible some day, so for now the branch argument is mostly just passed around and not used when interacting with NiFi Registry. 
   
   This PR on it's own is a little hard to test the branching changes, but the main thing would be making sure that no regressions happen when using NiFi Registry, which I did do a bit of testing. After this is merged then I can submit the GH Registry Client and then further testing can be done using branches there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1588963943


##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryBranch.java:
##########
@@ -0,0 +1,52 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+
+package org.apache.nifi.registry.flow;
+
+import java.util.Objects;
+
+public class FlowRegistryBranch {

Review Comment:
   Yeah, that makes sense. My reasons to ask this was a bit different:
   - It can be potentially an useful information for the users to show (it might help follow the lineage in the UI, without hopping to Git or other backend system)
   - With this question I wanted to validate my assumption with the code within NiFi does not have to do any kind of work with figuring out what the given branch contains (like from berofe and after branching off, wher the before is shared with the parent) but always expected to get a snapshot view of the actual content)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1587920491


##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryBranch.java:
##########
@@ -0,0 +1,52 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+
+package org.apache.nifi.registry.flow;
+
+import java.util.Objects;
+
+public class FlowRegistryBranch {

Review Comment:
   Currently there wasn't a need to know this since I wasn't envisioning any advanced features like merging back or rebasing etc from NiFi. All of these things would be done outside in GitHub, or whatever registry.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1588957056


##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryClient.java:
##########
@@ -71,30 +74,82 @@ public interface FlowRegistryClient extends ConfigurableComponent {
      */
     boolean isStorageLocationApplicable(FlowRegistryClientConfigurationContext context, String location);
 
+    /**
+     * Indicates if the registry supports branching.
+     *
+     * @param context Configuration context
+     * @return true if the registry supports branching, false otherwise
+     */
+    default boolean isBranchingSupported(final FlowRegistryClientConfigurationContext context) {
+        return false;
+    }
+
+    /**
+     * Get the available branches. Should return at least one branch that matches the response of getDefaultBranch.
+     *
+     * @param context Configuration context
+     * @return the set of available branches
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default Set<FlowRegistryBranch> getBranches(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+        return Set.of(getDefaultBranch(context));
+    }
+
+    /**
+     * Gets the default branch. Must return a non-null FlowRegistryBranch instance with a non-null name.
+     *
+     * @param context Configuration context
+     * @return the default branch
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default FlowRegistryBranch getDefaultBranch(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+        final FlowRegistryBranch branch = new FlowRegistryBranch();
+        branch.setName(DEFAULT_BRANCH_NAME);
+        return branch;
+    }
+
+    /**
+     * Creates a new branch from a given branch.
+     *
+     * @param context Configuration context
+     * @param createBranch the info for creating the branch
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default void createBranch(FlowRegistryClientConfigurationContext context, final CreateBranch createBranch) throws FlowRegistryException, IOException {

Review Comment:
   I think I understand the idea behind. With this, I would suggest to leave this part to a follow up when it is actually utilized.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1588977169


##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/BucketLocation.java:
##########
@@ -0,0 +1,54 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+
+package org.apache.nifi.registry.flow;
+
+/**
+ * Information for locating a bucket in a flow registry.
+ */
+public class BucketLocation {
+
+    private String branch;

Review Comment:
   Thank you for describing it in detail!
   
   The create branch related features misled me a bit. I was thinking we want to cover the "branch management" part of the lifecycle within NiFi UI. If that is not the case, please ignore my concerns, the workflow you want to cover seems just right.
   
   In case however we want to involve the branch management into NiFi (UI), I think it will need some further consideration (like adding support for breating buckets as well, or alike).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on PR #8726:
URL: https://github.com/apache/nifi/pull/8726#issuecomment-2097206304

   @simonbence Thanks for the continued review and feedback. If you think the changes are good after the latest update, it would be helpful to mark the PR as approved, in case no one else has a chance to review. Thanks again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1588943898


##########
nifi-extension-bundles/nifi-flow-registry-client-bundle/nifi-flow-registry-client-services/src/main/java/org/apache/nifi/registry/flow/NifiRegistryFlowRegistryClient.java:
##########
@@ -149,7 +149,7 @@ public boolean isStorageLocationApplicable(final FlowRegistryClientConfiguration
     }
 
     @Override
-    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context, final String branch) throws FlowRegistryException, IOException {

Review Comment:
   I wanted to point [here](https://github.com/bbende/nifi/blob/NIFI-13120/nifi-extension-bundles/nifi-flow-registry-client-bundle/nifi-flow-registry-client-services/src/main/java/org/apache/nifi/registry/flow/NifiRegistryFlowRegistryClient.java#L242-L247) as an example. The point was, here (even though it will aways be the default one) the branch is handled as it coukld be different and we might expect something else from the argument then the default.
   
   Thus it looks like it could have different value, which (in case of this particular implementation) is not true at this point. I think this "proper handling" can be misleading and might result unrealistic responses: it can return a snapshot with proper content but having metadata pointing to an arbitrary (non-existing) branch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1588954630


##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNode.java:
##########
@@ -401,13 +401,33 @@ public boolean isStorageLocationApplicable(final String location) {
     }
 
     @Override
-    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
-        return node.getBuckets(context);
+    public boolean isBranchingSupported() {
+        return node.isBranchingSupported();
     }
 
     @Override
-    public FlowRegistryBucket getBucket(final FlowRegistryClientUserContext context, final String bucketId) throws FlowRegistryException, IOException {
-        return node.getBucket(context, bucketId);
+    public Set<FlowRegistryBranch> getBranches(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
+        return node.getBranches(context);
+    }
+
+    @Override
+    public FlowRegistryBranch getDefaultBranch(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
+        return node.getDefaultBranch(context);
+    }
+
+    @Override
+    public void createBranch(final FlowRegistryClientUserContext context, final CreateBranch createBranch) throws FlowRegistryException, IOException {

Review Comment:
   What I was thinking about is that the interface is lack of arguments with "verb based" or "action based" argumetnts, rather it uses data driven approach. This might be resolved by naming, however I am not sure about `Info` (but for the full truth, I have no better naming at the moment)
   
   Your reasoning about the future development of the API I can perfectly stand with. Still, it looks quite alien to the other signatures from the interface.
   
   Maybe the best of both worlds might be to stick with the `fromBranch` and `toBranch` arguments but intead of string, they might have a type `Branch` or alike, which might be populated with further information in the case of need. Just moving on with this thought experiment, if we might need further information about where to branch from, it can have the type of `BranchingPoint`.
   
   Not sure this is gonna solve everything, but I kindly ask you to consider.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1587909319


##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNode.java:
##########
@@ -401,13 +401,33 @@ public boolean isStorageLocationApplicable(final String location) {
     }
 
     @Override
-    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
-        return node.getBuckets(context);
+    public boolean isBranchingSupported() {
+        return node.isBranchingSupported();
     }
 
     @Override
-    public FlowRegistryBucket getBucket(final FlowRegistryClientUserContext context, final String bucketId) throws FlowRegistryException, IOException {
-        return node.getBucket(context, bucketId);
+    public Set<FlowRegistryBranch> getBranches(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
+        return node.getBranches(context);
+    }
+
+    @Override
+    public FlowRegistryBranch getDefaultBranch(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
+        return node.getDefaultBranch(context);
+    }
+
+    @Override
+    public void createBranch(final FlowRegistryClientUserContext context, final CreateBranch createBranch) throws FlowRegistryException, IOException {

Review Comment:
   The `CreateBranch` object is still just a data driven pojo just like `RegisteredFlow`. I am happy to name it something that sounds less like a command, maybe just adding `Info` to the end - `CreateBranchInfo` ?
   
   I originally had the method as `createBranch(context, String fromBranch, String toBranch)`, but I made the arguments an object following similar thinking to the location objects, meaning - in the future if we have to add another argument, we don't have to break the API signature of `createBranch`, we just add another field to the `CreateBranch` object.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on PR #8726:
URL: https://github.com/apache/nifi/pull/8726#issuecomment-2090279180

   Hi @bbende ! It looks to be a very nice addition. Is there any discussion or description could help me understand the gist of it before looking into the code in detail?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1589565627


##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryBranch.java:
##########
@@ -0,0 +1,52 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+
+package org.apache.nifi.registry.flow;
+
+import java.util.Objects;
+
+public class FlowRegistryBranch {

Review Comment:
   I think this ok to mark as resolved, but let me know if not



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1590716881


##########
nifi-extension-bundles/nifi-flow-registry-client-bundle/nifi-flow-registry-client-services/src/main/java/org/apache/nifi/registry/flow/NifiRegistryFlowRegistryClient.java:
##########
@@ -149,7 +149,7 @@ public boolean isStorageLocationApplicable(final FlowRegistryClientConfiguration
     }
 
     @Override
-    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context, final String branch) throws FlowRegistryException, IOException {

Review Comment:
   I have no strong feelings about it but I see some possible benefit:
   - 1. It looks generally a good approach to not fully rely on client side data validation (due to possible changes there, etc.)
   - 2I did see some usage where NiFi API was used directly from scripts/automazation. While I do not think that this is the intended way (however by supporting "Request Auth Token" this seems inevidable) there is a possibility for this and I would be on the safe side even if the current version of the code could not done real harm with using non-existing branch.
   
   I am not sure which is the more forwarding way to tackle this. We can either add validation (it looks a bit much future proof but might be misleading as it would look like this is an argument we really consider here) or just enforce the default value (which comes with some hardwired thing, generally to avoid) but I think it would be safer to do some steps towards some of these directions.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1591711739


##########
nifi-extension-bundles/nifi-flow-registry-client-bundle/nifi-flow-registry-client-services/src/main/java/org/apache/nifi/registry/flow/NifiRegistryFlowRegistryClient.java:
##########
@@ -149,7 +149,7 @@ public boolean isStorageLocationApplicable(final FlowRegistryClientConfiguration
     }
 
     @Override
-    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context, final String branch) throws FlowRegistryException, IOException {

Review Comment:
   Thanks for clarifying. I went ahead and added validation within the NiFi Registry client that enforces that if any method receives a non-null branch, then it must match the default branch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1589333765


##########
nifi-extension-bundles/nifi-flow-registry-client-bundle/nifi-flow-registry-client-services/src/main/java/org/apache/nifi/registry/flow/NifiRegistryFlowRegistryClient.java:
##########
@@ -149,7 +149,7 @@ public boolean isStorageLocationApplicable(final FlowRegistryClientConfiguration
     }
 
     @Override
-    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context, final String branch) throws FlowRegistryException, IOException {

Review Comment:
   Are you saying that you think we should check the `branch` argument in all of the methods in `NiFiRegistryFlowRegistryClient` to validate that it matches the default branch?
   
   The only way this could happen is someone making API calls outside of the UI where they could submit a request for save/import with a branch name that wasn't the default branch NiFi Registry is expecting, but also it would just be ignored by all of the methods, so it wouldn't cause any issues. 
   
   One other note... I did not add `branch` to the NiFi Registry specific data model class like `VersionedFlow`, `VersionedFlowSnapshot`, and `VersionedFlowSnapshotMetadat`, because its not used for anything on registry side. So in your example, if someone calls `getFlowContents` with `branch` as `dev`, they will get back a null branch in the response since there wasn't a branch in the content returned from registry.
   
   Happy to improve something here, just want to make sure I'm following what you are suggesting.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1587603716


##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java:
##########
@@ -4022,7 +4027,7 @@ private VersionedFlowSynchronizationContext createGroupSynchronizationContext(fi
     }
 
     @Override
-    public void verifyCanSaveToFlowRegistry(final String registryId, final String bucketId, final String flowId, final String saveAction) {
+    public void verifyCanSaveToFlowRegistry(final String registryId, final String branch, final String bucketId, final String flowId, final String saveAction) {

Review Comment:
   Should not the newly created `FlowLocation` used? If I am correct, branch, bucket id and flow id covers just the same set of info as `FlowLocation`s state covers.



##########
nifi-extension-bundles/nifi-flow-registry-client-bundle/nifi-flow-registry-client-services/src/main/java/org/apache/nifi/registry/flow/NifiRegistryFlowRegistryClient.java:
##########
@@ -149,7 +149,7 @@ public boolean isStorageLocationApplicable(final FlowRegistryClientConfiguration
     }
 
     @Override
-    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context, final String branch) throws FlowRegistryException, IOException {

Review Comment:
   What is the expectation against requests for "non-existing" branches? (In case of the NiFiRegistry this should be always the default and I think it is planned to enforce this from UI but do we want to have server-side validation and failure for cases with different branch or we want branch parameter completely ignored for NiFiRegistry)
   
   The question comes from that in `getFlowContents` we do not ignore it. It could be shortcutted to be main but it is handled properly, which somewhat different from the branch handling in other methods.



##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryClient.java:
##########
@@ -71,30 +74,82 @@ public interface FlowRegistryClient extends ConfigurableComponent {
      */
     boolean isStorageLocationApplicable(FlowRegistryClientConfigurationContext context, String location);
 
+    /**
+     * Indicates if the registry supports branching.
+     *
+     * @param context Configuration context
+     * @return true if the registry supports branching, false otherwise
+     */
+    default boolean isBranchingSupported(final FlowRegistryClientConfigurationContext context) {
+        return false;
+    }
+
+    /**
+     * Get the available branches. Should return at least one branch that matches the response of getDefaultBranch.
+     *
+     * @param context Configuration context
+     * @return the set of available branches
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default Set<FlowRegistryBranch> getBranches(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+        return Set.of(getDefaultBranch(context));
+    }
+
+    /**
+     * Gets the default branch. Must return a non-null FlowRegistryBranch instance with a non-null name.

Review Comment:
   I think it might worth mentioning that in case of the impementation does not support branching, a default value will be returned for convinience.



##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/CreateBranch.java:
##########
@@ -0,0 +1,53 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+
+package org.apache.nifi.registry.flow;
+
+/**
+ * Information for creating a branch in a flow registry.
+ */
+public class CreateBranch {
+
+    private String fromBranch;
+    private String newBranch;
+
+    public CreateBranch() {
+    }
+
+    public CreateBranch(final String fromBranch, final String newBranch) {
+        this.fromBranch = fromBranch;

Review Comment:
   When we do create a branch, is it expected to be branched to the current "head" of the fromBranch or what is the expected behaviour? 



##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryClient.java:
##########
@@ -71,30 +74,82 @@ public interface FlowRegistryClient extends ConfigurableComponent {
      */
     boolean isStorageLocationApplicable(FlowRegistryClientConfigurationContext context, String location);
 
+    /**
+     * Indicates if the registry supports branching.
+     *
+     * @param context Configuration context
+     * @return true if the registry supports branching, false otherwise
+     */
+    default boolean isBranchingSupported(final FlowRegistryClientConfigurationContext context) {
+        return false;
+    }
+
+    /**
+     * Get the available branches. Should return at least one branch that matches the response of getDefaultBranch.
+     *
+     * @param context Configuration context
+     * @return the set of available branches
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default Set<FlowRegistryBranch> getBranches(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+        return Set.of(getDefaultBranch(context));
+    }
+
+    /**
+     * Gets the default branch. Must return a non-null FlowRegistryBranch instance with a non-null name.
+     *
+     * @param context Configuration context
+     * @return the default branch
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default FlowRegistryBranch getDefaultBranch(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+        final FlowRegistryBranch branch = new FlowRegistryBranch();
+        branch.setName(DEFAULT_BRANCH_NAME);
+        return branch;
+    }
+
+    /**
+     * Creates a new branch from a given branch.
+     *
+     * @param context Configuration context
+     * @param createBranch the info for creating the branch
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default void createBranch(FlowRegistryClientConfigurationContext context, final CreateBranch createBranch) throws FlowRegistryException, IOException {

Review Comment:
   Bucket creation is not supported in the client but spared for the actaul registry service. Is not this out of the concept of the client as well?



##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java:
##########
@@ -4040,14 +4045,20 @@ public void verifyCanSaveToFlowRegistry(final String registryId, final String bu
 
                 // Flow ID matches. We want to publish the Process Group as the next version of the Flow, so we must
                 // ensure that all other parameters match as well.
+
+                if (branch != null && !Objects.equals(branch, vci.getBranch())) {

Review Comment:
   This two statement could be merged into one with the following way:
   - `FlowLocation` might be able to proviede a `BucketLocation` instance (like a partial path)
   - We could compare `BucketLocation` instnaces
   I think that would simplify this, might communicate the intent in a more accurate manner and as we have the same error message we would not even loose details



##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/BucketLocation.java:
##########
@@ -0,0 +1,54 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+
+package org.apache.nifi.registry.flow;
+
+/**
+ * Information for locating a bucket in a flow registry.
+ */
+public class BucketLocation {
+
+    private String branch;

Review Comment:
   Is there any contract or expectation against the relationship between branch and bucket? Based on what I saw in the code, I assume the top level organization is the branch, so buckts are _witnin_ the branches. (I might be wrong, I do not remember seeing explicit mention of this) Is there any expectations like a bucket must exists under every brunch (but the flows/flow versions might differ)?
   
   If this level structure stands, it can have a negative affect on the user experience as buckets currently cannot be created from NiFi UI. So for example I have a flow F1 within bucket B1 which are in the "main" branch. Assume I want to move to a "dev" branch with a new version of F1. Multiple things might happen depending on our purposes:
   - If branches and buckets are ortogonal, the bucket will be available for every branches
   - If buckets (already existing under other branches) are created on the need basis, registry implementations needs to know about this expectation and support this
   - If we expect buckets to be in place, currently the users needs to move to the registry server beforehand and create the given bucket before committing.
   
   I think based on the code I have a good educated guess about the idea but I would find it beneficial to have a description about this somewhere (either in documentation or javadoc, I am not sure)



##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryBranch.java:
##########
@@ -0,0 +1,52 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+
+package org.apache.nifi.registry.flow;
+
+import java.util.Objects;
+
+public class FlowRegistryBranch {

Review Comment:
   Can it matter what is the parent branch? Or this is a detail to the registry implementation and we do not wish to know?



##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/StandardFlowRegistryClientNode.java:
##########
@@ -369,19 +391,21 @@ private RegisteredFlowSnapshot fetchFlowContents(final FlowRegistryClientUserCon
                                                      final boolean fetchRemoteFlows) throws FlowRegistryException {
 
         final String storageLocation = coordinates.getStorageLocation();
+        final String branch = coordinates.getBranch();
         final String bucketId = coordinates.getBucketId();
         final String flowId = coordinates.getFlowId();
         final String version = coordinates.getVersion();
+        final FlowVersionLocation flowVersionLocation = new FlowVersionLocation(branch, bucketId, flowId, version);
 
         final List<FlowRegistryClientNode> clientNodes = getRegistryClientsForInternalFlow(storageLocation);
         for (final FlowRegistryClientNode clientNode : clientNodes) {
             try {
-                logger.debug("Attempting to fetch flow for Bucket [{}] Flow [{}] Version [{}] using {}", bucketId, flowId, version, clientNode);
-                final FlowSnapshotContainer snapshotContainer = clientNode.getFlowContents(context, bucketId, flowId, version, fetchRemoteFlows);
+                logger.debug("Attempting to fetch flow from branch [{}] for Bucket [{}] Flow [{}] Version [{}] using {}", branch, bucketId, flowId, version, clientNode);
+                final FlowSnapshotContainer snapshotContainer = clientNode.getFlowContents(context, flowVersionLocation, fetchRemoteFlows);
                 final RegisteredFlowSnapshot snapshot = snapshotContainer.getFlowSnapshot();
                 coordinates.setRegistryId(clientNode.getIdentifier());
 
-                logger.debug("Successfully fetched flow for Bucket [{}] Flow [{}] Version [{}] using {}", bucketId, flowId, version, clientNode);
+                logger.debug("Successfully fetched flow from branch [{}] for Bucket [{}] Flow [{}] Version [{}] using {}", branch, bucketId, flowId, version, clientNode);

Review Comment:
   Minor: other parts of the location start with upper case. Should not branch do that too?



##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNode.java:
##########
@@ -401,13 +401,33 @@ public boolean isStorageLocationApplicable(final String location) {
     }
 
     @Override
-    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
-        return node.getBuckets(context);
+    public boolean isBranchingSupported() {
+        return node.isBranchingSupported();
     }
 
     @Override
-    public FlowRegistryBucket getBucket(final FlowRegistryClientUserContext context, final String bucketId) throws FlowRegistryException, IOException {
-        return node.getBucket(context, bucketId);
+    public Set<FlowRegistryBranch> getBranches(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
+        return node.getBranches(context);
+    }
+
+    @Override
+    public FlowRegistryBranch getDefaultBranch(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
+        return node.getDefaultBranch(context);
+    }
+
+    @Override
+    public void createBranch(final FlowRegistryClientUserContext context, final CreateBranch createBranch) throws FlowRegistryException, IOException {

Review Comment:
   Up to this point, the `FlowAnalyzingRegistryClientNode` did not have methods with argumetns serving as "command object". In cases where it depended on more complex structures, like in case of `RegisteredFlow`, these arguments were data driven (here: `RegisteredFlow` and not `RegisterFlow`). I think this makes the interface into somewhat inconsistent. I would suggest with providing two branch representation, even if they are Strings or their own type (as I see other methods go with String based representation, so if you go with their specific type, I suggest to update those methods as well)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1587922187


##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/CreateBranch.java:
##########
@@ -0,0 +1,53 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+
+package org.apache.nifi.registry.flow;
+
+/**
+ * Information for creating a branch in a flow registry.
+ */
+public class CreateBranch {
+
+    private String fromBranch;
+    private String newBranch;
+
+    public CreateBranch() {
+    }
+
+    public CreateBranch(final String fromBranch, final String newBranch) {
+        this.fromBranch = fromBranch;

Review Comment:
   That was my current thinking, but you are right that possibly having a commit to branch from would make sense. I will consider adding this if we decide to keep this method based on discussion in other comment about where this should be done outside.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on PR #8726:
URL: https://github.com/apache/nifi/pull/8726#issuecomment-2095492030

   Thanks for the follow up @bbende ! On my side I see only one open topic, the matter of input validation. As I mentioned I do not have very strong opinion about the matter but it would seem safer to add some further refinement to deal with the incoming argument. I listed some ideas might help shape your decision. It looks great otherwise and to me it looked like it does not break any current behaviour. Thanks for you work!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1587884737


##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/BucketLocation.java:
##########
@@ -0,0 +1,54 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+
+package org.apache.nifi.registry.flow;
+
+/**
+ * Information for locating a bucket in a flow registry.
+ */
+public class BucketLocation {
+
+    private String branch;

Review Comment:
   Great question...
   
   The way it is currently designed, branch is the first argument that needs to be chosen in order to then determine the buckets. For example, in Git/GitHub, a Bucket is just a top-level directory in the repo. So in order to know what top-level directories exist, you need to list the contents of the repo on a given branch.
   
   Also, in the current design, once a process group is under version control, either by `Import From Registry` or `Start Version Control`, at that point the branch of the PG can't change, just like you can't change Bucket. So you can't have a PG tracking to `Branch1 - Bucket1` and then all of a sudden change it to `Branch2` where you are worried about maybe `Bucket1` doesn't exist.
   
   The GitHub workflow I have been testing is something like the following...
   
   - Lets assume the Git repo has the branch `main` and already has a flow named `My Flow` in `My Bucket`
   - `Production` NiFi has a process group that imported this flow from `main`
   - Some user wants to make a change to this flow...
     - They go to GitHub, or command line Git, and they create a new branch from `main` called `flow-change-1` (or whatever)
     - At this point the branch `flow-change-1` has the same buckets and flows as `main`
     - They go the `Development` NiFi, or their laptop, and they `Import Flow From Registry` and choose the `flow-change-1` - `My Bucket` - `My Flow` - latest version
     - They make changes and commit the next version which commits to `flow-change-1` branch
     - They go to GitHub UI and raise a PR from `flow-change-1` to `main`
     - Someone else reviews the PR and merges to `main`
     - Someone else goes to `Production` NiFi and now sees a new version available for the PG tracking to `main`
     - They do Change Flow Version to the new version
     - At this point maybe the PG in the `Dev` NiFi is deleted because it was a temporary feature branch, or maybe this process it iterated for more changes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on PR #8726:
URL: https://github.com/apache/nifi/pull/8726#issuecomment-2090578311

   > Hi @simonbence ! I have been working on a GitHub based registry client and this evolved from that.
   > 
   > In my first iteration I had a property called "Branch" on the GH Registry Client, which does work fine, but it means that all operations on the client are tied to that one branch, which means just to import or save to a different branch you need multiple registry clients per branch which didn't feel correct.
   > 
   > Branch really needs to be another parameter to all the methods the way that we send in some combination of bucketId, flowId, version. So I refactored some of the methods to consolidate all the String arguments into "location" objects to reduce the number of arguments per method, and also to hopefully avoid needing to change method signatures in the future if a new argument is needed. The new branch methods all provide default implementations so any existing registry clients can ignore those if they don't care about branching.
   > 
   > I'm not sure when the UI can be updated, but the end goal would be for the screens where you save/import to/from a registry client, that you would see:
   > 
   > * Registry Client
   > * Branch
   > * Bucket
   > * Flow
   > * Versions list
   > 
   > For now branch is an optional field when saving/importing, and if not specified then the backend will call `getDefaultBranch` from the client and use that.
   > 
   > For NiFi Registry, I doubt that we will implement branching there, although its possible some day, so for now the branch argument is mostly just passed around and not used when interacting with NiFi Registry.
   > 
   > This PR on it's own is a little hard to test the branching changes, but the main thing would be making sure that no regressions happen when using NiFi Registry, which I did do a bit of testing. After this is merged then I can submit the GH Registry Client and then further testing can be done using branches there.
   
   Thanks for the quick and comprehensive answer! I really like the "location" idea. I left a couple of comments but I think part of my questions might have answered by this. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1587917581


##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryClient.java:
##########
@@ -71,30 +74,82 @@ public interface FlowRegistryClient extends ConfigurableComponent {
      */
     boolean isStorageLocationApplicable(FlowRegistryClientConfigurationContext context, String location);
 
+    /**
+     * Indicates if the registry supports branching.
+     *
+     * @param context Configuration context
+     * @return true if the registry supports branching, false otherwise
+     */
+    default boolean isBranchingSupported(final FlowRegistryClientConfigurationContext context) {
+        return false;
+    }
+
+    /**
+     * Get the available branches. Should return at least one branch that matches the response of getDefaultBranch.
+     *
+     * @param context Configuration context
+     * @return the set of available branches
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default Set<FlowRegistryBranch> getBranches(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+        return Set.of(getDefaultBranch(context));
+    }
+
+    /**
+     * Gets the default branch. Must return a non-null FlowRegistryBranch instance with a non-null name.
+     *
+     * @param context Configuration context
+     * @return the default branch
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default FlowRegistryBranch getDefaultBranch(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+        final FlowRegistryBranch branch = new FlowRegistryBranch();
+        branch.setName(DEFAULT_BRANCH_NAME);
+        return branch;
+    }
+
+    /**
+     * Creates a new branch from a given branch.
+     *
+     * @param context Configuration context
+     * @param createBranch the info for creating the branch
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default void createBranch(FlowRegistryClientConfigurationContext context, final CreateBranch createBranch) throws FlowRegistryException, IOException {

Review Comment:
   That is a fair point and it is making me think whether we should leave this out initially, since I did not plan to expose any API to use this yet, and we could always add it later with a default impl.
   
   The reason I added this was related to the Git workflow I described in a later comment. I was picturing a possible UX in NiFi UI where you go to `Import Flow From Registry` screen and you choose (Branch - Bucket - Flow - Version) and then an option to create a new branch from this and have your imported PG track to the new branch. Essentially "create my feature branch" to work on this, but maybe we should be punting this to an out of band process through GitHub UI or Git command line.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1587901450


##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/groups/StandardProcessGroup.java:
##########
@@ -4040,14 +4045,20 @@ public void verifyCanSaveToFlowRegistry(final String registryId, final String bu
 
                 // Flow ID matches. We want to publish the Process Group as the next version of the Flow, so we must
                 // ensure that all other parameters match as well.
+
+                if (branch != null && !Objects.equals(branch, vci.getBranch())) {

Review Comment:
   I'll take a look at this, but this is basically the same as the existing checks that are here to ensure that a new version being saved is not changing the bucketId/flowId/etc that it is already tracking against. The only difference here is that branch is optional when submitting the save, so we need to check that its not null. If its null, then the code during register will correctly select either the existing branch from VCI, or the default branch.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1587892003


##########
nifi-extension-bundles/nifi-flow-registry-client-bundle/nifi-flow-registry-client-services/src/main/java/org/apache/nifi/registry/flow/NifiRegistryFlowRegistryClient.java:
##########
@@ -149,7 +149,7 @@ public boolean isStorageLocationApplicable(final FlowRegistryClientConfiguration
     }
 
     @Override
-    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientConfigurationContext context, final String branch) throws FlowRegistryException, IOException {

Review Comment:
   I would say it is really up to the implementation to decide. We could have all the methods in NiFi registry check the incoming branch argument, and validate that it equals the same name as the default branch, but I don't think it is providing much value since none of the methods are using the branch.
   
   Can you clarify what you mean about `getFlowContents` ? I believe that method is unchanged in the NiFi Registry implementation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1589526553


##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/FlowRegistryClient.java:
##########
@@ -71,30 +74,82 @@ public interface FlowRegistryClient extends ConfigurableComponent {
      */
     boolean isStorageLocationApplicable(FlowRegistryClientConfigurationContext context, String location);
 
+    /**
+     * Indicates if the registry supports branching.
+     *
+     * @param context Configuration context
+     * @return true if the registry supports branching, false otherwise
+     */
+    default boolean isBranchingSupported(final FlowRegistryClientConfigurationContext context) {
+        return false;
+    }
+
+    /**
+     * Get the available branches. Should return at least one branch that matches the response of getDefaultBranch.
+     *
+     * @param context Configuration context
+     * @return the set of available branches
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default Set<FlowRegistryBranch> getBranches(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+        return Set.of(getDefaultBranch(context));
+    }
+
+    /**
+     * Gets the default branch. Must return a non-null FlowRegistryBranch instance with a non-null name.
+     *
+     * @param context Configuration context
+     * @return the default branch
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default FlowRegistryBranch getDefaultBranch(final FlowRegistryClientConfigurationContext context) throws FlowRegistryException, IOException {
+        final FlowRegistryBranch branch = new FlowRegistryBranch();
+        branch.setName(DEFAULT_BRANCH_NAME);
+        return branch;
+    }
+
+    /**
+     * Creates a new branch from a given branch.
+     *
+     * @param context Configuration context
+     * @param createBranch the info for creating the branch
+     *
+     * @throws FlowRegistryException If an issue happens during processing the request.
+     * @throws IOException If there is issue with the communication between NiFi and the Flow Registry.
+     */
+    default void createBranch(FlowRegistryClientConfigurationContext context, final CreateBranch createBranch) throws FlowRegistryException, IOException {

Review Comment:
   I agree, I am going to remove the `createBranch` method and `CreateBranch` object for now, which addresses some of the discussion/questions. It can be added in a non-breaking way later if we decide we ever want to create branches from NiFi, but I have a feeling we don't want to.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1589563432


##########
nifi-api/src/main/java/org/apache/nifi/registry/flow/CreateBranch.java:
##########
@@ -0,0 +1,53 @@
+/*
+ *
+ *  * Licensed to the Apache Software Foundation (ASF) under one or more
+ *  * contributor license agreements.  See the NOTICE file distributed with
+ *  * this work for additional information regarding copyright ownership.
+ *  * The ASF licenses this file to You under the Apache License, Version 2.0
+ *  * (the "License"); you may not use this file except in compliance with
+ *  * the License.  You may obtain a copy of the License at
+ *  *
+ *  *     http://www.apache.org/licenses/LICENSE-2.0
+ *  *
+ *  * Unless required by applicable law or agreed to in writing, software
+ *  * distributed under the License is distributed on an "AS IS" BASIS,
+ *  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ *  * See the License for the specific language governing permissions and
+ *  * limitations under the License.
+ *
+ */
+
+package org.apache.nifi.registry.flow;
+
+/**
+ * Information for creating a branch in a flow registry.
+ */
+public class CreateBranch {
+
+    private String fromBranch;
+    private String newBranch;
+
+    public CreateBranch() {
+    }
+
+    public CreateBranch(final String fromBranch, final String newBranch) {
+        this.fromBranch = fromBranch;

Review Comment:
   CreateBranch is removed for now



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] NIFI-13120 Introduce branch as a concept in registry client and VCI [nifi]

Posted by "bbende (via GitHub)" <gi...@apache.org>.
bbende commented on code in PR #8726:
URL: https://github.com/apache/nifi/pull/8726#discussion_r1589562736


##########
nifi-framework-bundle/nifi-framework/nifi-framework-components/src/main/java/org/apache/nifi/registry/flow/FlowAnalyzingRegistryClientNode.java:
##########
@@ -401,13 +401,33 @@ public boolean isStorageLocationApplicable(final String location) {
     }
 
     @Override
-    public Set<FlowRegistryBucket> getBuckets(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
-        return node.getBuckets(context);
+    public boolean isBranchingSupported() {
+        return node.isBranchingSupported();
     }
 
     @Override
-    public FlowRegistryBucket getBucket(final FlowRegistryClientUserContext context, final String bucketId) throws FlowRegistryException, IOException {
-        return node.getBucket(context, bucketId);
+    public Set<FlowRegistryBranch> getBranches(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
+        return node.getBranches(context);
+    }
+
+    @Override
+    public FlowRegistryBranch getDefaultBranch(final FlowRegistryClientUserContext context) throws FlowRegistryException, IOException {
+        return node.getDefaultBranch(context);
+    }
+
+    @Override
+    public void createBranch(final FlowRegistryClientUserContext context, final CreateBranch createBranch) throws FlowRegistryException, IOException {

Review Comment:
   Marking this as resolved since createBranch is removed for now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org