You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "simonbence (via GitHub)" <gi...@apache.org> on 2023/08/30 12:10:47 UTC
[GitHub] [nifi] simonbence opened a new pull request, #7661: NIFI-11772 Removing flow.xml support
simonbence opened a new pull request, #7661:
URL: https://github.com/apache/nifi/pull/7661
This is a draft pull request in order to finalize the approach. Things not aimed in this PR:
- Finalize documentation
- Toolkit changes
# Summary
[NIFI-11772](https://issues.apache.org/jira/browse/NIFI-11772)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [ ] Pull Request based on current revision of the `main` branch
- [ ] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `mvn clean install -P contrib-check`
- [ ] JDK 17
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11772 Removing flow.xml support [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1342713418
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -248,8 +235,6 @@ class ConfigEncryptionTool {
options.addOption(Option.builder("i").longOpt(OUTPUT_LOGIN_IDENTITY_PROVIDERS_ARG).hasArg(true).argName("file").desc("The destination login-identity-providers.xml file containing protected config values (will not modify input login-identity-providers.xml)").build())
options.addOption(Option.builder("a").longOpt(AUTHORIZERS_ARG).hasArg(true).argName("file").desc("The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)").build())
options.addOption(Option.builder("u").longOpt(OUTPUT_AUTHORIZERS_ARG).hasArg(true).argName("file").desc("The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)").build())
- options.addOption(Option.builder("f").longOpt(FLOW_XML_ARG).hasArg(true).argName("file").desc("The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)").build())
- options.addOption(Option.builder("g").longOpt(OUTPUT_FLOW_XML_ARG).hasArg(true).argName("file").desc("The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)").build())
Review Comment:
Thanks for the reply @simonbence. Handling the toolkit changes in a separate PR sounds good, but in that case, I recommend reverting this particular removal and keeping the toolkit changes as minimal as possible. As these argument names do not impact actual flow.xml.gz usage, leaving them in place seems fine for now, and more substantive changes can be handled separately. So from that perspective, I recommend limiting the toolkit changes to only those that are required for the removal of actual flow.xml.gz handling code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] simonbence commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1341127238
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -106,100 +95,57 @@ public void testStartupWithDifferentFlow() throws IOException, NiFiClientExcepti
node2.stop();
final File node2ConfDir = new File(node2.getInstanceDirectory(), "conf");
- final File flowXmlFile = new File(node2ConfDir, "flow.xml.gz");
- Files.deleteIfExists(flowXmlFile.toPath());
- Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.xml.gz"), flowXmlFile.toPath());
+ final File flowJsonFile = new File(node2ConfDir, "flow.json.gz");
+ Files.deleteIfExists(flowJsonFile.toPath());
+ Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.json.gz"), flowJsonFile.toPath());
node2.start(true);
- final File backupFile = getBackupFile(node2ConfDir);
- final NodeDTO node2Dto = getNodeDtoByApiPort(5672);
+ while (getNifiClient().getControllerClient().getNodes().getCluster().getNodes().stream().filter(n -> n.getStatus().equals("CONNECTED")).count() != 2) {
+ Thread.sleep(3000);
+ }
+ final File backupFile = getBackupFile(node2ConfDir);
verifyFlowContentsOnDisk(backupFile);
- disconnectNode(node2Dto);
verifyInMemoryFlowContents();
-
- // Reconnect the node so that we can properly shutdown
- reconnectNode(node2Dto);
Review Comment:
Thanks! I would believe the previous logic did not work properly as it was before remoeval (it did a REST call against the endpoint `cluster/nodes/${nodeId}` which behaves based on the accompanying `NodeEntity` / `NodeDTO`, but the status was not set to be DISCONNECTING, so in practice I think it did not disconnect), but thanks for the context! I will look how it behaves this way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11772 Removing flow.xml support [nifi]
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1348378366
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -106,124 +94,72 @@ public void testStartupWithDifferentFlow() throws IOException, NiFiClientExcepti
node2.stop();
final File node2ConfDir = new File(node2.getInstanceDirectory(), "conf");
- final File flowXmlFile = new File(node2ConfDir, "flow.xml.gz");
- Files.deleteIfExists(flowXmlFile.toPath());
- Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.xml.gz"), flowXmlFile.toPath());
+ final File flowJsonFile = new File(node2ConfDir, "flow.json.gz");
+ Files.deleteIfExists(flowJsonFile.toPath());
+ Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.json.gz"), flowJsonFile.toPath());
node2.start(true);
- final File backupFile = getBackupFile(node2ConfDir);
- final NodeDTO node2Dto = getNodeDtoByApiPort(5672);
+ waitForAllNodesConnected();
+ switchClientToNode(2);
+ final File backupFile = getBackupFile(node2ConfDir);
verifyFlowContentsOnDisk(backupFile);
- disconnectNode(node2Dto);
verifyInMemoryFlowContents();
-
- // Reconnect the node so that we can properly shutdown
- reconnectNode(node2Dto);
}
-
- private List<File> getFlowXmlFiles(final File confDir) {
- final File[] flowXmlFileArray = confDir.listFiles(file -> file.getName().startsWith("flow") && file.getName().endsWith(".xml.gz"));
- final List<File> flowXmlFiles = new ArrayList<>(Arrays.asList(flowXmlFileArray));
- return flowXmlFiles;
+ private List<File> getFlowJsonFiles(final File confDir) {
+ final File[] flowJsonFileArray = confDir.listFiles(file -> file.getName().startsWith("flow") && file.getName().endsWith(".json.gz"));
+ final List<File> flowJsonFiles = new ArrayList<>(Arrays.asList(flowJsonFileArray));
+ return flowJsonFiles;
}
private File getBackupFile(final File confDir) throws InterruptedException {
- waitFor(() -> getFlowXmlFiles(confDir).size() == 2);
-
- final List<File> flowXmlFiles = getFlowXmlFiles(confDir);
- assertEquals(2, flowXmlFiles.size());
+ waitFor(() -> getFlowJsonFiles(confDir).size() == 1);
- flowXmlFiles.removeIf(file -> file.getName().equals("flow.xml.gz"));
+ final List<File> flowJsonFiles = getFlowJsonFiles(confDir);
+ assertEquals(1, flowJsonFiles.size());
- assertEquals(1, flowXmlFiles.size());
- final File backupFile = flowXmlFiles.get(0);
- return backupFile;
+ return flowJsonFiles.iterator().next();
}
private void verifyFlowContentsOnDisk(final File backupFile) throws IOException {
- // Read the flow and make sure that the backup looks the same as the original. We don't just do a byte comparison because the compression may result in different
- // gzipped bytes and because if the two flows do differ, we want to have the String representation so that we can compare to see how they are different.
- final String flowXml = readFlow(backupFile);
- final String expectedFlow = readFlow(new File("src/test/resources/flows/mismatched-flows/flow1.xml.gz"));
-
- assertEquals(expectedFlow, flowXml);
-
// Verify some of the values that were persisted to disk
final File confDir = backupFile.getParentFile();
- final String loadedFlow = readFlow(new File(confDir, "flow.xml.gz"));
+ final String loadedFlow = readFlow(new File(confDir, "flow.json.gz"));
Review Comment:
https://issues.apache.org/jira/browse/NIFI-12176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1340487721
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -106,100 +95,57 @@ public void testStartupWithDifferentFlow() throws IOException, NiFiClientExcepti
node2.stop();
final File node2ConfDir = new File(node2.getInstanceDirectory(), "conf");
- final File flowXmlFile = new File(node2ConfDir, "flow.xml.gz");
- Files.deleteIfExists(flowXmlFile.toPath());
- Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.xml.gz"), flowXmlFile.toPath());
+ final File flowJsonFile = new File(node2ConfDir, "flow.json.gz");
+ Files.deleteIfExists(flowJsonFile.toPath());
+ Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.json.gz"), flowJsonFile.toPath());
node2.start(true);
- final File backupFile = getBackupFile(node2ConfDir);
- final NodeDTO node2Dto = getNodeDtoByApiPort(5672);
+ while (getNifiClient().getControllerClient().getNodes().getCluster().getNodes().stream().filter(n -> n.getStatus().equals("CONNECTED")).count() != 2) {
+ Thread.sleep(3000);
+ }
+ final File backupFile = getBackupFile(node2ConfDir);
verifyFlowContentsOnDisk(backupFile);
- disconnectNode(node2Dto);
verifyInMemoryFlowContents();
-
- // Reconnect the node so that we can properly shutdown
- reconnectNode(node2Dto);
Review Comment:
Is there a reason that we're changing the logic here?
Previously, the idea was to disconnect Node 2 so that any request made to the node would be a response directly from that node rather than a cluster response; then issue the request to Node 2 and verify that its flow is correct. Then, it would reconnect the node so that the shutdown logic would complete successfully.
The logic here has now changed to "verify that the cluster's flow is correct", which is not the same as "verify that Node 2's flow is correct."
That said, there is now a better way to handle this, by doing something like:
```
switchClientToNode(2);
getNifiClient().getFlowClient(DO_NOT_REPLICATE)...
```
This way, we don't have to disconnect the node, but we can issue requests directly to that node. But I think we need to either restore the logic or use the DO_NOT_REPLICATE to ensure that we're verifying Node 2's flow.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11772 Removing flow.xml support [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1342714169
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -385,26 +370,26 @@ class ConfigEncryptionTool {
}
}
- if (commandLine.hasOption(FLOW_XML_ARG)) {
- if (isVerbose) {
- logger.info("Handling encryption of flow.xml.gz")
- }
- flowXmlPath = commandLine.getOptionValue(FLOW_XML_ARG)
- outputFlowXmlPath = commandLine.getOptionValue(OUTPUT_FLOW_XML_ARG, flowXmlPath)
- handlingFlowXml = true
-
- newFlowAlgorithm = commandLine.getOptionValue(NEW_FLOW_ALGORITHM_ARG)
- newFlowProvider = commandLine.getOptionValue(NEW_FLOW_PROVIDER_ARG)
-
- if (flowXmlPath == outputFlowXmlPath) {
- // TODO: Add confirmation pause and provide -y flag to offer no-interaction mode?
- logger.warn("The source flow.xml.gz and destination flow.xml.gz are identical [${outputFlowXmlPath}] so the original will be overwritten")
- }
-
- if (!commandLine.hasOption(NIFI_PROPERTIES_ARG)) {
- printUsageAndThrow("In order to migrate a flow.xml.gz, a nifi.properties file must also be specified via '-n'/'--${NIFI_PROPERTIES_ARG}'.", ExitCode.INVALID_ARGS)
- }
- }
+// if (commandLine.hasOption(FLOW_XML_ARG)) {
+// if (isVerbose) {
+// logger.info("Handling encryption of flow.xml.gz")
+// }
+// flowXmlPath = commandLine.getOptionValue(FLOW_XML_ARG)
+// outputFlowXmlPath = commandLine.getOptionValue(OUTPUT_FLOW_XML_ARG, flowXmlPath)
+// handlingFlowXml = true
+//
+// newFlowAlgorithm = commandLine.getOptionValue(NEW_FLOW_ALGORITHM_ARG)
+// newFlowProvider = commandLine.getOptionValue(NEW_FLOW_PROVIDER_ARG)
+//
+// if (flowXmlPath == outputFlowXmlPath) {
+// // TODO: Add confirmation pause and provide -y flag to offer no-interaction mode?
+// logger.warn("The source flow.xml.gz and destination flow.xml.gz are identical [${outputFlowXmlPath}] so the original will be overwritten")
+// }
+//
+// if (!commandLine.hasOption(NIFI_PROPERTIES_ARG)) {
+// printUsageAndThrow("In order to migrate a flow.xml.gz, a nifi.properties file must also be specified via '-n'/'--${NIFI_PROPERTIES_ARG}'.", ExitCode.INVALID_ARGS)
+// }
+// }
Review Comment:
As noted in relation to the arguments, it looks like this can be uncommented for this PR, and then other changes can be handled separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on PR #7661:
URL: https://github.com/apache/nifi/pull/7661#issuecomment-1731987969
> Thanks for thoroughly reviewing @exceptionfactory . I'll plan on reviewing this in more depth next week as well. In the meantime, I do agree with your comment above re: deleting the antiquated integration tests.
Thanks for the confirmation on the integration tests @markap14!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11772 Removing flow.xml support [nifi]
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1350586845
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -236,20 +172,9 @@ private void verifyInMemoryFlowContents() throws NiFiClientException, IOExceptio
assertEquals("65b8f293-016e-1000-7b8f-6c6752fa921b", affectedComponent.getId());
assertEquals(AffectedComponentDTO.COMPONENT_TYPE_PROCESSOR, affectedComponent.getReferenceType());
- // The original Controller Service, whose UUID ended with 00 should be removed and a new one inherited.
- final ControllerServicesEntity controllerLevelServices = node2Client.getFlowClient().getControllerServices();
- assertEquals(1, controllerLevelServices.getControllerServices().size());
-
- final ControllerServiceEntity firstService = controllerLevelServices.getControllerServices().iterator().next();
- assertFalse(firstService.getId().endsWith("00"));
- }
-
- private PropertyEncryptor createEncryptorFromProperties(Properties properties) {
- final NiFiProperties niFiProperties = NiFiProperties.createBasicNiFiProperties(null, properties);
-
- final String propertiesKey = niFiProperties.getProperty(NiFiProperties.SENSITIVE_PROPS_KEY);
- final String propertiesAlgorithm = niFiProperties.getProperty(NiFiProperties.SENSITIVE_PROPS_ALGORITHM);
- return new PropertyEncryptorBuilder(propertiesKey).setAlgorithm(propertiesAlgorithm).build();
+ final ControllerServicesEntity controllerLevelServices = getNifiClient().getFlowClient(DO_NOT_REPLICATE).getControllerServices();
+ final Set<ControllerServiceEntity> controllerServices = controllerLevelServices.getControllerServices();
+ assertEquals(2, controllerServices.size());
Review Comment:
So you're saying that when using the XML form, the final result was 1 Controller Service. With the JSON form, it has two Controller Services, because it didn't remove the service from the first flow?
If that is so, I would say that it's a bug. So I'd say that it would make sense to:
1) Have the system test assert that the size is equal to 1.
2) Use @Disabled on the system test because we know it's not working
3) File a Jira to fix the issue and re-enable the system test.
This way, we can continue on and get this PR merged and avoid tackling other (potentially complex) bugs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1340472803
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/audit/PortAuditor.java:
##########
@@ -93,14 +91,8 @@ public Port updatePortAdvice(ProceedingJoinPoint proceedingJoinPoint, PortDTO po
final int maxConcurrentTasks = port.getMaxConcurrentTasks();
final Set<String> existingUsers = new HashSet<>();
- final Set<String> existingGroups = new HashSet<>();
boolean isPublicPort = false;
Review Comment:
It looks like `isPublicPort` will always be false here, as you've removed the `if (port instanceof PublicPort) {... }`. I think we need to instead set `boolean isPublicPort = port instanceof PublicPort;`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11772 Removing flow.xml support [nifi]
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1342367861
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -248,8 +235,6 @@ class ConfigEncryptionTool {
options.addOption(Option.builder("i").longOpt(OUTPUT_LOGIN_IDENTITY_PROVIDERS_ARG).hasArg(true).argName("file").desc("The destination login-identity-providers.xml file containing protected config values (will not modify input login-identity-providers.xml)").build())
options.addOption(Option.builder("a").longOpt(AUTHORIZERS_ARG).hasArg(true).argName("file").desc("The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)").build())
options.addOption(Option.builder("u").longOpt(OUTPUT_AUTHORIZERS_ARG).hasArg(true).argName("file").desc("The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)").build())
- options.addOption(Option.builder("f").longOpt(FLOW_XML_ARG).hasArg(true).argName("file").desc("The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)").build())
- options.addOption(Option.builder("g").longOpt(OUTPUT_FLOW_XML_ARG).hasArg(true).argName("file").desc("The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)").build())
Review Comment:
Yes, as mentioned in the description, originally I wanted to do the toolkit changes in a follow up story after clarifying the cleanup and general changes within this PR. Depending on your answer for my comment above hovere I will change this part accordingly. For further toolkit related changes I suggest to move them in a followup PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11772 Removing flow.xml support [nifi]
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1347743734
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -106,124 +94,72 @@ public void testStartupWithDifferentFlow() throws IOException, NiFiClientExcepti
node2.stop();
final File node2ConfDir = new File(node2.getInstanceDirectory(), "conf");
- final File flowXmlFile = new File(node2ConfDir, "flow.xml.gz");
- Files.deleteIfExists(flowXmlFile.toPath());
- Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.xml.gz"), flowXmlFile.toPath());
+ final File flowJsonFile = new File(node2ConfDir, "flow.json.gz");
+ Files.deleteIfExists(flowJsonFile.toPath());
+ Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.json.gz"), flowJsonFile.toPath());
node2.start(true);
- final File backupFile = getBackupFile(node2ConfDir);
- final NodeDTO node2Dto = getNodeDtoByApiPort(5672);
+ waitForAllNodesConnected();
+ switchClientToNode(2);
+ final File backupFile = getBackupFile(node2ConfDir);
verifyFlowContentsOnDisk(backupFile);
- disconnectNode(node2Dto);
verifyInMemoryFlowContents();
-
- // Reconnect the node so that we can properly shutdown
- reconnectNode(node2Dto);
}
-
- private List<File> getFlowXmlFiles(final File confDir) {
- final File[] flowXmlFileArray = confDir.listFiles(file -> file.getName().startsWith("flow") && file.getName().endsWith(".xml.gz"));
- final List<File> flowXmlFiles = new ArrayList<>(Arrays.asList(flowXmlFileArray));
- return flowXmlFiles;
+ private List<File> getFlowJsonFiles(final File confDir) {
+ final File[] flowJsonFileArray = confDir.listFiles(file -> file.getName().startsWith("flow") && file.getName().endsWith(".json.gz"));
+ final List<File> flowJsonFiles = new ArrayList<>(Arrays.asList(flowJsonFileArray));
+ return flowJsonFiles;
}
private File getBackupFile(final File confDir) throws InterruptedException {
- waitFor(() -> getFlowXmlFiles(confDir).size() == 2);
-
- final List<File> flowXmlFiles = getFlowXmlFiles(confDir);
- assertEquals(2, flowXmlFiles.size());
+ waitFor(() -> getFlowJsonFiles(confDir).size() == 1);
- flowXmlFiles.removeIf(file -> file.getName().equals("flow.xml.gz"));
+ final List<File> flowJsonFiles = getFlowJsonFiles(confDir);
+ assertEquals(1, flowJsonFiles.size());
- assertEquals(1, flowXmlFiles.size());
- final File backupFile = flowXmlFiles.get(0);
- return backupFile;
+ return flowJsonFiles.iterator().next();
}
private void verifyFlowContentsOnDisk(final File backupFile) throws IOException {
- // Read the flow and make sure that the backup looks the same as the original. We don't just do a byte comparison because the compression may result in different
- // gzipped bytes and because if the two flows do differ, we want to have the String representation so that we can compare to see how they are different.
- final String flowXml = readFlow(backupFile);
- final String expectedFlow = readFlow(new File("src/test/resources/flows/mismatched-flows/flow1.xml.gz"));
-
- assertEquals(expectedFlow, flowXml);
-
// Verify some of the values that were persisted to disk
final File confDir = backupFile.getParentFile();
- final String loadedFlow = readFlow(new File(confDir, "flow.xml.gz"));
+ final String loadedFlow = readFlow(new File(confDir, "flow.json.gz"));
Review Comment:
By some reason there is some misbehaviour with the id handling of the registry clients. At the time of implementation there was some necessary duplication of it, as we needed to support both the legacy solution (getId) and the solution derived from the component handling (getIdentifier). It looks like, in case of synchronization this might cause some issues as id is used for both instanceIdentifier and Identifier. That can couse the following error if we have different registry clients in the synchronized flows:
`Proposed Flow is not inheritable by the flow controller because of differences in missing components: Current flow has missing components that are not considered missing in the proposed flow (65b71b80-016e-1000-7131-40b0976ac45966)`
This is something was discovered within the effors led to this PR but not part of the scope or direct result of the changes. I will raise a separate apache ticket and PR for a fix on this.
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -236,20 +172,9 @@ private void verifyInMemoryFlowContents() throws NiFiClientException, IOExceptio
assertEquals("65b8f293-016e-1000-7b8f-6c6752fa921b", affectedComponent.getId());
assertEquals(AffectedComponentDTO.COMPONENT_TYPE_PROCESSOR, affectedComponent.getReferenceType());
- // The original Controller Service, whose UUID ended with 00 should be removed and a new one inherited.
- final ControllerServicesEntity controllerLevelServices = node2Client.getFlowClient().getControllerServices();
- assertEquals(1, controllerLevelServices.getControllerServices().size());
-
- final ControllerServiceEntity firstService = controllerLevelServices.getControllerServices().iterator().next();
- assertFalse(firstService.getId().endsWith("00"));
- }
-
- private PropertyEncryptor createEncryptorFromProperties(Properties properties) {
- final NiFiProperties niFiProperties = NiFiProperties.createBasicNiFiProperties(null, properties);
-
- final String propertiesKey = niFiProperties.getProperty(NiFiProperties.SENSITIVE_PROPS_KEY);
- final String propertiesAlgorithm = niFiProperties.getProperty(NiFiProperties.SENSITIVE_PROPS_ALGORITHM);
- return new PropertyEncryptorBuilder(propertiesKey).setAlgorithm(propertiesAlgorithm).build();
+ final ControllerServicesEntity controllerLevelServices = getNifiClient().getFlowClient(DO_NOT_REPLICATE).getControllerServices();
+ final Set<ControllerServiceEntity> controllerServices = controllerLevelServices.getControllerServices();
+ assertEquals(2, controllerServices.size());
Review Comment:
I had some troubles with this assertation as it did persistently behaved differently as in case of the XML based synchronization. I looked into the implementation and I found that, the JSON based synchronization behaves differently for the flow itself and the controller/management level parts of the flow definition. While the items part of the flow (root group) might be "replaced" (added and removed) when id is changed, and that part of the test still covers the behaviour correctly, the methods like `inheritControllerServices` within `VersionedFlowSynchronizer` seemingly work differently and does not remove items not appearant in the proposed flow. This is why in case of JSON sync, the resuolt contains the controller service from both flow1 and flow2. @markap14 could you please confirm this behaviour from the part of the `VersionedFlowSynchronizer` is deliberate? Thanks! (Also, thank you for highlighting that the original change sometimes checked on the flow of node 1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] simonbence commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1338595625
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -106,16 +97,16 @@ public void testStartupWithDifferentFlow() throws IOException, NiFiClientExcepti
node2.stop();
final File node2ConfDir = new File(node2.getInstanceDirectory(), "conf");
- final File flowXmlFile = new File(node2ConfDir, "flow.xml.gz");
+ final File flowXmlFile = new File(node2ConfDir, "flow.json.gz");
Files.deleteIfExists(flowXmlFile.toPath());
- Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.xml.gz"), flowXmlFile.toPath());
+ Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.json.gz"), flowXmlFile.toPath());
node2.start(true);
final File backupFile = getBackupFile(node2ConfDir);
final NodeDTO node2Dto = getNodeDtoByApiPort(5672);
- verifyFlowContentsOnDisk(backupFile);
+// verifyFlowContentsOnDisk(backupFile);
Review Comment:
I adjusted this, please check the new commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] simonbence commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1338595072
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/test/resources/properties/conf/duplicates.nifi.properties:
##########
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nifi.flow.configuration.file=./conf/flow.xml.gz
nifi.flow.configuration.file=./conf/flow.json.gz
+nifi.flow.configuration.file=./conf/flow2.json.gz
Review Comment:
Yes. According to the test uses this config, it is intentional to have two values for the same key
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on PR #7661:
URL: https://github.com/apache/nifi/pull/7661#issuecomment-1731986859
Thanks for thoroughly reviewing @exceptionfactory . I'll plan on reviewing this in more depth next week as well. In the meantime, I do agree with your comment above re: deleting the antiquated integration tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1341981733
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -385,26 +370,26 @@ class ConfigEncryptionTool {
}
}
- if (commandLine.hasOption(FLOW_XML_ARG)) {
- if (isVerbose) {
- logger.info("Handling encryption of flow.xml.gz")
- }
- flowXmlPath = commandLine.getOptionValue(FLOW_XML_ARG)
- outputFlowXmlPath = commandLine.getOptionValue(OUTPUT_FLOW_XML_ARG, flowXmlPath)
- handlingFlowXml = true
-
- newFlowAlgorithm = commandLine.getOptionValue(NEW_FLOW_ALGORITHM_ARG)
- newFlowProvider = commandLine.getOptionValue(NEW_FLOW_PROVIDER_ARG)
-
- if (flowXmlPath == outputFlowXmlPath) {
- // TODO: Add confirmation pause and provide -y flag to offer no-interaction mode?
- logger.warn("The source flow.xml.gz and destination flow.xml.gz are identical [${outputFlowXmlPath}] so the original will be overwritten")
- }
-
- if (!commandLine.hasOption(NIFI_PROPERTIES_ARG)) {
- printUsageAndThrow("In order to migrate a flow.xml.gz, a nifi.properties file must also be specified via '-n'/'--${NIFI_PROPERTIES_ARG}'.", ExitCode.INVALID_ARGS)
- }
- }
+// if (commandLine.hasOption(FLOW_XML_ARG)) {
+// if (isVerbose) {
+// logger.info("Handling encryption of flow.xml.gz")
+// }
+// flowXmlPath = commandLine.getOptionValue(FLOW_XML_ARG)
+// outputFlowXmlPath = commandLine.getOptionValue(OUTPUT_FLOW_XML_ARG, flowXmlPath)
+// handlingFlowXml = true
+//
+// newFlowAlgorithm = commandLine.getOptionValue(NEW_FLOW_ALGORITHM_ARG)
+// newFlowProvider = commandLine.getOptionValue(NEW_FLOW_PROVIDER_ARG)
+//
+// if (flowXmlPath == outputFlowXmlPath) {
+// // TODO: Add confirmation pause and provide -y flag to offer no-interaction mode?
+// logger.warn("The source flow.xml.gz and destination flow.xml.gz are identical [${outputFlowXmlPath}] so the original will be overwritten")
+// }
+//
+// if (!commandLine.hasOption(NIFI_PROPERTIES_ARG)) {
+// printUsageAndThrow("In order to migrate a flow.xml.gz, a nifi.properties file must also be specified via '-n'/'--${NIFI_PROPERTIES_ARG}'.", ExitCode.INVALID_ARGS)
+// }
+// }
Review Comment:
This block of code needs to be uncommented and working. Although the code says `flow.xml.gz`, it also works with `flow.json.gz`, so names should be adjusted accordingly.
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -248,8 +235,6 @@ class ConfigEncryptionTool {
options.addOption(Option.builder("i").longOpt(OUTPUT_LOGIN_IDENTITY_PROVIDERS_ARG).hasArg(true).argName("file").desc("The destination login-identity-providers.xml file containing protected config values (will not modify input login-identity-providers.xml)").build())
options.addOption(Option.builder("a").longOpt(AUTHORIZERS_ARG).hasArg(true).argName("file").desc("The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)").build())
options.addOption(Option.builder("u").longOpt(OUTPUT_AUTHORIZERS_ARG).hasArg(true).argName("file").desc("The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)").build())
- options.addOption(Option.builder("f").longOpt(FLOW_XML_ARG).hasArg(true).argName("file").desc("The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)").build())
- options.addOption(Option.builder("g").longOpt(OUTPUT_FLOW_XML_ARG).hasArg(true).argName("file").desc("The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)").build())
Review Comment:
These options need to be retained and changed to indicate flow.json.gz. We will need separate work on the toolkit in general, but at least for this PR, it should continue to function with flow.json.gz.
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -722,7 +707,7 @@ class ConfigEncryptionTool {
final PropertyEncryptor inputEncryptor = new PropertyEncryptorBuilder(existingFlowPassword).setAlgorithm(existingAlgorithm).build()
final PropertyEncryptor outputEncryptor = new PropertyEncryptorBuilder(newFlowPassword).setAlgorithm(newAlgorithm).build()
- final FlowEncryptor flowEncryptor = new StandardFlowEncryptor()
+ final FlowEncryptor flowEncryptor = new XmlFlowEncryptor()
Review Comment:
I may not have communicated clearly in the previous comment, but `XmlFlowEncryptor` can be removed and `StandardFlowEncryptor` should be used in this location.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] simonbence commented on pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on PR #7661:
URL: https://github.com/apache/nifi/pull/7661#issuecomment-1737396683
Hi @exceptionfactory ! Thank you very much for the effort you did put on the review! I added some changes. Also answering your question:
`
One general question is whether we should switch the configuration property or keep the json.file property. Relying on the json.file property name would simplify upgrading without having to change property values. On the other hand, it is a simple manual change that seems reasonable. Thoughts?
`
I was thinking on this myself as well, but I deciced to use the original property name without remarking the file format because from now on, this should be the default format thus it looks both unnecessary and confusing to highlight this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1340475473
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -106,100 +95,57 @@ public void testStartupWithDifferentFlow() throws IOException, NiFiClientExcepti
node2.stop();
final File node2ConfDir = new File(node2.getInstanceDirectory(), "conf");
- final File flowXmlFile = new File(node2ConfDir, "flow.xml.gz");
- Files.deleteIfExists(flowXmlFile.toPath());
- Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.xml.gz"), flowXmlFile.toPath());
+ final File flowJsonFile = new File(node2ConfDir, "flow.json.gz");
+ Files.deleteIfExists(flowJsonFile.toPath());
+ Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.json.gz"), flowJsonFile.toPath());
node2.start(true);
- final File backupFile = getBackupFile(node2ConfDir);
- final NodeDTO node2Dto = getNodeDtoByApiPort(5672);
+ while (getNifiClient().getControllerClient().getNodes().getCluster().getNodes().stream().filter(n -> n.getStatus().equals("CONNECTED")).count() != 2) {
+ Thread.sleep(3000);
+ }
Review Comment:
This should all be replaced with `waitForAllNodesConnected();`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] markap14 commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1340476051
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -106,100 +95,57 @@ public void testStartupWithDifferentFlow() throws IOException, NiFiClientExcepti
node2.stop();
final File node2ConfDir = new File(node2.getInstanceDirectory(), "conf");
- final File flowXmlFile = new File(node2ConfDir, "flow.xml.gz");
- Files.deleteIfExists(flowXmlFile.toPath());
- Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.xml.gz"), flowXmlFile.toPath());
+ final File flowJsonFile = new File(node2ConfDir, "flow.json.gz");
+ Files.deleteIfExists(flowJsonFile.toPath());
+ Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.json.gz"), flowJsonFile.toPath());
node2.start(true);
- final File backupFile = getBackupFile(node2ConfDir);
- final NodeDTO node2Dto = getNodeDtoByApiPort(5672);
+ while (getNifiClient().getControllerClient().getNodes().getCluster().getNodes().stream().filter(n -> n.getStatus().equals("CONNECTED")).count() != 2) {
+ Thread.sleep(3000);
+ }
+ final File backupFile = getBackupFile(node2ConfDir);
verifyFlowContentsOnDisk(backupFile);
- disconnectNode(node2Dto);
verifyInMemoryFlowContents();
-
- // Reconnect the node so that we can properly shutdown
- reconnectNode(node2Dto);
}
-
- private List<File> getFlowXmlFiles(final File confDir) {
- final File[] flowXmlFileArray = confDir.listFiles(file -> file.getName().startsWith("flow") && file.getName().endsWith(".xml.gz"));
- final List<File> flowXmlFiles = new ArrayList<>(Arrays.asList(flowXmlFileArray));
- return flowXmlFiles;
+ private List<File> getFlowJsonFiles(final File confDir) {
+ final File[] flowJsonFileArray = confDir.listFiles(file -> file.getName().startsWith("flow") && file.getName().endsWith(".json.gz"));
+ final List<File> flowJsonFiles = new ArrayList<>(Arrays.asList(flowJsonFileArray));
+ return flowJsonFiles;
}
private File getBackupFile(final File confDir) throws InterruptedException {
- waitFor(() -> getFlowXmlFiles(confDir).size() == 2);
-
- final List<File> flowXmlFiles = getFlowXmlFiles(confDir);
- assertEquals(2, flowXmlFiles.size());
-
- flowXmlFiles.removeIf(file -> file.getName().equals("flow.xml.gz"));
+ waitFor(() -> getFlowJsonFiles(confDir).size() == 1);
+ final List<File> flowXmlFiles = getFlowJsonFiles(confDir);
Review Comment:
Variable name should have been `flowJsonFiles`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] simonbence commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1338605481
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -715,20 +695,20 @@ class ConfigEncryptionTool {
* @param existingProvider the {@link java.security.Provider} to use (defaults to BC)
* @return the encrypted XML content as an InputStream
*/
- private InputStream migrateFlowXmlContent(InputStream flowXmlContent, String existingFlowPassword, String newFlowPassword, String existingAlgorithm = DEFAULT_FLOW_ALGORITHM, String newAlgorithm = DEFAULT_FLOW_ALGORITHM) {
- File tempFlowXmlFile = new File(getTemporaryFlowXmlFile(outputFlowXmlPath).toString())
- final OutputStream flowOutputStream = getFlowOutputStream(tempFlowXmlFile, flowXmlContent instanceof GZIPInputStream)
-
- final PropertyEncryptor inputEncryptor = new PropertyEncryptorBuilder(existingFlowPassword).setAlgorithm(existingAlgorithm).build()
- final PropertyEncryptor outputEncryptor = new PropertyEncryptorBuilder(newFlowPassword).setAlgorithm(newAlgorithm).build()
-
- final FlowEncryptor flowEncryptor = new StandardFlowEncryptor()
- flowEncryptor.processFlow(flowXmlContent, flowOutputStream, inputEncryptor, outputEncryptor)
-
- // Overwrite the original flow file with the migrated flow file
- Files.move(tempFlowXmlFile.toPath(), Paths.get(outputFlowXmlPath), StandardCopyOption.ATOMIC_MOVE)
- loadFlowXml(outputFlowXmlPath)
- }
+// private InputStream migrateFlowXmlContent(InputStream flowXmlContent, String existingFlowPassword, String newFlowPassword, String existingAlgorithm = DEFAULT_FLOW_ALGORITHM, String newAlgorithm = DEFAULT_FLOW_ALGORITHM) {
Review Comment:
I am not sure if this is the right way to move forward. This is a method specific to the XML flow definition and needs the already deleted XML based flow encryptor. I would find it better to remove it alltogether. As a tentative step I did add back the XmlFlowEncryptor but not to it's original place. Right not it is situated within this module until it's future is not decided. I did not yet put much effort to test this as it is now, because 1. I would like to hear your opinion before 2. originally the changes on the toolkit was planned for a followup story. But based on your answer I will adjust my actions on the matter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11772 Removing flow.xml support [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1342717006
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -722,7 +707,7 @@ class ConfigEncryptionTool {
final PropertyEncryptor inputEncryptor = new PropertyEncryptorBuilder(existingFlowPassword).setAlgorithm(existingAlgorithm).build()
final PropertyEncryptor outputEncryptor = new PropertyEncryptorBuilder(newFlowPassword).setAlgorithm(newAlgorithm).build()
- final FlowEncryptor flowEncryptor = new StandardFlowEncryptor()
+ final FlowEncryptor flowEncryptor = new XmlFlowEncryptor()
Review Comment:
Thanks for the reply and clarifying the implementations. In this case, it looks like the implementation should be changed to the JsonFlowEncryptor. The `migrateFlowXmlContent()` method can be renamed to `migrateFlowContent()` and that seems sufficient for this PR. How does that sound?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] simonbence commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1338598808
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -124,61 +115,62 @@ public void testStartupWithDifferentFlow() throws IOException, NiFiClientExcepti
}
- private List<File> getFlowXmlFiles(final File confDir) {
- final File[] flowXmlFileArray = confDir.listFiles(file -> file.getName().startsWith("flow") && file.getName().endsWith(".xml.gz"));
- final List<File> flowXmlFiles = new ArrayList<>(Arrays.asList(flowXmlFileArray));
- return flowXmlFiles;
+ private List<File> getFlowJsonFiles(final File confDir) {
+ final File[] flowJsonFileArray = confDir.listFiles(file -> file.getName().startsWith("flow") && file.getName().endsWith(".json.gz"));
+ final List<File> flowJsonFiles = new ArrayList<>(Arrays.asList(flowJsonFileArray));
+ return flowJsonFiles;
}
private File getBackupFile(final File confDir) throws InterruptedException {
- waitFor(() -> getFlowXmlFiles(confDir).size() == 2);
+ waitFor(() -> getFlowJsonFiles(confDir).size() == 2);
- final List<File> flowXmlFiles = getFlowXmlFiles(confDir);
+ final List<File> flowXmlFiles = getFlowJsonFiles(confDir);
assertEquals(2, flowXmlFiles.size());
- flowXmlFiles.removeIf(file -> file.getName().equals("flow.xml.gz"));
+ flowXmlFiles.removeIf(file -> file.getName().equals("flow.json.gz"));
assertEquals(1, flowXmlFiles.size());
final File backupFile = flowXmlFiles.get(0);
return backupFile;
}
- private void verifyFlowContentsOnDisk(final File backupFile) throws IOException {
- // Read the flow and make sure that the backup looks the same as the original. We don't just do a byte comparison because the compression may result in different
- // gzipped bytes and because if the two flows do differ, we want to have the String representation so that we can compare to see how they are different.
- final String flowXml = readFlow(backupFile);
- final String expectedFlow = readFlow(new File("src/test/resources/flows/mismatched-flows/flow1.xml.gz"));
-
- assertEquals(expectedFlow, flowXml);
-
- // Verify some of the values that were persisted to disk
- final File confDir = backupFile.getParentFile();
- final String loadedFlow = readFlow(new File(confDir, "flow.xml.gz"));
-
- final StandardDocumentProvider documentProvider = new StandardDocumentProvider();
- final Document document = documentProvider.parse(new ByteArrayInputStream(loadedFlow.getBytes(StandardCharsets.UTF_8)));
- final Element rootElement = (Element) document.getElementsByTagName("flowController").item(0);
- final FlowEncodingVersion encodingVersion = FlowEncodingVersion.parse(rootElement);
-
- final NiFiInstance node2 = getNiFiInstance().getNodeInstance(2);
- final PropertyEncryptor encryptor = createEncryptorFromProperties(node2.getProperties());
- final Element rootGroupElement = (Element) rootElement.getElementsByTagName("rootGroup").item(0);
-
- final ProcessGroupDTO groupDto = FlowFromDOMFactory.getProcessGroup(null, rootGroupElement, encryptor, encodingVersion);
- final Set<ProcessGroupDTO> childGroupDtos = groupDto.getContents().getProcessGroups();
- assertEquals(1, childGroupDtos.size());
-
- final ProcessGroupDTO childGroup = childGroupDtos.iterator().next();
- assertFalse(childGroup.getId().endsWith("00"));
- final FlowSnippetDTO childContents = childGroup.getContents();
-
- final Set<ProcessorDTO> childProcessors = childContents.getProcessors();
- assertEquals(1, childProcessors.size());
-
- final ProcessorDTO procDto = childProcessors.iterator().next();
- assertFalse(procDto.getId().endsWith("00"));
- assertFalse(procDto.getName().endsWith("00"));
- }
+ // TODO we need a JSON analogue for this
Review Comment:
I fixed the test within the story. Please take a look. There are some things you might want to ask about, especially the removeal of the disconnection. As far as I understood it 1. did not work as expected even before the PR 2. was unnecessary for the given test scenario. I might miss something so please mention if you disagree with any of the changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi] exceptionfactory commented on a diff in pull request #7661: NIFI-11772 Removing flow.xml support
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1332246606
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/resources/conf/nifi.properties:
##########
@@ -69,6 +69,10 @@ nifi.web.https.host=
nifi.web.https.port=
nifi.web.jetty.working.directory=./target/work/jetty
+# security properties #
+nifi.sensitive.props.key=key
+nifi.sensitive.props.algorithm=PBEWITHMD5AND256BITAES-CBC-OPENSSL
Review Comment:
This algorithm is deprecated and no longer supported on the main branch, so these two properties can be removed.
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -248,8 +228,8 @@ class ConfigEncryptionTool {
options.addOption(Option.builder("i").longOpt(OUTPUT_LOGIN_IDENTITY_PROVIDERS_ARG).hasArg(true).argName("file").desc("The destination login-identity-providers.xml file containing protected config values (will not modify input login-identity-providers.xml)").build())
options.addOption(Option.builder("a").longOpt(AUTHORIZERS_ARG).hasArg(true).argName("file").desc("The authorizers.xml file containing unprotected config values (will be overwritten unless -u is specified)").build())
options.addOption(Option.builder("u").longOpt(OUTPUT_AUTHORIZERS_ARG).hasArg(true).argName("file").desc("The destination authorizers.xml file containing protected config values (will not modify input authorizers.xml)").build())
- options.addOption(Option.builder("f").longOpt(FLOW_XML_ARG).hasArg(true).argName("file").desc("The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)").build())
- options.addOption(Option.builder("g").longOpt(OUTPUT_FLOW_XML_ARG).hasArg(true).argName("file").desc("The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)").build())
+// options.addOption(Option.builder("f").longOpt(FLOW_XML_ARG).hasArg(true).argName("file").desc("The flow.xml.gz file currently protected with old password (will be overwritten unless -g is specified)").build())
+// options.addOption(Option.builder("g").longOpt(OUTPUT_FLOW_XML_ARG).hasArg(true).argName("file").desc("The destination flow.xml.gz file containing protected config values (will not modify input flow.xml.gz)").build())
Review Comment:
These commented lines should be removed.
##########
nifi-commons/nifi-flow-encryptor/src/main/java/org/apache/nifi/flow/encryptor/JsonFlowEncryptor.java:
##########
@@ -33,9 +34,13 @@ public class JsonFlowEncryptor extends AbstractFlowEncryptor {
@Override
public void processFlow(final InputStream inputStream, final OutputStream outputStream,
final PropertyEncryptor inputEncryptor, final PropertyEncryptor outputEncryptor) {
+
+ final BufferedInputStream bufferedInputStream = new BufferedInputStream(inputStream);
+ bufferedInputStream.mark(1);
Review Comment:
This addition does not appear necessary. The only reason for it in the standard class was to support content type detection.
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -106,16 +97,16 @@ public void testStartupWithDifferentFlow() throws IOException, NiFiClientExcepti
node2.stop();
final File node2ConfDir = new File(node2.getInstanceDirectory(), "conf");
- final File flowXmlFile = new File(node2ConfDir, "flow.xml.gz");
+ final File flowXmlFile = new File(node2ConfDir, "flow.json.gz");
Files.deleteIfExists(flowXmlFile.toPath());
- Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.xml.gz"), flowXmlFile.toPath());
+ Files.copy(Paths.get("src/test/resources/flows/mismatched-flows/flow2.json.gz"), flowXmlFile.toPath());
node2.start(true);
final File backupFile = getBackupFile(node2ConfDir);
final NodeDTO node2Dto = getNodeDtoByApiPort(5672);
- verifyFlowContentsOnDisk(backupFile);
+// verifyFlowContentsOnDisk(backupFile);
Review Comment:
Is this verify method not working?
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/resources/FlowConfiguration.xsd:
##########
@@ -410,8 +410,6 @@
<xs:extension base="PortType">
<xs:sequence>
<xs:element name="maxConcurrentTasks" type="xs:positiveInteger" minOccurs="0" ></xs:element>
Review Comment:
This entire file can be deleted.
##########
minifi/minifi-integration-tests/src/test/resources/conf/nifi.properties:
##########
@@ -14,7 +14,6 @@
# limitations under the License.
# Core Properties #
-nifi.flow.configuration.file=./target/flow.xml.gz
Review Comment:
Should this be flow.json.gz?
##########
nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-properties-loader/src/test/resources/properties/conf/duplicates.nifi.properties:
##########
@@ -13,7 +13,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-nifi.flow.configuration.file=./conf/flow.xml.gz
nifi.flow.configuration.file=./conf/flow.json.gz
+nifi.flow.configuration.file=./conf/flow2.json.gz
Review Comment:
Is this change correct?
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -715,20 +695,20 @@ class ConfigEncryptionTool {
* @param existingProvider the {@link java.security.Provider} to use (defaults to BC)
* @return the encrypted XML content as an InputStream
*/
- private InputStream migrateFlowXmlContent(InputStream flowXmlContent, String existingFlowPassword, String newFlowPassword, String existingAlgorithm = DEFAULT_FLOW_ALGORITHM, String newAlgorithm = DEFAULT_FLOW_ALGORITHM) {
- File tempFlowXmlFile = new File(getTemporaryFlowXmlFile(outputFlowXmlPath).toString())
- final OutputStream flowOutputStream = getFlowOutputStream(tempFlowXmlFile, flowXmlContent instanceof GZIPInputStream)
-
- final PropertyEncryptor inputEncryptor = new PropertyEncryptorBuilder(existingFlowPassword).setAlgorithm(existingAlgorithm).build()
- final PropertyEncryptor outputEncryptor = new PropertyEncryptorBuilder(newFlowPassword).setAlgorithm(newAlgorithm).build()
-
- final FlowEncryptor flowEncryptor = new StandardFlowEncryptor()
- flowEncryptor.processFlow(flowXmlContent, flowOutputStream, inputEncryptor, outputEncryptor)
-
- // Overwrite the original flow file with the migrated flow file
- Files.move(tempFlowXmlFile.toPath(), Paths.get(outputFlowXmlPath), StandardCopyOption.ATOMIC_MOVE)
- loadFlowXml(outputFlowXmlPath)
- }
+// private InputStream migrateFlowXmlContent(InputStream flowXmlContent, String existingFlowPassword, String newFlowPassword, String existingAlgorithm = DEFAULT_FLOW_ALGORITHM, String newAlgorithm = DEFAULT_FLOW_ALGORITHM) {
Review Comment:
This needs to be uncommented and working before going forward.
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/FlowSynchronizationIT.java:
##########
@@ -546,7 +546,7 @@ private void restartWithOnlySingleFlowPersistenceFile(final String filenameToDel
node2.stop();
final File confDir = new File(node2.getInstanceDirectory(), "conf");
- assertEquals(1, confDir.listFiles(file -> file.getName().equals("flow.xml.gz")).length);
+ //assertEquals(1, confDir.listFiles(file -> file.getName().equals("flow.xml.gz")).length);
Review Comment:
This commented line should be removed.
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -124,61 +115,62 @@ public void testStartupWithDifferentFlow() throws IOException, NiFiClientExcepti
}
- private List<File> getFlowXmlFiles(final File confDir) {
- final File[] flowXmlFileArray = confDir.listFiles(file -> file.getName().startsWith("flow") && file.getName().endsWith(".xml.gz"));
- final List<File> flowXmlFiles = new ArrayList<>(Arrays.asList(flowXmlFileArray));
- return flowXmlFiles;
+ private List<File> getFlowJsonFiles(final File confDir) {
+ final File[] flowJsonFileArray = confDir.listFiles(file -> file.getName().startsWith("flow") && file.getName().endsWith(".json.gz"));
+ final List<File> flowJsonFiles = new ArrayList<>(Arrays.asList(flowJsonFileArray));
+ return flowJsonFiles;
}
private File getBackupFile(final File confDir) throws InterruptedException {
- waitFor(() -> getFlowXmlFiles(confDir).size() == 2);
+ waitFor(() -> getFlowJsonFiles(confDir).size() == 2);
- final List<File> flowXmlFiles = getFlowXmlFiles(confDir);
+ final List<File> flowXmlFiles = getFlowJsonFiles(confDir);
assertEquals(2, flowXmlFiles.size());
- flowXmlFiles.removeIf(file -> file.getName().equals("flow.xml.gz"));
+ flowXmlFiles.removeIf(file -> file.getName().equals("flow.json.gz"));
assertEquals(1, flowXmlFiles.size());
final File backupFile = flowXmlFiles.get(0);
return backupFile;
}
- private void verifyFlowContentsOnDisk(final File backupFile) throws IOException {
- // Read the flow and make sure that the backup looks the same as the original. We don't just do a byte comparison because the compression may result in different
- // gzipped bytes and because if the two flows do differ, we want to have the String representation so that we can compare to see how they are different.
- final String flowXml = readFlow(backupFile);
- final String expectedFlow = readFlow(new File("src/test/resources/flows/mismatched-flows/flow1.xml.gz"));
-
- assertEquals(expectedFlow, flowXml);
-
- // Verify some of the values that were persisted to disk
- final File confDir = backupFile.getParentFile();
- final String loadedFlow = readFlow(new File(confDir, "flow.xml.gz"));
-
- final StandardDocumentProvider documentProvider = new StandardDocumentProvider();
- final Document document = documentProvider.parse(new ByteArrayInputStream(loadedFlow.getBytes(StandardCharsets.UTF_8)));
- final Element rootElement = (Element) document.getElementsByTagName("flowController").item(0);
- final FlowEncodingVersion encodingVersion = FlowEncodingVersion.parse(rootElement);
-
- final NiFiInstance node2 = getNiFiInstance().getNodeInstance(2);
- final PropertyEncryptor encryptor = createEncryptorFromProperties(node2.getProperties());
- final Element rootGroupElement = (Element) rootElement.getElementsByTagName("rootGroup").item(0);
-
- final ProcessGroupDTO groupDto = FlowFromDOMFactory.getProcessGroup(null, rootGroupElement, encryptor, encodingVersion);
- final Set<ProcessGroupDTO> childGroupDtos = groupDto.getContents().getProcessGroups();
- assertEquals(1, childGroupDtos.size());
-
- final ProcessGroupDTO childGroup = childGroupDtos.iterator().next();
- assertFalse(childGroup.getId().endsWith("00"));
- final FlowSnippetDTO childContents = childGroup.getContents();
-
- final Set<ProcessorDTO> childProcessors = childContents.getProcessors();
- assertEquals(1, childProcessors.size());
-
- final ProcessorDTO procDto = childProcessors.iterator().next();
- assertFalse(procDto.getId().endsWith("00"));
- assertFalse(procDto.getName().endsWith("00"));
- }
+ // TODO we need a JSON analogue for this
Review Comment:
We should remove this method and create a new Jira issue instead of commenting out the method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11772 Removing flow.xml support [nifi]
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1342363037
##########
nifi-toolkit/nifi-toolkit-encrypt-config/src/main/groovy/org/apache/nifi/properties/ConfigEncryptionTool.groovy:
##########
@@ -722,7 +707,7 @@ class ConfigEncryptionTool {
final PropertyEncryptor inputEncryptor = new PropertyEncryptorBuilder(existingFlowPassword).setAlgorithm(existingAlgorithm).build()
final PropertyEncryptor outputEncryptor = new PropertyEncryptorBuilder(newFlowPassword).setAlgorithm(newAlgorithm).build()
- final FlowEncryptor flowEncryptor = new StandardFlowEncryptor()
+ final FlowEncryptor flowEncryptor = new XmlFlowEncryptor()
Review Comment:
In this case I am not 100% sure if I get your intent. `StandardFlowEncryptor` is (actually: was) a just a composite implementation which helped NiFi decide is the `XmlFlowEncryptor` or the `JsonFlowEncryptor` needs to be used. As `XmlFlowEncryptor` is not there anymore `StandardFlowEncryptor` was deleted as well. In the other hand this toolkit method needs an encryptor capable to work with XML, let it be `XmlFlowEncryptor` directly or `StandardFlowEncryptor` (which will call `XmlFlowEncryptor` in the background). So if we are willing to keep providing this particular service (`migrateFlowXmlContent`) we need the `XmlFlowEncryptor` in place. If this is your aim, I would limit it's scope to the toolkit (I added back this way) and would not add back the composite (`StandardFlowEncryptor`) as it provides no feature during the standard run of NiFi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11772 Removing flow.xml support [nifi]
Posted by "simonbence (via GitHub)" <gi...@apache.org>.
simonbence commented on code in PR #7661:
URL: https://github.com/apache/nifi/pull/7661#discussion_r1351838491
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/clustering/JoinClusterWithDifferentFlow.java:
##########
@@ -236,20 +172,9 @@ private void verifyInMemoryFlowContents() throws NiFiClientException, IOExceptio
assertEquals("65b8f293-016e-1000-7b8f-6c6752fa921b", affectedComponent.getId());
assertEquals(AffectedComponentDTO.COMPONENT_TYPE_PROCESSOR, affectedComponent.getReferenceType());
- // The original Controller Service, whose UUID ended with 00 should be removed and a new one inherited.
- final ControllerServicesEntity controllerLevelServices = node2Client.getFlowClient().getControllerServices();
- assertEquals(1, controllerLevelServices.getControllerServices().size());
-
- final ControllerServiceEntity firstService = controllerLevelServices.getControllerServices().iterator().next();
- assertFalse(firstService.getId().endsWith("00"));
- }
-
- private PropertyEncryptor createEncryptorFromProperties(Properties properties) {
- final NiFiProperties niFiProperties = NiFiProperties.createBasicNiFiProperties(null, properties);
-
- final String propertiesKey = niFiProperties.getProperty(NiFiProperties.SENSITIVE_PROPS_KEY);
- final String propertiesAlgorithm = niFiProperties.getProperty(NiFiProperties.SENSITIVE_PROPS_ALGORITHM);
- return new PropertyEncryptorBuilder(propertiesKey).setAlgorithm(propertiesAlgorithm).build();
+ final ControllerServicesEntity controllerLevelServices = getNifiClient().getFlowClient(DO_NOT_REPLICATE).getControllerServices();
+ final Set<ControllerServiceEntity> controllerServices = controllerLevelServices.getControllerServices();
+ assertEquals(2, controllerServices.size());
Review Comment:
I think that is correct. The original test had assertion to check for 1 Controller Service. As I mentioned there are a lot of moving parts (prod code, test code, test data) but based on what I did see, there is a behaviour in difference compared to the XML. I created a ticket for this: https://issues.apache.org/jira/browse/NIFI-12203
I will push a commit soon in order to adjust the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-11772 Removing flow.xml support [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory closed pull request #7661: NIFI-11772 Removing flow.xml support
URL: https://github.com/apache/nifi/pull/7661
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org