You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "markap14 (via GitHub)" <gi...@apache.org> on 2024/03/26 20:02:00 UTC
[PR] NIFI-12959: Support loading python processors from NARs [nifi]
markap14 opened a new pull request, #8573:
URL: https://github.com/apache/nifi/pull/8573
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-00000](https://issues.apache.org/jira/browse/NIFI-00000)
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [ ] Pull Request based on current revision of the `main` branch
- [ ] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [ ] Build completed using `mvn clean install -P contrib-check`
- [ ] JDK 21
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [ ] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on code in PR #8573:
URL: https://github.com/apache/nifi/pull/8573#discussion_r1551643116
##########
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##########
@@ -234,8 +240,18 @@ private Process launchPythonProcess(final int listeningPort, final String authTo
final List<String> commands = new ArrayList<>();
commands.add(pythonCommand);
+ if (!isUseVirtualEnv()) {
+ // If not using venv, we will not launch a separate virtual environment, so we need to use the -S
+ // flag in order to prevent the Python process from using the installation's site-packages. This provides
+ // proper dependency isolation to the Python process.
+ commands.add("-S");
+ }
String pythonPath = pythonApiDirectory.getAbsolutePath();
+ final String absolutePath = virtualEnvHome.getAbsolutePath();
+ final File dependenciesDir = new File(new File(absolutePath), "NAR-INF/bundled-dependencies");
Review Comment:
Good catch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #8573:
URL: https://github.com/apache/nifi/pull/8573#discussion_r1541949562
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/python/PythonNarIT.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.tests.system.python;
+
+import org.apache.nifi.tests.system.NiFiInstanceFactory;
+import org.apache.nifi.tests.system.NiFiSystemIT;
+import org.apache.nifi.toolkit.cli.impl.client.nifi.NiFiClientException;
+import org.apache.nifi.web.api.entity.ConnectionEntity;
+import org.apache.nifi.web.api.entity.ProcessorEntity;
+import org.junit.jupiter.api.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.StandardCopyOption;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PythonNarIT extends NiFiSystemIT {
+
+ @Override
+ public NiFiInstanceFactory getInstanceFactory() {
+ return createPythonicInstanceFactory();
+ }
+
+ @Override
+ protected boolean isAllowFactoryReuse() {
+ return false;
+ }
+
+ @Test
+ public void testRunProcessorWithBundledDependencies() throws IOException, NiFiClientException, InterruptedException {
+ final File nifiHome = getNiFiInstance().getInstanceDirectory();
+ final File lib = new File(nifiHome, "lib");
+ final File pythonNars = new File(lib, "python-nars");
+ final File[] narFiles = pythonNars.listFiles((dir, name) -> name.endsWith(".nar"));
+ assertNotNull(narFiles);
+
+ // Copy the python nar files to the lib directory so that they will be loaded on restart.
+ for (final File narFile : narFiles) {
+ Files.copy(narFile.toPath(), lib.toPath().resolve(narFile.getName()), StandardCopyOption.REPLACE_EXISTING);
+ }
+
+ // Delete all python extensions from the python/extensions directory to ensure they are not loaded.
+ final File pythonExtensions = new File(nifiHome, "python/extensions");
+ final File[] extensionFiles = pythonExtensions.listFiles();
+ assertNotNull(extensionFiles);
+ for (final File extensionFile : extensionFiles) {
+ deleteRecursively(extensionFile);
+ }
+
+ // Restart NiFi
+ getNiFiInstance().stop();
+ getNiFiInstance().start(true);
+
+ // Create instance of the WriteNumpyVersionBundledDependency processor, and connect a GenerateFlowFile to it
+ final ProcessorEntity generate = getClientUtil().createProcessor("GenerateFlowFile");
+ ProcessorEntity writeVersion = getClientUtil().createPythonProcessor("WriteJson5VersionBundledDependency");
+ writeVersion = getClientUtil().setAutoTerminatedRelationships(writeVersion, "failure");
+ final ConnectionEntity generateToWriteVersion = getClientUtil().createConnection(generate, writeVersion, "success");
+
+ // Create a TerminateFlowFile processor and connect the WriteNumpyVersionBundledDependency processor to it
+ final ProcessorEntity terminate = getClientUtil().createProcessor("TerminateFlowFile");
+ final ConnectionEntity writeVersionToTerminate = getClientUtil().createConnection(writeVersion, terminate, "success");
+
+ // Wait for processor validation to complete
+ getClientUtil().waitForValidProcessor(generate.getId());
+ getClientUtil().waitForValidProcessor(writeVersion.getId());
+
+ // Run the flow
+ getClientUtil().startProcessor(generate);
+ waitForQueueCount(generateToWriteVersion.getId(), 1);
+ getClientUtil().startProcessor(writeVersion);
+ waitForQueueCount(writeVersionToTerminate.getId(), 1);
+
+ // Verify the output
+ final String contents = getClientUtil().getFlowFileContentAsUtf8(writeVersionToTerminate.getId(), 0);
+ assertEquals("qpzry9x8gf2tvdw0s3jn54khce6mua7l", contents);
Review Comment:
Recommend calling this string out as a static variable named `BECH32_CHARSET`.
##########
nifi-system-tests/nifi-system-test-suite/src/test/java/org/apache/nifi/tests/system/python/PythonNarIT.java:
##########
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.tests.system.python;
+
+import org.apache.nifi.tests.system.NiFiInstanceFactory;
+import org.apache.nifi.tests.system.NiFiSystemIT;
+import org.apache.nifi.toolkit.cli.impl.client.nifi.NiFiClientException;
+import org.apache.nifi.web.api.entity.ConnectionEntity;
+import org.apache.nifi.web.api.entity.ProcessorEntity;
+import org.junit.jupiter.api.Test;
+
+import java.io.File;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.StandardCopyOption;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class PythonNarIT extends NiFiSystemIT {
+
+ @Override
+ public NiFiInstanceFactory getInstanceFactory() {
+ return createPythonicInstanceFactory();
+ }
+
+ @Override
+ protected boolean isAllowFactoryReuse() {
+ return false;
+ }
+
+ @Test
+ public void testRunProcessorWithBundledDependencies() throws IOException, NiFiClientException, InterruptedException {
+ final File nifiHome = getNiFiInstance().getInstanceDirectory();
+ final File lib = new File(nifiHome, "lib");
+ final File pythonNars = new File(lib, "python-nars");
+ final File[] narFiles = pythonNars.listFiles((dir, name) -> name.endsWith(".nar"));
+ assertNotNull(narFiles);
+
+ // Copy the python nar files to the lib directory so that they will be loaded on restart.
+ for (final File narFile : narFiles) {
+ Files.copy(narFile.toPath(), lib.toPath().resolve(narFile.getName()), StandardCopyOption.REPLACE_EXISTING);
+ }
+
+ // Delete all python extensions from the python/extensions directory to ensure they are not loaded.
+ final File pythonExtensions = new File(nifiHome, "python/extensions");
+ final File[] extensionFiles = pythonExtensions.listFiles();
+ assertNotNull(extensionFiles);
+ for (final File extensionFile : extensionFiles) {
+ deleteRecursively(extensionFile);
+ }
+
+ // Restart NiFi
+ getNiFiInstance().stop();
+ getNiFiInstance().start(true);
+
+ // Create instance of the WriteNumpyVersionBundledDependency processor, and connect a GenerateFlowFile to it
+ final ProcessorEntity generate = getClientUtil().createProcessor("GenerateFlowFile");
+ ProcessorEntity writeVersion = getClientUtil().createPythonProcessor("WriteJson5VersionBundledDependency");
Review Comment:
Note that this reference should be changed to match the updated Processor name.
##########
nifi-system-tests/nifi-python-test-extensions-nar/src/main/resources/WriteJson5VersionBundledDependency.py:
##########
@@ -0,0 +1,30 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import bech32
+from nifiapi.flowfiletransform import FlowFileTransform, FlowFileTransformResult
+
+
+class WriteJson5VersionBundledDependency(FlowFileTransform):
Review Comment:
This Processor should be renamed to remove `Json5` since it is using `bech32`:
```suggestion
class WriteLibraryPropertyBundledDependency(FlowFileTransform):
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory closed pull request #8573: NIFI-12959: Support loading python processors from NARs
URL: https://github.com/apache/nifi/pull/8573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on PR #8573:
URL: https://github.com/apache/nifi/pull/8573#issuecomment-2022845935
@dan-s1 thanks great catch. I removed some carrige-return-newline combos in that file and my IDE did something I didn't expect :) Fixed that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
Posted by "pvillard31 (via GitHub)" <gi...@apache.org>.
pvillard31 commented on PR #8573:
URL: https://github.com/apache/nifi/pull/8573#issuecomment-2022800770
Yeah, I must admit that I tried to figure out a way to use ParseDocument with NiFi in a container and I didn't manage to find a clean way to bring Tesseract into it. I feel like it would probably be easier to have it as a side container or something but I didn't find a way to piece everything together so I was naively (without too much hopes) thinking that this improvement may help :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
Posted by "samer1977 (via GitHub)" <gi...@apache.org>.
samer1977 commented on PR #8573:
URL: https://github.com/apache/nifi/pull/8573#issuecomment-2409218084
Hi ,
When I try to follow the folder structure:
```
my-nar.nar
+-- META-INF/
+-- MANIFEST.MF
+-- NAR-INF/
+-- bundled-dependencies/
+-- dependency1
+-- dependency2
+-- etc.
+-- MyProcessor.py
```
I keep getting the following error:
`java.io.FileNotFoundException: work\nar\extensions\nifi-MyProcessor-nar-2.0.0-M4.nar-unpacked\MyProcessor.py (The system cannot find the path specified)`
Please advise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
Posted by "exceptionfactory (via GitHub)" <gi...@apache.org>.
exceptionfactory commented on code in PR #8573:
URL: https://github.com/apache/nifi/pull/8573#discussion_r1550777551
##########
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##########
@@ -224,6 +226,10 @@ private String generateAuthToken() {
return Base64.getEncoder().encodeToString(bytes);
}
+ private boolean isUseVirtualEnv() {
Review Comment:
It looks like the only use of this method is a negated reference. Can this be removed and replaced with `if (packagedWithDependencies)`?
##########
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##########
@@ -234,8 +240,18 @@ private Process launchPythonProcess(final int listeningPort, final String authTo
final List<String> commands = new ArrayList<>();
commands.add(pythonCommand);
+ if (!isUseVirtualEnv()) {
+ // If not using venv, we will not launch a separate virtual environment, so we need to use the -S
+ // flag in order to prevent the Python process from using the installation's site-packages. This provides
+ // proper dependency isolation to the Python process.
+ commands.add("-S");
+ }
String pythonPath = pythonApiDirectory.getAbsolutePath();
+ final String absolutePath = virtualEnvHome.getAbsolutePath();
+ final File dependenciesDir = new File(new File(absolutePath), "NAR-INF/bundled-dependencies");
Review Comment:
Is the intent to add `NAR-INF/bundled-dependencies` in all cases? It seems like it should be conditional on whether the Processor is packaged with dependencies.
##########
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/main/java/org/apache/nifi/py4j/PythonProcess.java:
##########
@@ -287,6 +309,13 @@ String resolvePythonCommand() throws IOException {
private void setupEnvironment() throws IOException {
+ // Environment creation is only necessary if using PIP. Otherwise, the Process requires no outside dependencies, other than those
+ // provided in the package and thus we can simply include those packages in the PYTHON_PATH.
+ if (!isUseVirtualEnv()) {
+ logger.debug("Will not create Python Virtual Environment because PIP is disabled in nifi.properties");
Review Comment:
Very minor, but general usage of `pip` is all lowercase and opposed to all uppercase. More importantly though, the message does not reflect the behavior, as it is only based on the the presence of packaged dependencies.
```suggestion
logger.debug("Python Virtual Environment not created: Python Processor packaged with dependencies");
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
Posted by "markap14 (via GitHub)" <gi...@apache.org>.
markap14 commented on PR #8573:
URL: https://github.com/apache/nifi/pull/8573#issuecomment-2022733504
@pvillard31 no, it wouldn't really help there. The issue there is that a native library like Tesseract would need to be compiled for the correct OS and architecture. So in order to include that, we'd need to include many copies of the library. And currently, we do not set any sort of environment variables telling it to search for native libraries in the NAR's unpacked directory. It might be worth exploring as a future improvement, though. Even though it may not help for something like ParseDocument, it might be helpful at least for custom nars, where perhaps you know that you're only going to run in containers or on a specific architecture/os so you can include the appropriate library for the custom nar?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
Posted by "dan-s1 (via GitHub)" <gi...@apache.org>.
dan-s1 commented on code in PR #8573:
URL: https://github.com/apache/nifi/pull/8573#discussion_r1540151244
##########
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/test/java/org/apache/nifi/py4j/PythonProcessTest.java:
##########
@@ -56,51 +56,46 @@ class PythonProcessTest {
@BeforeEach
public void setUp() {
- this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, "Controller", "Controller");
+ this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, false, "Controller", "Controller");
+ }
+
+ @Test
+ void testUsesConfiguredValueWhenPackagedWithDependencies() throws IOException {
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
+ final PythonProcess process = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, true, "Controller", "Controller");
+ assertEquals(PYTHON_CMD, process.resolvePythonCommand());
}
@Test
void testResolvePythonCommandWindows() throws IOException {
- final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
- assertTrue(scriptsDir.mkdir());
+ final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR); assertTrue(scriptsDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandUnix() throws IOException {
- final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
- assertTrue(binDir.mkdir());
+ final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR); assertTrue(binDir.mkdir());
Review Comment:
```suggestion
final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
assertTrue(binDir.mkdir());
```
##########
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/test/java/org/apache/nifi/py4j/PythonProcessTest.java:
##########
@@ -56,51 +56,46 @@ class PythonProcessTest {
@BeforeEach
public void setUp() {
- this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, "Controller", "Controller");
+ this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, false, "Controller", "Controller");
+ }
+
+ @Test
+ void testUsesConfiguredValueWhenPackagedWithDependencies() throws IOException {
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
+ final PythonProcess process = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, true, "Controller", "Controller");
+ assertEquals(PYTHON_CMD, process.resolvePythonCommand());
}
@Test
void testResolvePythonCommandWindows() throws IOException {
- final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
- assertTrue(scriptsDir.mkdir());
+ final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR); assertTrue(scriptsDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandUnix() throws IOException {
- final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
- assertTrue(binDir.mkdir());
+ final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR); assertTrue(binDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(UNIX_BIN_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(UNIX_BIN_DIR); assertEquals(expected, result);
Review Comment:
```suggestion
final String expected = getExpectedBinaryPath(UNIX_BIN_DIR);
assertEquals(expected, result);
```
##########
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/test/java/org/apache/nifi/py4j/PythonProcessTest.java:
##########
@@ -56,51 +56,46 @@ class PythonProcessTest {
@BeforeEach
public void setUp() {
- this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, "Controller", "Controller");
+ this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, false, "Controller", "Controller");
+ }
+
+ @Test
+ void testUsesConfiguredValueWhenPackagedWithDependencies() throws IOException {
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
+ final PythonProcess process = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, true, "Controller", "Controller");
+ assertEquals(PYTHON_CMD, process.resolvePythonCommand());
}
@Test
void testResolvePythonCommandWindows() throws IOException {
- final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
- assertTrue(scriptsDir.mkdir());
+ final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR); assertTrue(scriptsDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandUnix() throws IOException {
- final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
- assertTrue(binDir.mkdir());
+ final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR); assertTrue(binDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(UNIX_BIN_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(UNIX_BIN_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandPreferBin() throws IOException {
- final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
- assertTrue(binDir.mkdir());
- final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
- assertTrue(scriptsDir.mkdir());
+ final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR); assertTrue(binDir.mkdir()); final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR); assertTrue(scriptsDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(UNIX_BIN_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(UNIX_BIN_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandNone() {
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- assertThrows(IOException.class, ()-> this.pythonProcess.resolvePythonCommand());
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); assertThrows(IOException.class, () -> this.pythonProcess.resolvePythonCommand());
Review Comment:
```suggestion
when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
assertThrows(IOException.class, () -> this.pythonProcess.resolvePythonCommand());
```
##########
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/test/java/org/apache/nifi/py4j/PythonProcessTest.java:
##########
@@ -56,51 +56,46 @@ class PythonProcessTest {
@BeforeEach
public void setUp() {
- this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, "Controller", "Controller");
+ this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, false, "Controller", "Controller");
+ }
+
+ @Test
+ void testUsesConfiguredValueWhenPackagedWithDependencies() throws IOException {
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
+ final PythonProcess process = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, true, "Controller", "Controller");
+ assertEquals(PYTHON_CMD, process.resolvePythonCommand());
}
@Test
void testResolvePythonCommandWindows() throws IOException {
- final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
- assertTrue(scriptsDir.mkdir());
+ final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR); assertTrue(scriptsDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandUnix() throws IOException {
- final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
- assertTrue(binDir.mkdir());
+ final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR); assertTrue(binDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(UNIX_BIN_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(UNIX_BIN_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandPreferBin() throws IOException {
- final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
- assertTrue(binDir.mkdir());
- final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
- assertTrue(scriptsDir.mkdir());
+ final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR); assertTrue(binDir.mkdir()); final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR); assertTrue(scriptsDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(UNIX_BIN_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(UNIX_BIN_DIR); assertEquals(expected, result);
Review Comment:
```suggestion
final String expected = getExpectedBinaryPath(UNIX_BIN_DIR);
assertEquals(expected, result);
```
##########
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/test/java/org/apache/nifi/py4j/PythonProcessTest.java:
##########
@@ -56,51 +56,46 @@ class PythonProcessTest {
@BeforeEach
public void setUp() {
- this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, "Controller", "Controller");
+ this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, false, "Controller", "Controller");
+ }
+
+ @Test
+ void testUsesConfiguredValueWhenPackagedWithDependencies() throws IOException {
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
+ final PythonProcess process = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, true, "Controller", "Controller");
+ assertEquals(PYTHON_CMD, process.resolvePythonCommand());
}
@Test
void testResolvePythonCommandWindows() throws IOException {
- final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
- assertTrue(scriptsDir.mkdir());
+ final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR); assertTrue(scriptsDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandUnix() throws IOException {
- final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
- assertTrue(binDir.mkdir());
+ final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR); assertTrue(binDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(UNIX_BIN_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(UNIX_BIN_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandPreferBin() throws IOException {
- final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
- assertTrue(binDir.mkdir());
- final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
- assertTrue(scriptsDir.mkdir());
+ final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR); assertTrue(binDir.mkdir()); final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR); assertTrue(scriptsDir.mkdir());
Review Comment:
```suggestion
final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
assertTrue(binDir.mkdir());
final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
assertTrue(scriptsDir.mkdir());
```
##########
nifi-nar-bundles/nifi-py4j-bundle/nifi-py4j-bridge/src/test/java/org/apache/nifi/py4j/PythonProcessTest.java:
##########
@@ -56,51 +56,46 @@ class PythonProcessTest {
@BeforeEach
public void setUp() {
- this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, "Controller", "Controller");
+ this.pythonProcess = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, false, "Controller", "Controller");
+ }
+
+ @Test
+ void testUsesConfiguredValueWhenPackagedWithDependencies() throws IOException {
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
+ final PythonProcess process = new PythonProcess(this.pythonProcessConfig, this.controllerServiceTypeLookup, virtualEnvHome, true, "Controller", "Controller");
+ assertEquals(PYTHON_CMD, process.resolvePythonCommand());
}
@Test
void testResolvePythonCommandWindows() throws IOException {
- final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
- assertTrue(scriptsDir.mkdir());
+ final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR); assertTrue(scriptsDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(WINDOWS_SCRIPTS_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandUnix() throws IOException {
- final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
- assertTrue(binDir.mkdir());
+ final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR); assertTrue(binDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
- final String expected = getExpectedBinaryPath(UNIX_BIN_DIR);
- assertEquals(expected, result);
+ final String expected = getExpectedBinaryPath(UNIX_BIN_DIR); assertEquals(expected, result);
}
@Test
void testResolvePythonCommandPreferBin() throws IOException {
- final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR);
- assertTrue(binDir.mkdir());
- final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR);
- assertTrue(scriptsDir.mkdir());
+ final File binDir = new File(virtualEnvHome, UNIX_BIN_DIR); assertTrue(binDir.mkdir()); final File scriptsDir = new File(virtualEnvHome, WINDOWS_SCRIPTS_DIR); assertTrue(scriptsDir.mkdir());
- when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
- final String result = this.pythonProcess.resolvePythonCommand();
+ when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD); final String result = this.pythonProcess.resolvePythonCommand();
Review Comment:
```suggestion
when(pythonProcessConfig.getPythonCommand()).thenReturn(PYTHON_CMD);
final String result = this.pythonProcess.resolvePythonCommand();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12959: Support loading python processors from NARs [nifi]
Posted by "pvillard31 (via GitHub)" <gi...@apache.org>.
pvillard31 commented on PR #8573:
URL: https://github.com/apache/nifi/pull/8573#issuecomment-2022389088
Nice improvement Mark! Would this approach help with a processor like ParseDocument where Tesseract has to be installed on the node first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org