You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "martinzink (via GitHub)" <gi...@apache.org> on 2025/10/08 14:11:08 UTC

[PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

martinzink opened a new pull request, #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044

   The docker tests are a great way to do integration testing but it became a bit clunky over the years.
   I've recreated the framework with simplicity in mind. Its now be able to create a flow (without concrete processor classes) and do simple checking with them. Any domain specific checking or containers should be moved into the respective extension.
   
   This will enable us to use the framework to test separetly built extensions in the future (upcoming stable c api).
   And hopefully this will make writing of these tests to be a bit more straight forward.
   
   ---
   
   Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
   
   In order to streamline the review of the contribution we ask you
   to ensure the following steps have been taken:
   
   ### For all changes:
   - [ ] Is there a JIRA ticket associated with this PR? Is it referenced
        in the commit message?
   
   - [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
   
   - [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
   
   - [ ] Is your initial contribution a single, squashed commit?
   
   ### For code changes:
   - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
   - [ ] If applicable, have you updated the LICENSE file?
   - [ ] If applicable, have you updated the NOTICE file?
   
   ### For documentation related changes:
   - [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
   
   ### Note:
   Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2503211896


##########
extensions/aws/tests/features/steps/s3_server_container.py:
##########


Review Comment:
   my comment about the quotes was a reply to a thread in the second copilot review, that's why you can't reply here. I'll resolve it again.



##########
extensions/aws/tests/features/steps/s3_server_container.py:
##########


Review Comment:
   my comment about the quotes was a reply to a thread in the second copilot review, that's why you can't reply here. I'll mark it resolved again.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits closed pull request #2044: MINIFICPP-2624 Modularized behave tests part 1
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469167631


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:

Review Comment:
   👍 https://github.com/apache/nifi-minifi-cpp/commit/d58939bf209e4ed1cf9428404b1b1843a970bf69#diff-b9acc60a55b3f931c457c238886505df8e3401fcb942eae49005cac3f96c397bL102



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469940859


##########
behave_framework/src/minifi_test_framework/containers/docker_image_builder.py:
##########
@@ -0,0 +1,88 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import tempfile
+from docker.models.images import Image
+from typing import Optional
+
+import docker

Review Comment:
   I'm not that familiar with it either, it was just more logical to me to list all the `import` package expressions first then the `from` package imports, but the auto format should be okay.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466564300


##########
extensions/sql/tests/features/environment.py:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import os
+from textwrap import dedent
+
+from minifi_test_framework.containers.docker_container_builder import DockerContainerBuilder
+from minifi_test_framework.core.hooks import common_before_scenario
+from minifi_test_framework.core.hooks import common_after_scenario
+from minifi_test_framework.core.hooks import get_minifi_container_image
+
+
+def before_all(context):
+    minifi_tag_prefix = os.environ['MINIFI_TAG_PREFIX']

Review Comment:
   👍 good catch thanks
   [fix sql tests](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/288a426c498e846335d0f8c423712c03e5df2612)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2527608350


##########
behave_framework/src/minifi_test_framework/steps/core_steps.py:
##########
@@ -0,0 +1,56 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import random
+import string
+
+import humanfriendly
+from behave import when, step
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+@when("both instances start up")
+@when("all instances start up")
+def step_impl(context: MinifiTestContext):
+    for container in context.containers:
+        assert container.deploy()
+    assert context.minifi_container.deploy() or context.minifi_container.get_logs()

Review Comment:
   👍 [fix log_app_output](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/2878db03199a1186e3a7f2fbdf24130a2e6dc91d)



##########
behave_framework/src/minifi_test_framework/core/helpers.py:
##########
@@ -0,0 +1,77 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from __future__ import annotations
+
+import logging
+import time
+from collections.abc import Callable
+
+import docker
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+def log_due_to_failure(context: MinifiTestContext | None):
+    if context is not None:
+        for container in context.containers:
+            container.log_app_output()
+        context.minifi_container.log_app_output()
+
+
+def wait_for_condition(condition: Callable[[], bool], timeout_seconds: float, bail_condition: Callable[[], bool],
+                       context: MinifiTestContext | None) -> bool:
+    if bail_condition():
+        logging.warning("Bail condition evaluated to 'True', aborting wait.")
+        log_due_to_failure(context)
+        return False
+    start_time = time.monotonic()
+    try:
+        while time.monotonic() - start_time < timeout_seconds:
+            if condition():
+                return True
+            if bail_condition():
+                logging.warning("Bail condition evaluated to 'True', aborting wait.")
+                log_due_to_failure(context)
+                return False
+            remaining_time = timeout_seconds - (time.monotonic() - start_time)
+            target_sleep = 1.0 if timeout_seconds < 30 else 10.0

Review Comment:
   👍 [use flat 1 sec wait in wait_for_condition](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/c0b8997cd8648d7a6e12646c496fbc1060a36512)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2523011396


##########
behave_framework/src/minifi_test_framework/steps/core_steps.py:
##########
@@ -0,0 +1,56 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import random
+import string
+
+import humanfriendly
+from behave import when, step
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+@when("both instances start up")
+@when("all instances start up")
+def step_impl(context: MinifiTestContext):
+    for container in context.containers:
+        assert container.deploy()
+    assert context.minifi_container.deploy() or context.minifi_container.get_logs()

Review Comment:
   Here the `context.minifi_container.get_logs()` only returns the log string but does not print the logs, I suppose this should be `log_app_output`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466581544


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""
+
+        exit_code, output = self.exec_run(command)
+
+        return exit_code == 0
+
+    def directory_contains_file_with_regex(self, directory_path: str, regex_str: str) -> bool:
+        if not self.container:
+            return False
+
+        safe_dir_path = shlex.quote(directory_path)
+        safe_regex_str = shlex.quote(regex_str)
+
+        command = (
+            f"find {safe_dir_path} -maxdepth 1 -type f -print0 | "
+            f"xargs -0 -r grep -l -E -- {safe_regex_str}"
+        )
+
+        exit_code, output = self.exec_run(f"sh -c \"{command}\"")
+
+        return exit_code == 0
+
+    def path_with_content_exists(self, path: str, content: str) -> bool:
+        count_command = f"sh -c 'cat {path} | grep \"^{content}$\" | wc -l'"
+        exit_code, output = self.exec_run(count_command)
+        if exit_code != 0:
+            logging.error(f"Error running command '{count_command}': {output}")
+            return False
+
+        try:
+            file_count = int(output.strip())
+        except (ValueError, IndexError):
+            logging.error(f"Error parsing output '{output}' from command '{count_command}'")
+            return False
+        return file_count == 1
+
+    def directory_has_single_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        count_command = f"sh -c 'find {directory_path} -maxdepth 1 -type f | wc -l'"
+        exit_code, output = self.exec_run(count_command)
+
+        if exit_code != 0:
+            logging.error(f"Error running command '{count_command}': {output}")
+            return False
+
+        try:
+            file_count = int(output.strip())
+        except (ValueError, IndexError):
+            logging.error(f"Error parsing output '{output}' from command '{count_command}'")
+            return False
+
+        if file_count != 1:
+            logging.error(f"{directory_path} has too many or too few ({file_count}) files")
+            return False
+
+        content_command = f"sh -c 'cat {directory_path}/*'"
+        exit_code, output = self.exec_run(content_command)
+
+        if exit_code != 0:
+            logging.error(f"Error running command '{content_command}': {output}")
+            return False
+
+        actual_content = output.strip()
+        logging.debug(f"Comparing: '{actual_content}' vs {expected_content}")
+        return actual_content == expected_content.strip()
+
+    def get_logs(self) -> str:
+        logging.debug("Getting logs from container '%s'", self.container_name)
+        if not self.container:
+            return ""
+        logs_as_bytes = self.container.logs()
+        return logs_as_bytes.decode('utf-8')
+
+    @property
+    def exited(self) -> bool:
+        if not self.container:
+            return False
+        try:
+            self.container.reload()
+            return self.container.status == 'exited'
+        except docker.errors.NotFound:
+            self.container = None
+            return False
+        except Exception:
+            return False
+
+    def _get_stats(self) -> Dict[str, Any]:
+        self.container.reload()
+        return self.container.stats(stream=False)
+
+    def get_number_of_files(self, directory_path: str) -> int:
+        if not self.container:
+            return False
+
+        count_command = f"sh -c 'find {directory_path} -maxdepth 1 -type f | wc -l'"
+        exit_code, output = self.exec_run(count_command)
+
+        if exit_code != 0:
+            logging.error(f"Error running command '{count_command}': {output}")
+            return False

Review Comment:
   👍 [get_number_of_files return -1 instead of False](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/f62a2c69d59de26e7a1813baef7922a6bf729f16)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466578939


##########
docker/test/integration/features/consumekafka.feature:
##########


Review Comment:
   [remove already rewritten tests from legacy consumekafka.feature](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/4fb0b25a8e6ec170f4e6771ab412c713130c75d1)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466586508


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):

Review Comment:
   https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/d58939bf209e4ed1cf9428404b1b1843a970bf69#diff-b9acc60a55b3f931c457c238886505df8e3401fcb942eae49005cac3f96c397bR104



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2499384152


##########
.github/workflows/create-release-artifacts.yml:
##########
@@ -0,0 +1,42 @@
+name: "Create Release Artifacts"
+
+on: [workflow_dispatch]
+env:
+  CMAKE_FLAGS: -DCMAKE_BUILD_TYPE=Release -DCI_BUILD=ON -DENABLE_ALL=ON -DMINIFI_FAIL_ON_WARNINGS=OFF -DDOCKER_BUILD_ONLY=ON

Review Comment:
   release artifacts shouldn't be CI_BUILD=ON



##########
docker/RunBehaveTests.sh:
##########
@@ -0,0 +1,200 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,

Review Comment:
   quotes in the license headers shouldn't be escaped, they're just comments



##########
extensions/azure/tests/features/environment.py:
##########


Review Comment:
   A comment describing the purpose of mechanics like this would be nice.



##########
extensions/aws/tests/features/steps/s3_server_container.py:
##########


Review Comment:
   multiple files are missing license header



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2486843415


##########
.github/workflows/ci.yml:
##########
@@ -535,6 +535,64 @@ jobs:
         with:
           name: behavex_output
           path: build/behavex_output
+  modular_docker_tests:
+    name: "Modular Docker integration tests (x86_64)"
+    needs: docker_build
+    runs-on: ubuntu-24.04
+    timeout-minutes: 180
+    steps:
+      - id: checkout
+        uses: actions/checkout@v4
+      - id: run_cmake
+        name: Run CMake
+        run: |
+          mkdir build
+          cd build
+          cmake ${DOCKER_CMAKE_FLAGS} ..
+      - name: Download artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: minifi_docker
+          path: build
+      - name: Load Docker image
+        run: |
+          docker load --input ./build/minifi_docker.tar
+      - id: install_deps
+        name: Install dependencies for Docker Verify
+        run: |
+          sudo apt update
+          sudo apt install -y python3-virtualenv
+      - id: free_disk_space
+        run: |
+          # We can gain additional disk space on the Ubuntu runners thanks to these suggestions:
+          # https://github.com/actions/runner-images/issues/2840#issuecomment-790492173
+          # https://github.com/actions/runner-images/issues/2606#issuecomment-772683150
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /usr/local/lib/android
+          sudo rm -rf /opt/ghc
+          sudo rm -rf "/usr/local/share/boost"
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY"
+      - id: test
+        name: Docker Verify
+        working-directory: ./build
+        run: make docker-verify-modular

Review Comment:
   good idea, I've changed the typing package imports here https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/7cb26f05f6806ba8cbb7e40c8ef078352ddc10d6
   and added documentation here https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/6fa1ee3f51471ff12bbf83c1367749f3725678b7



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466577946


##########
extensions/kafka/tests/features/consumekafka.feature:
##########


Review Comment:
   👍 [rename consue_kafka.feature to consumekafka.feature](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/60da88eb3af202740773be49983e06be86c0c300)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466573159


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""

Review Comment:
   good idea, [shlex](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/2535836773fd283f881e1775faea20e3c6681310)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#issuecomment-3385077374

   I've also separated the verify package job into 2 workflow so we dont need to rebuilt the package to run the tests.
   
   https://github.com/martinzink/nifi-minifi-cpp/actions/runs/18370101534 🟢 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2514737387


##########
behave_framework/src/minifi_test_framework/containers/file.py:
##########
@@ -0,0 +1,23 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+class File:
+    def __init__(self, path, host_filename, content):
+        self.path = path
+        self.host_filename = host_filename

Review Comment:
   In my experience this seems to be unnecessary, we could just have a single member named `path` that includes the whole file path. That would be align with the `HostFile`'s definition where only a single path is defined for both the container path and the host path.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466588589


##########
extensions/aws/tests/features/s3.feature:
##########
@@ -19,89 +19,74 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
   As a user of MiNiFi
   I need to have PutS3Object and DeleteS3Object processors
 
-  Background:
-    Given the content of "/tmp/output" is monitored
-
   Scenario: A MiNiFi instance transfers encoded data to s3
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
-    And a PutS3Object processor set up to communicate with an s3 server
-    And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And the "success" relationship of the GetFile processor is connected to the PutS3Object
-    And the "success" relationship of the PutS3Object processor is connected to the PutFile
-
-    And a s3 server is set up in correspondence with the PutS3Object
-
-    When both instances start up
-
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
-    And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
-    And the Minifi logs contain the following message: "in a single upload" in less than 10 seconds
-
-  Scenario: A MiNiFi instance transfers encoded data to s3 in FIPS mode
-    Given OpenSSL FIPS mode is enabled in MiNiFi
-    And a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
+    And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object
+    And PutFile's success relationship is auto-terminated
 
-    And a s3 server is set up in correspondence with the PutS3Object
+    And an s3 server is set up in correspondence with the PutS3Object
 
     When both instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 seconds
     And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
     And the Minifi logs contain the following message: "in a single upload" in less than 10 seconds
 
   Scenario: A MiNiFi instance transfers encoded data through a http proxy to s3
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
-    And these processor properties are set to match the http proxy:
-    | processor name  | property name  | property value                  |
-    | PutS3Object     | Proxy Host     | http-proxy-${feature_id}        |
-    | PutS3Object     | Proxy Port     | 3128                            |
-    | PutS3Object     | Proxy Username | admin                           |
-    | PutS3Object     | Proxy Password | test101                         |
+    And these processor properties are set
+      | processor name  | property name  | property value                  |
+      | PutS3Object     | Proxy Host     | http-proxy-s3-1                 |

Review Comment:
   good idea, https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/b4557124d87c6c7f035ead0fa2d1758585756213



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469230877


##########
behave_framework/src/minifi_test_framework/steps/core_steps.py:
##########
@@ -0,0 +1,55 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import humanfriendly
+import logging
+import random
+import string
+from behave import when, step
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+@when("both instances start up")
+@when("all instances start up")
+def step_impl(context: MinifiTestContext):
+    for container in context.containers:
+        assert container.deploy()
+    assert context.minifi_container.deploy() or context.minifi_container.get_logs()
+    logging.debug("All instances started up")
+
+
+@when("the MiNiFi instance starts up")
+def step_impl(context):
+    assert context.minifi_container.deploy()
+    logging.debug("All instances started up")

Review Comment:
   👍 https://github.com/apache/nifi-minifi-cpp/commit/d58939bf209e4ed1cf9428404b1b1843a970bf69#diff-87e7792003d4f9003905f53f9b09d3086ad0080104bb6bf18d3eeac2f7a3648eR40



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2503057555


##########
extensions/azure/tests/features/environment.py:
##########


Review Comment:
   good idea 👍 [added explanatory comments to environment.pys](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/1c9e4c846f612c52e324d46298a19061bfa84623)



##########
.github/workflows/create-release-artifacts.yml:
##########
@@ -0,0 +1,42 @@
+name: "Create Release Artifacts"
+
+on: [workflow_dispatch]
+env:
+  CMAKE_FLAGS: -DCMAKE_BUILD_TYPE=Release -DCI_BUILD=ON -DENABLE_ALL=ON -DMINIFI_FAIL_ON_WARNINGS=OFF -DDOCKER_BUILD_ONLY=ON

Review Comment:
   good idea, [DCI_BUILD=OFF in create-release-artifacts.yml](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/47827dceca23d3eafe26f349921d037b8629fcd5)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2503192792


##########
extensions/llamacpp/tests/features/steps/steps.py:
##########
@@ -0,0 +1,4 @@
+from minifi_test_framework.steps import checking_steps  # noqa: F401

Review Comment:
   This file is still missing the license header while all other similar files have one.
   ```suggestion
   # Licensed to the Apache Software Foundation (ASF) under one or more
   # contributor license agreements.  See the NOTICE file distributed with
   # this work for additional information regarding copyright ownership.
   # The ASF licenses this file to You under the Apache License, Version 2.0
   # (the "License"); you may not use this file except in compliance with
   # the License.  You may obtain a copy of the License at
   #
   #     http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   # See the License for the specific language governing permissions and
   # limitations under the License.
   
   from minifi_test_framework.steps import checking_steps  # noqa: F401
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2416221427


##########
extensions/standard-processors/tests/features/attributes_to_json.feature:
##########
@@ -15,17 +15,16 @@
 
 @CORE
 Feature: Writing attribute data using AttributesToJSON processor
-  Background:
-    Given the content of "/tmp/output" is monitored
 
   Scenario: Write selected attribute data to file
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with filename "test_file.log" and content "test_data" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file ("test_file.log") with the content "test_data"
     And a AttributesToJSON processor with the "Attributes List" property set to "filename,invalid"
     And the "Destination" property of the AttributesToJSON processor is set to "flowfile-content"
     And the "Null Value" property of the AttributesToJSON processor is set to "true"
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the AttributesToJSON
     And the "success" relationship of the AttributesToJSON processor is connected to the PutFile
+    And PutFile's success relationship is auto-terminated
     When the MiNiFi instance starts up
-    Then a flowfile with the JSON content "{"filename":"test_file.log","invalid":null}" is placed in the monitored directory in less than 10 seconds
+    Then there is a single file with "{"invalid":null,"filename":"test_file.log"}" content in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Inconsistent time unit abbreviation: should be 'seconds' instead of 'sec' to match other test files.
   ```suggestion
       Then there is a single file with "{"invalid":null,"filename":"test_file.log"}" content in the "/tmp/output" directory in less than 20 seconds
   ```



##########
extensions/standard-processors/tests/features/syslog_listener.feature:
##########
@@ -43,4 +40,4 @@ Feature: Minifi C++ can act as a syslog listener
     And the "success" relationship of the ListenSyslog processor is connected to the PutFile
 
     When both instances start up
-    Then at least one flowfile is placed in the monitored directory in less than 10 seconds
+    Then there are at least 1 files is in the "/tmp/output" directory in less than 10 seconds

Review Comment:
   Grammar error: 'files is' should be 'file is' or 'files are'.
   ```suggestion
       Then there is at least 1 file in the "/tmp/output" directory in less than 10 seconds
   ```



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -111,83 +96,88 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
 
     When both instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object bucket on the s3 server is empty
+    Then there is a single file with "test" content in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Multiple inconsistent time unit abbreviations: should be 'seconds' instead of 'sec' to match other test files.



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -19,89 +19,74 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
   As a user of MiNiFi
   I need to have PutS3Object and DeleteS3Object processors
 
-  Background:
-    Given the content of "/tmp/output" is monitored
-
   Scenario: A MiNiFi instance transfers encoded data to s3
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
-    And a PutS3Object processor set up to communicate with an s3 server
-    And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And the "success" relationship of the GetFile processor is connected to the PutS3Object
-    And the "success" relationship of the PutS3Object processor is connected to the PutFile
-
-    And a s3 server is set up in correspondence with the PutS3Object
-
-    When both instances start up
-
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
-    And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
-    And the Minifi logs contain the following message: "in a single upload" in less than 10 seconds
-
-  Scenario: A MiNiFi instance transfers encoded data to s3 in FIPS mode
-    Given OpenSSL FIPS mode is enabled in MiNiFi
-    And a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
+    And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object
+    And PutFile's success relationship is auto-terminated
 
-    And a s3 server is set up in correspondence with the PutS3Object
+    And an s3 server is set up in correspondence with the PutS3Object
 
     When both instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 sec
     And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
     And the Minifi logs contain the following message: "in a single upload" in less than 10 seconds
 
   Scenario: A MiNiFi instance transfers encoded data through a http proxy to s3
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
-    And these processor properties are set to match the http proxy:
-    | processor name  | property name  | property value                  |
-    | PutS3Object     | Proxy Host     | http-proxy-${feature_id}        |
-    | PutS3Object     | Proxy Port     | 3128                            |
-    | PutS3Object     | Proxy Username | admin                           |
-    | PutS3Object     | Proxy Password | test101                         |
+    And these processor properties are set
+      | processor name  | property name  | property value                  |
+      | PutS3Object     | Proxy Host     | http-proxy-s3-1                 |
+      | PutS3Object     | Proxy Port     | 3128                            |
+      | PutS3Object     | Proxy Username | admin                           |
+      | PutS3Object     | Proxy Password | test101                         |
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
+    And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object
+    And PutFile's success relationship is auto-terminated
 
     And a s3 server is set up in correspondence with the PutS3Object
     And the http proxy server is set up
     When all instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Multiple inconsistent time unit abbreviations: should be 'seconds' instead of 'sec' to match other test files.



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -111,83 +96,88 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
 
     When both instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object bucket on the s3 server is empty
+    Then there is a single file with "test" content in the "/tmp/output" directory in less than 20 sec
+    And the object bucket on the s3 server is empty in less than 10 seconds
 
   Scenario: Deletion of a s3 object through a proxy-server succeeds
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
     And a DeleteS3Object processor set up to communicate with the same s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And these processor properties are set to match the http proxy:
+    And these processor properties are set
       | processor name  | property name  | property value                  |
-      | DeleteS3Object  | Proxy Host     | http-proxy-${feature_id}        |
+      | DeleteS3Object  | Proxy Host     | http-proxy-s3-4                 |
       | DeleteS3Object  | Proxy Port     | 3128                            |
       | DeleteS3Object  | Proxy Username | admin                           |
       | DeleteS3Object  | Proxy Password | test101                         |
-    And the processors are connected up as described here:
+    And the processors are connected up as described here
       | source name    | relationship name | destination name |
       | GetFile        | success           | PutS3Object      |
+      | PutS3Object    | failure           | PutS3Object      |
       | PutS3Object    | success           | DeleteS3Object   |
       | DeleteS3Object | success           | PutFile          |
+      | PutFile        | success           | auto-terminated  |
 
     And a s3 server is set up in correspondence with the PutS3Object
     And the http proxy server is set up
 
     When all instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object bucket on the s3 server is empty
-    And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 sec
+    And the object bucket on the s3 server is empty in less than 10 seconds
+    And no errors were generated on the http-proxy regarding "http://s3-server-s3-4:9090/test_bucket/test_object_key"
 
   Scenario: A MiNiFi instance can download s3 bucket objects directly
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "test" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "test"
     And a PutS3Object processor set up to communicate with an s3 server
-    And the "success" relationship of the GetFile processor is connected to the PutS3Object
-
-    Given a GenerateFlowFile processor with the "File Size" property set to "1 kB" in a "secondary" flow
+    And a GenerateFlowFile processor with the "File Size" property set to "1 kB"
     And a FetchS3Object processor set up to communicate with the same s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And the processors are connected up as described here:
+    And the processors are connected up as described here
       | source name      | relationship name | destination name |
       | GenerateFlowFile | success           | FetchS3Object    |
       | FetchS3Object    | success           | PutFile          |
+      | PutFile          | success           | auto-terminated  |
+      | PutFile          | failure           | auto-terminated  |
+      | GetFile          | success           | PutS3Object      |
+      | PutS3Object      | success           | auto-terminated  |
+      | PutS3Object      | failure           | PutS3Object      |
 
     And a s3 server is set up in correspondence with the PutS3Object
 
     When all instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+    Then there is a single file with "test" content in the "/tmp/output" directory in less than 20 sec
 
   Scenario: A MiNiFi instance can download s3 bucket objects via a http-proxy
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "test" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "test"
     And a PutS3Object processor set up to communicate with an s3 server
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
 
-    Given a GenerateFlowFile processor with the "File Size" property set to "1 kB" in a "secondary" flow
+    Given a GenerateFlowFile processor with the "File Size" property set to "1 kB"
     And a FetchS3Object processor set up to communicate with the same s3 server
-    And these processor properties are set to match the http proxy:
+    And these processor properties are set
       | processor name | property name  | property value                  |
-      | FetchS3Object  | Proxy Host     | http-proxy-${feature_id}        |
+      | FetchS3Object  | Proxy Host     | http-proxy-s3-6                 |
       | FetchS3Object  | Proxy Port     | 3128                            |
       | FetchS3Object  | Proxy Username | admin                           |
       | FetchS3Object  | Proxy Password | test101                         |
     And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And the processors are connected up as described here:
+    And the processors are connected up as described here
       | source name      | relationship name | destination name |
       | GenerateFlowFile | success           | FetchS3Object    |
       | FetchS3Object    | success           | PutFile          |
 
     And a s3 server is set up in correspondence with the PutS3Object
-    And a http proxy server is set up accordingly
+    And the http proxy server is set up
 
     When all instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
+    Then there is a single file with "test" content in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Multiple inconsistent time unit abbreviations: should be 'seconds' instead of 'sec' to match other test files.



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -268,7 +260,7 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And the http proxy server is set up
     When all instances start up
 
-    Then 1 flowfile is placed in the monitored directory in 120 seconds
+    Then there is 1 file in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Multiple inconsistent time unit abbreviations: should be 'seconds' instead of 'sec' to match other test files.
   ```suggestion
       Then there is 1 file in the "/tmp/output" directory in less than 20 seconds
   ```



##########
extensions/sql/tests/features/sql.feature:
##########
@@ -29,13 +26,13 @@ Feature: Executing SQL operations from MiNiFi-C++
     And the "success" relationship of the GenerateFlowFile processor is connected to the UpdateAttribute
     And the "success" relationship of the UpdateAttribute processor is connected to the PutSQL
     And an ODBCService is setup up for PutSQL with the name "ODBCService"
-    And a PostgreSQL server is set up
+    And PutSQL's success relationship is auto-terminated
+    And a PostgresSQL server is set up

Review Comment:
   Inconsistent naming: 'PostgresSQL' should be 'PostgreSQL' (standard name for the database).



##########
extensions/sql/tests/features/sql.feature:
##########
@@ -61,9 +58,9 @@ Feature: Executing SQL operations from MiNiFi-C++
     And the "success" relationship of the UpdateAttribute processor is connected to the ExecuteSQL
     And the "success" relationship of the ExecuteSQL processor is connected to the PutFile
     And an ODBCService is setup up for ExecuteSQL with the name "ODBCService"
-    And a PostgreSQL server is set up
+    And a PostgresSQL server is set up

Review Comment:
   Inconsistent naming: 'PostgresSQL' should be 'PostgreSQL' (standard name for the database).



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -199,64 +189,66 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And the "Object Key" property of the PutS3Object processor is set to "${filename}"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
 
-    Given a ListS3 processor in the "secondary" flow
+    Given a ListS3 processor set up to communicate with the same s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the ListS3 processor is connected to the PutFile
 
     And a s3 server is set up in correspondence with the PutS3Object
 
     When all instances start up
 
-    Then 2 flowfiles are placed in the monitored directory in 120 seconds
+    Then there are 2 files in the "/tmp/output" directory in less than 20 sec
 
   Scenario: A MiNiFi instance can list an S3 bucket objects via a http-proxy
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "test" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "test"
     And a PutS3Object processor set up to communicate with an s3 server
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
 
-    Given a ListS3 processor in the "secondary" flow
-    And these processor properties are set to match the http proxy:
+    Given a ListS3 processor set up to communicate with the same s3 server
+    And these processor properties are set
       | processor name | property name  | property value                  |
-      | ListS3         | Proxy Host     | http-proxy-${feature_id}        |
+      | ListS3         | Proxy Host     | http-proxy-s3-8                 |
       | ListS3         | Proxy Port     | 3128                            |
       | ListS3         | Proxy Username | admin                           |
       | ListS3         | Proxy Password | test101                         |
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the ListS3 processor is connected to the PutFile
 
     And a s3 server is set up in correspondence with the PutS3Object
-    And a http proxy server is set up accordingly
+    And the http proxy server is set up
 
     When all instances start up
 
-    Then 1 flowfile is placed in the monitored directory in 120 seconds
-    And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket"
+    Then there is 1 file in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Multiple inconsistent time unit abbreviations: should be 'seconds' instead of 'sec' to match other test files.
   ```suggestion
       Then there is 1 file in the "/tmp/output" directory in less than 20 seconds
   ```



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -111,83 +96,88 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
 
     When both instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object bucket on the s3 server is empty
+    Then there is a single file with "test" content in the "/tmp/output" directory in less than 20 sec
+    And the object bucket on the s3 server is empty in less than 10 seconds
 
   Scenario: Deletion of a s3 object through a proxy-server succeeds
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
     And a DeleteS3Object processor set up to communicate with the same s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And these processor properties are set to match the http proxy:
+    And these processor properties are set
       | processor name  | property name  | property value                  |
-      | DeleteS3Object  | Proxy Host     | http-proxy-${feature_id}        |
+      | DeleteS3Object  | Proxy Host     | http-proxy-s3-4                 |
       | DeleteS3Object  | Proxy Port     | 3128                            |
       | DeleteS3Object  | Proxy Username | admin                           |
       | DeleteS3Object  | Proxy Password | test101                         |
-    And the processors are connected up as described here:
+    And the processors are connected up as described here
       | source name    | relationship name | destination name |
       | GetFile        | success           | PutS3Object      |
+      | PutS3Object    | failure           | PutS3Object      |
       | PutS3Object    | success           | DeleteS3Object   |
       | DeleteS3Object | success           | PutFile          |
+      | PutFile        | success           | auto-terminated  |
 
     And a s3 server is set up in correspondence with the PutS3Object
     And the http proxy server is set up
 
     When all instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object bucket on the s3 server is empty
-    And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Multiple inconsistent time unit abbreviations: should be 'seconds' instead of 'sec' to match other test files.



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -111,83 +96,88 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
 
     When both instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object bucket on the s3 server is empty
+    Then there is a single file with "test" content in the "/tmp/output" directory in less than 20 sec
+    And the object bucket on the s3 server is empty in less than 10 seconds
 
   Scenario: Deletion of a s3 object through a proxy-server succeeds
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
     And a DeleteS3Object processor set up to communicate with the same s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And these processor properties are set to match the http proxy:
+    And these processor properties are set
       | processor name  | property name  | property value                  |
-      | DeleteS3Object  | Proxy Host     | http-proxy-${feature_id}        |
+      | DeleteS3Object  | Proxy Host     | http-proxy-s3-4                 |
       | DeleteS3Object  | Proxy Port     | 3128                            |
       | DeleteS3Object  | Proxy Username | admin                           |
       | DeleteS3Object  | Proxy Password | test101                         |
-    And the processors are connected up as described here:
+    And the processors are connected up as described here
       | source name    | relationship name | destination name |
       | GetFile        | success           | PutS3Object      |
+      | PutS3Object    | failure           | PutS3Object      |
       | PutS3Object    | success           | DeleteS3Object   |
       | DeleteS3Object | success           | PutFile          |
+      | PutFile        | success           | auto-terminated  |
 
     And a s3 server is set up in correspondence with the PutS3Object
     And the http proxy server is set up
 
     When all instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object bucket on the s3 server is empty
-    And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 sec
+    And the object bucket on the s3 server is empty in less than 10 seconds
+    And no errors were generated on the http-proxy regarding "http://s3-server-s3-4:9090/test_bucket/test_object_key"
 
   Scenario: A MiNiFi instance can download s3 bucket objects directly
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "test" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "test"
     And a PutS3Object processor set up to communicate with an s3 server
-    And the "success" relationship of the GetFile processor is connected to the PutS3Object
-
-    Given a GenerateFlowFile processor with the "File Size" property set to "1 kB" in a "secondary" flow
+    And a GenerateFlowFile processor with the "File Size" property set to "1 kB"
     And a FetchS3Object processor set up to communicate with the same s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And the processors are connected up as described here:
+    And the processors are connected up as described here
       | source name      | relationship name | destination name |
       | GenerateFlowFile | success           | FetchS3Object    |
       | FetchS3Object    | success           | PutFile          |
+      | PutFile          | success           | auto-terminated  |
+      | PutFile          | failure           | auto-terminated  |
+      | GetFile          | success           | PutS3Object      |
+      | PutS3Object      | success           | auto-terminated  |
+      | PutS3Object      | failure           | PutS3Object      |
 
     And a s3 server is set up in correspondence with the PutS3Object
 
     When all instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+    Then there is a single file with "test" content in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Multiple inconsistent time unit abbreviations: should be 'seconds' instead of 'sec' to match other test files.



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -19,89 +19,74 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
   As a user of MiNiFi
   I need to have PutS3Object and DeleteS3Object processors
 
-  Background:
-    Given the content of "/tmp/output" is monitored
-
   Scenario: A MiNiFi instance transfers encoded data to s3
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
-    And a PutS3Object processor set up to communicate with an s3 server
-    And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And the "success" relationship of the GetFile processor is connected to the PutS3Object
-    And the "success" relationship of the PutS3Object processor is connected to the PutFile
-
-    And a s3 server is set up in correspondence with the PutS3Object
-
-    When both instances start up
-
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
-    And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
-    And the Minifi logs contain the following message: "in a single upload" in less than 10 seconds
-
-  Scenario: A MiNiFi instance transfers encoded data to s3 in FIPS mode
-    Given OpenSSL FIPS mode is enabled in MiNiFi
-    And a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
+    And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object
+    And PutFile's success relationship is auto-terminated
 
-    And a s3 server is set up in correspondence with the PutS3Object
+    And an s3 server is set up in correspondence with the PutS3Object
 
     When both instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Inconsistent time unit abbreviation: should be 'seconds' instead of 'sec' to match other test files.
   ```suggestion
       Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 seconds
   ```



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -199,64 +189,66 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And the "Object Key" property of the PutS3Object processor is set to "${filename}"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
 
-    Given a ListS3 processor in the "secondary" flow
+    Given a ListS3 processor set up to communicate with the same s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the ListS3 processor is connected to the PutFile
 
     And a s3 server is set up in correspondence with the PutS3Object
 
     When all instances start up
 
-    Then 2 flowfiles are placed in the monitored directory in 120 seconds
+    Then there are 2 files in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Multiple inconsistent time unit abbreviations: should be 'seconds' instead of 'sec' to match other test files.



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -19,89 +19,74 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
   As a user of MiNiFi
   I need to have PutS3Object and DeleteS3Object processors
 
-  Background:
-    Given the content of "/tmp/output" is monitored
-
   Scenario: A MiNiFi instance transfers encoded data to s3
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
-    And a PutS3Object processor set up to communicate with an s3 server
-    And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And the "success" relationship of the GetFile processor is connected to the PutS3Object
-    And the "success" relationship of the PutS3Object processor is connected to the PutFile
-
-    And a s3 server is set up in correspondence with the PutS3Object
-
-    When both instances start up
-
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
-    And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
-    And the Minifi logs contain the following message: "in a single upload" in less than 10 seconds
-
-  Scenario: A MiNiFi instance transfers encoded data to s3 in FIPS mode
-    Given OpenSSL FIPS mode is enabled in MiNiFi
-    And a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
+    And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object
+    And PutFile's success relationship is auto-terminated
 
-    And a s3 server is set up in correspondence with the PutS3Object
+    And an s3 server is set up in correspondence with the PutS3Object
 
     When both instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 sec
     And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
     And the Minifi logs contain the following message: "in a single upload" in less than 10 seconds
 
   Scenario: A MiNiFi instance transfers encoded data through a http proxy to s3
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
-    And these processor properties are set to match the http proxy:
-    | processor name  | property name  | property value                  |
-    | PutS3Object     | Proxy Host     | http-proxy-${feature_id}        |
-    | PutS3Object     | Proxy Port     | 3128                            |
-    | PutS3Object     | Proxy Username | admin                           |
-    | PutS3Object     | Proxy Password | test101                         |
+    And these processor properties are set
+      | processor name  | property name  | property value                  |
+      | PutS3Object     | Proxy Host     | http-proxy-s3-1                 |
+      | PutS3Object     | Proxy Port     | 3128                            |
+      | PutS3Object     | Proxy Username | admin                           |
+      | PutS3Object     | Proxy Password | test101                         |
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
+    And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object
+    And PutFile's success relationship is auto-terminated
 
     And a s3 server is set up in correspondence with the PutS3Object
     And the http proxy server is set up
     When all instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 sec
     And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
-    And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket/test_object_key"
+    And no errors were generated on the http-proxy regarding "http://s3-server-s3-1:9090/test_bucket/test_object_key"
 
   Scenario: A MiNiFi instance can remove s3 bucket objects
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
     And a DeleteS3Object processor set up to communicate with the same s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And the processors are connected up as described here:
+    And the processors are connected up as described here
       | source name    | relationship name | destination name |
       | GetFile        | success           | PutS3Object      |
       | PutS3Object    | success           | DeleteS3Object   |
+      | PutS3Object    | failure           | PutS3Object      |
       | DeleteS3Object | success           | PutFile          |
+      | PutFile        | success           | auto-terminated  |
 
     And a s3 server is set up in correspondence with the PutS3Object
 
     When both instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object bucket on the s3 server is empty
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Multiple inconsistent time unit abbreviations: should be 'seconds' instead of 'sec' to match other test files.



##########
extensions/sql/tests/features/sql.feature:
##########
@@ -73,9 +70,10 @@ Feature: Executing SQL operations from MiNiFi-C++
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the QueryDatabaseTable processor is connected to the PutFile
     And an ODBCService is setup up for QueryDatabaseTable with the name "ODBCService"
-    And a PostgreSQL server is set up
+    And a PostgresSQL server is set up

Review Comment:
   Inconsistent naming: 'PostgresSQL' should be 'PostgreSQL' (standard name for the database).



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -199,64 +189,66 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
     And the "Object Key" property of the PutS3Object processor is set to "${filename}"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
 
-    Given a ListS3 processor in the "secondary" flow
+    Given a ListS3 processor set up to communicate with the same s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the ListS3 processor is connected to the PutFile
 
     And a s3 server is set up in correspondence with the PutS3Object
 
     When all instances start up
 
-    Then 2 flowfiles are placed in the monitored directory in 120 seconds
+    Then there are 2 files in the "/tmp/output" directory in less than 20 sec
 
   Scenario: A MiNiFi instance can list an S3 bucket objects via a http-proxy
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "test" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "test"
     And a PutS3Object processor set up to communicate with an s3 server
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
 
-    Given a ListS3 processor in the "secondary" flow
-    And these processor properties are set to match the http proxy:
+    Given a ListS3 processor set up to communicate with the same s3 server
+    And these processor properties are set
       | processor name | property name  | property value                  |
-      | ListS3         | Proxy Host     | http-proxy-${feature_id}        |
+      | ListS3         | Proxy Host     | http-proxy-s3-8                 |
       | ListS3         | Proxy Port     | 3128                            |
       | ListS3         | Proxy Username | admin                           |
       | ListS3         | Proxy Password | test101                         |
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the ListS3 processor is connected to the PutFile
 
     And a s3 server is set up in correspondence with the PutS3Object
-    And a http proxy server is set up accordingly
+    And the http proxy server is set up
 
     When all instances start up
 
-    Then 1 flowfile is placed in the monitored directory in 120 seconds
-    And no errors were generated on the http-proxy regarding "http://s3-server-${feature_id}:9090/test_bucket"
+    Then there is 1 file in the "/tmp/output" directory in less than 20 sec
+    And no errors were generated on the http-proxy regarding "http://s3-server-s3-8:9090/test_bucket"
 
   Scenario: A MiNiFi instance transfers data in multiple parts to s3
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file of size 16MB is present in "/tmp/input"
+    And there is a 6MB file at the "/tmp/input" directory and we keep track of the hash of that
     And a PutS3Object processor set up to communicate with an s3 server
     And the "Multipart Threshold" property of the PutS3Object processor is set to "5 MB"
     And the "Multipart Part Size" property of the PutS3Object processor is set to "5 MB"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
     And a s3 server is set up in correspondence with the PutS3Object
+
     When all instances start up
-    Then 1 flowfile is placed in the monitored directory in 120 seconds
+
+    Then there is 1 file in the "/tmp/output" directory in less than 20 sec

Review Comment:
   Multiple inconsistent time unit abbreviations: should be 'seconds' instead of 'sec' to match other test files.
   ```suggestion
       Then there is 1 file in the "/tmp/output" directory in less than 20 seconds
   ```



##########
extensions/sql/tests/features/sql.feature:
##########
@@ -45,9 +42,9 @@ Feature: Executing SQL operations from MiNiFi-C++
     And the "success" relationship of the UpdateAttribute processor is connected to the ExecuteSQL
     And the "success" relationship of the ExecuteSQL processor is connected to the PutFile
     And an ODBCService is setup up for ExecuteSQL with the name "ODBCService"
-    And a PostgreSQL server is set up
+    And a PostgresSQL server is set up

Review Comment:
   Inconsistent naming: 'PostgresSQL' should be 'PostgreSQL' (standard name for the database).
   ```suggestion
       And a PostgreSQL server is set up
   ```



##########
extensions/sql/tests/features/sql.feature:
##########
@@ -85,6 +83,6 @@ Feature: Executing SQL operations from MiNiFi-C++
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the QueryDatabaseTable processor is connected to the PutFile
     And an ODBCService is setup up for QueryDatabaseTable with the name "ODBCService"
-    And a PostgreSQL server is set up
+    And a PostgresSQL server is set up

Review Comment:
   Inconsistent naming: 'PostgresSQL' should be 'PostgreSQL' (standard name for the database).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469162252


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:

Review Comment:
   nope, good idea, https://github.com/apache/nifi-minifi-cpp/commit/d58939bf209e4ed1cf9428404b1b1843a970bf69#diff-b9acc60a55b3f931c457c238886505df8e3401fcb942eae49005cac3f96c397bL51



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466579649


##########
docker/RunBehaveTests.sh:
##########
@@ -0,0 +1,200 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+die()
+{
+  local _ret="${2:-1}"
+  test "${_PRINT_HELP:-no}" = yes && print_help >&2
+  echo "$1" >&2
+  exit "${_ret}"
+}
+
+_positionals=()
+_arg_image_tag_prefix=
+_arg_tags_to_exclude=
+_arg_parallel_processes=3
+
+print_help()
+{
+  printf '%s\n' "Runs the provided behave tests in a containerized environment"
+  printf 'Usage: %s [--image-tag-prefix <arg>] [-h|--help] <minifi_version> <tags_to_run> ...\n' "$0"
+  printf '\t%s\n' "<minifi_version>: the version of minifi"
+  printf '\t%s\n' "<tags_to_run>: include these tags"
+  printf '\t%s\n' "--tags_to_exclude: optional tags that should be skipped (no default)"
+  printf '\t%s\n' "--image-tag-prefix: optional prefix to the docker tag (no default)"
+  printf '\t%s\n' "--parallel_processes: optional argument that specifies the number of parallel processes that can be executed simultaneously. (default: 3)"
+  printf '\t%s\n' "--fips: enables FIPS mode by default"
+  printf '\t%s\n' "-h, --help: Prints help"
+}
+
+
+parse_commandline()
+{
+  _positionals_count=0
+  _arg_fips=false  # Default to false
+  while test $# -gt 0
+  do
+    _key="$1"
+    case "$_key" in
+      --image-tag-prefix)
+        test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
+        _arg_image_tag_prefix="$2"
+        shift
+        ;;
+      --image-tag-prefix=*)
+        _arg_image_tag_prefix="${_key##--image-tag-prefix=}"
+        ;;
+      --tags_to_exclude)
+        test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
+        _arg_tags_to_exclude="$2"
+        shift
+        ;;
+      --tags_to_exclude=*)
+        _arg_tags_to_exclude="${_key##--tags_to_exclude=}"
+        ;;
+      --parallel_processes)
+        test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
+        _arg_parallel_processes="$2"
+        shift
+        ;;
+      --parallel_processes=*)
+        _arg_parallel_processes="${_key##--parallel_processes=}"
+        ;;
+      --fips)
+        _arg_fips=true  # Set boolean flag to true when argument is present
+        ;;
+      -h|--help)
+        print_help
+        exit 0
+        ;;
+      -h*)
+        print_help
+        exit 0
+        ;;
+      *)
+        _last_positional="$1"
+        _positionals+=("$_last_positional")
+        _positionals_count=$((_positionals_count + 1))
+        ;;
+    esac
+    shift
+  done
+}
+
+
+
+handle_passed_args_count()
+{
+  local _required_args_string="'minifi_version' and 'tags_to_run'"
+  test "${_positionals_count}" -ge 2 || _PRINT_HELP=yes die "FATAL ERROR: Not enough positional arguments - we require at least 2 (namely: $_required_args_string), but got only ${_positionals_count}." 1
+}
+
+
+assign_positional_args()
+{
+  local _positional_name _shift_for=$1
+  _positional_names="_arg_minifi_version _arg_tags_to_run "
+
+  shift "$_shift_for"
+  for _positional_name in ${_positional_names}
+  do
+    test $# -gt 0 || break
+    eval "$_positional_name=\${1}" || die "Error during argument parsing." 1
+    shift
+  done
+}
+
+parse_commandline "$@"
+handle_passed_args_count
+assign_positional_args 1 "${_positionals[@]}"
+
+docker_dir="$( cd "${0%/*}" && pwd )"
+
+# shellcheck disable=SC2154
+export MINIFI_VERSION=${_arg_minifi_version}
+if test -z "$_arg_image_tag_prefix"
+then
+  export MINIFI_TAG_PREFIX=""
+else
+  export MINIFI_TAG_PREFIX=${_arg_image_tag_prefix}-
+fi
+
+  if [ "$_arg_fips" = true ]; then
+    export MINIFI_FIPS="true"
+  else
+    export MINIFI_FIPS="false"
+  fi
+
+# Create virtual environment for testing
+if [[ ! -d ./behave_venv ]]; then
+  echo "Creating virtual environment in ./behave_venv" 1>&2
+  virtualenv --python=python3 ./behave_venv
+fi
+
+echo "Activating virtual environment..." 1>&2
+# shellcheck disable=SC1091
+. ./behave_venv/bin/activate
+pip install --trusted-host pypi.python.org --upgrade pip setuptools
+
+# Install test dependencies
+echo "Installing test dependencies..." 1>&2
+
+# hint include/library paths if homewbrew is in use
+if brew list 2> /dev/null | grep openssl > /dev/null 2>&1; then
+  echo "Using homebrew paths for openssl" 1>&2
+  LDFLAGS="-L$(brew --prefix openssl@1.1)/lib"
+  export LDFLAGS
+  CFLAGS="-I$(brew --prefix openssl@1.1)/include"
+  export CFLAGS
+  SWIG_FEATURES="-cpperraswarn -includeall -I$(brew --prefix openssl@1.1)/include"
+  export SWIG_FEATURES
+fi
+
+if ! command swig -version &> /dev/null; then
+  echo "Swig could not be found on your system (dependency of m2crypto python library). Please install swig to continue."
+  exit 1
+fi
+
+pip install -e "${docker_dir}/../behave_framework"
+
+export TMPDIR="/tmp/behavex_ci_${RANDOM}"
+mkdir -p "$TMPDIR"
+
+export TEMP="$TMPDIR/temp"
+export LOGS="$TMPDIR/logs"
+
+BEHAVE_OPTS=(--show-progress-bar --logging-level DEBUG --parallel-processes "${_arg_parallel_processes}" --parallel-scheme feature -o "${PWD}/behavex_output_2" -t "${_arg_tags_to_run}")

Review Comment:
   [RunBehaveTests.sh with INFO](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/40b5649ba8c639f6ab80d043bdf34aa621bf27e7)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466572045


##########
behave_framework/src/minifi_test_framework/containers/minifi_container.py:
##########
@@ -0,0 +1,85 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from docker.models.networks import Network
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.minifi.flow_definition import FlowDefinition
+from typing import Dict
+
+from .container import Container
+
+
+class MinifiContainer(Container):
+    def __init__(self, image_name: str, scenario_id: str, network: Network):
+        super().__init__(image_name, f"minifi-{scenario_id}", network)
+        self.flow_config_str: str = ""
+        self.flow_definition = FlowDefinition()
+        self.properties: Dict[str, str] = {}
+        self.log_properties: Dict[str, str] = {}
+
+        self.is_fhs = 'MINIFI_INSTALLATION_TYPE=FHS' in str(self.client.images.get(image_name).history())
+
+        self._fill_default_properties()
+        self._fill_default_log_properties()
+
+    def deploy(self) -> bool:
+        if self.is_fhs:
+            self.files.append(File("/etc/nifi-minifi-cpp", "config.yml", self.flow_definition.to_yaml()))
+            self.files.append(File("/etc/nifi-minifi-cpp", "minifi.properties", self._get_properties_file_content()))
+            self.files.append(
+                File("/etc/nifi-minifi-cpp", "minifi-log.properties", self._get_log_properties_file_content()))
+        else:
+            self.files.append(File("/opt/minifi/minifi-current/conf", "config.yml", self.flow_definition.to_yaml()))
+            self.files.append(
+                File("/opt/minifi/minifi-current/conf", "minifi.properties", self._get_properties_file_content()))
+            self.files.append(File("/opt/minifi/minifi-current/conf", "minifi-log.properties",
+                                   self._get_log_properties_file_content()))

Review Comment:
   that file should be there on the container and we probably dont need to change it during testing 



##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""

Review Comment:
   good idea 👍 https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/621be5a3eb5f4905a8b3f832d8dd5411de022267#diff-b9acc60a55b3f931c457c238886505df8e3401fcb942eae49005cac3f96c397bR110



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469166388


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")

Review Comment:
   🤷 fixed in https://github.com/apache/nifi-minifi-cpp/commit/d58939bf209e4ed1cf9428404b1b1843a970bf69#diff-b9acc60a55b3f931c457c238886505df8e3401fcb942eae49005cac3f96c397bL84



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2514811529


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,289 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+import json
+import logging
+import os
+import shlex
+import tempfile
+
+import docker
+from docker.models.networks import Network
+
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"

Review Comment:
   This also seems to cause some issues in some containers when running them as root. I think we should default to `None` here to use the default image user instead.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2425611713


##########
docker/RunBehaveTests.sh:
##########
@@ -0,0 +1,200 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,

Review Comment:
   Inconsistent escaping: should be 'the License. You may obtain a copy of the License at' without escaped quotes
   ```suggestion
   # (the "License"); you may not use this file except in compliance with
   # the License.  You may obtain a copy of the License at
   #
   #     http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   ```



##########
docker/RunBehaveTests.sh:
##########
@@ -0,0 +1,200 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,

Review Comment:
   Inconsistent escaping: should be 'AS IS' without escaped quotes
   ```suggestion
   # (the "License"); you may not use this file except in compliance with
   # the License.  You may obtain a copy of the License at
   #
   #     http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing, software
   # distributed under the License is distributed on an "AS IS" BASIS,
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2452074198


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name

Review Comment:
   nitpick, but could use os.path.join for these path definitions instead



##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""
+
+        exit_code, output = self.exec_run(command)
+
+        return exit_code == 0
+
+    def directory_contains_file_with_regex(self, directory_path: str, regex_str: str) -> bool:
+        if not self.container:
+            return False
+
+        safe_dir_path = shlex.quote(directory_path)
+        safe_regex_str = shlex.quote(regex_str)
+
+        command = (
+            f"find {safe_dir_path} -maxdepth 1 -type f -print0 | "
+            f"xargs -0 -r grep -l -E -- {safe_regex_str}"
+        )
+
+        exit_code, output = self.exec_run(f"sh -c \"{command}\"")
+
+        return exit_code == 0
+
+    def path_with_content_exists(self, path: str, content: str) -> bool:
+        count_command = f"sh -c 'cat {path} | grep \"^{content}$\" | wc -l'"
+        exit_code, output = self.exec_run(count_command)
+        if exit_code != 0:
+            logging.error(f"Error running command '{count_command}': {output}")
+            return False
+
+        try:
+            file_count = int(output.strip())
+        except (ValueError, IndexError):
+            logging.error(f"Error parsing output '{output}' from command '{count_command}'")
+            return False
+        return file_count == 1
+
+    def directory_has_single_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        count_command = f"sh -c 'find {directory_path} -maxdepth 1 -type f | wc -l'"
+        exit_code, output = self.exec_run(count_command)
+
+        if exit_code != 0:
+            logging.error(f"Error running command '{count_command}': {output}")
+            return False
+
+        try:
+            file_count = int(output.strip())
+        except (ValueError, IndexError):
+            logging.error(f"Error parsing output '{output}' from command '{count_command}'")
+            return False
+
+        if file_count != 1:
+            logging.error(f"{directory_path} has too many or too few ({file_count}) files")
+            return False
+
+        content_command = f"sh -c 'cat {directory_path}/*'"
+        exit_code, output = self.exec_run(content_command)
+
+        if exit_code != 0:
+            logging.error(f"Error running command '{content_command}': {output}")
+            return False
+
+        actual_content = output.strip()
+        logging.debug(f"Comparing: '{actual_content}' vs {expected_content}")
+        return actual_content == expected_content.strip()
+
+    def get_logs(self) -> str:
+        logging.debug("Getting logs from container '%s'", self.container_name)
+        if not self.container:
+            return ""
+        logs_as_bytes = self.container.logs()
+        return logs_as_bytes.decode('utf-8')
+
+    @property
+    def exited(self) -> bool:
+        if not self.container:
+            return False
+        try:
+            self.container.reload()
+            return self.container.status == 'exited'
+        except docker.errors.NotFound:
+            self.container = None
+            return False
+        except Exception:
+            return False
+
+    def _get_stats(self) -> Dict[str, Any]:
+        self.container.reload()
+        return self.container.stats(stream=False)
+
+    def get_number_of_files(self, directory_path: str) -> int:
+        if not self.container:
+            return False
+
+        count_command = f"sh -c 'find {directory_path} -maxdepth 1 -type f | wc -l'"
+        exit_code, output = self.exec_run(count_command)
+
+        if exit_code != 0:
+            logging.error(f"Error running command '{count_command}': {output}")
+            return False

Review Comment:
   Instead of `return False` it should return -1



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2495290732


##########
extensions/kafka/tests/features/steps/steps.py:
##########
@@ -0,0 +1,85 @@
+import binascii
+
+from behave import step, when, given
+
+
+from minifi_test_framework.steps import checking_steps        # noqa: F401
+from minifi_test_framework.steps import configuration_steps   # noqa: F401
+from minifi_test_framework.steps import core_steps            # noqa: F401
+from minifi_test_framework.steps import flow_building_steps   # noqa: F401
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+from minifi_test_framework.minifi.processor import Processor
+from kafka_server_container import KafkaServer
+
+
+@step("a {processor_name} processor set up to communicate with an Azure blob storage")
+def step_impl(context: MinifiTestContext, processor_name: str):
+    processor = Processor(processor_name, processor_name)
+    hostname = f"http://azure-storage-server-{context.scenario_id}"
+    processor.add_property('Container Name', 'test-container')
+    processor.add_property('Connection String', 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint={hostname}:10000/devstoreaccount1;QueueEndpoint={hostname}:10001/devstoreaccount1;'.format(hostname=hostname))
+    processor.add_property('Blob', 'test-blob')
+    processor.add_property('Create Container', 'true')
+    context.minifi_container.flow_definition.add_processor(processor)

Review Comment:
   Yeah, I've been tricked by that too, I thought only a few scenarios are missing from the kafka part, but then I realized just the skeleton of it is ready :sweat_smile: I started working on that one though, I'll create a separate PR of it when ready



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2523808803


##########
extensions/azure/tests/features/steps/azure_server_container.py:
##########
@@ -0,0 +1,103 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+
+from docker.errors import ContainerError
+from minifi_test_framework.containers.container import Container
+from minifi_test_framework.core.helpers import run_cmd_in_docker_image
+from minifi_test_framework.core.helpers import wait_for_condition
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+class AzureServerContainer(Container):
+    def __init__(self, test_context: MinifiTestContext):
+        super().__init__("mcr.microsoft.com/azure-storage/azurite:3.35.0",
+                         f"azure-storage-server-{test_context.scenario_id}",
+                         test_context.network)
+        azure_storage_hostname = f"azure-storage-server-{test_context.scenario_id}"
+        self.azure_connection_string = "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;" \
+                                       f"BlobEndpoint=http://{azure_storage_hostname}:10000/devstoreaccount1;QueueEndpoint=http://{azure_storage_hostname}:10001/devstoreaccount1;"
+
+    def deploy(self):
+        super().deploy()
+        finished_str = "Azurite Queue service is successfully listening at"
+        return wait_for_condition(condition=lambda: finished_str in self.get_logs(),
+                                  timeout_seconds=15,
+                                  bail_condition=lambda: self.exited,
+                                  context=None)
+
+    def check_azure_storage_server_data(self, test_data):
+        (code, output) = self.exec_run(["find", "/data/__blobstorage__", "-type", "f"])
+        if code != 0:
+            return False
+        data_file = output.strip()
+        (code, file_data) = self.exec_run(["cat", data_file])
+        return code == 0 and test_data in file_data
+
+    def add_test_blob(self, blob_name, content="test_data", with_snapshot=False) -> bool:
+        cmd_create = (f'az storage container create --name "test-container" '
+                      f'--connection-string "{self.azure_connection_string}"')
+        try:
+            run_cmd_in_docker_image("mcr.microsoft.com/azure-cli", cmd_create, self.network.name)
+        except ContainerError as e:
+            logging.error(e)
+            return False
+
+        cmd_upload = (f'az storage blob upload --container-name "test-container" '
+                      f'--name "{blob_name}" --data "{content}" '
+                      f'--connection-string "{self.azure_connection_string}"')
+        try:
+            run_cmd_in_docker_image("mcr.microsoft.com/azure-cli", cmd_upload, self.network.name)
+        except ContainerError as e:
+            logging.error(e)
+            return False
+
+        if with_snapshot:
+            cmd_snapshot = (f'az storage blob snapshot --container-name "test-container" '
+                            f'--name "{blob_name}" '
+                            f'--connection-string "{self.azure_connection_string}"')
+            try:
+                run_cmd_in_docker_image("mcr.microsoft.com/azure-cli", cmd_snapshot, self.network.name)
+            except ContainerError as e:
+                logging.error(e)
+                return False
+
+        return True
+
+    def __get_blob_and_snapshot_count(self):
+        cmd = (f'az storage blob list --container-name "test-container" '
+               f'--include deleted --query "length(@)" --output tsv '
+               f'--connection-string "{self.azure_connection_string}"')

Review Comment:
   for some reason calling this via list and not str the result is None not the stdout, (it still throws when returning with non zero err code) so I've left this as a string 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2465090952


##########
behave_framework/src/minifi_test_framework/core/helpers.py:
##########
@@ -0,0 +1,58 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from __future__ import annotations
+
+import logging
+import time
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+from typing import Callable
+
+
+def log_due_to_failure(context: MinifiTestContext | None):
+    if context is not None:
+        for container in context.containers:
+            container.log_app_output()
+        context.minifi_container.log_app_output()
+
+
+def wait_for_condition(condition: Callable[[], bool], timeout_seconds: float, bail_condition: Callable[[], bool],
+                       context: MinifiTestContext | None) -> bool:
+    if bail_condition():
+        logging.warning("Bail condition evaluated to 'True', aborting wait.")
+        log_due_to_failure(context)
+        return False
+    start_time = time.monotonic()
+    try:
+        while time.monotonic() - start_time < timeout_seconds:
+            if condition():
+                return True
+            if bail_condition():
+                logging.warning("Bail condition evaluated to 'True', aborting wait.")
+                log_due_to_failure(context)
+                return False
+            remaining_time = timeout_seconds - (time.monotonic() - start_time)
+            sleep_time = min(timeout_seconds / 10, remaining_time)

Review Comment:
   I think it would be better to have a fixed sleep time here like 1 second, if we have a way too big timeout then it will be a too rare check, but if we have less than 10 seconds it will be too frequent in my opinion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466580757


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name

Review Comment:
   👍 [use os.path.join](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/d1181583e9c06124316c1aa2d8ce81a25b213788)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466585354


##########
.github/workflows/ci.yml:
##########
@@ -535,6 +535,64 @@ jobs:
         with:
           name: behavex_output
           path: build/behavex_output
+  modular_docker_tests:
+    name: "Modular Docker integration tests (x86_64)"
+    needs: docker_build
+    runs-on: ubuntu-24.04
+    timeout-minutes: 180
+    steps:
+      - id: checkout
+        uses: actions/checkout@v4
+      - id: run_cmake
+        name: Run CMake
+        run: |
+          mkdir build
+          cd build
+          cmake ${DOCKER_CMAKE_FLAGS} ..
+      - name: Download artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: minifi_docker
+          path: build
+      - name: Load Docker image
+        run: |
+          docker load --input ./build/minifi_docker.tar
+      - id: install_deps
+        name: Install dependencies for Docker Verify
+        run: |
+          sudo apt update
+          sudo apt install -y python3-virtualenv
+      - id: free_disk_space
+        run: |
+          # We can gain additional disk space on the Ubuntu runners thanks to these suggestions:
+          # https://github.com/actions/runner-images/issues/2840#issuecomment-790492173
+          # https://github.com/actions/runner-images/issues/2606#issuecomment-772683150
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /usr/local/lib/android
+          sudo rm -rf /opt/ghc
+          sudo rm -rf "/usr/local/share/boost"
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY"
+      - id: test
+        name: Docker Verify
+        working-directory: ./build
+        run: make docker-verify-modular

Review Comment:
   good idea, https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/d58939bf209e4ed1cf9428404b1b1843a970bf69#diff-d32084965dae4bf43c2eaf159a05404724fca492ab691c00a373205e441aab15R4



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2498547347


##########
extensions/standard-processors/tests/features/syslog_listener.feature:
##########
@@ -30,9 +27,10 @@ Feature: Minifi C++ can act as a syslog listener
     And the "Replacement Value" property of the ReplaceText processor is set to "${syslog.timestamp}	${syslog.priority}	${reverseDnsLookup(${syslog.sender}, 100)}	${syslog.msg}"
     And the "success" relationship of the ListenSyslog processor is connected to the ReplaceText
     And the "success" relationship of the ReplaceText processor is connected to the PutFile
+    And PutFile's success relationship is auto-terminated
 
     When both instances start up
-    Then at least one flowfile's content match the following regex: "^((?:(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}(?:\.\d+)?))(Z|[\+-]\d{2}:\d{2})?)\t13\t.*\tsample_log$" in less than 10 seconds
+    Then at least one file in "/tmp/output" content match the following regex: "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.*13.*sample_log$" in less than 1000 seconds

Review Comment:
   good catch 👍 [revert 1000s to 10s in syslog_listener.feature](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/6874bc92777fe04c928a74db9e44434e7a1ab624)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#issuecomment-3497920849

   removed the "low-impact" label because while its description treats all test-only changes as "low-impact", and it's unlikely to introduce bugs, a new test framework isn't really "low impact" in the original sense of the phrase. If you disagree, feel free to add back.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2468792518


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""

Review Comment:
   I like these changes, but I would still add another test for when the directory exists but it is empty, since `grep` prints an error message in this case, too:
   ```
   $ mkdir /tmp/foo
   $ grep -l -F 'bar' /tmp/foo/*
   /usr/bin/grep: /tmp/foo/*: No such file or directory
   $ touch /tmp/foo/qux
   $ grep -l -F 'bar' /tmp/foo/*
   $ 
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469163928


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass

Review Comment:
   this is the happy path, since normally we should clean up after ourselves so its expected that we dont find the container



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469239046


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""

Review Comment:
   thats a good point, how about this? [check dir empty](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/270bc944f5169c9a141134a5c5f55908d15979a8)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2444917902


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")

Review Comment:
   Why do we print here instead of using logging?



##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:

Review Comment:
   Do we need this check if we just iterate through all of the files? If those lists are empty we don't do anything anyway



##########
extensions/standard-processors/tests/features/attributes_to_json.feature:
##########
@@ -15,17 +15,16 @@
 
 @CORE
 Feature: Writing attribute data using AttributesToJSON processor
-  Background:
-    Given the content of "/tmp/output" is monitored
 
   Scenario: Write selected attribute data to file
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with filename "test_file.log" and content "test_data" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file ("test_file.log") with the content "test_data"
     And a AttributesToJSON processor with the "Attributes List" property set to "filename,invalid"
     And the "Destination" property of the AttributesToJSON processor is set to "flowfile-content"
     And the "Null Value" property of the AttributesToJSON processor is set to "true"
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the AttributesToJSON
     And the "success" relationship of the AttributesToJSON processor is connected to the PutFile
+    And PutFile's success relationship is auto-terminated
     When the MiNiFi instance starts up
-    Then a flowfile with the JSON content "{"filename":"test_file.log","invalid":null}" is placed in the monitored directory in less than 10 seconds
+    Then there is a single file with "{"invalid":null,"filename":"test_file.log"}" content in the "/tmp/output" directory in less than 20 seconds

Review Comment:
   I would keep the JSON content check instead of raw file content check as the JSON field order is not guaranteed.



##########
.github/workflows/ci.yml:
##########
@@ -535,6 +535,64 @@ jobs:
         with:
           name: behavex_output
           path: build/behavex_output
+  modular_docker_tests:
+    name: "Modular Docker integration tests (x86_64)"
+    needs: docker_build
+    runs-on: ubuntu-24.04
+    timeout-minutes: 180
+    steps:
+      - id: checkout
+        uses: actions/checkout@v4
+      - id: run_cmake
+        name: Run CMake
+        run: |
+          mkdir build
+          cd build
+          cmake ${DOCKER_CMAKE_FLAGS} ..
+      - name: Download artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: minifi_docker
+          path: build
+      - name: Load Docker image
+        run: |
+          docker load --input ./build/minifi_docker.tar
+      - id: install_deps
+        name: Install dependencies for Docker Verify
+        run: |
+          sudo apt update
+          sudo apt install -y python3-virtualenv
+      - id: free_disk_space
+        run: |
+          # We can gain additional disk space on the Ubuntu runners thanks to these suggestions:
+          # https://github.com/actions/runner-images/issues/2840#issuecomment-790492173
+          # https://github.com/actions/runner-images/issues/2606#issuecomment-772683150
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /usr/local/lib/android
+          sudo rm -rf /opt/ghc
+          sudo rm -rf "/usr/local/share/boost"
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY"
+      - id: test
+        name: Docker Verify
+        working-directory: ./build
+        run: make docker-verify-modular

Review Comment:
   I think we should document how the modular docker tests should be run using the make target, and also manually a specific feature set. We should also document the required python environment and the required python version. In some places we are using the `|` pipe character which would require python 3.10, but other times we are using Optional instead which can be used with python 3.9. We should decide and if we require python 3.10 then maybe we may not need the `typing` package at all but we could use the built in type hints.



##########
behave_framework/src/minifi_test_framework/containers/docker_container_builder.py:
##########
@@ -0,0 +1,88 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import tempfile
+from docker.models.images import Image
+from typing import Optional
+
+import docker

Review Comment:
   nitpick, but I would move it up the top next to the other imports



##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:

Review Comment:
   nitpick, but we could keep it consistent with the other check by just using `if not self.container:` here too



##########
behave_framework/src/minifi_test_framework/steps/core_steps.py:
##########
@@ -0,0 +1,55 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import humanfriendly
+import logging
+import random
+import string
+from behave import when, step
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+@when("both instances start up")
+@when("all instances start up")
+def step_impl(context: MinifiTestContext):
+    for container in context.containers:
+        assert container.deploy()
+    assert context.minifi_container.deploy() or context.minifi_container.get_logs()
+    logging.debug("All instances started up")
+
+
+@when("the MiNiFi instance starts up")
+def step_impl(context):
+    assert context.minifi_container.deploy()
+    logging.debug("All instances started up")

Review Comment:
   I think this log doesn't belong here



##########
behave_framework/src/minifi_test_framework/containers/docker_container_builder.py:
##########
@@ -0,0 +1,88 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import tempfile
+from docker.models.images import Image
+from typing import Optional
+
+import docker
+
+
+class DockerContainerBuilder:

Review Comment:
   I would rename this DockerImageBuilder as it builds images not containers



##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass

Review Comment:
   We could write a warning here



##########
extensions/aws/tests/features/s3.feature:
##########
@@ -19,89 +19,74 @@ Feature: Sending data from MiNiFi-C++ to an AWS server
   As a user of MiNiFi
   I need to have PutS3Object and DeleteS3Object processors
 
-  Background:
-    Given the content of "/tmp/output" is monitored
-
   Scenario: A MiNiFi instance transfers encoded data to s3
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
-    And a PutS3Object processor set up to communicate with an s3 server
-    And a PutFile processor with the "Directory" property set to "/tmp/output"
-    And the "success" relationship of the GetFile processor is connected to the PutS3Object
-    And the "success" relationship of the PutS3Object processor is connected to the PutFile
-
-    And a s3 server is set up in correspondence with the PutS3Object
-
-    When both instances start up
-
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
-    And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
-    And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
-    And the Minifi logs contain the following message: "in a single upload" in less than 10 seconds
-
-  Scenario: A MiNiFi instance transfers encoded data to s3 in FIPS mode
-    Given OpenSSL FIPS mode is enabled in MiNiFi
-    And a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the PutS3Object
     And the "success" relationship of the PutS3Object processor is connected to the PutFile
+    And the "failure" relationship of the PutS3Object processor is connected to the PutS3Object
+    And PutFile's success relationship is auto-terminated
 
-    And a s3 server is set up in correspondence with the PutS3Object
+    And an s3 server is set up in correspondence with the PutS3Object
 
     When both instances start up
 
-    Then a flowfile with the content "test" is placed in the monitored directory in less than 60 seconds
+    Then there is a single file with "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" content in the "/tmp/output" directory in less than 20 seconds
     And the object on the s3 server is "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And the object content type on the s3 server is "application/octet-stream" and the object metadata matches use metadata
     And the Minifi logs contain the following message: "in a single upload" in less than 10 seconds
 
   Scenario: A MiNiFi instance transfers encoded data through a http proxy to s3
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file with the content "LH_O#L|FD<FASD{FO#@$#$%^ \"#\"$L%:\"@#$L\":test_data#$#%#$%?{\"F{"
     And a PutS3Object processor set up to communicate with an s3 server
-    And these processor properties are set to match the http proxy:
-    | processor name  | property name  | property value                  |
-    | PutS3Object     | Proxy Host     | http-proxy-${feature_id}        |
-    | PutS3Object     | Proxy Port     | 3128                            |
-    | PutS3Object     | Proxy Username | admin                           |
-    | PutS3Object     | Proxy Password | test101                         |
+    And these processor properties are set
+      | processor name  | property name  | property value                  |
+      | PutS3Object     | Proxy Host     | http-proxy-s3-1                 |

Review Comment:
   Would it be possible to substitute the scenario name automatically instead of figuring it out manually when writing the test? If there are a large number of test scenarios it would be cumbersome and error prone to figure out the current index of the scenario, or if an additional scenario is added in between it would need to be changed.



##########
behave_framework/src/minifi_test_framework/core/hooks.py:
##########
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import logging
+import os
+from behave.model import Scenario
+from behave.runner import Context
+from minifi_test_framework.containers.minifi_container import MinifiContainer
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+import docker

Review Comment:
   nitpick, could be moved up to the first imports



##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):

Review Comment:
   We should define a type hint for the return value like `-> Tuple[int | None, str]



##########
behave_framework/src/minifi_test_framework/core/helpers.py:
##########
@@ -0,0 +1,58 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from __future__ import annotations
+
+import logging
+import time
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+from typing import Callable
+
+
+def log_due_to_failure(context: MinifiTestContext | None):
+    if context is not None:
+        for container in context.containers:
+            container.log_app_output()
+        context.minifi_container.log_app_output()
+
+
+def wait_for_condition(condition: Callable[[], bool], timeout_seconds: float, bail_condition: Callable[[], bool],
+                       context: MinifiTestContext | None) -> bool:
+    if bail_condition():
+        logging.warning("Bail condition evaluated to 'True', aborting wait.")
+        log_due_to_failure(context)
+        return False
+    start_time = time.monotonic()
+    try:
+        while time.monotonic() - start_time < timeout_seconds:
+            if condition():
+                return True
+            if bail_condition():
+                logging.warning("Bail condition evaluated to 'True', aborting wait.")
+                log_due_to_failure(context)
+                return False
+            remaining_time = timeout_seconds - (time.monotonic() - start_time)
+            sleep_time = min(timeout_seconds / 10, remaining_time)
+            if sleep_time > 0:
+                time.sleep(sleep_time)
+    except (Exception,):

Review Comment:
   Could this be just `except Exception:`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2514153353


##########
extensions/azure/tests/features/steps/azure_server_container.py:
##########
@@ -0,0 +1,103 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+
+from docker.errors import ContainerError
+from minifi_test_framework.containers.container import Container
+from minifi_test_framework.core.helpers import run_cmd_in_docker_image
+from minifi_test_framework.core.helpers import wait_for_condition
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+class AzureServerContainer(Container):
+    def __init__(self, test_context: MinifiTestContext):
+        super().__init__("mcr.microsoft.com/azure-storage/azurite:3.35.0",
+                         f"azure-storage-server-{test_context.scenario_id}",
+                         test_context.network)
+        azure_storage_hostname = f"azure-storage-server-{test_context.scenario_id}"
+        self.azure_connection_string = "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;" \
+                                       f"BlobEndpoint=http://{azure_storage_hostname}:10000/devstoreaccount1;QueueEndpoint=http://{azure_storage_hostname}:10001/devstoreaccount1;"
+
+    def deploy(self):
+        super().deploy()
+        finished_str = "Azurite Queue service is successfully listening at"
+        return wait_for_condition(condition=lambda: finished_str in self.get_logs(),
+                                  timeout_seconds=15,
+                                  bail_condition=lambda: self.exited,
+                                  context=None)
+
+    def check_azure_storage_server_data(self, test_data):
+        (code, output) = self.exec_run(["find", "/data/__blobstorage__", "-type", "f"])
+        if code != 0:
+            return False
+        data_file = output.strip()
+        (code, file_data) = self.exec_run(["cat", data_file])
+        return code == 0 and test_data in file_data
+
+    def add_test_blob(self, blob_name, content="test_data", with_snapshot=False) -> bool:
+        cmd_create = (f'az storage container create --name "test-container" '
+                      f'--connection-string "{self.azure_connection_string}"')
+        try:
+            run_cmd_in_docker_image("mcr.microsoft.com/azure-cli", cmd_create, self.network.name)
+        except ContainerError as e:
+            logging.error(e)
+            return False
+
+        cmd_upload = (f'az storage blob upload --container-name "test-container" '
+                      f'--name "{blob_name}" --data "{content}" '
+                      f'--connection-string "{self.azure_connection_string}"')
+        try:
+            run_cmd_in_docker_image("mcr.microsoft.com/azure-cli", cmd_upload, self.network.name)
+        except ContainerError as e:
+            logging.error(e)
+            return False
+
+        if with_snapshot:
+            cmd_snapshot = (f'az storage blob snapshot --container-name "test-container" '
+                            f'--name "{blob_name}" '
+                            f'--connection-string "{self.azure_connection_string}"')
+            try:
+                run_cmd_in_docker_image("mcr.microsoft.com/azure-cli", cmd_snapshot, self.network.name)
+            except ContainerError as e:
+                logging.error(e)
+                return False
+
+        return True
+
+    def __get_blob_and_snapshot_count(self):
+        cmd = (f'az storage blob list --container-name "test-container" '
+               f'--include deleted --query "length(@)" --output tsv '
+               f'--connection-string "{self.azure_connection_string}"')

Review Comment:
   ```suggestion
           cmd = ["az", "storage", "blob", "list", "--container-name", "test-container",
                  "--include", "deleted", "--query", '"length(@)"', "--output", "tsv",
                  "--connection-string", self.azure_connection_string]
   ```



##########
behave_framework/src/minifi_test_framework/core/helpers.py:
##########
@@ -0,0 +1,73 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from __future__ import annotations
+
+import logging
+import time
+from collections.abc import Callable
+
+import docker
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+def log_due_to_failure(context: MinifiTestContext | None):
+    if context is not None:
+        for container in context.containers:
+            container.log_app_output()
+        context.minifi_container.log_app_output()
+
+
+def wait_for_condition(condition: Callable[[], bool], timeout_seconds: float, bail_condition: Callable[[], bool],
+                       context: MinifiTestContext | None) -> bool:
+    if bail_condition():
+        logging.warning("Bail condition evaluated to 'True', aborting wait.")
+        log_due_to_failure(context)
+        return False
+    start_time = time.monotonic()
+    try:
+        while time.monotonic() - start_time < timeout_seconds:
+            if condition():
+                return True
+            if bail_condition():
+                logging.warning("Bail condition evaluated to 'True', aborting wait.")
+                log_due_to_failure(context)
+                return False
+            remaining_time = timeout_seconds - (time.monotonic() - start_time)
+            target_sleep = 1.0 if timeout_seconds < 30 else 10.0
+            sleep_time = min(target_sleep, remaining_time)
+            if sleep_time > 0:
+                time.sleep(sleep_time)
+    except (Exception,):
+        logging.warning("Exception while waiting for condition")
+        log_due_to_failure(context)
+        return False
+    logging.warning("Timed out after %d seconds while waiting for condition", timeout_seconds)
+    log_due_to_failure(context)
+    return False
+
+
+def run_cmd_in_docker_image(image_name: str, cmd: str, network: str) -> str:
+    client = docker.from_env()
+    output = client.containers.run(image=image_name,
+                                   command=["/bin/sh", "-c", cmd],
+                                   remove=True,
+                                   stdout=True,
+                                   stderr=True,
+                                   network=network,
+                                   detach=False)
+    return output.decode("utf-8")

Review Comment:
   I would avoid forcing a shell layer on every caller when it's not necessary, and have cmd be directly forwarded as either a str or list to the command kwarg
   ```suggestion
   def run_cmd_in_docker_image(image_name: str, cmd: str|list, network: str) -> str:
       client = docker.from_env()
       output = client.containers.run(image=image_name,
                                      command=cmd,
                                      remove=True,
                                      stdout=True,
                                      stderr=True,
                                      network=network,
                                      detach=False)
       return output.decode("utf-8")
       
   def run_shell_cmd_in_docker_image(image_name: str, cmd: str, network: str) -> str:
       return run_cmd_in_docker_image(image_name, ["/bin/sh", "-c", cmd], network)
   ```
   
   Also if @lordgamez has figured out how to fail on non-zero exit code, that would be a nice improvement.



##########
extensions/azure/tests/features/steps/azure_server_container.py:
##########
@@ -0,0 +1,103 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+
+from docker.errors import ContainerError
+from minifi_test_framework.containers.container import Container
+from minifi_test_framework.core.helpers import run_cmd_in_docker_image
+from minifi_test_framework.core.helpers import wait_for_condition
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+class AzureServerContainer(Container):
+    def __init__(self, test_context: MinifiTestContext):
+        super().__init__("mcr.microsoft.com/azure-storage/azurite:3.35.0",
+                         f"azure-storage-server-{test_context.scenario_id}",
+                         test_context.network)
+        azure_storage_hostname = f"azure-storage-server-{test_context.scenario_id}"
+        self.azure_connection_string = "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;" \
+                                       f"BlobEndpoint=http://{azure_storage_hostname}:10000/devstoreaccount1;QueueEndpoint=http://{azure_storage_hostname}:10001/devstoreaccount1;"
+
+    def deploy(self):
+        super().deploy()
+        finished_str = "Azurite Queue service is successfully listening at"
+        return wait_for_condition(condition=lambda: finished_str in self.get_logs(),
+                                  timeout_seconds=15,
+                                  bail_condition=lambda: self.exited,
+                                  context=None)
+
+    def check_azure_storage_server_data(self, test_data):
+        (code, output) = self.exec_run(["find", "/data/__blobstorage__", "-type", "f"])
+        if code != 0:
+            return False
+        data_file = output.strip()
+        (code, file_data) = self.exec_run(["cat", data_file])
+        return code == 0 and test_data in file_data
+
+    def add_test_blob(self, blob_name, content="test_data", with_snapshot=False) -> bool:
+        cmd_create = (f'az storage container create --name "test-container" '
+                      f'--connection-string "{self.azure_connection_string}"')
+        try:
+            run_cmd_in_docker_image("mcr.microsoft.com/azure-cli", cmd_create, self.network.name)
+        except ContainerError as e:
+            logging.error(e)
+            return False
+
+        cmd_upload = (f'az storage blob upload --container-name "test-container" '
+                      f'--name "{blob_name}" --data "{content}" '
+                      f'--connection-string "{self.azure_connection_string}"')
+        try:
+            run_cmd_in_docker_image("mcr.microsoft.com/azure-cli", cmd_upload, self.network.name)
+        except ContainerError as e:
+            logging.error(e)
+            return False
+
+        if with_snapshot:
+            cmd_snapshot = (f'az storage blob snapshot --container-name "test-container" '
+                            f'--name "{blob_name}" '
+                            f'--connection-string "{self.azure_connection_string}"')

Review Comment:
   ```suggestion
               cmd_snapshot = ["az", "storage", "blob", "snapshot", "--container-name", "test-container",
                               "--name", blob_name, "--connection-string", self.azure_connection_string]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2514249977


##########
behave_framework/src/minifi_test_framework/core/helpers.py:
##########
@@ -0,0 +1,73 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from __future__ import annotations
+
+import logging
+import time
+from collections.abc import Callable
+
+import docker
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+def log_due_to_failure(context: MinifiTestContext | None):
+    if context is not None:
+        for container in context.containers:
+            container.log_app_output()
+        context.minifi_container.log_app_output()
+
+
+def wait_for_condition(condition: Callable[[], bool], timeout_seconds: float, bail_condition: Callable[[], bool],
+                       context: MinifiTestContext | None) -> bool:
+    if bail_condition():
+        logging.warning("Bail condition evaluated to 'True', aborting wait.")
+        log_due_to_failure(context)
+        return False
+    start_time = time.monotonic()
+    try:
+        while time.monotonic() - start_time < timeout_seconds:
+            if condition():
+                return True
+            if bail_condition():
+                logging.warning("Bail condition evaluated to 'True', aborting wait.")
+                log_due_to_failure(context)
+                return False
+            remaining_time = timeout_seconds - (time.monotonic() - start_time)
+            target_sleep = 1.0 if timeout_seconds < 30 else 10.0
+            sleep_time = min(target_sleep, remaining_time)
+            if sleep_time > 0:
+                time.sleep(sleep_time)
+    except (Exception,):
+        logging.warning("Exception while waiting for condition")
+        log_due_to_failure(context)
+        return False
+    logging.warning("Timed out after %d seconds while waiting for condition", timeout_seconds)
+    log_due_to_failure(context)
+    return False
+
+
+def run_cmd_in_docker_image(image_name: str, cmd: str, network: str) -> str:
+    client = docker.from_env()
+    output = client.containers.run(image=image_name,
+                                   command=["/bin/sh", "-c", cmd],
+                                   remove=True,
+                                   stdout=True,
+                                   stderr=True,
+                                   network=network,
+                                   detach=False)
+    return output.decode("utf-8")

Review Comment:
   yeah, we discussed it yesterday this one will throw when encountering non-zero exit code thus failing the test



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466584526


##########
behave_framework/src/minifi_test_framework/core/helpers.py:
##########
@@ -0,0 +1,58 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from __future__ import annotations
+
+import logging
+import time
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+from typing import Callable
+
+
+def log_due_to_failure(context: MinifiTestContext | None):
+    if context is not None:
+        for container in context.containers:
+            container.log_app_output()
+        context.minifi_container.log_app_output()
+
+
+def wait_for_condition(condition: Callable[[], bool], timeout_seconds: float, bail_condition: Callable[[], bool],
+                       context: MinifiTestContext | None) -> bool:
+    if bail_condition():
+        logging.warning("Bail condition evaluated to 'True', aborting wait.")
+        log_due_to_failure(context)
+        return False
+    start_time = time.monotonic()
+    try:
+        while time.monotonic() - start_time < timeout_seconds:
+            if condition():
+                return True
+            if bail_condition():
+                logging.warning("Bail condition evaluated to 'True', aborting wait.")
+                log_due_to_failure(context)
+                return False
+            remaining_time = timeout_seconds - (time.monotonic() - start_time)
+            sleep_time = min(timeout_seconds / 10, remaining_time)
+            if sleep_time > 0:
+                time.sleep(sleep_time)
+    except (Exception,):

Review Comment:
   flake8 doesnt like that :/ 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466587203


##########
behave_framework/src/minifi_test_framework/containers/docker_container_builder.py:
##########
@@ -0,0 +1,88 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import tempfile
+from docker.models.images import Image
+from typing import Optional
+
+import docker
+
+
+class DockerContainerBuilder:

Review Comment:
   good idea, https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/d58939bf209e4ed1cf9428404b1b1843a970bf69#diff-fb4d88af14495f7974249113edc8926f80a768cd21f7bd7b00458a290513ab12R26



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2495269825


##########
extensions/kafka/tests/features/steps/steps.py:
##########
@@ -0,0 +1,85 @@
+import binascii
+
+from behave import step, when, given
+
+
+from minifi_test_framework.steps import checking_steps        # noqa: F401
+from minifi_test_framework.steps import configuration_steps   # noqa: F401
+from minifi_test_framework.steps import core_steps            # noqa: F401
+from minifi_test_framework.steps import flow_building_steps   # noqa: F401
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+from minifi_test_framework.minifi.processor import Processor
+from kafka_server_container import KafkaServer
+
+
+@step("a {processor_name} processor set up to communicate with an Azure blob storage")
+def step_impl(context: MinifiTestContext, processor_name: str):
+    processor = Processor(processor_name, processor_name)
+    hostname = f"http://azure-storage-server-{context.scenario_id}"
+    processor.add_property('Container Name', 'test-container')
+    processor.add_property('Connection String', 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint={hostname}:10000/devstoreaccount1;QueueEndpoint={hostname}:10001/devstoreaccount1;'.format(hostname=hostname))
+    processor.add_property('Blob', 'test-blob')
+    processor.add_property('Create Container', 'true')
+    context.minifi_container.flow_definition.add_processor(processor)

Review Comment:
   yeah and not just that this modular kafka part is not close to being ready (it wasnt run thats why i didnt notice) i reverted the kafka to the legacy framework and removed this modular part 👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2470001478


##########
.github/workflows/ci.yml:
##########
@@ -535,6 +535,64 @@ jobs:
         with:
           name: behavex_output
           path: build/behavex_output
+  modular_docker_tests:
+    name: "Modular Docker integration tests (x86_64)"
+    needs: docker_build
+    runs-on: ubuntu-24.04
+    timeout-minutes: 180
+    steps:
+      - id: checkout
+        uses: actions/checkout@v4
+      - id: run_cmake
+        name: Run CMake
+        run: |
+          mkdir build
+          cd build
+          cmake ${DOCKER_CMAKE_FLAGS} ..
+      - name: Download artifact
+        uses: actions/download-artifact@v4
+        with:
+          name: minifi_docker
+          path: build
+      - name: Load Docker image
+        run: |
+          docker load --input ./build/minifi_docker.tar
+      - id: install_deps
+        name: Install dependencies for Docker Verify
+        run: |
+          sudo apt update
+          sudo apt install -y python3-virtualenv
+      - id: free_disk_space
+        run: |
+          # We can gain additional disk space on the Ubuntu runners thanks to these suggestions:
+          # https://github.com/actions/runner-images/issues/2840#issuecomment-790492173
+          # https://github.com/actions/runner-images/issues/2606#issuecomment-772683150
+          sudo rm -rf /usr/share/dotnet
+          sudo rm -rf /usr/local/lib/android
+          sudo rm -rf /opt/ghc
+          sudo rm -rf "/usr/local/share/boost"
+          sudo rm -rf "$AGENT_TOOLSDIRECTORY"
+      - id: test
+        name: Docker Verify
+        working-directory: ./build
+        run: make docker-verify-modular

Review Comment:
   I would also add a markdown file with the steps for setting up the virtualenv and use behavex to run a single extension's test scenarios, maybe an additional helper script to make it easier.
   
   Also how much effort would it take to remove all `typing` package imports and use the built in types for the type hints if we want to use the modernized type hints of python 3.10 everywhere, like list[], tuple[], | etc. instead of typing.List[], typing.Tuple[], typing.Optiona[]? Could that be changed in this PR?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469983384


##########
extensions/sql/tests/features/sql.feature:
##########
@@ -29,13 +26,13 @@ Feature: Executing SQL operations from MiNiFi-C++
     And the "success" relationship of the GenerateFlowFile processor is connected to the UpdateAttribute
     And the "success" relationship of the UpdateAttribute processor is connected to the PutSQL
     And an ODBCService is setup up for PutSQL with the name "ODBCService"
+    And PutSQL's success relationship is auto-terminated
     And a PostgreSQL server is set up
     When all instances start up
     Then the query "SELECT * FROM test_table WHERE int_col = 42" returns 1 rows in less than 60 seconds on the PostgreSQL server

Review Comment:
   I think this test fails in the CI due to the issues I also encountered when running this test:
   - The check searches for "rows" in the output while if a single line is inserted it only returns the output "1 row" so "rows" is not found
   - When uniqueness is not guaranteed in the table the same line is inserted multiple times so the query might return multiple lines instead of the single row that is checked
   
   I added some changes to fix these issues in the `extensions/sql/tests/features/sql.feature` and `extensions/sql/tests/features/steps/postgress_server_container.py` files to fix these issues in https://github.com/apache/nifi-minifi-cpp/pull/2052/ that could be added to this PR



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469945741


##########
behave_framework/src/minifi_test_framework/core/helpers.py:
##########
@@ -0,0 +1,58 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from __future__ import annotations
+
+import logging
+import time
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+from typing import Callable
+
+
+def log_due_to_failure(context: MinifiTestContext | None):
+    if context is not None:
+        for container in context.containers:
+            container.log_app_output()
+        context.minifi_container.log_app_output()
+
+
+def wait_for_condition(condition: Callable[[], bool], timeout_seconds: float, bail_condition: Callable[[], bool],
+                       context: MinifiTestContext | None) -> bool:
+    if bail_condition():
+        logging.warning("Bail condition evaluated to 'True', aborting wait.")
+        log_due_to_failure(context)
+        return False
+    start_time = time.monotonic()
+    try:
+        while time.monotonic() - start_time < timeout_seconds:
+            if condition():
+                return True
+            if bail_condition():
+                logging.warning("Bail condition evaluated to 'True', aborting wait.")
+                log_due_to_failure(context)
+                return False
+            remaining_time = timeout_seconds - (time.monotonic() - start_time)
+            sleep_time = min(timeout_seconds / 10, remaining_time)

Review Comment:
   Looks good to me :+1: 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469159630


##########
extensions/standard-processors/tests/features/attributes_to_json.feature:
##########
@@ -15,17 +15,16 @@
 
 @CORE
 Feature: Writing attribute data using AttributesToJSON processor
-  Background:
-    Given the content of "/tmp/output" is monitored
 
   Scenario: Write selected attribute data to file
     Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
-    And a file with filename "test_file.log" and content "test_data" is present in "/tmp/input"
+    And a directory at "/tmp/input" has a file ("test_file.log") with the content "test_data"
     And a AttributesToJSON processor with the "Attributes List" property set to "filename,invalid"
     And the "Destination" property of the AttributesToJSON processor is set to "flowfile-content"
     And the "Null Value" property of the AttributesToJSON processor is set to "true"
     And a PutFile processor with the "Directory" property set to "/tmp/output"
     And the "success" relationship of the GetFile processor is connected to the AttributesToJSON
     And the "success" relationship of the AttributesToJSON processor is connected to the PutFile
+    And PutFile's success relationship is auto-terminated
     When the MiNiFi instance starts up
-    Then a flowfile with the JSON content "{"filename":"test_file.log","invalid":null}" is placed in the monitored directory in less than 10 seconds
+    Then there is a single file with "{"invalid":null,"filename":"test_file.log"}" content in the "/tmp/output" directory in less than 20 seconds

Review Comment:
   good idea, https://github.com/apache/nifi-minifi-cpp/commit/6f9dc2cecae94a8c5d4c86b089c60aa3f5c1484e



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466568188


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)

Review Comment:
   [move makedirs outside of loop](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/fcb25c7f405d45abb2f968553352f6692ad47c99)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2527633484


##########
behave_framework/src/minifi_test_framework/containers/file.py:
##########
@@ -0,0 +1,23 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+class File:
+    def __init__(self, path, host_filename, content):
+        self.path = path
+        self.host_filename = host_filename

Review Comment:
   👍 [merge file path and filename](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/1d3eaccde60f3e3905f15fd0f2772a9b8acc4010)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2523806705


##########
behave_framework/src/minifi_test_framework/core/helpers.py:
##########
@@ -0,0 +1,73 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from __future__ import annotations
+
+import logging
+import time
+from collections.abc import Callable
+
+import docker
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+def log_due_to_failure(context: MinifiTestContext | None):
+    if context is not None:
+        for container in context.containers:
+            container.log_app_output()
+        context.minifi_container.log_app_output()
+
+
+def wait_for_condition(condition: Callable[[], bool], timeout_seconds: float, bail_condition: Callable[[], bool],
+                       context: MinifiTestContext | None) -> bool:
+    if bail_condition():
+        logging.warning("Bail condition evaluated to 'True', aborting wait.")
+        log_due_to_failure(context)
+        return False
+    start_time = time.monotonic()
+    try:
+        while time.monotonic() - start_time < timeout_seconds:
+            if condition():
+                return True
+            if bail_condition():
+                logging.warning("Bail condition evaluated to 'True', aborting wait.")
+                log_due_to_failure(context)
+                return False
+            remaining_time = timeout_seconds - (time.monotonic() - start_time)
+            target_sleep = 1.0 if timeout_seconds < 30 else 10.0
+            sleep_time = min(target_sleep, remaining_time)
+            if sleep_time > 0:
+                time.sleep(sleep_time)
+    except (Exception,):
+        logging.warning("Exception while waiting for condition")
+        log_due_to_failure(context)
+        return False
+    logging.warning("Timed out after %d seconds while waiting for condition", timeout_seconds)
+    log_due_to_failure(context)
+    return False
+
+
+def run_cmd_in_docker_image(image_name: str, cmd: str, network: str) -> str:
+    client = docker.from_env()
+    output = client.containers.run(image=image_name,
+                                   command=["/bin/sh", "-c", cmd],
+                                   remove=True,
+                                   stdout=True,
+                                   stderr=True,
+                                   network=network,
+                                   detach=False)
+    return output.decode("utf-8")

Review Comment:
   I've included your suggestion [review changes](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/221aa3751eedf787dc330ea3267066e0d99cc9f8)



##########
extensions/azure/tests/features/steps/azure_server_container.py:
##########
@@ -0,0 +1,103 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+
+from docker.errors import ContainerError
+from minifi_test_framework.containers.container import Container
+from minifi_test_framework.core.helpers import run_cmd_in_docker_image
+from minifi_test_framework.core.helpers import wait_for_condition
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+class AzureServerContainer(Container):
+    def __init__(self, test_context: MinifiTestContext):
+        super().__init__("mcr.microsoft.com/azure-storage/azurite:3.35.0",
+                         f"azure-storage-server-{test_context.scenario_id}",
+                         test_context.network)
+        azure_storage_hostname = f"azure-storage-server-{test_context.scenario_id}"
+        self.azure_connection_string = "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;" \
+                                       f"BlobEndpoint=http://{azure_storage_hostname}:10000/devstoreaccount1;QueueEndpoint=http://{azure_storage_hostname}:10001/devstoreaccount1;"
+
+    def deploy(self):
+        super().deploy()
+        finished_str = "Azurite Queue service is successfully listening at"
+        return wait_for_condition(condition=lambda: finished_str in self.get_logs(),
+                                  timeout_seconds=15,
+                                  bail_condition=lambda: self.exited,
+                                  context=None)
+
+    def check_azure_storage_server_data(self, test_data):
+        (code, output) = self.exec_run(["find", "/data/__blobstorage__", "-type", "f"])
+        if code != 0:
+            return False
+        data_file = output.strip()
+        (code, file_data) = self.exec_run(["cat", data_file])
+        return code == 0 and test_data in file_data
+
+    def add_test_blob(self, blob_name, content="test_data", with_snapshot=False) -> bool:
+        cmd_create = (f'az storage container create --name "test-container" '
+                      f'--connection-string "{self.azure_connection_string}"')
+        try:
+            run_cmd_in_docker_image("mcr.microsoft.com/azure-cli", cmd_create, self.network.name)
+        except ContainerError as e:
+            logging.error(e)
+            return False
+
+        cmd_upload = (f'az storage blob upload --container-name "test-container" '
+                      f'--name "{blob_name}" --data "{content}" '
+                      f'--connection-string "{self.azure_connection_string}"')
+        try:
+            run_cmd_in_docker_image("mcr.microsoft.com/azure-cli", cmd_upload, self.network.name)
+        except ContainerError as e:
+            logging.error(e)
+            return False
+
+        if with_snapshot:
+            cmd_snapshot = (f'az storage blob snapshot --container-name "test-container" '
+                            f'--name "{blob_name}" '
+                            f'--connection-string "{self.azure_connection_string}"')

Review Comment:
   I've included your suggestion [review changes](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/221aa3751eedf787dc330ea3267066e0d99cc9f8)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2518781780


##########
behave_framework/src/minifi_test_framework/core/helpers.py:
##########
@@ -0,0 +1,77 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from __future__ import annotations
+
+import logging
+import time
+from collections.abc import Callable
+
+import docker
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+def log_due_to_failure(context: MinifiTestContext | None):
+    if context is not None:
+        for container in context.containers:
+            container.log_app_output()
+        context.minifi_container.log_app_output()
+
+
+def wait_for_condition(condition: Callable[[], bool], timeout_seconds: float, bail_condition: Callable[[], bool],
+                       context: MinifiTestContext | None) -> bool:
+    if bail_condition():
+        logging.warning("Bail condition evaluated to 'True', aborting wait.")
+        log_due_to_failure(context)
+        return False
+    start_time = time.monotonic()
+    try:
+        while time.monotonic() - start_time < timeout_seconds:
+            if condition():
+                return True
+            if bail_condition():
+                logging.warning("Bail condition evaluated to 'True', aborting wait.")
+                log_due_to_failure(context)
+                return False
+            remaining_time = timeout_seconds - (time.monotonic() - start_time)
+            target_sleep = 1.0 if timeout_seconds < 30 else 10.0

Review Comment:
   After a while I realized that this gets problematic in some cases. If we have a 30 or 60 second timeout for a check because for example it's much slower in the CI environment it also causes that in your own environment the check could take 10 or 20 seconds when it would finish after 2 or 12 seconds. It may also take at least 10 seconds to fail when you increase the timeout for debugging and it can be not trivial to realize why it takes that long. I would just keep it simple and use a 1 second sleep between checks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469236036


##########
behave_framework/src/minifi_test_framework/containers/docker_image_builder.py:
##########
@@ -0,0 +1,88 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import tempfile
+from docker.models.images import Image
+from typing import Optional
+
+import docker

Review Comment:
   good idea, to be honest im not that familiar with python best practices when it comes to formatting, went ahead and reformatted and optimized imports via PyCharm in [auto format via PyCharm](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/1fe9bd2f06b2ca21645b2dd7548a706dcb4631cd)



##########
behave_framework/src/minifi_test_framework/core/hooks.py:
##########
@@ -0,0 +1,59 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import logging
+import os
+from behave.model import Scenario
+from behave.runner import Context
+from minifi_test_framework.containers.minifi_container import MinifiContainer
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+import docker

Review Comment:
   good idea, to be honest im not that familiar with python best practices when it comes to formatting, went ahead and reformatted and optimized imports via PyCharm in [auto format via PyCharm](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/1fe9bd2f06b2ca21645b2dd7548a706dcb4631cd)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466567035


##########
.github/workflows/verify-package.yml:
##########
@@ -94,17 +134,14 @@ jobs:
       - run: |
           cd build && cmake ${DOCKER_CMAKE_FLAGS} ..
           VERIFY_CMD="${{ matrix.platform.verify_cmd }}"
-          if [[ "${{ inputs.enable_fips }}" == "true" ]]; then
-            VERIFY_CMD="${VERIFY_CMD}-fips"
-          fi
           ${{ matrix.platform.build_cmd }} && $VERIFY_CMD
 
       - name: Test Reporter
         if: always()
         uses: phoenix-actions/test-reporting@f957cd93fc2d848d556fa0d03c57bc79127b6b5e  # v15
         with:
           name: "${{ matrix.platform.name }} (${{ matrix.arch }})${{ inputs.enable_fips && ' (FIPS Mode)' || '' }}"
-          path: build/behavex_output/behave/*.xml
+          path: build/behavex_output_2/behave/*.xml

Review Comment:
   [rename behavex_output_2 to behavex_output_modular](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/5aeb0399852daa305fc8fcd9c8910efb1200ecd0)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2468792518


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""

Review Comment:
   I like these changes, but I would still add another test for when the directory exists but it is empty, since `grep` prints an error message in this case, too:
   ```
   $ mkdir /tmp/foo
   $ grep -l -F 'bar' /tmp/foo/*
   /usr/bin/grep: /tmp/foo/*: No such file or directory
   $ touch /tmp/foo/qux
   $ grep -l -F 'bar' /tmp/foo/*
   $ _
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2468792518


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""

Review Comment:
   I like these changes, but I would also check if the directory exists but it is empty, since `grep` prints an error message in this case, too:
   ```
   $ mkdir /tmp/foo
   $ grep -l -F 'bar' /tmp/foo/*
   /usr/bin/grep: /tmp/foo/*: No such file or directory
   $ touch /tmp/foo/qux
   $ grep -l -F 'bar' /tmp/foo/*
   $ 
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2468792518


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""

Review Comment:
   I like these changes, but I would still add another test for when the directory exists but it is empty, since `grep` prints an error message in this case, too:
   ```
   $ mkdir /tmp/foo
   $ grep -l -F 'bar' /tmp/foo/*
   /usr/bin/grep: /tmp/foo/*: No such file or directory
   $ touch /tmp/foo/qux
   $ grep -l -F 'bar' /tmp/foo/*
   $ 
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#issuecomment-3502048837

   > removed the "low-impact" label because while its description treats all test-only changes as "low-impact", and it's unlikely to introduce bugs, a new test framework isn't really "low impact" in the original sense of the phrase. If you disagree, feel free to add back.
   
   agreed 100%, I think the low impact should be unanimous, so if anyone disagrees it should be removed, and you also convinced me about this 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2498380897


##########
extensions/standard-processors/tests/features/syslog_listener.feature:
##########
@@ -30,9 +27,10 @@ Feature: Minifi C++ can act as a syslog listener
     And the "Replacement Value" property of the ReplaceText processor is set to "${syslog.timestamp}	${syslog.priority}	${reverseDnsLookup(${syslog.sender}, 100)}	${syslog.msg}"
     And the "success" relationship of the ListenSyslog processor is connected to the ReplaceText
     And the "success" relationship of the ReplaceText processor is connected to the PutFile
+    And PutFile's success relationship is auto-terminated
 
     When both instances start up
-    Then at least one flowfile's content match the following regex: "^((?:(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}(?:\.\d+)?))(Z|[\+-]\d{2}:\d{2})?)\t13\t.*\tsample_log$" in less than 10 seconds
+    Then at least one file in "/tmp/output" content match the following regex: "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.*13.*sample_log$" in less than 1000 seconds

Review Comment:
   Can the 1000 seconds timeout be changed back to 10 seconds?



##########
extensions/standard-processors/tests/features/syslog_listener.feature:
##########
@@ -30,9 +27,10 @@ Feature: Minifi C++ can act as a syslog listener
     And the "Replacement Value" property of the ReplaceText processor is set to "${syslog.timestamp}	${syslog.priority}	${reverseDnsLookup(${syslog.sender}, 100)}	${syslog.msg}"
     And the "success" relationship of the ListenSyslog processor is connected to the ReplaceText
     And the "success" relationship of the ReplaceText processor is connected to the PutFile
+    And PutFile's success relationship is auto-terminated
 
     When both instances start up
-    Then at least one flowfile's content match the following regex: "^((?:(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}(?:\.\d+)?))(Z|[\+-]\d{2}:\d{2})?)\t13\t.*\tsample_log$" in less than 10 seconds
+    Then at least one file in "/tmp/output" content match the following regex: "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.*13.*sample_log$" in less than 1000 seconds

Review Comment:
   Can the 1000 seconds timeout be changed back to 10 seconds?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2498380897


##########
extensions/standard-processors/tests/features/syslog_listener.feature:
##########
@@ -30,9 +27,10 @@ Feature: Minifi C++ can act as a syslog listener
     And the "Replacement Value" property of the ReplaceText processor is set to "${syslog.timestamp}	${syslog.priority}	${reverseDnsLookup(${syslog.sender}, 100)}	${syslog.msg}"
     And the "success" relationship of the ListenSyslog processor is connected to the ReplaceText
     And the "success" relationship of the ReplaceText processor is connected to the PutFile
+    And PutFile's success relationship is auto-terminated
 
     When both instances start up
-    Then at least one flowfile's content match the following regex: "^((?:(\d{4}-\d{2}-\d{2})T(\d{2}:\d{2}:\d{2}(?:\.\d+)?))(Z|[\+-]\d{2}:\d{2})?)\t13\t.*\tsample_log$" in less than 10 seconds
+    Then at least one file in "/tmp/output" content match the following regex: "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.*13.*sample_log$" in less than 1000 seconds

Review Comment:
   Can the 1000 seconds timeout be changed back to 10 seconds?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2444977857


##########
.github/workflows/verify-package.yml:
##########
@@ -94,17 +134,14 @@ jobs:
       - run: |
           cd build && cmake ${DOCKER_CMAKE_FLAGS} ..
           VERIFY_CMD="${{ matrix.platform.verify_cmd }}"
-          if [[ "${{ inputs.enable_fips }}" == "true" ]]; then
-            VERIFY_CMD="${VERIFY_CMD}-fips"
-          fi
           ${{ matrix.platform.build_cmd }} && $VERIFY_CMD
 
       - name: Test Reporter
         if: always()
         uses: phoenix-actions/test-reporting@f957cd93fc2d848d556fa0d03c57bc79127b6b5e  # v15
         with:
           name: "${{ matrix.platform.name }} (${{ matrix.arch }})${{ inputs.enable_fips && ' (FIPS Mode)' || '' }}"
-          path: build/behavex_output/behave/*.xml
+          path: build/behavex_output_2/behave/*.xml

Review Comment:
   Can we change `_2` to `_modular` or something similar?



##########
docker/RunBehaveTests.sh:
##########
@@ -0,0 +1,200 @@
+#!/bin/bash
+
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the \"License\"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an \"AS IS\" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+
+die()
+{
+  local _ret="${2:-1}"
+  test "${_PRINT_HELP:-no}" = yes && print_help >&2
+  echo "$1" >&2
+  exit "${_ret}"
+}
+
+_positionals=()
+_arg_image_tag_prefix=
+_arg_tags_to_exclude=
+_arg_parallel_processes=3
+
+print_help()
+{
+  printf '%s\n' "Runs the provided behave tests in a containerized environment"
+  printf 'Usage: %s [--image-tag-prefix <arg>] [-h|--help] <minifi_version> <tags_to_run> ...\n' "$0"
+  printf '\t%s\n' "<minifi_version>: the version of minifi"
+  printf '\t%s\n' "<tags_to_run>: include these tags"
+  printf '\t%s\n' "--tags_to_exclude: optional tags that should be skipped (no default)"
+  printf '\t%s\n' "--image-tag-prefix: optional prefix to the docker tag (no default)"
+  printf '\t%s\n' "--parallel_processes: optional argument that specifies the number of parallel processes that can be executed simultaneously. (default: 3)"
+  printf '\t%s\n' "--fips: enables FIPS mode by default"
+  printf '\t%s\n' "-h, --help: Prints help"
+}
+
+
+parse_commandline()
+{
+  _positionals_count=0
+  _arg_fips=false  # Default to false
+  while test $# -gt 0
+  do
+    _key="$1"
+    case "$_key" in
+      --image-tag-prefix)
+        test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
+        _arg_image_tag_prefix="$2"
+        shift
+        ;;
+      --image-tag-prefix=*)
+        _arg_image_tag_prefix="${_key##--image-tag-prefix=}"
+        ;;
+      --tags_to_exclude)
+        test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
+        _arg_tags_to_exclude="$2"
+        shift
+        ;;
+      --tags_to_exclude=*)
+        _arg_tags_to_exclude="${_key##--tags_to_exclude=}"
+        ;;
+      --parallel_processes)
+        test $# -lt 2 && die "Missing value for the optional argument '$_key'." 1
+        _arg_parallel_processes="$2"
+        shift
+        ;;
+      --parallel_processes=*)
+        _arg_parallel_processes="${_key##--parallel_processes=}"
+        ;;
+      --fips)
+        _arg_fips=true  # Set boolean flag to true when argument is present
+        ;;
+      -h|--help)
+        print_help
+        exit 0
+        ;;
+      -h*)
+        print_help
+        exit 0
+        ;;
+      *)
+        _last_positional="$1"
+        _positionals+=("$_last_positional")
+        _positionals_count=$((_positionals_count + 1))
+        ;;
+    esac
+    shift
+  done
+}
+
+
+
+handle_passed_args_count()
+{
+  local _required_args_string="'minifi_version' and 'tags_to_run'"
+  test "${_positionals_count}" -ge 2 || _PRINT_HELP=yes die "FATAL ERROR: Not enough positional arguments - we require at least 2 (namely: $_required_args_string), but got only ${_positionals_count}." 1
+}
+
+
+assign_positional_args()
+{
+  local _positional_name _shift_for=$1
+  _positional_names="_arg_minifi_version _arg_tags_to_run "
+
+  shift "$_shift_for"
+  for _positional_name in ${_positional_names}
+  do
+    test $# -gt 0 || break
+    eval "$_positional_name=\${1}" || die "Error during argument parsing." 1
+    shift
+  done
+}
+
+parse_commandline "$@"
+handle_passed_args_count
+assign_positional_args 1 "${_positionals[@]}"
+
+docker_dir="$( cd "${0%/*}" && pwd )"
+
+# shellcheck disable=SC2154
+export MINIFI_VERSION=${_arg_minifi_version}
+if test -z "$_arg_image_tag_prefix"
+then
+  export MINIFI_TAG_PREFIX=""
+else
+  export MINIFI_TAG_PREFIX=${_arg_image_tag_prefix}-
+fi
+
+  if [ "$_arg_fips" = true ]; then
+    export MINIFI_FIPS="true"
+  else
+    export MINIFI_FIPS="false"
+  fi
+
+# Create virtual environment for testing
+if [[ ! -d ./behave_venv ]]; then
+  echo "Creating virtual environment in ./behave_venv" 1>&2
+  virtualenv --python=python3 ./behave_venv
+fi
+
+echo "Activating virtual environment..." 1>&2
+# shellcheck disable=SC1091
+. ./behave_venv/bin/activate
+pip install --trusted-host pypi.python.org --upgrade pip setuptools
+
+# Install test dependencies
+echo "Installing test dependencies..." 1>&2
+
+# hint include/library paths if homewbrew is in use
+if brew list 2> /dev/null | grep openssl > /dev/null 2>&1; then
+  echo "Using homebrew paths for openssl" 1>&2
+  LDFLAGS="-L$(brew --prefix openssl@1.1)/lib"
+  export LDFLAGS
+  CFLAGS="-I$(brew --prefix openssl@1.1)/include"
+  export CFLAGS
+  SWIG_FEATURES="-cpperraswarn -includeall -I$(brew --prefix openssl@1.1)/include"
+  export SWIG_FEATURES
+fi
+
+if ! command swig -version &> /dev/null; then
+  echo "Swig could not be found on your system (dependency of m2crypto python library). Please install swig to continue."
+  exit 1
+fi
+
+pip install -e "${docker_dir}/../behave_framework"
+
+export TMPDIR="/tmp/behavex_ci_${RANDOM}"
+mkdir -p "$TMPDIR"
+
+export TEMP="$TMPDIR/temp"
+export LOGS="$TMPDIR/logs"
+
+BEHAVE_OPTS=(--show-progress-bar --logging-level DEBUG --parallel-processes "${_arg_parallel_processes}" --parallel-scheme feature -o "${PWD}/behavex_output_2" -t "${_arg_tags_to_run}")

Review Comment:
   Do we need `DEBUG` logs? This used to be `INFO` in `DockerVerify.sh`.



##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""

Review Comment:
   we could check if the directory is empty (and return False if it is) to avoid the "No such file or directory" messages from grep



##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)

Review Comment:
   the `makedirs` could be outside the loop



##########
extensions/aws/tests/features/steps/s3_server_container.py:
##########
@@ -0,0 +1,62 @@
+import json
+import logging
+
+from minifi_test_framework.containers.container import Container
+from minifi_test_framework.core.helpers import wait_for_condition
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+class S3ServerContainer(Container):
+    def __init__(self, test_context: MinifiTestContext):
+        super().__init__("adobe/s3mock:3.12.0", f"s3-server-{test_context.scenario_id}", test_context.network)
+        self.environment.append("initialBuckets=test_bucket")
+
+    def deploy(self):
+        super().deploy()
+        finished_str = "Started S3MockApplication"
+        return wait_for_condition(
+            condition=lambda: finished_str in self.get_logs(),
+            timeout_seconds=15,
+            bail_condition=lambda: self.exited,
+            context=None)
+
+    def check_kinesis_server_record_data(self, container_name, record_data):

Review Comment:
   These checks used to have a `@retry_check()` annotation; is that no longer needed?



##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,266 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+import logging
+import os
+import shlex
+import tempfile
+from docker.models.networks import Network
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+from typing import Dict, Any, List
+
+import docker
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"
+        self.client = docker.from_env()
+        self.container = None
+        self.files: List[File] = []
+        self.dirs: List[Directory] = []
+        self.host_files: List[HostFile] = []
+        self.volumes = {}
+        self.command = None
+        self._temp_dir = None
+        self.ports = None
+        self.environment: List[str] = []
+
+    def deploy(self) -> bool:
+        self._temp_dir = tempfile.TemporaryDirectory()
+
+        if len(self.files) != 0 or len(self.dirs) != 0 or len(self.host_files) != 0:
+            for file in self.files:
+                temp_path = self._temp_dir.name + "/" + file.host_filename
+                with open(temp_path, "w") as temp_file:
+                    temp_file.write(file.content)
+                self.volumes[temp_path] = {
+                    "bind": file.path + "/" + file.host_filename,
+                    "mode": file.mode
+                }
+            for directory in self.dirs:
+                temp_path = self._temp_dir.name + directory.path
+                for file_name, content in directory.files.items():
+                    file_path = temp_path + "/" + file_name
+                    os.makedirs(temp_path, exist_ok=True)
+                    with open(file_path, "w") as temp_file:
+                        temp_file.write(content)
+                self.volumes[temp_path] = {
+                    "bind": directory.path,
+                    "mode": directory.mode
+                }
+            for host_file in self.host_files:
+                self.volumes[host_file.container_path] = {
+                    "bind": host_file.host_path,
+                    "mode": host_file.mode
+                }
+
+        try:
+            existing_container = self.client.containers.get(self.container_name)
+            logging.warning(f"Found existing container '{self.container_name}'. Removing it first.")
+            existing_container.remove(force=True)
+        except docker.errors.NotFound:
+            pass
+        try:
+            print(f"Creating and starting container '{self.container_name}'...")
+            self.container = self.client.containers.run(
+                image=self.image_name,
+                name=self.container_name,
+                ports=self.ports,
+                environment=self.environment,
+                volumes=self.volumes,
+                network=self.network.name,
+                command=self.command,
+                user=self.user,
+                detach=True  # Always run in the background
+            )
+        except Exception as e:
+            logging.error(f"Error starting container: {e}")
+            raise
+        return True
+
+    def clean_up(self):
+        if self.container is not None:
+            self.container.remove(force=True)
+
+    def exec_run(self, command):
+        if self.container:
+            (code, output) = self.container.exec_run(command)
+            return code, output.decode("utf-8")
+        return None, "Container not running."
+
+    def directory_contains_file_with_content(self, directory_path: str, expected_content: str) -> bool:
+        if not self.container:
+            return False
+
+        escaped_content = expected_content.replace("\"", "\\\"")
+
+        command = f"sh -c \"grep -l -F -- '{escaped_content}' {directory_path}/*\""

Review Comment:
   It would be better to use `shlex.quote` (which I didn't know, but you use it below) in all functions in this file. (Eg. what happens if `expected_content` contains a single quote?)
   
   untested suggestion:
   ```suggestion
           quoted_content = shlex.quote(expected_content)
   
           command = "sh -c {}".format(shlex.quote(f"grep -l -F -- {quoted_content} {directory_path}/*")
   ```



##########
behave_framework/src/minifi_test_framework/containers/minifi_container.py:
##########
@@ -0,0 +1,85 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from docker.models.networks import Network
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.minifi.flow_definition import FlowDefinition
+from typing import Dict
+
+from .container import Container
+
+
+class MinifiContainer(Container):
+    def __init__(self, image_name: str, scenario_id: str, network: Network):
+        super().__init__(image_name, f"minifi-{scenario_id}", network)
+        self.flow_config_str: str = ""
+        self.flow_definition = FlowDefinition()
+        self.properties: Dict[str, str] = {}
+        self.log_properties: Dict[str, str] = {}
+
+        self.is_fhs = 'MINIFI_INSTALLATION_TYPE=FHS' in str(self.client.images.get(image_name).history())
+
+        self._fill_default_properties()
+        self._fill_default_log_properties()
+
+    def deploy(self) -> bool:
+        if self.is_fhs:
+            self.files.append(File("/etc/nifi-minifi-cpp", "config.yml", self.flow_definition.to_yaml()))
+            self.files.append(File("/etc/nifi-minifi-cpp", "minifi.properties", self._get_properties_file_content()))
+            self.files.append(
+                File("/etc/nifi-minifi-cpp", "minifi-log.properties", self._get_log_properties_file_content()))
+        else:
+            self.files.append(File("/opt/minifi/minifi-current/conf", "config.yml", self.flow_definition.to_yaml()))
+            self.files.append(
+                File("/opt/minifi/minifi-current/conf", "minifi.properties", self._get_properties_file_content()))
+            self.files.append(File("/opt/minifi/minifi-current/conf", "minifi-log.properties",
+                                   self._get_log_properties_file_content()))

Review Comment:
   is `minifi-uid.properties` not needed here?



##########
docker/test/integration/features/consume_kafka.feature:
##########


Review Comment:
   You have already converted some of these tests in `extensions/kafka/tests/features/consumekafka.feature`. I think the tests which are in `consumekafka.feature` should be removed from here.



##########
extensions/kafka/tests/features/consumekafka.feature:
##########


Review Comment:
   Is the `consume_kafka.feature` → `consumekafka.feature` name change intentional? If not, then I would prefer the old name, because it's easier to read.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2465199537


##########
extensions/sql/tests/features/environment.py:
##########
@@ -0,0 +1,83 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import os
+from textwrap import dedent
+
+from minifi_test_framework.containers.docker_container_builder import DockerContainerBuilder
+from minifi_test_framework.core.hooks import common_before_scenario
+from minifi_test_framework.core.hooks import common_after_scenario
+from minifi_test_framework.core.hooks import get_minifi_container_image
+
+
+def before_all(context):
+    minifi_tag_prefix = os.environ['MINIFI_TAG_PREFIX']

Review Comment:
   This fails if sql tests are run independently, should be changed to:
   ```suggestion
       minifi_tag_prefix = os.environ['MINIFI_TAG_PREFIX'] if 'MINIFI_TAG_PREFIX' in os.environ else ''
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2466576337


##########
extensions/aws/tests/features/steps/s3_server_container.py:
##########
@@ -0,0 +1,62 @@
+import json
+import logging
+
+from minifi_test_framework.containers.container import Container
+from minifi_test_framework.core.helpers import wait_for_condition
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+
+
+class S3ServerContainer(Container):
+    def __init__(self, test_context: MinifiTestContext):
+        super().__init__("adobe/s3mock:3.12.0", f"s3-server-{test_context.scenario_id}", test_context.network)
+        self.environment.append("initialBuckets=test_bucket")
+
+    def deploy(self):
+        super().deploy()
+        finished_str = "Started S3MockApplication"
+        return wait_for_condition(
+            condition=lambda: finished_str in self.get_logs(),
+            timeout_seconds=15,
+            bail_condition=lambda: self.exited,
+            context=None)
+
+    def check_kinesis_server_record_data(self, container_name, record_data):

Review Comment:
   since these checks dont seem to fail since I guess not, if these start to fail we could use them in a wait_for_condition



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2469270079


##########
behave_framework/src/minifi_test_framework/core/helpers.py:
##########
@@ -0,0 +1,58 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+
+from __future__ import annotations
+
+import logging
+import time
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+from typing import Callable
+
+
+def log_due_to_failure(context: MinifiTestContext | None):
+    if context is not None:
+        for container in context.containers:
+            container.log_app_output()
+        context.minifi_container.log_app_output()
+
+
+def wait_for_condition(condition: Callable[[], bool], timeout_seconds: float, bail_condition: Callable[[], bool],
+                       context: MinifiTestContext | None) -> bool:
+    if bail_condition():
+        logging.warning("Bail condition evaluated to 'True', aborting wait.")
+        log_due_to_failure(context)
+        return False
+    start_time = time.monotonic()
+    try:
+        while time.monotonic() - start_time < timeout_seconds:
+            if condition():
+                return True
+            if bail_condition():
+                logging.warning("Bail condition evaluated to 'True', aborting wait.")
+                log_due_to_failure(context)
+                return False
+            remaining_time = timeout_seconds - (time.monotonic() - start_time)
+            sleep_time = min(timeout_seconds / 10, remaining_time)

Review Comment:
   👍 how about this? [change sleep length in wait_for_condition](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/c20dd7cd41b83b103db99b5753c8cfb30facd39d)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2495202713


##########
extensions/kafka/tests/features/steps/steps.py:
##########
@@ -0,0 +1,85 @@
+import binascii
+
+from behave import step, when, given
+
+
+from minifi_test_framework.steps import checking_steps        # noqa: F401
+from minifi_test_framework.steps import configuration_steps   # noqa: F401
+from minifi_test_framework.steps import core_steps            # noqa: F401
+from minifi_test_framework.steps import flow_building_steps   # noqa: F401
+from minifi_test_framework.core.minifi_test_context import MinifiTestContext
+from minifi_test_framework.minifi.processor import Processor
+from kafka_server_container import KafkaServer
+
+
+@step("a {processor_name} processor set up to communicate with an Azure blob storage")
+def step_impl(context: MinifiTestContext, processor_name: str):
+    processor = Processor(processor_name, processor_name)
+    hostname = f"http://azure-storage-server-{context.scenario_id}"
+    processor.add_property('Container Name', 'test-container')
+    processor.add_property('Connection String', 'DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint={hostname}:10000/devstoreaccount1;QueueEndpoint={hostname}:10001/devstoreaccount1;'.format(hostname=hostname))
+    processor.add_property('Blob', 'test-blob')
+    processor.add_property('Create Container', 'true')
+    context.minifi_container.flow_definition.add_processor(processor)

Review Comment:
   This seems to have been added here by mistake



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2503065812


##########
extensions/aws/tests/features/steps/s3_server_container.py:
##########


Review Comment:
   👍 fixed them in [add missing LICENSEs](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/f70324039669780232cccc002576b856745f34d7)
   
   also i cant reply to your comment so ill do it here:
   it was copy pasted from somewhere else, now i fixed it everywhere 👍 
   [dont escape quotes in comments](https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/59d7d45c86532b1805af7464152deec8bf00725c)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2527632778


##########
behave_framework/src/minifi_test_framework/containers/container.py:
##########
@@ -0,0 +1,289 @@
+#
+#  Licensed to the Apache Software Foundation (ASF) under one or more
+#  contributor license agreements.  See the NOTICE file distributed with
+#  this work for additional information regarding copyright ownership.
+#  The ASF licenses this file to You under the Apache License, Version 2.0
+#  (the "License"); you may not use this file except in compliance with
+#  the License.  You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+#  Unless required by applicable law or agreed to in writing, software
+#  distributed under the License is distributed on an "AS IS" BASIS,
+#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#  See the License for the specific language governing permissions and
+#  limitations under the License.
+#
+import json
+import logging
+import os
+import shlex
+import tempfile
+
+import docker
+from docker.models.networks import Network
+
+from minifi_test_framework.containers.directory import Directory
+from minifi_test_framework.containers.file import File
+from minifi_test_framework.containers.host_file import HostFile
+
+
+class Container:
+    def __init__(self, image_name: str, container_name: str, network: Network):
+        self.image_name: str = image_name
+        self.container_name: str = container_name
+        self.network: Network = network
+        self.user: str = "0:0"

Review Comment:
   i tried them without root but that also caused issues, I think we can cross that bridge when we get there and make this confuigrable in the followup PR-s if neccessary



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "martinzink (via GitHub)" <gi...@apache.org>.
martinzink commented on code in PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#discussion_r2472647242


##########
extensions/sql/tests/features/sql.feature:
##########
@@ -29,13 +26,13 @@ Feature: Executing SQL operations from MiNiFi-C++
     And the "success" relationship of the GenerateFlowFile processor is connected to the UpdateAttribute
     And the "success" relationship of the UpdateAttribute processor is connected to the PutSQL
     And an ODBCService is setup up for PutSQL with the name "ODBCService"
+    And PutSQL's success relationship is auto-terminated
     And a PostgreSQL server is set up
     When all instances start up
     Then the query "SELECT * FROM test_table WHERE int_col = 42" returns 1 rows in less than 60 seconds on the PostgreSQL server

Review Comment:
   you are right, thanks, I've included your patch here https://github.com/apache/nifi-minifi-cpp/pull/2044/commits/d7469f7214b9f2b4c44bec6b7c7f3540668914a3



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINIFICPP-2624 Modularized behave tests part 1 [nifi-minifi-cpp]

Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on PR #2044:
URL: https://github.com/apache/nifi-minifi-cpp/pull/2044#issuecomment-3450139877

   When I ran the docker tests on my own branch due to my own changes I had some failures in both the old and the new docker tests. The report of the new modular tests were okay, but when I opened the test report html it was empty with 0 test scenarios and features listed. It should be checked too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org