You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2022/11/28 15:25:23 UTC
[GitHub] [nifi-minifi-cpp] lordgamez opened a new pull request, #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
lordgamez opened a new pull request, #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461
This PR contains multiple changes and refactoring in the docker test environment:
- Moved docker cluster specific classes to separate `cluster` directory under `test/integration` directory
- minifi.properties and all additional configuration files are now mounted instead of building a separate image for every config
- Added `MinifiOptions` to specify minifi deployment options like additional properties
- New minifi test image is only build if additonal packages are needed (in case of SQL tests)
- Removed inheritance hierarchy for `DockerTestCluster`
- `SingleNodeDockerCluster` renamed to `ContainerStore` and only contains container deployment specific functionality
- Extension specific checkers are extracted to separate checker classes
- Docker communicator extracted for communicated with docker containers
- Moved general container operations to `Container` base class
- Flushing container logs in case of all assertion failures
https://issues.apache.org/jira/browse/MINIFICPP-1983
-------------------------
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on code in PR #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461#discussion_r1073705141
##########
docker/test/integration/cluster/checkers/PostgresChecker.py:
##########
@@ -0,0 +1,18 @@
+import time
+
+
+class PostgresChecker:
+ def __init__(self, container_communicator):
+ self.container_communicator = container_communicator
+
+ def __query_postgres_server(self, postgresql_container_name, query, number_of_rows):
+ (code, output) = self.container_communicator.execute_command(postgresql_container_name, ["psql", "-U", "postgres", "-c", query])
+ return code == 0 and str(number_of_rows) + " rows" in output
+
+ def check_query_results(self, postgresql_container_name, query, number_of_rows, timeout_seconds):
+ start_time = time.perf_counter()
+ while (time.perf_counter() - start_time) < timeout_seconds:
+ if self.__query_postgres_server(postgresql_container_name, query, number_of_rows):
+ return True
+ time.sleep(2)
+ return False
Review Comment:
could we use the `retry_check` wrapper for this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by GitBox <gi...@apache.org>.
lordgamez commented on code in PR #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461#discussion_r1080937969
##########
docker/test/integration/cluster/containers/Container.py:
##########
@@ -63,13 +63,25 @@ def log_source(self):
return LogSource.FROM_DOCKER_CONTAINER
def stop(self):
- raise NotImplementedError()
+ logging.info('Stopping minifi docker container "%s"...', self.name)
Review Comment:
That should be not be minifi specific, fixed log messages in d5329427a1cb89b2cc9d99a85251b62aa0e8a5d4
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on code in PR #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461#discussion_r1080974318
##########
docker/test/integration/cluster/checkers/PostgresChecker.py:
##########
@@ -0,0 +1,18 @@
+import time
+
+
+class PostgresChecker:
+ def __init__(self, container_communicator):
+ self.container_communicator = container_communicator
+
+ def __query_postgres_server(self, postgresql_container_name, query, number_of_rows):
+ (code, output) = self.container_communicator.execute_command(postgresql_container_name, ["psql", "-U", "postgres", "-c", query])
+ return code == 0 and str(number_of_rows) + " rows" in output
+
+ def check_query_results(self, postgresql_container_name, query, number_of_rows, timeout_seconds):
+ start_time = time.perf_counter()
+ while (time.perf_counter() - start_time) < timeout_seconds:
+ if self.__query_postgres_server(postgresql_container_name, query, number_of_rows):
+ return True
+ time.sleep(2)
+ return False
Review Comment:
makes sense, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by GitBox <gi...@apache.org>.
szaszm commented on code in PR #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461#discussion_r1082582759
##########
docker/test/integration/cluster/checkers/AzureChecker.py:
##########
@@ -0,0 +1,74 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import logging
+import time
+from azure.storage.blob import BlobServiceClient
+from azure.core.exceptions import ResourceExistsError
+from utils import retry_check
+
+
+class AzureChecker:
+ AZURE_CONNECTION_STRING = \
+ ("DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;"
+ "BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;")
+
+ def __init__(self, container_communicator):
+ self.container_communicator = container_communicator
+ self.blob_service_client = BlobServiceClient.from_connection_string(AzureChecker.AZURE_CONNECTION_STRING)
+
+ @retry_check()
+ def check_azure_storage_server_data(self, container_name, test_data):
+ (code, output) = self.container_communicator.execute_command(container_name, ["find", "/data/__blobstorage__", "-type", "f"])
+ if code != 0:
+ return False
+ data_file = output.strip()
+ (code, file_data) = self.container_communicator.execute_command(container_name, ["cat", data_file])
+ return code == 0 and test_data in file_data
+
+ def add_test_blob(self, blob_name, content="", with_snapshot=False):
+ try:
+ self.blob_service_client.create_container("test-container")
+ except ResourceExistsError:
+ logging.debug('test-container already exists')
+
+ blob_client = self.blob_service_client.get_blob_client(container="test-container", blob=blob_name)
+ blob_client.upload_blob(content)
+
+ if with_snapshot:
+ blob_client.create_snapshot()
+
+ def __get_blob_and_snapshot_count(self):
+ container_client = self.blob_service_client.get_container_client("test-container")
+ return len(list(container_client.list_blobs(include=['deleted'])))
+
+ def check_azure_blob_and_snapshot_count(self, blob_and_snapshot_count, timeout_seconds):
+ start_time = time.perf_counter()
+ while True:
+ if self.__get_blob_and_snapshot_count() == blob_and_snapshot_count:
+ return True
+ time.sleep(1)
+ if timeout_seconds < (time.perf_counter() - start_time):
+ break
Review Comment:
I was thinking about something like this. Untested example, and my python skills are not the greatest, so treat this as pseudocode. :)
```python
def wait_for(action, timeout, predicate_transform = lambda x: x, check_period = 1):
start_time = time.perf_counter()
while True:
result = action()
if predicate_transform(result):
return (True, result)
time.sleep(check_period)
if timeout_seconds < (time.perf_counter() - start_time):
break
return (False, None)
(happened, _) = wait_for(lambda: self.__get_blob_and_snapshot_count() == blob_and_snapshot_count, timeout_seconds)
# and the more complex example
def get_logs():
logging.info('Waiting for app-logs `%s` in container `%s`', log_entry, container_name)
return self.get_app_log(container_name)
def predicate(x):
(status, logs) = x
return logs is not None or status == 'exited'
(happened, (status, logs)) = wait_for(get_logs, timeout_seconds, predicate)
if not happened or status == 'exited':
return False
# ...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a diff in pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on code in PR #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461#discussion_r1073709877
##########
docker/test/integration/cluster/containers/Container.py:
##########
@@ -63,13 +63,25 @@ def log_source(self):
return LogSource.FROM_DOCKER_CONTAINER
def stop(self):
- raise NotImplementedError()
+ logging.info('Stopping minifi docker container "%s"...', self.name)
Review Comment:
is this a base class which other, non-minifi, containers inherit from? are those "minifi docker container"s as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by GitBox <gi...@apache.org>.
szaszm commented on code in PR #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461#discussion_r1082555652
##########
docker/test/integration/cluster/checkers/AzureChecker.py:
##########
@@ -0,0 +1,74 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import logging
+import time
+from azure.storage.blob import BlobServiceClient
+from azure.core.exceptions import ResourceExistsError
+from utils import retry_check
+
+
+class AzureChecker:
+ AZURE_CONNECTION_STRING = \
+ ("DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;"
+ "BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;")
+
+ def __init__(self, container_communicator):
+ self.container_communicator = container_communicator
+ self.blob_service_client = BlobServiceClient.from_connection_string(AzureChecker.AZURE_CONNECTION_STRING)
+
+ @retry_check()
+ def check_azure_storage_server_data(self, container_name, test_data):
+ (code, output) = self.container_communicator.execute_command(container_name, ["find", "/data/__blobstorage__", "-type", "f"])
+ if code != 0:
+ return False
+ data_file = output.strip()
+ (code, file_data) = self.container_communicator.execute_command(container_name, ["cat", data_file])
+ return code == 0 and test_data in file_data
+
+ def add_test_blob(self, blob_name, content="", with_snapshot=False):
+ try:
+ self.blob_service_client.create_container("test-container")
+ except ResourceExistsError:
+ logging.debug('test-container already exists')
+
+ blob_client = self.blob_service_client.get_blob_client(container="test-container", blob=blob_name)
+ blob_client.upload_blob(content)
+
+ if with_snapshot:
+ blob_client.create_snapshot()
+
+ def __get_blob_and_snapshot_count(self):
+ container_client = self.blob_service_client.get_container_client("test-container")
+ return len(list(container_client.list_blobs(include=['deleted'])))
+
+ def check_azure_blob_and_snapshot_count(self, blob_and_snapshot_count, timeout_seconds):
+ start_time = time.perf_counter()
+ while True:
+ if self.__get_blob_and_snapshot_count() == blob_and_snapshot_count:
+ return True
+ time.sleep(1)
+ if timeout_seconds < (time.perf_counter() - start_time):
+ break
Review Comment:
Did you consider extracting the waiting for condition with timeout logic to a higher order function? These loops in d465264 are similar enough, and we could get rid of some code duplication, but I'm not insisting, +1 even if you don't want to change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by GitBox <gi...@apache.org>.
lordgamez commented on code in PR #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461#discussion_r1082521737
##########
docker/test/integration/cluster/checkers/AzureChecker.py:
##########
@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import logging
+import time
+from azure.storage.blob import BlobServiceClient
+from azure.core.exceptions import ResourceExistsError
+from utils import retry_check
+
+
+class AzureChecker:
+ AZURE_CONNECTION_STRING = \
+ ("DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;"
+ "BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;")
+
+ def __init__(self, container_communicator):
+ self.container_communicator = container_communicator
+ self.blob_service_client = BlobServiceClient.from_connection_string(AzureChecker.AZURE_CONNECTION_STRING)
+
+ @retry_check()
+ def check_azure_storage_server_data(self, container_name, test_data):
+ (code, output) = self.container_communicator.execute_command(container_name, ["find", "/data/__blobstorage__", "-type", "f"])
+ if code != 0:
+ return False
+ data_file = output.strip()
+ (code, file_data) = self.container_communicator.execute_command(container_name, ["cat", data_file])
+ return code == 0 and test_data in file_data
+
+ def add_test_blob(self, blob_name, content="", with_snapshot=False):
+ try:
+ self.blob_service_client.create_container("test-container")
+ except ResourceExistsError:
+ logging.debug('test-container already exists')
+
+ blob_client = self.blob_service_client.get_blob_client(container="test-container", blob=blob_name)
+ blob_client.upload_blob(content)
+
+ if with_snapshot:
+ blob_client.create_snapshot()
+
+ def __get_blob_and_snapshot_count(self):
+ container_client = self.blob_service_client.get_container_client("test-container")
+ return len(list(container_client.list_blobs(include=['deleted'])))
+
+ def check_azure_blob_and_snapshot_count(self, blob_and_snapshot_count, timeout_seconds):
+ start_time = time.perf_counter()
+ while (time.perf_counter() - start_time) < timeout_seconds:
+ if self.__get_blob_and_snapshot_count() == blob_and_snapshot_count:
+ return True
+ time.sleep(1)
+ return False
+
+ def check_azure_blob_storage_is_empty(self, timeout_seconds):
+ start_time = time.perf_counter()
+ while (time.perf_counter() - start_time) < timeout_seconds:
+ if self.__get_blob_and_snapshot_count() == 0:
+ return True
+ time.sleep(1)
+ return False
Review Comment:
Updated in d465264923264b0b3f4c9317ed0f5c50adc9729c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits closed pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits closed pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by GitBox <gi...@apache.org>.
lordgamez commented on code in PR #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461#discussion_r1080939840
##########
docker/test/integration/cluster/checkers/PostgresChecker.py:
##########
@@ -0,0 +1,18 @@
+import time
+
+
+class PostgresChecker:
+ def __init__(self, container_communicator):
+ self.container_communicator = container_communicator
+
+ def __query_postgres_server(self, postgresql_container_name, query, number_of_rows):
+ (code, output) = self.container_communicator.execute_command(postgresql_container_name, ["psql", "-U", "postgres", "-c", query])
+ return code == 0 and str(number_of_rows) + " rows" in output
+
+ def check_query_results(self, postgresql_container_name, query, number_of_rows, timeout_seconds):
+ start_time = time.perf_counter()
+ while (time.perf_counter() - start_time) < timeout_seconds:
+ if self.__query_postgres_server(postgresql_container_name, query, number_of_rows):
+ return True
+ time.sleep(2)
+ return False
Review Comment:
Because the timeout is a function parameter that cannot be used as the parameter of the annotation, so I would stick with this solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a diff in pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by GitBox <gi...@apache.org>.
szaszm commented on code in PR #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461#discussion_r1081182871
##########
docker/test/integration/cluster/checkers/AzureChecker.py:
##########
@@ -0,0 +1,70 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import logging
+import time
+from azure.storage.blob import BlobServiceClient
+from azure.core.exceptions import ResourceExistsError
+from utils import retry_check
+
+
+class AzureChecker:
+ AZURE_CONNECTION_STRING = \
+ ("DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;"
+ "BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;")
+
+ def __init__(self, container_communicator):
+ self.container_communicator = container_communicator
+ self.blob_service_client = BlobServiceClient.from_connection_string(AzureChecker.AZURE_CONNECTION_STRING)
+
+ @retry_check()
+ def check_azure_storage_server_data(self, container_name, test_data):
+ (code, output) = self.container_communicator.execute_command(container_name, ["find", "/data/__blobstorage__", "-type", "f"])
+ if code != 0:
+ return False
+ data_file = output.strip()
+ (code, file_data) = self.container_communicator.execute_command(container_name, ["cat", data_file])
+ return code == 0 and test_data in file_data
+
+ def add_test_blob(self, blob_name, content="", with_snapshot=False):
+ try:
+ self.blob_service_client.create_container("test-container")
+ except ResourceExistsError:
+ logging.debug('test-container already exists')
+
+ blob_client = self.blob_service_client.get_blob_client(container="test-container", blob=blob_name)
+ blob_client.upload_blob(content)
+
+ if with_snapshot:
+ blob_client.create_snapshot()
+
+ def __get_blob_and_snapshot_count(self):
+ container_client = self.blob_service_client.get_container_client("test-container")
+ return len(list(container_client.list_blobs(include=['deleted'])))
+
+ def check_azure_blob_and_snapshot_count(self, blob_and_snapshot_count, timeout_seconds):
+ start_time = time.perf_counter()
+ while (time.perf_counter() - start_time) < timeout_seconds:
+ if self.__get_blob_and_snapshot_count() == blob_and_snapshot_count:
+ return True
+ time.sleep(1)
+ return False
+
+ def check_azure_blob_storage_is_empty(self, timeout_seconds):
+ start_time = time.perf_counter()
+ while (time.perf_counter() - start_time) < timeout_seconds:
+ if self.__get_blob_and_snapshot_count() == 0:
+ return True
+ time.sleep(1)
+ return False
Review Comment:
It would probably make sense to check at least once even if timeout is 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a diff in pull request #1461: MINIFICPP-1983 Mount files in MiNiFi test containers and additional refactoring
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1461:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1461#discussion_r1083976564
##########
docker/test/integration/cluster/checkers/AzureChecker.py:
##########
@@ -0,0 +1,74 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import logging
+import time
+from azure.storage.blob import BlobServiceClient
+from azure.core.exceptions import ResourceExistsError
+from utils import retry_check
+
+
+class AzureChecker:
+ AZURE_CONNECTION_STRING = \
+ ("DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;"
+ "BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;QueueEndpoint=http://127.0.0.1:10001/devstoreaccount1;")
+
+ def __init__(self, container_communicator):
+ self.container_communicator = container_communicator
+ self.blob_service_client = BlobServiceClient.from_connection_string(AzureChecker.AZURE_CONNECTION_STRING)
+
+ @retry_check()
+ def check_azure_storage_server_data(self, container_name, test_data):
+ (code, output) = self.container_communicator.execute_command(container_name, ["find", "/data/__blobstorage__", "-type", "f"])
+ if code != 0:
+ return False
+ data_file = output.strip()
+ (code, file_data) = self.container_communicator.execute_command(container_name, ["cat", data_file])
+ return code == 0 and test_data in file_data
+
+ def add_test_blob(self, blob_name, content="", with_snapshot=False):
+ try:
+ self.blob_service_client.create_container("test-container")
+ except ResourceExistsError:
+ logging.debug('test-container already exists')
+
+ blob_client = self.blob_service_client.get_blob_client(container="test-container", blob=blob_name)
+ blob_client.upload_blob(content)
+
+ if with_snapshot:
+ blob_client.create_snapshot()
+
+ def __get_blob_and_snapshot_count(self):
+ container_client = self.blob_service_client.get_container_client("test-container")
+ return len(list(container_client.list_blobs(include=['deleted'])))
+
+ def check_azure_blob_and_snapshot_count(self, blob_and_snapshot_count, timeout_seconds):
+ start_time = time.perf_counter()
+ while True:
+ if self.__get_blob_and_snapshot_count() == blob_and_snapshot_count:
+ return True
+ time.sleep(1)
+ if timeout_seconds < (time.perf_counter() - start_time):
+ break
Review Comment:
Thanks, I think it's a good idea! Although I only prefer to use it in trivial cases, so I would not complicate the utility function with the result transformation. Updated in 5ca3ac0037c3df330cd8cfe2df58c2b0169483b7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org