You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nifi.apache.org by fg...@apache.org on 2023/01/23 18:34:01 UTC
[nifi-minifi-cpp] branch main updated (2173e13e1 -> 06d1d1610)
This is an automated email from the ASF dual-hosted git repository.
fgerlits pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
from 2173e13e1 MINIFICPP-1987 Configuring processor metrics with regular expressions
new 78a5c85b8 MINIFICPP-1965 Add CMAKE flags to select malloc implementation
new 572e7e928 MINIFICPP-2032 Add support for new AWS regions
new 06d1d1610 MINIFICPP-2004 Remove default property values from kubernetes examples
The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CMakeLists.txt | 28 ++--
PROCESSORS.md | 155 +++++++++++----------
cmake/BundledAwsSdkCpp.cmake | 15 +-
cmake/{ExpectedLite.cmake => MiMalloc.cmake} | 13 +-
cmake/MiNiFiOptions.cmake | 9 ++
cmake/{Abseil.cmake => RpMalloc.cmake} | 18 ++-
docker/requirements.txt | 2 +-
.../integration/MiNiFi_integration_test_driver.py | 14 +-
.../features/core_functionality.feature | 10 ++
.../integration/minifi/core/DockerTestCluster.py | 29 +++-
docker/test/integration/minifi/core/utils.py | 24 ++++
docker/test/integration/steps/steps.py | 64 ++++++---
.../daemon-set-log-collection.yml | 6 -
.../sidecar-log-collection.yml | 6 -
extensions/aws/processors/S3Processor.h | 5 +
.../processors/S3ProcessorStaticDefinitions.cpp | 7 +-
extensions/aws/s3/S3ClientRequestSender.cpp | 28 ++--
extensions/aws/s3/S3ClientRequestSender.h | 14 +-
extensions/aws/s3/S3Wrapper.h | 6 +-
libminifi/test/aws-tests/PutS3ObjectTests.cpp | 4 +-
minifi_main/CMakeLists.txt | 8 +-
thirdparty/aws-sdk-cpp/bundle-openssl.patch | 46 ++++++
.../aws-sdk-cpp/c++20-compilation-fixes.patch | 27 +---
thirdparty/aws-sdk-cpp/dll-export-injection.patch | 34 +++--
thirdparty/aws-sdk-cpp/shutdown-fix.patch | 9 +-
25 files changed, 357 insertions(+), 224 deletions(-)
copy cmake/{ExpectedLite.cmake => MiMalloc.cmake} (75%)
copy cmake/{Abseil.cmake => RpMalloc.cmake} (61%)
create mode 100644 thirdparty/aws-sdk-cpp/bundle-openssl.patch
[nifi-minifi-cpp] 03/03: MINIFICPP-2004 Remove default property values from kubernetes examples
Posted by fg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fgerlits pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 06d1d161018eef80cc1a5f9cc96bc043646156d6
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Mon Jan 16 16:18:43 2023 +0100
MINIFICPP-2004 Remove default property values from kubernetes examples
Signed-off-by: Ferenc Gerlits <fg...@gmail.com>
This closes #1493
---
.../daemon-set-log-collection/daemon-set-log-collection.yml | 6 ------
.../kubernetes/sidecar-log-collection/sidecar-log-collection.yml | 6 ------
2 files changed, 12 deletions(-)
diff --git a/examples/kubernetes/daemon-set-log-collection/daemon-set-log-collection.yml b/examples/kubernetes/daemon-set-log-collection/daemon-set-log-collection.yml
index 01e2afa84..4fa6d9079 100644
--- a/examples/kubernetes/daemon-set-log-collection/daemon-set-log-collection.yml
+++ b/examples/kubernetes/daemon-set-log-collection/daemon-set-log-collection.yml
@@ -21,17 +21,11 @@ data:
logger.root=INFO,stderr
logger.org::apache::nifi::minifi=INFO,stderr
minifi.properties: |
- nifi.flow.configuration.file=./conf/config.yml
nifi.administrative.yield.duration=30 sec
nifi.bored.yield.duration=100 millis
- nifi.extension.path=../extensions/*
- nifi.provenance.repository.directory.default=${MINIFI_HOME}/provenance_repository
nifi.provenance.repository.max.storage.time=1 MIN
nifi.provenance.repository.max.storage.size=1 MB
- nifi.flowfile.repository.directory.default=${MINIFI_HOME}/flowfile_repository
- nifi.flowfile.checkpoint.directory.default=${MINIFI_HOME}/flowfile_checkpoint
- nifi.database.content.repository.directory.default=${MINIFI_HOME}/content_repository
nifi.provenance.repository.class.name=NoOpRepository
nifi.content.repository.class.name=DatabaseContentRepository
config.yml: |
diff --git a/examples/kubernetes/sidecar-log-collection/sidecar-log-collection.yml b/examples/kubernetes/sidecar-log-collection/sidecar-log-collection.yml
index 6574b896f..b831fe474 100644
--- a/examples/kubernetes/sidecar-log-collection/sidecar-log-collection.yml
+++ b/examples/kubernetes/sidecar-log-collection/sidecar-log-collection.yml
@@ -23,17 +23,11 @@ data:
logger.root=INFO,stderr
logger.org::apache::nifi::minifi=INFO,stderr
minifi.properties: |
- nifi.flow.configuration.file=./conf/config.yml
nifi.administrative.yield.duration=30 sec
nifi.bored.yield.duration=100 millis
- nifi.extension.path=../extensions/*
- nifi.provenance.repository.directory.default=${MINIFI_HOME}/provenance_repository
nifi.provenance.repository.max.storage.time=1 MIN
nifi.provenance.repository.max.storage.size=1 MB
- nifi.flowfile.repository.directory.default=${MINIFI_HOME}/flowfile_repository
- nifi.flowfile.checkpoint.directory.default=${MINIFI_HOME}/flowfile_checkpoint
- nifi.database.content.repository.directory.default=${MINIFI_HOME}/content_repository
nifi.provenance.repository.class.name=NoOpRepository
nifi.content.repository.class.name=DatabaseContentRepository
config.yml: |
[nifi-minifi-cpp] 01/03: MINIFICPP-1965 Add CMAKE flags to select malloc implementation
Posted by fg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fgerlits pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 78a5c85b8499b055489f9e5c4ab41101b72ed293
Author: Martin Zink <ma...@apache.org>
AuthorDate: Sun Oct 23 21:13:38 2022 +0200
MINIFICPP-1965 Add CMAKE flags to select malloc implementation
Signed-off-by: Ferenc Gerlits <fg...@gmail.com>
This closes #1456
---
CMakeLists.txt | 24 ++++++--
cmake/MiMalloc.cmake | 25 +++++++++
cmake/MiNiFiOptions.cmake | 9 +++
cmake/RpMalloc.cmake | 32 +++++++++++
docker/requirements.txt | 2 +-
.../integration/MiNiFi_integration_test_driver.py | 14 ++++-
.../features/core_functionality.feature | 10 ++++
.../integration/minifi/core/DockerTestCluster.py | 29 +++++++++-
docker/test/integration/minifi/core/utils.py | 24 ++++++++
docker/test/integration/steps/steps.py | 64 ++++++++++++++--------
minifi_main/CMakeLists.txt | 5 +-
11 files changed, 204 insertions(+), 34 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index 31d1a3d7c..b7166f2c8 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -200,14 +200,26 @@ set(PASSTHROUGH_CMAKE_ARGS -DANDROID_ABI=${ANDROID_ABI}
-G${CMAKE_GENERATOR}
)
-# jemalloc
+if(CUSTOM_MALLOC)
+ if (CUSTOM_MALLOC STREQUAL jemalloc)
+ include(BundledJemalloc)
+ use_bundled_jemalloc(${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR})
+ set(CUSTOM_MALLOC_LIB JeMalloc::JeMalloc)
+ elseif (CUSTOM_MALLOC STREQUAL mimalloc)
+ include(MiMalloc)
+ set(CUSTOM_MALLOC_LIB mimalloc)
+ elseif (CUSTOM_MALLOC STREQUAL rpmalloc)
+ include(RpMalloc)
+ set(CUSTOM_MALLOC_LIB rpmalloc)
+ else()
+ message(FATAL_ERROR "Invalid CUSTOM_MALLOC")
+ endif()
+else()
+ message(VERBOSE "No custom malloc implementation")
+endif()
+
if(NOT WIN32)
if (ENABLE_JNI)
- if (NOT DISABLE_JEMALLOC)
- include(BundledJemalloc)
- use_bundled_jemalloc(${CMAKE_CURRENT_SOURCE_DIR} ${CMAKE_CURRENT_BINARY_DIR})
- message("jemalloc found at ${JEMALLOC_LIBRARIES}")
- endif()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DENABLE_JNI")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DENABLE_JNI")
endif()
diff --git a/cmake/MiMalloc.cmake b/cmake/MiMalloc.cmake
new file mode 100644
index 000000000..ec12daa0d
--- /dev/null
+++ b/cmake/MiMalloc.cmake
@@ -0,0 +1,25 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+include(FetchContent)
+FetchContent_Declare(
+ mimalloc
+ URL https://github.com/microsoft/mimalloc/archive/refs/tags/v2.0.6.tar.gz
+ URL_HASH SHA256=9f05c94cc2b017ed13698834ac2a3567b6339a8bde27640df5a1581d49d05ce5
+)
+FetchContent_MakeAvailable(mimalloc)
diff --git a/cmake/MiNiFiOptions.cmake b/cmake/MiNiFiOptions.cmake
index cfca8b213..b04e3157f 100644
--- a/cmake/MiNiFiOptions.cmake
+++ b/cmake/MiNiFiOptions.cmake
@@ -28,6 +28,12 @@ function(add_minifi_dependent_option OPTION_NAME OPTION_DESCRIPTION OPTION_VALUE
set(MINIFI_OPTIONS ${MINIFI_OPTIONS} PARENT_SCOPE)
endfunction()
+function(set_minifi_cache_variable VARIABLE_NAME VARIABLE_VALUE DOCSTRING)
+ set(${VARIABLE_NAME} ${VARIABLE_VALUE} CACHE STRING ${DOCSTRING})
+ list(APPEND MINIFI_OPTIONS ${VARIABLE_NAME})
+ set(MINIFI_OPTIONS ${MINIFI_OPTIONS} PARENT_SCOPE)
+endfunction()
+
add_minifi_option(CI_BUILD "Build is used for CI." OFF)
add_minifi_option(SKIP_TESTS "Skips building all tests." OFF)
add_minifi_option(DOCKER_BUILD_ONLY "Disables all targets except docker build scripts. Ideal for systems without an up-to-date compiler." OFF)
@@ -117,6 +123,9 @@ add_minifi_option(ENABLE_TEST_PROCESSORS "Enables test processors" OFF)
add_minifi_option(ENABLE_PROMETHEUS "Enables Prometheus support." OFF)
add_minifi_option(DISABLE_JEMALLOC "Disables jemalloc." OFF)
+set_minifi_cache_variable(CUSTOM_MALLOC OFF "Overwrite malloc implementation.")
+set_property(CACHE CUSTOM_MALLOC PROPERTY STRINGS "jemalloc" "mimalloc" "rpmalloc" OFF)
+
if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
add_minifi_option(ENABLE_PROCFS "Enables the procfs extension." ON)
endif()
diff --git a/cmake/RpMalloc.cmake b/cmake/RpMalloc.cmake
new file mode 100644
index 000000000..17ab3166e
--- /dev/null
+++ b/cmake/RpMalloc.cmake
@@ -0,0 +1,32 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+include(FetchContent)
+FetchContent_Declare(
+ rpmalloc
+ URL https://github.com/mjansson/rpmalloc/archive/refs/tags/1.4.4.tar.gz
+ URL_HASH SHA256=3859620c03e6473f0b3f16a4e965e7c049594253f70e8370fb9caa0e4118accb
+)
+FetchContent_GetProperties(rpmalloc)
+
+if(NOT rpmalloc_POPULATED)
+ FetchContent_Populate(rpmalloc)
+ add_library(rpmalloc ${rpmalloc_SOURCE_DIR}/rpmalloc/rpmalloc.c)
+ target_include_directories(rpmalloc PUBLIC ${rpmalloc_SOURCE_DIR}/rpmalloc)
+ target_compile_definitions(rpmalloc PRIVATE ENABLE_OVERRIDE=1 ENABLE_PRELOAD=1)
+endif()
diff --git a/docker/requirements.txt b/docker/requirements.txt
index 2dda64d12..8efcc2861 100644
--- a/docker/requirements.txt
+++ b/docker/requirements.txt
@@ -1,5 +1,4 @@
behave==1.2.6
-pytimeparse==1.1.8
docker==5.0.0
kafka-python==2.0.2
confluent-kafka==1.7.0
@@ -9,3 +8,4 @@ watchdog==2.1.2
pyopenssl==23.0.0
azure-storage-blob==12.9.0
prometheus-api-client==0.5.0
+humanfriendly==10.0
diff --git a/docker/test/integration/MiNiFi_integration_test_driver.py b/docker/test/integration/MiNiFi_integration_test_driver.py
index 328a7b60b..f16a29d32 100644
--- a/docker/test/integration/MiNiFi_integration_test_driver.py
+++ b/docker/test/integration/MiNiFi_integration_test_driver.py
@@ -33,7 +33,7 @@ from minifi.validators.NoContentCheckFileNumberValidator import NoContentCheckFi
from minifi.validators.NumFileRangeValidator import NumFileRangeValidator
from minifi.validators.SingleJSONFileOutputValidator import SingleJSONFileOutputValidator
-from minifi.core.utils import decode_escaped_str
+from minifi.core.utils import decode_escaped_str, get_minifi_pid, get_peak_memory_usage
class MiNiFi_integration_test:
@@ -311,3 +311,15 @@ class MiNiFi_integration_test:
def check_processor_metric_on_prometheus(self, metric_class, timeout_seconds, processor_name):
assert self.cluster.wait_for_processor_metric_on_prometheus(metric_class, timeout_seconds, processor_name)
+
+ def check_if_peak_memory_usage_exceeded(self, minimum_peak_memory_usage: int, timeout_seconds: int) -> None:
+ assert self.cluster.wait_for_peak_memory_usage_to_exceed(minimum_peak_memory_usage, timeout_seconds)
+
+ def check_if_memory_usage_is_below(self, maximum_memory_usage: int, timeout_seconds: int) -> None:
+ assert self.cluster.wait_for_memory_usage_to_drop_below(maximum_memory_usage, timeout_seconds)
+
+ def check_memory_usage_compared_to_peak(self, peak_multiplier: float, timeout_seconds: int) -> None:
+ peak_memory = get_peak_memory_usage(get_minifi_pid())
+ assert (peak_memory is not None)
+ assert (1.0 > peak_multiplier > 0.0)
+ assert self.cluster.wait_for_memory_usage_to_drop_below(peak_memory * peak_multiplier, timeout_seconds)
diff --git a/docker/test/integration/features/core_functionality.feature b/docker/test/integration/features/core_functionality.feature
index 3f3330e99..ce5ce667f 100644
--- a/docker/test/integration/features/core_functionality.feature
+++ b/docker/test/integration/features/core_functionality.feature
@@ -40,3 +40,13 @@ Feature: Core flow functionalities
Given a GenerateFlowFile processor with the name "generateFlowFile" in the "minifi-cpp-with-provenance-repo" flow with engine "minifi-cpp-with-provenance-repo"
When the MiNiFi instance starts up
Then the "minifi-cpp-with-provenance-repo" flow has a log line matching "MiNiFi started" in less than 30 seconds
+
+ Scenario: Memory usage returns after peak usage
+ Given a GenerateFlowFile processor with the "Batch Size" property set to "50000"
+ And the "Data Format" property of the GenerateFlowFile processor is set to "Text"
+ And the scheduling period of the GenerateFlowFile processor is set to "1 hours"
+ And the "Unique FlowFiles" property of the GenerateFlowFile processor is set to "false"
+ And the "Custom Text" property of the GenerateFlowFile processor is set to "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur tellus quam, sagittis quis ante ac, finibus ornare lectus. Morbi libero mauris, mollis sed mi at."
+ When all instances start up
+ Then the peak memory usage of the agent is more than 130 MB in less than 20 seconds
+ And the memory usage of the agent decreases to 70% peak usage in less than 20 seconds
diff --git a/docker/test/integration/minifi/core/DockerTestCluster.py b/docker/test/integration/minifi/core/DockerTestCluster.py
index e89895dd0..0d03f8037 100644
--- a/docker/test/integration/minifi/core/DockerTestCluster.py
+++ b/docker/test/integration/minifi/core/DockerTestCluster.py
@@ -25,7 +25,7 @@ import tempfile
from .LogSource import LogSource
from .SingleNodeDockerCluster import SingleNodeDockerCluster
from .PrometheusChecker import PrometheusChecker
-from .utils import retry_check
+from .utils import retry_check, get_peak_memory_usage, get_minifi_pid, get_memory_usage
from azure.storage.blob import BlobServiceClient
from azure.core.exceptions import ResourceExistsError
@@ -318,3 +318,30 @@ class DockerTestCluster(SingleNodeDockerCluster):
def wait_for_processor_metric_on_prometheus(self, metric_class, timeout_seconds, processor_name):
return PrometheusChecker().wait_for_processor_metric_on_prometheus(metric_class, timeout_seconds, processor_name)
+
+ def wait_for_peak_memory_usage_to_exceed(self, minimum_peak_memory_usage: int, timeout_seconds: int) -> bool:
+ start_time = time.perf_counter()
+ while (time.perf_counter() - start_time) < timeout_seconds:
+ current_peak_memory_usage = get_peak_memory_usage(get_minifi_pid())
+ if current_peak_memory_usage is None:
+ logging.warning("Failed to determine peak memory usage")
+ return False
+ if current_peak_memory_usage > minimum_peak_memory_usage:
+ return True
+ time.sleep(1)
+ logging.warning(f"Peak memory usage ({current_peak_memory_usage}) didnt exceed minimum asserted peak memory usage {minimum_peak_memory_usage}")
+ return False
+
+ def wait_for_memory_usage_to_drop_below(self, max_memory_usage: int, timeout_seconds: int) -> bool:
+ start_time = time.perf_counter()
+ while (time.perf_counter() - start_time) < timeout_seconds:
+ current_memory_usage = get_memory_usage(get_minifi_pid())
+ if current_memory_usage is None:
+ logging.warning("Failed to determine memory usage")
+ return False
+ if current_memory_usage < max_memory_usage:
+ return True
+ current_memory_usage = get_memory_usage(get_minifi_pid())
+ time.sleep(1)
+ logging.warning(f"Memory usage ({current_memory_usage}) is more than the maximum asserted memory usage ({max_memory_usage})")
+ return False
diff --git a/docker/test/integration/minifi/core/utils.py b/docker/test/integration/minifi/core/utils.py
index f559a2b18..c0a5d34c1 100644
--- a/docker/test/integration/minifi/core/utils.py
+++ b/docker/test/integration/minifi/core/utils.py
@@ -17,6 +17,8 @@
import time
import functools
import os
+import subprocess
+from typing import Optional
def retry_check(max_tries=5, retry_interval=1):
@@ -57,3 +59,25 @@ def decode_escaped_str(str):
def is_temporary_output_file(filepath):
return filepath.split(os.path.sep)[-1][0] == '.'
+
+
+def get_minifi_pid() -> int:
+ return int(subprocess.run(["pidof", "-s", "minifi"], capture_output=True).stdout)
+
+
+def get_peak_memory_usage(pid: int) -> Optional[int]:
+ with open("/proc/" + str(pid) + "/status") as stat_file:
+ for line in stat_file:
+ if "VmHWM" in line:
+ peak_resident_set_size = [int(s) for s in line.split() if s.isdigit()].pop()
+ return peak_resident_set_size * 1024
+ return None
+
+
+def get_memory_usage(pid: int) -> Optional[int]:
+ with open("/proc/" + str(pid) + "/status") as stat_file:
+ for line in stat_file:
+ if "VmRSS" in line:
+ resident_set_size = [int(s) for s in line.split() if s.isdigit()].pop()
+ return resident_set_size * 1024
+ return None
diff --git a/docker/test/integration/steps/steps.py b/docker/test/integration/steps/steps.py
index 6ae9b5904..52b80bcb0 100644
--- a/docker/test/integration/steps/steps.py
+++ b/docker/test/integration/steps/steps.py
@@ -28,12 +28,12 @@ from minifi.controllers.KubernetesControllerService import KubernetesControllerS
from behave import given, then, when
from behave.model_describe import ModelDescriptor
from pydoc import locate
-from pytimeparse.timeparse import timeparse
import logging
import time
import uuid
import binascii
+import humanfriendly
from kafka import KafkaProducer
from confluent_kafka.admin import AdminClient, NewTopic
@@ -67,8 +67,10 @@ def step_impl(context, processor_type, processor_name, property_name, property_v
__create_processor(context, processor_type, processor_name, property_name, property_value, minifi_container_name)
-@given("a {processor_type} processor with the name \"{processor_name}\" and the \"{property_name}\" property set to \"{property_value}\" in a \"{minifi_container_name}\" flow with engine \"{engine_name}\"")
-@given("a {processor_type} processor with the name \"{processor_name}\" and the \"{property_name}\" property set to \"{property_value}\" in the \"{minifi_container_name}\" flow with engine \"{engine_name}\"")
+@given(
+ "a {processor_type} processor with the name \"{processor_name}\" and the \"{property_name}\" property set to \"{property_value}\" in a \"{minifi_container_name}\" flow with engine \"{engine_name}\"")
+@given(
+ "a {processor_type} processor with the name \"{processor_name}\" and the \"{property_name}\" property set to \"{property_value}\" in the \"{minifi_container_name}\" flow with engine \"{engine_name}\"")
def step_impl(context, processor_type, processor_name, property_name, property_value, minifi_container_name, engine_name):
__create_processor(context, processor_type, processor_name, property_name, property_value, minifi_container_name, engine_name)
@@ -195,10 +197,10 @@ def step_impl(context, processor_name, max_concurrent_tasks):
@given("the \"{property_name}\" property of the {processor_name} processor is set to match {key_attribute_encoding} encoded kafka message key \"{message_key}\"")
def step_impl(context, property_name, processor_name, key_attribute_encoding, message_key):
encoded_key = ""
- if(key_attribute_encoding.lower() == "hex"):
+ if (key_attribute_encoding.lower() == "hex"):
# Hex is presented upper-case to be in sync with NiFi
encoded_key = binascii.hexlify(message_key.encode("utf-8")).upper()
- elif(key_attribute_encoding.lower() == "(not set)"):
+ elif (key_attribute_encoding.lower() == "(not set)"):
encoded_key = message_key.encode("utf-8")
else:
encoded_key = message_key.encode(key_attribute_encoding)
@@ -725,19 +727,19 @@ def step_impl(context):
@then("a flowfile with the content '{content}' is placed in the monitored directory in less than {duration}")
@then("{number_of_flow_files:d} flowfiles with the content \"{content}\" are placed in the monitored directory in less than {duration}")
def step_impl(context, content, duration, number_of_flow_files=1):
- context.test.check_for_multiple_files_generated(number_of_flow_files, timeparse(duration), [content])
+ context.test.check_for_multiple_files_generated(number_of_flow_files, humanfriendly.parse_timespan(duration), [content])
@then("a flowfile with the JSON content \"{content}\" is placed in the monitored directory in less than {duration}")
@then("a flowfile with the JSON content '{content}' is placed in the monitored directory in less than {duration}")
def step_impl(context, content, duration):
- context.test.check_for_single_json_file_with_content_generated(content, timeparse(duration))
+ context.test.check_for_single_json_file_with_content_generated(content, humanfriendly.parse_timespan(duration))
@then("at least one flowfile with the content \"{content}\" is placed in the monitored directory in less than {duration}")
@then("at least one flowfile with the content '{content}' is placed in the monitored directory in less than {duration}")
def step_impl(context, content, duration):
- context.test.check_for_at_least_one_file_with_content_generated(content, timeparse(duration))
+ context.test.check_for_at_least_one_file_with_content_generated(content, humanfriendly.parse_timespan(duration))
@then("{num_flowfiles} flowfiles are placed in the monitored directory in less than {duration}")
@@ -745,49 +747,49 @@ def step_impl(context, num_flowfiles, duration):
if num_flowfiles == 0:
context.execute_steps("""no files are placed in the monitored directory in {duration} of running time""".format(duration=duration))
return
- context.test.check_for_num_files_generated(int(num_flowfiles), timeparse(duration))
+ context.test.check_for_num_files_generated(int(num_flowfiles), humanfriendly.parse_timespan(duration))
@then("at least one flowfile is placed in the monitored directory in less than {duration}")
def step_impl(context, duration):
- context.test.check_for_num_file_range_generated(1, float('inf'), timeparse(duration))
+ context.test.check_for_num_file_range_generated(1, float('inf'), humanfriendly.parse_timespan(duration))
@then("one flowfile with the contents \"{content}\" is placed in the monitored directory in less than {duration}")
def step_impl(context, content, duration):
- context.test.check_for_multiple_files_generated(1, timeparse(duration), [content])
+ context.test.check_for_multiple_files_generated(1, humanfriendly.parse_timespan(duration), [content])
@then("two flowfiles with the contents \"{content_1}\" and \"{content_2}\" are placed in the monitored directory in less than {duration}")
def step_impl(context, content_1, content_2, duration):
- context.test.check_for_multiple_files_generated(2, timeparse(duration), [content_1, content_2])
+ context.test.check_for_multiple_files_generated(2, humanfriendly.parse_timespan(duration), [content_1, content_2])
@then("flowfiles with these contents are placed in the monitored directory in less than {duration}: \"{contents}\"")
def step_impl(context, duration, contents):
contents_arr = contents.split(",")
- context.test.check_for_multiple_files_generated(0, timeparse(duration), contents_arr)
+ context.test.check_for_multiple_files_generated(0, humanfriendly.parse_timespan(duration), contents_arr)
@then("after a wait of {duration}, at least {lower_bound:d} and at most {upper_bound:d} flowfiles are produced and placed in the monitored directory")
def step_impl(context, lower_bound, upper_bound, duration):
- context.test.check_for_num_file_range_generated(lower_bound, upper_bound, timeparse(duration))
+ context.test.check_for_num_file_range_generated(lower_bound, upper_bound, humanfriendly.parse_timespan(duration))
@then("{number_of_files:d} flowfiles are placed in the monitored directory in {duration}")
@then("{number_of_files:d} flowfile is placed in the monitored directory in {duration}")
def step_impl(context, number_of_files, duration):
- context.test.check_for_multiple_files_generated(number_of_files, timeparse(duration))
+ context.test.check_for_multiple_files_generated(number_of_files, humanfriendly.parse_timespan(duration))
@then("at least one empty flowfile is placed in the monitored directory in less than {duration}")
def step_impl(context, duration):
- context.test.check_for_an_empty_file_generated(timeparse(duration))
+ context.test.check_for_an_empty_file_generated(humanfriendly.parse_timespan(duration))
@then("no files are placed in the monitored directory in {duration} of running time")
def step_impl(context, duration):
- context.test.check_for_no_files_generated(timeparse(duration))
+ context.test.check_for_no_files_generated(humanfriendly.parse_timespan(duration))
@then("no errors were generated on the http-proxy regarding \"{url}\"")
@@ -854,7 +856,7 @@ def step_impl(context, query, number_of_rows, timeout_seconds):
@then("the Minifi logs contain the following message: \"{log_message}\" in less than {duration}")
def step_impl(context, log_message, duration):
- context.test.check_minifi_log_contents(log_message, timeparse(duration))
+ context.test.check_minifi_log_contents(log_message, humanfriendly.parse_timespan(duration))
@then("the Minifi logs contain the following message: \"{log_message}\" {count:d} times after {seconds:d} seconds")
@@ -870,12 +872,12 @@ def step_impl(context, log_message, seconds):
@then("the Minifi logs match the following regex: \"{regex}\" in less than {duration}")
def step_impl(context, regex, duration):
- context.test.check_minifi_log_matches_regex(regex, timeparse(duration))
+ context.test.check_minifi_log_matches_regex(regex, humanfriendly.parse_timespan(duration))
@then("the OPC UA server logs contain the following message: \"{log_message}\" in less than {duration}")
def step_impl(context, log_message, duration):
- context.test.check_container_log_contents("opcua-server", log_message, timeparse(duration))
+ context.test.check_container_log_contents("opcua-server", log_message, humanfriendly.parse_timespan(duration))
# MQTT
@@ -891,7 +893,7 @@ def step_impl(context, log_count, log_pattern):
@then("the \"{minifi_container_name}\" flow has a log line matching \"{log_pattern}\" in less than {duration}")
def step_impl(context, minifi_container_name, log_pattern, duration):
- context.test.check_container_log_matches_regex(minifi_container_name, log_pattern, timeparse(duration), count=1)
+ context.test.check_container_log_matches_regex(minifi_container_name, log_pattern, humanfriendly.parse_timespan(duration), count=1)
@then("an MQTT broker is deployed in correspondence with the PublishMQTT")
@@ -981,14 +983,30 @@ def step_impl(context):
@then("the MiNiFi C2 server logs contain the following message: \"{log_message}\" in less than {duration}")
def step_impl(context, log_message, duration):
- context.test.check_container_log_contents("minifi-c2-server", log_message, timeparse(duration))
+ context.test.check_container_log_contents("minifi-c2-server", log_message, humanfriendly.parse_timespan(duration))
@then("the MiNiFi C2 SSL server logs contain the following message: \"{log_message}\" in less than {duration}")
def step_impl(context, log_message, duration):
- context.test.check_container_log_contents("minifi-c2-server-ssl", log_message, timeparse(duration))
+ context.test.check_container_log_contents("minifi-c2-server-ssl", log_message, humanfriendly.parse_timespan(duration))
@given(u'a MiNiFi C2 server is set up with SSL')
def step_impl(context):
context.test.acquire_container("minifi-c2-server", "minifi-c2-server-ssl")
+
+
+# MiNiFi memory usage
+@then(u'the peak memory usage of the agent is more than {size} in less than {duration}')
+def step_impl(context, size: str, duration: str) -> None:
+ context.test.check_if_peak_memory_usage_exceeded(humanfriendly.parse_size(size), humanfriendly.parse_timespan(duration))
+
+
+@then(u'the memory usage of the agent is less than {size} in less than {duration}')
+def step_impl(context, size: str, duration: str) -> None:
+ context.test.check_if_memory_usage_is_below(humanfriendly.parse_size(size), humanfriendly.parse_timespan(duration))
+
+
+@then(u'the memory usage of the agent decreases to {peak_usage_percent}% peak usage in less than {duration}')
+def step_impl(context, peak_usage_percent: str, duration: str) -> None:
+ context.test.check_memory_usage_compared_to_peak(float(peak_usage_percent) * 0.01, humanfriendly.parse_timespan(duration))
diff --git a/minifi_main/CMakeLists.txt b/minifi_main/CMakeLists.txt
index ea7c9e381..beb2be852 100644
--- a/minifi_main/CMakeLists.txt
+++ b/minifi_main/CMakeLists.txt
@@ -52,8 +52,9 @@ endif(NOT USE_SHARED_LIBS)
target_link_libraries(minifiexe Threads::Threads)
target_link_libraries(minifiexe yaml-cpp)
-if(NOT WIN32 AND ENABLE_JNI AND NOT DISABLE_JEMALLOC)
- target_link_libraries(minifiexe JeMalloc::JeMalloc)
+if(CUSTOM_MALLOC_LIB)
+ message(VERBOSE "Using custom malloc lib ${CUSTOM_MALLOC_LIB} for minifiexe")
+ target_link_libraries(minifiexe ${CUSTOM_MALLOC_LIB})
endif()
if (WIN32)
[nifi-minifi-cpp] 02/03: MINIFICPP-2032 Add support for new AWS regions
Posted by fg...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
fgerlits pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 572e7e928c3dbad075d00c185f1d3e5c808b305d
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Wed Jan 11 09:19:34 2023 +0100
MINIFICPP-2032 Add support for new AWS regions
Signed-off-by: Ferenc Gerlits <fg...@gmail.com>
This closes #1491
---
CMakeLists.txt | 4 -
PROCESSORS.md | 155 +++++++++++----------
cmake/BundledAwsSdkCpp.cmake | 15 +-
extensions/aws/processors/S3Processor.h | 5 +
.../processors/S3ProcessorStaticDefinitions.cpp | 7 +-
extensions/aws/s3/S3ClientRequestSender.cpp | 28 ++--
extensions/aws/s3/S3ClientRequestSender.h | 14 +-
extensions/aws/s3/S3Wrapper.h | 6 +-
libminifi/test/aws-tests/PutS3ObjectTests.cpp | 4 +-
minifi_main/CMakeLists.txt | 3 +
thirdparty/aws-sdk-cpp/bundle-openssl.patch | 46 ++++++
.../aws-sdk-cpp/c++20-compilation-fixes.patch | 27 +---
thirdparty/aws-sdk-cpp/dll-export-injection.patch | 34 +++--
thirdparty/aws-sdk-cpp/shutdown-fix.patch | 9 +-
14 files changed, 192 insertions(+), 165 deletions(-)
diff --git a/CMakeLists.txt b/CMakeLists.txt
index b7166f2c8..35cf83db7 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -109,10 +109,6 @@ if (WIN32)
add_compile_options(/W3 /utf-8)
endif()
-if (WIN32)
- add_definitions(-DSERVICE_NAME="Apache NiFi MINiFi")
-endif()
-
if (NOT PORTABLE)
if(MSVC)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} /arch:AVX2")
diff --git a/PROCESSORS.md b/PROCESSORS.md
index c8cee2b17..31e0de24d 100644
--- a/PROCESSORS.md
+++ b/PROCESSORS.md
@@ -524,22 +524,22 @@ Deletes FlowFiles on an Amazon S3 Bucket. If attempting to delete a file that do
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | <div style="width:7em">Default Value</div> | <div style="width:8em">Allowable Values</div> | Description [...]
-|----------------------------------|--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------- [...]
-| Object Key | | | The key of t [...]
-| **Bucket** | | | The S3 bucke [...]
-| Access Key | | | AWS account [...]
-| Secret Key | | | AWS account [...]
-| Credentials File | | | Path to a fi [...]
-| AWS Credentials Provider service | | | The name of [...]
-| **Region** | us-west-2 | af-south-1<br/>ap-east-1<br/>ap-northeast-1<br/>ap-northeast-2<br/>ap-northeast-3<br/>ap-south-1<br/>ap-southeast-1<br/>ap-southeast-2<br/>ca-central-1<br/>cn-north-1<br/>cn-northwest-1<br/>eu-central-1<br/>eu-north-1<br/>eu-south-1<br/>eu-west-1<br/>eu-west-2<br/>eu-west-3<br/>me-south-1<br/>sa-east-1<br/>us-east-1<br/>us-east-2<br/>us-gov-east-1<br/>us-gov-west-1<br/>us-west-1<br/>us-west-2 | AWS Region [...]
-| **Communications Timeout** | 30 sec | | Sets the tim [...]
-| Endpoint Override URL | | | Endpoint URL [...]
-| Proxy Host | | | Proxy host n [...]
-| Proxy Port | | | The port num [...]
-| Proxy Username | | | Username to [...]
-| Proxy Password | | | Password to [...]
-| Version | | | The Version [...]
+| Name | <div style="width:7em">Default Value</div> | <div style="width:8em">Allowable Values</div> [...]
+|----------------------------------|--------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| Object Key | | [...]
+| **Bucket** | | [...]
+| Access Key | | [...]
+| Secret Key | | [...]
+| Credentials File | | [...]
+| AWS Credentials Provider service | | [...]
+| **Region** | us-west-2 | af-south-1<br/>ap-east-1<br/>ap-northeast-1<br/>ap-northeast-2<br/>ap-northeast-3<br/>ap-south-1<br/>ap-southeast-1<br/>ap-southeast-2<br/>ap-southeast-3<br/>ca-central-1<br/>cn-north-1<br/>cn-northwest-1<br/>eu-central-1<br/>eu-north-1<br/>eu-south-1<br/>eu-west-1<br/>eu-west-2<br/>eu-west-3<br/>me-central-1<br/>me-south-1<br/>sa-east-1<br/>us-east-1<br/>us-east-2<br/>us-gov-east-1<br/>us-gov-west-1<br/>us [...]
+| **Communications Timeout** | 30 sec | [...]
+| Endpoint Override URL | | [...]
+| Proxy Host | | [...]
+| Proxy Port | | [...]
+| Proxy Username | | [...]
+| Proxy Password | | [...]
+| Version | | [...]
### Relationships
| Name | Description |
@@ -829,23 +829,23 @@ Retrieves the contents of an S3 Object and writes it to the content of a FlowFil
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | <div style="width:7em">Default Value</div> | <div style="width:8em">Allowable Values</div> | Description [...]
-|----------------------------------|--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------- [...]
-| Object Key | | | The key of t [...]
-| **Bucket** | | | The S3 bucke [...]
-| Access Key | | | AWS account [...]
-| Secret Key | | | AWS account [...]
-| Credentials File | | | Path to a fi [...]
-| AWS Credentials Provider service | | | The name of [...]
-| **Region** | us-west-2 | af-south-1<br/>ap-east-1<br/>ap-northeast-1<br/>ap-northeast-2<br/>ap-northeast-3<br/>ap-south-1<br/>ap-southeast-1<br/>ap-southeast-2<br/>ca-central-1<br/>cn-north-1<br/>cn-northwest-1<br/>eu-central-1<br/>eu-north-1<br/>eu-south-1<br/>eu-west-1<br/>eu-west-2<br/>eu-west-3<br/>me-south-1<br/>sa-east-1<br/>us-east-1<br/>us-east-2<br/>us-gov-east-1<br/>us-gov-west-1<br/>us-west-1<br/>us-west-2 | AWS Region [...]
-| **Communications Timeout** | 30 sec | | Sets the tim [...]
-| Endpoint Override URL | | | Endpoint URL [...]
-| Proxy Host | | | Proxy host n [...]
-| Proxy Port | | | The port num [...]
-| Proxy Username | | | Username to [...]
-| Proxy Password | | | Password to [...]
-| Version | | | The Version [...]
-| **Requester Pays** | false | | If true, ind [...]
+| Name | <div style="width:7em">Default Value</div> | <div style="width:8em">Allowable Values</div> [...]
+|----------------------------------|--------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| Object Key | | [...]
+| **Bucket** | | [...]
+| Access Key | | [...]
+| Secret Key | | [...]
+| Credentials File | | [...]
+| AWS Credentials Provider service | | [...]
+| **Region** | us-west-2 | af-south-1<br/>ap-east-1<br/>ap-northeast-1<br/>ap-northeast-2<br/>ap-northeast-3<br/>ap-south-1<br/>ap-southeast-1<br/>ap-southeast-2<br/>ap-southeast-3<br/>ca-central-1<br/>cn-north-1<br/>cn-northwest-1<br/>eu-central-1<br/>eu-north-1<br/>eu-south-1<br/>eu-west-1<br/>eu-west-2<br/>eu-west-3<br/>me-central-1<br/>me-south-1<br/>sa-east-1<br/>us-east-1<br/>us-east-2<br/>us-gov-east-1<br/>us-gov-west-1<br/>us [...]
+| **Communications Timeout** | 30 sec | [...]
+| Endpoint Override URL | | [...]
+| Proxy Host | | [...]
+| Proxy Port | | [...]
+| Proxy Username | | [...]
+| Proxy Password | | [...]
+| Version | | [...]
+| **Requester Pays** | false | [...]
### Relationships
| Name | Description |
@@ -1419,27 +1419,27 @@ Retrieves a listing of objects from an S3 bucket. For each object that is listed
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | Default Value | Allowable Values | Description [...]
-|----------------------------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------ [...]
-| **Bucket** | | | The S3 bucket<br/>**Supports Expression L [...]
-| Access Key | | | AWS account access key<br/>**Supports Exp [...]
-| Secret Key | | | AWS account secret key<br/>**Supports Exp [...]
-| Credentials File | | | Path to a file containing AWS access key [...]
-| AWS Credentials Provider service | | | The name of the AWS Credentials Provider [...]
-| **Region** | us-west-2 | af-south-1<br/>ap-east-1<br/>ap-northeast-1<br/>ap-northeast-2<br/>ap-northeast-3<br/>ap-south-1<br/>ap-southeast-1<br/>ap-southeast-2<br/>ca-central-1<br/>cn-north-1<br/>cn-northwest-1<br/>eu-central-1<br/>eu-north-1<br/>eu-south-1<br/>eu-west-1<br/>eu-west-2<br/>eu-west-3<br/>me-south-1<br/>sa-east-1<br/>us-east-1<br/>us-east-2<br/>us-gov-east-1<br/>us-gov-west-1<br/>us-west-1<br/>us-west-2 | AWS Region [...]
-| **Communications Timeout** | 30 sec | | Sets the timeout of the communication bet [...]
-| Endpoint Override URL | | | Endpoint URL to use instead of the AWS de [...]
-| Proxy Host | | | Proxy host name or IP<br/>**Supports Expr [...]
-| Proxy Port | | | The port number of the proxy host<br/>**S [...]
-| Proxy Username | | | Username to set when authenticating again [...]
-| Proxy Password | | | Password to set when authenticating again [...]
-| Delimiter | | | The string used to delimit directories wi [...]
-| Prefix | | | The prefix used to filter the object list [...]
-| **Use Versions** | false | | Specifies whether to use S3 versions, if [...]
-| **Minimum Object Age** | 0 sec | | The minimum age that an S3 object must be [...]
-| **Write Object Tags** | false | | If set to 'true', the tags associated wit [...]
-| **Write User Metadata** | false | | If set to 'true', the user defined metada [...]
-| **Requester Pays** | false | | If true, indicates that the requester con [...]
+| Name | Default Value | Allowable Values [...]
+|----------------------------------|---------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| **Bucket** | | [...]
+| Access Key | | [...]
+| Secret Key | | [...]
+| Credentials File | | [...]
+| AWS Credentials Provider service | | [...]
+| **Region** | us-west-2 | af-south-1<br/>ap-east-1<br/>ap-northeast-1<br/>ap-northeast-2<br/>ap-northeast-3<br/>ap-south-1<br/>ap-southeast-1<br/>ap-southeast-2<br/>ap-southeast-3<br/>ca-central-1<br/>cn-north-1<br/>cn-northwest-1<br/>eu-central-1<br/>eu-north-1<br/>eu-south-1<br/>eu-west-1<br/>eu-west-2<br/>eu-west-3<br/>me-central-1<br/>me-south-1<br/>sa-east-1<br/>us-east-1<br/>us-east-2<br/>us-gov-east-1<br/>us-gov-west-1<br/>us-iso-east-1<br/>us-isob-east- [...]
+| **Communications Timeout** | 30 sec | [...]
+| Endpoint Override URL | | [...]
+| Proxy Host | | [...]
+| Proxy Port | | [...]
+| Proxy Username | | [...]
+| Proxy Password | | [...]
+| Delimiter | | [...]
+| Prefix | | [...]
+| **Use Versions** | false | [...]
+| **Minimum Object Age** | 0 sec | [...]
+| **Write Object Tags** | false | [...]
+| **Write User Metadata** | false | [...]
+| **Requester Pays** | false | [...]
## ListSFTP
@@ -2053,29 +2053,30 @@ Puts FlowFiles to an Amazon S3 Bucket. The upload uses either the PutS3Object me
In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
-| Name | <div style="width:7em">Default Value</div> | <div style="width:8em">Allowable Values</div> | Description [...]
-|----------------------------------|--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------- [...]
-| Object Key | | | The key of t [...]
-| **Bucket** | | | The S3 bucke [...]
-| Content Type | | | Sets the Con [...]
-| **Use Default Credentials** | false | | If true, use [...]
-| Access Key | | | AWS account [...]
-| Secret Key | | | AWS account [...]
-| Credentials File | | | Path to a fi [...]
-| AWS Credentials Provider service | | | The name of [...]
-| **Storage Class** | Standard | Standard<br/>ReducedRedundancy<br/>StandardIA<br/>OnezoneIA<br/>IntelligentTiering<br/>Glacier<br/>DeepArchive | AWS S3 Stora [...]
-| **Region** | us-west-2 | af-south-1<br/>ap-east-1<br/>ap-northeast-1<br/>ap-northeast-2<br/>ap-northeast-3<br/>ap-south-1<br/>ap-southeast-1<br/>ap-southeast-2<br/>ca-central-1<br/>cn-north-1<br/>cn-northwest-1<br/>eu-central-1<br/>eu-north-1<br/>eu-south-1<br/>eu-west-1<br/>eu-west-2<br/>eu-west-3<br/>me-south-1<br/>sa-east-1<br/>us-east-1<br/>us-east-2<br/>us-gov-east-1<br/>us-gov-west-1<br/>us-west-1<br/>us-west-2 | AWS Region [...]
-| **Communications Timeout** | 30 sec | | Sets the tim [...]
-| FullControl User List | | | A comma-sepa [...]
-| Read Permission User List | | | A comma-sepa [...]
-| Read ACL User List | | | A comma-sepa [...]
-| Write ACL User List | | | A comma-sepa [...]
-| Endpoint Override URL | | | Endpoint URL [...]
-| **Server Side Encryption** | None | None<br/>AES256<br/>aws_kms | Specifies th [...]
-| Proxy Host | | | Proxy host n [...]
-| Proxy Port | | | The port num [...]
-| Proxy Username | | | Username to [...]
-| Proxy Password | | | Password to [...]
+| Name | <div style="width:7em">Default Value</div> | <div style="width:8em">Allowable Values</div> [...]
+|----------------------------------|--------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
+| Object Key | | [...]
+| **Bucket** | | [...]
+| Content Type | | [...]
+| **Use Default Credentials** | false | [...]
+| Access Key | | [...]
+| Secret Key | | [...]
+| Credentials File | | [...]
+| AWS Credentials Provider service | | [...]
+| **Storage Class** | Standard | Standard<br/>ReducedRedundancy<br/>StandardIA<br/>OnezoneIA<br/>IntelligentTiering<br/>Glacier<br/>DeepArchive [...]
+| **Region** | us-west-2 | af-south-1<br/>ap-east-1<br/>ap-northeast-1<br/>ap-northeast-2<br/>ap-northeast-3<br/>ap-south-1<br/>ap-southeast-1<br/>ap-southeast-2<br/>ap-southeast-3<br/>ca-central-1<br/>cn-north-1<br/>cn-northwest-1<br/>eu-central-1<br/>eu-north-1<br/>eu-south-1<br/>eu-west-1<br/>eu-west-2<br/>eu-west-3<br/>me-central-1<br/>me-south-1<br/>sa-east-1<br/>us-east-1<br/>us-east-2<br/>us-gov-east-1<br/>us-gov-west-1<br/>us [...]
+| **Communications Timeout** | 30 sec | [...]
+| FullControl User List | | [...]
+| Read Permission User List | | [...]
+| Read ACL User List | | [...]
+| Write ACL User List | | [...]
+| Canned ACL | | [...]
+| Endpoint Override URL | | [...]
+| **Server Side Encryption** | None | None<br/>AES256<br/>aws_kms [...]
+| Proxy Host | | [...]
+| Proxy Port | | [...]
+| Proxy Username | | [...]
+| Proxy Password | | [...]
### Relationships
| Name | Description |
diff --git a/cmake/BundledAwsSdkCpp.cmake b/cmake/BundledAwsSdkCpp.cmake
index aeb072c25..67c028533 100644
--- a/cmake/BundledAwsSdkCpp.cmake
+++ b/cmake/BundledAwsSdkCpp.cmake
@@ -19,10 +19,12 @@ function(use_bundled_libaws SOURCE_DIR BINARY_DIR)
set(PATCH_FILE1 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/c++20-compilation-fixes.patch")
set(PATCH_FILE2 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/dll-export-injection.patch")
set(PATCH_FILE3 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/shutdown-fix.patch")
+ set(PATCH_FILE4 "${SOURCE_DIR}/thirdparty/aws-sdk-cpp/bundle-openssl.patch")
set(AWS_SDK_CPP_PATCH_COMMAND ${Bash_EXECUTABLE} -c "set -x &&\
(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE1}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE1}\") &&\
(\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE2}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE2}\") &&\
- (\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE3}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE3}\") ")
+ (\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE3}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE3}\") &&\
+ (\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE4}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE4}\") ")
if (WIN32)
set(LIBDIR "lib")
@@ -56,6 +58,7 @@ function(use_bundled_libaws SOURCE_DIR BINARY_DIR)
"${LIBDIR}/${PREFIX}aws-c-auth.${SUFFIX}"
"${LIBDIR}/${PREFIX}aws-c-cal.${SUFFIX}"
"${LIBDIR}/${PREFIX}aws-c-compression.${SUFFIX}"
+ "${LIBDIR}/${PREFIX}aws-c-sdkutils.${SUFFIX}"
"${LIBDIR}/${PREFIX}aws-cpp-sdk-core.${SUFFIX}"
"${LIBDIR}/${PREFIX}aws-cpp-sdk-s3.${SUFFIX}")
@@ -80,7 +83,7 @@ function(use_bundled_libaws SOURCE_DIR BINARY_DIR)
ExternalProject_Add(
aws-sdk-cpp-external
GIT_REPOSITORY "https://github.com/aws/aws-sdk-cpp.git"
- GIT_TAG "1.9.65"
+ GIT_TAG "1.10.48"
UPDATE_COMMAND git submodule update --init --recursive
SOURCE_DIR "${BINARY_DIR}/thirdparty/aws-sdk-cpp-src"
INSTALL_DIR "${BINARY_DIR}/thirdparty/libaws-install"
@@ -114,6 +117,7 @@ function(use_bundled_libaws SOURCE_DIR BINARY_DIR)
set_target_properties(AWS::s2n PROPERTIES IMPORTED_LOCATION "${BINARY_DIR}/thirdparty/libaws-install/${LIBDIR}/${PREFIX}s2n.${SUFFIX}")
add_dependencies(AWS::s2n aws-sdk-cpp-external)
target_include_directories(AWS::s2n INTERFACE ${LIBAWS_INCLUDE_DIR})
+ target_link_libraries(AWS::s2n INTERFACE OpenSSL::Crypto)
endif()
add_library(AWS::aws-c-io STATIC IMPORTED)
@@ -167,11 +171,16 @@ function(use_bundled_libaws SOURCE_DIR BINARY_DIR)
add_dependencies(AWS::aws-c-compression aws-sdk-cpp-external)
target_include_directories(AWS::aws-c-compression INTERFACE ${LIBAWS_INCLUDE_DIR})
+ add_library(AWS::aws-c-sdkutils STATIC IMPORTED)
+ set_target_properties(AWS::aws-c-sdkutils PROPERTIES IMPORTED_LOCATION "${BINARY_DIR}/thirdparty/libaws-install/${LIBDIR}/${PREFIX}aws-c-sdkutils.${SUFFIX}")
+ add_dependencies(AWS::aws-c-sdkutils aws-sdk-cpp-external)
+ target_include_directories(AWS::aws-c-sdkutils INTERFACE ${LIBAWS_INCLUDE_DIR})
+
add_library(AWS::aws-crt-cpp STATIC IMPORTED)
set_target_properties(AWS::aws-crt-cpp PROPERTIES IMPORTED_LOCATION "${BINARY_DIR}/thirdparty/libaws-install/${LIBDIR}/${PREFIX}aws-crt-cpp.${SUFFIX}")
add_dependencies(AWS::aws-crt-cpp aws-sdk-cpp-external)
target_include_directories(AWS::aws-crt-cpp INTERFACE ${LIBAWS_INCLUDE_DIR})
- target_link_libraries(AWS::aws-crt-cpp INTERFACE AWS::aws-c-io AWS::aws-c-s3 AWS::aws-c-mqtt AWS::aws-c-http AWS::aws-c-cal AWS::aws-c-compression)
+ target_link_libraries(AWS::aws-crt-cpp INTERFACE AWS::aws-c-io AWS::aws-c-s3 AWS::aws-c-mqtt AWS::aws-c-http AWS::aws-c-cal AWS::aws-c-compression AWS::aws-c-sdkutils)
add_library(AWS::aws-cpp-sdk-core STATIC IMPORTED)
set_target_properties(AWS::aws-cpp-sdk-core PROPERTIES IMPORTED_LOCATION "${BINARY_DIR}/thirdparty/libaws-install/${LIBDIR}/${PREFIX}aws-cpp-sdk-core.${SUFFIX}")
diff --git a/extensions/aws/processors/S3Processor.h b/extensions/aws/processors/S3Processor.h
index 0e7b1c780..5be00bc84 100644
--- a/extensions/aws/processors/S3Processor.h
+++ b/extensions/aws/processors/S3Processor.h
@@ -48,6 +48,7 @@ constexpr const char *AP_NORTHEAST_3 = "ap-northeast-3";
constexpr const char *AP_SOUTH_1 = "ap-south-1";
constexpr const char *AP_SOUTHEAST_1 = "ap-southeast-1";
constexpr const char *AP_SOUTHEAST_2 = "ap-southeast-2";
+constexpr const char *AP_SOUTHEAST_3 = "ap-southeast-3";
constexpr const char *CA_CENTRAL_1 = "ca-central-1";
constexpr const char *CN_NORTH_1 = "cn-north-1";
constexpr const char *CN_NORTHWEST_1 = "cn-northwest-1";
@@ -57,12 +58,16 @@ constexpr const char *EU_SOUTH_1 = "eu-south-1";
constexpr const char *EU_WEST_1 = "eu-west-1";
constexpr const char *EU_WEST_2 = "eu-west-2";
constexpr const char *EU_WEST_3 = "eu-west-3";
+constexpr const char *ME_CENTRAL_1 = "me-central-1";
constexpr const char *ME_SOUTH_1 = "me-south-1";
constexpr const char *SA_EAST_1 = "sa-east-1";
constexpr const char *US_EAST_1 = "us-east-1";
constexpr const char *US_EAST_2 = "us-east-2";
constexpr const char *US_GOV_EAST_1 = "us-gov-east-1";
constexpr const char *US_GOV_WEST_1 = "us-gov-west-1";
+constexpr const char *US_ISO_EAST_1 = "us-iso-east-1";
+constexpr const char *US_ISOB_EAST_1 = "us-isob-east-1";
+constexpr const char *US_ISO_WEST_1 = "us-iso-west-1";
constexpr const char *US_WEST_1 = "us-west-1";
constexpr const char *US_WEST_2 = "us-west-2";
diff --git a/extensions/aws/processors/S3ProcessorStaticDefinitions.cpp b/extensions/aws/processors/S3ProcessorStaticDefinitions.cpp
index 1bc96788f..f3abed5d1 100644
--- a/extensions/aws/processors/S3ProcessorStaticDefinitions.cpp
+++ b/extensions/aws/processors/S3ProcessorStaticDefinitions.cpp
@@ -33,9 +33,10 @@ namespace org::apache::nifi::minifi::aws::processors {
const std::set<std::string> S3Processor::REGIONS({region::AF_SOUTH_1, region::AP_EAST_1, region::AP_NORTHEAST_1,
region::AP_NORTHEAST_2, region::AP_NORTHEAST_3, region::AP_SOUTH_1, region::AP_SOUTHEAST_1, region::AP_SOUTHEAST_2,
- region::CA_CENTRAL_1, region::CN_NORTH_1, region::CN_NORTHWEST_1, region::EU_CENTRAL_1, region::EU_NORTH_1,
- region::EU_SOUTH_1, region::EU_WEST_1, region::EU_WEST_2, region::EU_WEST_3, region::ME_SOUTH_1, region::SA_EAST_1,
- region::US_EAST_1, region::US_EAST_2, region::US_GOV_EAST_1, region::US_GOV_WEST_1, region::US_WEST_1, region::US_WEST_2});
+ region::AP_SOUTHEAST_3, region::CA_CENTRAL_1, region::CN_NORTH_1, region::CN_NORTHWEST_1, region::EU_CENTRAL_1,
+ region::EU_NORTH_1, region::EU_SOUTH_1, region::EU_WEST_1, region::EU_WEST_2, region::EU_WEST_3, region::ME_CENTRAL_1,
+ region::ME_SOUTH_1, region::SA_EAST_1, region::US_EAST_1, region::US_EAST_2, region::US_GOV_EAST_1, region::US_GOV_WEST_1,
+ region::US_ISO_EAST_1, region::US_ISOB_EAST_1, region::US_ISO_WEST_1, region::US_WEST_1, region::US_WEST_2});
const core::Property S3Processor::Bucket(
core::PropertyBuilder::createProperty("Bucket")
diff --git a/extensions/aws/s3/S3ClientRequestSender.cpp b/extensions/aws/s3/S3ClientRequestSender.cpp
index 66d415f2c..d24322e41 100644
--- a/extensions/aws/s3/S3ClientRequestSender.cpp
+++ b/extensions/aws/s3/S3ClientRequestSender.cpp
@@ -21,18 +21,13 @@
#include <aws/s3/S3Client.h>
-namespace org {
-namespace apache {
-namespace nifi {
-namespace minifi {
-namespace aws {
-namespace s3 {
+namespace org::apache::nifi::minifi::aws::s3 {
std::optional<Aws::S3::Model::PutObjectResult> S3ClientRequestSender::sendPutObjectRequest(
const Aws::S3::Model::PutObjectRequest& request,
const Aws::Auth::AWSCredentials& credentials,
const Aws::Client::ClientConfiguration& client_config) {
- Aws::S3::S3Client s3_client(credentials, client_config);
+ Aws::S3::S3Client s3_client(credentials, client_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true);
auto outcome = s3_client.PutObject(request);
if (outcome.IsSuccess()) {
@@ -48,7 +43,7 @@ bool S3ClientRequestSender::sendDeleteObjectRequest(
const Aws::S3::Model::DeleteObjectRequest& request,
const Aws::Auth::AWSCredentials& credentials,
const Aws::Client::ClientConfiguration& client_config) {
- Aws::S3::S3Client s3_client(credentials, client_config);
+ Aws::S3::S3Client s3_client(credentials, client_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true);
Aws::S3::Model::DeleteObjectOutcome outcome = s3_client.DeleteObject(request);
if (outcome.IsSuccess()) {
@@ -67,7 +62,7 @@ std::optional<Aws::S3::Model::GetObjectResult> S3ClientRequestSender::sendGetObj
const Aws::S3::Model::GetObjectRequest& request,
const Aws::Auth::AWSCredentials& credentials,
const Aws::Client::ClientConfiguration& client_config) {
- Aws::S3::S3Client s3_client(credentials, client_config);
+ Aws::S3::S3Client s3_client(credentials, client_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true);
auto outcome = s3_client.GetObject(request);
if (outcome.IsSuccess()) {
@@ -83,7 +78,7 @@ std::optional<Aws::S3::Model::ListObjectsV2Result> S3ClientRequestSender::sendLi
const Aws::S3::Model::ListObjectsV2Request& request,
const Aws::Auth::AWSCredentials& credentials,
const Aws::Client::ClientConfiguration& client_config) {
- Aws::S3::S3Client s3_client(credentials, client_config);
+ Aws::S3::S3Client s3_client(credentials, client_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true);
auto outcome = s3_client.ListObjectsV2(request);
if (outcome.IsSuccess()) {
@@ -99,7 +94,7 @@ std::optional<Aws::S3::Model::ListObjectVersionsResult> S3ClientRequestSender::s
const Aws::S3::Model::ListObjectVersionsRequest& request,
const Aws::Auth::AWSCredentials& credentials,
const Aws::Client::ClientConfiguration& client_config) {
- Aws::S3::S3Client s3_client(credentials, client_config);
+ Aws::S3::S3Client s3_client(credentials, client_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true);
auto outcome = s3_client.ListObjectVersions(request);
if (outcome.IsSuccess()) {
@@ -115,7 +110,7 @@ std::optional<Aws::S3::Model::GetObjectTaggingResult> S3ClientRequestSender::sen
const Aws::S3::Model::GetObjectTaggingRequest& request,
const Aws::Auth::AWSCredentials& credentials,
const Aws::Client::ClientConfiguration& client_config) {
- Aws::S3::S3Client s3_client(credentials, client_config);
+ Aws::S3::S3Client s3_client(credentials, client_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true);
auto outcome = s3_client.GetObjectTagging(request);
if (outcome.IsSuccess()) {
@@ -131,7 +126,7 @@ std::optional<Aws::S3::Model::HeadObjectResult> S3ClientRequestSender::sendHeadO
const Aws::S3::Model::HeadObjectRequest& request,
const Aws::Auth::AWSCredentials& credentials,
const Aws::Client::ClientConfiguration& client_config) {
- Aws::S3::S3Client s3_client(credentials, client_config);
+ Aws::S3::S3Client s3_client(credentials, client_config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, true);
auto outcome = s3_client.HeadObject(request);
if (outcome.IsSuccess()) {
@@ -143,9 +138,4 @@ std::optional<Aws::S3::Model::HeadObjectResult> S3ClientRequestSender::sendHeadO
}
}
-} // namespace s3
-} // namespace aws
-} // namespace minifi
-} // namespace nifi
-} // namespace apache
-} // namespace org
+} // namespace org::apache::nifi::minifi::aws::s3
diff --git a/extensions/aws/s3/S3ClientRequestSender.h b/extensions/aws/s3/S3ClientRequestSender.h
index 01b0e5c37..769d7c6d1 100644
--- a/extensions/aws/s3/S3ClientRequestSender.h
+++ b/extensions/aws/s3/S3ClientRequestSender.h
@@ -23,12 +23,7 @@
#include "S3RequestSender.h"
-namespace org {
-namespace apache {
-namespace nifi {
-namespace minifi {
-namespace aws {
-namespace s3 {
+namespace org::apache::nifi::minifi::aws::s3 {
class S3ClientRequestSender : public S3RequestSender {
public:
@@ -62,9 +57,4 @@ class S3ClientRequestSender : public S3RequestSender {
const Aws::Client::ClientConfiguration& client_config) override;
};
-} // namespace s3
-} // namespace aws
-} // namespace minifi
-} // namespace nifi
-} // namespace apache
-} // namespace org
+} // namespace org::apache::nifi::minifi::aws::s3
diff --git a/extensions/aws/s3/S3Wrapper.h b/extensions/aws/s3/S3Wrapper.h
index da37564fa..038d0d029 100644
--- a/extensions/aws/s3/S3Wrapper.h
+++ b/extensions/aws/s3/S3Wrapper.h
@@ -97,9 +97,9 @@ struct PutObjectResult {
};
struct RequestParameters {
- RequestParameters(Aws::Auth::AWSCredentials creds, Aws::Client::ClientConfiguration config)
- : credentials(std::move(creds))
- , client_config(std::move(config)) {}
+ RequestParameters(Aws::Auth::AWSCredentials creds, Aws::Client::ClientConfiguration config)
+ : credentials(std::move(creds)),
+ client_config(std::move(config)) {}
Aws::Auth::AWSCredentials credentials;
Aws::Client::ClientConfiguration client_config;
diff --git a/libminifi/test/aws-tests/PutS3ObjectTests.cpp b/libminifi/test/aws-tests/PutS3ObjectTests.cpp
index 854e5e87e..884bfc7e7 100644
--- a/libminifi/test/aws-tests/PutS3ObjectTests.cpp
+++ b/libminifi/test/aws-tests/PutS3ObjectTests.cpp
@@ -154,7 +154,7 @@ TEST_CASE_METHOD(PutS3ObjectTestsFixture, "Set non-default client configuration"
plan->setProperty(update_attribute, "test.contentType", "application/tar", true);
plan->setProperty(s3_processor, "Content Type", "${test.contentType}");
plan->setProperty(s3_processor, "Storage Class", "ReducedRedundancy");
- plan->setProperty(s3_processor, "Region", minifi::aws::processors::region::US_EAST_1);
+ plan->setProperty(s3_processor, "Region", minifi::aws::processors::region::AP_SOUTHEAST_3);
plan->setProperty(s3_processor, "Communications Timeout", "10 Sec");
plan->setProperty(update_attribute, "test.endpoint", "http://localhost:1234", true);
plan->setProperty(s3_processor, "Endpoint Override URL", "${test.endpoint}");
@@ -167,7 +167,7 @@ TEST_CASE_METHOD(PutS3ObjectTestsFixture, "Set non-default client configuration"
REQUIRE(mock_s3_request_sender_ptr->put_object_request.GetContentType() == "application/tar");
REQUIRE(mock_s3_request_sender_ptr->put_object_request.GetStorageClass() == Aws::S3::Model::StorageClass::REDUCED_REDUNDANCY);
REQUIRE(mock_s3_request_sender_ptr->put_object_request.GetServerSideEncryption() == Aws::S3::Model::ServerSideEncryption::AES256);
- REQUIRE(mock_s3_request_sender_ptr->getClientConfig().region == minifi::aws::processors::region::US_EAST_1);
+ REQUIRE(mock_s3_request_sender_ptr->getClientConfig().region == minifi::aws::processors::region::AP_SOUTHEAST_3);
REQUIRE(mock_s3_request_sender_ptr->getClientConfig().connectTimeoutMs == 10000);
REQUIRE(mock_s3_request_sender_ptr->getClientConfig().endpointOverride == "http://localhost:1234");
REQUIRE(mock_s3_request_sender_ptr->getPutObjectRequestBody() == INPUT_DATA);
diff --git a/minifi_main/CMakeLists.txt b/minifi_main/CMakeLists.txt
index beb2be852..95014e934 100644
--- a/minifi_main/CMakeLists.txt
+++ b/minifi_main/CMakeLists.txt
@@ -68,6 +68,9 @@ target_link_libraries(minifiexe spdlog libsodium gsl-lite ${LIBMINIFI})
set_target_properties(minifiexe PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin")
set_target_properties(minifiexe PROPERTIES OUTPUT_NAME minifi)
set_target_properties(minifiexe PROPERTIES ENABLE_EXPORTS True)
+if (WIN32)
+ target_compile_definitions(minifiexe PUBLIC SERVICE_NAME="Apache NiFi MINiFi")
+endif()
if (NOT WIN32)
add_custom_command(TARGET minifiexe POST_BUILD
diff --git a/thirdparty/aws-sdk-cpp/bundle-openssl.patch b/thirdparty/aws-sdk-cpp/bundle-openssl.patch
new file mode 100644
index 000000000..4262dbeb2
--- /dev/null
+++ b/thirdparty/aws-sdk-cpp/bundle-openssl.patch
@@ -0,0 +1,46 @@
+diff --git a/cmake/external_dependencies.cmake b/cmake/external_dependencies.cmake
+index 20feb6de51..aa63b4633c 100644
+--- a/cmake/external_dependencies.cmake
++++ b/cmake/external_dependencies.cmake
+@@ -34,14 +34,14 @@ elseif(ENABLE_OPENSSL_ENCRYPTION)
+ add_definitions(-DENABLE_OPENSSL_ENCRYPTION)
+ message(STATUS "Encryption: LibCrypto")
+
+- set(CRYPTO_TARGET_NAME "AWS::crypto")
++ set(CRYPTO_TARGET_NAME "OpenSSL::Crypto")
+ if(PLATFORM_ANDROID AND ANDROID_BUILD_OPENSSL)
+ set(BUILD_OPENSSL 1)
+ set(CRYPTO_TARGET_NAME "crypto")
+ set(USE_OPENSSL ON)
+ message(STATUS " Building Openssl as part of AWS SDK")
+ else()
+- find_package(crypto REQUIRED)
++ find_package(OpenSSL REQUIRED)
+ endif()
+ set(CRYPTO_LIBS ${CRYPTO_TARGET_NAME} ${ZLIB_LIBRARIES})
+ # ssl depends on libcrypto
+diff --git a/crt/aws-crt-cpp/crt/s2n/CMakeLists.txt b/crt/aws-crt-cpp/crt/s2n/CMakeLists.txt
+index f5ee8379..9dcfc7ab 100644
+--- a/crt/aws-crt-cpp/crt/s2n/CMakeLists.txt
++++ b/crt/aws-crt-cpp/crt/s2n/CMakeLists.txt
+@@ -116,7 +116,7 @@ else()
+ # supports it and set proper compiler flags to be added later to the
+ # Kyber compilation units.
+ if(${CMAKE_SYSTEM_PROCESSOR} MATCHES "^(x86_64|amd64|AMD64)$")
+-
++
+ set(KYBER512R3_AVX2_BMI2_FLAGS "-mavx2 -mavx -mbmi2")
+ try_compile(KYBER512R3_AVX2_BMI2_SUPPORTED
+ ${CMAKE_BINARY_DIR}
+@@ -380,9 +380,9 @@ if (TARGET crypto)
+ message(STATUS "S2N found target: crypto")
+ set(LINK_LIB "crypto")
+ else()
+- find_package(crypto REQUIRED)
++ find_package(OpenSSL REQUIRED)
+ message(STATUS "Using libcrypto from the cmake path")
+- set(LINK_LIB "AWS::crypto")
++ set(LINK_LIB "OpenSSL::Crypto")
+ endif()
+
+ if (S2N_INTERN_LIBCRYPTO)
diff --git a/thirdparty/aws-sdk-cpp/c++20-compilation-fixes.patch b/thirdparty/aws-sdk-cpp/c++20-compilation-fixes.patch
index 8a8a5bc06..f13d28f61 100644
--- a/thirdparty/aws-sdk-cpp/c++20-compilation-fixes.patch
+++ b/thirdparty/aws-sdk-cpp/c++20-compilation-fixes.patch
@@ -1,26 +1,7 @@
-diff -ruN orig/aws-cpp-sdk-core/source/utils/crypto/bcrypt/CryptoImpl.cpp patched/aws-cpp-sdk-core/source/utils/crypto/bcrypt/CryptoImpl.cpp
---- orig/aws-cpp-sdk-core/source/utils/crypto/bcrypt/CryptoImpl.cpp 2021-08-03 20:29:03.231072140 +0200
-+++ patched/aws-cpp-sdk-core/source/utils/crypto/bcrypt/CryptoImpl.cpp 2021-08-03 20:30:21.510721218 +0200
-@@ -11,6 +11,7 @@
- #include <aws/core/utils/Outcome.h>
- #include <aws/core/utils/crypto/Hash.h>
- #include <aws/core/utils/HashingUtils.h>
-+#include <aws/core/utils/StringUtils.h>
- #include <atomic>
- #include <bcrypt.h>
- #include <winternl.h>
-@@ -126,7 +127,7 @@
- NTSTATUS status = BCryptOpenAlgorithmProvider(&m_algorithmHandle, algorithmName, MS_PRIMITIVE_PROVIDER, isHMAC ? BCRYPT_ALG_HANDLE_HMAC_FLAG : 0);
- if (!NT_SUCCESS(status))
- {
-- AWS_LOGSTREAM_ERROR(logTag, "Failed initializing BCryptOpenAlgorithmProvider for " << algorithmName);
-+ AWS_LOGSTREAM_ERROR(logTag, "Failed initializing BCryptOpenAlgorithmProvider for " << StringUtils::FromWString(algorithmName));
- return;
- }
-
-diff -ruN orig/crt/aws-crt-cpp/include/aws/crt/StringView.h patched/crt/aws-crt-cpp/include/aws/crt/StringView.h
---- orig/crt/aws-crt-cpp/include/aws/crt/StringView.h 2021-08-03 20:29:04.615065881 +0200
-+++ patched/crt/aws-crt-cpp/include/aws/crt/StringView.h 2021-08-03 20:30:38.030647928 +0200
+diff --git a/crt/aws-crt-cpp/include/aws/crt/StringView.h b/crt/aws-crt-cpp/include/aws/crt/StringView.h
+index a0fec83..6d0b997 100644
+--- a/crt/aws-crt-cpp/include/aws/crt/StringView.h
++++ b/crt/aws-crt-cpp/include/aws/crt/StringView.h
@@ -13,6 +13,7 @@
#include <iterator>
#include <limits>
diff --git a/thirdparty/aws-sdk-cpp/dll-export-injection.patch b/thirdparty/aws-sdk-cpp/dll-export-injection.patch
index b3304b0fd..8f8979d59 100644
--- a/thirdparty/aws-sdk-cpp/dll-export-injection.patch
+++ b/thirdparty/aws-sdk-cpp/dll-export-injection.patch
@@ -1,7 +1,8 @@
-diff -rupN aws-sdk-cpp-src/aws-cpp-sdk-core/CMakeLists.txt aws-sdk-cpp-src-patched/aws-cpp-sdk-core/CMakeLists.txt
---- aws-sdk-cpp-src/aws-cpp-sdk-core/CMakeLists.txt 2021-07-19 13:25:02.472816900 +0200
-+++ aws-sdk-cpp-src-patched/aws-cpp-sdk-core/CMakeLists.txt 2021-07-19 13:24:17.986828500 +0200
-@@ -507,6 +507,10 @@ if(USE_WINDOWS_DLL_SEMANTICS)
+diff --git a/aws-cpp-sdk-core/CMakeLists.txt b/aws-cpp-sdk-core/CMakeLists.txt
+index ab20cec848..d945c84000 100644
+--- a/aws-cpp-sdk-core/CMakeLists.txt
++++ b/aws-cpp-sdk-core/CMakeLists.txt
+@@ -550,6 +550,10 @@ if(USE_WINDOWS_DLL_SEMANTICS)
endif()
endif()
@@ -12,9 +13,10 @@ diff -rupN aws-sdk-cpp-src/aws-cpp-sdk-core/CMakeLists.txt aws-sdk-cpp-src-patch
set_compiler_flags(${PROJECT_NAME})
set_compiler_warnings(${PROJECT_NAME})
-diff -rupN aws-sdk-cpp-src/aws-cpp-sdk-core/include/aws/core/Core_EXPORTS.h aws-sdk-cpp-src-patched/aws-cpp-sdk-core/include/aws/core/Core_EXPORTS.h
---- aws-sdk-cpp-src/aws-cpp-sdk-core/include/aws/core/Core_EXPORTS.h 2021-07-19 13:25:02.473815700 +0200
-+++ aws-sdk-cpp-src-patched/aws-cpp-sdk-core/include/aws/core/Core_EXPORTS.h 2021-07-19 13:24:18.070776900 +0200
+diff --git a/aws-cpp-sdk-core/include/aws/core/Core_EXPORTS.h b/aws-cpp-sdk-core/include/aws/core/Core_EXPORTS.h
+index 1d3e312a60..74b111df96 100644
+--- a/aws-cpp-sdk-core/include/aws/core/Core_EXPORTS.h
++++ b/aws-cpp-sdk-core/include/aws/core/Core_EXPORTS.h
@@ -18,7 +18,7 @@
#else // AWS_CORE_EXPORTS
#define AWS_CORE_API __declspec(dllimport)
@@ -23,11 +25,12 @@ diff -rupN aws-sdk-cpp-src/aws-cpp-sdk-core/include/aws/core/Core_EXPORTS.h aws-
+ #elif !defined(AWS_CORE_API)
#define AWS_CORE_API
#endif // USE_IMPORT_EXPORT
- #else // defined (USE_WINDOWS_DLL_SEMANTICS) || defined (_WIN32)
-diff -rupN aws-sdk-cpp-src/aws-cpp-sdk-s3/CMakeLists.txt aws-sdk-cpp-src-patched/aws-cpp-sdk-s3/CMakeLists.txt
---- aws-sdk-cpp-src/aws-cpp-sdk-s3/CMakeLists.txt 2021-07-19 13:25:02.474814900 +0200
-+++ aws-sdk-cpp-src-patched/aws-cpp-sdk-s3/CMakeLists.txt 2021-07-19 13:23:53.913102900 +0200
-@@ -59,6 +59,10 @@ if(USE_WINDOWS_DLL_SEMANTICS AND BUILD_S
+ #define AWS_CORE_LOCAL
+diff --git a/aws-cpp-sdk-s3/CMakeLists.txt b/aws-cpp-sdk-s3/CMakeLists.txt
+index f44ba56642..ee40d8bf44 100644
+--- a/aws-cpp-sdk-s3/CMakeLists.txt
++++ b/aws-cpp-sdk-s3/CMakeLists.txt
+@@ -59,6 +59,10 @@ if(USE_WINDOWS_DLL_SEMANTICS AND BUILD_SHARED_LIBS)
target_compile_definitions(${PROJECT_NAME} PRIVATE "AWS_S3_EXPORTS")
endif()
@@ -38,9 +41,10 @@ diff -rupN aws-sdk-cpp-src/aws-cpp-sdk-s3/CMakeLists.txt aws-sdk-cpp-src-patched
target_include_directories(${PROJECT_NAME} PUBLIC
$<BUILD_INTERFACE:${CMAKE_CURRENT_SOURCE_DIR}/include>
$<INSTALL_INTERFACE:include>)
-diff -rupN aws-sdk-cpp-src/aws-cpp-sdk-s3/include/aws/s3/S3_EXPORTS.h aws-sdk-cpp-src-patched/aws-cpp-sdk-s3/include/aws/s3/S3_EXPORTS.h
---- aws-sdk-cpp-src/aws-cpp-sdk-s3/include/aws/s3/S3_EXPORTS.h 2021-07-19 13:25:02.475814200 +0200
-+++ aws-sdk-cpp-src-patched/aws-cpp-sdk-s3/include/aws/s3/S3_EXPORTS.h 2021-07-19 13:23:53.918099900 +0200
+diff --git a/aws-cpp-sdk-s3/include/aws/s3/S3_EXPORTS.h b/aws-cpp-sdk-s3/include/aws/s3/S3_EXPORTS.h
+index b520ae4341..7d325214bc 100644
+--- a/aws-cpp-sdk-s3/include/aws/s3/S3_EXPORTS.h
++++ b/aws-cpp-sdk-s3/include/aws/s3/S3_EXPORTS.h
@@ -21,7 +21,7 @@
#else
#define AWS_S3_API __declspec(dllimport)
diff --git a/thirdparty/aws-sdk-cpp/shutdown-fix.patch b/thirdparty/aws-sdk-cpp/shutdown-fix.patch
index e8bafa585..90ccc400c 100644
--- a/thirdparty/aws-sdk-cpp/shutdown-fix.patch
+++ b/thirdparty/aws-sdk-cpp/shutdown-fix.patch
@@ -1,7 +1,8 @@
-diff -rupN aws-sdk-cpp-src/crt/aws-crt-cpp/crt/aws-c-io/source/event_loop.c aws-sdk-cpp-src-patched/crt/aws-crt-cpp/crt/aws-c-io/source/event_loop.c
---- aws-sdk-cpp-src/crt/aws-crt-cpp/crt/aws-c-io/source/event_loop.c 2021-08-09 08:48:51.141882100 +0200
-+++ aws-sdk-cpp-src-patched/crt/aws-crt-cpp/crt/aws-c-io/source/event_loop.c 2021-08-09 08:48:08.231072500 +0200
-@@ -114,7 +114,7 @@ static struct aws_event_loop_group *s_ev
+diff --git a/crt/aws-crt-cpp/crt/aws-c-io/source/event_loop.c b/crt/aws-crt-cpp/crt/aws-c-io/source/event_loop.c
+index c37aaf6..2a45c21 100644
+--- a/crt/aws-crt-cpp/crt/aws-c-io/source/event_loop.c
++++ b/crt/aws-crt-cpp/crt/aws-c-io/source/event_loop.c
+@@ -114,7 +114,7 @@ static struct aws_event_loop_group *s_event_loop_group_new(
el_group->allocator = alloc;
aws_ref_count_init(