You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nifi.apache.org by ma...@apache.org on 2024/03/12 14:40:50 UTC
(nifi-minifi-cpp) branch main updated (21dcd4d09 -> fc9e50769)
This is an automated email from the ASF dual-hosted git repository.
martinzink pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
from 21dcd4d09 MINIFICPP-2278 Add custom relationship support for python processors
new 3776bd071 MINIFICPP-2296 - Install service on macOS
new 03b504ef1 MINIFICPP-2302 Upgrade github actions to Node.js 20 (or latest available) versions
new 545236fd0 MINIFICPP-2298 Make RocksDB options configurable - Also set keep_log_file_num default value to 5
new 5d9cd59a9 MINIFICPP-2307 Fix libsodium url parameter passing for CMake 3.29
new dd8e54e51 MINIFICPP-2192 Document user requirements for Windows service installation
new fc9e50769 MINIFICPP-1757 remove problematic linker flags on macos/gcc
The 6 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/ci.yml | 68 +++++------
CONFIGURE.md | 22 ++++
Windows.md | 15 +++
bin/minifi.plist | 25 ++++
bin/minifi.sh | 128 ++++++++++++++++-----
cmake/BundledLibSodium.cmake | 4 +-
cmake/BundledRocksDB.cmake | 14 ++-
cmake/Extensions.cmake | 2 +-
controller/CMakeLists.txt | 2 +-
encrypt-config/CMakeLists.txt | 2 +-
.../rocksdb-repos/DatabaseContentRepository.cpp | 8 +-
extensions/rocksdb-repos/FlowFileRepository.cpp | 7 +-
extensions/rocksdb-repos/ProvenanceRepository.cpp | 7 +-
.../controllers/RocksDbStateStorage.cpp | 7 +-
.../rocksdb-repos/database/RocksDatabase.cpp | 12 +-
extensions/rocksdb-repos/database/RocksDatabase.h | 5 +-
.../rocksdb-repos/database/RocksDbInstance.cpp | 29 +++--
.../rocksdb-repos/database/RocksDbInstance.h | 4 +-
extensions/rocksdb-repos/database/RocksDbUtils.cpp | 25 ++++
extensions/rocksdb-repos/database/RocksDbUtils.h | 3 +
.../tests/DBContentRepositoryTests.cpp | 2 +-
.../rocksdb-repos/tests/RocksDBStreamTests.cpp | 2 +-
extensions/rocksdb-repos/tests/RocksDBTests.cpp | 38 +++---
.../rocksdb-repos/tests/RocksDBUtilsTests.cpp | 46 ++++++++
libminifi/include/properties/Configuration.h | 5 +
libminifi/include/properties/Properties.h | 1 -
minifi_main/CMakeLists.txt | 2 +-
.../rocksdb/dboptions_equality_operator.patch | 21 ++++
28 files changed, 379 insertions(+), 127 deletions(-)
create mode 100644 bin/minifi.plist
create mode 100644 extensions/rocksdb-repos/tests/RocksDBUtilsTests.cpp
create mode 100644 thirdparty/rocksdb/dboptions_equality_operator.patch
(nifi-minifi-cpp) 03/06: MINIFICPP-2298 Make RocksDB options configurable - Also set keep_log_file_num default value to 5
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martinzink pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 545236fd06b613f2d2b38f74ff9ee85df9190d59
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Tue Mar 12 13:51:58 2024 +0100
MINIFICPP-2298 Make RocksDB options configurable
- Also set keep_log_file_num default value to 5
Closes #1731
Signed-off-by: Martin Zink <ma...@apache.org>
---
CONFIGURE.md | 22 +++++++++++
cmake/BundledRocksDB.cmake | 14 ++++---
.../rocksdb-repos/DatabaseContentRepository.cpp | 8 ++--
extensions/rocksdb-repos/FlowFileRepository.cpp | 7 ++--
extensions/rocksdb-repos/ProvenanceRepository.cpp | 7 ++--
.../controllers/RocksDbStateStorage.cpp | 7 ++--
.../rocksdb-repos/database/RocksDatabase.cpp | 12 +++---
extensions/rocksdb-repos/database/RocksDatabase.h | 5 ++-
.../rocksdb-repos/database/RocksDbInstance.cpp | 29 ++++++++++----
.../rocksdb-repos/database/RocksDbInstance.h | 4 +-
extensions/rocksdb-repos/database/RocksDbUtils.cpp | 25 ++++++++++++
extensions/rocksdb-repos/database/RocksDbUtils.h | 3 ++
.../tests/DBContentRepositoryTests.cpp | 2 +-
.../rocksdb-repos/tests/RocksDBStreamTests.cpp | 2 +-
extensions/rocksdb-repos/tests/RocksDBTests.cpp | 38 +++++++++---------
.../rocksdb-repos/tests/RocksDBUtilsTests.cpp | 46 ++++++++++++++++++++++
libminifi/include/properties/Configuration.h | 5 +++
libminifi/include/properties/Properties.h | 1 -
.../rocksdb/dboptions_equality_operator.patch | 21 ++++++++++
19 files changed, 198 insertions(+), 60 deletions(-)
diff --git a/CONFIGURE.md b/CONFIGURE.md
index 16469db58..2df8d695a 100644
--- a/CONFIGURE.md
+++ b/CONFIGURE.md
@@ -425,6 +425,28 @@ Rocksdb has an option to run compaction at specific intervals not just when need
nifi.flowfile.repository.rocksdb.compaction.period=2 min
nifi.database.content.repository.rocksdb.compaction.period=2 min
+### Global RocksDB options
+
+There are a few options for RocksDB that are set for all used RocksDB databases in MiNiFi:
+ `create_if_missing` is set to `true`
+ `use_direct_io_for_flush_and_compaction` is set to `true`
+ `use_direct_reads` is set to `true`
+ `keep_log_file_num` is set to `5`
+
+Any RocksDB option can be set or overriden using the `nifi.global.rocksdb.options.` prefix in the minifi.properties file.
+
+ # in minifi.properties
+ nifi.global.rocksdb.options.keep_log_file_num=7
+ nifi.global.rocksdb.options.atomic_flush=true
+
+RocksDB options can also be overridden for a specific repository using the `nifi.<repository>.rocksdb.options.` prefix in the minifi.properties file.
+
+ # in minifi.properties
+ nifi.flowfile.repository.rocksdb.options.keep_log_file_num=3
+ nifi.content.repository.rocksdb.options.atomic_flush=true
+ nifi.provenance.repository.rocksdb.options.use_direct_reads=false
+ nifi.state.storage.rocksdb.options.use_direct_io_for_flush_and_compaction=false
+
#### Shared database
It is also possible to use a single database to store multiple repositories with the `minifidb://` scheme.
diff --git a/cmake/BundledRocksDB.cmake b/cmake/BundledRocksDB.cmake
index f1dc063d3..557d44df0 100644
--- a/cmake/BundledRocksDB.cmake
+++ b/cmake/BundledRocksDB.cmake
@@ -27,9 +27,12 @@ function(use_bundled_rocksdb SOURCE_DIR BINARY_DIR)
endif()
# Patch to fix build issue on ARM7 architecture: https://github.com/facebook/rocksdb/issues/8609#issuecomment-1009572506
- set(PATCH_FILE "${SOURCE_DIR}/thirdparty/rocksdb/arm7.patch")
- set(PC ${Bash_EXECUTABLE} -c "set -x && \
- (\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE}\")")
+ set(PATCH_FILE_1 "${SOURCE_DIR}/thirdparty/rocksdb/arm7.patch")
+ set(PATCH_FILE_2 "${SOURCE_DIR}/thirdparty/rocksdb/dboptions_equality_operator.patch")
+ set(PC ${Bash_EXECUTABLE} -c "set -x &&\
+ (\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE_1}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE_1}\") &&\
+ (\"${Patch_EXECUTABLE}\" -p1 -R -s -f --dry-run -i \"${PATCH_FILE_2}\" || \"${Patch_EXECUTABLE}\" -p1 -N -i \"${PATCH_FILE_2}\") ")
+
# Define byproducts
if (WIN32)
set(BYPRODUCT "lib/rocksdb.lib")
@@ -49,7 +52,6 @@ function(use_bundled_rocksdb SOURCE_DIR BINARY_DIR)
-DUSE_RTTI=1
-DROCKSDB_BUILD_SHARED=OFF
-DFAIL_ON_WARNINGS=OFF
- -DCMAKE_CXX_STANDARD=17 # RocksDB fails to build in C++20 mode on GCC 11: https://godbolt.org/z/YeMcEzs8W
)
if(PORTABLE)
list(APPEND ROCKSDB_CMAKE_ARGS -DPORTABLE=ON)
@@ -71,8 +73,8 @@ function(use_bundled_rocksdb SOURCE_DIR BINARY_DIR)
# Build project
ExternalProject_Add(
rocksdb-external
- URL "https://github.com/facebook/rocksdb/archive/refs/tags/v8.1.1.tar.gz"
- URL_HASH "SHA256=9102704e169cfb53e7724a30750eeeb3e71307663852f01fa08d5a320e6155a8"
+ URL "https://github.com/facebook/rocksdb/archive/refs/tags/v8.10.2.tar.gz"
+ URL_HASH "SHA256=44b6ec2f4723a0d495762da245d4a59d38704e0d9d3d31c45af4014bee853256"
SOURCE_DIR "${BINARY_DIR}/thirdparty/rocksdb-src"
CMAKE_ARGS ${ROCKSDB_CMAKE_ARGS}
PATCH_COMMAND ${PC}
diff --git a/extensions/rocksdb-repos/DatabaseContentRepository.cpp b/extensions/rocksdb-repos/DatabaseContentRepository.cpp
index 65781fef9..b1cd9859c 100644
--- a/extensions/rocksdb-repos/DatabaseContentRepository.cpp
+++ b/extensions/rocksdb-repos/DatabaseContentRepository.cpp
@@ -56,10 +56,7 @@ bool DatabaseContentRepository::initialize(const std::shared_ptr<minifi::Configu
setCompactionPeriod(configuration);
auto set_db_opts = [encrypted_env] (minifi::internal::Writable<rocksdb::DBOptions>& db_opts) {
- db_opts.set(&rocksdb::DBOptions::create_if_missing, true);
- db_opts.set(&rocksdb::DBOptions::use_direct_io_for_flush_and_compaction, true);
- db_opts.set(&rocksdb::DBOptions::use_direct_reads, true);
- db_opts.set(&rocksdb::DBOptions::error_if_exists, false);
+ minifi::internal::setCommonRocksDbOptions(db_opts);
if (encrypted_env) {
db_opts.set(&rocksdb::DBOptions::env, encrypted_env.get(), EncryptionEq{});
} else {
@@ -74,7 +71,8 @@ bool DatabaseContentRepository::initialize(const std::shared_ptr<minifi::Configu
cf_opts.compression = *compression_type;
}
};
- db_ = minifi::internal::RocksDatabase::create(set_db_opts, set_cf_opts, directory_);
+ db_ = minifi::internal::RocksDatabase::create(set_db_opts, set_cf_opts, directory_,
+ minifi::internal::getRocksDbOptionsToOverride(configuration, Configure::nifi_content_repository_rocksdb_options));
if (db_->open()) {
logger_->log_debug("NiFi Content DB Repository database open {} success", directory_);
is_valid_ = true;
diff --git a/extensions/rocksdb-repos/FlowFileRepository.cpp b/extensions/rocksdb-repos/FlowFileRepository.cpp
index 67f08cba5..ca81c795a 100644
--- a/extensions/rocksdb-repos/FlowFileRepository.cpp
+++ b/extensions/rocksdb-repos/FlowFileRepository.cpp
@@ -185,9 +185,7 @@ bool FlowFileRepository::initialize(const std::shared_ptr<Configure> &configure)
logger_->log_info("Using {} FlowFileRepository", encrypted_env ? "encrypted" : "plaintext");
auto db_options = [encrypted_env] (minifi::internal::Writable<rocksdb::DBOptions>& options) {
- options.set(&rocksdb::DBOptions::create_if_missing, true);
- options.set(&rocksdb::DBOptions::use_direct_io_for_flush_and_compaction, true);
- options.set(&rocksdb::DBOptions::use_direct_reads, true);
+ minifi::internal::setCommonRocksDbOptions(options);
if (encrypted_env) {
options.set(&rocksdb::DBOptions::env, encrypted_env.get(), EncryptionEq{});
} else {
@@ -210,7 +208,8 @@ bool FlowFileRepository::initialize(const std::shared_ptr<Configure> &configure)
cf_opts.compression = *compression_type;
}
};
- db_ = minifi::internal::RocksDatabase::create(db_options, cf_options, directory_);
+ db_ = minifi::internal::RocksDatabase::create(db_options, cf_options, directory_,
+ minifi::internal::getRocksDbOptionsToOverride(configure, Configure::nifi_flowfile_repository_rocksdb_options));
if (db_->open()) {
logger_->log_debug("NiFi FlowFile Repository database open {} success", directory_);
return true;
diff --git a/extensions/rocksdb-repos/ProvenanceRepository.cpp b/extensions/rocksdb-repos/ProvenanceRepository.cpp
index 42bf840fa..68f404fe3 100644
--- a/extensions/rocksdb-repos/ProvenanceRepository.cpp
+++ b/extensions/rocksdb-repos/ProvenanceRepository.cpp
@@ -40,9 +40,7 @@ bool ProvenanceRepository::initialize(const std::shared_ptr<org::apache::nifi::m
logger_->log_debug("MiNiFi Provenance Max Storage Time: [{}]", max_partition_millis_);
auto db_options = [] (minifi::internal::Writable<rocksdb::DBOptions>& db_opts) {
- db_opts.set(&rocksdb::DBOptions::create_if_missing, true);
- db_opts.set(&rocksdb::DBOptions::use_direct_io_for_flush_and_compaction, true);
- db_opts.set(&rocksdb::DBOptions::use_direct_reads, true);
+ minifi::internal::setCommonRocksDbOptions(db_opts);
};
// Rocksdb write buffers act as a log of database operation: grow till reaching the limit, serialized after
@@ -60,7 +58,8 @@ bool ProvenanceRepository::initialize(const std::shared_ptr<org::apache::nifi::m
}
};
- db_ = minifi::internal::RocksDatabase::create(db_options, cf_options, directory_);
+ db_ = minifi::internal::RocksDatabase::create(db_options, cf_options, directory_,
+ minifi::internal::getRocksDbOptionsToOverride(config, Configure::nifi_provenance_repository_rocksdb_options));
if (db_->open()) {
logger_->log_debug("MiNiFi Provenance Repository database open {} success", directory_);
} else {
diff --git a/extensions/rocksdb-repos/controllers/RocksDbStateStorage.cpp b/extensions/rocksdb-repos/controllers/RocksDbStateStorage.cpp
index a07e1a451..965165ced 100644
--- a/extensions/rocksdb-repos/controllers/RocksDbStateStorage.cpp
+++ b/extensions/rocksdb-repos/controllers/RocksDbStateStorage.cpp
@@ -67,9 +67,7 @@ void RocksDbStateStorage::onEnable() {
logger_->log_info("Using {} RocksDbStateStorage", encrypted_env ? "encrypted" : "plaintext");
auto set_db_opts = [encrypted_env] (internal::Writable<rocksdb::DBOptions>& db_opts) {
- db_opts.set(&rocksdb::DBOptions::create_if_missing, true);
- db_opts.set(&rocksdb::DBOptions::use_direct_io_for_flush_and_compaction, true);
- db_opts.set(&rocksdb::DBOptions::use_direct_reads, true);
+ minifi::internal::setCommonRocksDbOptions(db_opts);
if (encrypted_env) {
db_opts.set(&rocksdb::DBOptions::env, encrypted_env.get(), core::repository::EncryptionEq{});
} else {
@@ -81,7 +79,8 @@ void RocksDbStateStorage::onEnable() {
cf_opts.write_buffer_size = 8ULL << 20U;
cf_opts.min_write_buffer_number_to_merge = 1;
};
- db_ = minifi::internal::RocksDatabase::create(set_db_opts, set_cf_opts, directory_);
+ db_ = minifi::internal::RocksDatabase::create(set_db_opts, set_cf_opts, directory_,
+ minifi::internal::getRocksDbOptionsToOverride(configuration_, Configure::nifi_state_storage_rocksdb_options));
if (db_->open()) {
logger_->log_trace("Successfully opened RocksDB database at {}", directory_.c_str());
} else {
diff --git a/extensions/rocksdb-repos/database/RocksDatabase.cpp b/extensions/rocksdb-repos/database/RocksDatabase.cpp
index 78d31db18..c4a166a6b 100644
--- a/extensions/rocksdb-repos/database/RocksDatabase.cpp
+++ b/extensions/rocksdb-repos/database/RocksDatabase.cpp
@@ -29,7 +29,8 @@ namespace org::apache::nifi::minifi::internal {
std::shared_ptr<core::logging::Logger> RocksDatabase::logger_ = core::logging::LoggerFactory<RocksDatabase>::getLogger();
-std::unique_ptr<RocksDatabase> RocksDatabase::create(const DBOptionsPatch& db_options_patch, const ColumnFamilyOptionsPatch& cf_options_patch, const std::string& uri, RocksDbMode mode) {
+std::unique_ptr<RocksDatabase> RocksDatabase::create(const DBOptionsPatch& db_options_patch, const ColumnFamilyOptionsPatch& cf_options_patch, const std::string& uri,
+ const std::unordered_map<std::string, std::string>& db_config_override, RocksDbMode mode) {
const std::string scheme = "minifidb://";
logger_->log_info("Acquiring database handle '{}'", uri);
@@ -56,7 +57,7 @@ std::unique_ptr<RocksDatabase> RocksDatabase::create(const DBOptionsPatch& db_op
if (mode == RocksDbMode::ReadOnly) {
// no need to cache anything with read-only databases
- return std::make_unique<RocksDatabase>(std::make_shared<RocksDbInstance>(db_path, mode), db_column, db_options_patch, cf_options_patch);
+ return std::make_unique<RocksDatabase>(std::make_shared<RocksDbInstance>(db_path, mode), db_column, db_options_patch, cf_options_patch, db_config_override);
}
static std::mutex mtx;
@@ -74,12 +75,13 @@ std::unique_ptr<RocksDatabase> RocksDatabase::create(const DBOptionsPatch& db_op
logger_->log_info("Using previously opened rocksdb instance '{}'", db_path);
}
}
- return std::make_unique<RocksDatabase>(instance, db_column, db_options_patch, cf_options_patch);
+ return std::make_unique<RocksDatabase>(instance, db_column, db_options_patch, cf_options_patch, db_config_override);
}
-RocksDatabase::RocksDatabase(std::shared_ptr<RocksDbInstance> db, std::string column, const DBOptionsPatch& db_options_patch, const ColumnFamilyOptionsPatch& cf_options_patch)
+RocksDatabase::RocksDatabase(std::shared_ptr<RocksDbInstance> db, std::string column, const DBOptionsPatch& db_options_patch, const ColumnFamilyOptionsPatch& cf_options_patch,
+ const std::unordered_map<std::string, std::string>& db_config_override)
: column_(std::move(column)), db_(std::move(db)) {
- db_->registerColumnConfig(column_, db_options_patch, cf_options_patch);
+ db_->registerColumnConfig(column_, db_options_patch, cf_options_patch, db_config_override);
}
RocksDatabase::~RocksDatabase() {
diff --git a/extensions/rocksdb-repos/database/RocksDatabase.h b/extensions/rocksdb-repos/database/RocksDatabase.h
index acaa3a6d4..7bcb229e9 100644
--- a/extensions/rocksdb-repos/database/RocksDatabase.h
+++ b/extensions/rocksdb-repos/database/RocksDatabase.h
@@ -21,6 +21,7 @@
#include <memory>
#include <optional>
#include <string>
+#include <unordered_map>
#include "rocksdb/db.h"
#include "logging/Logger.h"
@@ -40,9 +41,11 @@ class RocksDatabase {
static std::unique_ptr<RocksDatabase> create(const DBOptionsPatch& db_options_patch,
const ColumnFamilyOptionsPatch& cf_options_patch,
const std::string& uri,
+ const std::unordered_map<std::string, std::string>& db_config_override,
RocksDbMode mode = RocksDbMode::ReadWrite);
- RocksDatabase(std::shared_ptr<RocksDbInstance> db, std::string column, const DBOptionsPatch& db_options_patch, const ColumnFamilyOptionsPatch& cf_options_patch);
+ RocksDatabase(std::shared_ptr<RocksDbInstance> db, std::string column, const DBOptionsPatch& db_options_patch, const ColumnFamilyOptionsPatch& cf_options_patch,
+ const std::unordered_map<std::string, std::string>& db_config_override);
RocksDatabase(const RocksDatabase&) = delete;
RocksDatabase(RocksDatabase&&) = delete;
diff --git a/extensions/rocksdb-repos/database/RocksDbInstance.cpp b/extensions/rocksdb-repos/database/RocksDbInstance.cpp
index 0bab95c1d..4c23e0ec8 100644
--- a/extensions/rocksdb-repos/database/RocksDbInstance.cpp
+++ b/extensions/rocksdb-repos/database/RocksDbInstance.cpp
@@ -42,9 +42,11 @@ void RocksDbInstance::invalidate(const std::lock_guard<std::mutex>&) {
impl_.reset();
}
-void RocksDbInstance::registerColumnConfig(const std::string& column, const DBOptionsPatch& db_options_patch, const ColumnFamilyOptionsPatch& cf_options_patch) {
+void RocksDbInstance::registerColumnConfig(const std::string& column, const DBOptionsPatch& db_options_patch, const ColumnFamilyOptionsPatch& cf_options_patch,
+ const std::unordered_map<std::string, std::string>& db_config_override) {
std::lock_guard<std::mutex> db_guard{mtx_};
logger_->log_trace("Registering column '{}' in database '{}'", column, db_name_);
+ db_config_override_ = db_config_override;
auto [_, inserted] = column_configs_.insert({column, ColumnConfig{.dbo_patch = db_options_patch, .cfo_patch = cf_options_patch}});
if (!inserted) {
throw std::runtime_error("Configuration is already registered for column '" + column + "'");
@@ -60,8 +62,14 @@ void RocksDbInstance::registerColumnConfig(const std::string& column, const DBOp
Writable<rocksdb::DBOptions> db_opts_writer(db_opts_copy);
if (db_options_patch) {
db_options_patch(db_opts_writer);
- if (db_opts_writer.isModified()) {
- logger_->log_trace("Requested a difference DBOptions than the one that was used to open the database");
+ rocksdb::ConfigOptions conf_options;
+ conf_options.sanity_level = rocksdb::ConfigOptions::kSanityLevelLooselyCompatible;
+ auto db_config_override_status = rocksdb::GetDBOptionsFromMap(conf_options, db_opts_copy, db_config_override_, &db_opts_copy);
+ if (!db_config_override_status.ok()) {
+ throw std::runtime_error("Failed to override RocksDB options from minifi.properties file: " + db_config_override_status.ToString());
+ }
+ if (db_opts_copy != db_options_) {
+ logger_->log_trace("Requested a different DBOptions than the one that was used to open the database");
return true;
}
}
@@ -130,7 +138,7 @@ std::optional<OpenRocksDb> RocksDbInstance::open(const std::string& column) {
}
db_options_ = rocksdb::DBOptions{};
std::vector<rocksdb::ColumnFamilyDescriptor> cf_descriptors;
- rocksdb::Status option_status = rocksdb::LoadLatestOptions(conf_options, db_name_, &db_options_, &cf_descriptors);
+ rocksdb::Status latest_option_status = rocksdb::LoadLatestOptions(conf_options, db_name_, &db_options_, &cf_descriptors);
{
// apply the database options patchers
Writable<rocksdb::DBOptions> db_options_writer(db_options_);
@@ -148,14 +156,19 @@ std::optional<OpenRocksDb> RocksDbInstance::open(const std::string& column) {
}
}
}
- if (option_status.ok()) {
+ auto db_config_override_status = rocksdb::GetDBOptionsFromMap(conf_options, db_options_, db_config_override_, &db_options_);
+ if (!db_config_override_status.ok()) {
+ logger_->log_error("Failed to override RocksDB options from minifi.properties file: {}", db_config_override_status.ToString());
+ return std::nullopt;
+ }
+ if (latest_option_status.ok()) {
logger_->log_trace("Found existing database '{}', checking compatibility", db_name_);
rocksdb::Status compat_status = rocksdb::CheckOptionsCompatibility(conf_options, db_name_, db_options_, cf_descriptors);
if (!compat_status.ok()) {
logger_->log_error("Incompatible database options: {}", compat_status.ToString());
return std::nullopt;
}
- } else if (option_status.IsNotFound()) {
+ } else if (latest_option_status.IsNotFound()) {
logger_->log_trace("Database at '{}' not found, creating", db_name_);
rocksdb::ColumnFamilyOptions default_cf_options;
if (auto it = column_configs_.find("default"); it != column_configs_.end()) {
@@ -164,8 +177,8 @@ std::optional<OpenRocksDb> RocksDbInstance::open(const std::string& column) {
}
}
cf_descriptors.emplace_back("default", default_cf_options);
- } else if (!option_status.ok()) {
- logger_->log_error("Couldn't query database '{}' for options: '{}'", db_name_, option_status.ToString());
+ } else if (!latest_option_status.ok()) {
+ logger_->log_error("Couldn't query database '{}' for options: '{}'", db_name_, latest_option_status.ToString());
return std::nullopt;
}
std::vector<rocksdb::ColumnFamilyHandle*> column_handles;
diff --git a/extensions/rocksdb-repos/database/RocksDbInstance.h b/extensions/rocksdb-repos/database/RocksDbInstance.h
index 182324958..b4796c4eb 100644
--- a/extensions/rocksdb-repos/database/RocksDbInstance.h
+++ b/extensions/rocksdb-repos/database/RocksDbInstance.h
@@ -61,11 +61,13 @@ class RocksDbInstance {
void invalidate(const std::lock_guard<std::mutex>&);
- void registerColumnConfig(const std::string& column, const DBOptionsPatch& db_options_patch, const ColumnFamilyOptionsPatch& cf_options_patch);
+ void registerColumnConfig(const std::string& column, const DBOptionsPatch& db_options_patch, const ColumnFamilyOptionsPatch& cf_options_patch,
+ const std::unordered_map<std::string, std::string>& db_config_override);
void unregisterColumnConfig(const std::string& column);
rocksdb::DBOptions db_options_;
const std::string db_name_;
+ std::unordered_map<std::string, std::string> db_config_override_;
const RocksDbMode mode_;
std::mutex mtx_;
diff --git a/extensions/rocksdb-repos/database/RocksDbUtils.cpp b/extensions/rocksdb-repos/database/RocksDbUtils.cpp
index d3834dfef..223cbb27e 100644
--- a/extensions/rocksdb-repos/database/RocksDbUtils.cpp
+++ b/extensions/rocksdb-repos/database/RocksDbUtils.cpp
@@ -51,4 +51,29 @@ std::optional<rocksdb::CompressionType> readConfiguredCompressionType(const std:
#endif
}
+
+void setCommonRocksDbOptions(Writable<rocksdb::DBOptions>& db_opts) {
+ db_opts.set(&rocksdb::DBOptions::create_if_missing, true);
+ db_opts.set(&rocksdb::DBOptions::use_direct_io_for_flush_and_compaction, true);
+ db_opts.set(&rocksdb::DBOptions::use_direct_reads, true);
+ db_opts.set(&rocksdb::DBOptions::keep_log_file_num, 5);
+}
+
+std::unordered_map<std::string, std::string> getRocksDbOptionsToOverride(const std::shared_ptr<Configure> &configuration, std::string_view custom_db_prefix) {
+ std::unordered_map<std::string, std::string> options;
+ const auto addOverrideOptions = [&configuration, &options](std::string_view prefix) {
+ if (prefix.empty()) {
+ return;
+ }
+ for (const auto& [key, value] : configuration->getProperties()) {
+ if (key.starts_with(prefix)) {
+ options[key.substr(prefix.size())] = value;
+ }
+ }
+ };
+ addOverrideOptions(Configuration::nifi_global_rocksdb_options);
+ addOverrideOptions(custom_db_prefix);
+ return options;
+}
+
} // namespace org::apache::nifi::minifi::internal
diff --git a/extensions/rocksdb-repos/database/RocksDbUtils.h b/extensions/rocksdb-repos/database/RocksDbUtils.h
index b5748c117..0deac2c85 100644
--- a/extensions/rocksdb-repos/database/RocksDbUtils.h
+++ b/extensions/rocksdb-repos/database/RocksDbUtils.h
@@ -22,6 +22,7 @@
#include <optional>
#include <memory>
#include <string>
+#include <unordered_map>
#include "rocksdb/db.h"
#include "rocksdb/options.h"
@@ -71,5 +72,7 @@ using DBOptionsPatch = std::function<void(Writable<rocksdb::DBOptions>&)>;
using ColumnFamilyOptionsPatch = std::function<void(rocksdb::ColumnFamilyOptions&)>;
std::optional<rocksdb::CompressionType> readConfiguredCompressionType(const std::shared_ptr<Configure> &configuration, const std::string& config_key);
+void setCommonRocksDbOptions(Writable<rocksdb::DBOptions>& db_opts);
+std::unordered_map<std::string, std::string> getRocksDbOptionsToOverride(const std::shared_ptr<Configure> &configuration, std::string_view custom_db_prefix);
} // namespace org::apache::nifi::minifi::internal
diff --git a/extensions/rocksdb-repos/tests/DBContentRepositoryTests.cpp b/extensions/rocksdb-repos/tests/DBContentRepositoryTests.cpp
index d46601930..5f9eb7b25 100644
--- a/extensions/rocksdb-repos/tests/DBContentRepositoryTests.cpp
+++ b/extensions/rocksdb-repos/tests/DBContentRepositoryTests.cpp
@@ -330,7 +330,7 @@ TEST_CASE("ProcessSession::read can read zero length flowfiles without crash (Ro
}
size_t getDbSize(const std::filesystem::path& dir) {
- auto db = minifi::internal::RocksDatabase::create({}, {}, dir.string());
+ auto db = minifi::internal::RocksDatabase::create({}, {}, dir.string(), {});
auto opendb = db->open();
REQUIRE(opendb);
diff --git a/extensions/rocksdb-repos/tests/RocksDBStreamTests.cpp b/extensions/rocksdb-repos/tests/RocksDBStreamTests.cpp
index 4b8e2563d..e9433e576 100644
--- a/extensions/rocksdb-repos/tests/RocksDBStreamTests.cpp
+++ b/extensions/rocksdb-repos/tests/RocksDBStreamTests.cpp
@@ -34,7 +34,7 @@ class RocksDBStreamTest : TestController {
auto set_cf_opts = [] (rocksdb::ColumnFamilyOptions& cf_opts) {
cf_opts.merge_operator = std::make_shared<core::repository::StringAppender>();
};
- db = minifi::internal::RocksDatabase::create(set_db_opts, set_cf_opts, dbPath);
+ db = minifi::internal::RocksDatabase::create(set_db_opts, set_cf_opts, dbPath, {});
REQUIRE(db->open());
}
diff --git a/extensions/rocksdb-repos/tests/RocksDBTests.cpp b/extensions/rocksdb-repos/tests/RocksDBTests.cpp
index 51cea8e7c..8a923a051 100644
--- a/extensions/rocksdb-repos/tests/RocksDBTests.cpp
+++ b/extensions/rocksdb-repos/tests/RocksDBTests.cpp
@@ -70,7 +70,7 @@ void new_db_opts(minifi::internal::Writable<rocksdb::DBOptions>& db_opts) {
}
TEST_CASE_METHOD(RocksDBTest, "Malformed database uri - Missing column name", "[rocksDBTest1]") {
- auto db = minifi::internal::RocksDatabase::create({}, {}, "minifidb://malformed");
+ auto db = minifi::internal::RocksDatabase::create({}, {}, "minifidb://malformed", {});
REQUIRE(!db);
REQUIRE(utils::verifyLogLinePresenceInPollTime(
std::chrono::seconds{1}, "Couldn't detect the column name in 'minifidb://malformed'"));
@@ -78,7 +78,7 @@ TEST_CASE_METHOD(RocksDBTest, "Malformed database uri - Missing column name", "[
TEST_CASE_METHOD(RocksDBTest, "Can write to default column", "[rocksDBTest2]") {
{
- auto db = minifi::internal::RocksDatabase::create(new_db_opts, {}, db_dir);
+ auto db = minifi::internal::RocksDatabase::create(new_db_opts, {}, db_dir, {});
auto opendb = db->open();
opendb->Put(rocksdb::WriteOptions{}, "fruit", "apple");
}
@@ -92,7 +92,7 @@ TEST_CASE_METHOD(RocksDBTest, "Can write to default column", "[rocksDBTest2]") {
TEST_CASE_METHOD(RocksDBTest, "Can write to specific column using the rocksdb uri scheme", "[rocksDBTest3]") {
{
- auto db = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/column_one");
+ auto db = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/column_one", {});
auto opendb = db->open();
opendb->Put(rocksdb::WriteOptions{}, "fruit", "apple");
}
@@ -106,11 +106,11 @@ TEST_CASE_METHOD(RocksDBTest, "Can write to specific column using the rocksdb ur
TEST_CASE_METHOD(RocksDBTest, "Can write to two specific columns at once", "[rocksDBTest4]") {
{
- auto db1 = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/column_one");
+ auto db1 = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/column_one", {});
auto opendb1 = db1->open();
REQUIRE(opendb1);
opendb1->Put(rocksdb::WriteOptions{}, "fruit", "apple");
- auto db2 = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/column_two");
+ auto db2 = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/column_two", {});
auto opendb2 = db2->open();
REQUIRE(opendb2);
opendb2->Put(rocksdb::WriteOptions{}, "animal", "penguin");
@@ -128,11 +128,11 @@ TEST_CASE_METHOD(RocksDBTest, "Can write to two specific columns at once", "[roc
TEST_CASE_METHOD(RocksDBTest, "Can write to the default and a specific column at once", "[rocksDBTest5]") {
{
- auto db1 = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/column_one");
+ auto db1 = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/column_one", {});
auto opendb1 = db1->open();
REQUIRE(opendb1);
opendb1->Put(rocksdb::WriteOptions{}, "fruit", "apple");
- auto db2 = minifi::internal::RocksDatabase::create(new_db_opts, {}, db_dir);
+ auto db2 = minifi::internal::RocksDatabase::create(new_db_opts, {}, db_dir, {});
auto opendb2 = db2->open();
REQUIRE(opendb2);
opendb2->Put(rocksdb::WriteOptions{}, "animal", "penguin");
@@ -149,13 +149,13 @@ TEST_CASE_METHOD(RocksDBTest, "Can write to the default and a specific column at
}
TEST_CASE_METHOD(RocksDBTest, "Logged if the options are incompatible with an existing column family", "[rocksDBTest6]") {
- auto db = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/column_one");
+ auto db = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/column_one", {});
REQUIRE(db->open());
// implicitly created the "default" column family, but with the default options
auto cf_opts = [] (rocksdb::ColumnFamilyOptions& cf_opts) {
cf_opts.merge_operator = std::make_shared<minifi::core::repository::StringAppender>();
};
- auto default_db = minifi::internal::RocksDatabase::create(new_db_opts, cf_opts, "minifidb://" + db_dir + "/default");
+ auto default_db = minifi::internal::RocksDatabase::create(new_db_opts, cf_opts, "minifidb://" + db_dir + "/default", {});
REQUIRE(default_db->open());
REQUIRE(utils::verifyLogLinePresenceInPollTime(
std::chrono::seconds{1}, "Could not determine if we definitely need to reopen or we are definitely safe, requesting reopen"));
@@ -170,16 +170,16 @@ TEST_CASE_METHOD(RocksDBTest, "Error is logged if different DBOptions are used",
db_opts.set(&rocksdb::DBOptions::create_if_missing, true);
db_opts.set(&rocksdb::DBOptions::manual_wal_flush, true);
};
- auto col_1 = minifi::internal::RocksDatabase::create(db_opt_1, {}, "minifidb://" + db_dir + "/column_one");
+ auto col_1 = minifi::internal::RocksDatabase::create(db_opt_1, {}, "minifidb://" + db_dir + "/column_one", {});
REQUIRE(col_1->open());
- auto col_2 = minifi::internal::RocksDatabase::create(db_opt_2, {}, "minifidb://" + db_dir + "/column_two");
+ auto col_2 = minifi::internal::RocksDatabase::create(db_opt_2, {}, "minifidb://" + db_dir + "/column_two", {});
REQUIRE_FALSE(col_2->open());
REQUIRE(utils::verifyLogLinePresenceInPollTime(
std::chrono::seconds{1}, "Conflicting database options requested for '" + db_dir + "'"));
}
TEST_CASE_METHOD(RocksDBTest, "Sanity check: merge fails without merge_operator", "[rocksDBTest8]") {
- auto db = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/col_one");
+ auto db = minifi::internal::RocksDatabase::create(new_db_opts, {}, "minifidb://" + db_dir + "/col_one", {});
REQUIRE(db);
auto opendb = db->open();
@@ -203,7 +203,7 @@ TEST_CASE_METHOD(RocksDBTest, "Column options are applied", "[rocksDBTest9]") {
SECTION("Implicit default column") {
db_uri = db_dir;
}
- auto db = minifi::internal::RocksDatabase::create(new_db_opts, cf_opts, db_uri);
+ auto db = minifi::internal::RocksDatabase::create(new_db_opts, cf_opts, db_uri, {});
REQUIRE(db);
auto opendb = db->open();
@@ -238,25 +238,25 @@ TEST_CASE_METHOD(RocksDBTest, "Error is logged if different encryption keys are
<< "encryption.key.two=" << "905D7B95EF44DC27C87FFBC4DFDE376DAE604D55DB2C5496DEEF5236362DE62E" << "\n";
auto db_opt_1 = createEncrSetter(home_dir, "one", "encryption.key.one");
- auto col_1 = minifi::internal::RocksDatabase::create(db_opt_1, {}, "minifidb://" + db_dir + "/column_one");
+ auto col_1 = minifi::internal::RocksDatabase::create(db_opt_1, {}, "minifidb://" + db_dir + "/column_one", {});
REQUIRE(col_1->open());
SECTION("Using the same encryption key is OK") {
auto db_opt_2 = createEncrSetter(home_dir, "two", "encryption.key.one");
- auto col_2 = minifi::internal::RocksDatabase::create(db_opt_2, {}, "minifidb://" + db_dir + "/column_two");
+ auto col_2 = minifi::internal::RocksDatabase::create(db_opt_2, {}, "minifidb://" + db_dir + "/column_two", {});
REQUIRE(col_2->open());
}
SECTION("Using different encryption key") {
auto db_opt_2 = createEncrSetter(home_dir, "two", "encryption.key.two");
- auto col_2 = minifi::internal::RocksDatabase::create(db_opt_2, {}, "minifidb://" + db_dir + "/column_two");
+ auto col_2 = minifi::internal::RocksDatabase::create(db_opt_2, {}, "minifidb://" + db_dir + "/column_two", {});
REQUIRE_FALSE(col_2->open());
REQUIRE(utils::verifyLogLinePresenceInPollTime(
std::chrono::seconds{1}, "Conflicting database options requested for '" + db_dir + "'"));
}
SECTION("Using no encryption key") {
- auto col_2 = minifi::internal::RocksDatabase::create(withDefaultEnv, {}, "minifidb://" + db_dir + "/column_two");
+ auto col_2 = minifi::internal::RocksDatabase::create(withDefaultEnv, {}, "minifidb://" + db_dir + "/column_two", {});
REQUIRE_FALSE(col_2->open());
REQUIRE(utils::verifyLogLinePresenceInPollTime(
std::chrono::seconds{1}, "Conflicting database options requested for '" + db_dir + "'"));
@@ -265,7 +265,7 @@ TEST_CASE_METHOD(RocksDBTest, "Error is logged if different encryption keys are
TEST_CASE_METHOD(RocksDBTest, "RocksDb works correctly on changed (but compatible) ColumnFamilyOptions change", "[rocksDBTest11]") {
{
- auto col1 = minifi::internal::RocksDatabase::create(new_db_opts, {}, db_dir);
+ auto col1 = minifi::internal::RocksDatabase::create(new_db_opts, {}, db_dir, {});
auto opendb = col1->open();
REQUIRE(opendb);
REQUIRE(opendb->Put({}, "fruit", "apple").ok());
@@ -275,7 +275,7 @@ TEST_CASE_METHOD(RocksDBTest, "RocksDb works correctly on changed (but compatibl
cf_opts.OptimizeForPointLookup(4);
};
- auto col2 = minifi::internal::RocksDatabase::create(new_db_opts, cf_opts, db_dir);
+ auto col2 = minifi::internal::RocksDatabase::create(new_db_opts, cf_opts, db_dir, {});
REQUIRE(col2);
{
auto opendb = col2->open();
diff --git a/extensions/rocksdb-repos/tests/RocksDBUtilsTests.cpp b/extensions/rocksdb-repos/tests/RocksDBUtilsTests.cpp
new file mode 100644
index 000000000..9b7f662f5
--- /dev/null
+++ b/extensions/rocksdb-repos/tests/RocksDBUtilsTests.cpp
@@ -0,0 +1,46 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <unordered_map>
+#include <string>
+
+#include "TestBase.h"
+#include "Catch.h"
+#include "../database/RocksDbUtils.h"
+#include "properties/Configure.h"
+
+namespace org::apache::nifi::minifi::test {
+
+TEST_CASE("Get global RocksDB options to override", "[rocksdbutils]") {
+ std::unordered_map<std::string, std::string> expected_options {
+ {"keep_log_file_num", "2"},
+ {"table_cache_numshardbits", "456"},
+ {"create_if_missing", "false"}
+ };
+
+ auto config = std::make_shared<minifi::Configure>();
+ config->set("nifi.c2.rest.url", "http://localhost:8080");
+ config->set("nifi.global.rocksdb.options.table_cache_numshardbits", "456");
+ config->set("nifi.global.rocksdb.options.keep_log_file_num", "123");
+ config->set("nifi.flowfile.repository.rocksdb.options.keep_log_file_num", "2");
+ config->set("nifi.flowfile.repository.rocksdb.options.create_if_missing", "false");
+ config->set("nifi.c2.enable", "true");
+ auto result = org::apache::nifi::minifi::internal::getRocksDbOptionsToOverride(config, minifi::Configure::nifi_flowfile_repository_rocksdb_options);
+ REQUIRE(result == expected_options);
+}
+
+} // namespace org::apache::nifi::minifi::test
diff --git a/libminifi/include/properties/Configuration.h b/libminifi/include/properties/Configuration.h
index 579adca34..e50bc3115 100644
--- a/libminifi/include/properties/Configuration.h
+++ b/libminifi/include/properties/Configuration.h
@@ -36,6 +36,11 @@ class Configuration : public Properties {
Configuration() : Properties("MiNiFi configuration") {}
static constexpr const char *nifi_volatile_repository_options = "nifi.volatile.repository.options.";
+ static constexpr const char *nifi_global_rocksdb_options = "nifi.global.rocksdb.options.";
+ static constexpr const char *nifi_flowfile_repository_rocksdb_options = "nifi.flowfile.repository.rocksdb.options.";
+ static constexpr const char *nifi_content_repository_rocksdb_options = "nifi.content.repository.rocksdb.options.";
+ static constexpr const char *nifi_provenance_repository_rocksdb_options = "nifi.provenance.repository.rocksdb.options.";
+ static constexpr const char *nifi_state_storage_rocksdb_options = "nifi.state.storage.rocksdb.options.";
// nifi.flow.configuration.file
static constexpr const char *nifi_default_directory = "nifi.default.directory";
diff --git a/libminifi/include/properties/Properties.h b/libminifi/include/properties/Properties.h
index e86ec4b5e..6d427d64f 100644
--- a/libminifi/include/properties/Properties.h
+++ b/libminifi/include/properties/Properties.h
@@ -139,7 +139,6 @@ class Properties {
std::filesystem::path getFilePath() const;
- protected:
std::map<std::string, std::string> getProperties() const;
private:
diff --git a/thirdparty/rocksdb/dboptions_equality_operator.patch b/thirdparty/rocksdb/dboptions_equality_operator.patch
new file mode 100644
index 000000000..2efb91ea0
--- /dev/null
+++ b/thirdparty/rocksdb/dboptions_equality_operator.patch
@@ -0,0 +1,21 @@
+diff --git a/include/rocksdb/options.h b/include/rocksdb/options.h
+index ae5ed2c26..0038c6bff 100644
+--- a/include/rocksdb/options.h
++++ b/include/rocksdb/options.h
+@@ -397,6 +397,7 @@ struct DbPath {
+
+ DbPath() : target_size(0) {}
+ DbPath(const std::string& p, uint64_t t) : path(p), target_size(t) {}
++ bool operator==(const DbPath& other) const = default;
+ };
+
+ extern const char* kHostnameForDbHostId;
+@@ -1008,6 +1009,8 @@ struct DBOptions {
+ // Create DBOptions from Options
+ explicit DBOptions(const Options& options);
+
++ bool operator==(const DBOptions& other) const = default;
++
+ void Dump(Logger* log) const;
+
+ // Allows OS to incrementally sync files to disk while they are being
(nifi-minifi-cpp) 06/06: MINIFICPP-1757 remove problematic linker flags on macos/gcc
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martinzink pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit fc9e50769378744bdb19bd706aa2a075878f5614
Author: Marton Szasz <sz...@apache.org>
AuthorDate: Tue Mar 12 13:53:21 2024 +0100
MINIFICPP-1757 remove problematic linker flags on macos/gcc
Closes #1741
Signed-off-by: Martin Zink <ma...@apache.org>
---
cmake/Extensions.cmake | 2 +-
controller/CMakeLists.txt | 2 +-
encrypt-config/CMakeLists.txt | 2 +-
minifi_main/CMakeLists.txt | 2 +-
4 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/cmake/Extensions.cmake b/cmake/Extensions.cmake
index 176b08f97..41d0cc9a9 100644
--- a/cmake/Extensions.cmake
+++ b/cmake/Extensions.cmake
@@ -70,7 +70,7 @@ macro(register_extension extension-name extension-display-name extension-guard d
if(WIN32)
install(TARGETS ${extension-name} RUNTIME DESTINATION extensions COMPONENT ${component-name})
else()
- if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
+ if("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" AND NOT APPLE)
target_link_options(${extension-name} PRIVATE "-Wl,--disable-new-dtags")
endif()
if (APPLE)
diff --git a/controller/CMakeLists.txt b/controller/CMakeLists.txt
index 4eaa1cf07..1e632584c 100644
--- a/controller/CMakeLists.txt
+++ b/controller/CMakeLists.txt
@@ -48,7 +48,7 @@ set_target_properties(minificontroller PROPERTIES
RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin")
if (NOT WIN32)
- if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
+ if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" AND NOT APPLE)
target_link_options(minificontroller PRIVATE "-Wl,--disable-new-dtags")
endif()
if (APPLE)
diff --git a/encrypt-config/CMakeLists.txt b/encrypt-config/CMakeLists.txt
index 1b8129fb8..cfa436564 100644
--- a/encrypt-config/CMakeLists.txt
+++ b/encrypt-config/CMakeLists.txt
@@ -31,7 +31,7 @@ set_target_properties(encrypt-config PROPERTIES OUTPUT_NAME encrypt-config)
set_target_properties(encrypt-config PROPERTIES RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin")
if (NOT WIN32)
- if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
+ if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" AND NOT APPLE)
target_link_options(encrypt-config PRIVATE "-Wl,--disable-new-dtags")
endif()
if (APPLE)
diff --git a/minifi_main/CMakeLists.txt b/minifi_main/CMakeLists.txt
index 9039d146c..337b472ea 100644
--- a/minifi_main/CMakeLists.txt
+++ b/minifi_main/CMakeLists.txt
@@ -83,7 +83,7 @@ else()
endif()
if (NOT WIN32)
- if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
+ if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU" AND NOT APPLE)
target_link_options(minifiexe PRIVATE "-Wl,--disable-new-dtags")
endif()
if (APPLE)
(nifi-minifi-cpp) 05/06: MINIFICPP-2192 Document user requirements for Windows service installation
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martinzink pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit dd8e54e517e52ff05142c34490d159dbb2f8cb72
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Tue Mar 12 13:52:59 2024 +0100
MINIFICPP-2192 Document user requirements for Windows service installation
Closes #1737
Signed-off-by: Martin Zink <ma...@apache.org>
---
Windows.md | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/Windows.md b/Windows.md
index 82fb21a8d..c1f7ca7ee 100644
--- a/Windows.md
+++ b/Windows.md
@@ -126,3 +126,18 @@ copy minifi_main\Release\minifi.exe minifi_main\
cpack
ctest -C Release
```
+
+## Using MiNiFi C++ as a Windows service
+
+Building and packaging MiNiFi C++ results in an MSI installer. This installer can be used to install MiNiFi C++ as a Windows service. Depending on the build options, specific extensions can be selected for installation.
+
+The installer also provides an option to specify a service account to be used for the Windows service. By default, the `LocalSystem` account is used. If you want to specify a different account, make sure to provide the account name in the `DOMAIN\username` format.
+
+**NOTE:** To start the Windows service using the specified account, the account must have the `Log on as a service` right. If this right is missing, the following error event will appear in the system logs:
+
+```
+The Apache NiFi MiNiFi service was unable to log on as .\username with the currently configured password due to the following error:
+Logon failure: the user has not been granted the requested logon type at this computer.
+```
+
+This right can be granted using the Local Security Policy tool (`secpol.msc`). Navigate to `Local Policies` -> `User Rights Assignment` -> `Log on as a service` and add the account.
(nifi-minifi-cpp) 01/06: MINIFICPP-2296 - Install service on macOS
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martinzink pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 3776bd071c3728c438a9841eba5510b2dfa7cd75
Author: Adam Debreceni <ad...@apache.org>
AuthorDate: Tue Mar 12 13:50:49 2024 +0100
MINIFICPP-2296 - Install service on macOS
Closes #1726
Signed-off-by: Martin Zink <ma...@apache.org>
---
bin/minifi.plist | 25 +++++++++++
bin/minifi.sh | 128 +++++++++++++++++++++++++++++++++++++++++++------------
2 files changed, 126 insertions(+), 27 deletions(-)
diff --git a/bin/minifi.plist b/bin/minifi.plist
new file mode 100644
index 000000000..567229c6c
--- /dev/null
+++ b/bin/minifi.plist
@@ -0,0 +1,25 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+ <key>Label</key>
+ <string>org.apache.nifi.minifi</string>
+ <key>ProgramArguments</key>
+ <array>
+ <string>/opt/minifi-cpp/bin/minifi</string>
+ </array>
+ <key>EnvironmentVariables</key>
+ <dict>
+ <key>MINIFI_HOME</key>
+ <string>/opt/minifi-cpp</string>
+ </dict>
+ <key>RunAtLoad</key>
+ <true/>
+ <key>KeepAlive</key>
+ <false/>
+ <key>StandardOutPath</key>
+ <string>/var/log/minifi.log</string>
+ <key>StandardErrorPath</key>
+ <string>/var/log/minifi.err</string>
+</dict>
+</plist>
diff --git a/bin/minifi.sh b/bin/minifi.sh
index 33a395a53..b3f15ab68 100755
--- a/bin/minifi.sh
+++ b/bin/minifi.sh
@@ -54,18 +54,35 @@ detectOS() {
export LDR_CNTRL=MAXDATA=0xB0000000@DSA
echo ${LDR_CNTRL}
fi
+
+ if [ "${cygwin}" = "true" ]; then
+ echo 'Apache MiNiFi as a service is not supported on Cygwin.'
+ exit 1
+ fi
}
-install() {
- detectOS
- if [ "${darwin}" = "true" ] || [ "${cygwin}" = "true" ]; then
- echo 'Installing Apache MiNiFi as a service is not supported on OS X or Cygwin.'
+install_macos() {
+ target_dir="/Library/LaunchDaemons"
+ echo "Installing MiNiFi as a LaunchDaemon to ${target_dir}"
+ cp -fv "${bin_dir}"/minifi.plist "${target_dir}"
+ chmod 644 "${target_dir}"/minifi.plist
+ sed -i '' "s|/opt/minifi-cpp|${MINIFI_HOME}|g" "${target_dir}"/minifi.plist
+ launchctl load -w "${target_dir}"/minifi.plist
+}
+
+uninstall_macos() {
+ launchctl unload "/Library/LaunchDaemons/minifi.plist" 2>/dev/null
+ rm -fv "/Library/LaunchDaemons/minifi.plist" || :
+}
+
+check_service_installed_macos() {
+ if [ ! -f "/Library/LaunchDaemons/minifi.plist" ]; then
+ echo "MiNiFi is not installed as a service. Please run 'minifi.sh install' first."
exit 1
fi
+}
- echo "Uninstalling any previous versions"
- uninstall
-
+install_linux() {
target_dir="/usr/local/lib/systemd/system"
mkdir -p "${target_dir}" || die "Unable to create ${target_dir}. Cannot install MiNiFi as a service."
@@ -77,13 +94,7 @@ install() {
systemctl enable minifi.service
}
-uninstall() {
- detectOS
- if [ "${darwin}" = "true" ] || [ "${cygwin}" = "true" ]; then
- echo 'Apache MiNiFi as a service is not supported on OS X or Cygwin.'
- exit 1
- fi
-
+uninstall_linux() {
# Uninstall legacy init.d service files, if exists
rm -fv "/etc/init.d/minifi" || :
rm -fv "/etc/rc2.d/S65minifi" || :
@@ -94,39 +105,102 @@ uninstall() {
systemctl daemon-reload
}
-check_service_installed() {
+check_service_installed_linux() {
if [ ! -f "/usr/local/lib/systemd/system/minifi.service" ]; then
echo "MiNiFi is not installed as a service. Please run 'minifi.sh install' first."
exit 1
fi
}
+install() {
+ echo "Uninstalling any previous versions"
+ uninstall
+
+ detectOS
+ if [ "${darwin}" = "true" ]; then
+ install_macos
+ else
+ install_linux
+ fi
+}
+
+uninstall() {
+ detectOS
+ if [ "${darwin}" = "true" ]; then
+ uninstall_macos
+ else
+ uninstall_linux
+ fi
+}
+
+check_service_installed() {
+ detectOS
+ if [ "${darwin}" = "true" ]; then
+ check_service_installed_macos
+ else
+ check_service_installed_linux
+ fi
+}
+
+start_service() {
+ check_service_installed
+ if [ "${darwin}" = "true" ]; then
+ launchctl start org.apache.nifi.minifi
+ else
+ systemctl start minifi.service
+ fi
+ echo "MiNiFi started"
+}
+
+stop_service() {
+ check_service_installed
+ if [ "${darwin}" = "true" ]; then
+ launchctl stop org.apache.nifi.minifi
+ else
+ systemctl stop minifi.service
+ fi
+ echo "MiNiFi stopped"
+}
+
+restart_service() {
+ check_service_installed
+ if [ "${darwin}" = "true" ]; then
+ launchctl stop org.apache.nifi.minifi
+ launchctl start org.apache.nifi.minifi
+ else
+ systemctl restart minifi.service
+ fi
+ echo "MiNiFi restarted"
+}
+
+status_service() {
+ check_service_installed
+ if [ "${darwin}" = "true" ]; then
+ launchctl list | grep org.apache.nifi.minifi
+ else
+ systemctl status minifi.service
+ fi
+}
+
case "$1" in
start)
- check_service_installed
- systemctl start minifi.service
- echo "MiNiFi started"
+ start_service
;;
stop)
- check_service_installed
- systemctl stop minifi.service
- echo "MiNiFi stopped"
+ stop_service
;;
run)
exec "${minifi_executable}"
;;
status)
- check_service_installed
- systemctl status minifi.service
+ status_service
;;
restart)
- check_service_installed
- systemctl restart minifi.service
- echo "MiNiFi restarted"
+ restart_service
;;
install)
install "$@"
- echo "Service minifi installed. Please start it using 'minifi.sh start' or 'systemctl start minifi.service'"
+ echo "Service minifi installed. Please start it using 'minifi.sh start'"
;;
uninstall)
uninstall "$@"
(nifi-minifi-cpp) 02/06: MINIFICPP-2302 Upgrade github actions to Node.js 20 (or latest available) versions
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martinzink pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 03b504ef1c587d59ce230ac1f78ad5e1ae04dc62
Author: Ferenc Gerlits <fg...@gmail.com>
AuthorDate: Tue Mar 12 13:51:27 2024 +0100
MINIFICPP-2302 Upgrade github actions to Node.js 20 (or latest available) versions
Closes #1730
Signed-off-by: Martin Zink <ma...@apache.org>
---
.github/workflows/ci.yml | 68 ++++++++++++++++++++++++------------------------
1 file changed, 34 insertions(+), 34 deletions(-)
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 52e111703..97fbb76ed 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -13,9 +13,9 @@ jobs:
timeout-minutes: 180
steps:
- id: checkout
- uses: actions/checkout@v3
+ uses: actions/checkout@v4
- name: cache restore
- uses: actions/cache/restore@v3
+ uses: actions/cache/restore@v4
with:
path: ${{ env.CCACHE_DIR }}
key: macos-xcode-ccache-${{github.ref}}-${{github.sha}}
@@ -42,7 +42,7 @@ jobs:
# CPPFLAGS are not recognized by cmake, so we have to force them to CFLAGS and CXXFLAGS to have flex 2.6 working
./bootstrap.sh -e -t && cd build && cmake -G Ninja -DCMAKE_BUILD_TYPE=Release -DCI_BUILD=ON -DCMAKE_C_FLAGS="${CPPFLAGS} ${CFLAGS}" -DCMAKE_CXX_FLAGS="${CPPFLAGS} ${CXXFLAGS}" -DENABLE_PYTHON_SCRIPTING=ON -DENABLE_LUA_SCRIPTING=ON -DENABLE_SQL=ON -DUSE_REAL_ODBC_TEST_DRIVER=ON -DENABLE_AZURE=ON -DENABLE_GCP=ON -DENABLE_OPENCV=ON -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_RULE_MESSAGES=OFF -DSTRICT_GSL_CHECKS=AUDIT -DFAIL_ON_WARNINGS=ON .. && cmake --build . --parallel 4
- name: cache save
- uses: actions/cache/save@v3
+ uses: actions/cache/save@v4
if: always()
with:
path: ${{ env.CCACHE_DIR }}
@@ -59,12 +59,12 @@ jobs:
if: ${{ failure() && steps.test.conclusion == 'failure' }}
run: |
if [ "$(ls -A /cores)" ]; then echo "CORES_EXIST=true" >> $GITHUB_ENV; fi
- - uses: actions/upload-artifact@v3.1.2
+ - uses: actions/upload-artifact@v4
if: ${{ failure() && env.CORES_EXIST == 'true' }}
with:
name: macos-coredumps
path: /cores
- - uses: actions/upload-artifact@v3.1.2
+ - uses: actions/upload-artifact@v4
if: ${{ failure() && env.CORES_EXIST == 'true' }}
with:
name: macos-binaries
@@ -79,9 +79,9 @@ jobs:
- name: Support longpaths
run: git config --system core.longpaths true
- id: checkout
- uses: actions/checkout@v3
+ uses: actions/checkout@v4
- name: sccache cache restore
- uses: actions/cache/restore@v3
+ uses: actions/cache/restore@v4
with:
path: ~/AppData/Local/Mozilla/sccache/cache
key: ${{ runner.os }}-2022-sccache-${{ github.ref }}-${{ github.sha }}
@@ -90,11 +90,11 @@ jobs:
${{ runner.os }}-2022-sccache-refs/heads/main
${{ runner.os }}-2022-sccache
- name: Run sccache-cache
- uses: mozilla-actions/sccache-action@v0.0.3
+ uses: mozilla-actions/sccache-action@v0.0.4
- name: Install ninja-build tool
- uses: seanmiddleditch/gha-setup-ninja@v3
+ uses: seanmiddleditch/gha-setup-ninja@v4
- name: Set up Python
- uses: actions/setup-python@v4
+ uses: actions/setup-python@v5
with:
python-version: '3.11'
- name: Set up Lua
@@ -119,7 +119,7 @@ jobs:
sccache --show-stats
shell: cmd
- name: sccache cache save
- uses: actions/cache/save@v3
+ uses: actions/cache/save@v4
if: always()
with:
path: ~/AppData/Local/Mozilla/sccache/cache
@@ -136,9 +136,9 @@ jobs:
timeout-minutes: 120
steps:
- id: checkout
- uses: actions/checkout@v3
+ uses: actions/checkout@v4
- name: cache restore
- uses: actions/cache/restore@v3
+ uses: actions/cache/restore@v4
with:
path: ${{ env.CCACHE_DIR }}
key: ubuntu-20.04-ccache-${{github.ref}}-${{github.sha}}
@@ -162,7 +162,7 @@ jobs:
-DENABLE_LUA_SCRIPTING=OFF -DENABLE_PYTHON_SCRIPTING=OFF -DENABLE_MQTT=OFF -DENABLE_ELASTICSEARCH=OFF -DENABLE_KUBERNETES=OFF -DENABLE_OPC=OFF ..
make -j$(nproc) VERBOSE=1
- name: cache save
- uses: actions/cache/save@v3
+ uses: actions/cache/save@v4
if: always()
with:
path: ${{ env.CCACHE_DIR }}
@@ -177,12 +177,12 @@ jobs:
if: ${{ failure() && steps.test.conclusion == 'failure' }}
run: |
if [ "$(ls -A /var/lib/apport/coredump/)" ]; then echo "CORES_EXIST=true" >> $GITHUB_ENV; fi
- - uses: actions/upload-artifact@v3.1.2
+ - uses: actions/upload-artifact@v4
if: ${{ failure() && env.CORES_EXIST == 'true' }}
with:
name: ubuntu-coredumps
path: /var/lib/apport/coredump/
- - uses: actions/upload-artifact@v3.1.2
+ - uses: actions/upload-artifact@v4
if: ${{ failure() && env.CORES_EXIST == 'true' }}
with:
name: ubuntu-binaries
@@ -193,9 +193,9 @@ jobs:
timeout-minutes: 240
steps:
- id: checkout
- uses: actions/checkout@v3
+ uses: actions/checkout@v4
- name: cache restore
- uses: actions/cache/restore@v3
+ uses: actions/cache/restore@v4
with:
path: ${{ env.CCACHE_DIR }}
key: ubuntu-22.04-all-clang-ccache-${{github.ref}}-${{github.sha}}
@@ -238,7 +238,7 @@ jobs:
-DENABLE_GRAFANA_LOKI=ON -DCMAKE_EXPORT_COMPILE_COMMANDS=ON ..
cmake --build . --parallel $(nproc)
- id: cache_save
- uses: actions/cache/save@v3
+ uses: actions/cache/save@v4
if: always()
with:
path: ${{ env.CCACHE_DIR }}
@@ -256,7 +256,7 @@ jobs:
- id: flake8_check
run: cd build && make flake8
- id: files
- uses: Ana06/get-changed-files@v2.2.0
+ uses: Ana06/get-changed-files@v2.3.0
continue-on-error: true
- name: clang-tidy
run: |
@@ -278,12 +278,12 @@ jobs:
if: ${{ failure() && steps.test.conclusion == 'failure' }}
run: |
if [ "$(ls -A /var/lib/apport/coredump/)" ]; then echo "CORES_EXIST=true" >> $GITHUB_ENV; fi
- - uses: actions/upload-artifact@v3.1.2
+ - uses: actions/upload-artifact@v4
if: ${{ failure() && env.CORES_EXIST == 'true' }}
with:
name: ubuntu-clang-coredumps
path: /var/lib/apport/coredump/
- - uses: actions/upload-artifact@v3.1.2
+ - uses: actions/upload-artifact@v4
if: ${{ failure() && env.CORES_EXIST == 'true' }}
with:
name: ubuntu-clang-binaries
@@ -294,9 +294,9 @@ jobs:
timeout-minutes: 180
steps:
- id: checkout
- uses: actions/checkout@v3
+ uses: actions/checkout@v4
- name: cache restore
- uses: actions/cache/restore@v3
+ uses: actions/cache/restore@v4
with:
path: ${{ env.CCACHE_DIR }}
key: centos-ccache-${{github.ref}}-${{github.sha}}
@@ -327,7 +327,7 @@ jobs:
-DENABLE_USB_CAMERA=ON -DENABLE_PYTHON_SCRIPTING=ON -DENABLE_LUA_SCRIPTING=ON -DENABLE_KUBERNETES=ON -DENABLE_GCP=ON -DENABLE_PROCFS=ON -DENABLE_PROMETHEUS=ON \
-DENABLE_ELASTICSEARCH=ON -DENABLE_GRAFANA_LOKI=ON -DDOCKER_SKIP_TESTS=OFF -DDOCKER_BUILD_ONLY=ON -DDOCKER_CCACHE_DUMP_LOCATION=${{ env.CCACHE_DIR }} .. && make centos-test
- name: cache save
- uses: actions/cache/save@v3
+ uses: actions/cache/save@v4
if: always()
with:
path: ${{ env.CCACHE_DIR }}
@@ -348,12 +348,12 @@ jobs:
docker cp centos-test:/opt/minifi/build/bin /tmp
sudo chmod -R 777 /cores
fi
- - uses: actions/upload-artifact@v3.1.2
+ - uses: actions/upload-artifact@v4
if: ${{ failure() && env.CORES_EXIST == 'true' }}
with:
name: centos-crashes
path: /cores
- - uses: actions/upload-artifact@v3.1.2
+ - uses: actions/upload-artifact@v4
if: ${{ failure() && env.CORES_EXIST == 'true' }}
with:
name: centos-binaries
@@ -364,9 +364,9 @@ jobs:
timeout-minutes: 180
steps:
- id: checkout
- uses: actions/checkout@v3
+ uses: actions/checkout@v4
- name: cache restore
- uses: actions/cache/restore@v3
+ uses: actions/cache/restore@v4
with:
path: ${{ env.CCACHE_DIR }}
key: docker-ccache-${{github.ref}}-${{github.sha}}
@@ -380,7 +380,7 @@ jobs:
cmake ${DOCKER_CMAKE_FLAGS} -DDOCKER_CCACHE_DUMP_LOCATION=${{ env.CCACHE_DIR }} ..
make docker
- name: cache save
- uses: actions/cache/save@v3
+ uses: actions/cache/save@v4
if: always()
with:
path: ${{ env.CCACHE_DIR }}
@@ -388,7 +388,7 @@ jobs:
- name: Save docker image
run: cd build && docker save -o minifi_docker.tar apacheminificpp:$(grep CMAKE_PROJECT_VERSION:STATIC CMakeCache.txt | cut -d "=" -f2)
- name: Upload artifact
- uses: actions/upload-artifact@v3
+ uses: actions/upload-artifact@v4
with:
name: minifi_docker
path: build/minifi_docker.tar
@@ -399,7 +399,7 @@ jobs:
timeout-minutes: 180
steps:
- id: checkout
- uses: actions/checkout@v3
+ uses: actions/checkout@v4
- id: run_cmake
name: Run CMake
run: |
@@ -407,7 +407,7 @@ jobs:
cd build
cmake ${DOCKER_CMAKE_FLAGS} ..
- name: Download artifact
- uses: actions/download-artifact@v3
+ uses: actions/download-artifact@v4
with:
name: minifi_docker
path: build
@@ -436,7 +436,7 @@ jobs:
list-suites: 'failed'
- name: Upload artifact
if: failure()
- uses: actions/upload-artifact@v3
+ uses: actions/upload-artifact@v4
with:
name: behavex_output
path: build/behavex_output
(nifi-minifi-cpp) 04/06: MINIFICPP-2307 Fix libsodium url parameter passing for CMake 3.29
Posted by ma...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
martinzink pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 5d9cd59a96ea2166b13b000735e0cef1b08a7976
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Tue Mar 12 13:52:31 2024 +0100
MINIFICPP-2307 Fix libsodium url parameter passing for CMake 3.29
Closes #1735
Signed-off-by: Martin Zink <ma...@apache.org>
---
cmake/BundledLibSodium.cmake | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/cmake/BundledLibSodium.cmake b/cmake/BundledLibSodium.cmake
index 3a3daf410..98d05b737 100644
--- a/cmake/BundledLibSodium.cmake
+++ b/cmake/BundledLibSodium.cmake
@@ -48,7 +48,7 @@ function(use_bundled_libsodium SOURCE_DIR BINARY_DIR)
if (WIN32)
ExternalProject_Add(
libsodium-external
- URL "${LIBSODIUM_OFFICIAL_MIRROR_URL} ${LIBSODIUM_GITHUB_MIRROR_URL} ${LIBSODIUM_GENTOO_MIRROR_URL}"
+ URL "${LIBSODIUM_OFFICIAL_MIRROR_URL}" "${LIBSODIUM_GITHUB_MIRROR_URL}" "${LIBSODIUM_GENTOO_MIRROR_URL}"
URL_HASH ${LIBSODIUM_URL_HASH}
SOURCE_DIR "${BINARY_DIR}/thirdparty/libsodium-src"
LIST_SEPARATOR % # This is needed for passing semicolon-separated lists
@@ -62,7 +62,7 @@ function(use_bundled_libsodium SOURCE_DIR BINARY_DIR)
ExternalProject_Add(
libsodium-external
- URL "${LIBSODIUM_OFFICIAL_MIRROR_URL} ${LIBSODIUM_GITHUB_MIRROR_URL} ${LIBSODIUM_GENTOO_MIRROR_URL}"
+ URL "${LIBSODIUM_OFFICIAL_MIRROR_URL}" "${LIBSODIUM_GITHUB_MIRROR_URL}" "${LIBSODIUM_GENTOO_MIRROR_URL}"
URL_HASH ${LIBSODIUM_URL_HASH}
BUILD_IN_SOURCE true
SOURCE_DIR "${BINARY_DIR}/thirdparty/libsodium-src"