You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by GitBox <gi...@apache.org> on 2021/07/22 13:09:35 UTC
[GitHub] [nifi-minifi-cpp] adamdebreceni opened a new pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
adamdebreceni opened a new pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r679191844
##########
File path: libminifi/test/script-tests/ExecutePythonProcessorTests.cpp
##########
@@ -119,7 +118,7 @@ class SimplePythonFlowFileTransferTest : public ExecutePythonProcessorTestBase {
const std::string output_dir = createTempDir(testController_.get());
auto executePythonProcessor = plan_->addProcessor("ExecutePythonProcessor", "executePythonProcessor");
- plan_->setProperty(executePythonProcessor, org::apache::nifi::minifi::python::processors::ExecutePythonProcessor::ScriptFile.getName(), getScriptFullPath("stateful_processor.py"));
+ plan_->setProperty(executePythonProcessor, "Script File", getScriptFullPath("stateful_processor.py"));
Review comment:
tried it, but since it uses pybind11 (which itself has hidden sprinkled all around) it wouldn't budge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680764621
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
> is the http-curl-extension not an extension? do we not initialize/deinitialize the extension?
It is, but it's already modeled by the cmake target and the resulting dynamic library. This is about initializing and deinitializing, and this doesn't need a class. All symbols of a shared object behave like members of a class.
> ExtensionManager cannot [...] query the name of the extension it is initializing
It always has the filename of the library it is loading, that can be used as a name. I don't think we need more than that.
I'm thinking about a model similar to GCC's constructor and destructor function attributes, but done in a portable way, since we control the loading code here. https://stackoverflow.com/a/2053078
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680720629
##########
File path: libminifi/include/utils/Export.h
##########
@@ -15,14 +15,21 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include "TemplateLoader.h"
-#include "core/FlowConfiguration.h"
-bool TemplateFactory::added = core::FlowConfiguration::add_static_func("createTemplateFactory");
+#pragma once
-extern "C" {
-
-void *createTemplateFactory(void) {
- return new TemplateFactory();
-}
-}
+#ifdef WIN32
+ #ifdef LIBMINIFI
+ #define MINIFIAPI __declspec(dllexport)
+ #else
+ #define MINIFIAPI __declspec(dllimport)
+ #endif
+ #ifdef MODULE_NAME
+ #define EXTENSIONAPI __declspec(dllexport)
+ #else
+ #define EXTENSIONAPI __declspec(dllimport)
+ #endif
+#else
+ #define MINIFIAPI
+ #define EXTENSIONAPI
+#endif
Review comment:
agree that sharing export macros is suboptimal, possibly in a separate PR, now WINDOWS_EXPORT_ALL_SYMBOLS does most of the heavy lifting, moving to manually annotating all classes and static members (mostly for the sake of tests) could be a significant effort
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r693909181
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
Review comment:
The only was to signal an error from constructors is through exceptions, so it's hard to avoid them there. But in general I think exceptions are a great way to signal error.
Enforcing verification criteria in the constructor would allow to have the result of future verification be a class invariant, thus reducing the class state. Reducing state means simpler code and less opportunity to forget handling a special case.
My exception usage preferences are quite the opposite of yours: [Using exceptions for errors only](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#e3-use-exceptions-for-error-handling-only), [including establishing invariants](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#e5-let-a-constructor-establish-an-invariant-and-throw-if-it-cannot), but not part of the happy path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r679838081
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
Review comment:
I would simplify this by only saying that them must go in source files. You could add it to the first paragraph as well.
##########
File path: conf/minifi.properties
##########
@@ -20,6 +20,9 @@ nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=100 millis
+# Comma separated path for the extension libraries. Relative path is relative to the minifi executable.
+nifi.extension.path=../extensions/*
Review comment:
Can we make this relative to MINIFI_HOME instead? I think it would be easier to use.
##########
File path: libminifi/test/script-tests/ExecutePythonProcessorTests.cpp
##########
@@ -53,7 +53,6 @@ class ExecutePythonProcessorTestBase {
testController_.reset(new TestController());
plan_ = testController_->createPlan();
logTestController_.setDebug<TestPlan>();
- logTestController_.setDebug<minifi::python::processors::ExecutePythonProcessor>();
Review comment:
Is there a reason for this removal?
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
Could we do that without subclassing? For example by registering extension init and deinit functions as needed instead of a class. Or registering a class whose ctor and dtor are init and deinit, but without subclassing. Or using a specially named pair of functions (e.g. init_extension() and deinit_extension()) that are called after dlopen by minifi, if they are defined.
I like to avoid inheritance unless it's by far the simplest solution, or I'm modeling is-a relationships. The style guide seems to be leaning in this direction as well: https://google.github.io/styleguide/cppguide.html#Inheritance
There's also this funny talk title: [Inheritance Is The Base Class of Evil](https://youtu.be/bIhUE5uUFOA) (Sean Parent)
##########
File path: libminifi/include/core/state/nodes/DeviceInformation.h
##########
@@ -39,6 +37,10 @@
#else
#pragma comment(lib, "iphlpapi.lib")
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN 1
+#endif
Review comment:
Maybe it would make sense to have this defined in cmake for our targets. Does this help with macros btw?
##########
File path: libminifi/include/utils/Export.h
##########
@@ -15,14 +15,21 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include "TemplateLoader.h"
-#include "core/FlowConfiguration.h"
-bool TemplateFactory::added = core::FlowConfiguration::add_static_func("createTemplateFactory");
+#pragma once
-extern "C" {
-
-void *createTemplateFactory(void) {
- return new TemplateFactory();
-}
-}
+#ifdef WIN32
+ #ifdef LIBMINIFI
+ #define MINIFIAPI __declspec(dllexport)
+ #else
+ #define MINIFIAPI __declspec(dllimport)
+ #endif
+ #ifdef MODULE_NAME
+ #define EXTENSIONAPI __declspec(dllexport)
+ #else
+ #define EXTENSIONAPI __declspec(dllimport)
+ #endif
+#else
+ #define MINIFIAPI
+ #define EXTENSIONAPI
+#endif
Review comment:
Instead of this approach, I think we should have an exports header for each of our targets, and use the target-specific macros. Extensions shouldn't share export macros IMO.
See this for a better explanation: https://youtu.be/m0DwB4OvDXk?t=610
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+#ifdef WIN32
+static const std::vector<std::string> path_separators{"/", "\\"};
+#else
+static const std::vector<std::string> path_separators{"/"};
+#endif
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern) {
+ pattern = utils::StringUtils::trim(pattern);
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
+ excluding = true;
+ pattern = utils::StringUtils::trim(pattern.substr(1));
+ }
+ if (pattern.empty()) {
+ logger_->log_error("Empty pattern");
+ return nullopt;
+ }
+ std::string exe_dir = get_executable_dir();
+ if (exe_dir.empty() && !isAbsolutePath(pattern.c_str())) {
+ logger_->log_error("Couldn't determine executable dir, relative pattern '%s' not supported", pattern);
+ return nullopt;
+ }
+ pattern = resolve(exe_dir, pattern);
+ auto segments = split(pattern, path_separators);
+ gsl_Expects(!segments.empty());
+ auto file_pattern = segments.back();
+ if (file_pattern == "**") {
+ file_pattern = "*";
+ } else {
+ segments.pop_back();
+ }
+ if (file_pattern == "." || file_pattern == "..") {
+ logger_->log_error("Invalid file pattern '%s'", file_pattern);
+ return nullopt;
+ }
+ bool after_wildcard = false;
+ for (const auto& segment : segments) {
+ if (after_wildcard && segment == "..") {
+ logger_->log_error("Parent accessor is not supported after wildcards");
+ return nullopt;
+ }
+ if (isGlobPattern(segment)) {
+ after_wildcard = true;
+ }
+ }
+ return FilePattern(segments, file_pattern, excluding);
+}
+
+std::string FileMatcher::FilePattern::getBaseDirectory() const {
+ std::string base_dir;
+ for (const auto& segment : directory_segments_) {
+ // ignore segments at or after wildcards
+ if (isGlobPattern(segment)) {
+ break;
+ }
+ base_dir += segment + get_separator();
+ }
+ return base_dir;
+}
+
+FileMatcher::FileMatcher(const std::string &patterns) {
+ for (auto&& pattern : split(patterns, {","})) {
+ if (auto&& p = FilePattern::fromPattern(pattern)) {
+ patterns_.push_back(std::move(p.value()));
+ }
+ }
+}
+
+template<typename It>
+static bool advance_if_not_equal(It& it, const It& end) {
+ if (it == end) {
+ return false;
+ }
+ ++it;
+ return true;
+}
+
+static bool is_this_dir(const std::string& dir) {
+ return dir.empty() || dir == ".";
+}
+
+template<typename It, typename Fn>
+static void skip_if(It& it, const It& end, const Fn& fn) {
+ while (it != end && fn(*it)) {
+ ++it;
+ }
+}
+
+static bool matchGlob(std::string::const_iterator pattern_it, std::string::const_iterator pattern_end, std::string::const_iterator value_it, std::string::const_iterator value_end) {
Review comment:
You might want to use `std::string_view`
##########
File path: libminifi/test/azure-tests/CMakeLists.txt
##########
@@ -30,9 +30,9 @@ FOREACH(testfile ${AZURE_INTEGRATION_TESTS})
target_compile_features(${testfilename} PRIVATE cxx_std_14)
createTests("${testfilename}")
target_link_libraries(${testfilename} ${CATCH_MAIN_LIB})
- target_wholearchive_library(${testfilename} minifi-azure)
- target_wholearchive_library(${testfilename} minifi-standard-processors)
- target_wholearchive_library(${testfilename} minifi-expression-language-extensions)
+ target_link_libraries(${testfilename} minifi-azure)
Review comment:
Incorrect indentation here. (not new)
##########
File path: libminifi/test/unit/MemoryUsageTest.cpp
##########
@@ -16,6 +16,9 @@
* limitations under the License.
*/
+// including the extensions messes up the memory usage
+#define EXTENSION_LIST ""
Review comment:
What does this do? What was the problem?
##########
File path: libminifi/test/aws-tests/CMakeLists.txt
##########
@@ -30,9 +30,9 @@ FOREACH(testfile ${AWS_INTEGRATION_TESTS})
target_include_directories(${testfilename} PRIVATE BEFORE "${CMAKE_SOURCE_DIR}/extensions/expression-language")
createTests("${testfilename}")
target_link_libraries(${testfilename} ${CATCH_MAIN_LIB})
- target_wholearchive_library(${testfilename} minifi-aws)
- target_wholearchive_library(${testfilename} minifi-standard-processors)
- target_wholearchive_library(${testfilename} minifi-expression-language-extensions)
+ target_link_libraries(${testfilename} minifi-aws)
Review comment:
Incorrect indentation (not new)
##########
File path: main/CMakeLists.txt
##########
@@ -39,16 +39,8 @@ if(WIN32)
endif()
add_executable(minifiexe ${MINIFIEXE_SOURCES})
-if (NOT USE_SHARED_LIBS)
- if (LIBC_STATIC)
- set_target_properties(minifiexe PROPERTIES LINK_SEARCH_START_STATIC 1)
- endif(LIBC_STATIC)
-endif(NOT USE_SHARED_LIBS)
Review comment:
Why are we throwing out static standard library support?
##########
File path: libminifi/test/LogUtils.h
##########
@@ -0,0 +1,51 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <sstream>
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "spdlog/sinks/sink.h"
+#include "spdlog/sinks/ostream_sink.h"
+
+class StringStreamSink : public spdlog::sinks::sink {
+ public:
+ explicit StringStreamSink(std::shared_ptr<std::ostringstream> stream, bool force_flush = false)
+ : stream_(std::move(stream)), sink_(*stream_, force_flush) {}
+
+ ~StringStreamSink() override = default;
+
+ void log(const spdlog::details::log_msg &msg) override {
+ sink_.log(msg);
+ }
+ void flush() override {
+ sink_.flush();
+ }
+ void set_pattern(const std::string &pattern) override {
+ sink_.set_pattern(pattern);
+ }
+ void set_formatter(std::unique_ptr<spdlog::formatter> sink_formatter) override {
+ sink_.set_formatter(std::move(sink_formatter));
+ }
+
+ private:
+ std::shared_ptr<std::ostringstream> stream_;
Review comment:
I'd prefer this to be an observer pointer, i.e. without ownership.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680775139
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
gcc const/destr (or any load-time running code DllMain) is not viable here as some extensions (e.g. python extension, jni extension) use the configuration object passed onto it to initialize themselves
also there could be interdependencies between extensions, for example loading `http-curl-extension` loads `civet-extension`, so if we only rely on what we load with `dlopen` we miss out on `civet-extension`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680778576
##########
File path: extensions/script/CMakeLists.txt
##########
@@ -68,13 +58,10 @@ if (ENABLE_LUA_SCRIPTING)
add_definitions(-DLUA_SUPPORT)
file(GLOB LUA_SOURCES "lua/*.cpp")
- add_library(minifi-lua-extensions STATIC ${LUA_SOURCES})
-
- target_link_libraries(minifi-lua-extensions ${LIBMINIFI})
- target_link_libraries(minifi-lua-extensions ${LUA_LIBRARIES} sol)
- target_link_libraries(minifi-script-extensions minifi-lua-extensions)
+ target_sources(minifi-script-extensions PRIVATE ${LUA_SOURCES})
- target_compile_features(minifi-lua-extensions PUBLIC cxx_std_14)
+ target_link_libraries(minifi-script-extensions ${LUA_LIBRARIES} sol)
+ target_compile_features(minifi-script-extensions PUBLIC cxx_std_14)
Review comment:
we don't have to have a python interpreter if we enable only the lua extension, those sources are not compiled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697374948
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::filesystem::path& path) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
Review comment:
reverted it back to `std::string` (the failing CI reminded my why it was `std::string`) `std::string::starts_with` is c++20 feature, and I would rather not start rewriting `StringUtils` to accept `std::string_view` wherever it can in this PR, so there is no way around at least one `std::string` conversion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r677585723
##########
File path: cmake/Extensions.cmake
##########
@@ -26,7 +26,27 @@ macro(register_extension extension-name)
get_property(extensions GLOBAL PROPERTY EXTENSION-OPTIONS)
set_property(GLOBAL APPEND PROPERTY EXTENSION-OPTIONS ${extension-name})
target_compile_definitions(${extension-name} PRIVATE "MODULE_NAME=${extension-name}")
- set_target_properties(${extension-name} PROPERTIES ENABLE_EXPORTS True)
+ set_target_properties(${extension-name} PROPERTIES
+ ENABLE_EXPORTS True
+ POSITION_INDEPENDENT_CODE ON)
+ if (WIN32)
+ set_target_properties(${extension-name} PROPERTIES
+ RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ WINDOWS_EXPORT_ALL_SYMBOLS TRUE)
+ else()
+ set_target_properties(${extension-name} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin")
+ endif()
+
+ if (WIN32)
+ install(TARGETS ${extension-name} RUNTIME DESTINATION extensions COMPONENT bin)
+ else()
+ if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
+ target_link_options(${extension-name} PRIVATE "-Wl,--disable-new-dtags")
+ endif()
+ set_target_properties(${extension-name} PROPERTIES INSTALL_RPATH "$ORIGIN")
Review comment:
does this work? no escaping of the `$` is needed?
there is also a BUILD_RPATH_USE_ORIGIN CMake option that may be useful here
EDIT: I tested with `make u20` and it does work, so never mind
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r683307707
##########
File path: libminifi/include/core/state/nodes/DeviceInformation.h
##########
@@ -39,6 +37,10 @@
#else
#pragma comment(lib, "iphlpapi.lib")
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN 1
+#endif
Review comment:
removed this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r708983346
##########
File path: extensions/jni/JNILoader.cpp
##########
@@ -15,16 +15,24 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include "JNILoader.h"
-#include "core/FlowConfiguration.h"
+#include "core/extension/Extension.h"
+#include "JVMCreator.h"
-bool JNIFactory::added = core::FlowConfiguration::add_static_func("createJNIFactory");
+namespace minifi = org::apache::nifi::minifi;
-extern "C" {
+static minifi::jni::JVMCreator& getJVMCreator() {
+ static minifi::jni::JVMCreator instance("JVMCreator");
+ return instance;
+}
-void *createJNIFactory(void) {
- return new JNIFactory();
+static bool init(const std::shared_ptr<org::apache::nifi::minifi::Configure>& config) {
+ getJVMCreator().configure(config);
+ return true;
}
+static void deinit() {
+ // TODO(adebreceni)
Review comment:
added some comment, it was not implemented before, and is outside the scope of this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697339788
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::filesystem::path& path) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ std::string filename = path.filename().string();
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ path.parent_path(),
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ auto candidates = utils::file::match(utils::file::FilePattern(pattern.value(), [&] (std::string_view subpattern, std::string_view error_msg) {
+ logger_->log_error("Error in subpattern '%s': %s", std::string{subpattern}, std::string{error_msg});
Review comment:
as we can't get a null terminated `const char*` from `std::string_view` I don't see how that would work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r699279695
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+ public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
+ }
+
+ public:
+ explicit FilePatternSegment(std::string pattern);
+
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ NOT_MATCHING // dir/file does not match pattern, do what you may
+ };
+
+ bool isExcluding() const {
+ return excluding_;
+ }
+
+ MatchResult match(const std::string& directory) const;
+
+ MatchResult match(const std::string& directory, const std::string& filename) const;
+
+ MatchResult match(const std::filesystem::path& path) const;
+ /**
+ * @return The lowermost parent directory without wildcards.
+ */
+ std::filesystem::path getBaseDirectory() const;
+
+ private:
+ enum class DirMatchResult {
+ NONE, // pattern does not match the directory (e.g. p = "/home/inner/*test", v = "/home/banana")
+ PARENT, // directory is a parent of the pattern (e.g. p = "/home/inner/*test", v = "/home/inner")
+ EXACT, // pattern exactly matches the directory (e.g. p = "/home/inner/*test", v = "/home/inner/cool_test")
+ TREE // pattern matches the whole subtree of the directory (e.g. p = "/home/**", v = "/home/banana")
+ };
+
+ using DirIt = std::filesystem::path::const_iterator;
+ static DirMatchResult matchDirectory(DirIt pattern_begin, DirIt pattern_end, DirIt value_begin, DirIt value_end);
+
+ std::filesystem::path directory_pattern_;
+ std::string file_pattern_;
+ bool excluding_;
+ };
+
+ using ErrorHandler = std::function<void(std::string_view /*subpattern*/, std::string_view /*error_message*/)>;
+
+ static void defaultErrorHandler(std::string_view subpattern, std::string_view error_message) {
+ std::string message = "Error in subpattern '";
+ message += subpattern;
+ message += "': ";
+ message += error_message;
+ throw FilePatternError(message);
+ }
+
+ public:
+ explicit FilePattern(const std::string& pattern, ErrorHandler error_handler = defaultErrorHandler);
+
+ private:
+ std::vector<FilePatternSegment> segments_;
+};
+
+std::set<std::filesystem::path> match(const FilePattern& pattern);
Review comment:
in light of the numbers I reverted back to `std::set` (in my own measurements I found that the runtime is dominated by the filesystem operations)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r708983717
##########
File path: thirdparty/aws-sdk-cpp/dll-export-injection.patch
##########
@@ -0,0 +1,52 @@
+diff -rupN aws-sdk-cpp-src/aws-cpp-sdk-core/CMakeLists.txt aws-sdk-cpp-src-patched/aws-cpp-sdk-core/CMakeLists.txt
Review comment:
I feel like the name of the file is pretty talkative :)
##########
File path: libminifi/test/unit/FilePatternTests.cpp
##########
@@ -0,0 +1,253 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#define CUSTOM_EXTENSION_INIT
+
+#include <filesystem>
+
+#include "../TestBase.h"
+#include "utils/file/FilePattern.h"
+#include "range/v3/view/transform.hpp"
+#include "range/v3/view/map.hpp"
+#include "range/v3/view/join.hpp"
+#include "range/v3/view/cache1.hpp"
+#include "range/v3/view/c_str.hpp"
+#include "range/v3/range/conversion.hpp"
+#include "range/v3/range.hpp"
Review comment:
removed the unused ones
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680915970
##########
File path: libminifi/test/unit/MemoryUsageTest.cpp
##########
@@ -16,6 +16,9 @@
* limitations under the License.
*/
+// including the extensions messes up the memory usage
+#define EXTENSION_LIST ""
Review comment:
It seems to be 30M, but I guess we just go over that memory usage with everything loaded.
Could you maybe update the comment to something along the lines of "Loading the extensions increases the memory usage, potentially going over the threshold that this test expects"? Or double the size to increase this threshold.
Maybe @martinzink can also comment on this issue, but I suspect there's not much more to add.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680768847
##########
File path: libminifi/test/LogUtils.h
##########
@@ -0,0 +1,51 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <sstream>
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "spdlog/sinks/sink.h"
+#include "spdlog/sinks/ostream_sink.h"
+
+class StringStreamSink : public spdlog::sinks::sink {
+ public:
+ explicit StringStreamSink(std::shared_ptr<std::ostringstream> stream, bool force_flush = false)
+ : stream_(std::move(stream)), sink_(*stream_, force_flush) {}
+
+ ~StringStreamSink() override = default;
+
+ void log(const spdlog::details::log_msg &msg) override {
+ sink_.log(msg);
+ }
+ void flush() override {
+ sink_.flush();
+ }
+ void set_pattern(const std::string &pattern) override {
+ sink_.set_pattern(pattern);
+ }
+ void set_formatter(std::unique_ptr<spdlog::formatter> sink_formatter) override {
+ sink_.set_formatter(std::move(sink_formatter));
+ }
+
+ private:
+ std::shared_ptr<std::ostringstream> stream_;
Review comment:
Ok, that's fair. Could you add a comment above the member with this info? I want to avoid having rare problems introduced by a possible future refactoring where we rethink lifetimes and reduce shared_ptr usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r679838081
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
Review comment:
I would simplify this by only saying that they must go in source files. You could add it to the first paragraph as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r691838093
##########
File path: extensions/standard-processors/tests/unit/ClassLoaderTests.cpp
##########
@@ -20,6 +20,8 @@
#include "core/Processor.h"
#include "core/ClassLoader.h"
#include "core/yaml/YamlConfiguration.h"
+#include "core/extension/ExtensionManager.h"
+#include "utils/file/FileUtils.h"
Review comment:
almost none of the included headers are used in this test, removed them
##########
File path: libminifi/include/core/Resource.h
##########
@@ -37,29 +41,64 @@ namespace core {
#define MKSOC(x) #x
#define MAKESTRING(x) MKSOC(x)
+static inline ClassLoader& getClassLoader() {
+#ifdef MODULE_NAME
+ return ClassLoader::getDefaultClassLoader().getClassLoader(MAKESTRING(MODULE_NAME));
+#else
+ return ClassLoader::getDefaultClassLoader();
+#endif
+}
+
template<class T>
class StaticClassType {
public:
- StaticClassType(const std::string &name, const std::string &description = "") { // NOLINT
+ StaticClassType(const std::string& name, const std::optional<std::string>& description, const std::vector<std::string>& construction_names)
+ : name_(name), construction_names_(construction_names) { // NOLINT
Review comment:
unclear, removed it
##########
File path: libminifi/include/utils/file/FileMatcher.h
##########
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FileMatcherTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FileMatcher {
+ friend struct ::FileMatcherTestAccessor;
+ class FilePattern {
+ FilePattern(std::vector<std::string> directory_segments, std::string file_pattern, bool excluding)
+ : directory_segments_(std::move(directory_segments)),
+ file_pattern_(std::move(file_pattern)),
+ excluding_(excluding) {}
+
+ public:
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ DONT_CARE // dir/file does not match pattern, do what you may
Review comment:
renamed it to `NOT_MATCHING`
##########
File path: libminifi/include/core/extension/ExtensionManager.h
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "core/logging/Logger.h"
+#include "DynamicLibrary.h"
Review comment:
replaced
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r683288063
##########
File path: libminifi/include/core/state/nodes/DeviceInformation.h
##########
@@ -39,6 +37,10 @@
#else
#pragma comment(lib, "iphlpapi.lib")
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN 1
+#endif
Review comment:
it seems you yourself have already added it in #967, but did not remove them from source/header files, we should remove all of them in a separate PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680708207
##########
File path: main/CMakeLists.txt
##########
@@ -39,16 +39,8 @@ if(WIN32)
endif()
add_executable(minifiexe ${MINIFIEXE_SOURCES})
-if (NOT USE_SHARED_LIBS)
- if (LIBC_STATIC)
- set_target_properties(minifiexe PROPERTIES LINK_SEARCH_START_STATIC 1)
- endif(LIBC_STATIC)
-endif(NOT USE_SHARED_LIBS)
Review comment:
LIBC_STATIC is marked as experimental, not tested in the ci, and we no longer build a single static minifiexe, readded this as in the future we might want to support both static and shared builds
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680770030
##########
File path: libminifi/test/unit/MemoryUsageTest.cpp
##########
@@ -16,6 +16,9 @@
* limitations under the License.
*/
+// including the extensions messes up the memory usage
+#define EXTENSION_LIST ""
Review comment:
You mean using more memory? Or what does it mean to "mess up" memory usage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r704552398
##########
File path: libminifi/test/unit/FilePatternTests.cpp
##########
@@ -0,0 +1,253 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#define CUSTOM_EXTENSION_INIT
+
+#include <filesystem>
+
+#include "../TestBase.h"
+#include "utils/file/FilePattern.h"
+#include "range/v3/view/transform.hpp"
+#include "range/v3/view/map.hpp"
+#include "range/v3/view/join.hpp"
+#include "range/v3/view/cache1.hpp"
+#include "range/v3/view/c_str.hpp"
+#include "range/v3/range/conversion.hpp"
+#include "range/v3/range.hpp"
+
+struct FilePatternTestAccessor {
+ using FilePatternSegment = fileutils::FilePattern::FilePatternSegment;
+};
+
+using FilePatternSegment = FilePatternTestAccessor::FilePatternSegment;
+using FilePattern = fileutils::FilePattern;
+
+#define REQUIRE_INCLUDE(val) REQUIRE((val == FilePatternSegment::MatchResult::INCLUDE))
+#define REQUIRE_EXCLUDE(val) REQUIRE((val == FilePatternSegment::MatchResult::EXCLUDE))
+#define REQUIRE_NOT_MATCHING(val) REQUIRE((val == FilePatternSegment::MatchResult::NOT_MATCHING))
+
+#ifdef WIN32
+static std::filesystem::path root{"C:\\"};
+#else
+static std::filesystem::path root{"/"};
+#endif
+
+TEST_CASE("Invalid paths") {
+ REQUIRE_THROWS(FilePatternSegment(""));
+ REQUIRE_THROWS(FilePatternSegment("."));
+ REQUIRE_THROWS(FilePatternSegment(".."));
+ REQUIRE_THROWS(FilePatternSegment("!"));
+ REQUIRE_THROWS(FilePatternSegment("!."));
+ REQUIRE_THROWS(FilePatternSegment("!.."));
+ // parent accessor after wildcard
+ FilePatternSegment("./../file.txt"); // sanity check
+ REQUIRE_THROWS(FilePatternSegment("./*/../file.txt"));
+}
+
+TEST_CASE("FilePattern reports error in correct subpattern") {
+ std::vector<std::pair<std::string, std::string>> invalid_subpattern{
Review comment:
This could use string_view instead of string for simpler generated code. In general, static strings don't need to be copied to the heap with std::string, because they will stay in static memory.
https://godbolt.org/z/8GGe3qfao (600 something asm lines with string_view)
vs
https://godbolt.org/z/zeW38svPa (1100 lines with string)
##########
File path: libminifi/src/core/extension/DynamicLibrary.cpp
##########
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <memory>
+#ifndef WIN32
+#include <dlfcn.h>
+#define DLL_EXPORT
+#else
+#include <system_error>
+#define WIN32_LEAN_AND_MEAN 1
+#include <Windows.h> // Windows specific libraries for collecting software metrics.
+#include <Psapi.h>
+#pragma comment(lib, "psapi.lib" )
+#define DLL_EXPORT __declspec(dllexport)
+#define RTLD_LAZY 0
+#define RTLD_NOW 0
+
+#define RTLD_GLOBAL (1 << 1)
+#define RTLD_LOCAL (1 << 2)
+#endif
+
+#include "core/extension/DynamicLibrary.h"
+#include "core/extension/Extension.h"
+#include "utils/GeneralUtils.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+std::shared_ptr<logging::Logger> DynamicLibrary::logger_ = logging::LoggerFactory<DynamicLibrary>::getLogger();
+
+DynamicLibrary::DynamicLibrary(std::string name, std::filesystem::path library_path)
+ : Module(std::move(name)),
+ library_path_(std::move(library_path)) {
+}
+
+bool DynamicLibrary::load() {
+ dlerror();
+ handle_ = dlopen(library_path_.string().c_str(), RTLD_NOW | RTLD_LOCAL);
+ if (!handle_) {
+ logger_->log_error("Failed to load extension '%s' at '%s': %s", name_, library_path_.string(), dlerror());
+ return false;
+ } else {
+ logger_->log_trace("Loaded extension '%s' at '%s'", name_, library_path_.string());
+ return true;
+ }
+}
+
+bool DynamicLibrary::unload() {
+ logger_->log_trace("Unloading library '%s' at '%s'", name_, library_path_.string());
+ if (!handle_) {
+ logger_->log_error("Extension does not have a handle_ '%s' at '%s'", name_, library_path_.string());
+ return true;
+ }
+ dlerror();
+ if (dlclose(handle_)) {
+ logger_->log_error("Failed to unload extension '%s' at '%': %s", name_, library_path_.string(), dlerror());
+ return false;
+ }
+ logger_->log_trace("Unloaded extension '%s' at '%s'", name_, library_path_.string());
+ handle_ = nullptr;
+ return true;
+}
+
+DynamicLibrary::~DynamicLibrary() = default;
+
+#ifdef WIN32
+
+void DynamicLibrary::store_error() {
+ auto error = GetLastError();
+
+ if (error == 0) {
+ error_str_ = "";
+ return;
+ }
+
+ current_error_ = std::system_category().message(error);
+}
+
+void* DynamicLibrary::dlsym(void* handle, const char* name) {
+ FARPROC symbol;
+
+ symbol = GetProcAddress((HMODULE)handle, name);
+
+ if (symbol == nullptr) {
+ store_error();
+
+ for (auto hndl : resource_mapping_) {
+ symbol = GetProcAddress((HMODULE)hndl.first, name);
+ if (symbol != nullptr) {
+ break;
+ }
+ }
+ }
+
+#ifdef _MSC_VER
+#pragma warning(suppress: 4054 )
+#endif
+ return reinterpret_cast<void*>(symbol);
+}
+
+const char* DynamicLibrary::dlerror() {
+ error_str_ = current_error_;
+
+ current_error_ = "";
+
+ return error_str_.c_str();
+}
+
+void* DynamicLibrary::dlopen(const char* file, int mode) {
+ HMODULE object;
+ uint32_t uMode = SetErrorMode(SEM_FAILCRITICALERRORS);
+ if (nullptr == file) {
+ HMODULE allModules[1024];
+ HANDLE current_process_handle = GetCurrentProcess();
+ DWORD cbNeeded;
+ object = GetModuleHandle(NULL);
+
+ if (!object) {
+ store_error();
+ }
+
+ if (EnumProcessModules(current_process_handle, allModules, sizeof(allModules), &cbNeeded) != 0) {
+ for (uint32_t i = 0; i < cbNeeded / sizeof(HMODULE); i++) {
+ // Get the full path to the module's file.
+ resource_mapping_.insert(std::make_pair(reinterpret_cast<void*>(allModules[i]), "minifi-system"));
Review comment:
nitpicking: static_cast is enough here (and a few other places), as it's the same as reinterpret_cast for void*.
> Any object pointer type T1* can be converted to another object pointer type cv T2*. This is exactly equivalent to static_cast<cv T2*>(static_cast<cv void*>(expression))
(from the [cppreference reinterpret_cast page](https://en.cppreference.com/w/cpp/language/reinterpret_cast))
##########
File path: libminifi/include/core/logging/Logger.h
##########
@@ -26,6 +26,7 @@
#include <iostream>
#include <vector>
#include <algorithm>
+#include <string_view>
Review comment:
I'm assuming this is a leftover from a suggestion to implement conditional_conversion for string_view that couldn't be done.
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,136 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
Review comment:
Consider marking getters `[[nodiscard]]`. Clang-tidy warns me for all of these with modernize-use-nodiscard through CLion.
##########
File path: libminifi/include/core/extension/DynamicLibrary.h
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <map>
+#include <string>
+#include <filesystem>
+
+#include "Module.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class DynamicLibrary : public Module {
+ friend class ExtensionManager;
+
+ public:
+ DynamicLibrary(std::string name, std::filesystem::path library_path);
+ ~DynamicLibrary() override;
+
+ private:
+#ifdef WIN32
+ std::map<void*, std::string> resource_mapping_;
+
+ std::string error_str_;
+ std::string current_error_;
+
+ void store_error();
+ void* dlsym(void* handle, const char* name);
+ const char* dlerror();
+ void* dlopen(const char* file, int mode);
+ int dlclose(void* handle);
+#endif
+
+ bool load();
+ bool unload();
+
+ std::filesystem::path library_path_;
+ gsl::owner<void*> handle_ = nullptr;
+
+ static std::shared_ptr<logging::Logger> logger_;
Review comment:
This issue seems to be back
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <utility>
+#include <memory>
+#include <filesystem>
+#include <set>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
Review comment:
Logger and OptionalUtils are not used in this file, so I think it's best to remove them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678175372
##########
File path: extensions/opencv/MotionDetector.cpp
##########
@@ -201,6 +201,8 @@ void MotionDetector::onTrigger(const std::shared_ptr<core::ProcessContext> &cont
void MotionDetector::notifyStop() {
}
+REGISTER_RESOURCE(MotionDetector, "Detect motion from captured images."); // NOLINT
Review comment:
as it was recently removed on main, it must be a rebase-artefact, deleted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r683288063
##########
File path: libminifi/include/core/state/nodes/DeviceInformation.h
##########
@@ -39,6 +37,10 @@
#else
#pragma comment(lib, "iphlpapi.lib")
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN 1
+#endif
Review comment:
~it seems you yourself have already added it in #967~ somebody have already added it, but did not remove them from source/header files, we should remove all of them in a separate PR
edit: you just modified the line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r683288063
##########
File path: libminifi/include/core/state/nodes/DeviceInformation.h
##########
@@ -39,6 +37,10 @@
#else
#pragma comment(lib, "iphlpapi.lib")
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN 1
+#endif
Review comment:
it seems you yourself have already added it in #967, but did not remove them from source/header files, we should remove all of them in a separate PR
##########
File path: libminifi/include/core/state/nodes/DeviceInformation.h
##########
@@ -39,6 +37,10 @@
#else
#pragma comment(lib, "iphlpapi.lib")
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN 1
+#endif
Review comment:
~it seems you yourself have already added it in #967~ somebody have already added it, but did not remove them from source/header files, we should remove all of them in a separate PR
edit: you just modified the line
##########
File path: libminifi/include/core/state/nodes/DeviceInformation.h
##########
@@ -39,6 +37,10 @@
#else
#pragma comment(lib, "iphlpapi.lib")
+#ifndef WIN32_LEAN_AND_MEAN
+#define WIN32_LEAN_AND_MEAN 1
+#endif
Review comment:
removed this one
##########
File path: libminifi/test/unit/MemoryUsageTest.cpp
##########
@@ -16,6 +16,9 @@
* limitations under the License.
*/
+// including the extensions messes up the memory usage
+#define EXTENSION_LIST ""
Review comment:
updated comment
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+#ifdef WIN32
+static const std::vector<std::string> path_separators{"/", "\\"};
+#else
+static const std::vector<std::string> path_separators{"/"};
+#endif
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern) {
+ pattern = utils::StringUtils::trim(pattern);
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
+ excluding = true;
+ pattern = utils::StringUtils::trim(pattern.substr(1));
+ }
+ if (pattern.empty()) {
+ logger_->log_error("Empty pattern");
+ return nullopt;
+ }
+ std::string exe_dir = get_executable_dir();
+ if (exe_dir.empty() && !isAbsolutePath(pattern.c_str())) {
+ logger_->log_error("Couldn't determine executable dir, relative pattern '%s' not supported", pattern);
+ return nullopt;
+ }
+ pattern = resolve(exe_dir, pattern);
+ auto segments = split(pattern, path_separators);
+ gsl_Expects(!segments.empty());
+ auto file_pattern = segments.back();
+ if (file_pattern == "**") {
+ file_pattern = "*";
+ } else {
+ segments.pop_back();
+ }
+ if (file_pattern == "." || file_pattern == "..") {
+ logger_->log_error("Invalid file pattern '%s'", file_pattern);
+ return nullopt;
+ }
+ bool after_wildcard = false;
+ for (const auto& segment : segments) {
+ if (after_wildcard && segment == "..") {
+ logger_->log_error("Parent accessor is not supported after wildcards");
+ return nullopt;
+ }
+ if (isGlobPattern(segment)) {
+ after_wildcard = true;
+ }
+ }
+ return FilePattern(segments, file_pattern, excluding);
+}
+
+std::string FileMatcher::FilePattern::getBaseDirectory() const {
+ std::string base_dir;
+ for (const auto& segment : directory_segments_) {
+ // ignore segments at or after wildcards
+ if (isGlobPattern(segment)) {
+ break;
+ }
+ base_dir += segment + get_separator();
+ }
+ return base_dir;
+}
+
+FileMatcher::FileMatcher(const std::string &patterns) {
+ for (auto&& pattern : split(patterns, {","})) {
+ if (auto&& p = FilePattern::fromPattern(pattern)) {
+ patterns_.push_back(std::move(p.value()));
+ }
+ }
+}
+
+template<typename It>
+static bool advance_if_not_equal(It& it, const It& end) {
+ if (it == end) {
+ return false;
+ }
+ ++it;
+ return true;
+}
+
+static bool is_this_dir(const std::string& dir) {
+ return dir.empty() || dir == ".";
+}
+
+template<typename It, typename Fn>
+static void skip_if(It& it, const It& end, const Fn& fn) {
+ while (it != end && fn(*it)) {
+ ++it;
+ }
+}
+
+static bool matchGlob(std::string::const_iterator pattern_it, std::string::const_iterator pattern_end, std::string::const_iterator value_it, std::string::const_iterator value_end) {
Review comment:
it seems that on my machine `Apple clang version 11.0.3` does not support that constructor even with the `-std=c++2a` flag, `substr` it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r691315887
##########
File path: conf/minifi.properties
##########
@@ -20,6 +20,9 @@ nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=100 millis
+# Comma separated path for the extension libraries. Relative path is relative to the minifi executable.
+nifi.extension.path=../extensions/*
Review comment:
I currently see two usual ways of launching minifi:
- The normal installation, where the MINIFI_HOME is usually /opt/minifi, and the binary is under MINIFI_HOME/bin
- After compilation, launching the compiled binary in a preexisting MINIFI_HOME
In the first case, either version is fine. In the second case, we would need extensions to be installed to build/extensions/ for them to work without extra effort, or we could load them from MINIFI_HOME/extensions/ in the proposed case.
Another use case that I could see happening in the future is standard linux packages:
- minifi binary under /usr/bin/minifi
- config files under /etc/minifi
- extensions under /usr/share/minifi/extensions, and I imagine MINIFI_HOME would be /usr/share/minifi.
I think the default would cover more use cases with a path relative to MINIFI_HOME, and we could avoid the "../" at the beginning of practically every path pattern. Given the above points, do you still prefer paths relative to the binary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r685136268
##########
File path: extensions/aws/controllerservices/AWSCredentialsService.h
##########
@@ -26,6 +26,7 @@
#include "utils/AWSInitializer.h"
#include "core/Resource.h"
+#include "utils/OptionalUtils.h"
Review comment:
This header is not used in the file.
##########
File path: Extensions.md
##########
@@ -16,108 +16,62 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars. Registration should happen in source files,
+as otherwise, including another extension's headers would introduce the same resources in the including extension
+as well, possibly shadowing its own resources.
Review comment:
```suggestion
capabilities (classes) available to the system through registrars. Registration must happen in source files, not headers.
```
If someone questions it or wants to understand the reason, they can look at the macro implementations and figure out. Or we can keep the details but move them to code comments. The improvement would be shorter, easier to follow documentation.
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
I didn't mean exactly gcc constructor/destructor, just similar with our code. But even that could work by adding a config getter in libminifi, although I think that would be unnecessary added complexity.
I also think that inheritance is unnecessary added complexity. We don't need to inherit data or implementation.
https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#c120-use-class-hierarchies-to-represent-concepts-with-inherent-hierarchical-structure-only
##########
File path: extensions/standard-processors/tests/unit/ClassLoaderTests.cpp
##########
@@ -20,6 +20,8 @@
#include "core/Processor.h"
#include "core/ClassLoader.h"
#include "core/yaml/YamlConfiguration.h"
+#include "core/extension/ExtensionManager.h"
+#include "utils/file/FileUtils.h"
Review comment:
Are these necessary?
##########
File path: libminifi/include/utils/file/FileMatcher.h
##########
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FileMatcherTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FileMatcher {
+ friend struct ::FileMatcherTestAccessor;
+ class FilePattern {
+ FilePattern(std::vector<std::string> directory_segments, std::string file_pattern, bool excluding)
+ : directory_segments_(std::move(directory_segments)),
+ file_pattern_(std::move(file_pattern)),
+ excluding_(excluding) {}
+
+ public:
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ DONT_CARE // dir/file does not match pattern, do what you may
Review comment:
I would rename DONT_CARE to NOT_MATCHING or similar to convey the meaning. It was strange reading DONT_CARE in the client code and wondering why would you not care.
##########
File path: libminifi/include/core/extension/ExtensionManager.h
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "core/logging/Logger.h"
+#include "DynamicLibrary.h"
Review comment:
This could be replaced with Module.h
##########
File path: libminifi/include/core/Resource.h
##########
@@ -37,29 +41,64 @@ namespace core {
#define MKSOC(x) #x
#define MAKESTRING(x) MKSOC(x)
+static inline ClassLoader& getClassLoader() {
+#ifdef MODULE_NAME
+ return ClassLoader::getDefaultClassLoader().getClassLoader(MAKESTRING(MODULE_NAME));
+#else
+ return ClassLoader::getDefaultClassLoader();
+#endif
+}
+
template<class T>
class StaticClassType {
public:
- StaticClassType(const std::string &name, const std::string &description = "") { // NOLINT
+ StaticClassType(const std::string& name, const std::optional<std::string>& description, const std::vector<std::string>& construction_names)
+ : name_(name), construction_names_(construction_names) { // NOLINT
Review comment:
Do we still need this NOLINT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697336147
##########
File path: libminifi/include/core/extension/DynamicLibrary.h
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <map>
+#include <string>
+#include <filesystem>
+
+#include "Module.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class DynamicLibrary : public Module {
+ friend class ExtensionManager;
+
+ public:
+ DynamicLibrary(std::string name, std::filesystem::path library_path);
+ ~DynamicLibrary() override;
+
+ private:
+#ifdef WIN32
+ std::map<void*, std::string> resource_mapping_;
+
+ std::string error_str_;
+ std::string current_error_;
+
+ void store_error();
+ void* dlsym(void* handle, const char* name);
+ const char* dlerror();
+ void* dlopen(const char* file, int mode);
+ int dlclose(void* handle);
Review comment:
both the interface and implementation have been moved verbatim from `ClassLoader`, we could refine it later, but as it is hidden in the `DynamicLibrary` I would leave it as is for now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r679835774
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern, bool log_errors) {
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
+ excluding = true;
+ pattern = pattern.substr(1);
+ }
+ if (pattern.empty()) {
+ if (log_errors) logger_->log_error("Empty pattern");
+ return nullopt;
+ }
+ std::string exe_dir = get_executable_dir();
+ if (exe_dir.empty() && !isAbsolutePath(pattern.c_str())) {
+ if (log_errors) logger_->log_error("Couldn't determine executable dir, relative pattern '%s' not supported", pattern);
+ return nullopt;
+ }
+ pattern = resolve(exe_dir, pattern);
+ auto segments = split(pattern, {"/", "\\"});
+ gsl_Expects(!segments.empty());
+ auto file_pattern = segments.back();
+ if (file_pattern == "**") {
+ file_pattern = "*";
+ } else {
+ segments.pop_back();
+ }
+ if (file_pattern == "." || file_pattern == "..") {
+ if (log_errors) logger_->log_error("Invalid file pattern '%s'", file_pattern);
+ return nullopt;
+ }
+ return FilePattern(segments, file_pattern, excluding);
+}
+
+std::string FileMatcher::FilePattern::getBaseDirectory() const {
+ std::string base_dir;
+ for (const auto& segment : directory_segments_) {
+ // ignore segments at or after wildcards
+ if (isGlobPattern(segment)) {
+ break;
+ }
+ base_dir += segment + get_separator();
+ }
+ return base_dir;
+}
+
+FileMatcher::FileMatcher(const std::string &patterns) {
+ for (auto&& pattern : split(patterns, {","})) {
+ if (auto&& p = FilePattern::fromPattern(pattern)) {
+ patterns_.push_back(std::move(p.value()));
+ }
+ }
+}
+
+template<typename It>
+static bool advance_if_not_equal(It& it, const It& end) {
+ if (it == end) {
+ return false;
+ }
+ ++it;
+ return true;
+}
+
+static bool is_this_dir(const std::string& dir) {
+ return dir.empty() || dir == ".";
+}
+
+template<typename It, typename Fn>
+static void skip_if(It& it, const It& end, const Fn& fn) {
+ while (it != end && fn(*it)) {
+ ++it;
+ }
+}
+
+static bool matchGlob(std::string::const_iterator pattern_it, std::string::const_iterator pattern_end, std::string::const_iterator value_it, std::string::const_iterator value_end) {
+ // match * and ?
+ for (; pattern_it != pattern_end; ++pattern_it) {
+ if (*pattern_it == '*') {
+ do {
+ if (matchGlob(std::next(pattern_it), pattern_end, value_it, value_end)) {
+ return true;
+ }
+ } while (advance_if_not_equal(value_it, value_end));
+ return false;
+ }
+ if (value_it == value_end) {
+ return false;
+ }
+ if (*pattern_it != '?' && *pattern_it != *value_it) {
+ return false;
+ }
+ ++value_it;
+ }
+ return value_it == value_end;
+}
+
+FileMatcher::FilePattern::DirMatchResult FileMatcher::FilePattern::matchDirectory(DirIt pattern_it, DirIt pattern_end, DirIt value_it, DirIt value_end) {
+ for (; pattern_it != pattern_end; ++pattern_it) {
+ if (is_this_dir(*pattern_it)) {
+ continue;
+ }
+ if (*pattern_it == "**") {
+ if (std::next(pattern_it) == pattern_end) {
+ return DirMatchResult::TREE;
+ }
+ bool matched_parent = false;
+ // any number of nested directories
+ do {
+ skip_if(value_it, value_end, is_this_dir);
+ auto result = matchDirectory(std::next(pattern_it), pattern_end, value_it, value_end);
+ if (result == DirMatchResult::TREE || result == DirMatchResult::EXACT) {
+ return result;
+ }
+ if (result == DirMatchResult::PARENT) {
+ // even though we have a parent match, there may be a "better" (exact, tree) match
+ matched_parent = true;
+ }
+ } while (advance_if_not_equal(value_it, value_end));
+ if (matched_parent) {
+ return DirMatchResult::PARENT;
+ }
+ return DirMatchResult::NONE;
+ }
+ skip_if(value_it, value_end, is_this_dir);
+ if (value_it == value_end) {
+ // we used up all the value segments but there are still pattern segments
+ return DirMatchResult::PARENT;
+ }
+ if (!matchGlob(pattern_it->begin(), pattern_it->end(), value_it->begin(), value_it->end())) {
+ return DirMatchResult::NONE;
+ }
+ ++value_it;
+ }
+ skip_if(value_it, value_end, is_this_dir);
+ if (value_it == value_end) {
+ // used up all pattern and value segments
+ return DirMatchResult::EXACT;
+ } else {
+ // used up all pattern segments but we still have value segments
+ return DirMatchResult::NONE;
+ }
+}
+
+bool FileMatcher::FilePattern::match(const std::string& directory, const optional<std::string>& filename) const {
+ auto value = split(directory, {"/", "\\"});
+ auto result = matchDirectory(directory_segments_.begin(), directory_segments_.end(), value.begin(), value.end());
+ if (!filename) {
+ if (excluding_) {
+ if (result == DirMatchResult::TREE && file_pattern_ == "*") {
+ // all files are excluded in this directory
+ return true;
Review comment:
I did some refactoring, not sure if it made things less complicated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r696678247
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::string getFullPath() const {
+ return utils::file::PathUtils::concat_path(dir, filename);
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::string& dir, const std::string& filename) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ dir,
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ std::vector<LibraryDescriptor> libraries;
+ utils::file::FileMatcher(pattern.value()).forEachFile([&] (const std::string& dir, const std::string& filename) {
+ std::optional<LibraryDescriptor> library = asDynamicLibrary(dir, filename);
+ if (library && library->verify(logger_)) {
+ libraries.push_back(std::move(library.value()));
+ }
+ return true;
+ });
+ for (const auto& library : libraries) {
+ auto module = std::make_unique<DynamicLibrary>(library.name, library.getFullPath());
+ active_module_ = module.get();
Review comment:
the loading extension needs to access the loading dll to register itself for subsequent initialization (or at least register itself somewhere it can be accessed by the ExtensionManager later, but it is better to "scope" the extension into the dlls they came from IMO)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680755093
##########
File path: libminifi/include/utils/Export.h
##########
@@ -15,14 +15,21 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include "TemplateLoader.h"
-#include "core/FlowConfiguration.h"
-bool TemplateFactory::added = core::FlowConfiguration::add_static_func("createTemplateFactory");
+#pragma once
-extern "C" {
-
-void *createTemplateFactory(void) {
- return new TemplateFactory();
-}
-}
+#ifdef WIN32
+ #ifdef LIBMINIFI
+ #define MINIFIAPI __declspec(dllexport)
+ #else
+ #define MINIFIAPI __declspec(dllimport)
+ #endif
+ #ifdef MODULE_NAME
+ #define EXTENSIONAPI __declspec(dllexport)
+ #else
+ #define EXTENSIONAPI __declspec(dllimport)
+ #endif
+#else
+ #define MINIFIAPI
+ #define EXTENSIONAPI
+#endif
Review comment:
also we will want to rework what we export anyway, so this is more like a temporary solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r693909181
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
Review comment:
The only was to signal an error from constructors is through exceptions, so it's hard to avoid them there. But in general I think exceptions are a great way to signal error.
Enforcing verification criteria in the constructor would allow to have the result of future verification be a class invariant, thus reducing the class state.
My exception usage preferences are quite the opposite of yours: [Using exceptions for errors only](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#e3-use-exceptions-for-error-handling-only), [including establishing invariants](https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#e5-let-a-constructor-establish-an-invariant-and-throw-if-it-cannot), but not part of the happy path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697342207
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
Review comment:
we can't turn a `std::string_view` directly to a null terminated const char* without allocation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r691837941
##########
File path: extensions/aws/controllerservices/AWSCredentialsService.h
##########
@@ -26,6 +26,7 @@
#include "utils/AWSInitializer.h"
#include "core/Resource.h"
+#include "utils/OptionalUtils.h"
Review comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r708994802
##########
File path: thirdparty/aws-sdk-cpp/dll-export-injection.patch
##########
@@ -0,0 +1,52 @@
+diff -rupN aws-sdk-cpp-src/aws-cpp-sdk-core/CMakeLists.txt aws-sdk-cpp-src-patched/aws-cpp-sdk-core/CMakeLists.txt
Review comment:
Yeah on second glance it should be explicit enough, it can be kept this way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697344625
##########
File path: libminifi/include/core/extension/DynamicLibrary.h
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <map>
+#include <string>
+#include <filesystem>
+
+#include "Module.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class DynamicLibrary : public Module {
+ friend class ExtensionManager;
+
+ public:
+ DynamicLibrary(std::string name, std::filesystem::path library_path);
+ ~DynamicLibrary() override;
+
+ private:
+#ifdef WIN32
+ std::map<void*, std::string> resource_mapping_;
+
+ std::string error_str_;
+ std::string current_error_;
+
+ void store_error();
+ void* dlsym(void* handle, const char* name);
+ const char* dlerror();
+ void* dlopen(const char* file, int mode);
+ int dlclose(void* handle);
+#endif
+
+ bool load();
+ bool unload();
+
+ std::filesystem::path library_path_;
+ gsl::owner<void*> handle_ = nullptr;
+
+ static std::shared_ptr<logging::Logger> logger_;
Review comment:
done
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::filesystem::path& path) {
Review comment:
moved it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r691878972
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
regarding the config access in static init, some/most tests link to shared libraries startup time when the config is not yet available (not like the tests use the config so we could return an empty config, but then they would differ semantically from the main application)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r696679051
##########
File path: libminifi/include/utils/file/FileMatcher.h
##########
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FileMatcherTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FileMatcher {
Review comment:
on gcc < 9, we now have to link to `stdc++fs` to use the `std::filesystem`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r677585723
##########
File path: cmake/Extensions.cmake
##########
@@ -26,7 +26,27 @@ macro(register_extension extension-name)
get_property(extensions GLOBAL PROPERTY EXTENSION-OPTIONS)
set_property(GLOBAL APPEND PROPERTY EXTENSION-OPTIONS ${extension-name})
target_compile_definitions(${extension-name} PRIVATE "MODULE_NAME=${extension-name}")
- set_target_properties(${extension-name} PROPERTIES ENABLE_EXPORTS True)
+ set_target_properties(${extension-name} PROPERTIES
+ ENABLE_EXPORTS True
+ POSITION_INDEPENDENT_CODE ON)
+ if (WIN32)
+ set_target_properties(${extension-name} PROPERTIES
+ RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ WINDOWS_EXPORT_ALL_SYMBOLS TRUE)
+ else()
+ set_target_properties(${extension-name} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin")
+ endif()
+
+ if (WIN32)
+ install(TARGETS ${extension-name} RUNTIME DESTINATION extensions COMPONENT bin)
+ else()
+ if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
+ target_link_options(${extension-name} PRIVATE "-Wl,--disable-new-dtags")
+ endif()
+ set_target_properties(${extension-name} PROPERTIES INSTALL_RPATH "$ORIGIN")
Review comment:
does this work? no escaping of the `$` is needed?
there is also a BUILD_RPATH_USE_ORIGIN CMake option that may be useful here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r693905746
##########
File path: libminifi/include/core/extension/ExtensionManager.h
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "core/logging/Logger.h"
+#include "Module.h"
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+/**
+ * Comma separated list of path patterns. Patterns prepended with "!" result in the exclusion
+ * of the extensions matching that pattern, unless some subsequent pattern re-enables it.
+ */
+static constexpr const char* nifi_extension_path = "nifi.extension.path";
+
+class ExtensionManager {
+ ExtensionManager();
+
+ public:
+ static ExtensionManager& get();
Review comment:
the extensions would need to register themselves somewhere (possible during startup time) so the need for a globally accessible reference to the ExtensionManager remains, also it does not make much sense to have multiple ExtensionManagers (so best to prevent that), moreover the destruction of an (the) ExtensionManager would not mean the unloading of libraries (works on some systems but not on linux), so the managed extensions would outlive their manager
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r691877148
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
removed the inheritance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697343808
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
Review comment:
I just realized this handler is unused, so removed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680918382
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+#ifdef WIN32
+static const std::vector<std::string> path_separators{"/", "\\"};
+#else
+static const std::vector<std::string> path_separators{"/"};
+#endif
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern) {
+ pattern = utils::StringUtils::trim(pattern);
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
+ excluding = true;
+ pattern = utils::StringUtils::trim(pattern.substr(1));
+ }
+ if (pattern.empty()) {
+ logger_->log_error("Empty pattern");
+ return nullopt;
+ }
+ std::string exe_dir = get_executable_dir();
+ if (exe_dir.empty() && !isAbsolutePath(pattern.c_str())) {
+ logger_->log_error("Couldn't determine executable dir, relative pattern '%s' not supported", pattern);
+ return nullopt;
+ }
+ pattern = resolve(exe_dir, pattern);
+ auto segments = split(pattern, path_separators);
+ gsl_Expects(!segments.empty());
+ auto file_pattern = segments.back();
+ if (file_pattern == "**") {
+ file_pattern = "*";
+ } else {
+ segments.pop_back();
+ }
+ if (file_pattern == "." || file_pattern == "..") {
+ logger_->log_error("Invalid file pattern '%s'", file_pattern);
+ return nullopt;
+ }
+ bool after_wildcard = false;
+ for (const auto& segment : segments) {
+ if (after_wildcard && segment == "..") {
+ logger_->log_error("Parent accessor is not supported after wildcards");
+ return nullopt;
+ }
+ if (isGlobPattern(segment)) {
+ after_wildcard = true;
+ }
+ }
+ return FilePattern(segments, file_pattern, excluding);
+}
+
+std::string FileMatcher::FilePattern::getBaseDirectory() const {
+ std::string base_dir;
+ for (const auto& segment : directory_segments_) {
+ // ignore segments at or after wildcards
+ if (isGlobPattern(segment)) {
+ break;
+ }
+ base_dir += segment + get_separator();
+ }
+ return base_dir;
+}
+
+FileMatcher::FileMatcher(const std::string &patterns) {
+ for (auto&& pattern : split(patterns, {","})) {
+ if (auto&& p = FilePattern::fromPattern(pattern)) {
+ patterns_.push_back(std::move(p.value()));
+ }
+ }
+}
+
+template<typename It>
+static bool advance_if_not_equal(It& it, const It& end) {
+ if (it == end) {
+ return false;
+ }
+ ++it;
+ return true;
+}
+
+static bool is_this_dir(const std::string& dir) {
+ return dir.empty() || dir == ".";
+}
+
+template<typename It, typename Fn>
+static void skip_if(It& it, const It& end, const Fn& fn) {
+ while (it != end && fn(*it)) {
+ ++it;
+ }
+}
+
+static bool matchGlob(std::string::const_iterator pattern_it, std::string::const_iterator pattern_end, std::string::const_iterator value_it, std::string::const_iterator value_end) {
Review comment:
#1142 seems to be quite close to merge, but I was originally thinking about substr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678101677
##########
File path: extensions/libarchive/ManipulateArchive.h
##########
@@ -27,6 +27,9 @@
#include "core/Processor.h"
#include "core/ProcessSession.h"
+#include "FocusArchiveEntry.h"
+#include "UnfocusArchiveEntry.h"
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678899881
##########
File path: libminifi/src/FlowController.cpp
##########
@@ -60,7 +60,7 @@ namespace minifi {
FlowController::FlowController(std::shared_ptr<core::Repository> provenance_repo, std::shared_ptr<core::Repository> flow_file_repo,
std::shared_ptr<Configure> configure, std::unique_ptr<core::FlowConfiguration> flow_configuration,
- std::shared_ptr<core::ContentRepository> content_repo, const std::string /*name*/, bool headless_mode,
Review comment:
you are right, the same thing (not initializing python and jni extension) can now be achieved by setting the extension list in the config accordingly (excluding these extensions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r679834067
##########
File path: libminifi/include/utils/file/FileUtils.h
##########
@@ -500,13 +500,26 @@ inline void addFilesMatchingExtension(const std::shared_ptr<logging::Logger> &lo
#endif
}
+inline std::string concat_path(const std::string& root, const std::string& child, bool force_posix = false) {
+ if (root.empty()) {
+ return child;
+ }
+ std::stringstream new_path;
+ if (root.back() == get_separator(force_posix)) {
+ new_path << root << child;
+ } else {
+ new_path << root << get_separator(force_posix) << child;
+ }
+ return new_path.str();
+}
+
/*
* Provides a platform-independent function to list a directory
* Callback is called for every file found: first argument is the path of the directory, second is the filename
* Return value of the callback is used to continue (true) or stop (false) listing
*/
Review comment:
added
##########
File path: libminifi/test/unit/FileMatcherTests.cpp
##########
@@ -0,0 +1,208 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#define CUSTOM_EXTENSION_LIST
+
+#include "../TestBase.h"
+#include "../Path.h"
+#include "utils/file/FileMatcher.h"
+
+struct FileMatcherTestAccessor {
+ using FilePattern = fileutils::FileMatcher::FilePattern;
+};
+
+using FilePattern = FileMatcherTestAccessor::FilePattern;
+using FileMatcher = fileutils::FileMatcher;
+
+TEST_CASE("Invalid paths") {
+ REQUIRE_FALSE(FilePattern::fromPattern(""));
+ REQUIRE_FALSE(FilePattern::fromPattern("."));
+ REQUIRE_FALSE(FilePattern::fromPattern(".."));
+ REQUIRE_FALSE(FilePattern::fromPattern("!"));
+ REQUIRE_FALSE(FilePattern::fromPattern("!."));
+ REQUIRE_FALSE(FilePattern::fromPattern("!.."));
+}
+
+TEST_CASE("Matching directories without globs") {
+#ifdef WIN32
+ utils::Path root{"C:\\"};
+#else
+ utils::Path root{"/"};
+#endif
+ auto pattern = FilePattern::fromPattern((root / "one" / "banana" / "file").str()).value();
+ REQUIRE(pattern.match((root / "one").str()));
+ REQUIRE(pattern.match((root / "one" / "banana").str()));
+ REQUIRE_FALSE(pattern.match((root / "two").str()));
+ REQUIRE_FALSE(pattern.match((root / "one" / "apple").str()));
+ REQUIRE_FALSE(pattern.match((root / "one" / "banana" / "file").str()));
Review comment:
added comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r693952215
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::string getFullPath() const {
+ return utils::file::PathUtils::concat_path(dir, filename);
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::string& dir, const std::string& filename) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ dir,
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ std::vector<LibraryDescriptor> libraries;
+ utils::file::FileMatcher(pattern.value()).forEachFile([&] (const std::string& dir, const std::string& filename) {
Review comment:
as I took a quick peek I see there are some quirks to `ranges`
could you expand on this with implementation? if it is possible to use ranges to make this more concise, you could
have a better shot at it
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::string getFullPath() const {
+ return utils::file::PathUtils::concat_path(dir, filename);
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::string& dir, const std::string& filename) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ dir,
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ std::vector<LibraryDescriptor> libraries;
+ utils::file::FileMatcher(pattern.value()).forEachFile([&] (const std::string& dir, const std::string& filename) {
Review comment:
as I took a quick peek I see there are some quirks to `ranges`
could you expand on this with implementation? if it is possible to use ranges to make this more concise, you could have a better shot at it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r694549392
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::string getFullPath() const {
+ return utils::file::PathUtils::concat_path(dir, filename);
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::string& dir, const std::string& filename) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ dir,
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ std::vector<LibraryDescriptor> libraries;
+ utils::file::FileMatcher(pattern.value()).forEachFile([&] (const std::string& dir, const std::string& filename) {
Review comment:
after consideration I reverted the change, as I do not believe it improves readability to justify the substantial change required to be included in an already bloated PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits closed pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits closed pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#issuecomment-920042078
the aws extension is now dynamically loaded into all tests (unless otherwise specified), which then tries to fetch metadata causing C2 tests to take longer, thus fail, we disabled this metadata retrieval in all tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680713185
##########
File path: libminifi/test/LogUtils.h
##########
@@ -0,0 +1,51 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <sstream>
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "spdlog/sinks/sink.h"
+#include "spdlog/sinks/ostream_sink.h"
+
+class StringStreamSink : public spdlog::sinks::sink {
+ public:
+ explicit StringStreamSink(std::shared_ptr<std::ostringstream> stream, bool force_flush = false)
+ : stream_(std::move(stream)), sink_(*stream_, force_flush) {}
+
+ ~StringStreamSink() override = default;
+
+ void log(const spdlog::details::log_msg &msg) override {
+ sink_.log(msg);
+ }
+ void flush() override {
+ sink_.flush();
+ }
+ void set_pattern(const std::string &pattern) override {
+ sink_.set_pattern(pattern);
+ }
+ void set_formatter(std::unique_ptr<spdlog::formatter> sink_formatter) override {
+ sink_.set_formatter(std::move(sink_formatter));
+ }
+
+ private:
+ std::shared_ptr<std::ostringstream> stream_;
Review comment:
unfortunately it is an owning ptr on purpose, I encountered crashes when LogTestController got destroyed, while a sink was still being used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697344848
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::filesystem::path& path) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
Review comment:
changed them to `std::string_view`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680913485
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+#ifdef WIN32
+static const std::vector<std::string> path_separators{"/", "\\"};
+#else
+static const std::vector<std::string> path_separators{"/"};
+#endif
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern) {
+ pattern = utils::StringUtils::trim(pattern);
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
+ excluding = true;
+ pattern = utils::StringUtils::trim(pattern.substr(1));
+ }
+ if (pattern.empty()) {
+ logger_->log_error("Empty pattern");
+ return nullopt;
+ }
+ std::string exe_dir = get_executable_dir();
+ if (exe_dir.empty() && !isAbsolutePath(pattern.c_str())) {
+ logger_->log_error("Couldn't determine executable dir, relative pattern '%s' not supported", pattern);
+ return nullopt;
+ }
+ pattern = resolve(exe_dir, pattern);
+ auto segments = split(pattern, path_separators);
+ gsl_Expects(!segments.empty());
+ auto file_pattern = segments.back();
+ if (file_pattern == "**") {
+ file_pattern = "*";
+ } else {
+ segments.pop_back();
+ }
+ if (file_pattern == "." || file_pattern == "..") {
+ logger_->log_error("Invalid file pattern '%s'", file_pattern);
+ return nullopt;
+ }
+ bool after_wildcard = false;
+ for (const auto& segment : segments) {
+ if (after_wildcard && segment == "..") {
+ logger_->log_error("Parent accessor is not supported after wildcards");
+ return nullopt;
+ }
+ if (isGlobPattern(segment)) {
+ after_wildcard = true;
+ }
+ }
+ return FilePattern(segments, file_pattern, excluding);
+}
+
+std::string FileMatcher::FilePattern::getBaseDirectory() const {
+ std::string base_dir;
+ for (const auto& segment : directory_segments_) {
+ // ignore segments at or after wildcards
+ if (isGlobPattern(segment)) {
+ break;
+ }
+ base_dir += segment + get_separator();
+ }
+ return base_dir;
+}
+
+FileMatcher::FileMatcher(const std::string &patterns) {
+ for (auto&& pattern : split(patterns, {","})) {
+ if (auto&& p = FilePattern::fromPattern(pattern)) {
+ patterns_.push_back(std::move(p.value()));
+ }
+ }
+}
+
+template<typename It>
+static bool advance_if_not_equal(It& it, const It& end) {
+ if (it == end) {
+ return false;
+ }
+ ++it;
+ return true;
+}
+
+static bool is_this_dir(const std::string& dir) {
+ return dir.empty() || dir == ".";
+}
+
+template<typename It, typename Fn>
+static void skip_if(It& it, const It& end, const Fn& fn) {
+ while (it != end && fn(*it)) {
+ ++it;
+ }
+}
+
+static bool matchGlob(std::string::const_iterator pattern_it, std::string::const_iterator pattern_end, std::string::const_iterator value_it, std::string::const_iterator value_end) {
Review comment:
rewrote it to use `string_view`, although using the iterator constructor for `std::string_view` would be nice, unfortunately not until c++20
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697344425
##########
File path: libminifi/include/core/logging/Logger.h
##########
@@ -70,6 +75,8 @@ inline T conditional_conversion(T t) {
return t;
}
+
+
Review comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680002877
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern, bool log_errors) {
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
+ excluding = true;
+ pattern = pattern.substr(1);
+ }
+ if (pattern.empty()) {
+ if (log_errors) logger_->log_error("Empty pattern");
+ return nullopt;
+ }
+ std::string exe_dir = get_executable_dir();
+ if (exe_dir.empty() && !isAbsolutePath(pattern.c_str())) {
+ if (log_errors) logger_->log_error("Couldn't determine executable dir, relative pattern '%s' not supported", pattern);
+ return nullopt;
+ }
+ pattern = resolve(exe_dir, pattern);
+ auto segments = split(pattern, {"/", "\\"});
+ gsl_Expects(!segments.empty());
+ auto file_pattern = segments.back();
+ if (file_pattern == "**") {
+ file_pattern = "*";
+ } else {
+ segments.pop_back();
+ }
+ if (file_pattern == "." || file_pattern == "..") {
+ if (log_errors) logger_->log_error("Invalid file pattern '%s'", file_pattern);
+ return nullopt;
+ }
+ return FilePattern(segments, file_pattern, excluding);
+}
+
+std::string FileMatcher::FilePattern::getBaseDirectory() const {
+ std::string base_dir;
+ for (const auto& segment : directory_segments_) {
+ // ignore segments at or after wildcards
+ if (isGlobPattern(segment)) {
+ break;
+ }
+ base_dir += segment + get_separator();
+ }
+ return base_dir;
+}
+
+FileMatcher::FileMatcher(const std::string &patterns) {
+ for (auto&& pattern : split(patterns, {","})) {
+ if (auto&& p = FilePattern::fromPattern(pattern)) {
+ patterns_.push_back(std::move(p.value()));
+ }
+ }
+}
+
+template<typename It>
+static bool advance_if_not_equal(It& it, const It& end) {
+ if (it == end) {
+ return false;
+ }
+ ++it;
+ return true;
+}
+
+static bool is_this_dir(const std::string& dir) {
+ return dir.empty() || dir == ".";
+}
+
+template<typename It, typename Fn>
+static void skip_if(It& it, const It& end, const Fn& fn) {
+ while (it != end && fn(*it)) {
+ ++it;
+ }
+}
+
+static bool matchGlob(std::string::const_iterator pattern_it, std::string::const_iterator pattern_end, std::string::const_iterator value_it, std::string::const_iterator value_end) {
+ // match * and ?
+ for (; pattern_it != pattern_end; ++pattern_it) {
+ if (*pattern_it == '*') {
+ do {
+ if (matchGlob(std::next(pattern_it), pattern_end, value_it, value_end)) {
+ return true;
+ }
+ } while (advance_if_not_equal(value_it, value_end));
+ return false;
+ }
+ if (value_it == value_end) {
+ return false;
+ }
+ if (*pattern_it != '?' && *pattern_it != *value_it) {
+ return false;
+ }
+ ++value_it;
+ }
+ return value_it == value_end;
+}
+
+FileMatcher::FilePattern::DirMatchResult FileMatcher::FilePattern::matchDirectory(DirIt pattern_it, DirIt pattern_end, DirIt value_it, DirIt value_end) {
+ for (; pattern_it != pattern_end; ++pattern_it) {
+ if (is_this_dir(*pattern_it)) {
+ continue;
+ }
+ if (*pattern_it == "**") {
+ if (std::next(pattern_it) == pattern_end) {
+ return DirMatchResult::TREE;
+ }
+ bool matched_parent = false;
+ // any number of nested directories
+ do {
+ skip_if(value_it, value_end, is_this_dir);
+ auto result = matchDirectory(std::next(pattern_it), pattern_end, value_it, value_end);
+ if (result == DirMatchResult::TREE || result == DirMatchResult::EXACT) {
+ return result;
+ }
+ if (result == DirMatchResult::PARENT) {
+ // even though we have a parent match, there may be a "better" (exact, tree) match
+ matched_parent = true;
+ }
+ } while (advance_if_not_equal(value_it, value_end));
+ if (matched_parent) {
+ return DirMatchResult::PARENT;
+ }
+ return DirMatchResult::NONE;
+ }
+ skip_if(value_it, value_end, is_this_dir);
+ if (value_it == value_end) {
+ // we used up all the value segments but there are still pattern segments
+ return DirMatchResult::PARENT;
+ }
+ if (!matchGlob(pattern_it->begin(), pattern_it->end(), value_it->begin(), value_it->end())) {
+ return DirMatchResult::NONE;
+ }
+ ++value_it;
+ }
+ skip_if(value_it, value_end, is_this_dir);
+ if (value_it == value_end) {
+ // used up all pattern and value segments
+ return DirMatchResult::EXACT;
+ } else {
+ // used up all pattern segments but we still have value segments
+ return DirMatchResult::NONE;
+ }
+}
+
+bool FileMatcher::FilePattern::match(const std::string& directory, const optional<std::string>& filename) const {
+ auto value = split(directory, {"/", "\\"});
+ auto result = matchDirectory(directory_segments_.begin(), directory_segments_.end(), value.begin(), value.end());
+ if (!filename) {
+ if (excluding_) {
+ if (result == DirMatchResult::TREE && file_pattern_ == "*") {
+ // all files are excluded in this directory
+ return true;
Review comment:
Yes, the new version is clearer. However, I would completely separate the directory matching from the file matching; there will be some duplication of code, but I think it is going to be easier to understand and maintain. This is what I have in mind: https://github.com/fgerlits/nifi-minifi-cpp/commit/c271b0f612b704bcb72e48afb709eaa4b8f3a377
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678172665
##########
File path: extensions/script/python/PythonObjectFactory.cpp
##########
@@ -15,9 +15,5 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include "PyProcCreator.h"
+#include "PythonObjectFactory.h"
Review comment:
There is nothing left here. Can this file be deleted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r683314053
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+#ifdef WIN32
+static const std::vector<std::string> path_separators{"/", "\\"};
+#else
+static const std::vector<std::string> path_separators{"/"};
+#endif
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern) {
+ pattern = utils::StringUtils::trim(pattern);
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
+ excluding = true;
+ pattern = utils::StringUtils::trim(pattern.substr(1));
+ }
+ if (pattern.empty()) {
+ logger_->log_error("Empty pattern");
+ return nullopt;
+ }
+ std::string exe_dir = get_executable_dir();
+ if (exe_dir.empty() && !isAbsolutePath(pattern.c_str())) {
+ logger_->log_error("Couldn't determine executable dir, relative pattern '%s' not supported", pattern);
+ return nullopt;
+ }
+ pattern = resolve(exe_dir, pattern);
+ auto segments = split(pattern, path_separators);
+ gsl_Expects(!segments.empty());
+ auto file_pattern = segments.back();
+ if (file_pattern == "**") {
+ file_pattern = "*";
+ } else {
+ segments.pop_back();
+ }
+ if (file_pattern == "." || file_pattern == "..") {
+ logger_->log_error("Invalid file pattern '%s'", file_pattern);
+ return nullopt;
+ }
+ bool after_wildcard = false;
+ for (const auto& segment : segments) {
+ if (after_wildcard && segment == "..") {
+ logger_->log_error("Parent accessor is not supported after wildcards");
+ return nullopt;
+ }
+ if (isGlobPattern(segment)) {
+ after_wildcard = true;
+ }
+ }
+ return FilePattern(segments, file_pattern, excluding);
+}
+
+std::string FileMatcher::FilePattern::getBaseDirectory() const {
+ std::string base_dir;
+ for (const auto& segment : directory_segments_) {
+ // ignore segments at or after wildcards
+ if (isGlobPattern(segment)) {
+ break;
+ }
+ base_dir += segment + get_separator();
+ }
+ return base_dir;
+}
+
+FileMatcher::FileMatcher(const std::string &patterns) {
+ for (auto&& pattern : split(patterns, {","})) {
+ if (auto&& p = FilePattern::fromPattern(pattern)) {
+ patterns_.push_back(std::move(p.value()));
+ }
+ }
+}
+
+template<typename It>
+static bool advance_if_not_equal(It& it, const It& end) {
+ if (it == end) {
+ return false;
+ }
+ ++it;
+ return true;
+}
+
+static bool is_this_dir(const std::string& dir) {
+ return dir.empty() || dir == ".";
+}
+
+template<typename It, typename Fn>
+static void skip_if(It& it, const It& end, const Fn& fn) {
+ while (it != end && fn(*it)) {
+ ++it;
+ }
+}
+
+static bool matchGlob(std::string::const_iterator pattern_it, std::string::const_iterator pattern_end, std::string::const_iterator value_it, std::string::const_iterator value_end) {
Review comment:
it seems that on my machine `Apple clang version 11.0.3` does not support that constructor even with the `-std=c++2a` flag, `substr` it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r705061409
##########
File path: libminifi/include/core/extension/DynamicLibrary.h
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <map>
+#include <string>
+#include <filesystem>
+
+#include "Module.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class DynamicLibrary : public Module {
+ friend class ExtensionManager;
+
+ public:
+ DynamicLibrary(std::string name, std::filesystem::path library_path);
+ ~DynamicLibrary() override;
+
+ private:
+#ifdef WIN32
+ std::map<void*, std::string> resource_mapping_;
+
+ std::string error_str_;
+ std::string current_error_;
+
+ void store_error();
+ void* dlsym(void* handle, const char* name);
+ const char* dlerror();
+ void* dlopen(const char* file, int mode);
+ int dlclose(void* handle);
+#endif
+
+ bool load();
+ bool unload();
+
+ std::filesystem::path library_path_;
+ gsl::owner<void*> handle_ = nullptr;
+
+ static std::shared_ptr<logging::Logger> logger_;
Review comment:
should be done now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r699111472
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+ public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
+ }
+
+ public:
+ explicit FilePatternSegment(std::string pattern);
+
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ NOT_MATCHING // dir/file does not match pattern, do what you may
+ };
+
+ bool isExcluding() const {
+ return excluding_;
+ }
+
+ MatchResult match(const std::string& directory) const;
+
+ MatchResult match(const std::string& directory, const std::string& filename) const;
+
+ MatchResult match(const std::filesystem::path& path) const;
+ /**
+ * @return The lowermost parent directory without wildcards.
+ */
+ std::filesystem::path getBaseDirectory() const;
+
+ private:
+ enum class DirMatchResult {
+ NONE, // pattern does not match the directory (e.g. p = "/home/inner/*test", v = "/home/banana")
+ PARENT, // directory is a parent of the pattern (e.g. p = "/home/inner/*test", v = "/home/inner")
+ EXACT, // pattern exactly matches the directory (e.g. p = "/home/inner/*test", v = "/home/inner/cool_test")
+ TREE // pattern matches the whole subtree of the directory (e.g. p = "/home/**", v = "/home/banana")
+ };
+
+ using DirIt = std::filesystem::path::const_iterator;
+ static DirMatchResult matchDirectory(DirIt pattern_begin, DirIt pattern_end, DirIt value_begin, DirIt value_end);
+
+ std::filesystem::path directory_pattern_;
+ std::string file_pattern_;
+ bool excluding_;
+ };
+
+ using ErrorHandler = std::function<void(std::string_view /*subpattern*/, std::string_view /*error_message*/)>;
+
+ static void defaultErrorHandler(std::string_view subpattern, std::string_view error_message) {
+ std::string message = "Error in subpattern '";
+ message += subpattern;
+ message += "': ";
+ message += error_message;
+ throw FilePatternError(message);
+ }
+
+ public:
+ explicit FilePattern(const std::string& pattern, ErrorHandler error_handler = defaultErrorHandler);
+
+ private:
+ std::vector<FilePatternSegment> segments_;
+};
+
+std::set<std::filesystem::path> match(const FilePattern& pattern);
Review comment:
I just prefer vector as the default unless we have a compelling reason not to use it. If you still feel that set is a better fit, then feel free to revert this change.
I also did some measurements and they didn't show a significant difference in performance:
```
--------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------
BM_MatchVector/4 150744 ns 148394 ns 4553
BM_MatchVector/8 534604 ns 526563 ns 1330
BM_MatchVector/64 31438122 ns 30888763 ns 23
BM_MatchVector/512 2112878189 ns 2075348002 ns 1
BM_MatchVector/2048 3325256795 ns 3258781806 ns 1
BM_MatchSet/4 151805 ns 149650 ns 4707
BM_MatchSet/8 534444 ns 527351 ns 1328
BM_MatchSet/64 31374911 ns 30869050 ns 23
BM_MatchSet/512 2110077546 ns 2075058340 ns 1
BM_MatchSet/2048 3297083600 ns 3242293291 ns 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r698395494
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+ public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
+ }
+
+ public:
+ explicit FilePatternSegment(std::string pattern);
+
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ NOT_MATCHING // dir/file does not match pattern, do what you may
+ };
+
+ bool isExcluding() const {
+ return excluding_;
+ }
+
+ MatchResult match(const std::string& directory) const;
+
+ MatchResult match(const std::string& directory, const std::string& filename) const;
+
+ MatchResult match(const std::filesystem::path& path) const;
+ /**
+ * @return The lowermost parent directory without wildcards.
+ */
+ std::filesystem::path getBaseDirectory() const;
+
+ private:
+ enum class DirMatchResult {
+ NONE, // pattern does not match the directory (e.g. p = "/home/inner/*test", v = "/home/banana")
+ PARENT, // directory is a parent of the pattern (e.g. p = "/home/inner/*test", v = "/home/inner")
+ EXACT, // pattern exactly matches the directory (e.g. p = "/home/inner/*test", v = "/home/inner/cool_test")
+ TREE // pattern matches the whole subtree of the directory (e.g. p = "/home/**", v = "/home/banana")
+ };
+
+ using DirIt = std::filesystem::path::const_iterator;
+ static DirMatchResult matchDirectory(DirIt pattern_begin, DirIt pattern_end, DirIt value_begin, DirIt value_end);
+
+ std::filesystem::path directory_pattern_;
+ std::string file_pattern_;
+ bool excluding_;
+ };
+
+ using ErrorHandler = std::function<void(std::string_view /*subpattern*/, std::string_view /*error_message*/)>;
+
+ static void defaultErrorHandler(std::string_view subpattern, std::string_view error_message) {
+ std::string message = "Error in subpattern '";
+ message += subpattern;
+ message += "': ";
+ message += error_message;
+ throw FilePatternError(message);
+ }
+
+ public:
+ explicit FilePattern(const std::string& pattern, ErrorHandler error_handler = defaultErrorHandler);
+
+ private:
+ std::vector<FilePatternSegment> segments_;
+};
+
+std::set<std::filesystem::path> match(const FilePattern& pattern);
Review comment:
rewrote it to return `std::vector` (but I still feel like we traded a container with inherent order/uniqueness guarantees for a hand-rolled one for unmeasured performance gains)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680775139
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
gcc const/destr (or any load-time running code DllMain) is not viable here as some extensions (e.g. python extension, jni extension) use the configuration object passed onto it to initialize themselves
also there could be interdependencies between extensions, for example loading `http-curl-extension` loads `civet-extension`, so if we only rely on what we load with `dlopen` we could miss out on `civet-extension`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r699279695
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+ public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
+ }
+
+ public:
+ explicit FilePatternSegment(std::string pattern);
+
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ NOT_MATCHING // dir/file does not match pattern, do what you may
+ };
+
+ bool isExcluding() const {
+ return excluding_;
+ }
+
+ MatchResult match(const std::string& directory) const;
+
+ MatchResult match(const std::string& directory, const std::string& filename) const;
+
+ MatchResult match(const std::filesystem::path& path) const;
+ /**
+ * @return The lowermost parent directory without wildcards.
+ */
+ std::filesystem::path getBaseDirectory() const;
+
+ private:
+ enum class DirMatchResult {
+ NONE, // pattern does not match the directory (e.g. p = "/home/inner/*test", v = "/home/banana")
+ PARENT, // directory is a parent of the pattern (e.g. p = "/home/inner/*test", v = "/home/inner")
+ EXACT, // pattern exactly matches the directory (e.g. p = "/home/inner/*test", v = "/home/inner/cool_test")
+ TREE // pattern matches the whole subtree of the directory (e.g. p = "/home/**", v = "/home/banana")
+ };
+
+ using DirIt = std::filesystem::path::const_iterator;
+ static DirMatchResult matchDirectory(DirIt pattern_begin, DirIt pattern_end, DirIt value_begin, DirIt value_end);
+
+ std::filesystem::path directory_pattern_;
+ std::string file_pattern_;
+ bool excluding_;
+ };
+
+ using ErrorHandler = std::function<void(std::string_view /*subpattern*/, std::string_view /*error_message*/)>;
+
+ static void defaultErrorHandler(std::string_view subpattern, std::string_view error_message) {
+ std::string message = "Error in subpattern '";
+ message += subpattern;
+ message += "': ";
+ message += error_message;
+ throw FilePatternError(message);
+ }
+
+ public:
+ explicit FilePattern(const std::string& pattern, ErrorHandler error_handler = defaultErrorHandler);
+
+ private:
+ std::vector<FilePatternSegment> segments_;
+};
+
+std::set<std::filesystem::path> match(const FilePattern& pattern);
Review comment:
in light of the numbers I reverted back to `std::set` (in my own measurements I found that the runtime is dominated by the filesystem operations)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
lordgamez commented on pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#issuecomment-919821985
Currently the C2 tests fail locally when I try to run them, have you encountered this before?
The following tests FAILED:
95 - C2JstackTest (Child aborted)
100 - C2RequestClassTest (Child aborted)
102 - C2VerifyHeartbeatAndStopSecure (Child aborted)
107 - C2VerifyHeartbeatAndStop (Child aborted)
108 - C2VerifyResourceConsumptionInHeartbeat (Child aborted)
115 - C2PauseResumeTest (Child aborted)
116 - C2LogHeartbeatTest (Child aborted)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680902055
##########
File path: libminifi/test/LogUtils.h
##########
@@ -0,0 +1,51 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <sstream>
+#include <memory>
+#include <string>
+#include <utility>
+
+#include "spdlog/sinks/sink.h"
+#include "spdlog/sinks/ostream_sink.h"
+
+class StringStreamSink : public spdlog::sinks::sink {
+ public:
+ explicit StringStreamSink(std::shared_ptr<std::ostringstream> stream, bool force_flush = false)
+ : stream_(std::move(stream)), sink_(*stream_, force_flush) {}
+
+ ~StringStreamSink() override = default;
+
+ void log(const spdlog::details::log_msg &msg) override {
+ sink_.log(msg);
+ }
+ void flush() override {
+ sink_.flush();
+ }
+ void set_pattern(const std::string &pattern) override {
+ sink_.set_pattern(pattern);
+ }
+ void set_formatter(std::unique_ptr<spdlog::formatter> sink_formatter) override {
+ sink_.set_formatter(std::move(sink_formatter));
+ }
+
+ private:
+ std::shared_ptr<std::ostringstream> stream_;
Review comment:
added comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r694717942
##########
File path: libminifi/include/utils/file/FileMatcher.h
##########
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FileMatcherTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FileMatcher {
Review comment:
The implementation could still involve breaking up the code to different classes, I would just prefer the interface to be a function, just like `std::regex_match` is one. I think FilePattern is a better analogy to `std::regex`, although it doesn't involve expensive regex compilation in this case. What do you think about `utils::file::match(std::vector<FilePattern>)`? `forEachFile` could become a loop at the usage side instead.
If the pattern is invalid, the FilePattern constructor could throw. If there are issues related to the filesystem, then the operation should throw. If a directory in a pattern doesn't exist, then that could be an empty match without failure, allowing us to load the rest of the extensions. There could be a function in ExtensionManager.cpp that converts the patterns string to a `vector<FilePattern>` and logs on error.
```
// ExtensionManager.cpp, anonymous namespace
std::vector<FilePattern> splitPatterns(std::string_view patterns, core::logging::Logger& logger) {
std::vector<FilePattern> result;
const auto patterns_vector = StringUtils::split(patterns, ",");
result.reserve(patterns_vector.size());
for (auto& pattern_str: patterns_vector) {
auto pattern = FilePattern::fromPattern(pattern_str); // I would consider making this function a constructor
// currently this returns an optional, but if it threw, this would become a try block
if (pattern) result.push_back(*pattern);
else logger->log_warn("Invalid pattern: %s", pattern_str);
}
result.shrink_to_fit();
return result;
}
// in ExtensionManager::initialize
const auto files = utils::file::match(splitPatterns(*pattern, logger_));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r677585723
##########
File path: cmake/Extensions.cmake
##########
@@ -26,7 +26,27 @@ macro(register_extension extension-name)
get_property(extensions GLOBAL PROPERTY EXTENSION-OPTIONS)
set_property(GLOBAL APPEND PROPERTY EXTENSION-OPTIONS ${extension-name})
target_compile_definitions(${extension-name} PRIVATE "MODULE_NAME=${extension-name}")
- set_target_properties(${extension-name} PROPERTIES ENABLE_EXPORTS True)
+ set_target_properties(${extension-name} PROPERTIES
+ ENABLE_EXPORTS True
+ POSITION_INDEPENDENT_CODE ON)
+ if (WIN32)
+ set_target_properties(${extension-name} PROPERTIES
+ RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ WINDOWS_EXPORT_ALL_SYMBOLS TRUE)
+ else()
+ set_target_properties(${extension-name} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin")
+ endif()
+
+ if (WIN32)
+ install(TARGETS ${extension-name} RUNTIME DESTINATION extensions COMPONENT bin)
+ else()
+ if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
+ target_link_options(${extension-name} PRIVATE "-Wl,--disable-new-dtags")
+ endif()
+ set_target_properties(${extension-name} PROPERTIES INSTALL_RPATH "$ORIGIN")
Review comment:
does this work? no escaping of the `$` is needed?
there is also a BUILD_RPATH_USE_ORIGIN CMake option that may be useful here
EDIT: I tested it with `make u20` and it does work, so never mind
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r698370235
##########
File path: libminifi/src/core/extension/DynamicLibrary.cpp
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <memory>
+#ifndef WIN32
+#include <dlfcn.h>
+#define DLL_EXPORT
+#else
+#define WIN32_LEAN_AND_MEAN 1
+#include <Windows.h> // Windows specific libraries for collecting software metrics.
+#include <Psapi.h>
+#pragma comment(lib, "psapi.lib" )
+#define DLL_EXPORT __declspec(dllexport)
+#define RTLD_LAZY 0
+#define RTLD_NOW 0
+
+#define RTLD_GLOBAL (1 << 1)
+#define RTLD_LOCAL (1 << 2)
+#endif
+
+#include "core/extension/DynamicLibrary.h"
+#include "core/extension/Extension.h"
+#include "utils/GeneralUtils.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+std::shared_ptr<logging::Logger> DynamicLibrary::logger_ = logging::LoggerFactory<DynamicLibrary>::getLogger();
+
+DynamicLibrary::DynamicLibrary(std::string name, std::filesystem::path library_path)
+ : Module(std::move(name)),
+ library_path_(std::move(library_path)) {
+}
+
+bool DynamicLibrary::load() {
+ dlerror();
+ handle_ = dlopen(library_path_.c_str(), RTLD_NOW | RTLD_LOCAL);
+ if (!handle_) {
+ logger_->log_error("Failed to load extension '%s' at '%s': %s", name_, library_path_, dlerror());
+ return false;
+ } else {
+ logger_->log_trace("Loaded extension '%s' at '%s'", name_, library_path_);
+ return true;
+ }
+}
+
+bool DynamicLibrary::unload() {
+ logger_->log_trace("Unloading library '%s' at '%s'", name_, library_path_);
+ if (!handle_) {
+ logger_->log_error("Extension does not have a handle_ '%s' at '%s'", name_, library_path_);
+ return true;
+ }
+ dlerror();
+ if (dlclose(handle_)) {
+ logger_->log_error("Failed to unload extension '%s' at '%': %s", name_, library_path_, dlerror());
+ return false;
+ }
+ logger_->log_trace("Unloaded extension '%s' at '%s'", name_, library_path_);
+ handle_ = nullptr;
+ return true;
+}
+
+DynamicLibrary::~DynamicLibrary() = default;
+
+#ifdef WIN32
+
+void DynamicLibrary::store_error() {
+ auto error = GetLastError();
+
+ if (error == 0) {
+ error_str_ = "";
+ return;
+ }
+
+ LPSTR messageBuffer = nullptr;
+ size_t size = FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
+ NULL, error, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPSTR)&messageBuffer, 0, NULL);
+
+ current_error_ = std::string(messageBuffer, size);
+
+ // Free the buffer.
+ LocalFree(messageBuffer);
+}
Review comment:
- FORMAT_MESSAGE_ALLOCATE_BUFFER: For allocation, this is handled internally with system_error
- FORMAT_MESSAGE_FROM_SYSTEM: Looks in the system table, makes FormatMessage usable with GetLastError(). Same as system_error
- FORMAT_MESSAGE_IGNORE_INSERTS: Also passed by system_error: https://github.com/microsoft/STL/blob/31bed7ae0d2ff22029e7ed3e8c58d6ee429974fc/stl/src/syserror.cpp#L119
- MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT): system_error uses the default behavior instead (passing 0), which is very similar to this and it's fine IMO.
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-formatmessage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r705059132
##########
File path: libminifi/include/core/logging/Logger.h
##########
@@ -26,6 +26,7 @@
#include <iostream>
#include <vector>
#include <algorithm>
+#include <string_view>
Review comment:
remove
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,118 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <utility>
+#include <memory>
+#include <filesystem>
+#include <set>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
Review comment:
removed
##########
File path: libminifi/src/core/extension/DynamicLibrary.cpp
##########
@@ -0,0 +1,195 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <memory>
+#ifndef WIN32
+#include <dlfcn.h>
+#define DLL_EXPORT
+#else
+#include <system_error>
+#define WIN32_LEAN_AND_MEAN 1
+#include <Windows.h> // Windows specific libraries for collecting software metrics.
+#include <Psapi.h>
+#pragma comment(lib, "psapi.lib" )
+#define DLL_EXPORT __declspec(dllexport)
+#define RTLD_LAZY 0
+#define RTLD_NOW 0
+
+#define RTLD_GLOBAL (1 << 1)
+#define RTLD_LOCAL (1 << 2)
+#endif
+
+#include "core/extension/DynamicLibrary.h"
+#include "core/extension/Extension.h"
+#include "utils/GeneralUtils.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+std::shared_ptr<logging::Logger> DynamicLibrary::logger_ = logging::LoggerFactory<DynamicLibrary>::getLogger();
+
+DynamicLibrary::DynamicLibrary(std::string name, std::filesystem::path library_path)
+ : Module(std::move(name)),
+ library_path_(std::move(library_path)) {
+}
+
+bool DynamicLibrary::load() {
+ dlerror();
+ handle_ = dlopen(library_path_.string().c_str(), RTLD_NOW | RTLD_LOCAL);
+ if (!handle_) {
+ logger_->log_error("Failed to load extension '%s' at '%s': %s", name_, library_path_.string(), dlerror());
+ return false;
+ } else {
+ logger_->log_trace("Loaded extension '%s' at '%s'", name_, library_path_.string());
+ return true;
+ }
+}
+
+bool DynamicLibrary::unload() {
+ logger_->log_trace("Unloading library '%s' at '%s'", name_, library_path_.string());
+ if (!handle_) {
+ logger_->log_error("Extension does not have a handle_ '%s' at '%s'", name_, library_path_.string());
+ return true;
+ }
+ dlerror();
+ if (dlclose(handle_)) {
+ logger_->log_error("Failed to unload extension '%s' at '%': %s", name_, library_path_.string(), dlerror());
+ return false;
+ }
+ logger_->log_trace("Unloaded extension '%s' at '%s'", name_, library_path_.string());
+ handle_ = nullptr;
+ return true;
+}
+
+DynamicLibrary::~DynamicLibrary() = default;
+
+#ifdef WIN32
+
+void DynamicLibrary::store_error() {
+ auto error = GetLastError();
+
+ if (error == 0) {
+ error_str_ = "";
+ return;
+ }
+
+ current_error_ = std::system_category().message(error);
+}
+
+void* DynamicLibrary::dlsym(void* handle, const char* name) {
+ FARPROC symbol;
+
+ symbol = GetProcAddress((HMODULE)handle, name);
+
+ if (symbol == nullptr) {
+ store_error();
+
+ for (auto hndl : resource_mapping_) {
+ symbol = GetProcAddress((HMODULE)hndl.first, name);
+ if (symbol != nullptr) {
+ break;
+ }
+ }
+ }
+
+#ifdef _MSC_VER
+#pragma warning(suppress: 4054 )
+#endif
+ return reinterpret_cast<void*>(symbol);
+}
+
+const char* DynamicLibrary::dlerror() {
+ error_str_ = current_error_;
+
+ current_error_ = "";
+
+ return error_str_.c_str();
+}
+
+void* DynamicLibrary::dlopen(const char* file, int mode) {
+ HMODULE object;
+ uint32_t uMode = SetErrorMode(SEM_FAILCRITICALERRORS);
+ if (nullptr == file) {
+ HMODULE allModules[1024];
+ HANDLE current_process_handle = GetCurrentProcess();
+ DWORD cbNeeded;
+ object = GetModuleHandle(NULL);
+
+ if (!object) {
+ store_error();
+ }
+
+ if (EnumProcessModules(current_process_handle, allModules, sizeof(allModules), &cbNeeded) != 0) {
+ for (uint32_t i = 0; i < cbNeeded / sizeof(HMODULE); i++) {
+ // Get the full path to the module's file.
+ resource_mapping_.insert(std::make_pair(reinterpret_cast<void*>(allModules[i]), "minifi-system"));
Review comment:
changed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680691995
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
is the http-curl-extension not an extension? do we not initialize/deinitialize the extension? I feel like having two free-floating functions is not really the approach to best model their semantics
the constr/destr => init/deinit correspondance is used in ExtensionInitializer, if we drop that, drop the inheritance and rename HttpCurlExtension to HttpCurlExtensionInitializier and then register a function like
```
static bool httpCurlRegistrar = (ExtensionManager.get().register[what? not Extension, not Init]([] {
static HttpCurlExtensionInitializer initializer;
}, true);
```
then ExtensionManager cannot for example query the name of the extension it is initializing, because it is not initializing an extension (that would be `extension->init()`) but it just calls a function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r679004197
##########
File path: libminifi/test/unit/FileMatcherTests.cpp
##########
@@ -0,0 +1,208 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#define CUSTOM_EXTENSION_LIST
+
+#include "../TestBase.h"
+#include "../Path.h"
+#include "utils/file/FileMatcher.h"
+
+struct FileMatcherTestAccessor {
+ using FilePattern = fileutils::FileMatcher::FilePattern;
+};
+
+using FilePattern = FileMatcherTestAccessor::FilePattern;
+using FileMatcher = fileutils::FileMatcher;
+
+TEST_CASE("Invalid paths") {
+ REQUIRE_FALSE(FilePattern::fromPattern(""));
+ REQUIRE_FALSE(FilePattern::fromPattern("."));
+ REQUIRE_FALSE(FilePattern::fromPattern(".."));
+ REQUIRE_FALSE(FilePattern::fromPattern("!"));
+ REQUIRE_FALSE(FilePattern::fromPattern("!."));
+ REQUIRE_FALSE(FilePattern::fromPattern("!.."));
+}
+
+TEST_CASE("Matching directories without globs") {
+#ifdef WIN32
+ utils::Path root{"C:\\"};
+#else
+ utils::Path root{"/"};
+#endif
+ auto pattern = FilePattern::fromPattern((root / "one" / "banana" / "file").str()).value();
+ REQUIRE(pattern.match((root / "one").str()));
+ REQUIRE(pattern.match((root / "one" / "banana").str()));
+ REQUIRE_FALSE(pattern.match((root / "two").str()));
+ REQUIRE_FALSE(pattern.match((root / "one" / "apple").str()));
+ REQUIRE_FALSE(pattern.match((root / "one" / "banana" / "file").str()));
Review comment:
Why doesn't this match? A comment could be useful.
##########
File path: libminifi/include/utils/file/FileUtils.h
##########
@@ -500,13 +500,26 @@ inline void addFilesMatchingExtension(const std::shared_ptr<logging::Logger> &lo
#endif
}
+inline std::string concat_path(const std::string& root, const std::string& child, bool force_posix = false) {
+ if (root.empty()) {
+ return child;
+ }
+ std::stringstream new_path;
+ if (root.back() == get_separator(force_posix)) {
+ new_path << root << child;
+ } else {
+ new_path << root << get_separator(force_posix) << child;
+ }
+ return new_path.str();
+}
+
/*
* Provides a platform-independent function to list a directory
* Callback is called for every file found: first argument is the path of the directory, second is the filename
* Return value of the callback is used to continue (true) or stop (false) listing
*/
Review comment:
Please add a comment here about what `dir_callback` is for.
##########
File path: libminifi/test/script-tests/ExecutePythonProcessorTests.cpp
##########
@@ -119,7 +118,7 @@ class SimplePythonFlowFileTransferTest : public ExecutePythonProcessorTestBase {
const std::string output_dir = createTempDir(testController_.get());
auto executePythonProcessor = plan_->addProcessor("ExecutePythonProcessor", "executePythonProcessor");
- plan_->setProperty(executePythonProcessor, org::apache::nifi::minifi::python::processors::ExecutePythonProcessor::ScriptFile.getName(), getScriptFullPath("stateful_processor.py"));
+ plan_->setProperty(executePythonProcessor, "Script File", getScriptFullPath("stateful_processor.py"));
Review comment:
I can see that `ExecutePythonProcessor::ScriptFile` is hidden, so you need to do this, but why is it hidden? Can we unhide it?
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern, bool log_errors) {
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
+ excluding = true;
+ pattern = pattern.substr(1);
+ }
+ if (pattern.empty()) {
+ if (log_errors) logger_->log_error("Empty pattern");
+ return nullopt;
+ }
+ std::string exe_dir = get_executable_dir();
+ if (exe_dir.empty() && !isAbsolutePath(pattern.c_str())) {
+ if (log_errors) logger_->log_error("Couldn't determine executable dir, relative pattern '%s' not supported", pattern);
+ return nullopt;
+ }
+ pattern = resolve(exe_dir, pattern);
+ auto segments = split(pattern, {"/", "\\"});
+ gsl_Expects(!segments.empty());
+ auto file_pattern = segments.back();
+ if (file_pattern == "**") {
+ file_pattern = "*";
+ } else {
+ segments.pop_back();
+ }
+ if (file_pattern == "." || file_pattern == "..") {
+ if (log_errors) logger_->log_error("Invalid file pattern '%s'", file_pattern);
+ return nullopt;
+ }
+ return FilePattern(segments, file_pattern, excluding);
+}
+
+std::string FileMatcher::FilePattern::getBaseDirectory() const {
+ std::string base_dir;
+ for (const auto& segment : directory_segments_) {
+ // ignore segments at or after wildcards
+ if (isGlobPattern(segment)) {
+ break;
+ }
+ base_dir += segment + get_separator();
+ }
+ return base_dir;
+}
+
+FileMatcher::FileMatcher(const std::string &patterns) {
+ for (auto&& pattern : split(patterns, {","})) {
+ if (auto&& p = FilePattern::fromPattern(pattern)) {
+ patterns_.push_back(std::move(p.value()));
+ }
+ }
+}
+
+template<typename It>
+static bool advance_if_not_equal(It& it, const It& end) {
+ if (it == end) {
+ return false;
+ }
+ ++it;
+ return true;
+}
+
+static bool is_this_dir(const std::string& dir) {
+ return dir.empty() || dir == ".";
+}
+
+template<typename It, typename Fn>
+static void skip_if(It& it, const It& end, const Fn& fn) {
+ while (it != end && fn(*it)) {
+ ++it;
+ }
+}
+
+static bool matchGlob(std::string::const_iterator pattern_it, std::string::const_iterator pattern_end, std::string::const_iterator value_it, std::string::const_iterator value_end) {
+ // match * and ?
+ for (; pattern_it != pattern_end; ++pattern_it) {
+ if (*pattern_it == '*') {
+ do {
+ if (matchGlob(std::next(pattern_it), pattern_end, value_it, value_end)) {
+ return true;
+ }
+ } while (advance_if_not_equal(value_it, value_end));
+ return false;
+ }
+ if (value_it == value_end) {
+ return false;
+ }
+ if (*pattern_it != '?' && *pattern_it != *value_it) {
+ return false;
+ }
+ ++value_it;
+ }
+ return value_it == value_end;
+}
+
+FileMatcher::FilePattern::DirMatchResult FileMatcher::FilePattern::matchDirectory(DirIt pattern_it, DirIt pattern_end, DirIt value_it, DirIt value_end) {
+ for (; pattern_it != pattern_end; ++pattern_it) {
+ if (is_this_dir(*pattern_it)) {
+ continue;
+ }
+ if (*pattern_it == "**") {
+ if (std::next(pattern_it) == pattern_end) {
+ return DirMatchResult::TREE;
+ }
+ bool matched_parent = false;
+ // any number of nested directories
+ do {
+ skip_if(value_it, value_end, is_this_dir);
+ auto result = matchDirectory(std::next(pattern_it), pattern_end, value_it, value_end);
+ if (result == DirMatchResult::TREE || result == DirMatchResult::EXACT) {
+ return result;
+ }
+ if (result == DirMatchResult::PARENT) {
+ // even though we have a parent match, there may be a "better" (exact, tree) match
+ matched_parent = true;
+ }
+ } while (advance_if_not_equal(value_it, value_end));
+ if (matched_parent) {
+ return DirMatchResult::PARENT;
+ }
+ return DirMatchResult::NONE;
+ }
+ skip_if(value_it, value_end, is_this_dir);
+ if (value_it == value_end) {
+ // we used up all the value segments but there are still pattern segments
+ return DirMatchResult::PARENT;
+ }
+ if (!matchGlob(pattern_it->begin(), pattern_it->end(), value_it->begin(), value_it->end())) {
+ return DirMatchResult::NONE;
+ }
+ ++value_it;
+ }
+ skip_if(value_it, value_end, is_this_dir);
+ if (value_it == value_end) {
+ // used up all pattern and value segments
+ return DirMatchResult::EXACT;
+ } else {
+ // used up all pattern segments but we still have value segments
+ return DirMatchResult::NONE;
+ }
+}
+
+bool FileMatcher::FilePattern::match(const std::string& directory, const optional<std::string>& filename) const {
+ auto value = split(directory, {"/", "\\"});
+ auto result = matchDirectory(directory_segments_.begin(), directory_segments_.end(), value.begin(), value.end());
+ if (!filename) {
+ if (excluding_) {
+ if (result == DirMatchResult::TREE && file_pattern_ == "*") {
+ // all files are excluded in this directory
+ return true;
Review comment:
I can see in the unit tests that this works, but I don't understand why we return `true` here. Maybe the comment could be expanded?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678265418
##########
File path: libminifi/include/core/extension/Extension.h
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <vector>
+#include <string>
+
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class Extension;
+
+using ExtensionConfig = std::shared_ptr<org::apache::nifi::minifi::Configure>;
+using ExtensionInit = bool(*)(Extension*, const ExtensionConfig&);
+
+class ExtensionInitializer;
+
+class Extension {
+ friend class ExtensionInitializer;
+ public:
+ explicit Extension(std::string name, ExtensionInit init);
+ virtual ~Extension();
+
+ bool initialize(const ExtensionConfig& config) {
+ return init_(this, config);
+ }
+
+ const std::string& getName() const {
+ return name_;
+ }
+
+ protected:
+ virtual bool doInitialize(const ExtensionConfig& /*config*/) {
Review comment:
added some comments to explain the difference, if that does not resolve the confusion, we could rename it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] martinzink commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
martinzink commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r681693116
##########
File path: libminifi/test/unit/MemoryUsageTest.cpp
##########
@@ -16,6 +16,9 @@
* limitations under the License.
*/
+// including the extensions messes up the memory usage
+#define EXTENSION_LIST ""
Review comment:
Yeah I think disabling the extensions like this is fine, however I agree with adding a more clear comment stating that loading extensions increases the baseline memory usage.
Or we could query the memory usage before the vector is created and after it. So we test the change in memory usage and not the absolute memory usage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r696504277
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
Review comment:
We can save an allocation by passing `const char*` (`error_message.c_str()`). Exceptions need to make a copy of the string internally regardless if it's std::string or anything else, due to special requirements.
##########
File path: libminifi/include/core/extension/DynamicLibrary.h
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <map>
+#include <string>
+#include <filesystem>
+
+#include "Module.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class DynamicLibrary : public Module {
+ friend class ExtensionManager;
+
+ public:
+ DynamicLibrary(std::string name, std::filesystem::path library_path);
+ ~DynamicLibrary() override;
+
+ private:
+#ifdef WIN32
+ std::map<void*, std::string> resource_mapping_;
+
+ std::string error_str_;
+ std::string current_error_;
+
+ void store_error();
+ void* dlsym(void* handle, const char* name);
+ const char* dlerror();
+ void* dlopen(const char* file, int mode);
+ int dlclose(void* handle);
+#endif
+
+ bool load();
+ bool unload();
+
+ std::filesystem::path library_path_;
+ gsl::owner<void*> handle_ = nullptr;
+
+ static std::shared_ptr<logging::Logger> logger_;
Review comment:
If we never need to change the logger instance, then this could be `const`.
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::filesystem::path& path) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
Review comment:
Did you consider `std::string_view` or `const char*`? It's fine as is, but relying on SSO gives me a somewhat uneasy feeling.
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+ public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
+ }
+
+ public:
+ explicit FilePatternSegment(std::string pattern);
+
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ NOT_MATCHING // dir/file does not match pattern, do what you may
+ };
+
+ bool isExcluding() const {
+ return excluding_;
+ }
+
+ MatchResult match(const std::string& directory) const;
+
+ MatchResult match(const std::string& directory, const std::string& filename) const;
+
+ MatchResult match(const std::filesystem::path& path) const;
+ /**
+ * @return The lowermost parent directory without wildcards.
+ */
+ std::filesystem::path getBaseDirectory() const;
+
+ private:
+ enum class DirMatchResult {
+ NONE, // pattern does not match the directory (e.g. p = "/home/inner/*test", v = "/home/banana")
+ PARENT, // directory is a parent of the pattern (e.g. p = "/home/inner/*test", v = "/home/inner")
+ EXACT, // pattern exactly matches the directory (e.g. p = "/home/inner/*test", v = "/home/inner/cool_test")
+ TREE // pattern matches the whole subtree of the directory (e.g. p = "/home/**", v = "/home/banana")
+ };
+
+ using DirIt = std::filesystem::path::const_iterator;
+ static DirMatchResult matchDirectory(DirIt pattern_begin, DirIt pattern_end, DirIt value_begin, DirIt value_end);
+
+ std::filesystem::path directory_pattern_;
+ std::string file_pattern_;
+ bool excluding_;
+ };
+
+ using ErrorHandler = std::function<void(std::string_view /*subpattern*/, std::string_view /*error_message*/)>;
+
+ static void defaultErrorHandler(std::string_view subpattern, std::string_view error_message) {
+ std::string message = "Error in subpattern '";
+ message += subpattern;
+ message += "': ";
+ message += error_message;
+ throw FilePatternError(message);
+ }
+
+ public:
+ explicit FilePattern(const std::string& pattern, ErrorHandler error_handler = defaultErrorHandler);
+
+ private:
+ std::vector<FilePatternSegment> segments_;
+};
+
+std::set<std::filesystem::path> match(const FilePattern& pattern);
Review comment:
Can we make this return a vector? Unless we need set specifically for some reason.
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::filesystem::path& path) {
Review comment:
Moving this inside the anonymous namespace would have the same effect as `static`. I would move it, since there is one already.
##########
File path: libminifi/src/core/extension/DynamicLibrary.cpp
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <memory>
+#ifndef WIN32
+#include <dlfcn.h>
+#define DLL_EXPORT
+#else
+#define WIN32_LEAN_AND_MEAN 1
+#include <Windows.h> // Windows specific libraries for collecting software metrics.
+#include <Psapi.h>
+#pragma comment(lib, "psapi.lib" )
+#define DLL_EXPORT __declspec(dllexport)
+#define RTLD_LAZY 0
+#define RTLD_NOW 0
+
+#define RTLD_GLOBAL (1 << 1)
+#define RTLD_LOCAL (1 << 2)
+#endif
+
+#include "core/extension/DynamicLibrary.h"
+#include "core/extension/Extension.h"
+#include "utils/GeneralUtils.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+std::shared_ptr<logging::Logger> DynamicLibrary::logger_ = logging::LoggerFactory<DynamicLibrary>::getLogger();
+
+DynamicLibrary::DynamicLibrary(std::string name, std::filesystem::path library_path)
+ : Module(std::move(name)),
+ library_path_(std::move(library_path)) {
+}
+
+bool DynamicLibrary::load() {
+ dlerror();
+ handle_ = dlopen(library_path_.string().c_str(), RTLD_NOW | RTLD_LOCAL);
+ if (!handle_) {
+ logger_->log_error("Failed to load extension '%s' at '%s': %s", name_, library_path_.string(), dlerror());
+ return false;
+ } else {
+ logger_->log_trace("Loaded extension '%s' at '%s'", name_, library_path_.string());
+ return true;
+ }
+}
+
+bool DynamicLibrary::unload() {
+ logger_->log_trace("Unloading library '%s' at '%s'", name_, library_path_.string());
+ if (!handle_) {
+ logger_->log_error("Extension does not have a handle_ '%s' at '%s'", name_, library_path_.string());
+ return true;
+ }
+ dlerror();
+ if (dlclose(handle_)) {
+ logger_->log_error("Failed to unload extension '%s' at '%': %s", name_, library_path_.string(), dlerror());
+ return false;
+ }
+ logger_->log_trace("Unloaded extension '%s' at '%s'", name_, library_path_.string());
+ handle_ = nullptr;
+ return true;
+}
+
+DynamicLibrary::~DynamicLibrary() = default;
+
+#ifdef WIN32
+
+void DynamicLibrary::store_error() {
+ auto error = GetLastError();
+
+ if (error == 0) {
+ error_str_ = "";
+ return;
+ }
+
+ LPSTR messageBuffer = nullptr;
+ size_t size = FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
+ NULL, error, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPSTR)&messageBuffer, 0, NULL);
+
+ current_error_ = std::string(messageBuffer, size);
+
+ // Free the buffer.
+ LocalFree(messageBuffer);
+}
+
+void* DynamicLibrary::dlsym(void* handle, const char* name) {
+ FARPROC symbol;
+
+ symbol = GetProcAddress((HMODULE)handle, name);
+
+ if (symbol == nullptr) {
+ store_error();
+
+ for (auto hndl : resource_mapping_) {
+ symbol = GetProcAddress((HMODULE)hndl.first, name);
+ if (symbol != nullptr) {
+ break;
+ }
+ }
+ }
+
+#ifdef _MSC_VER
+#pragma warning(suppress: 4054 )
+#endif
+ return reinterpret_cast<void*>(symbol);
+}
+
+const char* DynamicLibrary::dlerror() {
+ error_str_ = current_error_;
+
+ current_error_ = "";
+
+ return error_str_.c_str();
+}
+
+void* DynamicLibrary::dlopen(const char* file, int mode) {
+ HMODULE object;
+ uint32_t uMode = SetErrorMode(SEM_FAILCRITICALERRORS);
+ if (nullptr == file) {
+ HMODULE allModules[1024];
+ HANDLE current_process_id = GetCurrentProcess();
Review comment:
This is not really an id, but a handle. I would rename to current_process.
##########
File path: libminifi/include/core/extension/DynamicLibrary.h
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <map>
+#include <string>
+#include <filesystem>
+
+#include "Module.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class DynamicLibrary : public Module {
+ friend class ExtensionManager;
+
+ public:
+ DynamicLibrary(std::string name, std::filesystem::path library_path);
+ ~DynamicLibrary() override;
+
+ private:
+#ifdef WIN32
+ std::map<void*, std::string> resource_mapping_;
+
+ std::string error_str_;
+ std::string current_error_;
+
+ void store_error();
+ void* dlsym(void* handle, const char* name);
+ const char* dlerror();
+ void* dlopen(const char* file, int mode);
+ int dlclose(void* handle);
Review comment:
Instead of strictly replicating the POSIX API, I would consider changing the signature to return a unique_ptr and maybe signal error with exceptions or similar. This could save us some state space by not needing to store errors on windows.
I'm fine with leaving it as is, this is just an idea.
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::filesystem::path& path) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ std::string filename = path.filename().string();
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ path.parent_path(),
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ auto candidates = utils::file::match(utils::file::FilePattern(pattern.value(), [&] (std::string_view subpattern, std::string_view error_msg) {
+ logger_->log_error("Error in subpattern '%s': %s", std::string{subpattern}, std::string{error_msg});
Review comment:
It might be worth to make a `std::string_view` overload of conditional_conversion in Logger.h.
##########
File path: libminifi/include/core/logging/Logger.h
##########
@@ -70,6 +75,8 @@ inline T conditional_conversion(T t) {
return t;
}
+
+
Review comment:
Not sure if we need this many empty lines. :)
##########
File path: libminifi/src/core/extension/DynamicLibrary.cpp
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <memory>
+#ifndef WIN32
+#include <dlfcn.h>
+#define DLL_EXPORT
+#else
+#define WIN32_LEAN_AND_MEAN 1
+#include <Windows.h> // Windows specific libraries for collecting software metrics.
+#include <Psapi.h>
+#pragma comment(lib, "psapi.lib" )
+#define DLL_EXPORT __declspec(dllexport)
+#define RTLD_LAZY 0
+#define RTLD_NOW 0
+
+#define RTLD_GLOBAL (1 << 1)
+#define RTLD_LOCAL (1 << 2)
+#endif
+
+#include "core/extension/DynamicLibrary.h"
+#include "core/extension/Extension.h"
+#include "utils/GeneralUtils.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+std::shared_ptr<logging::Logger> DynamicLibrary::logger_ = logging::LoggerFactory<DynamicLibrary>::getLogger();
+
+DynamicLibrary::DynamicLibrary(std::string name, std::filesystem::path library_path)
+ : Module(std::move(name)),
+ library_path_(std::move(library_path)) {
+}
+
+bool DynamicLibrary::load() {
+ dlerror();
+ handle_ = dlopen(library_path_.c_str(), RTLD_NOW | RTLD_LOCAL);
+ if (!handle_) {
+ logger_->log_error("Failed to load extension '%s' at '%s': %s", name_, library_path_, dlerror());
+ return false;
+ } else {
+ logger_->log_trace("Loaded extension '%s' at '%s'", name_, library_path_);
+ return true;
+ }
+}
+
+bool DynamicLibrary::unload() {
+ logger_->log_trace("Unloading library '%s' at '%s'", name_, library_path_);
+ if (!handle_) {
+ logger_->log_error("Extension does not have a handle_ '%s' at '%s'", name_, library_path_);
+ return true;
+ }
+ dlerror();
+ if (dlclose(handle_)) {
+ logger_->log_error("Failed to unload extension '%s' at '%': %s", name_, library_path_, dlerror());
+ return false;
+ }
+ logger_->log_trace("Unloaded extension '%s' at '%s'", name_, library_path_);
+ handle_ = nullptr;
+ return true;
+}
+
+DynamicLibrary::~DynamicLibrary() = default;
+
+#ifdef WIN32
+
+void DynamicLibrary::store_error() {
+ auto error = GetLastError();
+
+ if (error == 0) {
+ error_str_ = "";
+ return;
+ }
+
+ LPSTR messageBuffer = nullptr;
+ size_t size = FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
+ NULL, error, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPSTR)&messageBuffer, 0, NULL);
+
+ current_error_ = std::string(messageBuffer, size);
+
+ // Free the buffer.
+ LocalFree(messageBuffer);
+}
Review comment:
Check out `get_last_socket_error_message` in ClientSocket.cpp. It's an easier way to get error messages, using system_error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680764621
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
> is the http-curl-extension not an extension? do we not initialize/deinitialize the extension?
It is, but it's already modeled by the cmake target and the resulting dynamic library. This is about initializing and deinitializing, and this doesn't need a class.
> ExtensionManager cannot [...] query the name of the extension it is initializing
It always has the filename of the library it is loading, that can be used as a name. I don't think we need more than that.
I'm thinking about a model similar to GCC's constructor and destructor function attributes, but done in a portable way, since we control the loading code here. https://stackoverflow.com/a/2053078
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r699279695
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+ public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
+ }
+
+ public:
+ explicit FilePatternSegment(std::string pattern);
+
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ NOT_MATCHING // dir/file does not match pattern, do what you may
+ };
+
+ bool isExcluding() const {
+ return excluding_;
+ }
+
+ MatchResult match(const std::string& directory) const;
+
+ MatchResult match(const std::string& directory, const std::string& filename) const;
+
+ MatchResult match(const std::filesystem::path& path) const;
+ /**
+ * @return The lowermost parent directory without wildcards.
+ */
+ std::filesystem::path getBaseDirectory() const;
+
+ private:
+ enum class DirMatchResult {
+ NONE, // pattern does not match the directory (e.g. p = "/home/inner/*test", v = "/home/banana")
+ PARENT, // directory is a parent of the pattern (e.g. p = "/home/inner/*test", v = "/home/inner")
+ EXACT, // pattern exactly matches the directory (e.g. p = "/home/inner/*test", v = "/home/inner/cool_test")
+ TREE // pattern matches the whole subtree of the directory (e.g. p = "/home/**", v = "/home/banana")
+ };
+
+ using DirIt = std::filesystem::path::const_iterator;
+ static DirMatchResult matchDirectory(DirIt pattern_begin, DirIt pattern_end, DirIt value_begin, DirIt value_end);
+
+ std::filesystem::path directory_pattern_;
+ std::string file_pattern_;
+ bool excluding_;
+ };
+
+ using ErrorHandler = std::function<void(std::string_view /*subpattern*/, std::string_view /*error_message*/)>;
+
+ static void defaultErrorHandler(std::string_view subpattern, std::string_view error_message) {
+ std::string message = "Error in subpattern '";
+ message += subpattern;
+ message += "': ";
+ message += error_message;
+ throw FilePatternError(message);
+ }
+
+ public:
+ explicit FilePattern(const std::string& pattern, ErrorHandler error_handler = defaultErrorHandler);
+
+ private:
+ std::vector<FilePatternSegment> segments_;
+};
+
+std::set<std::filesystem::path> match(const FilePattern& pattern);
Review comment:
in light of the numbers I reverted back to `std::set` (in my own measurements I found that the runtime is dominated by the filesystem operations)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680710429
##########
File path: libminifi/test/azure-tests/CMakeLists.txt
##########
@@ -30,9 +30,9 @@ FOREACH(testfile ${AZURE_INTEGRATION_TESTS})
target_compile_features(${testfilename} PRIVATE cxx_std_14)
createTests("${testfilename}")
target_link_libraries(${testfilename} ${CATCH_MAIN_LIB})
- target_wholearchive_library(${testfilename} minifi-azure)
- target_wholearchive_library(${testfilename} minifi-standard-processors)
- target_wholearchive_library(${testfilename} minifi-expression-language-extensions)
+ target_link_libraries(${testfilename} minifi-azure)
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r705060665
##########
File path: libminifi/test/unit/FilePatternTests.cpp
##########
@@ -0,0 +1,253 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#define CUSTOM_EXTENSION_INIT
+
+#include <filesystem>
+
+#include "../TestBase.h"
+#include "utils/file/FilePattern.h"
+#include "range/v3/view/transform.hpp"
+#include "range/v3/view/map.hpp"
+#include "range/v3/view/join.hpp"
+#include "range/v3/view/cache1.hpp"
+#include "range/v3/view/c_str.hpp"
+#include "range/v3/range/conversion.hpp"
+#include "range/v3/range.hpp"
+
+struct FilePatternTestAccessor {
+ using FilePatternSegment = fileutils::FilePattern::FilePatternSegment;
+};
+
+using FilePatternSegment = FilePatternTestAccessor::FilePatternSegment;
+using FilePattern = fileutils::FilePattern;
+
+#define REQUIRE_INCLUDE(val) REQUIRE((val == FilePatternSegment::MatchResult::INCLUDE))
+#define REQUIRE_EXCLUDE(val) REQUIRE((val == FilePatternSegment::MatchResult::EXCLUDE))
+#define REQUIRE_NOT_MATCHING(val) REQUIRE((val == FilePatternSegment::MatchResult::NOT_MATCHING))
+
+#ifdef WIN32
+static std::filesystem::path root{"C:\\"};
+#else
+static std::filesystem::path root{"/"};
+#endif
+
+TEST_CASE("Invalid paths") {
+ REQUIRE_THROWS(FilePatternSegment(""));
+ REQUIRE_THROWS(FilePatternSegment("."));
+ REQUIRE_THROWS(FilePatternSegment(".."));
+ REQUIRE_THROWS(FilePatternSegment("!"));
+ REQUIRE_THROWS(FilePatternSegment("!."));
+ REQUIRE_THROWS(FilePatternSegment("!.."));
+ // parent accessor after wildcard
+ FilePatternSegment("./../file.txt"); // sanity check
+ REQUIRE_THROWS(FilePatternSegment("./*/../file.txt"));
+}
+
+TEST_CASE("FilePattern reports error in correct subpattern") {
+ std::vector<std::pair<std::string, std::string>> invalid_subpattern{
Review comment:
changed
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,136 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
Review comment:
added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678269961
##########
File path: libminifi/include/core/extension/Extension.h
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <vector>
+#include <string>
+
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class Extension;
+
+using ExtensionConfig = std::shared_ptr<org::apache::nifi::minifi::Configure>;
+using ExtensionInit = bool(*)(Extension*, const ExtensionConfig&);
+
+class ExtensionInitializer;
+
+class Extension {
+ friend class ExtensionInitializer;
+ public:
+ explicit Extension(std::string name, ExtensionInit init);
+ virtual ~Extension();
+
+ bool initialize(const ExtensionConfig& config) {
+ return init_(this, config);
+ }
+
+ const std::string& getName() const {
+ return name_;
+ }
+
+ protected:
+ virtual bool doInitialize(const ExtensionConfig& /*config*/) {
Review comment:
OK, I think I understand what's going on now. It is quite complicated, but I can't see a way of simplifying it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697374948
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::filesystem::path& path) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
Review comment:
reverted it back to `std::string` (the failing CI reminded my why it was `std::string`) `std::string::starts_with` is c++20 feature, and we should refactor `StringUtils` to accept `std::string_view` wherever it can in a separate PR, so there is no way around at least one `std::string` conversion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r698379783
##########
File path: libminifi/src/core/extension/DynamicLibrary.cpp
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <memory>
+#ifndef WIN32
+#include <dlfcn.h>
+#define DLL_EXPORT
+#else
+#define WIN32_LEAN_AND_MEAN 1
+#include <Windows.h> // Windows specific libraries for collecting software metrics.
+#include <Psapi.h>
+#pragma comment(lib, "psapi.lib" )
+#define DLL_EXPORT __declspec(dllexport)
+#define RTLD_LAZY 0
+#define RTLD_NOW 0
+
+#define RTLD_GLOBAL (1 << 1)
+#define RTLD_LOCAL (1 << 2)
+#endif
+
+#include "core/extension/DynamicLibrary.h"
+#include "core/extension/Extension.h"
+#include "utils/GeneralUtils.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+std::shared_ptr<logging::Logger> DynamicLibrary::logger_ = logging::LoggerFactory<DynamicLibrary>::getLogger();
+
+DynamicLibrary::DynamicLibrary(std::string name, std::filesystem::path library_path)
+ : Module(std::move(name)),
+ library_path_(std::move(library_path)) {
+}
+
+bool DynamicLibrary::load() {
+ dlerror();
+ handle_ = dlopen(library_path_.c_str(), RTLD_NOW | RTLD_LOCAL);
+ if (!handle_) {
+ logger_->log_error("Failed to load extension '%s' at '%s': %s", name_, library_path_, dlerror());
+ return false;
+ } else {
+ logger_->log_trace("Loaded extension '%s' at '%s'", name_, library_path_);
+ return true;
+ }
+}
+
+bool DynamicLibrary::unload() {
+ logger_->log_trace("Unloading library '%s' at '%s'", name_, library_path_);
+ if (!handle_) {
+ logger_->log_error("Extension does not have a handle_ '%s' at '%s'", name_, library_path_);
+ return true;
+ }
+ dlerror();
+ if (dlclose(handle_)) {
+ logger_->log_error("Failed to unload extension '%s' at '%': %s", name_, library_path_, dlerror());
+ return false;
+ }
+ logger_->log_trace("Unloaded extension '%s' at '%s'", name_, library_path_);
+ handle_ = nullptr;
+ return true;
+}
+
+DynamicLibrary::~DynamicLibrary() = default;
+
+#ifdef WIN32
+
+void DynamicLibrary::store_error() {
+ auto error = GetLastError();
+
+ if (error == 0) {
+ error_str_ = "";
+ return;
+ }
+
+ LPSTR messageBuffer = nullptr;
+ size_t size = FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
+ NULL, error, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPSTR)&messageBuffer, 0, NULL);
+
+ current_error_ = std::string(messageBuffer, size);
+
+ // Free the buffer.
+ LocalFree(messageBuffer);
+}
Review comment:
replaced
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r691886386
##########
File path: conf/minifi.properties
##########
@@ -20,6 +20,9 @@ nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=100 millis
+# Comma separated path for the extension libraries. Relative path is relative to the minifi executable.
+nifi.extension.path=../extensions/*
Review comment:
regarding the second point: you would need to move the compiled extensions to `MINIFI_HOME/extensions`, I wouldn't call that "without extra effort", but you could specify `nifi.extension.path=*` and it would use the freshly built extensions while `${MINIFI_HOME}/extensions/*` would achieve the other use-case
I don't think we can reference the `EXE_DIR` from the configuration file, so if someone decides to use the extension relative to the exe they would have to hard code the exe dir
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680772568
##########
File path: libminifi/test/unit/MemoryUsageTest.cpp
##########
@@ -16,6 +16,9 @@
* limitations under the License.
*/
+// including the extensions messes up the memory usage
+#define EXTENSION_LIST ""
Review comment:
well, this is a test file, that creates a vector with `3'000'000` chars and then queries the memory used by the process, this test fails if we allow extensions to be loaded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r674783705
##########
File path: cmake/BuildTests.cmake
##########
@@ -69,24 +69,23 @@ function(createTests testName)
if (ENABLE_BINARY_DIFF)
target_include_directories(${testName} SYSTEM BEFORE PRIVATE "${CMAKE_SOURCE_DIR}/thirdparty/bsdiff/")
- endif(ENABLE_BINARY_DIFF)
+ endif(ENABLE_BINARY_DIFF)
- if (Boost_FOUND)
- target_include_directories(${testName} BEFORE PRIVATE "${Boost_INCLUDE_DIRS}")
- endif()
- target_link_libraries(${testName} ${CMAKE_DL_LIBS} ${TEST_BASE_LIB})
- target_link_libraries(${testName} core-minifi yaml-cpp spdlog Threads::Threads)
- if (NOT excludeBase)
- target_wholearchive_library(${testName} minifi)
+ if (Boost_FOUND)
+ target_include_directories(${testName} BEFORE PRIVATE "${Boost_INCLUDE_DIRS}")
endif()
- add_dependencies(${testName} minifiexe)
+ target_link_libraries(${testName} ${CMAKE_DL_LIBS} ${TEST_BASE_LIB})
+ target_link_libraries(${testName} core-minifi yaml-cpp spdlog Threads::Threads)
+ add_dependencies(${testName} minifiexe)
Review comment:
incorrect indentation here
##########
File path: extensions/coap/controllerservice/CoapConnector.h
##########
@@ -25,6 +25,7 @@
#include "core/logging/LoggerConfiguration.h"
#include "core/controller/ControllerService.h"
+#include "core/Resource.h"
Review comment:
I think this include should move to the .cpp file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r694090751
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::string getFullPath() const {
+ return utils::file::PathUtils::concat_path(dir, filename);
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::string& dir, const std::string& filename) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ dir,
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ std::vector<LibraryDescriptor> libraries;
+ utils::file::FileMatcher(pattern.value()).forEachFile([&] (const std::string& dir, const std::string& filename) {
Review comment:
19560f3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez edited a comment on pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
lordgamez edited a comment on pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#issuecomment-919821985
Currently the C2 tests fail locally when I try to run them, have you encountered this before?
The following tests FAILED:
95 - C2JstackTest (Child aborted)
100 - C2RequestClassTest (Child aborted)
102 - C2VerifyHeartbeatAndStopSecure (Child aborted)
107 - C2VerifyHeartbeatAndStop (Child aborted)
108 - C2VerifyResourceConsumptionInHeartbeat (Child aborted)
115 - C2PauseResumeTest (Child aborted)
116 - C2LogHeartbeatTest (Child aborted)
Platform: ubuntu 20.04
CMake command: `cmake -DENABLE_AWS=ON -DENABLE_LIBRDKAFKA=ON -DENABLE_AZURE=ON -DSKIP_TESTS= -DASAN_BUILD=OFF -DUSE_SHARED_LIBS=ON -DPORTABLE=ON -DBUILD_ROCKSDB=ON -DFAIL_ON_WARNINGS=ON -DBUILD_IDENTIFIER= -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_COMPILER=gcc-11 -DCMAKE_CXX_COMPILER=g++-11 ..`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678123041
##########
File path: extensions/script/CMakeLists.txt
##########
@@ -68,13 +58,10 @@ if (ENABLE_LUA_SCRIPTING)
add_definitions(-DLUA_SUPPORT)
file(GLOB LUA_SOURCES "lua/*.cpp")
- add_library(minifi-lua-extensions STATIC ${LUA_SOURCES})
-
- target_link_libraries(minifi-lua-extensions ${LIBMINIFI})
- target_link_libraries(minifi-lua-extensions ${LUA_LIBRARIES} sol)
- target_link_libraries(minifi-script-extensions minifi-lua-extensions)
+ target_sources(minifi-script-extensions PRIVATE ${LUA_SOURCES})
- target_compile_features(minifi-lua-extensions PUBLIC cxx_std_14)
+ target_link_libraries(minifi-script-extensions ${LUA_LIBRARIES} sol)
+ target_compile_features(minifi-script-extensions PUBLIC cxx_std_14)
Review comment:
tried separating them into `script-core.a` (from `extensions/script/*.cpp`), `lua.a` and `python.a`, and merging them into a dynamic lib, but did not find the right combination of cmake commands to avoid linker errors, the separation would be artificial at best anyway IMO, since e.g. `ExecuteScript.cpp` has a number of if-defs checking for lua and python supports
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r696675232
##########
File path: libminifi/include/utils/file/FileMatcher.h
##########
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FileMatcherTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FileMatcher {
Review comment:
rewrote the FilePattern to use `std::filesystem::path`, replaced FileMatcher with a `utils::file::match(const FilePattern&)` function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680770030
##########
File path: libminifi/test/unit/MemoryUsageTest.cpp
##########
@@ -16,6 +16,9 @@
* limitations under the License.
*/
+// including the extensions messes up the memory usage
+#define EXTENSION_LIST ""
Review comment:
You mean using more memory? Or breaking the code that queries memory usage? Or what does it mean to "mess up" memory usage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r694560917
##########
File path: libminifi/include/utils/file/FileMatcher.h
##########
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FileMatcherTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FileMatcher {
Review comment:
as the logic inside `FileMatcher` is pretty "involved", moving all of them into a single function would certainly cause more confusion, also we could think of `FileMatcher` as the equivalent of `std::regex`, as the pattern itself can contain errors (and `FileMatcher` logs them), this "error-state" is better represented construction time than usage time (here we could throw from the constructor, unclear if that is something we would like to do and stop loading the extension, currently it is on a "best-effort" basis)
should we load extensions even if we encounter an invalid pattern (and log errors) or should we refuse to load extension then? (the only problem with the latter is that then we won't load any extensions, not even the curl extension by which the c2-server could correct the pattern)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r693920970
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
Review comment:
I would be fine with a requirement that extension changes on disk must happen while the extensions are not loaded in minifi. They only need to change during minifi updates and possibly through C2 or registry fetches in the future. I consider them internals, so if someone randomly changes them, they are shooting themselves in the foot anyway. Just like you don't overwrite system dlls on windows while it's running. Undefined behavior.
Treating them like this would make them conceptually "const" and fine to base invariants on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r708980062
##########
File path: cmake/BundledAwsSdkCpp.cmake
##########
@@ -110,6 +119,9 @@ function(use_bundled_libaws SOURCE_DIR BINARY_DIR)
add_dependencies(AWS::aws-c-io aws-sdk-cpp-external)
target_include_directories(AWS::aws-c-io INTERFACE ${LIBAWS_INCLUDE_DIR})
target_link_libraries(AWS::aws-c-io INTERFACE AWS::aws-c-common)
+ if (WIN32)
+ target_link_libraries(AWS::aws-c-io INTERFACE ncrypt.lib)
Review comment:
simply put: it won't build without :)
they were always a dependency, just they were included transitively from somewhere, some dependencies have been severed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680724936
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern, bool log_errors) {
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
+ excluding = true;
+ pattern = pattern.substr(1);
+ }
+ if (pattern.empty()) {
+ if (log_errors) logger_->log_error("Empty pattern");
+ return nullopt;
+ }
+ std::string exe_dir = get_executable_dir();
+ if (exe_dir.empty() && !isAbsolutePath(pattern.c_str())) {
+ if (log_errors) logger_->log_error("Couldn't determine executable dir, relative pattern '%s' not supported", pattern);
+ return nullopt;
+ }
+ pattern = resolve(exe_dir, pattern);
+ auto segments = split(pattern, {"/", "\\"});
+ gsl_Expects(!segments.empty());
+ auto file_pattern = segments.back();
+ if (file_pattern == "**") {
+ file_pattern = "*";
+ } else {
+ segments.pop_back();
+ }
+ if (file_pattern == "." || file_pattern == "..") {
+ if (log_errors) logger_->log_error("Invalid file pattern '%s'", file_pattern);
+ return nullopt;
+ }
+ return FilePattern(segments, file_pattern, excluding);
+}
+
+std::string FileMatcher::FilePattern::getBaseDirectory() const {
+ std::string base_dir;
+ for (const auto& segment : directory_segments_) {
+ // ignore segments at or after wildcards
+ if (isGlobPattern(segment)) {
+ break;
+ }
+ base_dir += segment + get_separator();
+ }
+ return base_dir;
+}
+
+FileMatcher::FileMatcher(const std::string &patterns) {
+ for (auto&& pattern : split(patterns, {","})) {
+ if (auto&& p = FilePattern::fromPattern(pattern)) {
+ patterns_.push_back(std::move(p.value()));
+ }
+ }
+}
+
+template<typename It>
+static bool advance_if_not_equal(It& it, const It& end) {
+ if (it == end) {
+ return false;
+ }
+ ++it;
+ return true;
+}
+
+static bool is_this_dir(const std::string& dir) {
+ return dir.empty() || dir == ".";
+}
+
+template<typename It, typename Fn>
+static void skip_if(It& it, const It& end, const Fn& fn) {
+ while (it != end && fn(*it)) {
+ ++it;
+ }
+}
+
+static bool matchGlob(std::string::const_iterator pattern_it, std::string::const_iterator pattern_end, std::string::const_iterator value_it, std::string::const_iterator value_end) {
+ // match * and ?
+ for (; pattern_it != pattern_end; ++pattern_it) {
+ if (*pattern_it == '*') {
+ do {
+ if (matchGlob(std::next(pattern_it), pattern_end, value_it, value_end)) {
+ return true;
+ }
+ } while (advance_if_not_equal(value_it, value_end));
+ return false;
+ }
+ if (value_it == value_end) {
+ return false;
+ }
+ if (*pattern_it != '?' && *pattern_it != *value_it) {
+ return false;
+ }
+ ++value_it;
+ }
+ return value_it == value_end;
+}
+
+FileMatcher::FilePattern::DirMatchResult FileMatcher::FilePattern::matchDirectory(DirIt pattern_it, DirIt pattern_end, DirIt value_it, DirIt value_end) {
+ for (; pattern_it != pattern_end; ++pattern_it) {
+ if (is_this_dir(*pattern_it)) {
+ continue;
+ }
+ if (*pattern_it == "**") {
+ if (std::next(pattern_it) == pattern_end) {
+ return DirMatchResult::TREE;
+ }
+ bool matched_parent = false;
+ // any number of nested directories
+ do {
+ skip_if(value_it, value_end, is_this_dir);
+ auto result = matchDirectory(std::next(pattern_it), pattern_end, value_it, value_end);
+ if (result == DirMatchResult::TREE || result == DirMatchResult::EXACT) {
+ return result;
+ }
+ if (result == DirMatchResult::PARENT) {
+ // even though we have a parent match, there may be a "better" (exact, tree) match
+ matched_parent = true;
+ }
+ } while (advance_if_not_equal(value_it, value_end));
+ if (matched_parent) {
+ return DirMatchResult::PARENT;
+ }
+ return DirMatchResult::NONE;
+ }
+ skip_if(value_it, value_end, is_this_dir);
+ if (value_it == value_end) {
+ // we used up all the value segments but there are still pattern segments
+ return DirMatchResult::PARENT;
+ }
+ if (!matchGlob(pattern_it->begin(), pattern_it->end(), value_it->begin(), value_it->end())) {
+ return DirMatchResult::NONE;
+ }
+ ++value_it;
+ }
+ skip_if(value_it, value_end, is_this_dir);
+ if (value_it == value_end) {
+ // used up all pattern and value segments
+ return DirMatchResult::EXACT;
+ } else {
+ // used up all pattern segments but we still have value segments
+ return DirMatchResult::NONE;
+ }
+}
+
+bool FileMatcher::FilePattern::match(const std::string& directory, const optional<std::string>& filename) const {
+ auto value = split(directory, {"/", "\\"});
+ auto result = matchDirectory(directory_segments_.begin(), directory_segments_.end(), value.begin(), value.end());
+ if (!filename) {
+ if (excluding_) {
+ if (result == DirMatchResult::TREE && file_pattern_ == "*") {
+ // all files are excluded in this directory
+ return true;
Review comment:
I experimented with separating the two, but to my eyes it was little more cumbersome (as we have to iterate twice) but it could make things easier to read, cherry-picked your commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678081266
##########
File path: extensions/coap/CMakeLists.txt
##########
@@ -26,7 +26,7 @@ file(GLOB CSOURCES "nanofi/*.c")
file(GLOB SOURCES "*.cpp" "protocols/*.cpp" "processors/*.cpp" "controllerservice/*.cpp" "server/*.cpp" )
add_library(nanofi-coap-c STATIC ${CSOURCES})
-add_library(minifi-coap STATIC ${SOURCES})
+add_library(minifi-coap SHARED ${SOURCES})
set_property(TARGET minifi-coap PROPERTY POSITION_INDEPENDENT_CODE ON)
Review comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678420263
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern, bool log_errors) {
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
Review comment:
I would trim whitespace from around `pattern`, and also after `!`.
##########
File path: libminifi/include/utils/file/FileMatcher.h
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FileMatcherTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FileMatcher {
+ friend struct ::FileMatcherTestAccessor;
+ class FilePattern {
+ FilePattern(std::vector<std::string> directory_segments, std::string file_pattern, bool excluding)
+ : directory_segments_(std::move(directory_segments)),
+ file_pattern_(std::move(file_pattern)),
+ excluding_(excluding) {}
+
+ public:
+ static optional<FilePattern> fromPattern(std::string pattern, bool log_errors = true);
Review comment:
I couldn't find any place where we call this with `log_errors == false`. If there isn't one, the parameter could be removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678898651
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,273 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern, bool log_errors) {
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
Review comment:
done
##########
File path: libminifi/include/utils/file/FileMatcher.h
##########
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FileMatcherTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FileMatcher {
+ friend struct ::FileMatcherTestAccessor;
+ class FilePattern {
+ FilePattern(std::vector<std::string> directory_segments, std::string file_pattern, bool excluding)
+ : directory_segments_(std::move(directory_segments)),
+ file_pattern_(std::move(file_pattern)),
+ excluding_(excluding) {}
+
+ public:
+ static optional<FilePattern> fromPattern(std::string pattern, bool log_errors = true);
Review comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r693900637
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
Review comment:
I wouldn't call a library descriptor "invalid" if it fails our (future) verification logic (a SignedExtensionLibrary would be such a class), I generally like to avoid throwing exceptions (especially from constructors) if possible (I only try to use them to cut short deep computations where checking the return value at each step would be cumbersome)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r693916061
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
Review comment:
making "verified" seem like a class invariant would be a mistake IMO, as the moment verification finishes somebody could overwrite the file on the filesystem rendering the invariant invalid, but as we do not provide any verification logic we could circle back to this question when we actually implement it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680818699
##########
File path: extensions/script/CMakeLists.txt
##########
@@ -68,13 +58,10 @@ if (ENABLE_LUA_SCRIPTING)
add_definitions(-DLUA_SUPPORT)
file(GLOB LUA_SOURCES "lua/*.cpp")
- add_library(minifi-lua-extensions STATIC ${LUA_SOURCES})
-
- target_link_libraries(minifi-lua-extensions ${LIBMINIFI})
- target_link_libraries(minifi-lua-extensions ${LUA_LIBRARIES} sol)
- target_link_libraries(minifi-script-extensions minifi-lua-extensions)
+ target_sources(minifi-script-extensions PRIVATE ${LUA_SOURCES})
- target_compile_features(minifi-lua-extensions PUBLIC cxx_std_14)
+ target_link_libraries(minifi-script-extensions ${LUA_LIBRARIES} sol)
+ target_compile_features(minifi-script-extensions PUBLIC cxx_std_14)
Review comment:
Ok, and I just realized that even linking is conditional. Nevermind then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r679707731
##########
File path: libminifi/test/script-tests/ExecutePythonProcessorTests.cpp
##########
@@ -119,7 +118,7 @@ class SimplePythonFlowFileTransferTest : public ExecutePythonProcessorTestBase {
const std::string output_dir = createTempDir(testController_.get());
auto executePythonProcessor = plan_->addProcessor("ExecutePythonProcessor", "executePythonProcessor");
- plan_->setProperty(executePythonProcessor, org::apache::nifi::minifi::python::processors::ExecutePythonProcessor::ScriptFile.getName(), getScriptFullPath("stateful_processor.py"));
+ plan_->setProperty(executePythonProcessor, "Script File", getScriptFullPath("stateful_processor.py"));
Review comment:
OK, that's fine then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680709476
##########
File path: libminifi/test/script-tests/ExecutePythonProcessorTests.cpp
##########
@@ -53,7 +53,6 @@ class ExecutePythonProcessorTestBase {
testController_.reset(new TestController());
plan_ = testController_->createPlan();
logTestController_.setDebug<TestPlan>();
- logTestController_.setDebug<minifi::python::processors::ExecutePythonProcessor>();
Review comment:
yes, we cannot reference `ExecutePythonProcessor` from outside the script-extension as it contains members with hidden class type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r693920970
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
Review comment:
I would be fine with a requirement that extension changes on disk must happen while the extensions are not loaded in minifi. They only need to change during minifi updates and possibly through C2 or registry fetches in the future. I consider them internals, so if someone randomly changes them, they are shooting themselves in the foot anyway. Just like you don't overwrite system dlls on windows while it's running.
Treating them like this would make them conceptually "const" and fine to base invariants on.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697371004
##########
File path: libminifi/src/core/extension/DynamicLibrary.cpp
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <memory>
+#ifndef WIN32
+#include <dlfcn.h>
+#define DLL_EXPORT
+#else
+#define WIN32_LEAN_AND_MEAN 1
+#include <Windows.h> // Windows specific libraries for collecting software metrics.
+#include <Psapi.h>
+#pragma comment(lib, "psapi.lib" )
+#define DLL_EXPORT __declspec(dllexport)
+#define RTLD_LAZY 0
+#define RTLD_NOW 0
+
+#define RTLD_GLOBAL (1 << 1)
+#define RTLD_LOCAL (1 << 2)
+#endif
+
+#include "core/extension/DynamicLibrary.h"
+#include "core/extension/Extension.h"
+#include "utils/GeneralUtils.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+std::shared_ptr<logging::Logger> DynamicLibrary::logger_ = logging::LoggerFactory<DynamicLibrary>::getLogger();
+
+DynamicLibrary::DynamicLibrary(std::string name, std::filesystem::path library_path)
+ : Module(std::move(name)),
+ library_path_(std::move(library_path)) {
+}
+
+bool DynamicLibrary::load() {
+ dlerror();
+ handle_ = dlopen(library_path_.c_str(), RTLD_NOW | RTLD_LOCAL);
+ if (!handle_) {
+ logger_->log_error("Failed to load extension '%s' at '%s': %s", name_, library_path_, dlerror());
+ return false;
+ } else {
+ logger_->log_trace("Loaded extension '%s' at '%s'", name_, library_path_);
+ return true;
+ }
+}
+
+bool DynamicLibrary::unload() {
+ logger_->log_trace("Unloading library '%s' at '%s'", name_, library_path_);
+ if (!handle_) {
+ logger_->log_error("Extension does not have a handle_ '%s' at '%s'", name_, library_path_);
+ return true;
+ }
+ dlerror();
+ if (dlclose(handle_)) {
+ logger_->log_error("Failed to unload extension '%s' at '%': %s", name_, library_path_, dlerror());
+ return false;
+ }
+ logger_->log_trace("Unloaded extension '%s' at '%s'", name_, library_path_);
+ handle_ = nullptr;
+ return true;
+}
+
+DynamicLibrary::~DynamicLibrary() = default;
+
+#ifdef WIN32
+
+void DynamicLibrary::store_error() {
+ auto error = GetLastError();
+
+ if (error == 0) {
+ error_str_ = "";
+ return;
+ }
+
+ LPSTR messageBuffer = nullptr;
+ size_t size = FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
+ NULL, error, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPSTR)&messageBuffer, 0, NULL);
+
+ current_error_ = std::string(messageBuffer, size);
+
+ // Free the buffer.
+ LocalFree(messageBuffer);
+}
Review comment:
there are quite a number of flags passed on to `FormatMessageA`, I'm not sure that they would result in the same error message
##########
File path: libminifi/src/core/extension/DynamicLibrary.cpp
##########
@@ -0,0 +1,201 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include <memory>
+#ifndef WIN32
+#include <dlfcn.h>
+#define DLL_EXPORT
+#else
+#define WIN32_LEAN_AND_MEAN 1
+#include <Windows.h> // Windows specific libraries for collecting software metrics.
+#include <Psapi.h>
+#pragma comment(lib, "psapi.lib" )
+#define DLL_EXPORT __declspec(dllexport)
+#define RTLD_LAZY 0
+#define RTLD_NOW 0
+
+#define RTLD_GLOBAL (1 << 1)
+#define RTLD_LOCAL (1 << 2)
+#endif
+
+#include "core/extension/DynamicLibrary.h"
+#include "core/extension/Extension.h"
+#include "utils/GeneralUtils.h"
+#include "core/logging/LoggerConfiguration.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+std::shared_ptr<logging::Logger> DynamicLibrary::logger_ = logging::LoggerFactory<DynamicLibrary>::getLogger();
+
+DynamicLibrary::DynamicLibrary(std::string name, std::filesystem::path library_path)
+ : Module(std::move(name)),
+ library_path_(std::move(library_path)) {
+}
+
+bool DynamicLibrary::load() {
+ dlerror();
+ handle_ = dlopen(library_path_.string().c_str(), RTLD_NOW | RTLD_LOCAL);
+ if (!handle_) {
+ logger_->log_error("Failed to load extension '%s' at '%s': %s", name_, library_path_.string(), dlerror());
+ return false;
+ } else {
+ logger_->log_trace("Loaded extension '%s' at '%s'", name_, library_path_.string());
+ return true;
+ }
+}
+
+bool DynamicLibrary::unload() {
+ logger_->log_trace("Unloading library '%s' at '%s'", name_, library_path_.string());
+ if (!handle_) {
+ logger_->log_error("Extension does not have a handle_ '%s' at '%s'", name_, library_path_.string());
+ return true;
+ }
+ dlerror();
+ if (dlclose(handle_)) {
+ logger_->log_error("Failed to unload extension '%s' at '%': %s", name_, library_path_.string(), dlerror());
+ return false;
+ }
+ logger_->log_trace("Unloaded extension '%s' at '%s'", name_, library_path_.string());
+ handle_ = nullptr;
+ return true;
+}
+
+DynamicLibrary::~DynamicLibrary() = default;
+
+#ifdef WIN32
+
+void DynamicLibrary::store_error() {
+ auto error = GetLastError();
+
+ if (error == 0) {
+ error_str_ = "";
+ return;
+ }
+
+ LPSTR messageBuffer = nullptr;
+ size_t size = FormatMessageA(FORMAT_MESSAGE_ALLOCATE_BUFFER | FORMAT_MESSAGE_FROM_SYSTEM | FORMAT_MESSAGE_IGNORE_INSERTS,
+ NULL, error, MAKELANGID(LANG_NEUTRAL, SUBLANG_DEFAULT), (LPSTR)&messageBuffer, 0, NULL);
+
+ current_error_ = std::string(messageBuffer, size);
+
+ // Free the buffer.
+ LocalFree(messageBuffer);
+}
+
+void* DynamicLibrary::dlsym(void* handle, const char* name) {
+ FARPROC symbol;
+
+ symbol = GetProcAddress((HMODULE)handle, name);
+
+ if (symbol == nullptr) {
+ store_error();
+
+ for (auto hndl : resource_mapping_) {
+ symbol = GetProcAddress((HMODULE)hndl.first, name);
+ if (symbol != nullptr) {
+ break;
+ }
+ }
+ }
+
+#ifdef _MSC_VER
+#pragma warning(suppress: 4054 )
+#endif
+ return reinterpret_cast<void*>(symbol);
+}
+
+const char* DynamicLibrary::dlerror() {
+ error_str_ = current_error_;
+
+ current_error_ = "";
+
+ return error_str_.c_str();
+}
+
+void* DynamicLibrary::dlopen(const char* file, int mode) {
+ HMODULE object;
+ uint32_t uMode = SetErrorMode(SEM_FAILCRITICALERRORS);
+ if (nullptr == file) {
+ HMODULE allModules[1024];
+ HANDLE current_process_id = GetCurrentProcess();
Review comment:
renamed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680918382
##########
File path: libminifi/src/utils/file/FileMatcher.cpp
##########
@@ -0,0 +1,294 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "utils/file/FileMatcher.h"
+#include "utils/file/FileUtils.h"
+#include "utils/StringUtils.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+std::shared_ptr<core::logging::Logger> FileMatcher::FilePattern::logger_ = logging::LoggerFactory<FileMatcher::FilePattern>::getLogger();
+std::shared_ptr<core::logging::Logger> FileMatcher::logger_ = logging::LoggerFactory<FileMatcher>::getLogger();
+
+static bool isGlobPattern(const std::string& pattern) {
+ return pattern.find_first_of("?*") != std::string::npos;
+}
+
+static std::vector<std::string> split(const std::string& str, const std::vector<std::string>& delimiters) {
+ std::vector<std::string> result;
+
+ size_t prev_delim_end = 0;
+ size_t next_delim_begin = std::string::npos;
+ do {
+ for (const auto& delim : delimiters) {
+ next_delim_begin = str.find(delim, prev_delim_end);
+ if (next_delim_begin != std::string::npos) {
+ result.push_back(str.substr(prev_delim_end, next_delim_begin - prev_delim_end));
+ prev_delim_end = next_delim_begin + delim.length();
+ break;
+ }
+ }
+ } while (next_delim_begin != std::string::npos);
+ result.push_back(str.substr(prev_delim_end));
+ return result;
+}
+
+#ifdef WIN32
+static const std::vector<std::string> path_separators{"/", "\\"};
+#else
+static const std::vector<std::string> path_separators{"/"};
+#endif
+
+optional<FileMatcher::FilePattern> FileMatcher::FilePattern::fromPattern(std::string pattern) {
+ pattern = utils::StringUtils::trim(pattern);
+ bool excluding = false;
+ if (!pattern.empty() && pattern[0] == '!') {
+ excluding = true;
+ pattern = utils::StringUtils::trim(pattern.substr(1));
+ }
+ if (pattern.empty()) {
+ logger_->log_error("Empty pattern");
+ return nullopt;
+ }
+ std::string exe_dir = get_executable_dir();
+ if (exe_dir.empty() && !isAbsolutePath(pattern.c_str())) {
+ logger_->log_error("Couldn't determine executable dir, relative pattern '%s' not supported", pattern);
+ return nullopt;
+ }
+ pattern = resolve(exe_dir, pattern);
+ auto segments = split(pattern, path_separators);
+ gsl_Expects(!segments.empty());
+ auto file_pattern = segments.back();
+ if (file_pattern == "**") {
+ file_pattern = "*";
+ } else {
+ segments.pop_back();
+ }
+ if (file_pattern == "." || file_pattern == "..") {
+ logger_->log_error("Invalid file pattern '%s'", file_pattern);
+ return nullopt;
+ }
+ bool after_wildcard = false;
+ for (const auto& segment : segments) {
+ if (after_wildcard && segment == "..") {
+ logger_->log_error("Parent accessor is not supported after wildcards");
+ return nullopt;
+ }
+ if (isGlobPattern(segment)) {
+ after_wildcard = true;
+ }
+ }
+ return FilePattern(segments, file_pattern, excluding);
+}
+
+std::string FileMatcher::FilePattern::getBaseDirectory() const {
+ std::string base_dir;
+ for (const auto& segment : directory_segments_) {
+ // ignore segments at or after wildcards
+ if (isGlobPattern(segment)) {
+ break;
+ }
+ base_dir += segment + get_separator();
+ }
+ return base_dir;
+}
+
+FileMatcher::FileMatcher(const std::string &patterns) {
+ for (auto&& pattern : split(patterns, {","})) {
+ if (auto&& p = FilePattern::fromPattern(pattern)) {
+ patterns_.push_back(std::move(p.value()));
+ }
+ }
+}
+
+template<typename It>
+static bool advance_if_not_equal(It& it, const It& end) {
+ if (it == end) {
+ return false;
+ }
+ ++it;
+ return true;
+}
+
+static bool is_this_dir(const std::string& dir) {
+ return dir.empty() || dir == ".";
+}
+
+template<typename It, typename Fn>
+static void skip_if(It& it, const It& end, const Fn& fn) {
+ while (it != end && fn(*it)) {
+ ++it;
+ }
+}
+
+static bool matchGlob(std::string::const_iterator pattern_it, std::string::const_iterator pattern_end, std::string::const_iterator value_it, std::string::const_iterator value_end) {
Review comment:
#1142 seems to be quite close to merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r693909270
##########
File path: libminifi/include/core/extension/Module.h
##########
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "core/logging/Logger.h"
+#include "utils/gsl.h"
+#include "properties/Configure.h"
+#include "Extension.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+/**
+ * Represents an initializable component of the agent.
+ */
+class Module {
Review comment:
a module is either the executable or some dlopen-ed dynamic library, opening a dynamic library could trigger the transitive loading of other extensions (`http-curl` -> `civet`) so there could be multiple extensions for a single dlopen
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r693866860
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::string getFullPath() const {
+ return utils::file::PathUtils::concat_path(dir, filename);
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::string& dir, const std::string& filename) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ dir,
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ std::vector<LibraryDescriptor> libraries;
+ utils::file::FileMatcher(pattern.value()).forEachFile([&] (const std::string& dir, const std::string& filename) {
Review comment:
I think passing a `path` would simplify the code. It may be worth considering to break up this loop to a higher level transform chain afterwards.
Something like `utils::file::match(*pattern) | views::transform(&asDynamicLibrary) | views::filter([x => x && x.verify]) | views::transform(&dereference) | ranges::as<std::vector>();` (pseudocode)
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::string getFullPath() const {
+ return utils::file::PathUtils::concat_path(dir, filename);
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::string& dir, const std::string& filename) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ dir,
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ std::vector<LibraryDescriptor> libraries;
+ utils::file::FileMatcher(pattern.value()).forEachFile([&] (const std::string& dir, const std::string& filename) {
+ std::optional<LibraryDescriptor> library = asDynamicLibrary(dir, filename);
+ if (library && library->verify(logger_)) {
+ libraries.push_back(std::move(library.value()));
+ }
+ return true;
+ });
+ for (const auto& library : libraries) {
+ auto module = std::make_unique<DynamicLibrary>(library.name, library.getFullPath());
+ active_module_ = module.get();
Review comment:
Can we avoid leaking this pointer? The usage is not obvious at first sight, but we should probably pass the observer pointer down the call stack if it's needed somewhere else.
##########
File path: libminifi/include/core/extension/Module.h
##########
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "core/logging/Logger.h"
+#include "utils/gsl.h"
+#include "properties/Configure.h"
+#include "Extension.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+/**
+ * Represents an initializable component of the agent.
+ */
+class Module {
Review comment:
How is module different from extension? It looks like something that can contain multiple extensions, but do we actually need to make a distinction?
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
Review comment:
Does it make sense to allow the construction of invalid library descriptors? Shouldn't this be in a throwing constructor?
##########
File path: libminifi/include/core/extension/ExtensionManager.h
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "core/logging/Logger.h"
+#include "Module.h"
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+/**
+ * Comma separated list of path patterns. Patterns prepended with "!" result in the exclusion
+ * of the extensions matching that pattern, unless some subsequent pattern re-enables it.
+ */
+static constexpr const char* nifi_extension_path = "nifi.extension.path";
+
+class ExtensionManager {
+ ExtensionManager();
+
+ public:
+ static ExtensionManager& get();
Review comment:
Would it be possible/feasible to have a proper owner for ExtensionManager? It would be nice to have a clean lifetime and avoiding globals (including singleton). The structure I have in mind is extensions declaring their properties (init, deinit, maybe name), and the extension loading code reading and using those. I think that would be the cleanest way of supporting dynamic extensions.
##########
File path: libminifi/include/utils/file/FileMatcher.h
##########
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FileMatcherTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FileMatcher {
Review comment:
Could this be a function? Its name sounds like an operation, like a `utils::file::match` function, that returns a list (vector) of paths.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r698372311
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FilePattern.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::filesystem::path dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::filesystem::path getFullPath() const {
+ return dir / filename;
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::filesystem::path& path) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ std::string filename = path.filename().string();
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ path.parent_path(),
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ auto candidates = utils::file::match(utils::file::FilePattern(pattern.value(), [&] (std::string_view subpattern, std::string_view error_msg) {
+ logger_->log_error("Error in subpattern '%s': %s", std::string{subpattern}, std::string{error_msg});
Review comment:
True, I didn't think of this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680902181
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
Review comment:
changed it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680703902
##########
File path: conf/minifi.properties
##########
@@ -20,6 +20,9 @@ nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=100 millis
+# Comma separated path for the extension libraries. Relative path is relative to the minifi executable.
+nifi.extension.path=../extensions/*
Review comment:
the extensions are still essentially part of the agent, the MINIFI_HOME, as I understand, could be moved to anywhere on the system, I would like them (extensions) not to have to be moved with, just as we don't have to move the executable
(note: we can still use the ${MINIFI_HOME} variable in the path if need be just as with any other property)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r698373626
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+ public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
+ }
+
+ public:
+ explicit FilePatternSegment(std::string pattern);
+
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ NOT_MATCHING // dir/file does not match pattern, do what you may
+ };
+
+ bool isExcluding() const {
+ return excluding_;
+ }
+
+ MatchResult match(const std::string& directory) const;
+
+ MatchResult match(const std::string& directory, const std::string& filename) const;
+
+ MatchResult match(const std::filesystem::path& path) const;
+ /**
+ * @return The lowermost parent directory without wildcards.
+ */
+ std::filesystem::path getBaseDirectory() const;
+
+ private:
+ enum class DirMatchResult {
+ NONE, // pattern does not match the directory (e.g. p = "/home/inner/*test", v = "/home/banana")
+ PARENT, // directory is a parent of the pattern (e.g. p = "/home/inner/*test", v = "/home/inner")
+ EXACT, // pattern exactly matches the directory (e.g. p = "/home/inner/*test", v = "/home/inner/cool_test")
+ TREE // pattern matches the whole subtree of the directory (e.g. p = "/home/**", v = "/home/banana")
+ };
+
+ using DirIt = std::filesystem::path::const_iterator;
+ static DirMatchResult matchDirectory(DirIt pattern_begin, DirIt pattern_end, DirIt value_begin, DirIt value_end);
+
+ std::filesystem::path directory_pattern_;
+ std::string file_pattern_;
+ bool excluding_;
+ };
+
+ using ErrorHandler = std::function<void(std::string_view /*subpattern*/, std::string_view /*error_message*/)>;
+
+ static void defaultErrorHandler(std::string_view subpattern, std::string_view error_message) {
+ std::string message = "Error in subpattern '";
+ message += subpattern;
+ message += "': ";
+ message += error_message;
+ throw FilePatternError(message);
+ }
+
+ public:
+ explicit FilePattern(const std::string& pattern, ErrorHandler error_handler = defaultErrorHandler);
+
+ private:
+ std::vector<FilePatternSegment> segments_;
+};
+
+std::set<std::filesystem::path> match(const FilePattern& pattern);
Review comment:
We could still use std::sort and std::unique to guarantee uniqueness of a vector with the same complexity, but with better cache-friendlyness and less allocations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680714656
##########
File path: libminifi/test/unit/MemoryUsageTest.cpp
##########
@@ -16,6 +16,9 @@
* limitations under the License.
*/
+// including the extensions messes up the memory usage
+#define EXTENSION_LIST ""
Review comment:
exactly what the comment says, by default all tests load all built extensions unless we specify the EXTENSION_LIST macro by hand, and apparently loading all extension messed up the memory usage
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678304459
##########
File path: libminifi/include/core/ClassLoader.h
##########
@@ -56,182 +47,6 @@ namespace core {
#define RTLD_LOCAL (1 << 2)
#endif
-/**
- * Class used to provide a global initialization and deinitialization function for an ObjectFactory.
- * Calls to instances of all ObjectFactoryInitializers are done under a unique lock.
- */
-class ObjectFactoryInitializer {
- public:
- virtual ~ObjectFactoryInitializer() = default;
-
- /**
- * This function is be called before the ObjectFactory is used.
- * @return whether the initialization was successful. If false, deinitialize will NOT be called.
- */
- virtual bool initialize() = 0;
-
- /**
- * This function will be called after the ObjectFactory is not needed anymore.
- */
- virtual void deinitialize() = 0;
-};
-
-/**
- * Factory that is used as an interface for
- * creating processors from shared objects.
- */
-class ObjectFactory {
- public:
- ObjectFactory(const std::string &group) // NOLINT
- : group_(group) {
- }
-
- ObjectFactory() = default;
-
- /**
- * Virtual destructor.
- */
- virtual ~ObjectFactory() = default;
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual std::shared_ptr<CoreComponent> create(const std::string& /*name*/) {
- return nullptr;
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual CoreComponent *createRaw(const std::string& /*name*/) {
- return nullptr;
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual std::shared_ptr<CoreComponent> create(const std::string& /*name*/, const utils::Identifier& /*uuid*/) {
- return nullptr;
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual CoreComponent* createRaw(const std::string& /*name*/, const utils::Identifier& /*uuid*/) {
- return nullptr;
- }
-
- /**
- * Returns an initializer for the factory.
- */
- virtual std::unique_ptr<ObjectFactoryInitializer> getInitializer() {
- return nullptr;
- }
-
- /**
- * Gets the name of the object.
- * @return class name of processor
- */
- virtual std::string getName() = 0;
-
- virtual std::string getGroupName() const {
- return group_;
- }
-
- virtual std::string getClassName() = 0;
- /**
- * Gets the class name for the object
- * @return class name for the processor.
- */
- virtual std::vector<std::string> getClassNames() = 0;
-
- virtual std::unique_ptr<ObjectFactory> assign(const std::string &class_name) = 0;
-
- private:
- std::string group_;
-};
-/**
- * Factory that is used as an interface for
- * creating processors from shared objects.
- */
-template<class T>
-class DefautObjectFactory : public ObjectFactory {
- public:
- DefautObjectFactory() {
- className = core::getClassName<T>();
- }
-
- DefautObjectFactory(const std::string &group_name) // NOLINT
- : ObjectFactory(group_name) {
- className = core::getClassName<T>();
- }
- /**
- * Virtual destructor.
- */
- virtual ~DefautObjectFactory() = default;
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual std::shared_ptr<CoreComponent> create(const std::string &name) {
- std::shared_ptr<T> ptr = std::make_shared<T>(name);
- return std::static_pointer_cast<CoreComponent>(ptr);
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual std::shared_ptr<CoreComponent> create(const std::string &name, const utils::Identifier &uuid) {
- std::shared_ptr<T> ptr = std::make_shared<T>(name, uuid);
- return std::static_pointer_cast<CoreComponent>(ptr);
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual CoreComponent* createRaw(const std::string &name) {
- T *ptr = new T(name);
- return dynamic_cast<CoreComponent*>(ptr);
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual CoreComponent* createRaw(const std::string &name, const utils::Identifier &uuid) {
- T *ptr = new T(name, uuid);
- return dynamic_cast<CoreComponent*>(ptr);
- }
-
- /**
- * Gets the name of the object.
- * @return class name of processor
- */
- virtual std::string getName() {
- return className;
- }
-
- /**
- * Gets the class name for the object
- * @return class name for the processor.
- */
- virtual std::string getClassName() {
- return className;
- }
-
- virtual std::vector<std::string> getClassNames() {
- std::vector<std::string> container;
- container.push_back(className);
- return container;
- }
-
- virtual std::unique_ptr<ObjectFactory> assign(const std::string& /*class_name*/) {
- return nullptr;
- }
-
- protected:
- std::string className;
-};
-
/**
* Function that is used to create the
* processor factory from the shared object.
Review comment:
the `createFactory` typedef is no longer used, it can be removed
##########
File path: libminifi/include/core/ObjectFactory.h
##########
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <memory>
+#include "Core.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+
+/**
+ * Factory that is used as an interface for
+ * creating processors from shared objects.
+ */
+class ObjectFactory {
+ public:
+ ObjectFactory(const std::string &group) // NOLINT
Review comment:
old code, but `explicit` would be better than `NOLINT`
##########
File path: libminifi/src/agent/agent_docs.cpp
##########
@@ -0,0 +1,32 @@
+/*** Licensed to the Apache Software Foundation (ASF) under one or more
Review comment:
very minor, but there should be a newline after the second asterisk
##########
File path: libminifi/src/FlowController.cpp
##########
@@ -60,7 +60,7 @@ namespace minifi {
FlowController::FlowController(std::shared_ptr<core::Repository> provenance_repo, std::shared_ptr<core::Repository> flow_file_repo,
std::shared_ptr<Configure> configure, std::unique_ptr<core::FlowConfiguration> flow_configuration,
- std::shared_ptr<core::ContentRepository> content_repo, const std::string /*name*/, bool headless_mode,
Review comment:
it looks like `headless_mode` is no longer used, so this parameter can be removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680720629
##########
File path: libminifi/include/utils/Export.h
##########
@@ -15,14 +15,21 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include "TemplateLoader.h"
-#include "core/FlowConfiguration.h"
-bool TemplateFactory::added = core::FlowConfiguration::add_static_func("createTemplateFactory");
+#pragma once
-extern "C" {
-
-void *createTemplateFactory(void) {
- return new TemplateFactory();
-}
-}
+#ifdef WIN32
+ #ifdef LIBMINIFI
+ #define MINIFIAPI __declspec(dllexport)
+ #else
+ #define MINIFIAPI __declspec(dllimport)
+ #endif
+ #ifdef MODULE_NAME
+ #define EXTENSIONAPI __declspec(dllexport)
+ #else
+ #define EXTENSIONAPI __declspec(dllimport)
+ #endif
+#else
+ #define MINIFIAPI
+ #define EXTENSIONAPI
+#endif
Review comment:
agree that sharing export macros is suboptimal, possibly in a separate PR, now WINDOWS_EXPORT_ALL_SYMBOLS does most of the heavy lifting, moving to manually annotating all classes and static members (mostly for the sake of tests) is a significant effort
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r683309453
##########
File path: libminifi/test/unit/MemoryUsageTest.cpp
##########
@@ -16,6 +16,9 @@
* limitations under the License.
*/
+// including the extensions messes up the memory usage
+#define EXTENSION_LIST ""
Review comment:
updated comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r694090751
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::string getFullPath() const {
+ return utils::file::PathUtils::concat_path(dir, filename);
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::string& dir, const std::string& filename) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ dir,
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ std::vector<LibraryDescriptor> libraries;
+ utils::file::FileMatcher(pattern.value()).forEachFile([&] (const std::string& dir, const std::string& filename) {
Review comment:
~19560f3~ 02bab30
`utils::file::match` is just a quick and dirty implementation there. And one of the tricky things was realizing that an rvalue range does not satisfy viewable_range and therefore cannot be "piped" to view niebloids.
edit: replaced with 02bab30 because I messed up the first dereference version and because most of our tested compilers lack concepts support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r691878972
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
regarding the config access in static init, some/most tests link to shared libraries startup time when the config is not yet available
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r697341006
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+ public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
+ }
+
+ public:
+ explicit FilePatternSegment(std::string pattern);
+
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ NOT_MATCHING // dir/file does not match pattern, do what you may
+ };
+
+ bool isExcluding() const {
+ return excluding_;
+ }
+
+ MatchResult match(const std::string& directory) const;
+
+ MatchResult match(const std::string& directory, const std::string& filename) const;
+
+ MatchResult match(const std::filesystem::path& path) const;
+ /**
+ * @return The lowermost parent directory without wildcards.
+ */
+ std::filesystem::path getBaseDirectory() const;
+
+ private:
+ enum class DirMatchResult {
+ NONE, // pattern does not match the directory (e.g. p = "/home/inner/*test", v = "/home/banana")
+ PARENT, // directory is a parent of the pattern (e.g. p = "/home/inner/*test", v = "/home/inner")
+ EXACT, // pattern exactly matches the directory (e.g. p = "/home/inner/*test", v = "/home/inner/cool_test")
+ TREE // pattern matches the whole subtree of the directory (e.g. p = "/home/**", v = "/home/banana")
+ };
+
+ using DirIt = std::filesystem::path::const_iterator;
+ static DirMatchResult matchDirectory(DirIt pattern_begin, DirIt pattern_end, DirIt value_begin, DirIt value_end);
+
+ std::filesystem::path directory_pattern_;
+ std::string file_pattern_;
+ bool excluding_;
+ };
+
+ using ErrorHandler = std::function<void(std::string_view /*subpattern*/, std::string_view /*error_message*/)>;
+
+ static void defaultErrorHandler(std::string_view subpattern, std::string_view error_message) {
+ std::string message = "Error in subpattern '";
+ message += subpattern;
+ message += "': ";
+ message += error_message;
+ throw FilePatternError(message);
+ }
+
+ public:
+ explicit FilePattern(const std::string& pattern, ErrorHandler error_handler = defaultErrorHandler);
+
+ private:
+ std::vector<FilePatternSegment> segments_;
+};
+
+std::set<std::filesystem::path> match(const FilePattern& pattern);
Review comment:
I think it is better to guarantee some level of uniqueness for the return paths (I wrote 'some level' as of course two different paths could still refer to the same underlying file)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r705041686
##########
File path: libminifi/include/core/extension/DynamicLibrary.h
##########
@@ -0,0 +1,69 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <map>
+#include <string>
+#include <filesystem>
+
+#include "Module.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class DynamicLibrary : public Module {
+ friend class ExtensionManager;
+
+ public:
+ DynamicLibrary(std::string name, std::filesystem::path library_path);
+ ~DynamicLibrary() override;
+
+ private:
+#ifdef WIN32
+ std::map<void*, std::string> resource_mapping_;
+
+ std::string error_str_;
+ std::string current_error_;
+
+ void store_error();
+ void* dlsym(void* handle, const char* name);
+ const char* dlerror();
+ void* dlopen(const char* file, int mode);
+ int dlclose(void* handle);
+#endif
+
+ bool load();
+ bool unload();
+
+ std::filesystem::path library_path_;
+ gsl::owner<void*> handle_ = nullptr;
+
+ static std::shared_ptr<logging::Logger> logger_;
Review comment:
it seems it was never fixed, I made the ExtensionManager::logger_ const :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678112156
##########
File path: cmake/Extensions.cmake
##########
@@ -26,7 +26,27 @@ macro(register_extension extension-name)
get_property(extensions GLOBAL PROPERTY EXTENSION-OPTIONS)
set_property(GLOBAL APPEND PROPERTY EXTENSION-OPTIONS ${extension-name})
target_compile_definitions(${extension-name} PRIVATE "MODULE_NAME=${extension-name}")
- set_target_properties(${extension-name} PROPERTIES ENABLE_EXPORTS True)
+ set_target_properties(${extension-name} PROPERTIES
+ ENABLE_EXPORTS True
+ POSITION_INDEPENDENT_CODE ON)
+ if (WIN32)
+ set_target_properties(${extension-name} PROPERTIES
+ RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ WINDOWS_EXPORT_ALL_SYMBOLS TRUE)
+ else()
+ set_target_properties(${extension-name} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin")
+ endif()
+
+ if (WIN32)
+ install(TARGETS ${extension-name} RUNTIME DESTINATION extensions COMPONENT bin)
+ else()
+ if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
+ target_link_options(${extension-name} PRIVATE "-Wl,--disable-new-dtags")
Review comment:
without this, GCC sets the `RUNPATH` instead of the `RPATH`, there are differences in precedence and inheritance, not entirely clear what the root cause is, but without this test executables fail to start
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] fgerlits commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
fgerlits commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r677581930
##########
File path: extensions/coap/CMakeLists.txt
##########
@@ -26,7 +26,7 @@ file(GLOB CSOURCES "nanofi/*.c")
file(GLOB SOURCES "*.cpp" "protocols/*.cpp" "processors/*.cpp" "controllerservice/*.cpp" "server/*.cpp" )
add_library(nanofi-coap-c STATIC ${CSOURCES})
-add_library(minifi-coap STATIC ${SOURCES})
+add_library(minifi-coap SHARED ${SOURCES})
set_property(TARGET minifi-coap PROPERTY POSITION_INDEPENDENT_CODE ON)
Review comment:
I guess it doesn't matter, as the shared lib will be position-independent anyway, but you deleted this `set_property(... POSITION_INDEPENDENT_CODE ON)` line from the other extensions, so I would delete it from here, too
##########
File path: cmake/Extensions.cmake
##########
@@ -26,7 +26,27 @@ macro(register_extension extension-name)
get_property(extensions GLOBAL PROPERTY EXTENSION-OPTIONS)
set_property(GLOBAL APPEND PROPERTY EXTENSION-OPTIONS ${extension-name})
target_compile_definitions(${extension-name} PRIVATE "MODULE_NAME=${extension-name}")
- set_target_properties(${extension-name} PROPERTIES ENABLE_EXPORTS True)
+ set_target_properties(${extension-name} PROPERTIES
+ ENABLE_EXPORTS True
+ POSITION_INDEPENDENT_CODE ON)
+ if (WIN32)
+ set_target_properties(${extension-name} PROPERTIES
+ RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ WINDOWS_EXPORT_ALL_SYMBOLS TRUE)
+ else()
+ set_target_properties(${extension-name} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin")
+ endif()
+
+ if (WIN32)
+ install(TARGETS ${extension-name} RUNTIME DESTINATION extensions COMPONENT bin)
+ else()
+ if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
+ target_link_options(${extension-name} PRIVATE "-Wl,--disable-new-dtags")
+ endif()
+ set_target_properties(${extension-name} PROPERTIES INSTALL_RPATH "$ORIGIN")
Review comment:
does this work? no escaping of the `$` is needed?
##########
File path: cmake/Extensions.cmake
##########
@@ -26,7 +26,27 @@ macro(register_extension extension-name)
get_property(extensions GLOBAL PROPERTY EXTENSION-OPTIONS)
set_property(GLOBAL APPEND PROPERTY EXTENSION-OPTIONS ${extension-name})
target_compile_definitions(${extension-name} PRIVATE "MODULE_NAME=${extension-name}")
- set_target_properties(${extension-name} PROPERTIES ENABLE_EXPORTS True)
+ set_target_properties(${extension-name} PROPERTIES
+ ENABLE_EXPORTS True
+ POSITION_INDEPENDENT_CODE ON)
+ if (WIN32)
+ set_target_properties(${extension-name} PROPERTIES
+ RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
+ WINDOWS_EXPORT_ALL_SYMBOLS TRUE)
+ else()
+ set_target_properties(${extension-name} PROPERTIES LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin")
+ endif()
+
+ if (WIN32)
+ install(TARGETS ${extension-name} RUNTIME DESTINATION extensions COMPONENT bin)
+ else()
+ if ("${CMAKE_CXX_COMPILER_ID}" STREQUAL "GNU")
+ target_link_options(${extension-name} PRIVATE "-Wl,--disable-new-dtags")
Review comment:
why do we need `--disable-new-dtags`?
##########
File path: extensions/script/CMakeLists.txt
##########
@@ -68,13 +58,10 @@ if (ENABLE_LUA_SCRIPTING)
add_definitions(-DLUA_SUPPORT)
file(GLOB LUA_SOURCES "lua/*.cpp")
- add_library(minifi-lua-extensions STATIC ${LUA_SOURCES})
-
- target_link_libraries(minifi-lua-extensions ${LIBMINIFI})
- target_link_libraries(minifi-lua-extensions ${LUA_LIBRARIES} sol)
- target_link_libraries(minifi-script-extensions minifi-lua-extensions)
+ target_sources(minifi-script-extensions PRIVATE ${LUA_SOURCES})
- target_compile_features(minifi-lua-extensions PUBLIC cxx_std_14)
+ target_link_libraries(minifi-script-extensions ${LUA_LIBRARIES} sol)
+ target_compile_features(minifi-script-extensions PUBLIC cxx_std_14)
Review comment:
These used to be in a separate `minifi-lua-extensions` library, now they are part of `minifi-script-extensions`. Why?
##########
File path: libminifi/include/core/extension/Extension.h
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <vector>
+#include <string>
+
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class Extension;
+
+using ExtensionConfig = std::shared_ptr<org::apache::nifi::minifi::Configure>;
+using ExtensionInit = bool(*)(Extension*, const ExtensionConfig&);
+
+class ExtensionInitializer;
+
+class Extension {
+ friend class ExtensionInitializer;
+ public:
+ explicit Extension(std::string name, ExtensionInit init);
+ virtual ~Extension();
+
+ bool initialize(const ExtensionConfig& config) {
+ return init_(this, config);
+ }
+
+ const std::string& getName() const {
+ return name_;
+ }
+
+ protected:
Review comment:
It's a bit weird to make these two protected, and then undo this with the `friend` declaration. Why not just make them public?
##########
File path: extensions/opencv/MotionDetector.cpp
##########
@@ -201,6 +201,8 @@ void MotionDetector::onTrigger(const std::shared_ptr<core::ProcessContext> &cont
void MotionDetector::notifyStop() {
}
+REGISTER_RESOURCE(MotionDetector, "Detect motion from captured images."); // NOLINT
Review comment:
why is the `// NOLINT` needed?
##########
File path: libminifi/include/core/extension/Extension.h
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <vector>
+#include <string>
+
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class Extension;
+
+using ExtensionConfig = std::shared_ptr<org::apache::nifi::minifi::Configure>;
+using ExtensionInit = bool(*)(Extension*, const ExtensionConfig&);
+
+class ExtensionInitializer;
+
+class Extension {
+ friend class ExtensionInitializer;
+ public:
+ explicit Extension(std::string name, ExtensionInit init);
+ virtual ~Extension();
+
+ bool initialize(const ExtensionConfig& config) {
+ return init_(this, config);
+ }
+
+ const std::string& getName() const {
+ return name_;
+ }
+
+ protected:
+ virtual bool doInitialize(const ExtensionConfig& /*config*/) {
Review comment:
I find `initialize` / `doInitialize` confusing, as it's not clear from their names what the difference is. I don't have a great suggestion for better names; maybe keep `initialize` and rename `doInitialize` and `doDeinitialize` to `initializeClass` and `deinitializeClass`?
##########
File path: extensions/libarchive/ManipulateArchive.h
##########
@@ -27,6 +27,9 @@
#include "core/Processor.h"
#include "core/ProcessSession.h"
+#include "FocusArchiveEntry.h"
+#include "UnfocusArchiveEntry.h"
Review comment:
could these two includes be in the `cpp` file?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r699111472
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+ public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
+ }
+
+ public:
+ explicit FilePatternSegment(std::string pattern);
+
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ NOT_MATCHING // dir/file does not match pattern, do what you may
+ };
+
+ bool isExcluding() const {
+ return excluding_;
+ }
+
+ MatchResult match(const std::string& directory) const;
+
+ MatchResult match(const std::string& directory, const std::string& filename) const;
+
+ MatchResult match(const std::filesystem::path& path) const;
+ /**
+ * @return The lowermost parent directory without wildcards.
+ */
+ std::filesystem::path getBaseDirectory() const;
+
+ private:
+ enum class DirMatchResult {
+ NONE, // pattern does not match the directory (e.g. p = "/home/inner/*test", v = "/home/banana")
+ PARENT, // directory is a parent of the pattern (e.g. p = "/home/inner/*test", v = "/home/inner")
+ EXACT, // pattern exactly matches the directory (e.g. p = "/home/inner/*test", v = "/home/inner/cool_test")
+ TREE // pattern matches the whole subtree of the directory (e.g. p = "/home/**", v = "/home/banana")
+ };
+
+ using DirIt = std::filesystem::path::const_iterator;
+ static DirMatchResult matchDirectory(DirIt pattern_begin, DirIt pattern_end, DirIt value_begin, DirIt value_end);
+
+ std::filesystem::path directory_pattern_;
+ std::string file_pattern_;
+ bool excluding_;
+ };
+
+ using ErrorHandler = std::function<void(std::string_view /*subpattern*/, std::string_view /*error_message*/)>;
+
+ static void defaultErrorHandler(std::string_view subpattern, std::string_view error_message) {
+ std::string message = "Error in subpattern '";
+ message += subpattern;
+ message += "': ";
+ message += error_message;
+ throw FilePatternError(message);
+ }
+
+ public:
+ explicit FilePattern(const std::string& pattern, ErrorHandler error_handler = defaultErrorHandler);
+
+ private:
+ std::vector<FilePatternSegment> segments_;
+};
+
+std::set<std::filesystem::path> match(const FilePattern& pattern);
Review comment:
I just prefer vector as the default unless we have a compelling reason not to use it. If you still feel that set is a better fit, then feel free to revert this change.
I also did some measurements and they didn't show a significant difference in performance:
```
--------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------
BM_MatchVector/4 150744 ns 148394 ns 4553
BM_MatchVector/8 534604 ns 526563 ns 1330
BM_MatchVector/64 31438122 ns 30888763 ns 23
BM_MatchVector/512 2112878189 ns 2075348002 ns 1
BM_MatchVector/2048 3325256795 ns 3258781806 ns 1
BM_MatchSet/4 151805 ns 149650 ns 4707
BM_MatchSet/8 534444 ns 527351 ns 1328
BM_MatchSet/64 31374911 ns 30869050 ns 23
BM_MatchSet/512 2110077546 ns 2075058340 ns 1
BM_MatchSet/2048 3297083600 ns 3242293291 ns 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r708980201
##########
File path: extensions/azure/CMakeLists.txt
##########
@@ -34,6 +34,10 @@ target_link_libraries(minifi-azure ${LIBMINIFI} Threads::Threads)
target_link_libraries(minifi-azure CURL::libcurl LibXml2::LibXml2)
target_link_libraries(minifi-azure AZURE::azure-storage-blobs AZURE::azure-storage-common AZURE::azure-core)
+if (WIN32)
+ target_link_libraries(minifi-azure crypt32.lib bcrypt.lib)
Review comment:
same as: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r708980062
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r699111472
##########
File path: libminifi/include/utils/file/FilePattern.h
##########
@@ -0,0 +1,122 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+#include <filesystem>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FilePatternTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FilePatternError : public std::invalid_argument {
+ public:
+ explicit FilePatternError(const std::string& msg) : invalid_argument(msg) {}
+};
+
+class FilePattern {
+ friend struct ::FilePatternTestAccessor;
+
+ friend std::set<std::filesystem::path> match(const FilePattern& pattern);
+
+ class FilePatternSegmentError : public std::invalid_argument {
+ public:
+ explicit FilePatternSegmentError(const std::string& msg) : invalid_argument(msg) {}
+ };
+
+ class FilePatternSegment {
+ static void defaultSegmentErrorHandler(std::string_view error_message) {
+ throw FilePatternError(std::string{error_message});
+ }
+
+ public:
+ explicit FilePatternSegment(std::string pattern);
+
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ NOT_MATCHING // dir/file does not match pattern, do what you may
+ };
+
+ bool isExcluding() const {
+ return excluding_;
+ }
+
+ MatchResult match(const std::string& directory) const;
+
+ MatchResult match(const std::string& directory, const std::string& filename) const;
+
+ MatchResult match(const std::filesystem::path& path) const;
+ /**
+ * @return The lowermost parent directory without wildcards.
+ */
+ std::filesystem::path getBaseDirectory() const;
+
+ private:
+ enum class DirMatchResult {
+ NONE, // pattern does not match the directory (e.g. p = "/home/inner/*test", v = "/home/banana")
+ PARENT, // directory is a parent of the pattern (e.g. p = "/home/inner/*test", v = "/home/inner")
+ EXACT, // pattern exactly matches the directory (e.g. p = "/home/inner/*test", v = "/home/inner/cool_test")
+ TREE // pattern matches the whole subtree of the directory (e.g. p = "/home/**", v = "/home/banana")
+ };
+
+ using DirIt = std::filesystem::path::const_iterator;
+ static DirMatchResult matchDirectory(DirIt pattern_begin, DirIt pattern_end, DirIt value_begin, DirIt value_end);
+
+ std::filesystem::path directory_pattern_;
+ std::string file_pattern_;
+ bool excluding_;
+ };
+
+ using ErrorHandler = std::function<void(std::string_view /*subpattern*/, std::string_view /*error_message*/)>;
+
+ static void defaultErrorHandler(std::string_view subpattern, std::string_view error_message) {
+ std::string message = "Error in subpattern '";
+ message += subpattern;
+ message += "': ";
+ message += error_message;
+ throw FilePatternError(message);
+ }
+
+ public:
+ explicit FilePattern(const std::string& pattern, ErrorHandler error_handler = defaultErrorHandler);
+
+ private:
+ std::vector<FilePatternSegment> segments_;
+};
+
+std::set<std::filesystem::path> match(const FilePattern& pattern);
Review comment:
I just prefer vector as the default unless we have a compelling reason not to use it. If you still feel that set is a better fit, then feel free to revert this change.
I also did some measurements and they didn't show a significant difference in performance:
```
--------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------
BM_MatchVector/4 150744 ns 148394 ns 4553
BM_MatchVector/8 534604 ns 526563 ns 1330
BM_MatchVector/64 31438122 ns 30888763 ns 23
BM_MatchVector/512 2112878189 ns 2075348002 ns 1
BM_MatchVector/2048 3325256795 ns 3258781806 ns 1
BM_MatchSet/4 151805 ns 149650 ns 4707
BM_MatchSet/8 534444 ns 527351 ns 1328
BM_MatchSet/64 31374911 ns 30869050 ns 23
BM_MatchSet/512 2110077546 ns 2075058340 ns 1
BM_MatchSet/2048 3297083600 ns 3242293291 ns 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] lordgamez commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
lordgamez commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r708935767
##########
File path: extensions/azure/CMakeLists.txt
##########
@@ -34,6 +34,10 @@ target_link_libraries(minifi-azure ${LIBMINIFI} Threads::Threads)
target_link_libraries(minifi-azure CURL::libcurl LibXml2::LibXml2)
target_link_libraries(minifi-azure AZURE::azure-storage-blobs AZURE::azure-storage-common AZURE::azure-core)
+if (WIN32)
+ target_link_libraries(minifi-azure crypt32.lib bcrypt.lib)
Review comment:
The same question here, why do we need these additional libs now?
##########
File path: extensions/jni/JNILoader.cpp
##########
@@ -15,16 +15,24 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-#include "JNILoader.h"
-#include "core/FlowConfiguration.h"
+#include "core/extension/Extension.h"
+#include "JVMCreator.h"
-bool JNIFactory::added = core::FlowConfiguration::add_static_func("createJNIFactory");
+namespace minifi = org::apache::nifi::minifi;
-extern "C" {
+static minifi::jni::JVMCreator& getJVMCreator() {
+ static minifi::jni::JVMCreator instance("JVMCreator");
+ return instance;
+}
-void *createJNIFactory(void) {
- return new JNIFactory();
+static bool init(const std::shared_ptr<org::apache::nifi::minifi::Configure>& config) {
+ getJVMCreator().configure(config);
+ return true;
}
+static void deinit() {
+ // TODO(adebreceni)
Review comment:
Please specify if there is anything to be done here
##########
File path: cmake/BundledAwsSdkCpp.cmake
##########
@@ -110,6 +119,9 @@ function(use_bundled_libaws SOURCE_DIR BINARY_DIR)
add_dependencies(AWS::aws-c-io aws-sdk-cpp-external)
target_include_directories(AWS::aws-c-io INTERFACE ${LIBAWS_INCLUDE_DIR})
target_link_libraries(AWS::aws-c-io INTERFACE AWS::aws-c-common)
+ if (WIN32)
+ target_link_libraries(AWS::aws-c-io INTERFACE ncrypt.lib)
Review comment:
Just wondering, what changed that windows now requires ncrypt.lib to be linked?
##########
File path: libminifi/test/unit/FilePatternTests.cpp
##########
@@ -0,0 +1,253 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#define CUSTOM_EXTENSION_INIT
+
+#include <filesystem>
+
+#include "../TestBase.h"
+#include "utils/file/FilePattern.h"
+#include "range/v3/view/transform.hpp"
+#include "range/v3/view/map.hpp"
+#include "range/v3/view/join.hpp"
+#include "range/v3/view/cache1.hpp"
+#include "range/v3/view/c_str.hpp"
+#include "range/v3/range/conversion.hpp"
+#include "range/v3/range.hpp"
Review comment:
Are all these includes necessary?
##########
File path: thirdparty/aws-sdk-cpp/dll-export-injection.patch
##########
@@ -0,0 +1,52 @@
+diff -rupN aws-sdk-cpp-src/aws-cpp-sdk-core/CMakeLists.txt aws-sdk-cpp-src-patched/aws-cpp-sdk-core/CMakeLists.txt
Review comment:
I think this patch could use a comment for future reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r685136268
##########
File path: extensions/aws/controllerservices/AWSCredentialsService.h
##########
@@ -26,6 +26,7 @@
#include "utils/AWSInitializer.h"
#include "core/Resource.h"
+#include "utils/OptionalUtils.h"
Review comment:
This header is not used in the file.
##########
File path: Extensions.md
##########
@@ -16,108 +16,62 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars. Registration should happen in source files,
+as otherwise, including another extension's headers would introduce the same resources in the including extension
+as well, possibly shadowing its own resources.
Review comment:
```suggestion
capabilities (classes) available to the system through registrars. Registration must happen in source files, not headers.
```
If someone questions it or wants to understand the reason, they can look at the macro implementations and figure out. Or we can keep the details but move them to code comments. The improvement would be shorter, easier to follow documentation.
##########
File path: Extensions.md
##########
@@ -16,108 +16,64 @@
To enable all extensions for your platform, you may use -DENABLE_ALL=TRUE OR select the option to "Enable all Extensions" in the bootstrap script. [ReadMe](https://github.com/apache/nifi-minifi-cpp/#bootstrapping)
-# Extensions by example
-
-Extensions consist of modules that are conditionally built into your client. Reasons why you may wish to do this with your modules/processors
-
- - Do not with to make dependencies required or the lack thereof is a known/expected runtime condition.
- - You wish to allow users to exclude dependencies for a variety of reasons.
-
-# Extensions by example
-We've used HTTP-CURL as the first example. We've taken all libcURL runtime classes and placed them into an extensions folder
- - /extensions/http-curl
-
-This folder contains a CMakeLists file so that we can conditionally build it. In the case with libcURL, if the user does not have curl installed OR they specify -DDISABLE_CURL=true in the cmake build, the extensions will not be built. In this case, when the extension is not built, C2 REST protocols, InvokeHTTP, and an HTTP Client implementation will not be included.
-
-Your CMAKE file should build a static library, that will be included into the run time. This must be added with your conditional to the libminifi CMAKE, along with a platform specific whole archive inclusion. Note that this will ensure that despite no direct linkage being found by the compiler, we will include the code so that we can dynamically find your code.
-
-# Including your extension in the build
-There is a new function that can be used in the root cmake to build and included your extension. An example is based on the LibArchive extension. The createExtension function has 8 possible arguments. The first five arguments are required.
-The first argument specifies the variable controlling the exclusion of this extension, followed by the variable that
-is used when including it into conditional statements. The third argument is the pretty name followed by the description of the extension and the extension directory. The first optional argument is the test directory, which must also contain a CMakeLists.txt file. The seventh argument can be a conditional variable that tells us whether or not to add a third party subdirectory specified by the final extension.
-
-In the lib archive example, we provide all arguments, but your extension may only need the first five and the the test folder. The seventh and eighth arguments must be included in tandem.
-
-```cmake
-if ( NOT LibArchive_FOUND OR BUILD_LIBARCHIVE )
- set(BUILD_TP "TRUE")
-endif()
-createExtension(DISABLE_LIBARCHIVE
- ARCHIVE-EXTENSIONS
- "ARCHIVE EXTENSIONS"
- "This Enables libarchive functionality including MergeContent, CompressContent, and (Un)FocusArchiveEntry"
- "extensions/libarchive"
- "${TEST_DIR}/archive-tests"
- BUILD_TP
- "thirdparty/libarchive-3.3.2")
+# Extension internals
+Extensions are dynamic libraries loaded at runtime by the agent. An extension makes its
+capabilities (classes) available to the system through registrars.
+
+``` C++
+// register user-facing classes as
+REGISTER_RESOURCE(InvokeHTTP, "An HTTP client processor which can interact with a configurable HTTP Endpoint. "
+ "The destination URL and HTTP Method are configurable. FlowFile attributes are converted to HTTP headers and the "
+ "FlowFile contents are included as the body of the request (if the HTTP Method is PUT, POST or PATCH).");
+
+// register internal resources as
+REGISTER_INTERNAL_RESOURCE(HTTPClient);
```
-It is advised that you also add your extension to bootstrap.sh as that is the suggested method of configuring MiNiFi C++
-
-# C bindings
-To find your classes, you must adhere to a dlsym call back that adheres to the core::ObjectFactory class, like the one below. This object factory will return a list of classes, that we can instantiate through the class loader mechanism. Note that since we are including your code directly into our runtime, we will take care of dlopen and dlsym calls. A map from the class name to the object factory is kept in memory.
+If you decide to put them in a header file, you better make sure that there are no dependencies between extensions,
+as the inclusion of such header from extension "A", will force it to be defined in the including extension "B". This could result
+in shadowing each other's resources. Best to put them in source files.
+
+Some extensions (e.g. `http-curl`) require initialization before use. You need to subclass `Extension` and let the system know by using `REGISTER_EXTENSION`.
```C++
-class __attribute__((visibility("default"))) HttpCurlObjectFactory : public core::ObjectFactory {
+class HttpCurlExtension : core::extension::Extension {
public:
Review comment:
I didn't mean exactly gcc constructor/destructor, just similar with our code. But even that could work by adding a config getter in libminifi, although I think that would be unnecessary added complexity.
I also think that inheritance is unnecessary added complexity. We don't need to inherit data or implementation.
https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#c120-use-class-hierarchies-to-represent-concepts-with-inherent-hierarchical-structure-only
##########
File path: extensions/standard-processors/tests/unit/ClassLoaderTests.cpp
##########
@@ -20,6 +20,8 @@
#include "core/Processor.h"
#include "core/ClassLoader.h"
#include "core/yaml/YamlConfiguration.h"
+#include "core/extension/ExtensionManager.h"
+#include "utils/file/FileUtils.h"
Review comment:
Are these necessary?
##########
File path: libminifi/include/utils/file/FileMatcher.h
##########
@@ -0,0 +1,102 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <set>
+#include <utility>
+#include <memory>
+
+#include "utils/OptionalUtils.h"
+#include "core/logging/Logger.h"
+
+struct FileMatcherTestAccessor;
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace utils {
+namespace file {
+
+class FileMatcher {
+ friend struct ::FileMatcherTestAccessor;
+ class FilePattern {
+ FilePattern(std::vector<std::string> directory_segments, std::string file_pattern, bool excluding)
+ : directory_segments_(std::move(directory_segments)),
+ file_pattern_(std::move(file_pattern)),
+ excluding_(excluding) {}
+
+ public:
+ enum class MatchResult {
+ INCLUDE, // dir/file should be processed according to the pattern
+ EXCLUDE, // dir/file is explicitly rejected by the pattern
+ DONT_CARE // dir/file does not match pattern, do what you may
Review comment:
I would rename DONT_CARE to NOT_MATCHING or similar to convey the meaning. It was strange reading DONT_CARE in the client code and wondering why would you not care.
##########
File path: libminifi/include/core/extension/ExtensionManager.h
##########
@@ -0,0 +1,65 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "core/logging/Logger.h"
+#include "DynamicLibrary.h"
Review comment:
This could be replaced with Module.h
##########
File path: libminifi/include/core/Resource.h
##########
@@ -37,29 +41,64 @@ namespace core {
#define MKSOC(x) #x
#define MAKESTRING(x) MKSOC(x)
+static inline ClassLoader& getClassLoader() {
+#ifdef MODULE_NAME
+ return ClassLoader::getDefaultClassLoader().getClassLoader(MAKESTRING(MODULE_NAME));
+#else
+ return ClassLoader::getDefaultClassLoader();
+#endif
+}
+
template<class T>
class StaticClassType {
public:
- StaticClassType(const std::string &name, const std::string &description = "") { // NOLINT
+ StaticClassType(const std::string& name, const std::optional<std::string>& description, const std::vector<std::string>& construction_names)
+ : name_(name), construction_names_(construction_names) { // NOLINT
Review comment:
Do we still need this NOLINT?
##########
File path: conf/minifi.properties
##########
@@ -20,6 +20,9 @@ nifi.administrative.yield.duration=30 sec
# If a component has no work to do (is "bored"), how long should we wait before checking again for work?
nifi.bored.yield.duration=100 millis
+# Comma separated path for the extension libraries. Relative path is relative to the minifi executable.
+nifi.extension.path=../extensions/*
Review comment:
I currently see two usual ways of launching minifi:
- The normal installation, where the MINIFI_HOME is usually /opt/minifi, and the binary is under MINIFI_HOME/bin
- After compilation, launching the compiled binary in a preexisting MINIFI_HOME
In the first case, either version is fine. In the second case, we would need extensions to be installed to build/extensions/ for them to work without extra effort, or we could load them from MINIFI_HOME/extensions/ in the proposed case.
Another use case that I could see happening in the future is standard linux packages:
- minifi binary under /usr/bin/minifi
- config files under /etc/minifi
- extensions under /usr/share/minifi/extensions, and I imagine MINIFI_HOME would be /usr/share/minifi.
I think the default would cover more use cases with a path relative to MINIFI_HOME, and we could avoid the "../" at the beginning of practically every path pattern. Given the above points, do you still prefer paths relative to the binary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680713612
##########
File path: libminifi/test/aws-tests/CMakeLists.txt
##########
@@ -30,9 +30,9 @@ FOREACH(testfile ${AWS_INTEGRATION_TESTS})
target_include_directories(${testfilename} PRIVATE BEFORE "${CMAKE_SOURCE_DIR}/extensions/expression-language")
createTests("${testfilename}")
target_link_libraries(${testfilename} ${CATCH_MAIN_LIB})
- target_wholearchive_library(${testfilename} minifi-aws)
- target_wholearchive_library(${testfilename} minifi-standard-processors)
- target_wholearchive_library(${testfilename} minifi-expression-language-extensions)
+ target_link_libraries(${testfilename} minifi-aws)
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r680774954
##########
File path: extensions/script/CMakeLists.txt
##########
@@ -68,13 +58,10 @@ if (ENABLE_LUA_SCRIPTING)
add_definitions(-DLUA_SUPPORT)
file(GLOB LUA_SOURCES "lua/*.cpp")
- add_library(minifi-lua-extensions STATIC ${LUA_SOURCES})
-
- target_link_libraries(minifi-lua-extensions ${LIBMINIFI})
- target_link_libraries(minifi-lua-extensions ${LUA_LIBRARIES} sol)
- target_link_libraries(minifi-script-extensions minifi-lua-extensions)
+ target_sources(minifi-script-extensions PRIVATE ${LUA_SOURCES})
- target_compile_features(minifi-lua-extensions PUBLIC cxx_std_14)
+ target_link_libraries(minifi-script-extensions ${LUA_LIBRARIES} sol)
+ target_compile_features(minifi-script-extensions PUBLIC cxx_std_14)
Review comment:
I can imagine scenarios where a user wants lua scripting, but doesn't want to install a python interpreter on their embedded system. Or the other way around. So I don't agree with the separation being "artificial" in the sense of being unnecessary.
I can accept a new limitation because of this feature, if it's not worth your time to fix this issue. In this case, please adapt the main CMakeLists.txt and the bootstrap script to only have one option that toggles scripting support with both python and lua.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] szaszm commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
szaszm commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r694677927
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::string getFullPath() const {
+ return utils::file::PathUtils::concat_path(dir, filename);
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::string& dir, const std::string& filename) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ dir,
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ std::vector<LibraryDescriptor> libraries;
+ utils::file::FileMatcher(pattern.value()).forEachFile([&] (const std::string& dir, const std::string& filename) {
Review comment:
okay. Are you still considering working with `path`s instead of strings, or are you also against that approach?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678898892
##########
File path: libminifi/include/core/ObjectFactory.h
##########
@@ -0,0 +1,152 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <string>
+#include <memory>
+#include "Core.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+
+/**
+ * Factory that is used as an interface for
+ * creating processors from shared objects.
+ */
+class ObjectFactory {
+ public:
+ ObjectFactory(const std::string &group) // NOLINT
Review comment:
changed to explicit (also with `DefaultObjectFactory`)
##########
File path: libminifi/include/core/ClassLoader.h
##########
@@ -56,182 +47,6 @@ namespace core {
#define RTLD_LOCAL (1 << 2)
#endif
-/**
- * Class used to provide a global initialization and deinitialization function for an ObjectFactory.
- * Calls to instances of all ObjectFactoryInitializers are done under a unique lock.
- */
-class ObjectFactoryInitializer {
- public:
- virtual ~ObjectFactoryInitializer() = default;
-
- /**
- * This function is be called before the ObjectFactory is used.
- * @return whether the initialization was successful. If false, deinitialize will NOT be called.
- */
- virtual bool initialize() = 0;
-
- /**
- * This function will be called after the ObjectFactory is not needed anymore.
- */
- virtual void deinitialize() = 0;
-};
-
-/**
- * Factory that is used as an interface for
- * creating processors from shared objects.
- */
-class ObjectFactory {
- public:
- ObjectFactory(const std::string &group) // NOLINT
- : group_(group) {
- }
-
- ObjectFactory() = default;
-
- /**
- * Virtual destructor.
- */
- virtual ~ObjectFactory() = default;
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual std::shared_ptr<CoreComponent> create(const std::string& /*name*/) {
- return nullptr;
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual CoreComponent *createRaw(const std::string& /*name*/) {
- return nullptr;
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual std::shared_ptr<CoreComponent> create(const std::string& /*name*/, const utils::Identifier& /*uuid*/) {
- return nullptr;
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual CoreComponent* createRaw(const std::string& /*name*/, const utils::Identifier& /*uuid*/) {
- return nullptr;
- }
-
- /**
- * Returns an initializer for the factory.
- */
- virtual std::unique_ptr<ObjectFactoryInitializer> getInitializer() {
- return nullptr;
- }
-
- /**
- * Gets the name of the object.
- * @return class name of processor
- */
- virtual std::string getName() = 0;
-
- virtual std::string getGroupName() const {
- return group_;
- }
-
- virtual std::string getClassName() = 0;
- /**
- * Gets the class name for the object
- * @return class name for the processor.
- */
- virtual std::vector<std::string> getClassNames() = 0;
-
- virtual std::unique_ptr<ObjectFactory> assign(const std::string &class_name) = 0;
-
- private:
- std::string group_;
-};
-/**
- * Factory that is used as an interface for
- * creating processors from shared objects.
- */
-template<class T>
-class DefautObjectFactory : public ObjectFactory {
- public:
- DefautObjectFactory() {
- className = core::getClassName<T>();
- }
-
- DefautObjectFactory(const std::string &group_name) // NOLINT
- : ObjectFactory(group_name) {
- className = core::getClassName<T>();
- }
- /**
- * Virtual destructor.
- */
- virtual ~DefautObjectFactory() = default;
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual std::shared_ptr<CoreComponent> create(const std::string &name) {
- std::shared_ptr<T> ptr = std::make_shared<T>(name);
- return std::static_pointer_cast<CoreComponent>(ptr);
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual std::shared_ptr<CoreComponent> create(const std::string &name, const utils::Identifier &uuid) {
- std::shared_ptr<T> ptr = std::make_shared<T>(name, uuid);
- return std::static_pointer_cast<CoreComponent>(ptr);
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual CoreComponent* createRaw(const std::string &name) {
- T *ptr = new T(name);
- return dynamic_cast<CoreComponent*>(ptr);
- }
-
- /**
- * Create a shared pointer to a new processor.
- */
- virtual CoreComponent* createRaw(const std::string &name, const utils::Identifier &uuid) {
- T *ptr = new T(name, uuid);
- return dynamic_cast<CoreComponent*>(ptr);
- }
-
- /**
- * Gets the name of the object.
- * @return class name of processor
- */
- virtual std::string getName() {
- return className;
- }
-
- /**
- * Gets the class name for the object
- * @return class name for the processor.
- */
- virtual std::string getClassName() {
- return className;
- }
-
- virtual std::vector<std::string> getClassNames() {
- std::vector<std::string> container;
- container.push_back(className);
- return container;
- }
-
- virtual std::unique_ptr<ObjectFactory> assign(const std::string& /*class_name*/) {
- return nullptr;
- }
-
- protected:
- std::string className;
-};
-
/**
* Function that is used to create the
* processor factory from the shared object.
Review comment:
removed
##########
File path: libminifi/src/agent/agent_docs.cpp
##########
@@ -0,0 +1,32 @@
+/*** Licensed to the Apache Software Foundation (ASF) under one or more
Review comment:
added newline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678081716
##########
File path: extensions/coap/controllerservice/CoapConnector.h
##########
@@ -25,6 +25,7 @@
#include "core/logging/LoggerConfiguration.h"
#include "core/controller/ControllerService.h"
+#include "core/Resource.h"
Review comment:
moved all `#include "core/Resource.h"` from headers to source files
##########
File path: cmake/BuildTests.cmake
##########
@@ -69,24 +69,23 @@ function(createTests testName)
if (ENABLE_BINARY_DIFF)
target_include_directories(${testName} SYSTEM BEFORE PRIVATE "${CMAKE_SOURCE_DIR}/thirdparty/bsdiff/")
- endif(ENABLE_BINARY_DIFF)
+ endif(ENABLE_BINARY_DIFF)
- if (Boost_FOUND)
- target_include_directories(${testName} BEFORE PRIVATE "${Boost_INCLUDE_DIRS}")
- endif()
- target_link_libraries(${testName} ${CMAKE_DL_LIBS} ${TEST_BASE_LIB})
- target_link_libraries(${testName} core-minifi yaml-cpp spdlog Threads::Threads)
- if (NOT excludeBase)
- target_wholearchive_library(${testName} minifi)
+ if (Boost_FOUND)
+ target_include_directories(${testName} BEFORE PRIVATE "${Boost_INCLUDE_DIRS}")
endif()
- add_dependencies(${testName} minifiexe)
+ target_link_libraries(${testName} ${CMAKE_DL_LIBS} ${TEST_BASE_LIB})
+ target_link_libraries(${testName} core-minifi yaml-cpp spdlog Threads::Threads)
+ add_dependencies(${testName} minifiexe)
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r678168130
##########
File path: libminifi/include/core/extension/Extension.h
##########
@@ -0,0 +1,90 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#pragma once
+
+#include <memory>
+#include <vector>
+#include <string>
+
+#include "properties/Configure.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+class Extension;
+
+using ExtensionConfig = std::shared_ptr<org::apache::nifi::minifi::Configure>;
+using ExtensionInit = bool(*)(Extension*, const ExtensionConfig&);
+
+class ExtensionInitializer;
+
+class Extension {
+ friend class ExtensionInitializer;
+ public:
+ explicit Extension(std::string name, ExtensionInit init);
+ virtual ~Extension();
+
+ bool initialize(const ExtensionConfig& config) {
+ return init_(this, config);
+ }
+
+ const std::string& getName() const {
+ return name_;
+ }
+
+ protected:
Review comment:
I don't feel "them being protected" and "having a friend" conflicting, `Extension` is referenced in other places (`ExtensionManager`, `Module`) and `doInitialize` does not guarantee the same semantics `initialize` does, so it should not be part of the public interface, and is expected to be overridden by subclasses and only be called by `ExtensionInitializer`, we could probably achieve the same thing through some inheritance hierarchy although I feel like that would make it more complex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [nifi-minifi-cpp] adamdebreceni commented on a change in pull request #1138: MINIFICPP-1471 - Build extensions as dynamic libraries and dynamically load them
Posted by GitBox <gi...@apache.org>.
adamdebreceni commented on a change in pull request #1138:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1138#discussion_r694689039
##########
File path: libminifi/src/core/extension/ExtensionManager.cpp
##########
@@ -0,0 +1,134 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "core/extension/ExtensionManager.h"
+#include "core/logging/LoggerConfiguration.h"
+#include "utils/file/FileUtils.h"
+#include "core/extension/Executable.h"
+#include "utils/file/FileMatcher.h"
+#include "core/extension/DynamicLibrary.h"
+
+namespace org {
+namespace apache {
+namespace nifi {
+namespace minifi {
+namespace core {
+namespace extension {
+
+namespace {
+struct LibraryDescriptor {
+ std::string name;
+ std::string dir;
+ std::string filename;
+
+ bool verify(const std::shared_ptr<logging::Logger>& /*logger*/) const {
+ // TODO(adebreceni): check signature
+ return true;
+ }
+
+ std::string getFullPath() const {
+ return utils::file::PathUtils::concat_path(dir, filename);
+ }
+};
+} // namespace
+
+static std::optional<LibraryDescriptor> asDynamicLibrary(const std::string& dir, const std::string& filename) {
+#if defined(WIN32)
+ const std::string extension = ".dll";
+#elif defined(__APPLE__)
+ const std::string extension = ".dylib";
+#else
+ const std::string extension = ".so";
+#endif
+
+#ifdef WIN32
+ const std::string prefix = "";
+#else
+ const std::string prefix = "lib";
+#endif
+ if (!utils::StringUtils::startsWith(filename, prefix) || !utils::StringUtils::endsWith(filename, extension)) {
+ return {};
+ }
+ return LibraryDescriptor{
+ filename.substr(prefix.length(), filename.length() - extension.length() - prefix.length()),
+ dir,
+ filename
+ };
+}
+
+std::shared_ptr<logging::Logger> ExtensionManager::logger_ = logging::LoggerFactory<ExtensionManager>::getLogger();
+
+ExtensionManager::ExtensionManager() {
+ modules_.push_back(std::make_unique<Executable>());
+ active_module_ = modules_[0].get();
+}
+
+ExtensionManager& ExtensionManager::get() {
+ static ExtensionManager instance;
+ return instance;
+}
+
+bool ExtensionManager::initialize(const std::shared_ptr<Configure>& config) {
+ static bool initialized = ([&] {
+ logger_->log_trace("Initializing extensions");
+ // initialize executable
+ active_module_->initialize(config);
+ std::optional<std::string> pattern = config ? config->get(nifi_extension_path) : std::nullopt;
+ if (!pattern) return;
+ std::vector<LibraryDescriptor> libraries;
+ utils::file::FileMatcher(pattern.value()).forEachFile([&] (const std::string& dir, const std::string& filename) {
Review comment:
I tried changing it and it is definitely something that should be done, but some properties (implicit convertibility from `std::string`, platform dependent implicit string conversion, what it considers an "absolute" path) made me believe that moving to `path` should be done at once, or at least in their dedicated PR-s
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org