You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "lordgamez (via GitHub)" <gi...@apache.org> on 2024/05/21 10:23:32 UTC
[PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
lordgamez opened a new pull request, #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792
https://issues.apache.org/jira/browse/MINIFICPP-2367
-----------
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1668741513
##########
libminifi/include/core/ConfigurableComponent.h:
##########
@@ -136,6 +136,8 @@ class ConfigurableComponent {
*/
bool getDynamicProperty(const std::string& name, std::string &value) const;
+ bool getDynamicProperty(const std::string& name, core::Property &item) const;
Review Comment:
It is currently used in the tests to check if dynamic property sequences are handled properly. In the other overload only 1 value can be queried, but if the property has multiple values as a value sequence we need this overload to check those values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1627977636
##########
libminifi/include/core/ProcessGroup.h:
##########
@@ -217,6 +217,9 @@ class ProcessGroup : public CoreComponent {
void verify() const;
+ void setParameterContextName(const std::string &name);
+ std::string getParameterContextName() const;
+
Review Comment:
I think this doesn't need the extra indirection. The config parser can look up the parameter context, and store a pointer to it here, in the process group. What do you think? Instead of ParameterContextName, we would work with ParameterContext directly.
##########
libminifi/include/core/ConfigurableComponent.h:
##########
@@ -136,6 +136,8 @@ class ConfigurableComponent {
*/
bool getDynamicProperty(const std::string& name, std::string &value) const;
+ bool getDynamicProperty(const std::string& name, core::Property &item) const;
Review Comment:
What's the purpose of this overload?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits closed pull request #1792: MINIFICPP-2367 Add support for non-sensitive parameters
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1668693502
##########
extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp:
##########
@@ -1072,3 +1075,581 @@ TEST_CASE("Test serialization", "[YamlConfiguration]") {
const std::string serialized_flow_definition_masked = std::regex_replace(serialized_flow_definition, std::regex{"enc\\{.*\\}"}, "enc{...}");
CHECK(serialized_flow_definition_masked == TEST_FLOW_WITH_SENSITIVE_PROPERTIES_ENCRYPTED);
}
+
+TEST_CASE("Yaml configuration can use parameter contexts", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - name: batch_size
+ description: ''
+ value: 12
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ std::unique_ptr<core::ProcessGroup> flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+ auto* proc = flow->findProcessorByName("TailFile");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Batch Size") == "12");
+ REQUIRE(proc->getProperty("Lookup frequency") == "12 min");
+}
+
+TEST_CASE("Yaml config should not replace parameter from different parameter context", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376978
+ name: other-context
+ description: my other context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Parameter 'batch_size' not found");
+}
+
+TEST_CASE("Cannot use the same parameter context name twice", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 1
+Processors: []
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Parameter context name 'my-context' already exists, parameter context names must be unique!");
+}
+
+TEST_CASE("Cannot use the same parameter name within a parameter context twice", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - name: lookup.frequency
+ description: ''
+ value: 1 min
+Processors: []
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Parameter name 'lookup.frequency' already exists, parameter names must be unique within a parameter context!");
+}
+
+class DummyFlowYamlProcessor : public core::Processor {
+ public:
+ using core::Processor::Processor;
+
+ static constexpr const char* Description = "A processor that does nothing.";
+ static constexpr auto SimpleProperty = core::PropertyDefinitionBuilder<>::createProperty("Simple Property")
+ .withDescription("Just a simple string property")
+ .build();
+ static constexpr auto SensitiveProperty = core::PropertyDefinitionBuilder<>::createProperty("Sensitive Property")
+ .withDescription("Sensitive property")
+ .isSensitive(true)
+ .build();
+ static constexpr auto Properties = std::array<core::PropertyReference, 2>{SimpleProperty, SensitiveProperty};
+ static constexpr auto Relationships = std::array<core::RelationshipDefinition, 0>{};
+ static constexpr bool SupportsDynamicProperties = true;
+ static constexpr bool SupportsDynamicRelationships = true;
+ static constexpr core::annotation::Input InputRequirement = core::annotation::Input::INPUT_ALLOWED;
+ static constexpr bool IsSingleThreaded = false;
+ ADD_COMMON_VIRTUAL_FUNCTIONS_FOR_PROCESSORS
+
+ void initialize() override { setSupportedProperties(Properties); }
+};
+
+REGISTER_RESOURCE(DummyFlowYamlProcessor, Processor);
+
+TEST_CASE("Cannot use non-sensitive parameter in sensitive property", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flowconfig
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: my_value
+ description: ''
+ value: value1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: simple
+ Sensitive Property: "#{my_value}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Non-sensitive parameter 'my_value' cannot be referenced in a sensitive property");
+}
+
+TEST_CASE("Cannot use non-sensitive parameter in sensitive property value sequence", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flowconfig
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: my_value
+ description: ''
+ value: value1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: simple
+ Sensitive Property:
+ - value: first value
+ - value: "#{my_value}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Non-sensitive parameter 'my_value' cannot be referenced in a sensitive property");
+}
+
+TEST_CASE("Parameters can be used in nested process groups", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376456
+ name: sub-context
+ description: my sub context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 12
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: 1
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+Process Groups:
+ - id: 2a3aaf32-8574-4fa7-b720-84001f8dde43
+ name: Sub process group
+ Processors:
+ - id: 12304f28-0158-1000-0000-000000000000
+ name: SubTailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: 1 sec
+ Parameter Context Name: sub-context
+ )";
+
+ std::unique_ptr<core::ProcessGroup> flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+ auto* proc = flow->findProcessorByName("TailFile");
+ REQUIRE(proc);
+ CHECK(proc->getProperty("Batch Size") == "1");
+ CHECK(proc->getProperty("Lookup frequency") == "12 min");
+ auto* subproc = flow->findProcessorByName("SubTailFile");
+ REQUIRE(subproc);
+ CHECK(subproc->getProperty("Batch Size") == "12");
+ CHECK(subproc->getProperty("Lookup frequency") == "1 sec");
+}
+
+TEST_CASE("Subprocessgroups cannot inherit parameters from parent processgroup", "[YamlConfiguration]") {
Review Comment:
Yes, this is on purpose. In NiFi parameter contexts are not inherited between process groups. It has to be explicitly selected which parameter context should be used for each process group. There is a feature in NiFi which will be implemented later in MiNiFi C++ to use parameter context inheritance. That allows parameter contexts to be derived from other parameter contexts and using the derived contexts allows the process group to use the parameters from the base parameter context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1670605385
##########
libminifi/src/core/ParameterTokenParser.cpp:
##########
@@ -0,0 +1,105 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "core/ParameterTokenParser.h"
+
+#include <stdexcept>
+#include "utils/RegexUtils.h"
+
+namespace org::apache::nifi::minifi::core {
+
+void ParameterTokenParser::parse() {
+ minifi::utils::Regex expr("[-a-zA-Z0-9_\\. ]+");
+ ParseState state = ParseState::OutsideToken;
+ uint32_t token_start = 0;
+ uint32_t hashmark_length = 0;
+ for (uint32_t i = 0; i < input_.size(); ++i) {
+ if (input_[i] == '#') {
+ if (state == ParseState::OutsideToken) {
+ state = ParseState::InHashMark;
+ }
+ if (state != ParseState::InToken) {
+ ++hashmark_length;
+ }
+ } else if (input_[i] == '{') {
+ if (state == ParseState::InHashMark) {
+ token_start = i - hashmark_length;
+ state = ParseState::InToken;
+ }
+ } else if (input_[i] == '}') {
+ if (state == ParseState::InToken) {
+ state = ParseState::OutsideToken;
+ gsl_Assert(hashmark_length > 0);
+ if (hashmark_length % 2 == 0) {
+ tokens_.push_back(std::make_unique<EscapedToken>(token_start, i - token_start + 1, input_.substr(token_start + (hashmark_length / 2), i - token_start + 1 - (hashmark_length / 2))));
+ } else {
+ auto token_name = input_.substr(token_start + hashmark_length + 1, i - token_start - hashmark_length - 1);
+ if (token_name.empty() || !minifi::utils::regexMatch(token_name, expr)) {
+ throw ParameterException("Invalid token name: '" + token_name + "'. "
+ "Only alpha-numeric characters (a-z, A-Z, 0-9), hyphens ( - ), underscores ( _ ), periods ( . ), and spaces are allowed in token name.");
+ }
+ tokens_.push_back(std::make_unique<ReplaceableToken>(token_name, (hashmark_length - 1) / 2, token_start, i - token_start + 1));
+ token_start = 0;
+ }
+ } else {
+ state = ParseState::OutsideToken;
+ }
+ hashmark_length = 0;
+ } else {
+ if (state != ParseState::InToken) {
+ state = ParseState::OutsideToken;
+ hashmark_length = 0;
+ }
+ }
+ }
+}
+
+std::string ParameterTokenParser::replaceParameters(ParameterContext* parameter_context, bool is_sensitive) const {
+ if (tokens_.empty()) {
+ return input_;
+ }
+ std::string result;
+ uint32_t last_end = 0;
+ for (const auto& token : tokens_) {
+ result.append(input_.substr(last_end, token->getStart() - last_end));
+ if (token->getType() == ParameterToken::ParameterTokenType::Escaped) {
+ result.append(token->getValue().value());
+ last_end = token->getStart() + token->getSize();
+ continue;
+ }
+
+ if (!parameter_context) {
+ throw ParameterException("Property references a parameter in its value, but no parameter context was provided.");
+ }
Review Comment:
I'd consider moving the check outside and making the `parameter_context` parameter not nullable (reference).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1668618742
##########
CONFIGURE.md:
##########
@@ -121,8 +124,101 @@ It's recommended to create your configuration in YAML format or configure the ag
max concurrent tasks: 1
Properties:
+Besides YAML configuration format, MiNiFi C++ also supports JSON configuration. To see different uses cases in both formats, please refer to the [examples page](examples/README.md) for flow config examples.
+
**NOTE:** Make sure to specify id for each component (Processor, Connection, Controller, RPG etc.) to make sure that Apache MiNiFi C++ can reload the state after a process restart. The id should be unique in the flow configuration.
+### Parameter Contexts
+
+Processor properties in flow configurations can be parameterized using parameters defined in parameter contexts. Flow configurations can define parameter contexts that define parameter-value pairs to be reused in the flow configuration as per the following rules:
+ - Parameters within a process group can only be used if the parameter context is assigned to that process group.
+ - Each process group can be assigned only one parameter context.
+ - Parameter contexts can be assigned to multiple process groups.
+ - The assigned parameter context has the scope of the process group, but not its child process groups.
+ - The parameters can be used in the flow configuration by using the `#{parameterName}` syntax for processor properties.
+ - Only alpha-numeric characters (a-z, A-Z, 0-9), hyphens ( - ), underscores ( _ ), periods ( . ), and spaces are allowed in parameter name.
+ - `#` character can be used to escape the parameter syntax. E.g. if the `parameterName` parameter's value is `xxx` then `#{parameterName}` will be replaced with `xxx`, `##{parameterName}` will be replaced with `#{parameterName}`, and `#####{parameterName}` will be replaced with `##xxx`.
+ - Parameters can only be used in values of non-sensitive processor properties.
+
+An example for using parameters in a JSON configuration file:
+
+```json
+{
+ "parameterContexts": [
+ {
+ "identifier": "804e6b47-ea22-45cd-a472-545801db98e6",
+ "name": "root-process-group-context",
+ "description": "Root process group parameter context",
+ "parameters": [
+ {
+ "name": "tail_base_dir",
+ "description": "Base dir of tailed files",
+ "value": "/tmp/tail/file/path"
+ }
+ ]
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Tail test_file1.log",
+ "identifier": "83b58f9f-e661-4634-96fb-0e82b92becdf",
+ "type": "org.apache.nifi.minifi.processors.TailFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "properties": {
+ "File to Tail": "#{tail_base_dir}/test_file1.log"
+ }
+ },
+ {
+ "name": "Tail test_file2.log",
+ "identifier": "8a772a10-7c34-48e7-b152-b1a32c5db83e",
+ "type": "org.apache.nifi.minifi.processors.TailFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "properties": {
+ "File to Tail": "#{tail_base_dir}/test_file2.log"
+ }
+ }
+ ],
+ "parameterContextName": "root-process-group-context"
Review Comment:
Are the parameter contexts identified by name in nifi too?
##########
libminifi/src/core/flow/StructuredConfiguration.cpp:
##########
@@ -799,6 +862,19 @@ void StructuredConfiguration::parsePorts(const flow::Node& node, core::ProcessGr
}
}
+void StructuredConfiguration::parseParameterContextName(const flow::Node& node, core::ProcessGroup* parent) {
+ if (!parent) {
+ logger_->log_error("parseParameterContextName: no parent group was provided");
+ return;
+ }
Review Comment:
this could take a reference instead, that way no null checking is necessary
##########
libminifi/include/core/ParameterContext.h:
##########
@@ -0,0 +1,61 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include <string>
+#include <unordered_set>
+#include <optional>
+
+#include "Core.h"
+
+namespace org::apache::nifi::minifi::core {
+
+class ParameterException : public std::runtime_error {
Review Comment:
This could be derived from minifi::Exception. Maybe have its own type enum value
##########
extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp:
##########
@@ -1072,3 +1075,581 @@ TEST_CASE("Test serialization", "[YamlConfiguration]") {
const std::string serialized_flow_definition_masked = std::regex_replace(serialized_flow_definition, std::regex{"enc\\{.*\\}"}, "enc{...}");
CHECK(serialized_flow_definition_masked == TEST_FLOW_WITH_SENSITIVE_PROPERTIES_ENCRYPTED);
}
+
+TEST_CASE("Yaml configuration can use parameter contexts", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - name: batch_size
+ description: ''
+ value: 12
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ std::unique_ptr<core::ProcessGroup> flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+ auto* proc = flow->findProcessorByName("TailFile");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Batch Size") == "12");
+ REQUIRE(proc->getProperty("Lookup frequency") == "12 min");
+}
+
+TEST_CASE("Yaml config should not replace parameter from different parameter context", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376978
+ name: other-context
+ description: my other context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Parameter 'batch_size' not found");
+}
+
+TEST_CASE("Cannot use the same parameter context name twice", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 1
+Processors: []
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Parameter context name 'my-context' already exists, parameter context names must be unique!");
+}
+
+TEST_CASE("Cannot use the same parameter name within a parameter context twice", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - name: lookup.frequency
+ description: ''
+ value: 1 min
+Processors: []
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Parameter name 'lookup.frequency' already exists, parameter names must be unique within a parameter context!");
+}
+
+class DummyFlowYamlProcessor : public core::Processor {
+ public:
+ using core::Processor::Processor;
+
+ static constexpr const char* Description = "A processor that does nothing.";
+ static constexpr auto SimpleProperty = core::PropertyDefinitionBuilder<>::createProperty("Simple Property")
+ .withDescription("Just a simple string property")
+ .build();
+ static constexpr auto SensitiveProperty = core::PropertyDefinitionBuilder<>::createProperty("Sensitive Property")
+ .withDescription("Sensitive property")
+ .isSensitive(true)
+ .build();
+ static constexpr auto Properties = std::array<core::PropertyReference, 2>{SimpleProperty, SensitiveProperty};
+ static constexpr auto Relationships = std::array<core::RelationshipDefinition, 0>{};
+ static constexpr bool SupportsDynamicProperties = true;
+ static constexpr bool SupportsDynamicRelationships = true;
+ static constexpr core::annotation::Input InputRequirement = core::annotation::Input::INPUT_ALLOWED;
+ static constexpr bool IsSingleThreaded = false;
+ ADD_COMMON_VIRTUAL_FUNCTIONS_FOR_PROCESSORS
+
+ void initialize() override { setSupportedProperties(Properties); }
+};
+
+REGISTER_RESOURCE(DummyFlowYamlProcessor, Processor);
+
+TEST_CASE("Cannot use non-sensitive parameter in sensitive property", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flowconfig
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: my_value
+ description: ''
+ value: value1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: simple
+ Sensitive Property: "#{my_value}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Non-sensitive parameter 'my_value' cannot be referenced in a sensitive property");
+}
+
+TEST_CASE("Cannot use non-sensitive parameter in sensitive property value sequence", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flowconfig
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: my_value
+ description: ''
+ value: value1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: simple
+ Sensitive Property:
+ - value: first value
+ - value: "#{my_value}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Non-sensitive parameter 'my_value' cannot be referenced in a sensitive property");
+}
+
+TEST_CASE("Parameters can be used in nested process groups", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376456
+ name: sub-context
+ description: my sub context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 12
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: 1
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+Process Groups:
+ - id: 2a3aaf32-8574-4fa7-b720-84001f8dde43
+ name: Sub process group
+ Processors:
+ - id: 12304f28-0158-1000-0000-000000000000
+ name: SubTailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: 1 sec
+ Parameter Context Name: sub-context
+ )";
+
+ std::unique_ptr<core::ProcessGroup> flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+ auto* proc = flow->findProcessorByName("TailFile");
+ REQUIRE(proc);
+ CHECK(proc->getProperty("Batch Size") == "1");
+ CHECK(proc->getProperty("Lookup frequency") == "12 min");
+ auto* subproc = flow->findProcessorByName("SubTailFile");
+ REQUIRE(subproc);
+ CHECK(subproc->getProperty("Batch Size") == "12");
+ CHECK(subproc->getProperty("Lookup frequency") == "1 sec");
+}
+
+TEST_CASE("Subprocessgroups cannot inherit parameters from parent processgroup", "[YamlConfiguration]") {
Review Comment:
Is this on purpose? I thought they'd be inherited.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1668693502
##########
extensions/standard-processors/tests/unit/YamlConfigurationTests.cpp:
##########
@@ -1072,3 +1075,581 @@ TEST_CASE("Test serialization", "[YamlConfiguration]") {
const std::string serialized_flow_definition_masked = std::regex_replace(serialized_flow_definition, std::regex{"enc\\{.*\\}"}, "enc{...}");
CHECK(serialized_flow_definition_masked == TEST_FLOW_WITH_SENSITIVE_PROPERTIES_ENCRYPTED);
}
+
+TEST_CASE("Yaml configuration can use parameter contexts", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - name: batch_size
+ description: ''
+ value: 12
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ std::unique_ptr<core::ProcessGroup> flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+ auto* proc = flow->findProcessorByName("TailFile");
+ REQUIRE(proc);
+ REQUIRE(proc->getProperty("Batch Size") == "12");
+ REQUIRE(proc->getProperty("Lookup frequency") == "12 min");
+}
+
+TEST_CASE("Yaml config should not replace parameter from different parameter context", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376978
+ name: other-context
+ description: my other context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Parameter 'batch_size' not found");
+}
+
+TEST_CASE("Cannot use the same parameter context name twice", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 1
+Processors: []
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Parameter context name 'my-context' already exists, parameter context names must be unique!");
+}
+
+TEST_CASE("Cannot use the same parameter name within a parameter context twice", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - name: lookup.frequency
+ description: ''
+ value: 1 min
+Processors: []
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Parameter name 'lookup.frequency' already exists, parameter names must be unique within a parameter context!");
+}
+
+class DummyFlowYamlProcessor : public core::Processor {
+ public:
+ using core::Processor::Processor;
+
+ static constexpr const char* Description = "A processor that does nothing.";
+ static constexpr auto SimpleProperty = core::PropertyDefinitionBuilder<>::createProperty("Simple Property")
+ .withDescription("Just a simple string property")
+ .build();
+ static constexpr auto SensitiveProperty = core::PropertyDefinitionBuilder<>::createProperty("Sensitive Property")
+ .withDescription("Sensitive property")
+ .isSensitive(true)
+ .build();
+ static constexpr auto Properties = std::array<core::PropertyReference, 2>{SimpleProperty, SensitiveProperty};
+ static constexpr auto Relationships = std::array<core::RelationshipDefinition, 0>{};
+ static constexpr bool SupportsDynamicProperties = true;
+ static constexpr bool SupportsDynamicRelationships = true;
+ static constexpr core::annotation::Input InputRequirement = core::annotation::Input::INPUT_ALLOWED;
+ static constexpr bool IsSingleThreaded = false;
+ ADD_COMMON_VIRTUAL_FUNCTIONS_FOR_PROCESSORS
+
+ void initialize() override { setSupportedProperties(Properties); }
+};
+
+REGISTER_RESOURCE(DummyFlowYamlProcessor, Processor);
+
+TEST_CASE("Cannot use non-sensitive parameter in sensitive property", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flowconfig
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: my_value
+ description: ''
+ value: value1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: simple
+ Sensitive Property: "#{my_value}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Non-sensitive parameter 'my_value' cannot be referenced in a sensitive property");
+}
+
+TEST_CASE("Cannot use non-sensitive parameter in sensitive property value sequence", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: flowconfig
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: my_value
+ description: ''
+ value: value1
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.DummyFlowYamlProcessor
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Simple Property: simple
+ Sensitive Property:
+ - value: first value
+ - value: "#{my_value}"
+Controller Services: []
+Process Groups: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+NiFi Properties Overrides: {}
+ )";
+
+ REQUIRE_THROWS_WITH(yaml_config.getRootFromPayload(TEST_CONFIG_YAML), "Non-sensitive parameter 'my_value' cannot be referenced in a sensitive property");
+}
+
+TEST_CASE("Parameters can be used in nested process groups", "[YamlConfiguration]") {
+ ConfigurationTestController test_controller;
+ core::YamlConfiguration yaml_config(test_controller.getContext());
+
+ static const std::string TEST_CONFIG_YAML =
+ R"(
+MiNiFi Config Version: 3
+Flow Controller:
+ name: Simple TailFile
+Parameter Contexts:
+ - id: 721e10b7-8e00-3188-9a27-476cca376978
+ name: my-context
+ description: my parameter context
+ Parameters:
+ - name: lookup.frequency
+ description: ''
+ value: 12 min
+ - id: 123e10b7-8e00-3188-9a27-476cca376456
+ name: sub-context
+ description: my sub context
+ Parameters:
+ - name: batch_size
+ description: ''
+ value: 12
+Processors:
+- id: b0c04f28-0158-1000-0000-000000000000
+ name: TailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: 1
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: "#{lookup.frequency}"
+Controller Services: []
+Input Ports: []
+Output Ports: []
+Funnels: []
+Connections: []
+Parameter Context Name: my-context
+Process Groups:
+ - id: 2a3aaf32-8574-4fa7-b720-84001f8dde43
+ name: Sub process group
+ Processors:
+ - id: 12304f28-0158-1000-0000-000000000000
+ name: SubTailFile
+ class: org.apache.nifi.processors.standard.TailFile
+ max concurrent tasks: 1
+ scheduling strategy: TIMER_DRIVEN
+ scheduling period: 1 sec
+ auto-terminated relationships list: [success]
+ Properties:
+ Batch Size: "#{batch_size}"
+ File to Tail: ./logs/minifi-app.log
+ Initial Start Position: Beginning of File
+ tail-mode: Single file
+ Lookup frequency: 1 sec
+ Parameter Context Name: sub-context
+ )";
+
+ std::unique_ptr<core::ProcessGroup> flow = yaml_config.getRootFromPayload(TEST_CONFIG_YAML);
+ REQUIRE(flow);
+ auto* proc = flow->findProcessorByName("TailFile");
+ REQUIRE(proc);
+ CHECK(proc->getProperty("Batch Size") == "1");
+ CHECK(proc->getProperty("Lookup frequency") == "12 min");
+ auto* subproc = flow->findProcessorByName("SubTailFile");
+ REQUIRE(subproc);
+ CHECK(subproc->getProperty("Batch Size") == "12");
+ CHECK(subproc->getProperty("Lookup frequency") == "1 sec");
+}
+
+TEST_CASE("Subprocessgroups cannot inherit parameters from parent processgroup", "[YamlConfiguration]") {
Review Comment:
Yes, this is on purpose. In NiFi parameter contexts are not inherited between process groups. It has to be explicitly selected which parameter context should be used for each process group. There is a feature in NiFi which will be implemented later to use parameter context inheritance. That allows parameter contexts to be derived from other parameter contexts and using the derived contexts allows the process group to use the parameters from the base parameter context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1670609030
##########
libminifi/include/core/FlowConfiguration.h:
##########
@@ -134,6 +135,7 @@ class FlowConfiguration : public CoreComponent {
utils::ChecksumCalculator& getChecksumCalculator() { return checksum_calculator_; }
protected:
+ std::unordered_map<std::string, std::unique_ptr<ParameterContext>> parameter_contexts_;
Review Comment:
We could make this `gsl::not_null`, and not even store null contexts if they are even possible. And just give stronger guarantees to any code that uses this member.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1670704775
##########
libminifi/src/core/ParameterTokenParser.cpp:
##########
@@ -0,0 +1,105 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#include "core/ParameterTokenParser.h"
+
+#include <stdexcept>
+#include "utils/RegexUtils.h"
+
+namespace org::apache::nifi::minifi::core {
+
+void ParameterTokenParser::parse() {
+ minifi::utils::Regex expr("[-a-zA-Z0-9_\\. ]+");
+ ParseState state = ParseState::OutsideToken;
+ uint32_t token_start = 0;
+ uint32_t hashmark_length = 0;
+ for (uint32_t i = 0; i < input_.size(); ++i) {
+ if (input_[i] == '#') {
+ if (state == ParseState::OutsideToken) {
+ state = ParseState::InHashMark;
+ }
+ if (state != ParseState::InToken) {
+ ++hashmark_length;
+ }
+ } else if (input_[i] == '{') {
+ if (state == ParseState::InHashMark) {
+ token_start = i - hashmark_length;
+ state = ParseState::InToken;
+ }
+ } else if (input_[i] == '}') {
+ if (state == ParseState::InToken) {
+ state = ParseState::OutsideToken;
+ gsl_Assert(hashmark_length > 0);
+ if (hashmark_length % 2 == 0) {
+ tokens_.push_back(std::make_unique<EscapedToken>(token_start, i - token_start + 1, input_.substr(token_start + (hashmark_length / 2), i - token_start + 1 - (hashmark_length / 2))));
+ } else {
+ auto token_name = input_.substr(token_start + hashmark_length + 1, i - token_start - hashmark_length - 1);
+ if (token_name.empty() || !minifi::utils::regexMatch(token_name, expr)) {
+ throw ParameterException("Invalid token name: '" + token_name + "'. "
+ "Only alpha-numeric characters (a-z, A-Z, 0-9), hyphens ( - ), underscores ( _ ), periods ( . ), and spaces are allowed in token name.");
+ }
+ tokens_.push_back(std::make_unique<ReplaceableToken>(token_name, (hashmark_length - 1) / 2, token_start, i - token_start + 1));
+ token_start = 0;
+ }
+ } else {
+ state = ParseState::OutsideToken;
+ }
+ hashmark_length = 0;
+ } else {
+ if (state != ParseState::InToken) {
+ state = ParseState::OutsideToken;
+ hashmark_length = 0;
+ }
+ }
+ }
+}
+
+std::string ParameterTokenParser::replaceParameters(ParameterContext* parameter_context, bool is_sensitive) const {
+ if (tokens_.empty()) {
+ return input_;
+ }
+ std::string result;
+ uint32_t last_end = 0;
+ for (const auto& token : tokens_) {
+ result.append(input_.substr(last_end, token->getStart() - last_end));
+ if (token->getType() == ParameterToken::ParameterTokenType::Escaped) {
+ result.append(token->getValue().value());
+ last_end = token->getStart() + token->getSize();
+ continue;
+ }
+
+ if (!parameter_context) {
+ throw ParameterException("Property references a parameter in its value, but no parameter context was provided.");
+ }
Review Comment:
In some cases parameter context is not required here, when no parameter tokens are present only the escaped `#` characters are replaced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1668807387
##########
libminifi/include/core/ProcessGroup.h:
##########
@@ -217,6 +217,9 @@ class ProcessGroup : public CoreComponent {
void verify() const;
+ void setParameterContextName(const std::string &name);
+ std::string getParameterContextName() const;
+
Review Comment:
Good idea, updated in 7404acec9914456ec059be5b109e09d6c8dff9d2
##########
libminifi/src/core/flow/StructuredConfiguration.cpp:
##########
@@ -799,6 +862,19 @@ void StructuredConfiguration::parsePorts(const flow::Node& node, core::ProcessGr
}
}
+void StructuredConfiguration::parseParameterContextName(const flow::Node& node, core::ProcessGroup* parent) {
+ if (!parent) {
+ logger_->log_error("parseParameterContextName: no parent group was provided");
+ return;
+ }
Review Comment:
Updated in 7404acec9914456ec059be5b109e09d6c8dff9d2
##########
libminifi/include/core/ParameterContext.h:
##########
@@ -0,0 +1,61 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include <string>
+#include <unordered_set>
+#include <optional>
+
+#include "Core.h"
+
+namespace org::apache::nifi::minifi::core {
+
+class ParameterException : public std::runtime_error {
Review Comment:
Updated in 7404acec9914456ec059be5b109e09d6c8dff9d2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1668752869
##########
CONFIGURE.md:
##########
@@ -121,8 +124,101 @@ It's recommended to create your configuration in YAML format or configure the ag
max concurrent tasks: 1
Properties:
+Besides YAML configuration format, MiNiFi C++ also supports JSON configuration. To see different uses cases in both formats, please refer to the [examples page](examples/README.md) for flow config examples.
+
**NOTE:** Make sure to specify id for each component (Processor, Connection, Controller, RPG etc.) to make sure that Apache MiNiFi C++ can reload the state after a process restart. The id should be unique in the flow configuration.
+### Parameter Contexts
+
+Processor properties in flow configurations can be parameterized using parameters defined in parameter contexts. Flow configurations can define parameter contexts that define parameter-value pairs to be reused in the flow configuration as per the following rules:
+ - Parameters within a process group can only be used if the parameter context is assigned to that process group.
+ - Each process group can be assigned only one parameter context.
+ - Parameter contexts can be assigned to multiple process groups.
+ - The assigned parameter context has the scope of the process group, but not its child process groups.
+ - The parameters can be used in the flow configuration by using the `#{parameterName}` syntax for processor properties.
+ - Only alpha-numeric characters (a-z, A-Z, 0-9), hyphens ( - ), underscores ( _ ), periods ( . ), and spaces are allowed in parameter name.
+ - `#` character can be used to escape the parameter syntax. E.g. if the `parameterName` parameter's value is `xxx` then `#{parameterName}` will be replaced with `xxx`, `##{parameterName}` will be replaced with `#{parameterName}`, and `#####{parameterName}` will be replaced with `##xxx`.
+ - Parameters can only be used in values of non-sensitive processor properties.
+
+An example for using parameters in a JSON configuration file:
+
+```json
+{
+ "parameterContexts": [
+ {
+ "identifier": "804e6b47-ea22-45cd-a472-545801db98e6",
+ "name": "root-process-group-context",
+ "description": "Root process group parameter context",
+ "parameters": [
+ {
+ "name": "tail_base_dir",
+ "description": "Base dir of tailed files",
+ "value": "/tmp/tail/file/path"
+ }
+ ]
+ }
+ ],
+ "rootGroup": {
+ "name": "MiNiFi Flow",
+ "processors": [
+ {
+ "name": "Tail test_file1.log",
+ "identifier": "83b58f9f-e661-4634-96fb-0e82b92becdf",
+ "type": "org.apache.nifi.minifi.processors.TailFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "properties": {
+ "File to Tail": "#{tail_base_dir}/test_file1.log"
+ }
+ },
+ {
+ "name": "Tail test_file2.log",
+ "identifier": "8a772a10-7c34-48e7-b152-b1a32c5db83e",
+ "type": "org.apache.nifi.minifi.processors.TailFile",
+ "schedulingStrategy": "TIMER_DRIVEN",
+ "schedulingPeriod": "1000 ms",
+ "properties": {
+ "File to Tail": "#{tail_base_dir}/test_file2.log"
+ }
+ }
+ ],
+ "parameterContextName": "root-process-group-context"
Review Comment:
Yes, this works the same way in NiFi
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2367 Add support for non-sensitive parameters [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1792:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1792#discussion_r1670732275
##########
libminifi/include/core/FlowConfiguration.h:
##########
@@ -134,6 +135,7 @@ class FlowConfiguration : public CoreComponent {
utils::ChecksumCalculator& getChecksumCalculator() { return checksum_calculator_; }
protected:
+ std::unordered_map<std::string, std::unique_ptr<ParameterContext>> parameter_contexts_;
Review Comment:
Updated in b7ea62e8d312dcb41c2893263cdbfcc46ab22064
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org