You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "lordgamez (via GitHub)" <gi...@apache.org> on 2025/07/11 12:33:41 UTC
[PR] MINIFICPP-2587 Add Sparkplug support for ConsumeMQTT processor [nifi-minifi-cpp]
lordgamez opened a new pull request, #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990
- Extract protobuf library to be built separately
- Upgrade grpc library to v1.72.2
- Change RecordReader interface to expect InputStream instead of FlowFile and ProcessSession
- Add SparkplugReader record reader controller
- Add Record Reader and Record Writer properties to ConsumeMQTT processor
- Add "Add Attributes As Fields" to add flow file attributes as record fields if records are used in ConsumeMQTT
- Make ConsumeMQTT trigger unit testable
- Add new attributes to ConsumeMQTT created flow files to match NiFi
- Implement record reader and writer usage in ConsumeMQTT
https://issues.apache.org/jira/browse/MINIFICPP-2587
Depends on https://github.com/apache/nifi-minifi-cpp/pull/1968
--------------------
Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.
In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:
### For all changes:
- [ ] Is there a JIRA ticket associated with this PR? Is it referenced
in the commit message?
- [ ] Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
- [ ] Has your PR been rebased against the latest commit within the target branch (typically main)?
- [ ] Is your initial contribution a single, squashed commit?
### For code changes:
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the LICENSE file?
- [ ] If applicable, have you updated the NOTICE file?
### For documentation related changes:
- [ ] Have you ensured that format looks appropriate for the output in which it is rendered?
### Note:
Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2200619342
##########
CONTROLLERS.md:
##########
@@ -277,6 +278,20 @@ In the list below, the names of required properties appear in bold. Any other pr
| Password | | | The password used for authentication. Required if Username is set.<br/>**Sensitive Property: true** |
+## SparkplugReader
+
+### Description
+
+Reads Sparkplug B messages and turns them into individual Record objects. The reader expects a single Sparkplug B payload in a read operation, which is a protobuf-encoded binary message. This reader is typically used with MQTT processors like ConsumeMQTT.
Review Comment:
```suggestion
Reads Sparkplug B messages and turns them into individual Record objects. The reader expects a single Sparkplug B payload in a read operation, which is a protobuf-encoded binary message. This reader is designed to be used with ConsumeMQTT, since Sparkplug B is an MQTT-based protocol.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2254103403
##########
extensions/mqtt/processors/ConsumeMQTT.cpp:
##########
@@ -26,17 +25,44 @@
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
#include "core/Resource.h"
+#include "io/BufferStream.h"
+#include "utils/ProcessorConfigUtils.h"
#include "utils/StringUtils.h"
#include "utils/ValueParser.h"
-#include "utils/ProcessorConfigUtils.h"
namespace org::apache::nifi::minifi::processors {
+namespace {
+template<typename RecordSetIO>
+std::shared_ptr<RecordSetIO> getRecordSetIO(core::ProcessContext& context, const core::PropertyReference& property,
+ const utils::Identifier& processor_uuid) {
+ std::string service_name = context.getProperty(property).value_or("");
+ if (!IsNullOrEmpty(service_name)) {
+ auto record_set_io = std::dynamic_pointer_cast<RecordSetIO>(context.getControllerService(service_name, processor_uuid));
+ if (!record_set_io) {
+ throw Exception(ExceptionType::PROCESS_SCHEDULE_EXCEPTION, fmt::format("'{}' property is set to invalid controller service '{}'", property.name, service_name));
+ }
+ return record_set_io;
+ }
+ return nullptr;
+}
Review Comment:
My problem with that currently that we do not differentiate if the property was set, but it was invalid or if the property was not set at all. Maybe we should change the current utility to only return std::nullopt if the input is invalid, otherwise return nullptr, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2254146463
##########
extensions/mqtt/processors/ConsumeMQTT.cpp:
##########
@@ -26,17 +25,44 @@
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
#include "core/Resource.h"
+#include "io/BufferStream.h"
+#include "utils/ProcessorConfigUtils.h"
#include "utils/StringUtils.h"
#include "utils/ValueParser.h"
-#include "utils/ProcessorConfigUtils.h"
namespace org::apache::nifi::minifi::processors {
+namespace {
+template<typename RecordSetIO>
+std::shared_ptr<RecordSetIO> getRecordSetIO(core::ProcessContext& context, const core::PropertyReference& property,
+ const utils::Identifier& processor_uuid) {
+ std::string service_name = context.getProperty(property).value_or("");
+ if (!IsNullOrEmpty(service_name)) {
+ auto record_set_io = std::dynamic_pointer_cast<RecordSetIO>(context.getControllerService(service_name, processor_uuid));
+ if (!record_set_io) {
+ throw Exception(ExceptionType::PROCESS_SCHEDULE_EXCEPTION, fmt::format("'{}' property is set to invalid controller service '{}'", property.name, service_name));
+ }
+ return record_set_io;
+ }
+ return nullptr;
+}
Review Comment:
Good point. Another option would be to throw on invalid and return either nullopt or nullptr if the property is not set.
EDIT: we agreed that I will change `utils::parseOptionalControllerService()` to throw (and also to not use `optional`) in a separate PR. For now, this can stay as it is, and we will rebase one or the other PR depending on which gets merged first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2254294128
##########
extensions/mqtt/processors/ConsumeMQTT.h:
##########
@@ -92,15 +107,26 @@ class ConsumeMQTT : public processors::AbstractMQTTProcessor {
QueueBufferMaxMessage,
AttributeFromContentType,
TopicAliasMaximum,
- ReceiveMaximum
+ ReceiveMaximum,
+ RecordReader,
+ RecordWriter,
+ AddAttributesAsFields
}), AbstractMQTTProcessor::AdvancedProperties);
EXTENSIONAPI static constexpr auto Success = core::RelationshipDefinition{"success", "FlowFiles that are sent successfully to the destination are transferred to this relationship"};
EXTENSIONAPI static constexpr auto Relationships = std::array{Success};
EXTENSIONAPI static constexpr auto BrokerOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.broker", {}, "URI of the sending broker"};
EXTENSIONAPI static constexpr auto TopicOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.topic", {}, "Topic of the message"};
- EXTENSIONAPI static constexpr auto OutputAttributes = std::array<core::OutputAttributeReference, 2>{BrokerOutputAttribute, TopicOutputAttribute};
+ EXTENSIONAPI static constexpr auto TopicSegmentOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.topic.segment.n", {}, "The nth topic segment of the message"};
+ EXTENSIONAPI static constexpr auto QosOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.qos", {}, "The quality of service for this message."};
+ EXTENSIONAPI static constexpr auto IsDuplicateOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.isDuplicate", {},
+ "Whether or not this message might be a duplicate of one which has already been received."};
+ EXTENSIONAPI static constexpr auto IsRetainedOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.isRetained", {},
+ "Whether or not this message was from a current publisher, or was \"retained\" by the server as the last message published on the topic."};
+ EXTENSIONAPI static constexpr auto RecordCountOutputAttribute = core::OutputAttributeDefinition<0>{"record.count", {}, "The number of records received"};
+ EXTENSIONAPI static constexpr auto OutputAttributes = std::array<core::OutputAttributeReference, 7>{BrokerOutputAttribute, TopicOutputAttribute, TopicSegmentOutputAttribute,
+ QosOutputAttribute, IsDuplicateOutputAttribute, IsRetainedOutputAttribute, RecordCountOutputAttribute};
Review Comment:
Updated in https://github.com/apache/nifi-minifi-cpp/pull/1990/commits/7e2cad557da32447caada0536e44ada143104195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "szaszm (via GitHub)" <gi...@apache.org>.
szaszm commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2246494405
##########
PROCESSORS.md:
##########
@@ -433,10 +436,15 @@ In the list below, the names of required properties appear in bold. Any other pr
### Output Attributes
-| Attribute | Relationship | Description |
-|-------------|--------------|---------------------------|
-| mqtt.broker | | URI of the sending broker |
-| mqtt.topic | | Topic of the message |
+| Attribute | Relationship | Description |
+|----------------------|--------------|---------------------------------------------------------------------------------------------------------------------------------------|
+| mqtt.broker | | URI of the sending broker |
+| mqtt.topic | | Topic of the message |
+| mqtt.topic.segment.n | | The nth topic segment of the message |
+| mqtt.qos | | The quality of service for this message. |
+| mqtt.isDuplicate | | Whether or not this message might be a duplicate of one which has already been received. |
+| mqtt.isRetained | | Whether or not this message was from a current publisher, or was "retained" by the server as the last message published on the topic. |
Review Comment:
I'm only seeing mqtt.broker in the flow file attributes with the record reader/writer set, but I'm seeing similar record attributes with an underscore prefix. We should make it clear which attribute ends up where.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2254076180
##########
docker/test/integration/cluster/checkers/sparkplug_b_pb2.py:
##########
Review Comment:
I'm not sure if it's worth adding protoc as a required dependency for the test environment just for generating this file, so I think in this case it's easier just to add this file to the repo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2254114235
##########
extensions/mqtt/processors/ConsumeMQTT.cpp:
##########
@@ -58,8 +84,55 @@ void ConsumeMQTT::readProperties(core::ProcessContext& context) {
receive_maximum_ = gsl::narrow<uint16_t>(utils::parseU64Property(context, ReceiveMaximum));
}
-void ConsumeMQTT::onTriggerImpl(core::ProcessContext&, core::ProcessSession& session) {
- std::queue<SmartMessage> msg_queue = getReceivedMqttMessages();
+void ConsumeMQTT::addAttributesAsRecordFields(core::RecordSet& new_records, const std::queue<SmartMessage>& msg_queue) const {
+ if (!add_attributes_as_fields_) {
+ return;
+ }
+
+ for (auto& record: new_records) {
+ record.emplace("_topic", core::RecordField(msg_queue.front().topic));
+ auto topic_segments = utils::string::split(msg_queue.front().topic, "/");
+ for (size_t i = 0; i < topic_segments.size(); ++i) {
+ record.emplace("_topic.segment." + std::to_string(i), core::RecordField(topic_segments[i]));
+ }
+ record.emplace("_qos", core::RecordField(msg_queue.front().contents->qos));
+ record.emplace("_isDuplicate", core::RecordField(msg_queue.front().contents->dup > 0));
+ record.emplace("_isRetained", core::RecordField(msg_queue.front().contents->retained > 0));
+ }
+}
+
+void ConsumeMQTT::transferMessagesAsRecords(core::ProcessSession& session) {
+ gsl_Expects(record_set_reader_ && record_set_writer_);
+ auto msg_queue = getReceivedMqttMessages();
+ core::RecordSet record_set;
+ while (!msg_queue.empty()) {
+ io::BufferStream buffer_stream;
+ buffer_stream.write(reinterpret_cast<const uint8_t*>(msg_queue.front().contents->payload), gsl::narrow<size_t>(msg_queue.front().contents->payloadlen));
+ auto new_records_result = record_set_reader_->read(buffer_stream);
+ if (!new_records_result) {
+ logger_->log_error("Failed to read records from MQTT message: {}", new_records_result.error());
+ msg_queue.pop();
+ continue;
+ }
+ auto& new_records = new_records_result.value();
+ addAttributesAsRecordFields(new_records, msg_queue);
+ record_set.reserve(record_set.size() + new_records.size());
+ record_set.insert(record_set.end(), std::make_move_iterator(new_records.begin()), std::make_move_iterator(new_records.end()));
+ msg_queue.pop();
+ }
+ if (record_set.empty()) {
+ logger_->log_debug("No records to write, skipping FlowFile creation");
+ return;
+ }
+ std::shared_ptr<core::FlowFile> flow_file = session.create();
+ record_set_writer_->write(record_set, flow_file, session);
+ session.putAttribute(*flow_file, RecordCountOutputAttribute.name, std::to_string(record_set.size()));
+ session.putAttribute(*flow_file, BrokerOutputAttribute.name, uri_);
+ session.transfer(flow_file, Success);
Review Comment:
In this case multiple records can be written to the flow file with each record being a separate MQTT message while in the `tranferMessagesAsFlowFiles` every flow file is a single MQTT message. Those attributes are MQTT message specific, so they cannot be added to the flow file if it contains multiple MQTT messages. For this scenario we can set the `Add Attributes As Fields` property to set these attributes as record fields for each record in the flow file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2254146463
##########
extensions/mqtt/processors/ConsumeMQTT.cpp:
##########
@@ -26,17 +25,44 @@
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
#include "core/Resource.h"
+#include "io/BufferStream.h"
+#include "utils/ProcessorConfigUtils.h"
#include "utils/StringUtils.h"
#include "utils/ValueParser.h"
-#include "utils/ProcessorConfigUtils.h"
namespace org::apache::nifi::minifi::processors {
+namespace {
+template<typename RecordSetIO>
+std::shared_ptr<RecordSetIO> getRecordSetIO(core::ProcessContext& context, const core::PropertyReference& property,
+ const utils::Identifier& processor_uuid) {
+ std::string service_name = context.getProperty(property).value_or("");
+ if (!IsNullOrEmpty(service_name)) {
+ auto record_set_io = std::dynamic_pointer_cast<RecordSetIO>(context.getControllerService(service_name, processor_uuid));
+ if (!record_set_io) {
+ throw Exception(ExceptionType::PROCESS_SCHEDULE_EXCEPTION, fmt::format("'{}' property is set to invalid controller service '{}'", property.name, service_name));
+ }
+ return record_set_io;
+ }
+ return nullptr;
+}
Review Comment:
Good point. Another option would be to throw on invalid and return either nullopt or nullptr if the property is not set. Let's discuss it offline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2245787019
##########
docker/test/integration/cluster/checkers/sparkplug_b_pb2.py:
##########
Review Comment:
Could we generate this when building (or before running) the tests instead of having the generated code in the repo?
##########
docker/test/integration/features/steps/steps.py:
##########
@@ -435,6 +436,18 @@ def step_impl(context):
context.test.start('mqtt-broker')
+@given("a SparkplugBReader controller service is set up")
+def step_impl(context):
+ json_record_set_reader = SparkplugBReader("SparkplugBReader")
+ container = context.test.acquire_container(context=context, name="minifi-cpp-flow")
+ container.add_controller(json_record_set_reader)
Review Comment:
nitpicking, but this is not JSON:
```suggestion
record_set_reader = SparkplugBReader("SparkplugBReader")
container = context.test.acquire_container(context=context, name="minifi-cpp-flow")
container.add_controller(record_set_reader)
```
##########
extensions/mqtt/processors/ConsumeMQTT.cpp:
##########
@@ -58,8 +84,55 @@ void ConsumeMQTT::readProperties(core::ProcessContext& context) {
receive_maximum_ = gsl::narrow<uint16_t>(utils::parseU64Property(context, ReceiveMaximum));
}
-void ConsumeMQTT::onTriggerImpl(core::ProcessContext&, core::ProcessSession& session) {
- std::queue<SmartMessage> msg_queue = getReceivedMqttMessages();
+void ConsumeMQTT::addAttributesAsRecordFields(core::RecordSet& new_records, const std::queue<SmartMessage>& msg_queue) const {
+ if (!add_attributes_as_fields_) {
+ return;
+ }
+
+ for (auto& record: new_records) {
+ record.emplace("_topic", core::RecordField(msg_queue.front().topic));
+ auto topic_segments = utils::string::split(msg_queue.front().topic, "/");
+ for (size_t i = 0; i < topic_segments.size(); ++i) {
+ record.emplace("_topic.segment." + std::to_string(i), core::RecordField(topic_segments[i]));
+ }
+ record.emplace("_qos", core::RecordField(msg_queue.front().contents->qos));
+ record.emplace("_isDuplicate", core::RecordField(msg_queue.front().contents->dup > 0));
+ record.emplace("_isRetained", core::RecordField(msg_queue.front().contents->retained > 0));
+ }
+}
+
+void ConsumeMQTT::transferMessagesAsRecords(core::ProcessSession& session) {
+ gsl_Expects(record_set_reader_ && record_set_writer_);
+ auto msg_queue = getReceivedMqttMessages();
+ core::RecordSet record_set;
+ while (!msg_queue.empty()) {
+ io::BufferStream buffer_stream;
+ buffer_stream.write(reinterpret_cast<const uint8_t*>(msg_queue.front().contents->payload), gsl::narrow<size_t>(msg_queue.front().contents->payloadlen));
+ auto new_records_result = record_set_reader_->read(buffer_stream);
+ if (!new_records_result) {
+ logger_->log_error("Failed to read records from MQTT message: {}", new_records_result.error());
+ msg_queue.pop();
+ continue;
+ }
+ auto& new_records = new_records_result.value();
+ addAttributesAsRecordFields(new_records, msg_queue);
+ record_set.reserve(record_set.size() + new_records.size());
+ record_set.insert(record_set.end(), std::make_move_iterator(new_records.begin()), std::make_move_iterator(new_records.end()));
+ msg_queue.pop();
+ }
+ if (record_set.empty()) {
+ logger_->log_debug("No records to write, skipping FlowFile creation");
+ return;
+ }
+ std::shared_ptr<core::FlowFile> flow_file = session.create();
+ record_set_writer_->write(record_set, flow_file, session);
+ session.putAttribute(*flow_file, RecordCountOutputAttribute.name, std::to_string(record_set.size()));
+ session.putAttribute(*flow_file, BrokerOutputAttribute.name, uri_);
+ session.transfer(flow_file, Success);
Review Comment:
In `tranferMessagesAsFlowFiles`, we add a bunch of flow file attributes using `putUserPropertiesAsAttributes` and `fillAttributeFromContentType`. Are those not needed / not useful here?
##########
extensions/mqtt/processors/ConsumeMQTT.h:
##########
@@ -92,15 +107,26 @@ class ConsumeMQTT : public processors::AbstractMQTTProcessor {
QueueBufferMaxMessage,
AttributeFromContentType,
TopicAliasMaximum,
- ReceiveMaximum
+ ReceiveMaximum,
+ RecordReader,
+ RecordWriter,
+ AddAttributesAsFields
}), AbstractMQTTProcessor::AdvancedProperties);
EXTENSIONAPI static constexpr auto Success = core::RelationshipDefinition{"success", "FlowFiles that are sent successfully to the destination are transferred to this relationship"};
EXTENSIONAPI static constexpr auto Relationships = std::array{Success};
EXTENSIONAPI static constexpr auto BrokerOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.broker", {}, "URI of the sending broker"};
EXTENSIONAPI static constexpr auto TopicOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.topic", {}, "Topic of the message"};
- EXTENSIONAPI static constexpr auto OutputAttributes = std::array<core::OutputAttributeReference, 2>{BrokerOutputAttribute, TopicOutputAttribute};
+ EXTENSIONAPI static constexpr auto TopicSegmentOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.topic.segment.n", {}, "The nth topic segment of the message"};
+ EXTENSIONAPI static constexpr auto QosOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.qos", {}, "The quality of service for this message."};
+ EXTENSIONAPI static constexpr auto IsDuplicateOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.isDuplicate", {},
+ "Whether or not this message might be a duplicate of one which has already been received."};
+ EXTENSIONAPI static constexpr auto IsRetainedOutputAttribute = core::OutputAttributeDefinition<0>{"mqtt.isRetained", {},
+ "Whether or not this message was from a current publisher, or was \"retained\" by the server as the last message published on the topic."};
+ EXTENSIONAPI static constexpr auto RecordCountOutputAttribute = core::OutputAttributeDefinition<0>{"record.count", {}, "The number of records received"};
+ EXTENSIONAPI static constexpr auto OutputAttributes = std::array<core::OutputAttributeReference, 7>{BrokerOutputAttribute, TopicOutputAttribute, TopicSegmentOutputAttribute,
+ QosOutputAttribute, IsDuplicateOutputAttribute, IsRetainedOutputAttribute, RecordCountOutputAttribute};
Review Comment:
```suggestion
EXTENSIONAPI static constexpr auto OutputAttributes = std::to_array<core::OutputAttributeReference>({BrokerOutputAttribute, TopicOutputAttribute, TopicSegmentOutputAttribute,
QosOutputAttribute, IsDuplicateOutputAttribute, IsRetainedOutputAttribute, RecordCountOutputAttribute});
```
would be marginally nicer
##########
extensions/mqtt/processors/ConsumeMQTT.cpp:
##########
@@ -26,17 +25,44 @@
#include "core/ProcessContext.h"
#include "core/ProcessSession.h"
#include "core/Resource.h"
+#include "io/BufferStream.h"
+#include "utils/ProcessorConfigUtils.h"
#include "utils/StringUtils.h"
#include "utils/ValueParser.h"
-#include "utils/ProcessorConfigUtils.h"
namespace org::apache::nifi::minifi::processors {
+namespace {
+template<typename RecordSetIO>
+std::shared_ptr<RecordSetIO> getRecordSetIO(core::ProcessContext& context, const core::PropertyReference& property,
+ const utils::Identifier& processor_uuid) {
+ std::string service_name = context.getProperty(property).value_or("");
+ if (!IsNullOrEmpty(service_name)) {
+ auto record_set_io = std::dynamic_pointer_cast<RecordSetIO>(context.getControllerService(service_name, processor_uuid));
+ if (!record_set_io) {
+ throw Exception(ExceptionType::PROCESS_SCHEDULE_EXCEPTION, fmt::format("'{}' property is set to invalid controller service '{}'", property.name, service_name));
+ }
+ return record_set_io;
+ }
+ return nullptr;
+}
Review Comment:
we could use `utils::parseOptionalControllerService()` instead of this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "fgerlits (via GitHub)" <gi...@apache.org>.
fgerlits commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2254335901
##########
docker/test/integration/cluster/checkers/sparkplug_b_pb2.py:
##########
Review Comment:
OK, makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez closed pull request #1990: MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2254293299
##########
PROCESSORS.md:
##########
@@ -433,10 +436,15 @@ In the list below, the names of required properties appear in bold. Any other pr
### Output Attributes
-| Attribute | Relationship | Description |
-|-------------|--------------|---------------------------|
-| mqtt.broker | | URI of the sending broker |
-| mqtt.topic | | Topic of the message |
+| Attribute | Relationship | Description |
+|----------------------|--------------|---------------------------------------------------------------------------------------------------------------------------------------|
+| mqtt.broker | | URI of the sending broker |
+| mqtt.topic | | Topic of the message |
+| mqtt.topic.segment.n | | The nth topic segment of the message |
+| mqtt.qos | | The quality of service for this message. |
+| mqtt.isDuplicate | | Whether or not this message might be a duplicate of one which has already been received. |
+| mqtt.isRetained | | Whether or not this message was from a current publisher, or was "retained" by the server as the last message published on the topic. |
Review Comment:
Added info on this in the description in https://github.com/apache/nifi-minifi-cpp/pull/1990/commits/7e2cad557da32447caada0536e44ada143104195
##########
docker/test/integration/features/steps/steps.py:
##########
@@ -435,6 +436,18 @@ def step_impl(context):
context.test.start('mqtt-broker')
+@given("a SparkplugBReader controller service is set up")
+def step_impl(context):
+ json_record_set_reader = SparkplugBReader("SparkplugBReader")
+ container = context.test.acquire_container(context=context, name="minifi-cpp-flow")
+ container.add_controller(json_record_set_reader)
Review Comment:
Updated in https://github.com/apache/nifi-minifi-cpp/pull/1990/commits/7e2cad557da32447caada0536e44ada143104195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINIFICPP-2587 Add Sparkplug B support for ConsumeMQTT processor [nifi-minifi-cpp]
Posted by "lordgamez (via GitHub)" <gi...@apache.org>.
lordgamez commented on code in PR #1990:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1990#discussion_r2254292844
##########
CONTROLLERS.md:
##########
@@ -277,6 +278,20 @@ In the list below, the names of required properties appear in bold. Any other pr
| Password | | | The password used for authentication. Required if Username is set.<br/>**Sensitive Property: true** |
+## SparkplugReader
+
+### Description
+
+Reads Sparkplug B messages and turns them into individual Record objects. The reader expects a single Sparkplug B payload in a read operation, which is a protobuf-encoded binary message. This reader is typically used with MQTT processors like ConsumeMQTT.
Review Comment:
Updated in https://github.com/apache/nifi-minifi-cpp/pull/1990/commits/7e2cad557da32447caada0536e44ada143104195
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org