You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nifi.apache.org by lo...@apache.org on 2024/11/19 09:09:22 UTC
(nifi-minifi-cpp) branch MINIFICPP-2469 updated (65f320fe8 -> d15c53f64)
This is an automated email from the ASF dual-hosted git repository.
lordgamez pushed a change to branch MINIFICPP-2469
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
omit 65f320fe8 Add test logs
omit 07561ed03 Add extenstion docs and refactor docker tests
omit beee9a85c MINIFICPP-2469 Create GetCouchbaseKey processor
omit 564c4319e Fix test on Windows
omit c61f02727 Fix test after rebase
omit b2b2b6c29 Update extensions in README.md
omit 4f7229f23 Refactor tests
omit c1474222e Review update
omit d7301b238 Add Document Type property
omit 85705da00 Refactor
omit 1582d5dd2 Remove connection state management
omit 82686fd2a Fix clang issue and revise temporary failures
omit 3383cfb01 Review update
omit ae46b9c34 Handle temporary errors on connection initialization
omit b423b20ed MINIFICPP-2467 MINIFICPP-2468 PutCouchbaseKey processor and controller
add 9bcf7e494 MINIFICPP-2486 Simplify code around https
add 58dc93251 MINIFICPP-2477 Valgrind CI
add 02d4e630e MINIFICPP-2487 - Destroy PyException while holding GIL
add 6525dfe28 MINIFICPP-2377 Support process group level controller services
add 71a4239de MINIFICPP-2378 Add support for parameters in controller services
add 153c16f22 MINIFICPP-2279 Add support for RecordTransform NiFi python processors
add 8f53fbf6c MINIFICPP-2454 Create SplitRecord processor
add 6a2cc7f34 MINIFICPP-2488 Group metrics in Prometheus metric families by name
add a8f18ee6e MINIFICPP-2489 Send reason for rejecting flow update
add f21e3f23b MINIFICPP-2467 MINIFICPP-2468 PutCouchbaseKey processor and controller
add ef69c3d16 Handle temporary errors on connection initialization
add 25c417077 Review update
add 2fb968c16 Fix clang issue and revise temporary failures
add 24a1404af Remove connection state management
add 4ff422366 Refactor
add 8c532b977 Add Document Type property
add 66779e1cd Review update
add b2258155c Refactor tests
add 064fe6e30 Update extensions in README.md
add 60fe46781 Fix test after rebase
add 6fd33f7d7 Fix test on Windows
add c0a2e3ad0 Review update
add efb192fc5 Review update
add 30fea42cf Fix after rebase
new d6f4c0b57 MINIFICPP-2469 Create GetCouchbaseKey processor
new 76cc382a6 Add extenstion docs and refactor docker tests
new 0c23c85a8 Add test logs
new d15c53f64 Fix after rebase
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (65f320fe8)
\
N -- N -- N refs/heads/MINIFICPP-2469 (d15c53f64)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/memcheck_ci.yml | 54 +++++++
CMakeLists.txt | 4 +
PROCESSORS.md | 49 +++++-
Windows.md | 2 +-
cmake/Prometheus.cmake | 4 +-
controller/tests/ControllerTests.cpp | 10 +-
.../test/integration/cluster/DockerTestCluster.py | 3 +
docker/test/integration/cluster/ImageStore.py | 2 +
.../cluster/checkers/PrometheusChecker.py | 16 ++
.../cluster/containers/MinifiContainer.py | 5 +
.../features/MiNiFi_integration_test_driver.py | 3 +
.../integration/features/fetch_modbus_tcp.feature | 3 +-
.../test/integration/features/prometheus.feature | 1 +
docker/test/integration/features/python.feature | 24 +++
docker/test/integration/features/steps/steps.py | 25 ++-
.../JsonRecordSetReader.py} | 7 +-
.../minifi/controllers/JsonRecordSetWriter.py | 4 +-
.../{RotatingForwarder.py => SetRecordField.py} | 6 +-
.../integration/resources/python/SetRecordField.py | 57 +++++++
extensions/aws/processors/DeleteS3Object.h | 4 +-
extensions/aws/processors/FetchS3Object.h | 4 +-
extensions/aws/processors/PutS3Object.h | 4 +-
extensions/aws/processors/S3Processor.cpp | 4 +-
extensions/aws/tests/CMakeLists.txt | 1 +
extensions/aws/tests/PutS3ObjectTests.cpp | 2 +-
extensions/aws/tests/S3TestsFixture.h | 23 ++-
.../azure/processors/AzureStorageProcessorBase.cpp | 2 +-
.../azure/tests/AzureBlobStorageTestsFixture.h | 15 +-
.../azure/tests/AzureDataLakeStorageTestsFixture.h | 15 +-
extensions/azure/tests/CMakeLists.txt | 1 +
.../azure/tests/ListAzureBlobStorageTests.cpp | 7 +-
.../azure/tests/ListAzureDataLakeStorageTests.cpp | 7 +-
extensions/bustache/tests/ApplyTemplateTests.cpp | 8 +-
extensions/bustache/tests/CMakeLists.txt | 1 +
extensions/civetweb/tests/CMakeLists.txt | 1 +
extensions/civetweb/tests/ListenHTTPTests.cpp | 14 +-
.../controllerservices/CouchbaseClusterService.cpp | 23 ++-
.../controllerservices/CouchbaseClusterService.h | 23 ++-
.../couchbase/tests/GetCouchbaseKeyTests.cpp | 8 +-
.../couchbase/tests/PutCouchbaseKeyTests.cpp | 8 +-
extensions/elasticsearch/PostElasticsearch.cpp | 10 +-
extensions/elasticsearch/PostElasticsearch.h | 2 +
extensions/elasticsearch/tests/CMakeLists.txt | 1 +
.../elasticsearch/tests/PostElasticsearchTests.cpp | 5 +-
extensions/execute-process/test/CMakeLists.txt | 2 +
.../execute-process/test/ExecuteProcessTests.cpp | 6 +-
.../expression-language/tests/CMakeLists.txt | 1 +
.../tests/ProcessContextExprTests.cpp | 2 +-
extensions/gcp/processors/GCSProcessor.cpp | 6 +-
extensions/gcp/processors/GCSProcessor.h | 1 +
extensions/gcp/tests/DeleteGCSObjectTests.cpp | 6 +-
extensions/gcp/tests/FetchGCSObjectTests.cpp | 5 +-
extensions/gcp/tests/ListGCSBucketTests.cpp | 5 +-
extensions/gcp/tests/PutGCSObjectTests.cpp | 5 +-
extensions/grafana-loki/PushGrafanaLoki.cpp | 4 +-
extensions/grafana-loki/PushGrafanaLoki.h | 2 +-
extensions/grafana-loki/tests/CMakeLists.txt | 1 +
.../grafana-loki/tests/PushGrafanaLokiGrpcTest.cpp | 18 +--
.../grafana-loki/tests/PushGrafanaLokiRESTTest.cpp | 23 +--
.../processors/CollectKubernetesPodMetrics.cpp | 2 +-
extensions/kubernetes/tests/CMakeLists.txt | 1 +
extensions/libarchive/tests/FocusArchiveTests.cpp | 14 +-
.../libarchive/tests/ManipulateArchiveTests.cpp | 8 +-
extensions/libarchive/tests/MergeFileTests.cpp | 10 +-
extensions/librdkafka/tests/CMakeLists.txt | 1 +
extensions/librdkafka/tests/PublishKafkaTests.cpp | 4 +-
extensions/lua/tests/CMakeLists.txt | 1 +
.../TestExecuteScriptProcessorWithLuaScript.cpp | 54 +++----
extensions/mqtt/tests/CMakeLists.txt | 1 +
extensions/mqtt/tests/ConsumeMQTTTests.cpp | 2 +-
extensions/mqtt/tests/PublishMQTTTests.cpp | 2 +-
extensions/opencv/tests/CMakeLists.txt | 1 +
extensions/opencv/tests/CaptureRTSPFrameTest.cpp | 4 +-
extensions/pdh/tests/CMakeLists.txt | 1 +
.../pdh/tests/PerformanceDataMonitorTests.cpp | 4 +-
extensions/procfs/tests/CMakeLists.txt | 1 +
extensions/procfs/tests/ProcFsMonitorTests.cpp | 4 +-
.../prometheus/PrometheusMetricsPublisher.cpp | 25 ++-
extensions/prometheus/PrometheusMetricsPublisher.h | 4 +-
.../prometheus/PublishedMetricGaugeCollection.cpp | 36 +++--
.../prometheus/PublishedMetricGaugeCollection.h | 6 +-
extensions/prometheus/tests/CMakeLists.txt | 1 +
.../tests/PrometheusMetricsPublisherTest.cpp | 30 ++--
extensions/python/PYTHON.md | 4 +-
extensions/python/PythonBindings.cpp | 6 +-
extensions/python/PythonScriptEngine.h | 4 +-
.../pythonprocessors/nifiapi/flowfilesource.py | 6 -
.../pythonprocessors/nifiapi/flowfiletransform.py | 6 -
.../pythonprocessors/nifiapi/recordtransform.py | 152 ++++++++++++++++++
.../python/tests/ExecutePythonProcessorTests.cpp | 10 +-
.../TestExecuteScriptProcessorWithPythonScript.cpp | 54 +++----
extensions/python/types/PyProcessContext.cpp | 10 +-
extensions/python/types/PyProcessContext.h | 2 +
extensions/python/types/PyProcessSession.cpp | 36 +++--
extensions/python/types/PyProcessSession.h | 3 +-
extensions/python/types/PyRecordSetReader.cpp | 112 +++++++++++++
.../{PyOutputStream.h => PyRecordSetReader.h} | 20 ++-
extensions/python/types/PyRecordSetWriter.cpp | 114 +++++++++++++
.../{PyOutputStream.h => PyRecordSetWriter.h} | 20 ++-
extensions/python/types/Types.h | 9 ++
extensions/script/ExecuteScript.h | 3 +
extensions/sftp/tests/FetchSFTPTests.cpp | 8 +-
extensions/sftp/tests/ListSFTPTests.cpp | 8 +-
extensions/sftp/tests/ListThenFetchSFTPTests.cpp | 8 +-
extensions/sftp/tests/PutSFTPTests.cpp | 4 +-
extensions/smb/SmbConnectionControllerService.cpp | 2 +-
extensions/smb/tests/FetchSmbTests.cpp | 8 +-
extensions/smb/tests/ListAndFetchSmbTests.cpp | 8 +-
extensions/smb/tests/ListSmbTests.cpp | 8 +-
extensions/smb/tests/PutSmbTests.cpp | 12 +-
extensions/splunk/SplunkHECProcessor.cpp | 4 +-
extensions/splunk/SplunkHECProcessor.h | 2 +-
extensions/splunk/tests/PutSplunkHTTPTests.cpp | 14 +-
.../tests/QuerySplunkIndexingStatusTests.cpp | 14 +-
extensions/sql/processors/SQLProcessor.cpp | 2 +-
extensions/sql/tests/PutSQLTests.cpp | 1 -
extensions/sql/tests/SQLTestPlan.h | 6 +-
.../standard-processors/modbus/FetchModbusTcp.cpp | 4 +-
.../standard-processors/processors/GetTCP.cpp | 6 +-
extensions/standard-processors/processors/GetTCP.h | 2 +-
.../standard-processors/processors/InvokeHTTP.cpp | 3 +-
.../standard-processors/processors/PutTCP.cpp | 4 +-
.../processors/SegmentContent.h | 3 +-
.../standard-processors/processors/SplitRecord.cpp | 119 ++++++++++++++
.../standard-processors/processors/SplitRecord.h | 96 +++++++++++
.../standard-processors/processors/TailFile.cpp | 2 +-
.../tests/integration/InvokeHTTPTests.cpp | 54 +++----
.../tests/unit/AppendHostInfoTests.cpp | 18 +--
.../tests/unit/AttributeRollingWindowTests.cpp | 4 +-
.../tests/unit/AttributesToJSONTests.cpp | 10 +-
.../tests/unit/DefragmentTextTests.cpp | 32 ++--
.../tests/unit/ExtractTextTests.cpp | 16 +-
.../tests/unit/FetchFileTests.cpp | 6 +-
.../tests/unit/FlowJsonTests.cpp | 137 +++++++++++++++-
.../tests/unit/GenerateFlowFileTests.cpp | 25 +--
.../tests/unit/GetFileTests.cpp | 10 +-
.../standard-processors/tests/unit/GetTCPTests.cpp | 16 +-
.../tests/unit/HashContentTest.cpp | 21 ++-
.../tests/unit/JoltTransformJSONTests.cpp | 28 ++--
.../tests/unit/ListFileTests.cpp | 10 +-
.../tests/unit/ListenSyslogTests.cpp | 23 ++-
.../tests/unit/ListenTcpTests.cpp | 24 +--
.../tests/unit/ListenUDPTests.cpp | 12 +-
.../tests/unit/LogAttributeTests.cpp | 16 +-
.../tests/unit/ProcessorTests.cpp | 52 +++---
.../tests/unit/PutFileTests.cpp | 34 ++--
.../standard-processors/tests/unit/PutTCPTests.cpp | 8 +-
.../standard-processors/tests/unit/PutUDPTests.cpp | 4 +-
.../tests/unit/RecordSetTests.cpp | 177 +++++++++++++++++++++
.../tests/unit/ReplaceTextTests.cpp | 24 +--
.../tests/unit/RetryFlowFileTests.cpp | 14 +-
.../tests/unit/RouteTextTests.cpp | 4 +-
.../tests/unit/SegmentContentTests.cpp | 31 ++--
.../tests/unit/SplitContentTests.cpp | 96 +++++------
.../tests/unit/SplitRecordTests.cpp | 96 +++++++++++
.../tests/unit/SplitTextTests.cpp | 47 +++---
.../tests/unit/TailFileTests.cpp | 70 ++++----
.../tests/unit/YamlConfigurationTests.cpp | 123 +++++++++++++-
extensions/systemd/tests/ConsumeJournaldTest.cpp | 2 +-
libminifi/include/FlowController.h | 5 +-
libminifi/include/core/ProcessContext.h | 44 +----
libminifi/include/core/ProcessGroup.h | 6 +-
libminifi/include/core/Record.h | 22 ++-
libminifi/include/core/RecordField.h | 3 +
.../core/controller/ControllerServiceLookup.h | 11 +-
.../core/controller/ControllerServiceNodeMap.h | 50 ++----
.../core/controller/ControllerServiceProvider.h | 7 +-
.../ForwardingControllerServiceProvider.h | 8 +
.../include/core/flow/StructuredConfiguration.h | 2 +-
libminifi/include/core/state/UpdateController.h | 14 +-
libminifi/include/http/BaseHTTPClient.h | 3 -
libminifi/include/http/HTTPClient.h | 3 -
libminifi/src/FlowController.cpp | 24 +--
libminifi/src/RemoteProcessorGroupPort.cpp | 2 +-
libminifi/src/c2/C2Agent.cpp | 8 +-
libminifi/src/core/ProcessGroup.cpp | 64 +++++---
libminifi/src/core/RecordField.cpp | 99 ++++++++++++
.../core/controller/ControllerServiceNodeMap.cpp | 96 +++++++++++
.../core/controller/ControllerServiceProvider.cpp | 12 +-
.../controller/StandardControllerServiceNode.cpp | 2 +-
.../src/core/flow/StructuredConfiguration.cpp | 24 ++-
libminifi/src/http/HTTPClient.cpp | 15 --
libminifi/src/utils/net/Ssl.cpp | 2 +-
libminifi/test/flow-tests/FlowControllerTests.cpp | 2 +-
.../ControllerServiceIntegrationTests.cpp | 15 +-
.../libtest/unit/ContentRepositoryDependentTests.h | 2 +-
libminifi/test/libtest/unit/MockClasses.h | 42 +++--
libminifi/test/libtest/unit/RecordSetTesters.h | 2 +-
.../libtest/unit/SingleProcessorTestController.h | 37 ++---
libminifi/test/libtest/unit/TestBase.cpp | 82 ++++++----
libminifi/test/libtest/unit/TestBase.h | 46 +++---
libminifi/test/libtest/unit/TestUtils.h | 2 +-
.../test/resources/TestControllerServices.yml | 164 ++++++++++++-------
libminifi/test/unit/ProcessSessionTests.cpp | 2 +-
194 files changed, 2758 insertions(+), 1103 deletions(-)
create mode 100644 .github/workflows/memcheck_ci.yml
copy docker/test/integration/minifi/{core/Funnel.py => controllers/JsonRecordSetReader.py} (79%)
copy docker/test/integration/minifi/processors/{RotatingForwarder.py => SetRecordField.py} (89%)
create mode 100644 docker/test/integration/resources/python/SetRecordField.py
create mode 100644 extensions/python/pythonprocessors/nifiapi/recordtransform.py
create mode 100644 extensions/python/types/PyRecordSetReader.cpp
copy extensions/python/types/{PyOutputStream.h => PyRecordSetReader.h} (68%)
create mode 100644 extensions/python/types/PyRecordSetWriter.cpp
copy extensions/python/types/{PyOutputStream.h => PyRecordSetWriter.h} (68%)
create mode 100644 extensions/standard-processors/processors/SplitRecord.cpp
create mode 100644 extensions/standard-processors/processors/SplitRecord.h
create mode 100644 extensions/standard-processors/tests/unit/RecordSetTests.cpp
create mode 100644 extensions/standard-processors/tests/unit/SplitRecordTests.cpp
create mode 100644 libminifi/src/core/RecordField.cpp
create mode 100644 libminifi/src/core/controller/ControllerServiceNodeMap.cpp
(nifi-minifi-cpp) 03/04: Add test logs
Posted by lo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lordgamez pushed a commit to branch MINIFICPP-2469
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 0c23c85a8a1e18356f4ebba3267a8e2ed977574d
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Tue Oct 22 14:11:18 2024 +0200
Add test logs
---
extensions/couchbase/tests/GetCouchbaseKeyTests.cpp | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp b/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp
index 69f84aa06..d518ace03 100644
--- a/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp
+++ b/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp
@@ -38,6 +38,11 @@ struct ExpectedCallOptions {
class GetCouchbaseKeyTestController : public TestController {
public:
GetCouchbaseKeyTestController() {
+ LogTestController::getInstance().setDebug<TestPlan>();
+ LogTestController::getInstance().setDebug<minifi::core::Processor>();
+ LogTestController::getInstance().setTrace<minifi::core::ProcessSession>();
+ LogTestController::getInstance().setDebug<controllers::CouchbaseClusterService>();
+ LogTestController::getInstance().setDebug<processors::GetCouchbaseKey>();
auto controller_service_node = controller_.plan->addController("MockCouchbaseClusterService", "MockCouchbaseClusterService");
mock_couchbase_cluster_service_ = std::static_pointer_cast<MockCouchbaseClusterService>(controller_service_node->getControllerServiceImplementation());
proc_->setProperty(processors::GetCouchbaseKey::CouchbaseClusterControllerService, "MockCouchbaseClusterService");
(nifi-minifi-cpp) 01/04: MINIFICPP-2469 Create GetCouchbaseKey processor
Posted by lo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lordgamez pushed a commit to branch MINIFICPP-2469
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit d6f4c0b578d2edd43b9ba7317ff4a31614cfb191
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Tue Oct 8 16:54:16 2024 +0200
MINIFICPP-2469 Create GetCouchbaseKey processor
---
PROCESSORS.md | 40 +++++
.../cluster/containers/CouchbaseServerContainer.py | 4 +-
docker/test/integration/features/couchbase.feature | 108 ++++++++++++++
.../{PutCouchbaseKey.py => GetCouchbaseKey.py} | 8 +-
.../minifi/processors/PutCouchbaseKey.py | 2 +-
.../controllerservices/CouchbaseClusterService.cpp | 62 ++++++--
.../controllerservices/CouchbaseClusterService.h | 17 ++-
.../couchbase/processors/GetCouchbaseKey.cpp | 111 ++++++++++++++
.../{PutCouchbaseKey.h => GetCouchbaseKey.h} | 93 +++---------
extensions/couchbase/processors/PutCouchbaseKey.h | 1 -
.../couchbase/tests/GetCouchbaseKeyTests.cpp | 163 +++++++++++++++++++++
.../couchbase/tests/MockCouchbaseClusterService.h | 35 ++++-
libminifi/include/core/Processor.h | 4 +-
13 files changed, 555 insertions(+), 93 deletions(-)
diff --git a/PROCESSORS.md b/PROCESSORS.md
index eec26bbac..667130ab4 100644
--- a/PROCESSORS.md
+++ b/PROCESSORS.md
@@ -48,6 +48,7 @@ limitations under the License.
- [FetchSmb](#FetchSmb)
- [FocusArchiveEntry](#FocusArchiveEntry)
- [GenerateFlowFile](#GenerateFlowFile)
+- [GetCouchbaseKey](#GetCouchbaseKey)
- [GetFile](#GetFile)
- [GetTCP](#GetTCP)
- [HashContent](#HashContent)
@@ -1096,6 +1097,45 @@ In the list below, the names of required properties appear in bold. Any other pr
| success | success operational on the flow record |
+## GetCouchbaseKey
+
+### Description
+
+Get a document from Couchbase Server via Key/Value access. The ID of the document to fetch may be supplied by setting the <Document Id> property. NOTE: if the Document Id property is not set, the contents of the FlowFile will be read to determine the Document Id, which means that the contents of the entire FlowFile will be buffered in memory.
+
+### Properties
+
+In the list below, the names of required properties appear in bold. Any other properties (not in bold) are considered optional. The table also indicates any default values, and whether a property supports the NiFi Expression Language.
+
+| Name | Default Value | Allowable Values | Description |
+|------------------------------------------|---------------|----------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **Couchbase Cluster Controller Service** | | | A Couchbase Cluster Controller Service which manages connections to a Couchbase cluster. |
+| **Bucket Name** | default | | The name of bucket to access.<br/>**Supports Expression Language: true** |
+| Scope Name | | | Scope to use inside the bucket. If not specified, the _default scope is used.<br/>**Supports Expression Language: true** |
+| Collection Name | | | Collection to use inside the bucket scope. If not specified, the _default collection is used.<br/>**Supports Expression Language: true** |
+| **Document Type** | Json | Json<br/>Binary<br/>String | Content type of the retrieved value. |
+| Document Id | | | A static, fixed Couchbase document id, or an expression to construct the Couchbase document id.<br/>**Supports Expression Language: true** |
+| Put Value to Attribute | | | If set, the retrieved value will be put into an attribute of the FlowFile instead of a the content of the FlowFile. The attribute key to put to is determined by evaluating value of this property.<br/>**Supports Expression Language: true** |
+
+### Relationships
+
+| Name | Description |
+|----------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| success | Values retrieved from Couchbase Server are written as outgoing FlowFiles content or put into an attribute of the incoming FlowFile and routed to this relationship. |
+| failure | All FlowFiles failed to fetch from Couchbase Server and not retry-able are routed to this relationship. |
+| retry | All FlowFiles failed to fetch from Couchbase Server but can be retried are routed to this relationship. |
+| original | The original input FlowFile is routed to this relationship when the value is retrieved from Couchbase Server and routed to 'success'. |
+
+### Output Attributes
+
+| Attribute | Relationship | Description |
+|----------------------|--------------|---------------------------------------|
+| couchbase.bucket | success | Bucket where the document was stored. |
+| couchbase.doc.id | success | Id of the document. |
+| couchbase.doc.cas | success | CAS of the document. |
+| couchbase.doc.expiry | success | Expiration of the document. |
+
+
## GetFile
### Description
diff --git a/docker/test/integration/cluster/containers/CouchbaseServerContainer.py b/docker/test/integration/cluster/containers/CouchbaseServerContainer.py
index 1c4353046..9baf01f08 100644
--- a/docker/test/integration/cluster/containers/CouchbaseServerContainer.py
+++ b/docker/test/integration/cluster/containers/CouchbaseServerContainer.py
@@ -33,7 +33,7 @@ class CouchbaseServerContainer(Container):
["couchbase-cli", "cluster-init", "-c", "localhost", "--cluster-username", "Administrator", "--cluster-password", "password123", "--services", "data,index,query",
"--cluster-ramsize", "2048", "--cluster-index-ramsize", "256"],
["couchbase-cli", "bucket-create", "-c", "localhost", "--username", "Administrator", "--password", "password123", "--bucket", "test_bucket", "--bucket-type", "couchbase",
- "--bucket-ramsize", "1024"]
+ "--bucket-ramsize", "1024", "--max-ttl", "36000"]
]
for command in commands:
(code, _) = self.client.containers.get(self.name).exec_run(command)
@@ -47,7 +47,7 @@ class CouchbaseServerContainer(Container):
return
self.docker_container = self.client.containers.run(
- "couchbase:community-7.6.2",
+ "couchbase:enterprise-7.2.5",
detach=True,
name=self.name,
network=self.network.name,
diff --git a/docker/test/integration/features/couchbase.feature b/docker/test/integration/features/couchbase.feature
index 446821f25..d9a4bb664 100644
--- a/docker/test/integration/features/couchbase.feature
+++ b/docker/test/integration/features/couchbase.feature
@@ -65,3 +65,111 @@ Feature: Executing Couchbase operations from MiNiFi-C++
And the Minifi logs match the following regex: "key:couchbase.partition.uuid value:[1-9][0-9]*" in less than 1 seconds
And the Minifi logs match the following regex: "key:couchbase.partition.id value:[1-9][0-9]*" in less than 1 seconds
And a document with id "test_doc_id" in bucket "test_bucket" is present with data '{"field1": "value1"}' of type "Binary" in Couchbase
+
+ Scenario: A MiNiFi instance can get data from test bucket with GetCouchbaseKey processor
+ Given a Couchbase server is set up
+ And a GetFile processor with the "Input Directory" property set to "/tmp/input"
+ And a file with the content '{"field1": "value1", "field2": "value2"}' is present in '/tmp/input'
+ And a PutCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
+ And the "Document Id" property of the PutCouchbaseKey processor is set to "test_doc_id"
+ And a GetCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
+ And the "Document Id" property of the GetCouchbaseKey processor is set to "test_doc_id"
+ And a PutFile processor with the "Directory" property set to "/tmp/output"
+ And a LogAttribute processor with the "FlowFiles To Log" property set to "0"
+ And a CouchbaseClusterService is setup up for PutCouchbaseKey with the name "CouchbaseClusterService"
+ And a CouchbaseClusterService is setup up for GetCouchbaseKey with the name "CouchbaseClusterService"
+
+ And the "success" relationship of the GetFile processor is connected to the PutCouchbaseKey
+ And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey
+ And the "success" relationship of the GetCouchbaseKey processor is connected to the PutFile
+ And the "success" relationship of the PutFile processor is connected to the LogAttribute
+
+ When a Couchbase server is started
+ And all instances start up
+
+ Then a flowfile with the JSON content '{"field1": "value1", "field2": "value2"}' is placed in the monitored directory in less than 60 seconds
+ And the Minifi logs contain the following message: "key:couchbase.bucket value:test_bucket" in less than 10 seconds
+ And the Minifi logs contain the following message: "key:couchbase.doc.id value:test_doc_id" in less than 1 seconds
+ And the Minifi logs match the following regex: "key:couchbase.doc.cas value:[1-9][0-9]*" in less than 1 seconds
+ And the Minifi logs match the following regex: "key:couchbase.doc.expiry value:\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}" in less than 1 seconds
+
+ Scenario: A MiNiFi instance can get data from test bucket with GetCouchbaseKey processor using binary storage
+ Given a Couchbase server is set up
+ And a GetFile processor with the "Input Directory" property set to "/tmp/input"
+ And a file with the content '{"field1": "value1", "field2": "value2"}' is present in '/tmp/input'
+ And a PutCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
+ And the "Document Id" property of the PutCouchbaseKey processor is set to "test_doc_id"
+ And the "Document Type" property of the PutCouchbaseKey processor is set to "Binary"
+ And a GetCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
+ And the "Document Id" property of the GetCouchbaseKey processor is set to "test_doc_id"
+ And the "Document Type" property of the GetCouchbaseKey processor is set to "Binary"
+ And a PutFile processor with the "Directory" property set to "/tmp/output"
+ And a LogAttribute processor with the "FlowFiles To Log" property set to "0"
+ And a CouchbaseClusterService is setup up for PutCouchbaseKey with the name "CouchbaseClusterService"
+ And a CouchbaseClusterService is setup up for GetCouchbaseKey with the name "CouchbaseClusterService"
+
+ And the "success" relationship of the GetFile processor is connected to the PutCouchbaseKey
+ And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey
+ And the "success" relationship of the GetCouchbaseKey processor is connected to the PutFile
+ And the "success" relationship of the PutFile processor is connected to the LogAttribute
+
+ When a Couchbase server is started
+ And all instances start up
+
+ Then a flowfile with the JSON content '{"field1": "value1", "field2": "value2"}' is placed in the monitored directory in less than 60 seconds
+ And the Minifi logs contain the following message: "key:couchbase.bucket value:test_bucket" in less than 10 seconds
+ And the Minifi logs contain the following message: "key:couchbase.doc.id value:test_doc_id" in less than 1 seconds
+ And the Minifi logs match the following regex: "key:couchbase.doc.cas value:[1-9][0-9]*" in less than 1 seconds
+ And the Minifi logs match the following regex: "key:couchbase.doc.expiry value:\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}" in less than 1 seconds
+
+ Scenario: A MiNiFi instance can get data from test bucket with GetCouchbaseKey processor and put the result in an attribute
+ Given a Couchbase server is set up
+ And a GetFile processor with the "Input Directory" property set to "/tmp/input"
+ And a file with the content '{"field1": "value1", "field2": "value2"}' is present in '/tmp/input'
+ And a PutCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
+ And the "Document Id" property of the PutCouchbaseKey processor is set to "test_doc_id"
+ And the "Document Type" property of the PutCouchbaseKey processor is set to "String"
+ And a GetCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
+ And the "Document Id" property of the GetCouchbaseKey processor is set to "test_doc_id"
+ And the "Document Type" property of the GetCouchbaseKey processor is set to "String"
+ And the "Put Value to Attribute" property of the GetCouchbaseKey processor is set to "get_couchbase_result"
+ And a PutFile processor with the "Directory" property set to "/tmp/output"
+ And a LogAttribute processor with the "FlowFiles To Log" property set to "0"
+ And a CouchbaseClusterService is setup up for PutCouchbaseKey with the name "CouchbaseClusterService"
+ And a CouchbaseClusterService is setup up for GetCouchbaseKey with the name "CouchbaseClusterService"
+
+ And the "success" relationship of the GetFile processor is connected to the PutCouchbaseKey
+ And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey
+ And the "success" relationship of the GetCouchbaseKey processor is connected to the PutFile
+ And the "success" relationship of the PutFile processor is connected to the LogAttribute
+
+ When a Couchbase server is started
+ And all instances start up
+
+ Then a flowfile with the JSON content '{"field1": "value1", "field2": "value2"}' is placed in the monitored directory in less than 60 seconds
+ And the Minifi logs contain the following message: "key:couchbase.bucket value:test_bucket" in less than 10 seconds
+ And the Minifi logs contain the following message: "key:couchbase.doc.id value:test_doc_id" in less than 1 seconds
+ And the Minifi logs match the following regex: "key:couchbase.doc.cas value:[1-9][0-9]*" in less than 1 seconds
+ And the Minifi logs match the following regex: "key:couchbase.doc.expiry value:\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}" in less than 1 seconds
+ And the Minifi logs contain the following message: 'key:get_couchbase_result value:{"field1": "value1", "field2": "value2"}' in less than 1 seconds
+
+ Scenario: GetCouchbaseKey transfers FlowFile to failure relationship on Couchbase value type mismatch
+ Given a Couchbase server is set up
+ And a GetFile processor with the "Input Directory" property set to "/tmp/input"
+ And a file with the content '{"field1": "value1", "field2": "value2"}' is present in '/tmp/input'
+ And a PutCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
+ And the "Document Id" property of the PutCouchbaseKey processor is set to "test_doc_id"
+ And the "Document Type" property of the PutCouchbaseKey processor is set to "String"
+ And a GetCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
+ And the "Document Id" property of the GetCouchbaseKey processor is set to "test_doc_id"
+ And the "Document Type" property of the GetCouchbaseKey processor is set to "Binary"
+ And a CouchbaseClusterService is setup up for PutCouchbaseKey with the name "CouchbaseClusterService"
+ And a CouchbaseClusterService is setup up for GetCouchbaseKey with the name "CouchbaseClusterService"
+
+ And the "success" relationship of the GetFile processor is connected to the PutCouchbaseKey
+ And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey
+
+ When a Couchbase server is started
+ And all instances start up
+
+ Then the Minifi logs contain the following message: "Failed to get content for document 'test_doc_id' from collection 'test_bucket._default._default' with the following exception: 'raw_binary_transcoder expects document to have BINARY common flags" in less than 60 seconds
diff --git a/docker/test/integration/minifi/processors/PutCouchbaseKey.py b/docker/test/integration/minifi/processors/GetCouchbaseKey.py
similarity index 84%
copy from docker/test/integration/minifi/processors/PutCouchbaseKey.py
copy to docker/test/integration/minifi/processors/GetCouchbaseKey.py
index 5e94aaa07..0b48dcbdd 100644
--- a/docker/test/integration/minifi/processors/PutCouchbaseKey.py
+++ b/docker/test/integration/minifi/processors/GetCouchbaseKey.py
@@ -15,10 +15,10 @@
from ..core.Processor import Processor
-class PutCouchbaseKey(Processor):
+class GetCouchbaseKey(Processor):
def __init__(self, context, schedule={'scheduling strategy': 'EVENT_DRIVEN'}):
- super(PutCouchbaseKey, self).__init__(
+ super(GetCouchbaseKey, self).__init__(
context=context,
- clazz='PutCouchbaseKey',
- auto_terminate=['success', 'failure'],
+ clazz='GetCouchbaseKey',
+ auto_terminate=['success', 'failure', 'retry'],
schedule=schedule)
diff --git a/docker/test/integration/minifi/processors/PutCouchbaseKey.py b/docker/test/integration/minifi/processors/PutCouchbaseKey.py
index 5e94aaa07..341338771 100644
--- a/docker/test/integration/minifi/processors/PutCouchbaseKey.py
+++ b/docker/test/integration/minifi/processors/PutCouchbaseKey.py
@@ -20,5 +20,5 @@ class PutCouchbaseKey(Processor):
super(PutCouchbaseKey, self).__init__(
context=context,
clazz='PutCouchbaseKey',
- auto_terminate=['success', 'failure'],
+ auto_terminate=['success', 'failure', 'retry'],
schedule=schedule)
diff --git a/extensions/couchbase/controllerservices/CouchbaseClusterService.cpp b/extensions/couchbase/controllerservices/CouchbaseClusterService.cpp
index f320636fd..fd20c7a99 100644
--- a/extensions/couchbase/controllerservices/CouchbaseClusterService.cpp
+++ b/extensions/couchbase/controllerservices/CouchbaseClusterService.cpp
@@ -22,6 +22,7 @@
#include "couchbase/codec/raw_json_transcoder.hxx"
#include "core/Resource.h"
+#include "utils/TimeUtil.h"
namespace org::apache::nifi::minifi::couchbase {
@@ -86,16 +87,57 @@ nonstd::expected<CouchbaseUpsertResult, CouchbaseErrorType> CouchbaseClient::ups
document_id, collection.bucket_name, collection.scope_name, collection.collection_name, upsert_err.ec(), upsert_err.message());
return nonstd::make_unexpected(CouchbaseErrorType::FATAL);
} else {
- const uint64_t partition_uuid = (upsert_resp.mutation_token().has_value() ? upsert_resp.mutation_token()->partition_uuid() : 0);
- const uint64_t sequence_number = (upsert_resp.mutation_token().has_value() ? upsert_resp.mutation_token()->sequence_number() : 0);
- const uint16_t partition_id = (upsert_resp.mutation_token().has_value() ? upsert_resp.mutation_token()->partition_id() : 0);
- return CouchbaseUpsertResult {
- collection.bucket_name,
- upsert_resp.cas().value(),
- partition_uuid,
- sequence_number,
- partition_id
- };
+ CouchbaseUpsertResult result;
+ result.bucket_name = collection.bucket_name;
+ result.cas = upsert_resp.cas().value();
+ result.partition_uuid = (upsert_resp.mutation_token().has_value() ? upsert_resp.mutation_token()->partition_uuid() : 0);
+ result.sequence_number = (upsert_resp.mutation_token().has_value() ? upsert_resp.mutation_token()->sequence_number() : 0);
+ result.partition_id = (upsert_resp.mutation_token().has_value() ? upsert_resp.mutation_token()->partition_id() : 0);
+ return result;
+ }
+}
+
+nonstd::expected<CouchbaseGetResult, CouchbaseErrorType> CouchbaseClient::get(const CouchbaseCollection& collection,
+ const std::string& document_id, CouchbaseValueType return_type) {
+ auto collection_result = getCollection(collection);
+ if (!collection_result.has_value()) {
+ return nonstd::make_unexpected(collection_result.error());
+ }
+
+ ::couchbase::get_options options;
+ options.with_expiry(true);
+ auto [get_err, resp] = collection_result->get(document_id, options).get();
+ if (get_err.ec()) {
+ if (get_err.ec().value() == static_cast<int>(::couchbase::errc::common::unambiguous_timeout) || get_err.ec().value() == static_cast<int>(::couchbase::errc::common::ambiguous_timeout)) {
+ logger_->log_error("Failed to get document '{}' from collection '{}.{}.{}' due to timeout",
+ document_id, collection.bucket_name, collection.scope_name, collection.collection_name);
+ return nonstd::make_unexpected(CouchbaseErrorType::TEMPORARY);
+ }
+ std::string cause = get_err.cause() ? get_err.cause()->message() : "";
+ logger_->log_error("Failed to get document '{}' from collection '{}.{}.{}' with error code: '{}', message: '{}'",
+ document_id, collection.bucket_name, collection.scope_name, collection.collection_name, get_err.ec(), get_err.message());
+ return nonstd::make_unexpected(CouchbaseErrorType::FATAL);
+ } else {
+ try {
+ CouchbaseGetResult result;
+ result.bucket_name = collection.bucket_name;
+ result.cas = resp.cas().value();
+ if (return_type == CouchbaseValueType::Json) {
+ result.value = resp.content_as<::couchbase::codec::binary, ::couchbase::codec::raw_json_transcoder>();
+ } else if (return_type == CouchbaseValueType::String) {
+ result.value = resp.content_as<::couchbase::codec::raw_string_transcoder>();
+ } else {
+ result.value = resp.content_as<::couchbase::codec::raw_binary_transcoder>();
+ }
+ if (resp.expiry_time().has_value()) {
+ result.expiry = utils::timeutils::getTimeStr(*resp.expiry_time());
+ }
+ return result;
+ } catch(const std::exception& ex) {
+ logger_->log_error("Failed to get content for document '{}' from collection '{}.{}.{}' with the following exception: '{}'",
+ document_id, collection.bucket_name, collection.scope_name, collection.collection_name, ex.what());
+ return nonstd::make_unexpected(CouchbaseErrorType::FATAL);
+ }
}
}
diff --git a/extensions/couchbase/controllerservices/CouchbaseClusterService.h b/extensions/couchbase/controllerservices/CouchbaseClusterService.h
index 29fde7bde..f1174ef09 100644
--- a/extensions/couchbase/controllerservices/CouchbaseClusterService.h
+++ b/extensions/couchbase/controllerservices/CouchbaseClusterService.h
@@ -22,6 +22,7 @@
#include <string>
#include <utility>
#include <mutex>
+#include <variant>
#include "core/controller/ControllerService.h"
#include "core/PropertyDefinition.h"
@@ -39,9 +40,17 @@ struct CouchbaseCollection {
std::string collection_name;
};
-struct CouchbaseUpsertResult {
+struct CouchbaseCallResult {
std::string bucket_name;
std::uint64_t cas{0};
+};
+
+struct CouchbaseGetResult : public CouchbaseCallResult {
+ std::string expiry;
+ std::variant<std::vector<std::byte>, std::string> value;
+};
+
+struct CouchbaseUpsertResult : public CouchbaseCallResult {
std::uint64_t sequence_number{0};
std::uint64_t partition_uuid{0};
std::uint16_t partition_id{0};
@@ -75,6 +84,7 @@ class CouchbaseClient {
nonstd::expected<CouchbaseUpsertResult, CouchbaseErrorType> upsert(const CouchbaseCollection& collection, CouchbaseValueType document_type, const std::string& document_id,
const std::vector<std::byte>& buffer, const ::couchbase::upsert_options& options);
+ nonstd::expected<CouchbaseGetResult, CouchbaseErrorType> get(const CouchbaseCollection& collection, const std::string& document_id, CouchbaseValueType return_type);
nonstd::expected<void, CouchbaseErrorType> establishConnection();
void close();
@@ -151,6 +161,11 @@ class CouchbaseClusterService : public core::controller::ControllerService {
return client_->upsert(collection, document_type, document_id, buffer, options);
}
+ virtual nonstd::expected<CouchbaseGetResult, CouchbaseErrorType> get(const CouchbaseCollection& collection, const std::string& document_id, CouchbaseValueType return_type) {
+ gsl_Expects(client_);
+ return client_->get(collection, document_id, return_type);
+ }
+
static gsl::not_null<std::shared_ptr<CouchbaseClusterService>> getFromProperty(const core::ProcessContext& context, const core::PropertyReference& property);
private:
diff --git a/extensions/couchbase/processors/GetCouchbaseKey.cpp b/extensions/couchbase/processors/GetCouchbaseKey.cpp
new file mode 100644
index 000000000..89cf160f6
--- /dev/null
+++ b/extensions/couchbase/processors/GetCouchbaseKey.cpp
@@ -0,0 +1,111 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "GetCouchbaseKey.h"
+#include "utils/gsl.h"
+#include "core/Resource.h"
+#include "utils/ProcessorConfigUtils.h"
+
+namespace org::apache::nifi::minifi::couchbase::processors {
+
+void GetCouchbaseKey::onSchedule(core::ProcessContext& context, core::ProcessSessionFactory&) {
+ couchbase_cluster_service_ = controllers::CouchbaseClusterService::getFromProperty(context, GetCouchbaseKey::CouchbaseClusterControllerService);
+ document_type_ = utils::parseEnumProperty<CouchbaseValueType>(context, GetCouchbaseKey::DocumentType);
+}
+
+void GetCouchbaseKey::onTrigger(core::ProcessContext& context, core::ProcessSession& session) {
+ gsl_Expects(couchbase_cluster_service_);
+
+ auto flow_file = session.get();
+ if (!flow_file) {
+ context.yield();
+ return;
+ }
+
+ CouchbaseCollection collection;
+ if (!context.getProperty(BucketName, collection.bucket_name, flow_file.get()) || collection.bucket_name.empty()) {
+ logger_->log_error("Bucket '{}' is invalid or empty!", collection.bucket_name);
+ session.transfer(flow_file, Failure);
+ return;
+ }
+
+ if (!context.getProperty(ScopeName, collection.scope_name, flow_file.get()) || collection.scope_name.empty()) {
+ collection.scope_name = ::couchbase::scope::default_name;
+ }
+
+ if (!context.getProperty(CollectionName, collection.collection_name, flow_file.get()) || collection.collection_name.empty()) {
+ collection.collection_name = ::couchbase::collection::default_name;
+ }
+
+ std::string document_id;
+ if (!context.getProperty(DocumentId, document_id, flow_file.get()) || document_id.empty()) {
+ auto ff_content = session.readBuffer(flow_file).buffer;
+ document_id = std::string(reinterpret_cast<const char*>(ff_content.data()), ff_content.size());
+ }
+
+ if (document_id.empty()) {
+ logger_->log_error("Document ID is empty, transferring FlowFile to failure relationship");
+ session.transfer(flow_file, Failure);
+ return;
+ }
+
+ std::string attribute_to_put_result_to;
+ context.getProperty(PutValueToAttribute, attribute_to_put_result_to, flow_file.get());
+
+ if (auto get_result = couchbase_cluster_service_->get(collection, document_id, document_type_)) {
+ if (!attribute_to_put_result_to.empty()) {
+ if (document_type_ == CouchbaseValueType::String) {
+ session.putAttribute(*flow_file, attribute_to_put_result_to, std::get<std::string>(get_result->value));
+ } else {
+ auto& binary_data = std::get<std::vector<std::byte>>(get_result->value);
+ std::string str_value{reinterpret_cast<const char*>(binary_data.data()), binary_data.size()};
+ session.putAttribute(*flow_file, attribute_to_put_result_to, str_value);
+ }
+ } else {
+ session.write(flow_file, [&, this](const std::shared_ptr<io::OutputStream>& stream) -> int64_t {
+ if (document_type_ == CouchbaseValueType::String) {
+ auto& value = std::get<std::string>(get_result->value);
+ stream->write(value);
+ return gsl::narrow<int64_t>(value.size());
+ } else {
+ auto& value = std::get<std::vector<std::byte>>(get_result->value);
+ stream->write(value);
+ return gsl::narrow<int64_t>(value.size());
+ }
+ });
+ }
+
+ session.putAttribute(*flow_file, "couchbase.bucket", get_result->bucket_name);
+ session.putAttribute(*flow_file, "couchbase.doc.id", document_id);
+ session.putAttribute(*flow_file, "couchbase.doc.cas", std::to_string(get_result->cas));
+ session.putAttribute(*flow_file, "couchbase.doc.expiry", get_result->expiry);
+ session.transfer(flow_file, Success);
+ } else if (get_result.error() == CouchbaseErrorType::TEMPORARY) {
+ logger_->log_error("Failed to get document '{}' from collection '{}.{}.{}' due to timeout, transferring to retry relationship",
+ document_id, collection.bucket_name, collection.scope_name, collection.collection_name);
+ session.transfer(flow_file, Retry);
+ } else {
+ logger_->log_error("Failed to get document '{}' from collection '{}.{}.{}', transferring to failure relationship",
+ document_id, collection.bucket_name, collection.scope_name, collection.collection_name);
+ session.transfer(flow_file, Failure);
+ }
+}
+
+REGISTER_RESOURCE(GetCouchbaseKey, Processor);
+
+} // namespace org::apache::nifi::minifi::couchbase::processors
diff --git a/extensions/couchbase/processors/PutCouchbaseKey.h b/extensions/couchbase/processors/GetCouchbaseKey.h
similarity index 56%
copy from extensions/couchbase/processors/PutCouchbaseKey.h
copy to extensions/couchbase/processors/GetCouchbaseKey.h
index 56ba8c52f..c286561e4 100644
--- a/extensions/couchbase/processors/PutCouchbaseKey.h
+++ b/extensions/couchbase/processors/GetCouchbaseKey.h
@@ -23,56 +23,18 @@
#include "core/AbstractProcessor.h"
#include "core/ProcessSession.h"
-#include "utils/Enum.h"
#include "core/logging/LoggerConfiguration.h"
#include "CouchbaseClusterService.h"
-#include "couchbase/persist_to.hxx"
-#include "couchbase/replicate_to.hxx"
-
-namespace magic_enum::customize {
-
-template <>
-constexpr customize_t enum_name<::couchbase::persist_to>(::couchbase::persist_to value) noexcept {
- switch (value) {
- case ::couchbase::persist_to::none:
- return "NONE";
- case ::couchbase::persist_to::active:
- return "ACTIVE";
- case ::couchbase::persist_to::one:
- return "ONE";
- case ::couchbase::persist_to::two:
- return "TWO";
- case ::couchbase::persist_to::three:
- return "THREE";
- case ::couchbase::persist_to::four:
- return "FOUR";
- }
- return invalid_tag;
-}
-
-template <>
-constexpr customize_t enum_name<::couchbase::replicate_to>(::couchbase::replicate_to value) noexcept {
- switch (value) {
- case ::couchbase::replicate_to::none:
- return "NONE";
- case ::couchbase::replicate_to::one:
- return "ONE";
- case ::couchbase::replicate_to::two:
- return "TWO";
- case ::couchbase::replicate_to::three:
- return "THREE";
- }
- return invalid_tag;
-}
-} // namespace magic_enum::customize
namespace org::apache::nifi::minifi::couchbase::processors {
-class PutCouchbaseKey final : public core::AbstractProcessor<PutCouchbaseKey> {
+class GetCouchbaseKey final : public core::AbstractProcessor<GetCouchbaseKey> {
public:
- using core::AbstractProcessor<PutCouchbaseKey>::AbstractProcessor;
+ using core::AbstractProcessor<GetCouchbaseKey>::AbstractProcessor;
- EXTENSIONAPI static constexpr const char* Description = "Put a document to Couchbase Server via Key/Value access.";
+ EXTENSIONAPI static constexpr const char* Description = "Get a document from Couchbase Server via Key/Value access. The ID of the document to fetch may be supplied by setting the "
+ "<Document Id> property. NOTE: if the Document Id property is not set, the contents of the FlowFile will be read to determine the Document Id, which means that the contents of "
+ "the entire FlowFile will be buffered in memory.";
EXTENSIONAPI static constexpr auto CouchbaseClusterControllerService = core::PropertyDefinitionBuilder<>::createProperty("Couchbase Cluster Controller Service")
.withDescription("A Couchbase Cluster Controller Service which manages connections to a Couchbase cluster.")
@@ -94,27 +56,19 @@ class PutCouchbaseKey final : public core::AbstractProcessor<PutCouchbaseKey> {
.supportsExpressionLanguage(true)
.build();
EXTENSIONAPI static constexpr auto DocumentType = core::PropertyDefinitionBuilder<3>::createProperty("Document Type")
- .withDescription("Content type to store data as.")
+ .withDescription("Content type of the retrieved value.")
.isRequired(true)
.withDefaultValue(magic_enum::enum_name(CouchbaseValueType::Json))
.withAllowedValues(magic_enum::enum_names<CouchbaseValueType>())
.build();
EXTENSIONAPI static constexpr auto DocumentId = core::PropertyDefinitionBuilder<>::createProperty("Document Id")
- .withDescription("A static, fixed Couchbase document id, or an expression to construct the Couchbase document id. "
- "If not specified, either the FlowFile uuid attribute or if that's not found a generated uuid will be used.")
+ .withDescription("A static, fixed Couchbase document id, or an expression to construct the Couchbase document id.")
.supportsExpressionLanguage(true)
.build();
- EXTENSIONAPI static constexpr auto PersistTo = core::PropertyDefinitionBuilder<6>::createProperty("Persist To")
- .withDescription("Durability constraint about disk persistence.")
- .isRequired(true)
- .withDefaultValue(magic_enum::enum_name(::couchbase::persist_to::none))
- .withAllowedValues(magic_enum::enum_names<::couchbase::persist_to>())
- .build();
- EXTENSIONAPI static constexpr auto ReplicateTo = core::PropertyDefinitionBuilder<4>::createProperty("Replicate To")
- .withDescription("Durability constraint about replication.")
- .isRequired(true)
- .withDefaultValue(magic_enum::enum_name(::couchbase::replicate_to::none))
- .withAllowedValues(magic_enum::enum_names<::couchbase::replicate_to>())
+ EXTENSIONAPI static constexpr auto PutValueToAttribute = core::PropertyDefinitionBuilder<>::createProperty("Put Value to Attribute")
+ .withDescription("If set, the retrieved value will be put into an attribute of the FlowFile instead of a the content of the FlowFile. "
+ "The attribute key to put to is determined by evaluating value of this property.")
+ .supportsExpressionLanguage(true)
.build();
EXTENSIONAPI static constexpr auto Properties = std::to_array<core::PropertyReference>({
@@ -124,23 +78,23 @@ class PutCouchbaseKey final : public core::AbstractProcessor<PutCouchbaseKey> {
CollectionName,
DocumentType,
DocumentId,
- PersistTo,
- ReplicateTo
+ PutValueToAttribute
});
- EXTENSIONAPI static constexpr auto Success = core::RelationshipDefinition{"success", "All FlowFiles that are written to Couchbase Server are routed to this relationship."};
- EXTENSIONAPI static constexpr auto Failure = core::RelationshipDefinition{"failure", "All FlowFiles failed to be written to Couchbase Server and not retry-able are routed to this relationship."};
- EXTENSIONAPI static constexpr auto Retry = core::RelationshipDefinition{"retry", "All FlowFiles failed to be written to Couchbase Server but can be retried are routed to this relationship."};
- EXTENSIONAPI static constexpr auto Relationships = std::array{Success, Failure, Retry};
+ EXTENSIONAPI static constexpr auto Success = core::RelationshipDefinition{"success",
+ "Values retrieved from Couchbase Server are written as outgoing FlowFiles content or put into an attribute of the incoming FlowFile and routed to this relationship."};
+ EXTENSIONAPI static constexpr auto Failure = core::RelationshipDefinition{"failure", "All FlowFiles failed to fetch from Couchbase Server and not retry-able are routed to this relationship."};
+ EXTENSIONAPI static constexpr auto Retry = core::RelationshipDefinition{"retry", "All FlowFiles failed to fetch from Couchbase Server but can be retried are routed to this relationship."};
+ EXTENSIONAPI static constexpr auto Original = core::RelationshipDefinition{"original",
+ "The original input FlowFile is routed to this relationship when the value is retrieved from Couchbase Server and routed to 'success'."};
+ EXTENSIONAPI static constexpr auto Relationships = std::array{Success, Failure, Retry, Original};
EXTENSIONAPI static constexpr auto CouchbaseBucket = core::OutputAttributeDefinition<>{"couchbase.bucket", {Success}, "Bucket where the document was stored."};
EXTENSIONAPI static constexpr auto CouchbaseDocId = core::OutputAttributeDefinition<>{"couchbase.doc.id", {Success}, "Id of the document."};
EXTENSIONAPI static constexpr auto CouchbaseDocCas = core::OutputAttributeDefinition<>{"couchbase.doc.cas", {Success}, "CAS of the document."};
- EXTENSIONAPI static constexpr auto CouchbaseDocSequenceNumber = core::OutputAttributeDefinition<>{"couchbase.doc.sequence.number", {Success}, "Sequence number associated with the document."};
- EXTENSIONAPI static constexpr auto CouchbasePartitionUUID = core::OutputAttributeDefinition<>{"couchbase.partition.uuid", {Success}, "UUID of partition."};
- EXTENSIONAPI static constexpr auto CouchbasePartitionId = core::OutputAttributeDefinition<>{"couchbase.partition.id", {Success}, "ID of partition (also known as vBucket)."};
- EXTENSIONAPI static constexpr auto OutputAttributes = std::array<core::OutputAttributeReference, 6>{
- CouchbaseBucket, CouchbaseDocId, CouchbaseDocCas, CouchbaseDocSequenceNumber, CouchbasePartitionUUID, CouchbasePartitionId};
+ EXTENSIONAPI static constexpr auto CouchbaseDocExpiry = core::OutputAttributeDefinition<>{"couchbase.doc.expiry", {Success}, "Expiration of the document."};
+ EXTENSIONAPI static constexpr auto OutputAttributes = std::array<core::OutputAttributeReference, 4>{
+ CouchbaseBucket, CouchbaseDocId, CouchbaseDocCas, CouchbaseDocExpiry};
EXTENSIONAPI static constexpr bool SupportsDynamicProperties = false;
EXTENSIONAPI static constexpr bool SupportsDynamicRelationships = false;
@@ -152,10 +106,7 @@ class PutCouchbaseKey final : public core::AbstractProcessor<PutCouchbaseKey> {
private:
std::shared_ptr<controllers::CouchbaseClusterService> couchbase_cluster_service_;
- std::shared_ptr<core::logging::Logger> logger_ = core::logging::LoggerFactory<PutCouchbaseKey>::getLogger(uuid_);
CouchbaseValueType document_type_ = CouchbaseValueType::Json;
- ::couchbase::persist_to persist_to_ = ::couchbase::persist_to::none;
- ::couchbase::replicate_to replicate_to_ = ::couchbase::replicate_to::none;
};
} // namespace org::apache::nifi::minifi::couchbase::processors
diff --git a/extensions/couchbase/processors/PutCouchbaseKey.h b/extensions/couchbase/processors/PutCouchbaseKey.h
index 56ba8c52f..1472e3a20 100644
--- a/extensions/couchbase/processors/PutCouchbaseKey.h
+++ b/extensions/couchbase/processors/PutCouchbaseKey.h
@@ -152,7 +152,6 @@ class PutCouchbaseKey final : public core::AbstractProcessor<PutCouchbaseKey> {
private:
std::shared_ptr<controllers::CouchbaseClusterService> couchbase_cluster_service_;
- std::shared_ptr<core::logging::Logger> logger_ = core::logging::LoggerFactory<PutCouchbaseKey>::getLogger(uuid_);
CouchbaseValueType document_type_ = CouchbaseValueType::Json;
::couchbase::persist_to persist_to_ = ::couchbase::persist_to::none;
::couchbase::replicate_to replicate_to_ = ::couchbase::replicate_to::none;
diff --git a/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp b/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp
new file mode 100644
index 000000000..69f84aa06
--- /dev/null
+++ b/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp
@@ -0,0 +1,163 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "unit/TestBase.h"
+#include "unit/Catch.h"
+#include "unit/SingleProcessorTestController.h"
+#include "processors/GetCouchbaseKey.h"
+#include "MockCouchbaseClusterService.h"
+#include "core/Resource.h"
+
+namespace org::apache::nifi::minifi::couchbase::test {
+
+REGISTER_RESOURCE(MockCouchbaseClusterService, ControllerService);
+
+struct ExpectedCallOptions {
+ std::string bucket_name;
+ std::string scope_name;
+ std::string collection_name;
+ std::string doc_id;
+ couchbase::CouchbaseValueType document_type;
+};
+
+class GetCouchbaseKeyTestController : public TestController {
+ public:
+ GetCouchbaseKeyTestController() {
+ auto controller_service_node = controller_.plan->addController("MockCouchbaseClusterService", "MockCouchbaseClusterService");
+ mock_couchbase_cluster_service_ = std::static_pointer_cast<MockCouchbaseClusterService>(controller_service_node->getControllerServiceImplementation());
+ proc_->setProperty(processors::GetCouchbaseKey::CouchbaseClusterControllerService, "MockCouchbaseClusterService");
+ }
+
+ void verifyResults(const minifi::test::ProcessorTriggerResult& results, const minifi::core::Relationship& expected_result, const ExpectedCallOptions& expected_call_options,
+ const std::string& input) const {
+ std::shared_ptr<core::FlowFile> flow_file;
+ if (expected_result == processors::GetCouchbaseKey::Success) {
+ REQUIRE(results.at(processors::GetCouchbaseKey::Success).size() == 1);
+ REQUIRE(results.at(processors::GetCouchbaseKey::Failure).empty());
+ REQUIRE(results.at(processors::GetCouchbaseKey::Retry).empty());
+ flow_file = results.at(processors::GetCouchbaseKey::Success)[0];
+ } else if (expected_result == processors::GetCouchbaseKey::Failure) {
+ REQUIRE(results.at(processors::GetCouchbaseKey::Success).empty());
+ REQUIRE(results.at(processors::GetCouchbaseKey::Failure).size() == 1);
+ REQUIRE(results.at(processors::GetCouchbaseKey::Retry).empty());
+ flow_file = results.at(processors::GetCouchbaseKey::Failure)[0];
+ REQUIRE(LogTestController::getInstance().contains("Failed to get document", 1s));
+ } else {
+ REQUIRE(results.at(processors::GetCouchbaseKey::Success).empty());
+ REQUIRE(results.at(processors::GetCouchbaseKey::Failure).empty());
+ REQUIRE(results.at(processors::GetCouchbaseKey::Retry).size() == 1);
+ flow_file = results.at(processors::GetCouchbaseKey::Retry)[0];
+ }
+
+ auto get_collection_parameters = mock_couchbase_cluster_service_->getCollectionParameter();
+ CHECK(get_collection_parameters.bucket_name == expected_call_options.bucket_name);
+ CHECK(get_collection_parameters.collection_name == expected_call_options.collection_name);
+ CHECK(get_collection_parameters.scope_name == expected_call_options.scope_name);
+
+ auto get_parameters = mock_couchbase_cluster_service_->getGetParameters();
+ CHECK(get_parameters.document_id == expected_call_options.doc_id);
+ CHECK(get_parameters.document_type == expected_call_options.document_type);
+
+ if (expected_result != processors::GetCouchbaseKey::Success) {
+ return;
+ }
+
+ CHECK(flow_file->getAttribute("couchbase.bucket").value() == expected_call_options.bucket_name);
+ CHECK(flow_file->getAttribute("couchbase.doc.id").value() == expected_call_options.doc_id);
+ CHECK(flow_file->getAttribute("couchbase.doc.cas").value() == std::to_string(COUCHBASE_GET_RESULT_CAS));
+ CHECK(flow_file->getAttribute("couchbase.doc.expiry").value() == COUCHBASE_GET_RESULT_EXPIRY);
+ std::string value;
+ proc_->getProperty(processors::GetCouchbaseKey::PutValueToAttribute, value);
+ if (!value.empty()) {
+ CHECK(flow_file->getAttribute(value).value() == COUCHBASE_GET_RESULT_CONTENT);
+ CHECK(controller_.plan->getContent(flow_file) == input);
+ } else {
+ CHECK(controller_.plan->getContent(flow_file) == COUCHBASE_GET_RESULT_CONTENT);
+ }
+ }
+
+ protected:
+ std::shared_ptr<core::Processor> proc_ = std::make_shared<processors::GetCouchbaseKey>("GetCouchbaseKey");
+ minifi::test::SingleProcessorTestController controller_{proc_};
+ std::shared_ptr<MockCouchbaseClusterService> mock_couchbase_cluster_service_;
+};
+
+TEST_CASE_METHOD(GetCouchbaseKeyTestController, "Invalid Couchbase cluster controller service", "[getcouchbasekey]") {
+ proc_->setProperty(processors::GetCouchbaseKey::CouchbaseClusterControllerService, "invalid");
+ REQUIRE_THROWS_AS(controller_.trigger({minifi::test::InputFlowFileData{"couchbase_id"}}), minifi::Exception);
+}
+
+TEST_CASE_METHOD(GetCouchbaseKeyTestController, "Invalid bucket name", "[getcouchbasekey]") {
+ proc_->setProperty(processors::GetCouchbaseKey::BucketName, "");
+ auto results = controller_.trigger({minifi::test::InputFlowFileData{"couchbase_id"}});
+ REQUIRE(results[processors::GetCouchbaseKey::Failure].size() == 1);
+ REQUIRE(LogTestController::getInstance().contains("Bucket '' is invalid or empty!", 1s));
+}
+
+TEST_CASE_METHOD(GetCouchbaseKeyTestController, "Document ID is empty and no content is present to use", "[getcouchbasekey]") {
+ proc_->setProperty(processors::GetCouchbaseKey::BucketName, "mybucket");
+ auto results = controller_.trigger({minifi::test::InputFlowFileData{""}});
+ REQUIRE(results.at(processors::GetCouchbaseKey::Success).empty());
+ REQUIRE(results.at(processors::GetCouchbaseKey::Failure).size() == 1);
+ REQUIRE(results.at(processors::GetCouchbaseKey::Retry).empty());
+ REQUIRE(LogTestController::getInstance().contains("Document ID is empty, transferring FlowFile to failure relationship", 1s));
+}
+
+TEST_CASE_METHOD(GetCouchbaseKeyTestController, "Get succeeeds with default properties", "[getcouchbasekey]") {
+ proc_->setProperty(processors::GetCouchbaseKey::BucketName, "mybucket");
+ const std::string input = "couchbase_id";
+ auto results = controller_.trigger({minifi::test::InputFlowFileData{input}});
+ verifyResults(results, processors::GetCouchbaseKey::Success, ExpectedCallOptions{"mybucket", "_default", "_default", input, couchbase::CouchbaseValueType::Json}, input);
+}
+
+TEST_CASE_METHOD(GetCouchbaseKeyTestController, "Get succeeeds with optional properties", "[getcouchbasekey]") {
+ proc_->setProperty(processors::GetCouchbaseKey::BucketName, "mybucket");
+ proc_->setProperty(processors::GetCouchbaseKey::ScopeName, "scope1");
+ proc_->setProperty(processors::GetCouchbaseKey::CollectionName, "collection1");
+ proc_->setProperty(processors::GetCouchbaseKey::DocumentId, "important_doc");
+ proc_->setProperty(processors::GetCouchbaseKey::DocumentType, "Binary");
+ auto results = controller_.trigger({minifi::test::InputFlowFileData{""}});
+ verifyResults(results, processors::GetCouchbaseKey::Success, ExpectedCallOptions{"mybucket", "scope1", "collection1", "important_doc", couchbase::CouchbaseValueType::Binary}, "");
+}
+
+TEST_CASE_METHOD(GetCouchbaseKeyTestController, "Get fails with default properties", "[getcouchbasekey]") {
+ proc_->setProperty(processors::GetCouchbaseKey::BucketName, "mybucket");
+ mock_couchbase_cluster_service_->setGetError(CouchbaseErrorType::FATAL);
+ const std::string input = "couchbase_id";
+ auto results = controller_.trigger({minifi::test::InputFlowFileData{input}});
+ verifyResults(results, processors::GetCouchbaseKey::Failure, ExpectedCallOptions{"mybucket", "_default", "_default", input, couchbase::CouchbaseValueType::Json}, input);
+}
+
+TEST_CASE_METHOD(GetCouchbaseKeyTestController, "FlowFile is transferred to retry relationship when temporary error is returned", "[getcouchbasekey]") {
+ proc_->setProperty(processors::GetCouchbaseKey::BucketName, "mybucket");
+ mock_couchbase_cluster_service_->setGetError(CouchbaseErrorType::TEMPORARY);
+ const std::string input = "couchbase_id";
+ auto results = controller_.trigger({minifi::test::InputFlowFileData{input}});
+ verifyResults(results, processors::GetCouchbaseKey::Retry, ExpectedCallOptions{"mybucket", "_default", "_default", input, couchbase::CouchbaseValueType::Json}, input);
+}
+
+TEST_CASE_METHOD(GetCouchbaseKeyTestController, "Get result is written to attribute", "[getcouchbasekey]") {
+ proc_->setProperty(processors::GetCouchbaseKey::BucketName, "mybucket");
+ proc_->setProperty(processors::GetCouchbaseKey::DocumentType, "String");
+ proc_->setProperty(processors::GetCouchbaseKey::PutValueToAttribute, "myattribute");
+ const std::string input = "couchbase_id";
+ auto results = controller_.trigger({minifi::test::InputFlowFileData{input}});
+ verifyResults(results, processors::GetCouchbaseKey::Success, ExpectedCallOptions{"mybucket", "_default", "_default", input, couchbase::CouchbaseValueType::String}, input);
+}
+
+} // namespace org::apache::nifi::minifi::couchbase::test
diff --git a/extensions/couchbase/tests/MockCouchbaseClusterService.h b/extensions/couchbase/tests/MockCouchbaseClusterService.h
index a2757f16c..6566e85d4 100644
--- a/extensions/couchbase/tests/MockCouchbaseClusterService.h
+++ b/extensions/couchbase/tests/MockCouchbaseClusterService.h
@@ -29,6 +29,9 @@ const std::uint64_t COUCHBASE_PUT_RESULT_CAS = 9876;
const std::uint64_t COUCHBASE_PUT_RESULT_SEQUENCE_NUMBER = 345;
const std::uint64_t COUCHBASE_PUT_RESULT_PARTITION_UUID = 7890123456;
const std::uint16_t COUCHBASE_PUT_RESULT_PARTITION_ID = 1234;
+const std::string COUCHBASE_GET_RESULT_EXPIRY = "2024/10/14 09:37:43.000Z";
+const std::string COUCHBASE_GET_RESULT_CONTENT = "abc";
+const uint64_t COUCHBASE_GET_RESULT_CAS = 1234567;
struct UpsertParameters {
CouchbaseValueType document_type;
@@ -36,6 +39,10 @@ struct UpsertParameters {
std::vector<std::byte> buffer;
::couchbase::upsert_options options;
};
+struct GetParameters {
+ std::string document_id;
+ CouchbaseValueType document_type;
+};
class MockCouchbaseClusterService : public controllers::CouchbaseClusterService {
public:
@@ -57,7 +64,23 @@ class MockCouchbaseClusterService : public controllers::CouchbaseClusterService
if (upsert_error_) {
return nonstd::make_unexpected(*upsert_error_);
} else {
- return CouchbaseUpsertResult{collection_.bucket_name, COUCHBASE_PUT_RESULT_CAS, COUCHBASE_PUT_RESULT_SEQUENCE_NUMBER, COUCHBASE_PUT_RESULT_PARTITION_UUID, COUCHBASE_PUT_RESULT_PARTITION_ID};
+ return CouchbaseUpsertResult{{collection_.bucket_name, COUCHBASE_PUT_RESULT_CAS}, COUCHBASE_PUT_RESULT_SEQUENCE_NUMBER, COUCHBASE_PUT_RESULT_PARTITION_UUID, COUCHBASE_PUT_RESULT_PARTITION_ID};
+ }
+ }
+
+ nonstd::expected<CouchbaseGetResult, CouchbaseErrorType> get(const CouchbaseCollection& collection, const std::string& document_id, CouchbaseValueType document_type) override {
+ collection_ = collection;
+ get_parameters_.document_id = document_id;
+ get_parameters_.document_type = document_type;
+
+ if (get_error_) {
+ return nonstd::make_unexpected(*get_error_);
+ } else {
+ if (document_type == CouchbaseValueType::String) {
+ return CouchbaseGetResult{{collection_.bucket_name, COUCHBASE_GET_RESULT_CAS}, COUCHBASE_GET_RESULT_EXPIRY, COUCHBASE_GET_RESULT_CONTENT};
+ }
+ return CouchbaseGetResult{{collection_.bucket_name, COUCHBASE_GET_RESULT_CAS}, COUCHBASE_GET_RESULT_EXPIRY,
+ std::vector<std::byte>{static_cast<std::byte>('a'), static_cast<std::byte>('b'), static_cast<std::byte>('c')}};
}
}
@@ -65,6 +88,10 @@ class MockCouchbaseClusterService : public controllers::CouchbaseClusterService
return upsert_parameters_;
}
+ GetParameters getGetParameters() const {
+ return get_parameters_;
+ }
+
CouchbaseCollection getCollectionParameter() const {
return collection_;
}
@@ -73,9 +100,15 @@ class MockCouchbaseClusterService : public controllers::CouchbaseClusterService
upsert_error_ = upsert_error;
}
+ void setGetError(const CouchbaseErrorType get_error) {
+ get_error_ = get_error;
+ }
+
private:
CouchbaseCollection collection_;
UpsertParameters upsert_parameters_;
+ GetParameters get_parameters_;
std::optional<CouchbaseErrorType> upsert_error_;
+ std::optional<CouchbaseErrorType> get_error_;
};
} // namespace org::apache::nifi::minifi::couchbase::test
diff --git a/libminifi/include/core/Processor.h b/libminifi/include/core/Processor.h
index f720733e8..6f25c176c 100644
--- a/libminifi/include/core/Processor.h
+++ b/libminifi/include/core/Processor.h
@@ -238,6 +238,8 @@ class Processor : public Connectable, public ConfigurableComponent, public state
std::string cron_period_;
gsl::not_null<std::shared_ptr<ProcessorMetrics>> metrics_;
+ std::shared_ptr<logging::Logger> logger_;
+
private:
mutable std::mutex mutex_;
std::atomic<std::chrono::steady_clock::time_point> yield_expiration_{};
@@ -254,8 +256,6 @@ class Processor : public Connectable, public ConfigurableComponent, public state
// an outgoing connection allows us to reach these nodes
std::unordered_map<Connection*, std::unordered_set<Processor*>> reachable_processors_;
-
- std::shared_ptr<logging::Logger> logger_;
};
} // namespace core
(nifi-minifi-cpp) 04/04: Fix after rebase
Posted by lo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lordgamez pushed a commit to branch MINIFICPP-2469
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit d15c53f64e8b368f9b943b838e5dfed66f37f6c3
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Tue Nov 19 10:08:57 2024 +0100
Fix after rebase
---
extensions/couchbase/tests/GetCouchbaseKeyTests.cpp | 8 +++++---
1 file changed, 5 insertions(+), 3 deletions(-)
diff --git a/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp b/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp
index d518ace03..0d2e5b910 100644
--- a/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp
+++ b/extensions/couchbase/tests/GetCouchbaseKeyTests.cpp
@@ -37,7 +37,9 @@ struct ExpectedCallOptions {
class GetCouchbaseKeyTestController : public TestController {
public:
- GetCouchbaseKeyTestController() {
+ GetCouchbaseKeyTestController()
+ : controller_(std::make_unique<processors::GetCouchbaseKey>("GetCouchbaseKey")),
+ proc_(controller_.getProcessor()) {
LogTestController::getInstance().setDebug<TestPlan>();
LogTestController::getInstance().setDebug<minifi::core::Processor>();
LogTestController::getInstance().setTrace<minifi::core::ProcessSession>();
@@ -97,8 +99,8 @@ class GetCouchbaseKeyTestController : public TestController {
}
protected:
- std::shared_ptr<core::Processor> proc_ = std::make_shared<processors::GetCouchbaseKey>("GetCouchbaseKey");
- minifi::test::SingleProcessorTestController controller_{proc_};
+ minifi::test::SingleProcessorTestController controller_;
+ core::Processor* proc_ = nullptr;
std::shared_ptr<MockCouchbaseClusterService> mock_couchbase_cluster_service_;
};
(nifi-minifi-cpp) 02/04: Add extenstion docs and refactor docker tests
Posted by lo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
lordgamez pushed a commit to branch MINIFICPP-2469
in repository https://gitbox.apache.org/repos/asf/nifi-minifi-cpp.git
commit 76cc382a661b0a7faa40a0a9737204c1a1f0e622
Author: Gabor Gyimesi <ga...@gmail.com>
AuthorDate: Mon Oct 21 14:24:31 2024 +0200
Add extenstion docs and refactor docker tests
---
README.md | 2 +-
docker/test/integration/features/couchbase.feature | 32 +++++++++++-----------
2 files changed, 17 insertions(+), 17 deletions(-)
diff --git a/README.md b/README.md
index 069d142ad..3c659f335 100644
--- a/README.md
+++ b/README.md
@@ -75,7 +75,7 @@ The next table outlines CMAKE flags that correspond with MiNiFi extensions. Exte
| AWS | [AWSCredentialsService](CONTROLLERS.md#awscredentialsservice)<br/>[PutS3Object](PROCESSORS.md#puts3object)<br/>[DeleteS3Object](PROCESSORS.md#deletes3object)<br/>[FetchS3Object](PROCESSORS.md#fetchs3object)<br/>[ListS3](PROCESSORS.md#lists3) [...]
| Azure | [AzureStorageCredentialsService](CONTROLLERS.md#azurestoragecredentialsservice)<br/>[PutAzureBlobStorage](PROCESSORS.md#putazureblobstorage)<br/>[DeleteAzureBlobStorage](PROCESSORS.md#deleteazureblobstorage)<br/>[FetchAzureBlobStorage](PROCESSORS.md#fetchazureblobstorage)<br/>[ListAzureBlobStorage](PROCESSORS.md#listazureblobstorage)<br/>[PutAzureDataLakeStorage](PROCESSORS.md#putazuredatalakestorage)<br/>[DeleteAzureDataLakeStorage](PROCESSORS.md#del [...]
| CivetWeb | [ListenHTTP](PROCESSORS.md#listenhttp) [...]
-| Couchbase | [CouchbaseClusterService](CONTROLLERS.md#couchbaseclusterservice)<br/>[PutCouchbaseKey](PROCESSORS.md#putcouchbasekey) [...]
+| Couchbase | [CouchbaseClusterService](CONTROLLERS.md#couchbaseclusterservice)<br/>[PutCouchbaseKey](PROCESSORS.md#putcouchbasekey)<br/>[GetCouchbaseKey](PROCESSORS.md#getcouchbasekey) [...]
| Elasticsearch | [ElasticsearchCredentialsControllerService](CONTROLLERS.md#elasticsearchcredentialscontrollerservice)<br/>[PostElasticsearch](PROCESSORS.md#postelasticsearch) [...]
| ExecuteProcess (Linux and macOS) | [ExecuteProcess](PROCESSORS.md#executeprocess) [...]
| Google Cloud Platform | [DeleteGCSObject](PROCESSORS.md#deletegcsobject)<br>[FetchGCSObject](PROCESSORS.md#fetchgcsobject)<br>[GCPCredentialsControllerService](CONTROLLERS.md#gcpcredentialscontrollerservice)<br>[ListGCSBucket](PROCESSORS.md#listgcsbucket)<br>[PutGCSObject](PROCESSORS.md#putgcsobject) [...]
diff --git a/docker/test/integration/features/couchbase.feature b/docker/test/integration/features/couchbase.feature
index d9a4bb664..646eaacd9 100644
--- a/docker/test/integration/features/couchbase.feature
+++ b/docker/test/integration/features/couchbase.feature
@@ -67,17 +67,17 @@ Feature: Executing Couchbase operations from MiNiFi-C++
And a document with id "test_doc_id" in bucket "test_bucket" is present with data '{"field1": "value1"}' of type "Binary" in Couchbase
Scenario: A MiNiFi instance can get data from test bucket with GetCouchbaseKey processor
- Given a Couchbase server is set up
- And a GetFile processor with the "Input Directory" property set to "/tmp/input"
+ Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
And a file with the content '{"field1": "value1", "field2": "value2"}' is present in '/tmp/input'
And a PutCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
And the "Document Id" property of the PutCouchbaseKey processor is set to "test_doc_id"
+ And the "Couchbase Cluster Controller Service" property of the PutCouchbaseKey processor is set to "CouchbaseClusterService"
And a GetCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
And the "Document Id" property of the GetCouchbaseKey processor is set to "test_doc_id"
+ And the "Couchbase Cluster Controller Service" property of the GetCouchbaseKey processor is set to "CouchbaseClusterService"
And a PutFile processor with the "Directory" property set to "/tmp/output"
And a LogAttribute processor with the "FlowFiles To Log" property set to "0"
- And a CouchbaseClusterService is setup up for PutCouchbaseKey with the name "CouchbaseClusterService"
- And a CouchbaseClusterService is setup up for GetCouchbaseKey with the name "CouchbaseClusterService"
+ And a CouchbaseClusterService is setup up with the name "CouchbaseClusterService"
And the "success" relationship of the GetFile processor is connected to the PutCouchbaseKey
And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey
@@ -94,19 +94,19 @@ Feature: Executing Couchbase operations from MiNiFi-C++
And the Minifi logs match the following regex: "key:couchbase.doc.expiry value:\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}" in less than 1 seconds
Scenario: A MiNiFi instance can get data from test bucket with GetCouchbaseKey processor using binary storage
- Given a Couchbase server is set up
- And a GetFile processor with the "Input Directory" property set to "/tmp/input"
+ Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
And a file with the content '{"field1": "value1", "field2": "value2"}' is present in '/tmp/input'
And a PutCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
And the "Document Id" property of the PutCouchbaseKey processor is set to "test_doc_id"
And the "Document Type" property of the PutCouchbaseKey processor is set to "Binary"
+ And the "Couchbase Cluster Controller Service" property of the PutCouchbaseKey processor is set to "CouchbaseClusterService"
And a GetCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
And the "Document Id" property of the GetCouchbaseKey processor is set to "test_doc_id"
And the "Document Type" property of the GetCouchbaseKey processor is set to "Binary"
+ And the "Couchbase Cluster Controller Service" property of the GetCouchbaseKey processor is set to "CouchbaseClusterService"
And a PutFile processor with the "Directory" property set to "/tmp/output"
And a LogAttribute processor with the "FlowFiles To Log" property set to "0"
- And a CouchbaseClusterService is setup up for PutCouchbaseKey with the name "CouchbaseClusterService"
- And a CouchbaseClusterService is setup up for GetCouchbaseKey with the name "CouchbaseClusterService"
+ And a CouchbaseClusterService is setup up with the name "CouchbaseClusterService"
And the "success" relationship of the GetFile processor is connected to the PutCouchbaseKey
And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey
@@ -123,20 +123,20 @@ Feature: Executing Couchbase operations from MiNiFi-C++
And the Minifi logs match the following regex: "key:couchbase.doc.expiry value:\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{3}" in less than 1 seconds
Scenario: A MiNiFi instance can get data from test bucket with GetCouchbaseKey processor and put the result in an attribute
- Given a Couchbase server is set up
- And a GetFile processor with the "Input Directory" property set to "/tmp/input"
+ Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
And a file with the content '{"field1": "value1", "field2": "value2"}' is present in '/tmp/input'
And a PutCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
And the "Document Id" property of the PutCouchbaseKey processor is set to "test_doc_id"
And the "Document Type" property of the PutCouchbaseKey processor is set to "String"
+ And the "Couchbase Cluster Controller Service" property of the PutCouchbaseKey processor is set to "CouchbaseClusterService"
And a GetCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
And the "Document Id" property of the GetCouchbaseKey processor is set to "test_doc_id"
And the "Document Type" property of the GetCouchbaseKey processor is set to "String"
+ And the "Couchbase Cluster Controller Service" property of the GetCouchbaseKey processor is set to "CouchbaseClusterService"
And the "Put Value to Attribute" property of the GetCouchbaseKey processor is set to "get_couchbase_result"
And a PutFile processor with the "Directory" property set to "/tmp/output"
And a LogAttribute processor with the "FlowFiles To Log" property set to "0"
- And a CouchbaseClusterService is setup up for PutCouchbaseKey with the name "CouchbaseClusterService"
- And a CouchbaseClusterService is setup up for GetCouchbaseKey with the name "CouchbaseClusterService"
+ And a CouchbaseClusterService is setup up with the name "CouchbaseClusterService"
And the "success" relationship of the GetFile processor is connected to the PutCouchbaseKey
And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey
@@ -154,17 +154,17 @@ Feature: Executing Couchbase operations from MiNiFi-C++
And the Minifi logs contain the following message: 'key:get_couchbase_result value:{"field1": "value1", "field2": "value2"}' in less than 1 seconds
Scenario: GetCouchbaseKey transfers FlowFile to failure relationship on Couchbase value type mismatch
- Given a Couchbase server is set up
- And a GetFile processor with the "Input Directory" property set to "/tmp/input"
+ Given a GetFile processor with the "Input Directory" property set to "/tmp/input"
And a file with the content '{"field1": "value1", "field2": "value2"}' is present in '/tmp/input'
And a PutCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
And the "Document Id" property of the PutCouchbaseKey processor is set to "test_doc_id"
And the "Document Type" property of the PutCouchbaseKey processor is set to "String"
+ And the "Couchbase Cluster Controller Service" property of the PutCouchbaseKey processor is set to "CouchbaseClusterService"
And a GetCouchbaseKey processor with the "Bucket Name" property set to "test_bucket"
And the "Document Id" property of the GetCouchbaseKey processor is set to "test_doc_id"
And the "Document Type" property of the GetCouchbaseKey processor is set to "Binary"
- And a CouchbaseClusterService is setup up for PutCouchbaseKey with the name "CouchbaseClusterService"
- And a CouchbaseClusterService is setup up for GetCouchbaseKey with the name "CouchbaseClusterService"
+ And the "Couchbase Cluster Controller Service" property of the GetCouchbaseKey processor is set to "CouchbaseClusterService"
+ And a CouchbaseClusterService is setup up with the name "CouchbaseClusterService"
And the "success" relationship of the GetFile processor is connected to the PutCouchbaseKey
And the "success" relationship of the PutCouchbaseKey processor is connected to the GetCouchbaseKey