You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@nifi.apache.org by ex...@apache.org on 2024/05/12 01:54:20 UTC
(nifi) branch main updated (74731b5a59 -> 15696ad86a)
This is an automated email from the ASF dual-hosted git repository.
exceptionfactory pushed a change to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
from 74731b5a59 [NIFI-13212] handle canvas overflow on shorter screens (#8810)
new 6fc374b268 NIFI-13215 AWS SDK and Elasticsearch Dependencies Upgraded
new c66a3cf4ff NIFI-13214 Replaced deprecated Lucene IndexReader methods with StoredFields
new 22109de2ad NIFI-13216 Replaced deprecated GzipUtils methods
new 03005aa132 NIFI-13185 Removed manual and non-working Integration Tests
new 15696ad86a NIFI-13217 Updated ProcessSession Comments to List Immutable Attributes
The 5 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/integration-tests.yml | 1 -
.../operation/TransferDebugOperationHandler.java | 4 +-
.../org/apache/nifi/processor/ProcessSession.java | 7 +-
...andardHashiCorpVaultCommunicationServiceIT.java | 126 ----
.../AwsKmsSensitivePropertyProviderIT.java | 142 ----
...sSecretsManagerSensitivePropertyProviderIT.java | 133 ----
...zureKeyVaultKeySensitivePropertyProviderIT.java | 135 ----
.../GcpKmsSensitivePropertyProviderIT.java | 134 ----
.../aws/kinesis/firehose/ITPutKinesisFirehose.java | 286 --------
.../stream/ITConsumeKinesisStreamConnectAWS.java | 72 --
.../ITConsumeKinesisStreamEndpointOverride.java | 88 ---
.../aws/kinesis/stream/ITPutKinesisStream.java | 421 -----------
.../ITPutKinesisStreamWithEndpointOverride.java | 84 ---
.../nifi/processors/aws/lambda/ITPutLambda.java | 137 ----
.../apache/nifi/processors/aws/sns/ITPutSNS.java | 78 --
nifi-extension-bundles/nifi-aws-bundle/pom.xml | 2 +-
.../azure/AzureGraphUserGroupProviderIT.java | 219 ------
.../cosmos/document/ITPutAzureCosmosDBRecord.java | 160 -----
.../storage/AbstractAzureBlobStorage_v12IT.java | 195 -----
.../storage/AbstractAzureDataLakeStorageIT.java | 151 ----
.../azure/storage/AbstractAzureStorageIT.java | 132 ----
.../azure/storage/ITCopyAzureBlobStorage_v12.java | 122 ----
.../storage/ITDeleteAzureBlobStorage_v12.java | 194 -----
.../storage/ITDeleteAzureDataLakeStorage.java | 501 -------------
.../azure/storage/ITFetchAzureBlobStorage_v12.java | 232 ------
.../azure/storage/ITFetchAzureDataLakeStorage.java | 558 ---------------
.../azure/storage/ITListAzureBlobStorage_v12.java | 223 ------
.../azure/storage/ITListAzureDataLakeStorage.java | 534 --------------
.../azure/storage/ITMoveAzureDataLakeStorage.java | 374 ----------
.../azure/storage/ITPutAzureBlobStorage_v12.java | 385 ----------
.../azure/storage/ITPutAzureDataLakeStorage.java | 434 -----------
.../nifi/processors/box/AbstractBoxFileIT.java | 137 ----
.../apache/nifi/processors/box/FetchBoxFileIT.java | 137 ----
.../apache/nifi/processors/box/ListBoxFileIT.java | 143 ----
.../apache/nifi/processors/box/PutBoxFileIT.java | 120 ----
.../JsonConfigBasedBoxClientServiceIT.java | 87 ---
.../nifi/processors/dropbox/AbstractDropboxIT.java | 128 ----
.../nifi/processors/dropbox/FetchDropboxIT.java | 132 ----
.../nifi/processors/dropbox/ListDropboxIT.java | 136 ----
.../nifi/processors/dropbox/PutDropboxIT.java | 157 ----
.../nifi-elasticsearch-bundle/pom.xml | 4 +-
.../GremlinClientServiceControllerSettingsIT.java | 81 ---
...mlinClientServiceYamlSettingsAndBytecodeIT.java | 133 ----
.../jms/processors/JMSPublisherConsumerIT.java | 58 --
.../apache/nifi/processors/mongodb/GetMongoIT.java | 76 --
.../processors/mongodb/gridfs/PutGridFSIT.java | 11 -
.../prometheus/PrometheusReportingTaskIT.java | 18 -
.../provenance/index/lucene/LuceneEventIndex.java | 4 +-
.../nifi/provenance/index/lucene/QueryTask.java | 9 +-
.../apache/nifi/provenance/lucene/DocsReader.java | 4 +-
.../ITRedisDistributedMapCacheClientService.java | 486 -------------
.../nifi/redis/state/ITRedisStateProvider.java | 3 +-
.../salesforce/PutSalesforceObjectIT.java | 135 ----
.../salesforce/QuerySalesforceObjectIT.java | 99 ---
.../salesforce/util/SalesforceRestServiceIT.java | 78 --
.../nifi/processors/snowflake/SnowflakePipeIT.java | 122 ----
.../service/SnowflakeConnectionPoolIT.java | 87 ---
.../nifi-snowflake-bundle/pom.xml | 2 +-
.../processors/standard/AbstractConversionIT.java | 388 ----------
.../standard/ConversionWithExplicitSchemaIT.java | 88 ---
.../standard/ConversionWithSchemaInferenceIT.java | 51 --
.../TestConversions/data.int_float_string.json | 22 -
.../data.int_float_string.with_header.csv | 8 -
.../data.int_float_string.with_schema.avro | Bin 302 -> 0 bytes
.../data.int_float_string.with_schema.json.to.avro | Bin 320 -> 0 bytes
.../data.int_float_string.without_header.csv | 7 -
.../data.int_float_string.without_schema.avro | Bin 51 -> 0 bytes
.../TestConversions/data.int_float_string.xml | 31 -
.../resources/TestConversions/explicit.schema.json | 23 -
.../repository/StandardProcessSessionIT.java | 2 +
.../accesscontrol/ITConnectionAccessControl.java | 427 -----------
.../accesscontrol/ITCountersAccessControl.java | 84 ---
.../accesscontrol/ITFlowAccessControl.java | 276 -------
.../accesscontrol/ITFunnelAccessControl.java | 359 ----------
.../accesscontrol/ITInputPortAccessControl.java | 403 -----------
.../accesscontrol/ITLabelAccessControl.java | 400 -----------
.../accesscontrol/ITOutputPortAccessControl.java | 403 -----------
.../accesscontrol/ITProcessGroupAccessControl.java | 465 ------------
.../accesscontrol/ITProcessorAccessControl.java | 749 -------------------
.../anonymous/AbstractAnonymousUserTest.java | 57 --
.../anonymous/ITAllowDirectAnonymousAccess.java | 67 --
.../anonymous/ITAllowProxiedAnonymousAccess.java | 67 --
.../anonymous/ITPreventDirectAnonymousAccess.java | 57 --
.../anonymous/ITPreventProxiedAnonymousAccess.java | 58 --
.../nifi/registry/web/api/SecureDatabaseIT.java | 237 ------
.../web/api/SecureNiFiRegistryClientIT.java | 163 +----
.../nifi/registry/web/api/SecureProxyIT.java | 75 --
.../web/api/UnsecuredNiFiRegistryClientIT.java | 796 +--------------------
pom.xml | 114 +--
89 files changed, 29 insertions(+), 14699 deletions(-)
delete mode 100644 nifi-commons/nifi-hashicorp-vault/src/test/java/org/apache/nifi/vault/hashicorp/StandardHashiCorpVaultCommunicationServiceIT.java
delete mode 100644 nifi-commons/nifi-property-protection-aws/src/test/java/org/apache/nifi/properties/AwsKmsSensitivePropertyProviderIT.java
delete mode 100644 nifi-commons/nifi-property-protection-aws/src/test/java/org/apache/nifi/properties/AwsSecretsManagerSensitivePropertyProviderIT.java
delete mode 100644 nifi-commons/nifi-property-protection-azure/src/test/java/org/apache/nifi/properties/AzureKeyVaultKeySensitivePropertyProviderIT.java
delete mode 100644 nifi-commons/nifi-property-protection-gcp/src/test/java/org/apache/nifi/properties/GcpKmsSensitivePropertyProviderIT.java
delete mode 100644 nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/firehose/ITPutKinesisFirehose.java
delete mode 100644 nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITConsumeKinesisStreamConnectAWS.java
delete mode 100644 nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITConsumeKinesisStreamEndpointOverride.java
delete mode 100644 nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITPutKinesisStream.java
delete mode 100644 nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITPutKinesisStreamWithEndpointOverride.java
delete mode 100644 nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/lambda/ITPutLambda.java
delete mode 100644 nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/sns/ITPutSNS.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-graph-authorizer/src/test/java/org/apache/nifi/authorization/azure/AzureGraphUserGroupProviderIT.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/cosmos/document/ITPutAzureCosmosDBRecord.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureBlobStorage_v12IT.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureDataLakeStorageIT.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureStorageIT.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITCopyAzureBlobStorage_v12.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITDeleteAzureBlobStorage_v12.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITDeleteAzureDataLakeStorage.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITFetchAzureBlobStorage_v12.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITFetchAzureDataLakeStorage.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITListAzureBlobStorage_v12.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITListAzureDataLakeStorage.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITMoveAzureDataLakeStorage.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITPutAzureBlobStorage_v12.java
delete mode 100644 nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITPutAzureDataLakeStorage.java
delete mode 100644 nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/AbstractBoxFileIT.java
delete mode 100644 nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/FetchBoxFileIT.java
delete mode 100644 nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/ListBoxFileIT.java
delete mode 100644 nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/PutBoxFileIT.java
delete mode 100644 nifi-extension-bundles/nifi-box-bundle/nifi-box-services/src/test/java/org/apache/nifi/box/controllerservices/JsonConfigBasedBoxClientServiceIT.java
delete mode 100644 nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/AbstractDropboxIT.java
delete mode 100644 nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/FetchDropboxIT.java
delete mode 100644 nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/ListDropboxIT.java
delete mode 100644 nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/PutDropboxIT.java
delete mode 100644 nifi-extension-bundles/nifi-graph-bundle/nifi-other-graph-services/src/test/java/org/apache/nifi/graph/GremlinClientServiceControllerSettingsIT.java
delete mode 100644 nifi-extension-bundles/nifi-graph-bundle/nifi-other-graph-services/src/test/java/org/apache/nifi/graph/GremlinClientServiceYamlSettingsAndBytecodeIT.java
delete mode 100644 nifi-extension-bundles/nifi-redis-bundle/nifi-redis-extensions/src/test/java/org/apache/nifi/redis/service/ITRedisDistributedMapCacheClientService.java
delete mode 100644 nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/PutSalesforceObjectIT.java
delete mode 100644 nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/QuerySalesforceObjectIT.java
delete mode 100644 nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/util/SalesforceRestServiceIT.java
delete mode 100644 nifi-extension-bundles/nifi-snowflake-bundle/nifi-snowflake-processors/src/test/java/org/apache/nifi/processors/snowflake/SnowflakePipeIT.java
delete mode 100644 nifi-extension-bundles/nifi-snowflake-bundle/nifi-snowflake-services/src/test/java/org/apache/nifi/snowflake/service/SnowflakeConnectionPoolIT.java
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/AbstractConversionIT.java
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/ConversionWithExplicitSchemaIT.java
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/ConversionWithSchemaInferenceIT.java
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.json
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_header.csv
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_schema.avro
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_schema.json.to.avro
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.without_header.csv
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.without_schema.avro
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.xml
delete mode 100644 nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/explicit.schema.json
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITConnectionAccessControl.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITCountersAccessControl.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITFlowAccessControl.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITFunnelAccessControl.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITInputPortAccessControl.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITLabelAccessControl.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITOutputPortAccessControl.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITProcessGroupAccessControl.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITProcessorAccessControl.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/anonymous/AbstractAnonymousUserTest.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/anonymous/ITAllowDirectAnonymousAccess.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/anonymous/ITAllowProxiedAnonymousAccess.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/anonymous/ITPreventDirectAnonymousAccess.java
delete mode 100644 nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/anonymous/ITPreventProxiedAnonymousAccess.java
delete mode 100644 nifi-registry/nifi-registry-core/nifi-registry-web-api/src/test/java/org/apache/nifi/registry/web/api/SecureDatabaseIT.java
(nifi) 02/05: NIFI-13214 Replaced deprecated Lucene IndexReader methods with StoredFields
Posted by ex...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
exceptionfactory pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
commit c66a3cf4ff6b531ed87b65f54fe3f73d5e810f16
Author: dan-s1 <ds...@gmail.com>
AuthorDate: Fri May 10 20:27:50 2024 +0000
NIFI-13214 Replaced deprecated Lucene IndexReader methods with StoredFields
This closes #8815
Signed-off-by: David Handermann <ex...@apache.org>
---
.../apache/nifi/provenance/index/lucene/LuceneEventIndex.java | 4 +++-
.../java/org/apache/nifi/provenance/index/lucene/QueryTask.java | 9 ++++++---
.../main/java/org/apache/nifi/provenance/lucene/DocsReader.java | 4 +++-
3 files changed, 12 insertions(+), 5 deletions(-)
diff --git a/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/index/lucene/LuceneEventIndex.java b/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/index/lucene/LuceneEventIndex.java
index 45dabfd999..60f444f988 100644
--- a/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/index/lucene/LuceneEventIndex.java
+++ b/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/index/lucene/LuceneEventIndex.java
@@ -19,6 +19,7 @@ package org.apache.nifi.provenance.index.lucene;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.StoredFields;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanClause;
import org.apache.lucene.search.BooleanQuery;
@@ -333,8 +334,9 @@ public class LuceneEventIndex implements EventIndex {
try {
final IndexReader reader = searcher.getIndexSearcher().getIndexReader();
+ final StoredFields storedFields = reader.storedFields();
final int maxDocId = reader.maxDoc() - 1;
- final Document document = reader.document(maxDocId);
+ final Document document = storedFields.document(maxDocId);
final long eventId = document.getField(SearchableFields.Identifier.getSearchableFieldName()).numericValue().longValue();
logger.info("Determined that Max Event ID indexed for Partition {} is approximately {} based on index {}", partitionName, eventId, directory);
return eventId;
diff --git a/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/index/lucene/QueryTask.java b/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/index/lucene/QueryTask.java
index 368a19dc38..a6ef58a22b 100644
--- a/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/index/lucene/QueryTask.java
+++ b/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/index/lucene/QueryTask.java
@@ -18,6 +18,7 @@
package org.apache.nifi.provenance.index.lucene;
import org.apache.lucene.index.IndexReader;
+import org.apache.lucene.index.StoredFields;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.Sort;
import org.apache.lucene.search.SortField;
@@ -125,7 +126,9 @@ public class QueryTask implements Runnable {
// Query lucene
final IndexReader indexReader = searcher.getIndexSearcher().getIndexReader();
final TopDocs topDocs;
+ final StoredFields storedFields;
try {
+ storedFields = indexReader.storedFields();
// Sort based on document id, descending. This gives us most recent events first.
final Sort sort = new Sort(new SortField(null, SortField.Type.DOC, true));
@@ -152,7 +155,7 @@ public class QueryTask implements Runnable {
return;
}
- final Tuple<List<ProvenanceEventRecord>, Long> eventsAndTotalHits = readDocuments(topDocs, indexReader);
+ final Tuple<List<ProvenanceEventRecord>, Long> eventsAndTotalHits = readDocuments(topDocs, storedFields);
if (eventsAndTotalHits == null) {
queryResult.update(Collections.emptyList(), 0L);
@@ -174,7 +177,7 @@ public class QueryTask implements Runnable {
}
}
- private Tuple<List<ProvenanceEventRecord>, Long> readDocuments(final TopDocs topDocs, final IndexReader indexReader) {
+ private Tuple<List<ProvenanceEventRecord>, Long> readDocuments(final TopDocs topDocs, final StoredFields storedFields) {
// If no topDocs is supplied, just provide a Tuple that has no records and a hit count of 0.
if (topDocs == null || topDocs.totalHits.value == 0) {
return new Tuple<>(Collections.<ProvenanceEventRecord> emptyList(), 0L);
@@ -185,7 +188,7 @@ public class QueryTask implements Runnable {
.mapToInt(scoreDoc -> scoreDoc.doc)
.mapToObj(docId -> {
try {
- return indexReader.document(docId, LUCENE_FIELDS_TO_LOAD);
+ return storedFields.document(docId, LUCENE_FIELDS_TO_LOAD);
} catch (final Exception e) {
throw new SearchFailedException("Failed to read Provenance Events from Event File", e);
}
diff --git a/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/lucene/DocsReader.java b/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/lucene/DocsReader.java
index 0e96b6248f..e62900d324 100644
--- a/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/lucene/DocsReader.java
+++ b/nifi-extension-bundles/nifi-provenance-repository-bundle/nifi-persistent-provenance-repository/src/main/java/org/apache/nifi/provenance/lucene/DocsReader.java
@@ -33,6 +33,7 @@ import java.util.concurrent.atomic.AtomicInteger;
import org.apache.lucene.document.Document;
import org.apache.lucene.index.IndexReader;
import org.apache.lucene.index.IndexableField;
+import org.apache.lucene.index.StoredFields;
import org.apache.lucene.search.ScoreDoc;
import org.apache.lucene.search.TopDocs;
import org.apache.nifi.provenance.ProvenanceEventRecord;
@@ -58,10 +59,11 @@ public class DocsReader {
final ScoreDoc[] scoreDocs = topDocs.scoreDocs;
final int numDocs = Math.min(scoreDocs.length, maxResults);
final List<Document> docs = new ArrayList<>(numDocs);
+ final StoredFields storedFields = indexReader.storedFields();
for (int i = numDocs - 1; i >= 0; i--) {
final int docId = scoreDocs[i].doc;
- final Document d = indexReader.document(docId);
+ final Document d = storedFields.document(docId);
docs.add(d);
}
(nifi) 01/05: NIFI-13215 AWS SDK and Elasticsearch Dependencies Upgraded
Posted by ex...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
exceptionfactory pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
commit 6fc374b268726326a23a3914e0baa78abae0e514
Author: Joseph Witt <jo...@apache.org>
AuthorDate: Fri May 10 13:32:32 2024 -0700
NIFI-13215 AWS SDK and Elasticsearch Dependencies Upgraded
- Upgraded AWS SDK 1 from 1.12.718 to 1.12.719
- Upgraded AWS SDK 2 from 2.25.48 to 2.25.49
- Upgraded snowflake-ingest-sdk from 2.1.0 to 2.1.1
- Upgraded org.elasticsearch.client from 8.13.3 to 8.13.4
- Upgraded software.amazon.kinesis from 2.5.8 to 2.6.0
This closes #8814
Signed-off-by: David Handermann <ex...@apache.org>
---
nifi-extension-bundles/nifi-aws-bundle/pom.xml | 2 +-
nifi-extension-bundles/nifi-elasticsearch-bundle/pom.xml | 4 ++--
nifi-extension-bundles/nifi-snowflake-bundle/pom.xml | 2 +-
pom.xml | 4 ++--
4 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/nifi-extension-bundles/nifi-aws-bundle/pom.xml b/nifi-extension-bundles/nifi-aws-bundle/pom.xml
index ecd446169a..5ff4cc5e42 100644
--- a/nifi-extension-bundles/nifi-aws-bundle/pom.xml
+++ b/nifi-extension-bundles/nifi-aws-bundle/pom.xml
@@ -27,7 +27,7 @@
<packaging>pom</packaging>
<properties>
- <aws-kinesis-client-library-version>2.5.8</aws-kinesis-client-library-version>
+ <aws-kinesis-client-library-version>2.6.0</aws-kinesis-client-library-version>
</properties>
<modules>
diff --git a/nifi-extension-bundles/nifi-elasticsearch-bundle/pom.xml b/nifi-extension-bundles/nifi-elasticsearch-bundle/pom.xml
index bb26aea04f..6b9506f9b7 100644
--- a/nifi-extension-bundles/nifi-elasticsearch-bundle/pom.xml
+++ b/nifi-extension-bundles/nifi-elasticsearch-bundle/pom.xml
@@ -33,7 +33,7 @@ language governing permissions and limitations under the License. -->
</modules>
<properties>
- <elasticsearch.client.version>8.13.3</elasticsearch.client.version>
+ <elasticsearch.client.version>8.13.4</elasticsearch.client.version>
</properties>
<dependencyManagement>
@@ -88,7 +88,7 @@ language governing permissions and limitations under the License. -->
</activation>
<properties>
<!-- also update the default Elasticsearch version in nifi-elasticsearch-test-utils#src/main/java/org/apache/nifi/elasticsearch/integration/AbstractElasticsearchITBase.java-->
- <elasticsearch_docker_image>8.13.3</elasticsearch_docker_image>
+ <elasticsearch_docker_image>8.13.4</elasticsearch_docker_image>
<elasticsearch.elastic.password>s3cret</elasticsearch.elastic.password>
</properties>
<build>
diff --git a/nifi-extension-bundles/nifi-snowflake-bundle/pom.xml b/nifi-extension-bundles/nifi-snowflake-bundle/pom.xml
index acfa2c14f3..1aebd21592 100644
--- a/nifi-extension-bundles/nifi-snowflake-bundle/pom.xml
+++ b/nifi-extension-bundles/nifi-snowflake-bundle/pom.xml
@@ -34,7 +34,7 @@
<dependency>
<groupId>net.snowflake</groupId>
<artifactId>snowflake-ingest-sdk</artifactId>
- <version>2.1.0</version>
+ <version>2.1.1</version>
</dependency>
<dependency>
<groupId>net.snowflake</groupId>
diff --git a/pom.xml b/pom.xml
index 4b05e1fb5d..d6fd13dd13 100644
--- a/pom.xml
+++ b/pom.xml
@@ -109,8 +109,8 @@
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<inceptionYear>2014</inceptionYear>
- <com.amazonaws.version>1.12.718</com.amazonaws.version>
- <software.amazon.awssdk.version>2.25.48</software.amazon.awssdk.version>
+ <com.amazonaws.version>1.12.719</com.amazonaws.version>
+ <software.amazon.awssdk.version>2.25.49</software.amazon.awssdk.version>
<gson.version>2.10.1</gson.version>
<io.fabric8.kubernetes.client.version>6.12.1</io.fabric8.kubernetes.client.version>
<kotlin.version>1.9.24</kotlin.version>
(nifi) 05/05: NIFI-13217 Updated ProcessSession Comments to List Immutable Attributes
Posted by ex...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
exceptionfactory pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
commit 15696ad86a2716c20d3f1889daa1f23f6e11298b
Author: EndzeitBegins <16...@users.noreply.github.com>
AuthorDate: Sat May 11 14:25:08 2024 +0200
NIFI-13217 Updated ProcessSession Comments to List Immutable Attributes
This closes #8819
Signed-off-by: David Handermann <ex...@apache.org>
---
.../src/main/java/org/apache/nifi/processor/ProcessSession.java | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/nifi-api/src/main/java/org/apache/nifi/processor/ProcessSession.java b/nifi-api/src/main/java/org/apache/nifi/processor/ProcessSession.java
index b3a33dcb3f..e6516ca68e 100644
--- a/nifi-api/src/main/java/org/apache/nifi/processor/ProcessSession.java
+++ b/nifi-api/src/main/java/org/apache/nifi/processor/ProcessSession.java
@@ -430,7 +430,8 @@ public interface ProcessSession {
/**
* Removes the attribute with the given {@code key} from the given {@link FlowFile}.
* <p>
- * If the {@code key} is named {@code uuid}, this method will return the same FlowFile without removing any attribute.
+ * The attributes with the keys {@code uuid}, {@code path}, and {@code filename} will not be removed.
+ * If the {@code key} is one of those, this method will return the same FlowFile without removing any attribute.
*
* @param flowFile to update
* @param key of attribute to remove
@@ -447,7 +448,7 @@ public interface ProcessSession {
/**
* Removes the attributes with the given {@code keys} from the given {@link FlowFile}.
* <p>
- * If the set of keys contains the value {@code uuid}, this key will be ignored.
+ * The attributes with the keys {@code uuid}, {@code path}, and {@code filename} will not be removed.
*
* @param flowFile to update
* @param keys of attributes to remove
@@ -464,7 +465,7 @@ public interface ProcessSession {
/**
* Removes all attributes from the given {@link FlowFile} whose key matches the given pattern.
* <p>
- * If the pattern matches the key {@code uuid}, this key will not be removed.
+ * The attributes with the keys {@code uuid}, {@code path}, and {@code filename} will not be removed.
*
* @param flowFile to update
* @param keyPattern pattern to match each {@link FlowFile} attribute against; may be null, in which case no attribute is removed
(nifi) 04/05: NIFI-13185 Removed manual and non-working Integration Tests
Posted by ex...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
exceptionfactory pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
commit 03005aa13201f4b140751934799140e4c1a1749f
Author: Joseph Witt <jo...@apache.org>
AuthorDate: Fri May 10 16:30:29 2024 -0700
NIFI-13185 Removed manual and non-working Integration Tests
Removed the entire integration-tests-ci profile and reference in GitHub workflow.
Removed all tests which require specific local env configs/cloud credentials/ or simply do not work and thus have been blocked from the build for a long time.
Set Disabled annotation on a couple key tests which are by design manual and must be retained.
The move to testcontainers has been a dramatic improvement in test reliability/behaviors and should be a model for all such tests going forward.
This closes #8818
Signed-off-by: David Handermann <ex...@apache.org>
---
.github/workflows/integration-tests.yml | 1 -
...andardHashiCorpVaultCommunicationServiceIT.java | 126 ----
.../AwsKmsSensitivePropertyProviderIT.java | 142 ----
...sSecretsManagerSensitivePropertyProviderIT.java | 133 ----
...zureKeyVaultKeySensitivePropertyProviderIT.java | 135 ----
.../GcpKmsSensitivePropertyProviderIT.java | 134 ----
.../aws/kinesis/firehose/ITPutKinesisFirehose.java | 286 --------
.../stream/ITConsumeKinesisStreamConnectAWS.java | 72 --
.../ITConsumeKinesisStreamEndpointOverride.java | 88 ---
.../aws/kinesis/stream/ITPutKinesisStream.java | 421 -----------
.../ITPutKinesisStreamWithEndpointOverride.java | 84 ---
.../nifi/processors/aws/lambda/ITPutLambda.java | 137 ----
.../apache/nifi/processors/aws/sns/ITPutSNS.java | 78 --
.../azure/AzureGraphUserGroupProviderIT.java | 219 ------
.../cosmos/document/ITPutAzureCosmosDBRecord.java | 160 -----
.../storage/AbstractAzureBlobStorage_v12IT.java | 195 -----
.../storage/AbstractAzureDataLakeStorageIT.java | 151 ----
.../azure/storage/AbstractAzureStorageIT.java | 132 ----
.../azure/storage/ITCopyAzureBlobStorage_v12.java | 122 ----
.../storage/ITDeleteAzureBlobStorage_v12.java | 194 -----
.../storage/ITDeleteAzureDataLakeStorage.java | 501 -------------
.../azure/storage/ITFetchAzureBlobStorage_v12.java | 232 ------
.../azure/storage/ITFetchAzureDataLakeStorage.java | 558 ---------------
.../azure/storage/ITListAzureBlobStorage_v12.java | 223 ------
.../azure/storage/ITListAzureDataLakeStorage.java | 534 --------------
.../azure/storage/ITMoveAzureDataLakeStorage.java | 374 ----------
.../azure/storage/ITPutAzureBlobStorage_v12.java | 385 ----------
.../azure/storage/ITPutAzureDataLakeStorage.java | 434 -----------
.../nifi/processors/box/AbstractBoxFileIT.java | 137 ----
.../apache/nifi/processors/box/FetchBoxFileIT.java | 137 ----
.../apache/nifi/processors/box/ListBoxFileIT.java | 143 ----
.../apache/nifi/processors/box/PutBoxFileIT.java | 120 ----
.../JsonConfigBasedBoxClientServiceIT.java | 87 ---
.../nifi/processors/dropbox/AbstractDropboxIT.java | 128 ----
.../nifi/processors/dropbox/FetchDropboxIT.java | 132 ----
.../nifi/processors/dropbox/ListDropboxIT.java | 136 ----
.../nifi/processors/dropbox/PutDropboxIT.java | 157 ----
.../GremlinClientServiceControllerSettingsIT.java | 81 ---
...mlinClientServiceYamlSettingsAndBytecodeIT.java | 133 ----
.../jms/processors/JMSPublisherConsumerIT.java | 58 --
.../apache/nifi/processors/mongodb/GetMongoIT.java | 76 --
.../processors/mongodb/gridfs/PutGridFSIT.java | 11 -
.../prometheus/PrometheusReportingTaskIT.java | 18 -
.../ITRedisDistributedMapCacheClientService.java | 486 -------------
.../nifi/redis/state/ITRedisStateProvider.java | 3 +-
.../salesforce/PutSalesforceObjectIT.java | 135 ----
.../salesforce/QuerySalesforceObjectIT.java | 99 ---
.../salesforce/util/SalesforceRestServiceIT.java | 78 --
.../nifi/processors/snowflake/SnowflakePipeIT.java | 122 ----
.../service/SnowflakeConnectionPoolIT.java | 87 ---
.../processors/standard/AbstractConversionIT.java | 388 ----------
.../standard/ConversionWithExplicitSchemaIT.java | 88 ---
.../standard/ConversionWithSchemaInferenceIT.java | 51 --
.../TestConversions/data.int_float_string.json | 22 -
.../data.int_float_string.with_header.csv | 8 -
.../data.int_float_string.with_schema.avro | Bin 302 -> 0 bytes
.../data.int_float_string.with_schema.json.to.avro | Bin 320 -> 0 bytes
.../data.int_float_string.without_header.csv | 7 -
.../data.int_float_string.without_schema.avro | Bin 51 -> 0 bytes
.../TestConversions/data.int_float_string.xml | 31 -
.../resources/TestConversions/explicit.schema.json | 23 -
.../repository/StandardProcessSessionIT.java | 2 +
.../accesscontrol/ITConnectionAccessControl.java | 427 -----------
.../accesscontrol/ITCountersAccessControl.java | 84 ---
.../accesscontrol/ITFlowAccessControl.java | 276 -------
.../accesscontrol/ITFunnelAccessControl.java | 359 ----------
.../accesscontrol/ITInputPortAccessControl.java | 403 -----------
.../accesscontrol/ITLabelAccessControl.java | 400 -----------
.../accesscontrol/ITOutputPortAccessControl.java | 403 -----------
.../accesscontrol/ITProcessGroupAccessControl.java | 465 ------------
.../accesscontrol/ITProcessorAccessControl.java | 749 -------------------
.../anonymous/AbstractAnonymousUserTest.java | 57 --
.../anonymous/ITAllowDirectAnonymousAccess.java | 67 --
.../anonymous/ITAllowProxiedAnonymousAccess.java | 67 --
.../anonymous/ITPreventDirectAnonymousAccess.java | 57 --
.../anonymous/ITPreventProxiedAnonymousAccess.java | 58 --
.../nifi/registry/web/api/SecureDatabaseIT.java | 237 ------
.../web/api/SecureNiFiRegistryClientIT.java | 163 +----
.../nifi/registry/web/api/SecureProxyIT.java | 75 --
.../web/api/UnsecuredNiFiRegistryClientIT.java | 796 +--------------------
pom.xml | 110 ---
81 files changed, 5 insertions(+), 14683 deletions(-)
diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml
index 19daacc3cb..e711901459 100644
--- a/.github/workflows/integration-tests.yml
+++ b/.github/workflows/integration-tests.yml
@@ -62,7 +62,6 @@ env:
verify
-P skip-unit-tests
-P integration-tests
- -P integration-tests-ci
-P nifi-registry-integration-tests
MAVEN_BUILD_EXCLUDE_PROJECTS: >-
-pl -:minifi-assembly
diff --git a/nifi-commons/nifi-hashicorp-vault/src/test/java/org/apache/nifi/vault/hashicorp/StandardHashiCorpVaultCommunicationServiceIT.java b/nifi-commons/nifi-hashicorp-vault/src/test/java/org/apache/nifi/vault/hashicorp/StandardHashiCorpVaultCommunicationServiceIT.java
deleted file mode 100644
index 2080e147a8..0000000000
--- a/nifi-commons/nifi-hashicorp-vault/src/test/java/org/apache/nifi/vault/hashicorp/StandardHashiCorpVaultCommunicationServiceIT.java
+++ /dev/null
@@ -1,126 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.vault.hashicorp;
-
-import org.apache.nifi.vault.hashicorp.config.HashiCorpVaultProperties;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.nio.charset.StandardCharsets;
-import java.util.HashMap;
-import java.util.Map;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-/**
- * The simplest way to run this test is by installing Vault locally, then running:
- *
- * vault server -dev
- * vault secrets enable transit
- * vault secrets enable kv
- * vault secrets enable kv-v2
- * vault write -f transit/keys/nifi
- *
- * Make note of the Root Token and create a properties file with the contents:
- * vault.token=[Root Token]
- *
- * Then, set the system property -Dvault.auth.properties to the file path of the above properties file when
- * running the integration test.
- */
-public class StandardHashiCorpVaultCommunicationServiceIT {
-
- private static final String TRANSIT_KEY = "nifi";
-
- private HashiCorpVaultCommunicationService vcs;
-
- @BeforeEach
- public void init() {
- vcs = new StandardHashiCorpVaultCommunicationService(defaultServiceBuilder().build());
- }
-
- private HashiCorpVaultProperties.HashiCorpVaultPropertiesBuilder defaultServiceBuilder() {
- return new HashiCorpVaultProperties.HashiCorpVaultPropertiesBuilder()
- .setAuthPropertiesFilename(System.getProperty("vault.auth.properties"))
- .setUri("http://127.0.0.1:8200");
- }
-
- @Test
- public void testEncryptDecrypt() {
- this.runEncryptDecryptTest();
- }
-
- public void runEncryptDecryptTest() {
- String plaintext = "this is the plaintext";
-
- String ciphertext = vcs.encrypt(TRANSIT_KEY, plaintext.getBytes(StandardCharsets.UTF_8));
-
- byte[] decrypted = vcs.decrypt(TRANSIT_KEY, ciphertext);
-
- assertEquals(plaintext, new String(decrypted, StandardCharsets.UTF_8));
- }
-
- /**
- * Run <code>vault kv get kv/key</code> to see the secret
- */
- @Test
- public void testReadWriteSecret() {
- final String key = "key";
- final String value = "value";
-
- vcs.writeKeyValueSecret("kv", key, value);
-
- final String resultValue = vcs.readKeyValueSecret("kv", key).orElseThrow(() -> new NullPointerException("Missing secret for kv/key"));
- assertEquals(value, resultValue);
- }
-
- /**
- * Run <code>vault kv get kv_v2/key</code> to see the secret
- */
- @Test
- public void testReadWriteSecret_kv_v2() {
- final String key = "key";
- final String value = "value";
-
- vcs = new StandardHashiCorpVaultCommunicationService(defaultServiceBuilder().setKvVersion(2).build());
-
- vcs.writeKeyValueSecret("kv-v2", key, value);
-
- final String resultValue = vcs.readKeyValueSecret("kv-v2", key).orElseThrow(() -> new NullPointerException("Missing secret for kv/key"));
- assertEquals(value, resultValue);
- }
-
- /**
- * Run <code>vault kv get kv/secret</code> to see the secret
- */
- @Test
- public void testReadWriteSecretMap() {
- final String secretKey = "secret";
- final String key = "key";
- final String value = "value";
- final String key2 = "key2";
- final String value2 = "value2";
-
- final Map<String, String> keyValues = new HashMap<>();
- keyValues.put(key, value);
- keyValues.put(key2, value2);
-
- vcs.writeKeyValueSecretMap("kv", secretKey, keyValues);
-
- final Map<String, String> resultMap = vcs.readKeyValueSecretMap("kv", secretKey);
- assertEquals(keyValues, resultMap);
- }
-}
diff --git a/nifi-commons/nifi-property-protection-aws/src/test/java/org/apache/nifi/properties/AwsKmsSensitivePropertyProviderIT.java b/nifi-commons/nifi-property-protection-aws/src/test/java/org/apache/nifi/properties/AwsKmsSensitivePropertyProviderIT.java
deleted file mode 100644
index 810bea825c..0000000000
--- a/nifi-commons/nifi-property-protection-aws/src/test/java/org/apache/nifi/properties/AwsKmsSensitivePropertyProviderIT.java
+++ /dev/null
@@ -1,142 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.properties;
-
-import org.apache.nifi.properties.configuration.AwsKmsClientProvider;
-import org.junit.jupiter.api.AfterAll;
-import org.junit.jupiter.api.BeforeAll;
-import org.junit.jupiter.api.Test;
-import org.mockito.internal.util.io.IOUtil;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import software.amazon.awssdk.services.kms.KmsClient;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.Properties;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertNotEquals;
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-/**
- * To run this test, make sure to first configure sensitive credential information as in the following link
- * https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
- *
- * If you don't have a key then run:
- * aws kms create-key
- *
- * Take note of the key id or arn.
- *
- * Then, set the system property -Daws.kms.key.id to the either key id value or arn value
- *
- * The following settings are optional. If you have a default AWS configuration and credentials in ~/.aws then
- * it will take that. Otherwise you can set all of the following:
- * set the system property -Daws.access.key.id to the access key id
- * set the system property -Daws.secret.access.key to the secret access key
- * set the system property -Daws.region to the region
- *
- * After you are satisfied with the test, and you don't need the key, you may schedule key deletion with:
- * aws kms schedule-key-deletion --key-id "key id" --pending-window-in-days "number of days"
- *
- */
-
-public class AwsKmsSensitivePropertyProviderIT {
- private static final String SAMPLE_PLAINTEXT = "AWSKMSSensitivePropertyProviderIT SAMPLE-PLAINTEXT";
- private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
- private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
- private static final String REGION_KEY_PROPS_NAME = "aws.region";
- private static final String KMS_KEY_PROPS_NAME = "aws.kms.key.id";
-
- private static final String BOOTSTRAP_AWS_FILE_PROPS_NAME = "nifi.bootstrap.protection.aws.conf";
-
- private static final String EMPTY_PROPERTY = "";
-
- private static AwsKmsSensitivePropertyProvider spp;
-
- private static BootstrapProperties props;
-
- private static Path mockBootstrapConf, mockAWSBootstrapConf;
-
- private static final Logger logger = LoggerFactory.getLogger(AwsKmsSensitivePropertyProviderIT.class);
-
- private static void initializeBootstrapProperties() throws IOException{
- mockBootstrapConf = Files.createTempFile("bootstrap", ".conf").toAbsolutePath();
- mockAWSBootstrapConf = Files.createTempFile("bootstrap-aws", ".conf").toAbsolutePath();
- IOUtil.writeText(BOOTSTRAP_AWS_FILE_PROPS_NAME + "=" + mockAWSBootstrapConf.toAbsolutePath(), mockBootstrapConf.toFile());
-
- final Properties bootstrapProperties = new Properties();
- try (final InputStream inputStream = Files.newInputStream(mockBootstrapConf)) {
- bootstrapProperties.load(inputStream);
- props = new BootstrapProperties("nifi", bootstrapProperties, mockBootstrapConf);
- }
-
- String accessKey = System.getProperty(ACCESS_KEY_PROPS_NAME, EMPTY_PROPERTY);
- String secretKey = System.getProperty(SECRET_KEY_PROPS_NAME, EMPTY_PROPERTY);
- String region = System.getProperty(REGION_KEY_PROPS_NAME, EMPTY_PROPERTY);
- String keyId = System.getProperty(KMS_KEY_PROPS_NAME, EMPTY_PROPERTY);
-
- StringBuilder bootstrapConfText = new StringBuilder();
- String lineSeparator = System.getProperty("line.separator");
- bootstrapConfText.append(ACCESS_KEY_PROPS_NAME + "=" + accessKey);
- bootstrapConfText.append(lineSeparator + SECRET_KEY_PROPS_NAME + "=" + secretKey);
- bootstrapConfText.append(lineSeparator + REGION_KEY_PROPS_NAME + "=" + region);
- bootstrapConfText.append(lineSeparator + KMS_KEY_PROPS_NAME + "=" + keyId);
-
- IOUtil.writeText(bootstrapConfText.toString(), mockAWSBootstrapConf.toFile());
- }
-
- @BeforeAll
- public static void initOnce() throws IOException {
- initializeBootstrapProperties();
- assertNotNull(props);
- final AwsKmsClientProvider provider = new AwsKmsClientProvider();
- final Properties properties = provider.getClientProperties(props).orElse(null);
- final KmsClient kmsClient = provider.getClient(properties).orElse(null);
- spp = new AwsKmsSensitivePropertyProvider(kmsClient, properties);
- assertNotNull(spp);
- }
-
- @AfterAll
- public static void tearDownOnce() throws IOException {
- Files.deleteIfExists(mockBootstrapConf);
- Files.deleteIfExists(mockAWSBootstrapConf);
-
- spp.cleanUp();
- }
-
- @Test
- public void testEncryptDecrypt() {
- logger.info("Running testEncryptDecrypt of AWS KMS SPP integration test");
- runEncryptDecryptTest();
- logger.info("testEncryptDecrypt of AWS KMS SPP integration test completed");
- }
-
- private static void runEncryptDecryptTest() {
- logger.info("Plaintext: " + SAMPLE_PLAINTEXT);
- String protectedValue = spp.protect(SAMPLE_PLAINTEXT, ProtectedPropertyContext.defaultContext("property"));
- logger.info("Protected Value: " + protectedValue);
- String unprotectedValue = spp.unprotect(protectedValue, ProtectedPropertyContext.defaultContext("property"));
- logger.info("Unprotected Value: " + unprotectedValue);
-
- assertEquals(SAMPLE_PLAINTEXT, unprotectedValue);
- assertNotEquals(SAMPLE_PLAINTEXT, protectedValue);
- assertNotEquals(protectedValue, unprotectedValue);
- }
-}
diff --git a/nifi-commons/nifi-property-protection-aws/src/test/java/org/apache/nifi/properties/AwsSecretsManagerSensitivePropertyProviderIT.java b/nifi-commons/nifi-property-protection-aws/src/test/java/org/apache/nifi/properties/AwsSecretsManagerSensitivePropertyProviderIT.java
deleted file mode 100644
index 3a21561d7e..0000000000
--- a/nifi-commons/nifi-property-protection-aws/src/test/java/org/apache/nifi/properties/AwsSecretsManagerSensitivePropertyProviderIT.java
+++ /dev/null
@@ -1,133 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.properties;
-
-import org.apache.nifi.properties.configuration.AwsSecretsManagerClientProvider;
-import org.junit.jupiter.api.AfterAll;
-import org.junit.jupiter.api.BeforeAll;
-import org.junit.jupiter.api.Test;
-import org.mockito.internal.util.io.IOUtil;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-import software.amazon.awssdk.services.secretsmanager.SecretsManagerClient;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.Properties;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertNotEquals;
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-/**
- * To run this test, make sure to first configure sensitive credential information as in the following link
- * https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
- *
- * The following settings are optional. If you have a default AWS configuration and credentials in ~/.aws then
- * it will take that. Otherwise you can set all of the following:
- * set the system property -Daws.access.key.id to the access key id
- * set the system property -Daws.secret.access.key to the secret access key
- * set the system property -Daws.region to the region
- *
- * After you are satisfied with the test, and you don't need the key, you may schedule secret deletion with:
- * aws secretsmanager delete-secret --secret-id "default/property" --recovery-window-in-days 14
- *
- */
-
-public class AwsSecretsManagerSensitivePropertyProviderIT {
- private static final String SAMPLE_PLAINTEXT = "AWSSecretsManagerSensitivePropertyProviderIT SAMPLE-PLAINTEXT";
- private static final String ACCESS_KEY_PROPS_NAME = "aws.access.key.id";
- private static final String SECRET_KEY_PROPS_NAME = "aws.secret.access.key";
- private static final String REGION_KEY_PROPS_NAME = "aws.region";
-
- private static final String BOOTSTRAP_AWS_FILE_PROPS_NAME = "nifi.bootstrap.protection.aws.conf";
-
- private static final String EMPTY_PROPERTY = "";
-
- private static AwsSecretsManagerSensitivePropertyProvider spp;
-
- private static BootstrapProperties props;
-
- private static Path mockBootstrapConf, mockAWSBootstrapConf;
-
- private static final Logger logger = LoggerFactory.getLogger(AwsSecretsManagerSensitivePropertyProviderIT.class);
-
- private static void initializeBootstrapProperties() throws IOException{
- mockBootstrapConf = Files.createTempFile("bootstrap", ".conf").toAbsolutePath();
- mockAWSBootstrapConf = Files.createTempFile("bootstrap-aws", ".conf").toAbsolutePath();
- IOUtil.writeText(BOOTSTRAP_AWS_FILE_PROPS_NAME + "=" + mockAWSBootstrapConf.toAbsolutePath(), mockBootstrapConf.toFile());
-
- final Properties bootstrapProperties = new Properties();
- try (final InputStream inputStream = Files.newInputStream(mockBootstrapConf)) {
- bootstrapProperties.load(inputStream);
- props = new BootstrapProperties("nifi", bootstrapProperties, mockBootstrapConf);
- }
-
- String accessKey = System.getProperty(ACCESS_KEY_PROPS_NAME, EMPTY_PROPERTY);
- String secretKey = System.getProperty(SECRET_KEY_PROPS_NAME, EMPTY_PROPERTY);
- String region = System.getProperty(REGION_KEY_PROPS_NAME, EMPTY_PROPERTY);
-
- StringBuilder bootstrapConfText = new StringBuilder();
- String lineSeparator = System.getProperty("line.separator");
- bootstrapConfText.append(ACCESS_KEY_PROPS_NAME + "=" + accessKey);
- bootstrapConfText.append(lineSeparator + SECRET_KEY_PROPS_NAME + "=" + secretKey);
- bootstrapConfText.append(lineSeparator + REGION_KEY_PROPS_NAME + "=" + region);
-
- IOUtil.writeText(bootstrapConfText.toString(), mockAWSBootstrapConf.toFile());
- }
-
- @BeforeAll
- public static void initOnce() throws IOException {
- initializeBootstrapProperties();
- assertNotNull(props);
- final AwsSecretsManagerClientProvider provider = new AwsSecretsManagerClientProvider();
- final Properties properties = provider.getClientProperties(props).orElse(null);
- final SecretsManagerClient secretsManagerClient = provider.getClient(properties).orElse(null);
- spp = new AwsSecretsManagerSensitivePropertyProvider(secretsManagerClient);
- assertNotNull(spp);
- }
-
- @AfterAll
- public static void tearDownOnce() throws IOException {
- Files.deleteIfExists(mockBootstrapConf);
- Files.deleteIfExists(mockAWSBootstrapConf);
-
- spp.cleanUp();
- }
-
- @Test
- public void testEncryptDecrypt() {
- logger.info("Running testEncryptDecrypt of AWS Secrets Manager SPP integration test");
- runEncryptDecryptTest();
- logger.info("testEncryptDecrypt of AWS Secrets Manager SPP integration test completed");
- }
-
- private static void runEncryptDecryptTest() {
- final String propertyName = "property2";
- logger.info("Plaintext: " + SAMPLE_PLAINTEXT);
- String protectedValue = spp.protect(SAMPLE_PLAINTEXT, ProtectedPropertyContext.defaultContext(propertyName));
- logger.info("Protected Value: " + protectedValue);
- String unprotectedValue = spp.unprotect(protectedValue, ProtectedPropertyContext.defaultContext(propertyName));
- logger.info("Unprotected Value: " + unprotectedValue);
-
- assertEquals(SAMPLE_PLAINTEXT, unprotectedValue);
- assertNotEquals(SAMPLE_PLAINTEXT, protectedValue);
- assertNotEquals(protectedValue, unprotectedValue);
- }
-}
diff --git a/nifi-commons/nifi-property-protection-azure/src/test/java/org/apache/nifi/properties/AzureKeyVaultKeySensitivePropertyProviderIT.java b/nifi-commons/nifi-property-protection-azure/src/test/java/org/apache/nifi/properties/AzureKeyVaultKeySensitivePropertyProviderIT.java
deleted file mode 100644
index bd2e7e7e03..0000000000
--- a/nifi-commons/nifi-property-protection-azure/src/test/java/org/apache/nifi/properties/AzureKeyVaultKeySensitivePropertyProviderIT.java
+++ /dev/null
@@ -1,135 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.properties;
-
-import com.azure.security.keyvault.keys.cryptography.CryptographyClient;
-import org.apache.nifi.properties.configuration.AzureCryptographyClientProvider;
-import org.junit.jupiter.api.AfterAll;
-import org.junit.jupiter.api.BeforeAll;
-import org.junit.jupiter.api.Test;
-import org.mockito.internal.util.io.IOUtil;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.Properties;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertNotEquals;
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-/**
- * For the purposes of running this test, if you have a key, first make sure you have configured
- * credential information as in the following link
- * https://docs.microsoft.com/en-us/java/api/overview/azure/security-keyvault-keys-readme?view=azure-java-stable
- *
- * Then simply set the system property -Dazure.keyvault.key.id to the key identifier
- * when running the integration test.
- * The key identifier takes the form: "https://keyvault-name.vault.azure.net/keys/key-name/keyID"
- *
- * Otherwise, to create a key, it will be simplest through the Azure web portal. First create a resource group,
- * then a key vault tied to that resource group. Then you can create a key through the key vault.
- *
- * Then simply set the system property -Dazure.keyvault.key.id to the key identifier
- * when running the integration test.
- * The key identifier takes the form: "https://keyvault-name.vault.azure.net/keys/key-name/keyID"
- *
- * Another system property that needs to be set is the algorithm: set it as
- * -Dazure.keyvault.encryption.algorithm=algorithm
- * Available algorithms can be found in:
- * https://docs.microsoft.com/en-us/java/api/com.azure.security.keyvault.keys.cryptography.models.encryptionalgorithm
- */
-
-public class AzureKeyVaultKeySensitivePropertyProviderIT {
- private static final String SAMPLE_PLAINTEXT = "AzureKeyVaultKeySensitivePropertyProviderIT SAMPLE-PLAINTEXT";
- private static final String KEYVAULT_KEY_PROPS_NAME = "azure.keyvault.key.id";
- private static final String ENCRYPTION_ALGORITHM_PROPS_NAME = "azure.keyvault.encryption.algorithm";
-
- private static final String BOOTSTRAP_AZURE_FILE_PROPS_NAME = "nifi.bootstrap.protection.azure.keyvault.conf";
-
- private static final String EMPTY_PROPERTY = "";
-
- private static AzureKeyVaultKeySensitivePropertyProvider spp;
-
- private static BootstrapProperties props;
-
- private static Path mockBootstrapConf, mockAzureBootstrapConf;
-
- private static final Logger logger = LoggerFactory.getLogger(AzureKeyVaultKeySensitivePropertyProviderIT.class);
-
- private static void initializeBootstrapProperties() throws IOException {
- mockBootstrapConf = Files.createTempFile("bootstrap", ".conf").toAbsolutePath();
- mockAzureBootstrapConf = Files.createTempFile("bootstrap-azure", ".conf").toAbsolutePath();
- IOUtil.writeText(BOOTSTRAP_AZURE_FILE_PROPS_NAME + "=" + mockAzureBootstrapConf.toAbsolutePath(), mockBootstrapConf.toFile());
-
- final Properties bootstrapProperties = new Properties();
- try (final InputStream inputStream = Files.newInputStream(mockBootstrapConf)) {
- bootstrapProperties.load(inputStream);
- props = new BootstrapProperties("nifi", bootstrapProperties, mockBootstrapConf);
- }
-
- String keyId = System.getProperty(KEYVAULT_KEY_PROPS_NAME, EMPTY_PROPERTY);
- String algorithm = System.getProperty(ENCRYPTION_ALGORITHM_PROPS_NAME, EMPTY_PROPERTY);
-
- StringBuilder bootstrapConfText = new StringBuilder();
- String lineSeparator = System.getProperty("line.separator");
- bootstrapConfText.append(KEYVAULT_KEY_PROPS_NAME + "=" + keyId);
- bootstrapConfText.append(lineSeparator + ENCRYPTION_ALGORITHM_PROPS_NAME + "=" + algorithm);
- IOUtil.writeText(bootstrapConfText.toString(), mockAzureBootstrapConf.toFile());
- }
-
- @BeforeAll
- public static void initOnce() throws IOException {
- initializeBootstrapProperties();
- assertNotNull(props);
- final AzureCryptographyClientProvider provider = new AzureCryptographyClientProvider();
- final Properties properties = provider.getClientProperties(props).orElse(null);
- final CryptographyClient cryptographyClient = provider.getClient(properties).orElse(null);
- spp = new AzureKeyVaultKeySensitivePropertyProvider(cryptographyClient, properties);
- assertNotNull(spp);
- }
-
- @AfterAll
- public static void tearDownOnce() throws IOException {
- Files.deleteIfExists(mockBootstrapConf);
- Files.deleteIfExists(mockAzureBootstrapConf);
-
- spp.cleanUp();
- }
-
- @Test
- public void testEncryptDecrypt() {
- logger.info("Running testEncryptDecrypt of Azure Key Vault Key SPP integration test");
- runEncryptDecryptTest();
- logger.info("testEncryptDecrypt of Azure Key Vault Key SPP integration test completed");
- }
-
- private static void runEncryptDecryptTest() {
- logger.info("Plaintext: " + SAMPLE_PLAINTEXT);
- String protectedValue = spp.protect(SAMPLE_PLAINTEXT, ProtectedPropertyContext.defaultContext("property"));
- logger.info("Protected Value: " + protectedValue);
- String unprotectedValue = spp.unprotect(protectedValue, ProtectedPropertyContext.defaultContext("property"));
- logger.info("Unprotected Value: " + unprotectedValue);
-
- assertEquals(SAMPLE_PLAINTEXT, unprotectedValue);
- assertNotEquals(SAMPLE_PLAINTEXT, protectedValue);
- assertNotEquals(protectedValue, unprotectedValue);
- }
-}
diff --git a/nifi-commons/nifi-property-protection-gcp/src/test/java/org/apache/nifi/properties/GcpKmsSensitivePropertyProviderIT.java b/nifi-commons/nifi-property-protection-gcp/src/test/java/org/apache/nifi/properties/GcpKmsSensitivePropertyProviderIT.java
deleted file mode 100644
index a448381018..0000000000
--- a/nifi-commons/nifi-property-protection-gcp/src/test/java/org/apache/nifi/properties/GcpKmsSensitivePropertyProviderIT.java
+++ /dev/null
@@ -1,134 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.properties;
-
-import com.google.cloud.kms.v1.KeyManagementServiceClient;
-import org.apache.nifi.properties.configuration.GoogleKeyManagementServiceClientProvider;
-import org.junit.jupiter.api.AfterAll;
-import org.junit.jupiter.api.BeforeAll;
-import org.junit.jupiter.api.Test;
-import org.mockito.internal.util.io.IOUtil;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.IOException;
-import java.io.InputStream;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.Properties;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertNotEquals;
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-/**
- * To run this test, make sure to first configure sensitive credential information as in the following link
- * https://cloud.google.com/kms/docs/reference/libraries#cloud-console
- *
- * Create a project, keyring and key in the web console.
- *
- * Take note of the project name, location, keyring name and key name.
- *
- * Then, set the system properties as follows:
- * -Dgcp.kms.project="project"
- * -Dgcp.kms.location="location"
- * -Dgcp.kms.keyring="key ring name"
- * -Dgcp.kms.key="key name"
- * when running the integration tests
- */
-
-public class GcpKmsSensitivePropertyProviderIT {
- private static final String SAMPLE_PLAINTEXT = "GCPKMSSensitivePropertyProviderIT SAMPLE-PLAINTEXT";
- private static final String PROJECT_ID_PROPS_NAME = "gcp.kms.project";
- private static final String LOCATION_ID_PROPS_NAME = "gcp.kms.location";
- private static final String KEYRING_ID_PROPS_NAME = "gcp.kms.keyring";
- private static final String KEY_ID_PROPS_NAME = "gcp.kms.key";
- private static final String BOOTSTRAP_GCP_FILE_PROPS_NAME = "nifi.bootstrap.protection.gcp.kms.conf";
-
- private static final String EMPTY_PROPERTY = "";
-
- private static GcpKmsSensitivePropertyProvider spp;
-
- private static BootstrapProperties props;
-
- private static Path mockBootstrapConf, mockGCPBootstrapConf;
-
- private static final Logger logger = LoggerFactory.getLogger(GcpKmsSensitivePropertyProviderIT.class);
-
- private static void initializeBootstrapProperties() throws IOException{
- mockBootstrapConf = Files.createTempFile("bootstrap", ".conf").toAbsolutePath();
- mockGCPBootstrapConf = Files.createTempFile("bootstrap-gcp", ".conf").toAbsolutePath();
- IOUtil.writeText(BOOTSTRAP_GCP_FILE_PROPS_NAME + "=" + mockGCPBootstrapConf.toAbsolutePath(), mockBootstrapConf.toFile());
-
- final Properties bootstrapProperties = new Properties();
- try (final InputStream inputStream = Files.newInputStream(mockBootstrapConf)) {
- bootstrapProperties.load(inputStream);
- props = new BootstrapProperties("nifi", bootstrapProperties, mockBootstrapConf);
- }
-
- String projectId = System.getProperty(PROJECT_ID_PROPS_NAME, EMPTY_PROPERTY);
- String locationId = System.getProperty(LOCATION_ID_PROPS_NAME, EMPTY_PROPERTY);
- String keyringId = System.getProperty(KEYRING_ID_PROPS_NAME, EMPTY_PROPERTY);
- String keyId = System.getProperty(KEY_ID_PROPS_NAME, EMPTY_PROPERTY);
-
- StringBuilder bootstrapConfText = new StringBuilder();
- String lineSeparator = System.lineSeparator();
- bootstrapConfText.append(PROJECT_ID_PROPS_NAME).append("=").append(projectId).append(lineSeparator);
- bootstrapConfText.append(LOCATION_ID_PROPS_NAME).append("=").append(locationId).append(lineSeparator);
- bootstrapConfText.append(KEYRING_ID_PROPS_NAME).append("=").append(keyringId).append(lineSeparator);
- bootstrapConfText.append(KEY_ID_PROPS_NAME).append("=").append(keyId).append(lineSeparator);
- IOUtil.writeText(bootstrapConfText.toString(), mockGCPBootstrapConf.toFile());
- }
-
- @BeforeAll
- public static void initOnce() throws IOException {
- initializeBootstrapProperties();
- assertNotNull(props);
- final GoogleKeyManagementServiceClientProvider provider = new GoogleKeyManagementServiceClientProvider();
- final Properties clientProperties = provider.getClientProperties(props).orElse(null);
- final KeyManagementServiceClient client = provider.getClient(clientProperties).orElse(null);
- spp = new GcpKmsSensitivePropertyProvider(client, clientProperties);
- assertNotNull(spp);
- }
-
- @AfterAll
- public static void tearDownOnce() throws IOException {
- Files.deleteIfExists(mockBootstrapConf);
- Files.deleteIfExists(mockGCPBootstrapConf);
-
- spp.cleanUp();
- }
-
- @Test
- public void testEncryptDecrypt() {
- logger.info("Running testEncryptDecrypt of GCP KMS SPP integration test");
- runEncryptDecryptTest();
- logger.info("testEncryptDecrypt of GCP KMS SPP integration test completed");
- }
-
- private static void runEncryptDecryptTest() {
- logger.info("Plaintext: " + SAMPLE_PLAINTEXT);
- String protectedValue = spp.protect(SAMPLE_PLAINTEXT, ProtectedPropertyContext.defaultContext("property"));
- logger.info("Protected Value: " + protectedValue);
- String unprotectedValue = spp.unprotect(protectedValue, ProtectedPropertyContext.defaultContext("property"));
- logger.info("Unprotected Value: " + unprotectedValue);
-
- assertEquals(SAMPLE_PLAINTEXT, unprotectedValue);
- assertNotEquals(SAMPLE_PLAINTEXT, protectedValue);
- assertNotEquals(protectedValue, unprotectedValue);
- }
-}
diff --git a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/firehose/ITPutKinesisFirehose.java b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/firehose/ITPutKinesisFirehose.java
deleted file mode 100644
index e1f26b0f1f..0000000000
--- a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/firehose/ITPutKinesisFirehose.java
+++ /dev/null
@@ -1,286 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.aws.kinesis.firehose;
-
-import org.apache.nifi.processors.aws.s3.FetchS3Object;
-import org.apache.nifi.processors.aws.testutil.AuthUtils;
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.Assumptions;
-import org.junit.jupiter.api.BeforeAll;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.io.File;
-import java.util.List;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-/**
- * This test contains both unit and integration test (integration tests are ignored by default)
- */
-
-public class ITPutKinesisFirehose {
- private final byte[] ONE_MB = new byte[1000 * 1024];
- private TestRunner runner;
- protected final static String CREDENTIALS_FILE = System.getProperty("user.home") + "/aws-credentials.properties";
-
- @BeforeAll
- public static void testSkipping() {
- Assumptions.assumeTrue(new File(CREDENTIALS_FILE).exists());
- }
-
- @BeforeEach
- public void setUp() throws Exception {
- runner = TestRunners.newTestRunner(PutKinesisFirehose.class);
- AuthUtils.enableCredentialsFile(runner, CREDENTIALS_FILE);
- runner.setProperty(PutKinesisFirehose.KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, "testkinesis");
- }
-
-
- /**
- * Comment out ignore for integration tests (requires creds files)
- */
- @Test
- public void testIntegrationSuccess() throws Exception {
- runner.enqueue("test".getBytes());
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutKinesisFirehose.REL_SUCCESS, 1);
-
- final List<MockFlowFile> ffs = runner.getFlowFilesForRelationship(FetchS3Object.REL_SUCCESS);
- final MockFlowFile out = ffs.iterator().next();
- out.assertContentEquals("test");
- }
-
- /**
- * Comment out ignore for integration tests (requires creds files)
- */
- @Test
- public void testIntegrationFailedBadStreamName() throws Exception {
- runner.setProperty(PutKinesisFirehose.KINESIS_FIREHOSE_DELIVERY_STREAM_NAME, "bad-firehose-s3-test");
-
- runner.enqueue("test");
- runner.run();
- runner.assertAllFlowFilesTransferred(PutKinesisFirehose.REL_FAILURE, 1);
- }
-
- @Test
- public void testOneMessageWithMaxBufferSizeGreaterThan1MBOneSuccess() {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "2");
- runner.setProperty(PutKinesisFirehose.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.enqueue(ONE_MB);
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutKinesisFirehose.REL_SUCCESS, 1);
- }
-
- @Test
- public void testTwoMessageBatch5WithMaxBufferSize1MBRunOnceTwoMessageSent() {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "5");
- runner.setProperty(PutKinesisFirehose.MAX_MESSAGE_BUFFER_SIZE_MB, "2 MB");
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutKinesisFirehose.REL_SUCCESS, 2);
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- }
- }
-
- @Test
- public void testThreeMessageWithBatch10MaxBufferSize1MBTRunOnceTwoMessageSent() {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "10");
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutKinesisFirehose.REL_SUCCESS, 2);
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- for (final MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- }
- }
-
- @Test
- public void testTwoMessageWithBatch10MaxBufferSize1MBTRunOnceTwoMessageSent() {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "10");
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutKinesisFirehose.REL_SUCCESS, 2);
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- for (final MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- }
- }
-
- @Test
- public void testThreeMessageWithBatch2MaxBufferSize1MBRunTwiceThreeMessageSent() {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "2");
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.run(2);
-
- runner.assertAllFlowFilesTransferred(PutKinesisFirehose.REL_SUCCESS, 3);
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- }
- }
-
- @Test
- public void testThreeMessageHello2MBThereWithBatch10MaxBufferSize1MBRunOnceTwoMessageSuccessOneFailed() {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "10");
- runner.enqueue("hello".getBytes());
- runner.enqueue(ONE_MB);
- runner.enqueue("there".getBytes());
- runner.run();
-
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (final MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- }
-
- final List<MockFlowFile> flowFilesFailed = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_FAILURE);
- assertEquals(1,flowFilesFailed.size());
- for (final MockFlowFile flowFileFailed : flowFilesFailed) {
- flowFileFailed.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_ERROR_MESSAGE);
- }
- }
-
- @Test
- public void testTwoMessageHello2MBWithBatch10MaxBufferSize1MBRunOnceOneSuccessOneFailed() throws Exception {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "10");
- runner.enqueue("hello".getBytes());
- runner.enqueue(ONE_MB);
- runner.run(1, true, true);
-
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- assertEquals(1,flowFiles.size());
- for (final MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- flowFile.assertContentEquals("hello".getBytes());
- }
-
- final List<MockFlowFile> flowFilesFailed = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_FAILURE);
- assertEquals(1,flowFilesFailed.size());
- for (final MockFlowFile flowFileFailed : flowFilesFailed) {
- assertNotNull(flowFileFailed.getAttribute(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_ERROR_MESSAGE));
- }
- }
-
- @Test
- public void testTwoMessage2MBHelloWorldWithBatch10MaxBufferSize1MBRunOnceTwoMessageSent() throws Exception {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "10");
- runner.enqueue(ONE_MB);
- runner.enqueue("HelloWorld".getBytes());
- runner.run();
-
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- assertEquals(1,flowFiles.size());
- for (final MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- flowFile.assertContentEquals("HelloWorld".getBytes());
- }
-
- final List<MockFlowFile> flowFilesFailed = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_FAILURE);
- assertEquals(1,flowFilesFailed.size());
- for (final MockFlowFile flowFileFailed : flowFilesFailed) {
- assertNotNull(flowFileFailed.getAttribute(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_ERROR_MESSAGE));
- }
- }
-
- @Test
- public void testTwoMessageHelloWorldWithBatch10MaxBufferSize1MBRunOnceTwoMessageSent() throws Exception {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "10");
- runner.enqueue("Hello".getBytes());
- runner.enqueue("World".getBytes());
- runner.run();
-
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (final MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- }
- flowFiles.get(0).assertContentEquals("Hello".getBytes());
- flowFiles.get(1).assertContentEquals("World".getBytes());
-
- final List<MockFlowFile> flowFilesFailed = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_FAILURE);
- assertEquals(0,flowFilesFailed.size());
- }
-
- @Test
- public void testThreeMessageWithBatch5MaxBufferSize1MBRunOnceTwoMessageSent() {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "5");
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutKinesisFirehose.REL_SUCCESS, 2);
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (final MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- }
- }
-
- @Test
- public void test5MessageWithBatch10MaxBufferSize10MBRunOnce5MessageSent() {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "10");
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutKinesisFirehose.REL_SUCCESS, 5);
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- assertEquals(5,flowFiles.size());
- for (final MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- }
- }
-
- @Test
- public void test5MessageWithBatch2MaxBufferSize10MBRunOnce2MessageSent() {
- runner.setProperty(PutKinesisFirehose.BATCH_SIZE, "2");
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.enqueue(ONE_MB);
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutKinesisFirehose.REL_SUCCESS, 2);
- final List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisFirehose.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (final MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisFirehose.AWS_KINESIS_FIREHOSE_RECORD_ID);
- }
- }
-}
diff --git a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITConsumeKinesisStreamConnectAWS.java b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITConsumeKinesisStreamConnectAWS.java
deleted file mode 100644
index 2c61ebcb7b..0000000000
--- a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITConsumeKinesisStreamConnectAWS.java
+++ /dev/null
@@ -1,72 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.aws.kinesis.stream;
-
-import org.apache.nifi.processors.aws.credentials.provider.PropertiesCredentialsProvider;
-import org.apache.nifi.processors.aws.testutil.AuthUtils;
-import org.apache.nifi.reporting.InitializationException;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.Assumptions;
-import org.junit.jupiter.api.BeforeEach;
-import software.amazon.awssdk.http.apache.ApacheHttpClient;
-import software.amazon.awssdk.regions.Region;
-import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
-import software.amazon.awssdk.services.kinesis.KinesisClient;
-import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
-
-import java.io.File;
-
-public class ITConsumeKinesisStreamConnectAWS extends ITConsumeKinesisStream {
-
- private final static File CREDENTIALS_FILE = new File(System.getProperty("user.home") + "/aws-credentials.properties");
-
- @BeforeEach
- public void setUp() throws InterruptedException, InitializationException {
- Assumptions.assumeTrue(CREDENTIALS_FILE.exists());
-
- System.setProperty("aws.cborEnabled", "false");
-
- kinesis = KinesisClient.builder()
- .credentialsProvider(new PropertiesCredentialsProvider(CREDENTIALS_FILE))
- .httpClient(ApacheHttpClient.create())
- .region(Region.of(REGION))
- .build();
-
- kinesis.createStream(CreateStreamRequest.builder().streamName(KINESIS_STREAM_NAME).shardCount(1).build());
-
- dynamoDB = DynamoDbClient.builder()
- .credentialsProvider(new PropertiesCredentialsProvider(CREDENTIALS_FILE))
- .region(Region.of(REGION))
- .httpClient(ApacheHttpClient.create())
- .build();
-
- waitForKinesisToInitialize();
-
- runner = TestRunners.newTestRunner(ConsumeKinesisStream.class);
- AuthUtils.enableCredentialsFile(runner, CREDENTIALS_FILE.getAbsolutePath());
- runner.setProperty(ConsumeKinesisStream.APPLICATION_NAME, APPLICATION_NAME);
- runner.setProperty(ConsumeKinesisStream.KINESIS_STREAM_NAME, KINESIS_STREAM_NAME);
- runner.setProperty(ConsumeKinesisStream.AWS_CREDENTIALS_PROVIDER_SERVICE, "credentials-service");
- runner.setProperty(ConsumeKinesisStream.REGION, REGION);
- runner.setProperty(ConsumeKinesisStream.REPORT_CLOUDWATCH_METRICS, "false");
- runner.assertValid();
- }
-
- private void waitForKinesisToInitialize() throws InterruptedException {
- Thread.sleep(20_000);
- }
-}
diff --git a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITConsumeKinesisStreamEndpointOverride.java b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITConsumeKinesisStreamEndpointOverride.java
deleted file mode 100644
index 0b6e0f360e..0000000000
--- a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITConsumeKinesisStreamEndpointOverride.java
+++ /dev/null
@@ -1,88 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.aws.kinesis.stream;
-
-import org.apache.nifi.processors.aws.testutil.AuthUtils;
-import org.apache.nifi.reporting.InitializationException;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.Assumptions;
-import org.junit.jupiter.api.BeforeEach;
-import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
-import software.amazon.awssdk.auth.credentials.AwsCredentialsProvider;
-import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
-import software.amazon.awssdk.http.apache.ApacheHttpClient;
-import software.amazon.awssdk.regions.Region;
-import software.amazon.awssdk.services.dynamodb.DynamoDbClient;
-import software.amazon.awssdk.services.kinesis.KinesisClient;
-import software.amazon.awssdk.services.kinesis.model.CreateStreamRequest;
-
-import java.io.File;
-import java.net.URI;
-
-public class ITConsumeKinesisStreamEndpointOverride extends ITConsumeKinesisStream {
-
- private final static File CREDENTIALS_FILE = new File(System.getProperty("user.home") + "/aws-credentials.properties");
-
- private static final String ACCESS_KEY = "test";
- private static final String SECRET_KEY = "test";
- private static final String KINESIS_STREAM_NAME = "test-stream";
- private static final String APPLICATION_NAME = "test-application";
- private static final String LOCAL_STACK_KINESIS_ENDPOINT_OVERRIDE = "http://localhost:4566";
- private static final String LOCAL_STACK_DYNAMODB_ENDPOINT_OVERRIDE = "http://localhost:4566";
-
- private final AwsCredentialsProvider awsCredentialsProvider = StaticCredentialsProvider.create(
- AwsBasicCredentials.create(ACCESS_KEY, SECRET_KEY));
-
- @BeforeEach
- public void setUp() throws InterruptedException, InitializationException {
- Assumptions.assumeTrue(CREDENTIALS_FILE.exists());
- System.setProperty("aws.cborEnabled", "false");
-
- kinesis = KinesisClient.builder()
- .credentialsProvider(awsCredentialsProvider)
- .endpointOverride(URI.create(LOCAL_STACK_KINESIS_ENDPOINT_OVERRIDE))
- .httpClient(ApacheHttpClient.create())
- .region(Region.of(REGION))
- .build();
-
- kinesis.createStream(CreateStreamRequest.builder().streamName(KINESIS_STREAM_NAME).shardCount(1).build());
-
- dynamoDB = DynamoDbClient.builder()
- .credentialsProvider(awsCredentialsProvider)
- .endpointOverride(URI.create(LOCAL_STACK_DYNAMODB_ENDPOINT_OVERRIDE))
- .region(Region.of(REGION))
- .httpClient(ApacheHttpClient.create())
- .build();
-
- waitForKinesisToInitialize();
-
- runner = TestRunners.newTestRunner(ConsumeKinesisStream.class);
- AuthUtils.enableCredentialsFile(runner, CREDENTIALS_FILE.getAbsolutePath());
-
- runner.setProperty(ConsumeKinesisStream.APPLICATION_NAME, APPLICATION_NAME);
- runner.setProperty(ConsumeKinesisStream.KINESIS_STREAM_NAME, KINESIS_STREAM_NAME);
- runner.setProperty(ConsumeKinesisStream.REGION, REGION);
- runner.setProperty(ConsumeKinesisStream.REPORT_CLOUDWATCH_METRICS, "false");
- runner.setProperty(ConsumeKinesisStream.ENDPOINT_OVERRIDE, LOCAL_STACK_KINESIS_ENDPOINT_OVERRIDE + "/kinesis");
- runner.setProperty(ConsumeKinesisStream.DYNAMODB_ENDPOINT_OVERRIDE, LOCAL_STACK_DYNAMODB_ENDPOINT_OVERRIDE + "/dynamodb");
- runner.assertValid();
- }
-
- private void waitForKinesisToInitialize() throws InterruptedException {
- Thread.sleep(1000);
- }
-}
diff --git a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITPutKinesisStream.java b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITPutKinesisStream.java
deleted file mode 100644
index e57faba6f5..0000000000
--- a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITPutKinesisStream.java
+++ /dev/null
@@ -1,421 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.aws.kinesis.stream;
-
-import org.apache.nifi.processors.aws.kinesis.KinesisProcessorUtils;
-import org.apache.nifi.processors.aws.testutil.AuthUtils;
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.Assumptions;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.io.File;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-/**
- * This test contains both unit and integration test (integration tests are ignored by default).
- * Running integration tests may result in failures due to provisioned capacity of Kinesis stream based on number of shards.
- * The following integration tests run successfully with 10 shards. If increasing shards is not a possiblity, please reduce the size and
- * number of messages in the integration tests based AWS Kinesis provisioning pages calculations.
- */
-public class ITPutKinesisStream {
-
- private TestRunner runner;
- protected final static String CREDENTIALS_FILE = System.getProperty("user.home") + "/aws-credentials.properties";
-
- @BeforeEach
- public void setUp() {
- Assumptions.assumeTrue(new File(CREDENTIALS_FILE).exists());
-
- runner = TestRunners.newTestRunner(PutKinesisStream.class);
- runner.setProperty(PutKinesisStream.KINESIS_STREAM_NAME, "kstream");
- AuthUtils.enableCredentialsFile(runner, CREDENTIALS_FILE);
- runner.assertValid();
- }
-
-
- @Test
- public void testIntegrationSuccess() throws Exception {
- runner.enqueue("test".getBytes());
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 1);
-
- final List<MockFlowFile> ffs = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- final MockFlowFile out = ffs.iterator().next();
-
- out.assertContentEquals("test".getBytes());
- }
-
- @Test
- public void testIntegrationWithFixedPartitionSuccess() throws Exception {
- runner.setProperty(PutKinesisStream.KINESIS_PARTITION_KEY, "pfixed");
- runner.assertValid();
-
- runner.enqueue("test".getBytes());
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 1);
-
- final List<MockFlowFile> ffs = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- final MockFlowFile out = ffs.iterator().next();
-
- out.assertContentEquals("test".getBytes());
- }
-
- @Test
- public void testIntegrationWithDynamicPartitionSuccess() throws Exception {
- runner.setProperty(PutKinesisStream.KINESIS_PARTITION_KEY, "${partition}");
- runner.assertValid();
- Map<String,String> properties = new HashMap<>();
- properties.put("partition", "px");
- runner.enqueue("test".getBytes(), properties);
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 1);
-
- final List<MockFlowFile> ffs = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- final MockFlowFile out = ffs.iterator().next();
-
- out.assertContentEquals("test".getBytes());
- }
-
- @Test
- public void testIntegrationFailedBadStreamName() throws Exception {
- runner.setProperty(PutKinesisStream.KINESIS_STREAM_NAME, "bad-kstream");
- runner.assertValid();
-
- runner.enqueue("test".getBytes());
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_FAILURE, 1);
-
- }
-
- @Test
- public void testOneMessageWithMaxBufferSizeGreaterThan1MBOneSuccess() {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "2");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- byte [] bytes = new byte[(KinesisProcessorUtils.MAX_MESSAGE_SIZE)];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue(bytes);
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 1);
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(1,flowFiles.size());
- }
-
- @Test
- public void testOneMessageWithMaxBufferSizeGreaterThan1MBOneWithParameterPartitionSuccess() {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "2");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.setProperty(PutKinesisStream.KINESIS_PARTITION_KEY, "${partitionKey}");
- runner.assertValid();
- byte [] bytes = new byte[(KinesisProcessorUtils.MAX_MESSAGE_SIZE)];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- Map<String,String> props = new HashMap<>();
- props.put("partitionKey", "p1");
- runner.enqueue(bytes,props);
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 1);
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(1,flowFiles.size());
- }
-
- @Test
- public void testTwoMessageBatch5WithMaxBufferSize1MBRunOnceTwoMessageSent() {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "5");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "2 MB");
- runner.assertValid();
- byte [] bytes = new byte[(KinesisProcessorUtils.MAX_MESSAGE_SIZE)];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue(bytes);
- runner.enqueue(bytes.clone());
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 2);
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- }
- }
-
- @Test
- public void testThreeMessageWithBatch10MaxBufferSize1MBTRunOnceTwoMessageSent() {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "10");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- byte [] bytes = new byte[(KinesisProcessorUtils.MAX_MESSAGE_SIZE)];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue(bytes);
- runner.enqueue(bytes.clone());
- runner.enqueue(bytes.clone());
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 2);
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- }
- }
-
- @Test
- public void testTwoMessageWithBatch10MaxBufferSize1MBTRunOnceTwoMessageSent() {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "10");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- byte [] bytes = new byte[(KinesisProcessorUtils.MAX_MESSAGE_SIZE)];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue(bytes);
- runner.enqueue(bytes.clone());
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 2);
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- }
- }
-
- @Test
- public void testThreeMessageWithBatch2MaxBufferSize1MBRunTwiceThreeMessageSent() {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "2");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- byte [] bytes = new byte[(KinesisProcessorUtils.MAX_MESSAGE_SIZE)];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue(bytes);
- runner.enqueue(bytes.clone());
- runner.enqueue(bytes.clone());
- runner.run(2, true, true);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 3);
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(3,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- }
- }
-
- @Test
- public void testThreeMessageHello2MBThereWithBatch10MaxBufferSize1MBRunOnceTwoMessageSuccessOneFailed() {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "10");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- byte [] bytes = new byte[(KinesisProcessorUtils.MAX_MESSAGE_SIZE * 2)];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue("hello".getBytes());
- runner.enqueue(bytes);
- runner.enqueue("there".getBytes());
- runner.run(1, true, true);
-
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- }
-
- List<MockFlowFile> flowFilesFailed = runner.getFlowFilesForRelationship(PutKinesisStream.REL_FAILURE);
- assertEquals(1,flowFilesFailed.size());
- for (MockFlowFile flowFileFailed : flowFilesFailed) {
- assertNotNull(flowFileFailed.getAttribute(PutKinesisStream.AWS_KINESIS_ERROR_MESSAGE));
- }
- }
-
- @Test
- public void testTwoMessageHello2MBWithBatch10MaxBufferSize1MBRunOnceOneSuccessOneFailed() throws Exception {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "10");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- byte [] bytes = new byte[(KinesisProcessorUtils.MAX_MESSAGE_SIZE * 2)];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue("hello".getBytes());
- runner.enqueue(bytes);
- runner.run(1, true, true);
-
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(1,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- flowFile.assertContentEquals("hello".getBytes());
- }
-
- List<MockFlowFile> flowFilesFailed = runner.getFlowFilesForRelationship(PutKinesisStream.REL_FAILURE);
- assertEquals(1,flowFilesFailed.size());
- for (MockFlowFile flowFileFailed : flowFilesFailed) {
- assertNotNull(flowFileFailed.getAttribute(PutKinesisStream.AWS_KINESIS_ERROR_MESSAGE));
- }
- }
-
- @Test
- public void testTwoMessage2MBHelloWorldWithBatch10MaxBufferSize1MBRunOnceTwoMessageSent() throws Exception {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "10");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- byte [] bytes = new byte[(KinesisProcessorUtils.MAX_MESSAGE_SIZE * 2)];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue(bytes);
- runner.enqueue("HelloWorld".getBytes());
- runner.run(1, true, true);
-
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(1,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- flowFile.assertContentEquals("HelloWorld".getBytes());
- }
-
- List<MockFlowFile> flowFilesFailed = runner.getFlowFilesForRelationship(PutKinesisStream.REL_FAILURE);
- assertEquals(1,flowFilesFailed.size());
- for (MockFlowFile flowFileFailed : flowFilesFailed) {
- assertNotNull(flowFileFailed.getAttribute(PutKinesisStream.AWS_KINESIS_ERROR_MESSAGE));
- }
- }
-
- @Test
- public void testTwoMessageHelloWorldWithBatch10MaxBufferSize1MBRunOnceTwoMessageSent() throws Exception {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "10");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- runner.enqueue("Hello".getBytes());
- runner.enqueue("World".getBytes());
- runner.run(1, true, true);
-
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- }
- flowFiles.get(0).assertContentEquals("Hello".getBytes());
- flowFiles.get(1).assertContentEquals("World".getBytes());
-
- List<MockFlowFile> flowFilesFailed = runner.getFlowFilesForRelationship(PutKinesisStream.REL_FAILURE);
- assertEquals(0,flowFilesFailed.size());
- }
-
- @Test
- public void testThreeMessageWithBatch5MaxBufferSize1MBRunOnceTwoMessageSent() {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "5");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- byte [] bytes = new byte[(KinesisProcessorUtils.MAX_MESSAGE_SIZE)];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue(bytes);
- runner.enqueue(bytes.clone());
- runner.enqueue(bytes.clone());
- runner.run(1, true, true);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 2);
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- }
- }
-
- @Test
- public void test5MessageWithBatch10MaxBufferSize10MBRunOnce5MessageSent() {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "10");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- byte [] bytes = new byte[10];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue(bytes);
- runner.enqueue(bytes.clone());
- runner.enqueue(bytes.clone());
- runner.enqueue(bytes);
- runner.enqueue(bytes.clone());
- runner.run(1, true, true);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 5);
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(5,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- }
- }
-
- @Test
- public void test5MessageWithBatch2MaxBufferSize10MBRunOnce2MessageSent() {
- runner.setProperty(PutKinesisStream.BATCH_SIZE, "2");
- runner.setProperty(PutKinesisStream.MAX_MESSAGE_BUFFER_SIZE_MB, "1 MB");
- runner.assertValid();
- byte [] bytes = new byte[10];
- for (int i = 0; i < bytes.length; i++) {
- bytes[i] = 'a';
- }
- runner.enqueue(bytes);
- runner.enqueue(bytes.clone());
- runner.enqueue(bytes.clone());
- runner.enqueue(bytes);
- runner.enqueue(bytes.clone());
- runner.run(1, true, true);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 2);
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- assertEquals(2,flowFiles.size());
- for (MockFlowFile flowFile : flowFiles) {
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SEQUENCE_NUMBER);
- flowFile.assertAttributeExists(PutKinesisStream.AWS_KINESIS_SHARD_ID);
- }
- }
-}
diff --git a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITPutKinesisStreamWithEndpointOverride.java b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITPutKinesisStreamWithEndpointOverride.java
deleted file mode 100644
index 4dfae73087..0000000000
--- a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/kinesis/stream/ITPutKinesisStreamWithEndpointOverride.java
+++ /dev/null
@@ -1,84 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.aws.kinesis.stream;
-
-import org.apache.nifi.processors.aws.testutil.AuthUtils;
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Disabled;
-import org.junit.jupiter.api.Test;
-
-import java.util.List;
-
-import static com.amazonaws.SDKGlobalConfiguration.AWS_CBOR_DISABLE_SYSTEM_PROPERTY;
-
-// This integration test can be run against a mock Kenesis such as
-// https://github.com/mhart/kinesalite or https://github.com/localstack/localstack
-@Disabled("Required external service be running. Needs to be updated to make use of Localstack TestContainer")
-public class ITPutKinesisStreamWithEndpointOverride {
-
- private TestRunner runner;
-
- @BeforeEach
- public void setUp() {
- System.setProperty(AWS_CBOR_DISABLE_SYSTEM_PROPERTY, "true");
-
- runner = TestRunners.newTestRunner(PutKinesisStream.class);
- runner.setProperty(PutKinesisStream.KINESIS_STREAM_NAME, "test");
- runner.setProperty(PutKinesisStream.ENDPOINT_OVERRIDE, "http://localhost:4568");
- AuthUtils.enableAccessKey(runner, "accessKey", "secretKey");
- runner.assertValid();
- }
-
- @AfterEach
- public void tearDown() {
- runner = null;
-
- System.clearProperty(AWS_CBOR_DISABLE_SYSTEM_PROPERTY);
- }
-
- @Test
- public void testIntegrationSuccess() throws Exception {
- runner.assertValid();
-
- runner.enqueue("test".getBytes());
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_SUCCESS, 1);
-
- final List<MockFlowFile> ffs = runner.getFlowFilesForRelationship(PutKinesisStream.REL_SUCCESS);
- final MockFlowFile out = ffs.iterator().next();
-
- out.assertContentEquals("test".getBytes());
- }
-
- @Test
- public void testIntegrationFailedBadStreamName() throws Exception {
- runner.setProperty(PutKinesisStream.KINESIS_STREAM_NAME, "notfound");
- runner.assertValid();
-
- runner.enqueue("test".getBytes());
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutKinesisStream.REL_FAILURE, 1);
-
- }
-
-}
diff --git a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/lambda/ITPutLambda.java b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/lambda/ITPutLambda.java
deleted file mode 100644
index 6f01fb7294..0000000000
--- a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/lambda/ITPutLambda.java
+++ /dev/null
@@ -1,137 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.aws.lambda;
-
-import org.apache.nifi.processors.aws.testutil.AuthUtils;
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.Assumptions;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.io.File;
-import java.util.List;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-import static org.junit.jupiter.api.Assertions.assertNull;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-/**
- * This test contains both unit and integration test (integration tests are ignored by default)
- */
-public class ITPutLambda {
-
- private TestRunner runner;
- protected final static String CREDENTIALS_FILE = System.getProperty("user.home") + "/aws-credentials.properties";
-
- @BeforeEach
- public void setUp() {
- Assumptions.assumeTrue(new File(CREDENTIALS_FILE).exists());
-
- runner = TestRunners.newTestRunner(PutLambda.class);
- AuthUtils.enableCredentialsFile(runner, CREDENTIALS_FILE);
- runner.setProperty(PutLambda.AWS_LAMBDA_FUNCTION_NAME, "functionName");
- runner.assertValid();
- }
-
- @AfterEach
- public void tearDown() {
- runner = null;
- }
-
- @Test
- public void testSizeGreaterThan6MB() {
- runner = TestRunners.newTestRunner(PutLambda.class);
- AuthUtils.enableCredentialsFile(runner, CREDENTIALS_FILE);
- runner.setProperty(PutLambda.AWS_LAMBDA_FUNCTION_NAME, "hello");
- runner.assertValid();
- byte [] largeInput = new byte[6000001];
- runner.enqueue(largeInput);
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutLambda.REL_FAILURE, 1);
- }
-
- /**
- * Comment out ignore for integration tests (requires creds files)
- */
- @Test
- public void testIntegrationSuccess() {
- runner.setProperty(PutLambda.AWS_LAMBDA_FUNCTION_NAME, "hello");
- runner.assertValid();
-
- runner.enqueue("{\"test\":\"hi\"}".getBytes());
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutLambda.REL_SUCCESS, 1);
-
- final List<MockFlowFile> ffs = runner.getFlowFilesForRelationship(PutLambda.REL_SUCCESS);
- final MockFlowFile out = ffs.iterator().next();
- assertNull(out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_FUNCTION_ERROR), "Function error should be null " + out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_FUNCTION_ERROR));
- assertNotNull(out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_LOG), "log should not be null");
- assertEquals("200",out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_STATUS_CODE), "Status should be equal");
- }
-
- /**
- * Comment out ignore for integration tests (requires creds files)
- */
- @Test
- public void testIntegrationClientErrorBadMessageBody() {
- runner.setProperty(PutLambda.AWS_LAMBDA_FUNCTION_NAME, "hello");
- runner.assertValid();
-
- runner.enqueue("badbod".getBytes());
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutLambda.REL_FAILURE, 1);
- final List<MockFlowFile> ffs = runner.getFlowFilesForRelationship(PutLambda.REL_FAILURE);
- final MockFlowFile out = ffs.iterator().next();
- assertNull(out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_FUNCTION_ERROR), "Function error should be null since there is exception"
- + out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_FUNCTION_ERROR));
- assertNull(out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_LOG), "log should not be null");
- assertEquals(null,out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_STATUS_CODE), "Status should be equal");
- assertEquals("InvalidRequestContentException",out.getAttribute(PutLambda.AWS_LAMBDA_EXCEPTION_ERROR_CODE), "exception error code should be equal");
- assertEquals("400",out.getAttribute(PutLambda.AWS_LAMBDA_EXCEPTION_STATUS_CODE), "exception exception error code should be equal");
- assertTrue(out.getAttribute(PutLambda.AWS_LAMBDA_EXCEPTION_MESSAGE)
- .startsWith("Could not parse request body into json: Could not parse payload into json: Unrecognized token 'badbod': " +
- "was expecting (JSON String, Number, Array, Object or token 'null', 'true' or 'false')"), "exception exception error message should be start with");
- }
-
- /**
- * Comment out ignore for integration tests (requires creds files)
- */
- @Test
- public void testIntegrationFailedBadStreamName() {
- runner.setProperty(PutLambda.AWS_LAMBDA_FUNCTION_NAME, "bad-function-name");
- runner.assertValid();
-
- runner.enqueue("{\"test\":\"hi\"}".getBytes());
- runner.run(1);
-
- runner.assertAllFlowFilesTransferred(PutLambda.REL_FAILURE, 1);
- final List<MockFlowFile> ffs = runner.getFlowFilesForRelationship(PutLambda.REL_FAILURE);
- final MockFlowFile out = ffs.iterator().next();
- assertNull(out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_FUNCTION_ERROR), "Function error should be null since there is exception"
- + out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_FUNCTION_ERROR));
- assertNull(out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_LOG));
- assertEquals(null, out.getAttribute(PutLambda.AWS_LAMBDA_RESULT_STATUS_CODE));
-
- }
-}
diff --git a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/sns/ITPutSNS.java b/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/sns/ITPutSNS.java
deleted file mode 100644
index 499f24a05c..0000000000
--- a/nifi-extension-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/sns/ITPutSNS.java
+++ /dev/null
@@ -1,78 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.aws.sns;
-
-import org.apache.nifi.processors.aws.testutil.AuthUtils;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.Assumptions;
-import org.junit.jupiter.api.BeforeAll;
-import org.junit.jupiter.api.Test;
-
-import java.io.File;
-import java.io.IOException;
-import java.nio.file.Paths;
-import java.util.HashMap;
-import java.util.Map;
-
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-/**
- * Provides integration level testing with actual AWS S3 resources for {@link PutSNS} and requires additional configuration and resources to work.
- */
-public class ITPutSNS {
-
- private static final String CREDENTIALS_FILE = System.getProperty("user.home") + "/aws-credentials.properties";
- private final String TOPIC_ARN = "Add SNS ARN here";
-
- @BeforeAll
- public static void assumeCredentialsFileExists() {
- Assumptions.assumeTrue(new File(CREDENTIALS_FILE).exists());
- }
-
- @Test
- public void testPublish() throws IOException {
- final TestRunner runner = TestRunners.newTestRunner(new PutSNS());
- AuthUtils.enableCredentialsFile(runner, CREDENTIALS_FILE);
- runner.setProperty(PutSNS.ARN, TOPIC_ARN);
- assertTrue(runner.setProperty("DynamicProperty", "hello!").isValid());
-
- final Map<String, String> attrs = new HashMap<>();
- attrs.put("filename", "1.txt");
- runner.enqueue(Paths.get("src/test/resources/hello.txt"), attrs);
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutSNS.REL_SUCCESS, 1);
- }
-
- @Test
- public void testPublishWithCredentialsProviderService() throws Throwable {
- final TestRunner runner = TestRunners.newTestRunner(new PutSNS());
- runner.setProperty(PutSNS.ARN, TOPIC_ARN);
- assertTrue(runner.setProperty("DynamicProperty", "hello!").isValid());
-
- AuthUtils.enableCredentialsFile(runner, CREDENTIALS_FILE);
- runner.run(1);
-
- final Map<String, String> attrs = new HashMap<>();
- attrs.put("filename", "1.txt");
- runner.enqueue(Paths.get("src/test/resources/hello.txt"), attrs);
- runner.run();
-
- runner.assertAllFlowFilesTransferred(PutSNS.REL_SUCCESS, 1);
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-graph-authorizer/src/test/java/org/apache/nifi/authorization/azure/AzureGraphUserGroupProviderIT.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-graph-authorizer/src/test/java/org/apache/nifi/authorization/azure/AzureGraphUserGroupProviderIT.java
deleted file mode 100644
index f866971417..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-graph-authorizer/src/test/java/org/apache/nifi/authorization/azure/AzureGraphUserGroupProviderIT.java
+++ /dev/null
@@ -1,219 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.nifi.authorization.azure;
-
-import org.apache.nifi.authorization.AuthorizerConfigurationContext;
-import org.apache.nifi.authorization.Group;
-import org.apache.nifi.authorization.UserAndGroups;
-import org.apache.nifi.authorization.UserGroupProviderInitializationContext;
-import org.apache.nifi.util.MockPropertyValue;
-import org.apache.nifi.util.file.FileUtils;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-import org.mockito.Mockito;
-import org.slf4j.Logger;
-import org.slf4j.LoggerFactory;
-
-import java.io.FileInputStream;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Properties;
-import java.util.Set;
-
-import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertFalse;
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-public class AzureGraphUserGroupProviderIT {
- private static final Logger logger = LoggerFactory.getLogger(AzureGraphUserGroupProviderIT.class);
-
- private static final Properties CONFIG;
-
- private static final String CREDENTIALS_FILE = System.getProperty("user.home") + "/azure-aad-app-reg.PROPERTIES";
-
- static {
- CONFIG = new Properties();
- assertDoesNotThrow(() -> {
- final FileInputStream fis = new FileInputStream(CREDENTIALS_FILE);
- assertDoesNotThrow(() -> CONFIG.load(fis));
- FileUtils.closeQuietly(fis);
- });
- }
-
- protected static String getAuthorityEndpoint() {
- return CONFIG.getProperty("AUTHORITY_ENDPOINT");
- }
-
- protected static String getTenantId() {
- return CONFIG.getProperty("TENANT_ID");
- }
-
- protected static String getAppRegClientId() {
- return CONFIG.getProperty("APP_REG_CLIENT_ID");
- }
-
- protected static String getAppRegClientSecret() {
- return CONFIG.getProperty("APP_REG_CLIENT_SECRET");
- }
-
- protected static String getKnownTestUserName() {
- return CONFIG.getProperty("KNOWN_TEST_USER");
-
- }
-
- protected static String getKnownTestGroupName() {
- return CONFIG.getProperty("KNOWN_TEST_GROUP");
- }
-
- protected static String getGroupListInclusion() {
- return CONFIG.getProperty("GROUP_FILTER_LIST_INCLUSION");
- }
-
- private AuthorizerConfigurationContext authContext = Mockito.mock(AuthorizerConfigurationContext.class);
- private AzureGraphUserGroupProvider testingProvider;
- private UserGroupProviderInitializationContext initContext;
-
- @BeforeEach
- public void setup() {
- authContext = Mockito.mock(AuthorizerConfigurationContext.class);
- initContext = Mockito.mock(UserGroupProviderInitializationContext.class);
-
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.AUTHORITY_ENDPOINT_PROPERTY)))
- .thenReturn(new MockPropertyValue(AzureGraphUserGroupProviderIT.getAuthorityEndpoint()));
-
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.TENANT_ID_PROPERTY)))
- .thenReturn(new MockPropertyValue(AzureGraphUserGroupProviderIT.getTenantId()));
-
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.APP_REG_CLIENT_ID_PROPERTY)))
- .thenReturn(new MockPropertyValue(AzureGraphUserGroupProviderIT.getAppRegClientId()));
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.APP_REG_CLIENT_SECRET_PROPERTY)))
- .thenReturn(new MockPropertyValue(AzureGraphUserGroupProviderIT.getAppRegClientSecret()));
- }
-
- private void setupTestingProvider() {
- testingProvider = new AzureGraphUserGroupProvider();
- try {
- testingProvider.initialize(initContext);
- testingProvider.onConfigured(authContext);
- } catch (final Exception exc) {
- logger.error("Error during setup; tests cannot run on this system.");
- }
- }
-
-
- @AfterEach
- public void tearDown() {
- testingProvider.preDestruction();
- }
-
- @Test
- public void testWithGroupListFilter(){
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.GROUP_FILTER_LIST_PROPERTY)))
- .thenReturn(new MockPropertyValue(getGroupListInclusion()));
-
- setupTestingProvider();
-
- assertFalse(testingProvider.getGroups().isEmpty());
- assertFalse(testingProvider.getUsers().isEmpty());
- UserAndGroups uag = testingProvider.getUserAndGroups(getKnownTestUserName());
- assertNotNull(uag.getUser());
- assertFalse(uag.getGroups().isEmpty());
-
- }
-
- @Test
- public void testWithPaging(){
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.GROUP_FILTER_LIST_PROPERTY)))
- .thenReturn(new MockPropertyValue(getGroupListInclusion()));
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.PAGE_SIZE_PROPERTY)))
- .thenReturn(new MockPropertyValue("3")); // in the real scenario, this should be 20 or bigger.
-
- setupTestingProvider();
-
- assertFalse(testingProvider.getGroups().isEmpty());
- assertFalse(testingProvider.getUsers().isEmpty());
- UserAndGroups uag = testingProvider.getUserAndGroups(getKnownTestUserName());
- assertNotNull(uag.getUser());
- assertFalse(uag.getGroups().isEmpty());
-
- String knownGroupName = getKnownTestGroupName();
- List<Group> search = testingProvider.getGroups().stream().filter(g-> g.getName().equals(knownGroupName)).toList();
- assertFalse(search.isEmpty());
- }
-
- @Test
- public void testWithGroupFilterPrefix(){
- // make sure to set up a test group name whose name length is longer than 5
- String knownGroupName = getKnownTestGroupName();
- String prefix = knownGroupName.substring(0, 2);
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.GROUP_FILTER_PREFIX_PROPERTY)))
- .thenReturn(new MockPropertyValue(prefix));
-
- setupTestingProvider();
- assertFalse(testingProvider.getGroups().isEmpty());
- List<Group> search = testingProvider.getGroups().stream().filter(g-> g.getName().equals(knownGroupName)).toList();
- assertFalse(search.isEmpty());
- }
-
- @Test
- public void testWithGroupFilterSuffix(){
- // make sure to set up a test group name whose name length is longer than 5
- String knownGroupName = getKnownTestGroupName();
- String suffix = knownGroupName.substring(knownGroupName.length()-2);
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.GROUP_FILTER_SUFFIX_PROPERTY)))
- .thenReturn(new MockPropertyValue(suffix));
-
- setupTestingProvider();
- assertFalse(testingProvider.getGroups().isEmpty());
- List<Group> search = testingProvider.getGroups().stream().filter(g-> g.getName().equals(knownGroupName)).toList();
- assertFalse(search.isEmpty());
- }
-
- @Test
- public void testWithGroupFilterSubstring(){
- // make sure to set up a test group name whose name length is longer than 5
- String knownGroupName = getKnownTestGroupName();
- String substring = knownGroupName.substring(1, knownGroupName.length()-1);
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.GROUP_FILTER_SUBSTRING_PROPERTY)))
- .thenReturn(new MockPropertyValue(substring));
-
- setupTestingProvider();
- assertFalse(testingProvider.getGroups().isEmpty());
- List<Group> search = testingProvider.getGroups().stream().filter( g-> g.getName().equals(knownGroupName)).toList();
- assertFalse(search.isEmpty());
- }
-
- @Test
- public void testWithGroupFilterOperatorAndListInclusion(){
- // make sure to set up a test group name whose name length is longer than 5
- String knownGroupName = getKnownTestGroupName();
- String substring = knownGroupName.substring(1, knownGroupName.length()-1);
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.GROUP_FILTER_SUBSTRING_PROPERTY)))
- .thenReturn(new MockPropertyValue(substring));
- Mockito.when(authContext.getProperty(Mockito.eq(AzureGraphUserGroupProvider.GROUP_FILTER_LIST_PROPERTY)))
- .thenReturn(new MockPropertyValue(getGroupListInclusion()));
-
- setupTestingProvider();
- assertFalse(testingProvider.getGroups().isEmpty());
- Set<Group> search = new HashSet<>(testingProvider.getGroups());
- // check there is no duplicate group
- assertEquals(search.size(), testingProvider.getGroups().size());
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/cosmos/document/ITPutAzureCosmosDBRecord.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/cosmos/document/ITPutAzureCosmosDBRecord.java
deleted file mode 100644
index d596fa5ad7..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/cosmos/document/ITPutAzureCosmosDBRecord.java
+++ /dev/null
@@ -1,160 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.cosmos.document;
-
-import com.azure.cosmos.models.CosmosQueryRequestOptions;
-import com.fasterxml.jackson.databind.JsonNode;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.reporting.InitializationException;
-import org.apache.nifi.serialization.SimpleRecordSchema;
-import org.apache.nifi.serialization.record.MapRecord;
-import org.apache.nifi.serialization.record.MockRecordParser;
-import org.apache.nifi.serialization.record.RecordField;
-import org.apache.nifi.serialization.record.RecordFieldType;
-import org.apache.nifi.serialization.record.RecordSchema;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.logging.Logger;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public class ITPutAzureCosmosDBRecord extends ITAbstractAzureCosmosDBDocument {
- static Logger logger = Logger.getLogger(ITPutAzureCosmosDBRecord.class.getName());
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return PutAzureCosmosDBRecord.class;
- }
-
- @BeforeEach
- public void setUp() throws Exception {
- resetTestCosmosConnection();
- }
-
- @AfterEach
- public void cleanupTestCase() {
- try {
- clearTestData();
- closeClient();
- } catch (Exception ignored) {
-
- }
- }
-
- private List<JsonNode> getDataFromTestDB() {
- logger.info("getDataFromTestDB for test result validation");
- CosmosQueryRequestOptions queryOptions = new CosmosQueryRequestOptions();
-
- return container
- .queryItems("select * from c order by c._ts", queryOptions, JsonNode.class)
- .stream().toList();
- }
-
- private MockRecordParser recordReader;
-
- private void setupRecordReader() throws InitializationException {
- recordReader = new MockRecordParser();
- runner.addControllerService("reader", recordReader);
- runner.enableControllerService(recordReader);
- runner.setProperty(PutAzureCosmosDBRecord.RECORD_READER_FACTORY, "reader");
- }
-
- @Test
- public void testOnTriggerWithNestedRecords() throws InitializationException {
- setupRecordReader();
- recordReader.addSchemaField("id", RecordFieldType.STRING);
- recordReader.addSchemaField(TEST_COSMOS_PARTITION_KEY_FIELD_NAME, RecordFieldType.STRING);
-
- final List<RecordField> personFields = new ArrayList<>();
- final RecordField nameField = new RecordField("name", RecordFieldType.STRING.getDataType());
- final RecordField ageField = new RecordField("age", RecordFieldType.INT.getDataType());
- final RecordField sportField = new RecordField("sport", RecordFieldType.STRING.getDataType());
- personFields.add(nameField);
- personFields.add(ageField);
- personFields.add(sportField);
- final RecordSchema personSchema = new SimpleRecordSchema(personFields);
- recordReader.addSchemaField("person", RecordFieldType.RECORD);
-
- recordReader.addRecord("1", "A", new MapRecord(personSchema, new HashMap<String, Object>() {
- private static final long serialVersionUID = -3185956498135742190L;
-
- {
- put("name", "John Doe");
- put("age", 48);
- put("sport", "Soccer");
- }
- }));
- recordReader.addRecord("2", "B", new MapRecord(personSchema, new HashMap<String, Object>() {
- private static final long serialVersionUID = 1L;
-
- {
- put("name", "Jane Doe");
- put("age", 47);
- put("sport", "Tennis");
- }
- }));
- recordReader.addRecord("3", "A", new MapRecord(personSchema, new HashMap<String, Object>() {
- private static final long serialVersionUID = -1329194249439570573L;
-
- {
- put("name", "Sally Doe");
- put("age", 47);
- put("sport", "Curling");
- }
- }));
- recordReader.addRecord("4", "C", new MapRecord(personSchema, new HashMap<String, Object>() {
- private static final long serialVersionUID = -1329194249439570574L;
-
- {
- put("name", "Jimmy Doe");
- put("age", 14);
- put("sport", null);
- }
- }));
-
- runner.enqueue("");
- runner.run();
- runner.assertAllFlowFilesTransferred(PutAzureCosmosDBRecord.REL_SUCCESS, 1);
- assertEquals(4, getDataFromTestDB().size());
- }
-
- @Test
- public void testOnTriggerWithFlatRecords() throws InitializationException {
- setupRecordReader();
- recordReader.addSchemaField("id", RecordFieldType.STRING);
- recordReader.addSchemaField(TEST_COSMOS_PARTITION_KEY_FIELD_NAME, RecordFieldType.STRING);
- recordReader.addSchemaField("name", RecordFieldType.STRING);
- recordReader.addSchemaField("age", RecordFieldType.INT);
- recordReader.addSchemaField("sport", RecordFieldType.STRING);
-
- recordReader.addRecord("1", "A", "John Doe", 48, "Soccer");
- recordReader.addRecord("2", "B", "Jane Doe", 47, "Tennis");
- recordReader.addRecord("3", "B", "Sally Doe", 47, "Curling");
- recordReader.addRecord("4", "A", "Jimmy Doe", 14, null);
- recordReader.addRecord("5", "C", "Pizza Doe", 14, null);
-
- runner.enqueue("");
- runner.run();
- runner.assertAllFlowFilesTransferred(PutAzureCosmosDBRecord.REL_SUCCESS, 1);
- assertEquals(5, getDataFromTestDB().size());
- }
-}
\ No newline at end of file
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureBlobStorage_v12IT.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureBlobStorage_v12IT.java
deleted file mode 100644
index 1899dad4bc..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureBlobStorage_v12IT.java
+++ /dev/null
@@ -1,195 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import com.azure.core.cryptography.AsyncKeyEncryptionKey;
-import com.azure.security.keyvault.keys.cryptography.KeyEncryptionKeyClientBuilder;
-import com.azure.security.keyvault.keys.models.JsonWebKey;
-import com.azure.security.keyvault.keys.models.KeyOperation;
-import com.azure.storage.blob.BlobClient;
-import com.azure.storage.blob.BlobContainerClient;
-import com.azure.storage.blob.BlobServiceClient;
-import com.azure.storage.blob.BlobServiceClientBuilder;
-import com.azure.storage.blob.models.BlobType;
-import com.azure.storage.blob.specialized.cryptography.EncryptedBlobClientBuilder;
-import com.azure.storage.blob.specialized.cryptography.EncryptionVersion;
-import com.azure.storage.common.StorageSharedKeyCredential;
-import org.apache.commons.codec.binary.Hex;
-import org.apache.nifi.processors.azure.AbstractAzureBlobProcessor_v12;
-import org.apache.nifi.processors.azure.AzureServiceEndpoints;
-import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
-import org.apache.nifi.processors.azure.storage.utils.BlobAttributes;
-import org.apache.nifi.services.azure.storage.AzureStorageCredentialsControllerService_v12;
-import org.apache.nifi.services.azure.storage.AzureStorageCredentialsService_v12;
-import org.apache.nifi.services.azure.storage.AzureStorageCredentialsType;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-
-import javax.crypto.spec.SecretKeySpec;
-import java.io.ByteArrayInputStream;
-import java.io.UnsupportedEncodingException;
-import java.net.URLEncoder;
-import java.nio.charset.StandardCharsets;
-import java.util.Arrays;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.UUID;
-
-import static org.apache.nifi.processors.azure.AzureServiceEndpoints.DEFAULT_BLOB_ENDPOINT_SUFFIX;
-import static org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils.BLOB_STORAGE_CREDENTIALS_SERVICE;
-
-public abstract class AbstractAzureBlobStorage_v12IT extends AbstractAzureStorageIT {
- protected static final String SERVICE_ID = "credentials-service";
- protected static final String BLOB_NAME = "blob1";
- protected static final byte[] BLOB_DATA = "0123456789".getBytes(StandardCharsets.UTF_8);
- protected static final String KEY_ID_VALUE = "key:id";
- protected static final String KEY_64B_VALUE = "1234567890ABCDEF";
- protected static final String KEY_128B_VALUE = KEY_64B_VALUE + KEY_64B_VALUE;
- protected static final String KEY_192B_VALUE = KEY_128B_VALUE + KEY_64B_VALUE;
- protected static final String KEY_256B_VALUE = KEY_128B_VALUE + KEY_128B_VALUE;
- protected static final String KEY_384B_VALUE = KEY_256B_VALUE + KEY_128B_VALUE;
- protected static final String KEY_512B_VALUE = KEY_256B_VALUE + KEY_256B_VALUE;
-
- protected static final String EL_CONTAINER_NAME = "az.containername";
- protected static final String EL_BLOB_NAME = "az.blobname";
-
- private static final String TEST_CONTAINER_NAME_PREFIX = "nifi-test-container";
-
- private BlobServiceClient storageClient;
- private BlobContainerClient containerClient;
- private String containerName;
-
- @Override
- protected String getDefaultEndpointSuffix() {
- return DEFAULT_BLOB_ENDPOINT_SUFFIX;
- }
-
- @Override
- protected void setUpCredentials() throws Exception {
- AzureStorageCredentialsService_v12 service = new AzureStorageCredentialsControllerService_v12();
- runner.addControllerService(SERVICE_ID, service);
- runner.setProperty(service, AzureStorageUtils.ACCOUNT_NAME, getAccountName());
- if (getEndpointSuffix() != null) {
- runner.setProperty(service, AzureStorageUtils.ENDPOINT_SUFFIX, getEndpointSuffix());
- }
- runner.setProperty(service, AzureStorageUtils.CREDENTIALS_TYPE, AzureStorageCredentialsType.ACCOUNT_KEY);
- runner.setProperty(service, AzureStorageUtils.ACCOUNT_KEY, getAccountKey());
- runner.enableControllerService(service);
-
- runner.setProperty(BLOB_STORAGE_CREDENTIALS_SERVICE, SERVICE_ID);
- }
-
- @BeforeEach
- public void setUpAzureBlobStorage_v12IT() {
- containerName = generateContainerName();
-
- runner.setProperty(AzureStorageUtils.CONTAINER, containerName);
-
- storageClient = createStorageClient();
- containerClient = storageClient.createBlobContainer(containerName);
- }
-
- @AfterEach
- public void tearDownAzureBlobStorage_v12IT() {
- containerClient.delete();
- }
-
- protected String generateContainerName() {
- return String.format("%s-%s", TEST_CONTAINER_NAME_PREFIX, UUID.randomUUID());
- }
-
- protected BlobServiceClient getStorageClient() {
- return storageClient;
- }
-
- protected BlobContainerClient getContainerClient() {
- return containerClient;
- }
-
- protected String getContainerName() {
- return containerName;
- }
-
- private BlobServiceClient createStorageClient() {
- return new BlobServiceClientBuilder()
- .endpoint(AzureServiceEndpoints.getAzureBlobStorageEndpoint(getAccountName(), getEndpointSuffix()))
- .credential(new StorageSharedKeyCredential(getAccountName(), getAccountKey()))
- .buildClient();
- }
-
- protected BlobClient uploadBlob(String blobName, byte[] blobData) throws Exception {
- BlobClient blobClient = containerClient.getBlobClient(blobName);
- blobClient.upload(new ByteArrayInputStream(blobData), blobData.length);
-
- // waiting for the blob to be available
- Thread.sleep(1000);
-
- return blobClient;
- }
-
- protected BlobClient uploadBlobWithCSE(String blobName, byte[] blobData, String hexKey, String keyId, String keyWrapAlgorithm) throws Exception {
- BlobClient blobClient = containerClient.getBlobClient(blobName);
- byte[] keyBytes = Hex.decodeHex(hexKey.toCharArray());
- JsonWebKey localKey = JsonWebKey.fromAes(new SecretKeySpec(keyBytes, "AES"),
- Arrays.asList(KeyOperation.WRAP_KEY, KeyOperation.UNWRAP_KEY))
- .setId(keyId);
- AsyncKeyEncryptionKey akek = new KeyEncryptionKeyClientBuilder()
- .buildAsyncKeyEncryptionKey(localKey).block();
- BlobClient encryptedBlobClient = new EncryptedBlobClientBuilder(EncryptionVersion.V2)
- .key(akek, keyWrapAlgorithm)
- .blobClient(blobClient)
- .buildEncryptedBlobClient();
- encryptedBlobClient.upload(new ByteArrayInputStream(blobData), blobData.length);
-
- // waiting for the blob to be available
- Thread.sleep(1000);
-
- return encryptedBlobClient;
- }
-
- protected Map<String, String> initCommonExpressionLanguageAttributes() {
- Map<String, String> attributes = new HashMap<>();
- attributes.put(EL_CONTAINER_NAME, getContainerName());
- attributes.put(EL_BLOB_NAME, BLOB_NAME);
-
- runner.setProperty(AzureStorageUtils.CONTAINER, String.format("${%s}", EL_CONTAINER_NAME));
- runner.setProperty(AbstractAzureBlobProcessor_v12.BLOB_NAME, String.format("${%s}", EL_BLOB_NAME));
-
- return attributes;
- }
-
- protected void assertFlowFileCommonBlobAttributes(MockFlowFile flowFile, String containerName, String blobName) throws UnsupportedEncodingException {
- flowFile.assertAttributeEquals(BlobAttributes.ATTR_NAME_CONTAINER, containerName);
- flowFile.assertAttributeEquals(BlobAttributes.ATTR_NAME_BLOBNAME, blobName);
- flowFile.assertAttributeEquals(BlobAttributes.ATTR_NAME_PRIMARY_URI,
- String.format("https://%s.blob.core.windows.net/%s/%s", getAccountName(), containerName, URLEncoder.encode(
- blobName,
- StandardCharsets.US_ASCII
- ).replace("+", "%20").replace("%2F", "/"))
- );
- }
-
- protected void assertFlowFileResultBlobAttributes(MockFlowFile flowFile, int blobLength) {
- flowFile.assertAttributeExists(BlobAttributes.ATTR_NAME_ETAG);
- flowFile.assertAttributeEquals(BlobAttributes.ATTR_NAME_BLOBTYPE, BlobType.BLOCK_BLOB.toString());
- flowFile.assertAttributeEquals(BlobAttributes.ATTR_NAME_MIME_TYPE, "application/octet-stream");
- flowFile.assertAttributeExists(BlobAttributes.ATTR_NAME_LANG);
- flowFile.assertAttributeExists(BlobAttributes.ATTR_NAME_TIMESTAMP);
- flowFile.assertAttributeEquals(BlobAttributes.ATTR_NAME_LENGTH, String.valueOf(blobLength));
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureDataLakeStorageIT.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureDataLakeStorageIT.java
deleted file mode 100644
index fc5f2e6232..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureDataLakeStorageIT.java
+++ /dev/null
@@ -1,151 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import com.azure.storage.common.StorageSharedKeyCredential;
-import com.azure.storage.file.datalake.DataLakeDirectoryClient;
-import com.azure.storage.file.datalake.DataLakeFileClient;
-import com.azure.storage.file.datalake.DataLakeFileSystemClient;
-import com.azure.storage.file.datalake.DataLakeServiceClient;
-import com.azure.storage.file.datalake.DataLakeServiceClientBuilder;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
-import org.apache.nifi.services.azure.storage.ADLSCredentialsControllerService;
-import org.apache.nifi.services.azure.storage.ADLSCredentialsService;
-import org.apache.nifi.services.azure.storage.AzureStorageCredentialsType;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-
-import java.io.ByteArrayInputStream;
-import java.nio.charset.StandardCharsets;
-import java.util.UUID;
-
-import static org.apache.nifi.processors.azure.AzureServiceEndpoints.DEFAULT_ADLS_ENDPOINT_SUFFIX;
-
-public abstract class AbstractAzureDataLakeStorageIT extends AbstractAzureStorageIT {
-
- private static final String FILESYSTEM_NAME_PREFIX = "nifi-test-filesystem";
-
- protected static final String TEST_FILE_CONTENT = "ABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890";
-
- protected String fileSystemName;
- protected DataLakeFileSystemClient fileSystemClient;
-
- @Override
- protected String getDefaultEndpointSuffix() {
- return DEFAULT_ADLS_ENDPOINT_SUFFIX;
- }
-
- @Override
- protected void setUpCredentials() throws Exception {
- ADLSCredentialsService service = new ADLSCredentialsControllerService();
- runner.addControllerService("ADLSCredentials", service);
- runner.setProperty(service, AzureStorageUtils.CREDENTIALS_TYPE, AzureStorageCredentialsType.ACCOUNT_KEY);
- runner.setProperty(service, ADLSCredentialsControllerService.ACCOUNT_NAME, getAccountName());
- runner.setProperty(service, AzureStorageUtils.ACCOUNT_KEY, getAccountKey());
- runner.enableControllerService(service);
-
- runner.setProperty(AzureStorageUtils.ADLS_CREDENTIALS_SERVICE, "ADLSCredentials");
- }
-
- @BeforeEach
- public void setUpAzureDataLakeStorageIT() {
- fileSystemName = String.format("%s-%s", FILESYSTEM_NAME_PREFIX, UUID.randomUUID());
-
- runner.setProperty(AzureStorageUtils.FILESYSTEM, fileSystemName);
-
- DataLakeServiceClient storageClient = createStorageClient();
- fileSystemClient = storageClient.createFileSystem(fileSystemName);
- }
-
- @AfterEach
- public void tearDownAzureDataLakeStorageIT() {
- fileSystemClient.delete();
- }
-
- private DataLakeServiceClient createStorageClient() {
- return new DataLakeServiceClientBuilder()
- .endpoint("https://" + getAccountName() + ".dfs.core.windows.net")
- .credential(new StorageSharedKeyCredential(getAccountName(), getAccountKey()))
- .buildClient();
- }
-
- protected void createDirectory(String directory) {
- fileSystemClient.createDirectory(directory);
- }
-
- protected void uploadFile(String directory, String filename, String fileContent) {
- uploadFile(directory, filename, fileContent.getBytes(StandardCharsets.UTF_8));
- }
-
- protected void uploadFile(TestFile testFile) {
- uploadFile(testFile.getDirectory(), testFile.getFilename(), testFile.getFileContent());
- }
-
- protected void uploadFile(String directory, String filename, byte[] fileData) {
- DataLakeDirectoryClient directoryClient = fileSystemClient.getDirectoryClient(directory);
- DataLakeFileClient fileClient = directoryClient.createFile(filename);
-
- PutAzureDataLakeStorage.uploadContent(fileClient, new ByteArrayInputStream(fileData), fileData.length);
- }
-
- protected void createDirectoryAndUploadFile(String directory, String filename, String fileContent) {
- createDirectoryAndUploadFile(directory, filename, fileContent.getBytes(StandardCharsets.UTF_8));
- }
-
- protected void createDirectoryAndUploadFile(String directory, String filename, byte[] fileData) {
- createDirectory(directory);
-
- uploadFile(directory, filename, fileData);
- }
-
- protected void createDirectoryAndUploadFile(TestFile testFile) {
- createDirectoryAndUploadFile(testFile.getDirectory(), testFile.getFilename(), testFile.getFileContent());
- }
-
- protected static class TestFile {
- private final String directory;
- private final String filename;
- private final String fileContent;
-
- public TestFile(String directory, String filename, String fileContent) {
- this.directory = directory;
- this.filename = filename;
- this.fileContent = fileContent;
- }
-
- public TestFile(String directory, String filename) {
- this(directory, filename, TEST_FILE_CONTENT);
- }
-
- public String getDirectory() {
- return directory;
- }
-
- public String getFilename() {
- return filename;
- }
-
- public String getFileContent() {
- return fileContent;
- }
-
- public String getFilePath() {
- return StringUtils.isNotBlank(directory) ? String.format("%s/%s", directory, filename) : filename;
- }
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureStorageIT.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureStorageIT.java
deleted file mode 100644
index aab84320f1..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/AbstractAzureStorageIT.java
+++ /dev/null
@@ -1,132 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
-import org.apache.nifi.proxy.StandardProxyConfigurationService;
-import org.apache.nifi.reporting.InitializationException;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.apache.nifi.util.file.FileUtils;
-import org.junit.jupiter.api.BeforeEach;
-
-import java.io.FileInputStream;
-import java.util.Properties;
-
-import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
-
-public abstract class AbstractAzureStorageIT {
-
- protected static final byte[] EMPTY_CONTENT = new byte[0];
-
- private static final Properties CREDENTIALS_CONFIG;
- private static final Properties PROXY_CONFIG;
-
- private static final String CREDENTIALS_FILE = System.getProperty("user.home") + "/azure-credentials.PROPERTIES";
- private static final String PROXY_CONFIGURATION_FILE = System.getProperty("user.home") + "/proxy-configuration.PROPERTIES";
-
- static {
- CREDENTIALS_CONFIG = loadConfig(CREDENTIALS_FILE);
- PROXY_CONFIG = loadConfig(PROXY_CONFIGURATION_FILE);
- }
-
- private static Properties loadConfig(String configPath) {
- Properties loadedProperties = new Properties();
-
- assertDoesNotThrow(() -> {
- final FileInputStream fIS = new FileInputStream(configPath);
- assertDoesNotThrow(() -> loadedProperties.load(fIS));
- FileUtils.closeQuietly(fIS);
- });
-
- return loadedProperties;
- }
-
- protected String getAccountName() {
- return CREDENTIALS_CONFIG.getProperty("accountName");
- }
-
- protected String getAccountKey() {
- return CREDENTIALS_CONFIG.getProperty("accountKey");
- }
-
- protected String getEndpointSuffix() {
- String endpointSuffix = CREDENTIALS_CONFIG.getProperty("endpointSuffix");
- return endpointSuffix != null ? endpointSuffix : getDefaultEndpointSuffix();
- }
-
- protected String getProxyType() {
- return PROXY_CONFIG.getProperty("proxyType");
- }
-
- protected String getSocksVersion() {
- return PROXY_CONFIG.getProperty("socksVersion");
- }
-
- protected String getProxyServerHost() {
- return PROXY_CONFIG.getProperty("proxyServerHost");
- }
-
- protected String getProxyServerPort() {
- return PROXY_CONFIG.getProperty("proxyServerPort");
- }
-
- protected String getProxyUsername() {
- return PROXY_CONFIG.getProperty("proxyUsername");
- }
-
- protected String getProxyUserPassword() {
- return PROXY_CONFIG.getProperty("proxyUserPassword");
- }
-
- protected abstract String getDefaultEndpointSuffix();
-
- protected TestRunner runner;
-
- @BeforeEach
- public void setUpAzureStorageIT() throws Exception {
- runner = TestRunners.newTestRunner(getProcessorClass());
-
- setUpCredentials();
- }
-
- protected void setUpCredentials() throws Exception {
- runner.setProperty(AzureStorageUtils.ACCOUNT_NAME, getAccountName());
- runner.setProperty(AzureStorageUtils.ACCOUNT_KEY, getAccountKey());
- }
-
- protected abstract Class<? extends Processor> getProcessorClass();
-
- protected void configureProxyService() throws InitializationException {
- final StandardProxyConfigurationService proxyConfigurationService = new StandardProxyConfigurationService();
- runner.addControllerService("proxy-configuration-service", proxyConfigurationService);
-
- runner.setProperty(proxyConfigurationService, StandardProxyConfigurationService.PROXY_TYPE, getProxyType());
- runner.setProperty(proxyConfigurationService, StandardProxyConfigurationService.SOCKS_VERSION, getSocksVersion());
- runner.setProperty(proxyConfigurationService, StandardProxyConfigurationService.PROXY_SERVER_HOST, getProxyServerHost());
- runner.setProperty(proxyConfigurationService, StandardProxyConfigurationService.PROXY_SERVER_PORT, getProxyServerPort());
- runner.setProperty(proxyConfigurationService, StandardProxyConfigurationService.PROXY_USER_NAME, getProxyUsername());
- runner.setProperty(proxyConfigurationService, StandardProxyConfigurationService.PROXY_USER_PASSWORD, getProxyUserPassword());
-
- runner.assertValid(proxyConfigurationService);
-
- runner.enableControllerService(proxyConfigurationService);
-
- runner.setProperty(AzureStorageUtils.PROXY_CONFIGURATION_SERVICE, proxyConfigurationService.getIdentifier());
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITCopyAzureBlobStorage_v12.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITCopyAzureBlobStorage_v12.java
deleted file mode 100644
index d71b421ada..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITCopyAzureBlobStorage_v12.java
+++ /dev/null
@@ -1,122 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import com.azure.storage.blob.BlobClient;
-import com.azure.storage.blob.BlobContainerClient;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.provenance.ProvenanceEventRecord;
-import org.apache.nifi.provenance.ProvenanceEventType;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.params.ParameterizedTest;
-import org.junit.jupiter.params.provider.ValueSource;
-
-import java.util.Collections;
-import java.util.Map;
-import java.util.Set;
-import java.util.stream.Collectors;
-
-import static org.apache.nifi.processors.azure.storage.CopyAzureBlobStorage_v12.SOURCE_STORAGE_CREDENTIALS_SERVICE;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-public class ITCopyAzureBlobStorage_v12 extends AbstractAzureBlobStorage_v12IT {
- public static class ITProcessor extends CopyAzureBlobStorage_v12 {
- public boolean blobMetadataApplied = false;
- @Override
- protected void applyBlobMetadata(Map<String, String> attributes, BlobClient blobClient) {
- super.applyBlobMetadata(attributes, blobClient);
- blobMetadataApplied = true;
- }
- }
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return ITProcessor.class;
- }
-
- @BeforeEach
- public void setUp() {
- runner.setProperty(CopyAzureBlobStorage_v12.SOURCE_CONTAINER_NAME, getContainerName());
- runner.setProperty(CopyAzureBlobStorage_v12.SOURCE_BLOB_NAME, BLOB_NAME);
- }
-
- @ValueSource(booleans={true, false})
- @ParameterizedTest
- public void testPutBlobFromUrl(boolean proxied) throws Exception {
- if (proxied) {
- configureProxyService();
- }
-
- uploadBlob(BLOB_NAME, BLOB_DATA);
- final String destinationBlobName = BLOB_NAME + "-target";
- runner.setProperty(CopyAzureBlobStorage_v12.DESTINATION_BLOB_NAME, destinationBlobName);
- runner.setProperty(SOURCE_STORAGE_CREDENTIALS_SERVICE, SERVICE_ID);
- runner.setProperty(CopyAzureBlobStorage_v12.SOURCE_BLOB_NAME, BLOB_NAME);
-
- runProcessor(BLOB_DATA);
-
- assertSuccess(getContainerName(), destinationBlobName, BLOB_DATA);
- assertTrue(((ITProcessor) runner.getProcessor()).blobMetadataApplied);
- }
-
- private void runProcessor(byte[] data) {
- runProcessor(data, Collections.emptyMap());
- }
-
- private void runProcessor(byte[] data, Map<String, String> attributes) {
- runner.assertValid();
- runner.enqueue(data, attributes);
- runner.run();
- }
-
- private void assertSuccess(String containerName, String blobName, byte[] blobData) throws Exception {
- assertFlowFile(containerName, blobName, blobData);
- assertAzureBlob(containerName, blobName, blobData);
- assertProvenanceEvents();
- }
-
- private void assertFlowFile(String containerName, String blobName, byte[] blobData) throws Exception {
- runner.assertAllFlowFilesTransferred(CopyAzureBlobStorage_v12.REL_SUCCESS, 1);
-
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(CopyAzureBlobStorage_v12.REL_SUCCESS).getFirst();
-
- assertFlowFileCommonBlobAttributes(flowFile, containerName, blobName);
- if (blobData != null) {
- assertFlowFileResultBlobAttributes(flowFile, blobData.length);
- flowFile.assertContentEquals(blobData);
- }
- }
-
- private void assertAzureBlob(String containerName, String blobName, byte[] blobData) {
- BlobContainerClient containerClient = getStorageClient().getBlobContainerClient(containerName);
- BlobClient blobClient = containerClient.getBlobClient(blobName);
-
- assertTrue(blobClient.exists());
- assertEquals(blobData.length, blobClient.getProperties().getBlobSize());
- }
-
- private void assertProvenanceEvents() {
- Set<ProvenanceEventType> expectedEventTypes = Collections.singleton(ProvenanceEventType.SEND);
-
- Set<ProvenanceEventType> actualEventTypes = runner.getProvenanceEvents().stream()
- .map(ProvenanceEventRecord::getEventType)
- .collect(Collectors.toSet());
- assertEquals(expectedEventTypes, actualEventTypes);
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITDeleteAzureBlobStorage_v12.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITDeleteAzureBlobStorage_v12.java
deleted file mode 100644
index c39baa224e..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITDeleteAzureBlobStorage_v12.java
+++ /dev/null
@@ -1,194 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import com.azure.storage.blob.BlobClient;
-import com.azure.storage.blob.specialized.BlobClientBase;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processor.Relationship;
-import org.apache.nifi.provenance.ProvenanceEventRecord;
-import org.apache.nifi.provenance.ProvenanceEventType;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.util.Collections;
-import java.util.Map;
-import java.util.Set;
-import java.util.stream.Collectors;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertFalse;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-public class ITDeleteAzureBlobStorage_v12 extends AbstractAzureBlobStorage_v12IT {
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return DeleteAzureBlobStorage_v12.class;
- }
-
- @BeforeEach
- public void setUp() {
- runner.setProperty(DeleteAzureBlobStorage_v12.BLOB_NAME, BLOB_NAME);
- }
-
- @Test
- public void testDeleteBlobWithSimpleName() throws Exception {
- uploadBlob(BLOB_NAME, BLOB_DATA);
-
- runProcessor();
-
- assertSuccess(BLOB_NAME);
- }
-
- @Test
- public void testDeleteBlobWithSimpleNameUsingProxyConfigurationService() throws Exception {
- uploadBlob(BLOB_NAME, BLOB_DATA);
-
- configureProxyService();
-
- runProcessor();
-
- assertSuccess(BLOB_NAME);
- }
-
- @Test
- public void testDeleteBlobWithCompoundName() throws Exception {
- String blobName = "dir1/dir2/blob1";
- runner.setProperty(DeleteAzureBlobStorage_v12.BLOB_NAME, blobName);
- uploadBlob(blobName, BLOB_DATA);
-
- runProcessor();
-
- assertSuccess(blobName);
- }
-
- @Test
- public void testDeleteNonExistingBlob() throws Exception {
- runProcessor();
-
- assertSuccess(BLOB_NAME);
- }
-
- @Test
- public void testDeleteBlobWithSpacesInBlobName() throws Exception {
- String blobName = "dir 1/blob 1";
- runner.setProperty(DeleteAzureBlobStorage_v12.BLOB_NAME, blobName);
- uploadBlob(blobName, BLOB_DATA);
-
- runProcessor();
-
- assertSuccess(blobName);
- }
-
- @Test
- public void testDeleteBlobUsingExpressionLanguage() throws Exception {
- Map<String, String> attributes = initCommonExpressionLanguageAttributes();
-
- uploadBlob(BLOB_NAME, BLOB_DATA);
-
- runProcessor(attributes);
-
- assertSuccess(BLOB_NAME);
- }
-
- @Test
- public void testDeleteBlobWithSnapshotAndDeleteSnapshotsOptionNone() throws Exception {
- runner.setProperty(DeleteAzureBlobStorage_v12.DELETE_SNAPSHOTS_OPTION, DeleteAzureBlobStorage_v12.DELETE_SNAPSHOTS_NONE);
-
- BlobClient blobClient = uploadBlob(BLOB_NAME, BLOB_DATA);
- BlobClientBase snapshotClient = blobClient.createSnapshot();
-
- runProcessor();
-
- assertFailure(BLOB_NAME);
- assertTrue(snapshotClient.exists());
- }
-
- @Test
- public void testDeleteBlobWithSnapshotAndDeleteSnapshotsOptionInclude() throws Exception {
- runner.setProperty(DeleteAzureBlobStorage_v12.DELETE_SNAPSHOTS_OPTION, DeleteAzureBlobStorage_v12.DELETE_SNAPSHOTS_ALSO);
-
- BlobClient blobClient = uploadBlob(BLOB_NAME, BLOB_DATA);
- BlobClientBase snapshotClient = blobClient.createSnapshot();
-
- runProcessor();
-
- assertSuccess(BLOB_NAME);
- assertFalse(snapshotClient.exists());
- }
-
- @Test
- public void testDeleteBlobWithSnapshotAndDeleteSnapshotsOptionOnly() throws Exception {
- runner.setProperty(DeleteAzureBlobStorage_v12.DELETE_SNAPSHOTS_OPTION, DeleteAzureBlobStorage_v12.DELETE_SNAPSHOTS_ONLY);
-
- BlobClient blobClient = uploadBlob(BLOB_NAME, BLOB_DATA);
- BlobClientBase snapshotClient = blobClient.createSnapshot();
-
- runProcessor();
-
- assertFlowFile(DeleteAzureBlobStorage_v12.REL_SUCCESS);
- assertTrue(blobClient.exists());
- assertFalse(snapshotClient.exists());
- assertProvenanceEvents();
- }
-
- private void runProcessor() {
- runProcessor(Collections.emptyMap());
- }
-
- private void runProcessor(Map<String, String> attributes) {
- runner.assertValid();
- runner.enqueue(EMPTY_CONTENT, attributes);
- runner.run();
- }
-
- private void assertSuccess(String blobName) throws Exception {
- assertFlowFile(DeleteAzureBlobStorage_v12.REL_SUCCESS);
- assertAzureBlob(blobName, false);
- assertProvenanceEvents();
- }
-
- private void assertFailure(String blobName) throws Exception {
- assertFlowFile(DeleteAzureBlobStorage_v12.REL_FAILURE);
- assertAzureBlob(blobName, true);
- }
-
- private void assertFlowFile(Relationship relationship) throws Exception {
- runner.assertAllFlowFilesTransferred(relationship, 1);
-
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(relationship).getFirst();
-
- flowFile.assertContentEquals(EMPTY_CONTENT);
- }
-
- private void assertAzureBlob(String blobName, boolean expectedExists) {
- BlobClient blobClient = getContainerClient().getBlobClient(blobName);
-
- assertEquals(expectedExists, blobClient.exists());
- }
-
- private void assertProvenanceEvents() {
- Set<ProvenanceEventType> expectedEventTypes = Collections.singleton(ProvenanceEventType.REMOTE_INVOCATION);
-
- Set<ProvenanceEventType> actualEventTypes = runner.getProvenanceEvents().stream()
- .map(ProvenanceEventRecord::getEventType)
- .collect(Collectors.toSet());
- assertEquals(expectedEventTypes, actualEventTypes);
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITDeleteAzureDataLakeStorage.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITDeleteAzureDataLakeStorage.java
deleted file mode 100644
index fc30522ef6..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITDeleteAzureDataLakeStorage.java
+++ /dev/null
@@ -1,501 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import com.azure.storage.file.datalake.DataLakeDirectoryClient;
-import com.azure.storage.file.datalake.DataLakeFileClient;
-import com.azure.storage.file.datalake.models.DataLakeStorageException;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
-import org.apache.nifi.provenance.ProvenanceEventType;
-import org.apache.nifi.reporting.InitializationException;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.Test;
-
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-
-import static org.apache.nifi.processors.azure.storage.DeleteAzureDataLakeStorage.FS_TYPE_DIRECTORY;
-import static org.apache.nifi.processors.azure.storage.DeleteAzureDataLakeStorage.FS_TYPE_FILE;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertFalse;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-public class ITDeleteAzureDataLakeStorage extends AbstractAzureDataLakeStorageIT {
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return DeleteAzureDataLakeStorage.class;
- }
-
- @Test
- public void testDeleteDirectoryWithFiles() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, directory, null, inputFlowFileContent, inputFlowFileContent);
- }
-
- @Test
- public void testDeleteDirectoryWithFilesUsingProxyConfigurationService() throws InitializationException {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- configureProxyService();
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, directory, null, inputFlowFileContent, inputFlowFileContent);
- }
-
- @Test
- public void testDeleteEmptyDirectoryWithFSTypeDirectory() {
- // GIVEN
- String directory = "TestDirectory";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectory(directory);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, directory, null, inputFlowFileContent, inputFlowFileContent);
- }
-
- @Test
- public void testDeleteSubdirectory() {
- // GIVEN
- String parentDirectory = "TestParentDirectory";
- String childDirectory = "TestParentDirectory/TestChildDirectory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectory(parentDirectory);
- createDirectoryAndUploadFile(childDirectory, filename, fileContent);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, childDirectory, null, inputFlowFileContent, inputFlowFileContent);
- assertTrue(directoryExists(parentDirectory));
- }
-
- @Test
- public void testDeleteParentDirectory() {
- // GIVEN
- String parentDirectory = "TestParentDirectory";
- String childDirectory = "TestParentDirectory/TestChildDirectory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectory(parentDirectory);
- createDirectoryAndUploadFile(childDirectory, filename, fileContent);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, parentDirectory, null, inputFlowFileContent, inputFlowFileContent);
- assertFalse(directoryExists(childDirectory));
- }
-
- @Test
- public void testDeleteFileFromRoot() {
- // GIVEN
- String directory = "";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- uploadFile(directory, filename, fileContent);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, directory, filename, inputFlowFileContent, inputFlowFileContent);
- }
-
- @Test
- public void testDeleteFileFromDirectory() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, directory, filename, inputFlowFileContent, inputFlowFileContent);
- }
-
- @Test
- public void testDeleteFileFromDeepDirectory() {
- // GIVEN
- String directory = "Directory01/Directory02/Directory03/Directory04/Directory05/Directory06/Directory07/"
- + "Directory08/Directory09/Directory10/Directory11/Directory12/Directory13/Directory14/Directory15/"
- + "Directory16/Directory17/Directory18/Directory19/Directory20/TestDirectory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, directory, filename, inputFlowFileContent, inputFlowFileContent);
- }
-
- @Test
- public void testDeleteFileWithWhitespaceInFilename() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "A test file.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, directory, filename, inputFlowFileContent, inputFlowFileContent);
- }
-
- @Test
- public void testDeleteFileWithWhitespaceInDirectoryName() {
- // GIVEN
- String directory = "A Test Directory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, directory, filename, inputFlowFileContent, inputFlowFileContent);
- }
-
- @Test
- public void testDeleteEmptyDirectoryWithFSTypeFile() {
- // GIVEN
- String parentDirectory = "ParentDirectory";
- String childDirectory = "ChildDirectory";
- String inputFlowFileContent = "InputFlowFileContent";
-
- fileSystemClient.createDirectory(parentDirectory + "/" + childDirectory);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, parentDirectory, childDirectory, inputFlowFileContent, inputFlowFileContent);
- }
-
- @Test
- public void testDeleteFileCaseSensitiveFilename() {
- // GIVEN
- String directory = "TestDirectory";
- String filename1 = "testFile.txt";
- String filename2 = "testfile.txt";
- String fileContent1 = "ContentOfFile1";
- String fileContent2 = "ContentOfFile2";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename1, fileContent1);
- uploadFile(directory, filename2, fileContent2);
-
- // WHEN
- // THEN
- testSuccessfulDelete(fileSystemName, directory, filename1, inputFlowFileContent, inputFlowFileContent);
- assertTrue(fileExists(directory, filename2));
- }
-
- @Test
- public void testDeleteUsingExpressionLanguage() {
- // GIVEN
- String expLangFileSystem = "az.filesystem";
- String expLangDirectory = "az.directory";
- String expLangFilename = "az.filename";
-
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
-
- String inputFlowFileContent = "InputFlowFileContent";
-
- Map<String, String> attributes = new HashMap<>();
- attributes.put(expLangFileSystem, fileSystemName);
- attributes.put(expLangDirectory, directory);
- attributes.put(expLangFilename, filename);
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- // WHEN
- // THEN
- testSuccessfulDeleteWithExpressionLanguage("${" + expLangFileSystem + "}",
- "${" + expLangDirectory + "}",
- "${" + expLangFilename + "}",
- attributes,
- inputFlowFileContent,
- inputFlowFileContent,
- directory,
- filename);
- }
-
- @Test
- public void testDeleteUsingExpressionLanguageFileSystemIsNotSpecified() {
- // GIVEN
- String expLangFileSystem = "az.filesystem";
- String expLangDirectory = "az.directory";
- String expLangFilename = "az.filename";
-
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
-
- String inputFlowFileContent = "InputFlowFileContent";
-
- Map<String, String> attributes = new HashMap<>();
- attributes.put(expLangDirectory, directory);
- attributes.put(expLangFilename, filename);
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- // WHEN
- // THEN
- testFailedDeleteWithProcessException("${" + expLangFileSystem + "}",
- "${" + expLangDirectory + "}",
- "${" + expLangFilename + "}",
- attributes,
- inputFlowFileContent,
- inputFlowFileContent);
- assertTrue(fileExists(directory, filename));
- }
-
- @Test
- public void testDeleteUsingExpressionLanguageFilenameIsNotSpecified() {
- // GIVEN
- String expLangFileSystem = "az.filesystem";
- String expLangDirectory = "az.directory";
- String expLangFilename = "az.filename";
-
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
-
- String inputFlowFileContent = "InputFlowFileContent";
- Map<String, String> attributes = new HashMap<>();
- attributes.put(expLangFileSystem, fileSystemName);
- attributes.put(expLangDirectory, directory);
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- // WHEN
- // THEN
- testFailedDeleteWithProcessException("${" + expLangFileSystem + "}",
- "${" + expLangDirectory + "}",
- "${" + expLangFilename + "}",
- attributes,
- inputFlowFileContent,
- inputFlowFileContent);
- assertTrue(fileExists(directory, filename));
- }
-
- @Test
- public void testDeleteNonExistentFile() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- fileSystemClient.createDirectory(directory);
-
- // WHEN
- // THEN
- testFailedDelete(fileSystemName, directory, filename, inputFlowFileContent, inputFlowFileContent, 404);
- assertTrue(fileExists("", directory));
- }
-
- @Test
- public void testDeleteFileFromNonExistentDirectory() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- // WHEN
- // THEN
- testFailedDelete(fileSystemName, directory, filename, inputFlowFileContent, inputFlowFileContent, 404);
- }
-
- @Test
- public void testDeleteFileFromNonExistentFileSystem() {
- // GIVEN
- String fileSystem = "NonExistentFileSystem";
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- // WHEN
- // THEN
- testFailedDelete(fileSystem, directory, filename, inputFlowFileContent, inputFlowFileContent, 400);
- }
-
- @Test
- public void testDeleteNonEmptyDirectoryWithFSTypeFile() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- testFailedDelete(fileSystemName, "", directory, inputFlowFileContent, inputFlowFileContent, 409);
- assertTrue(fileExists(directory, filename));
- }
-
- @Test
- public void testDeleteFileWithInvalidFilename() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String invalidFilename = "test/\\File.txt";
- String fileContent = "AzureFileContent";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, fileContent);
-
- // WHEN
- // THEN
- testFailedDelete(fileSystemName, directory, invalidFilename, inputFlowFileContent, inputFlowFileContent, 400);
- assertTrue(fileExists(directory, filename));
- }
-
- private void testSuccessfulDelete(String fileSystem, String directory, String filename, String inputFlowFileContent, String expectedFlowFileContent) {
- testSuccessfulDeleteWithExpressionLanguage(fileSystem, directory, filename, Collections.emptyMap(), inputFlowFileContent, expectedFlowFileContent,
- directory, filename);
- }
-
- private void testSuccessfulDeleteWithExpressionLanguage(String expLangFileSystem, String expLangDirectory, String expLangFilename, Map<String, String> attributes,
- String inputFlowFileContent, String expectedFlowFileContent, String directory, String filename) {
- // GIVEN
- int expectedNumberOfProvenanceEvents = 1;
- ProvenanceEventType expectedEventType = ProvenanceEventType.REMOTE_INVOCATION;
-
- setRunnerProperties(expLangFileSystem, expLangDirectory, expLangFilename);
-
- // WHEN
- startRunner(inputFlowFileContent, attributes);
-
- // THEN
- assertSuccess(directory, filename, expectedFlowFileContent, expectedNumberOfProvenanceEvents, expectedEventType);
- }
-
- private void testFailedDelete(String fileSystem, String directory, String filename, String inputFlowFileContent, String expectedFlowFileContent, int expectedErrorCode) {
- // GIVEN
- setRunnerProperties(fileSystem, directory, filename);
-
- // WHEN
- startRunner(inputFlowFileContent, Collections.emptyMap());
-
- // THEN
- DataLakeStorageException e = (DataLakeStorageException) runner.getLogger().getErrorMessages().getFirst().getThrowable();
- assertEquals(expectedErrorCode, e.getStatusCode());
-
- assertFailure(expectedFlowFileContent);
- }
-
- private void testFailedDeleteWithProcessException(String fileSystem, String directory, String filename, Map<String, String> attributes,
- String inputFlowFileContent, String expectedFlowFileContent) {
- // GIVEN
- setRunnerProperties(fileSystem, directory, filename);
-
- // WHEN
- startRunner(inputFlowFileContent, attributes);
-
- // THEN
- Throwable exception = runner.getLogger().getErrorMessages().getFirst().getThrowable();
- assertEquals(ProcessException.class, exception.getClass());
-
- assertFailure(expectedFlowFileContent);
- }
-
- private boolean fileExists(String directory, String filename) {
- DataLakeDirectoryClient directoryClient = fileSystemClient.getDirectoryClient(directory);
- DataLakeFileClient fileClient = directoryClient.getFileClient(filename);
-
- return fileClient.exists();
- }
-
- private boolean directoryExists(String directory) {
- DataLakeDirectoryClient directoryClient = fileSystemClient.getDirectoryClient(directory);
-
- return directoryClient.exists();
- }
-
- private void setRunnerProperties(String fileSystem, String directory, String filename) {
- runner.setProperty(DeleteAzureDataLakeStorage.FILESYSTEM_OBJECT_TYPE, filename != null ? FS_TYPE_FILE : FS_TYPE_DIRECTORY);
- runner.setProperty(AzureStorageUtils.FILESYSTEM, fileSystem);
- runner.setProperty(AzureStorageUtils.DIRECTORY, directory);
- if (filename != null) {
- runner.setProperty(DeleteAzureDataLakeStorage.FILE, filename);
- }
- runner.assertValid();
- }
-
- private void startRunner(String inputFlowFileContent, Map<String, String> attributes) {
- runner.enqueue(inputFlowFileContent, attributes);
- runner.run();
- }
-
- private void assertSuccess(String directory, String filename, String expectedFlowFileContent, int expectedNumberOfProvenanceEvents, ProvenanceEventType expectedEventType) {
- runner.assertAllFlowFilesTransferred(DeleteAzureDataLakeStorage.REL_SUCCESS, 1);
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(DeleteAzureDataLakeStorage.REL_SUCCESS).getFirst();
- flowFile.assertContentEquals(expectedFlowFileContent);
-
- int actualNumberOfProvenanceEvents = runner.getProvenanceEvents().size();
- assertEquals(expectedNumberOfProvenanceEvents, actualNumberOfProvenanceEvents);
-
- ProvenanceEventType actualEventType = runner.getProvenanceEvents().getFirst().getEventType();
- assertEquals(expectedEventType, actualEventType);
-
- if (filename != null) {
- assertFalse(fileExists(directory, filename));
- } else {
- assertFalse(directoryExists(directory));
- }
- }
-
- private void assertFailure(String expectedFlowFileContent) {
- runner.assertAllFlowFilesTransferred(DeleteAzureDataLakeStorage.REL_FAILURE, 1);
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(DeleteAzureDataLakeStorage.REL_FAILURE).getFirst();
- flowFile.assertContentEquals(expectedFlowFileContent);
- }
-
-}
\ No newline at end of file
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITFetchAzureBlobStorage_v12.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITFetchAzureBlobStorage_v12.java
deleted file mode 100644
index b076217434..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITFetchAzureBlobStorage_v12.java
+++ /dev/null
@@ -1,232 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import com.azure.security.keyvault.keys.cryptography.models.KeyWrapAlgorithm;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processors.azure.ClientSideEncryptionSupport;
-import org.apache.nifi.processors.azure.storage.utils.ClientSideEncryptionMethod;
-import org.apache.nifi.provenance.ProvenanceEventRecord;
-import org.apache.nifi.provenance.ProvenanceEventType;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.Map;
-import java.util.Random;
-import java.util.Set;
-import java.util.stream.Collectors;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public class ITFetchAzureBlobStorage_v12 extends AbstractAzureBlobStorage_v12IT {
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return FetchAzureBlobStorage_v12.class;
- }
-
- @BeforeEach
- public void setUp() {
- runner.setProperty(DeleteAzureBlobStorage_v12.BLOB_NAME, BLOB_NAME);
- }
-
- @Test
- public void testFetchBlobWithSimpleName() throws Exception {
- uploadBlob(BLOB_NAME, BLOB_DATA);
-
- runProcessor();
-
- assertSuccess(BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testFetchBlobWithCSE() throws Exception {
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_TYPE, ClientSideEncryptionMethod.LOCAL);
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_ID, KEY_ID_VALUE);
- runner.setProperty(ClientSideEncryptionSupport.CSE_LOCAL_KEY, KEY_128B_VALUE);
- uploadBlobWithCSE(BLOB_NAME, BLOB_DATA, KEY_128B_VALUE, KEY_ID_VALUE, KeyWrapAlgorithm.A128KW.toString());
-
- runProcessor();
- //cannot validate blob size as azure api does not expose unencrypted data length
- assertFlowFile(BLOB_NAME, BLOB_DATA, null);
- assertProvenanceEvents();
- }
-
- @Test
- public void testFetchBlobWithSimpleNameUsingProxyConfigurationService() throws Exception {
- uploadBlob(BLOB_NAME, BLOB_DATA);
-
- configureProxyService();
-
- runProcessor();
-
- assertSuccess(BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testFetchBlobWithCompoundName() throws Exception {
- String blobName = "dir1/dir2/blob1";
- runner.setProperty(DeleteAzureBlobStorage_v12.BLOB_NAME, blobName);
- uploadBlob(blobName, BLOB_DATA);
-
- runProcessor();
-
- assertSuccess(blobName, BLOB_DATA);
- }
-
- @Test
- public void testFetchNonExistingBlob() throws Exception {
- runProcessor();
-
- assertFailure();
- }
-
- @Test
- public void testFetchBlobWithSpacesInBlobName() throws Exception {
- String blobName = "dir 1/blob 1";
- runner.setProperty(FetchAzureBlobStorage_v12.BLOB_NAME, blobName);
- uploadBlob(blobName, BLOB_DATA);
-
- runProcessor();
-
- assertSuccess(blobName, BLOB_DATA);
- }
-
- @Test
- public void testFetchEmptyBlob() throws Exception {
- uploadBlob(BLOB_NAME, EMPTY_CONTENT);
-
- runProcessor();
-
- assertSuccess(BLOB_NAME, EMPTY_CONTENT);
- }
-
- @Test
- public void testFetchBigBlob() throws Exception {
- Random random = new Random();
- byte[] blobData = new byte[120_000_000];
- random.nextBytes(blobData);
-
- uploadBlob(BLOB_NAME, blobData);
-
- runProcessor();
-
- assertSuccess(BLOB_NAME, blobData);
- }
-
- @Test
- public void testFetchBlobUsingExpressionLanguage() throws Exception {
- uploadBlob(BLOB_NAME, BLOB_DATA);
-
- Map<String, String> attributes = initCommonExpressionLanguageAttributes();
-
- runProcessor(attributes);
-
- assertSuccess(BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testFetchBlobRangeWithStart() throws Exception {
- int start = 3;
- runner.setProperty(FetchAzureBlobStorage_v12.RANGE_START, String.format("%d B", start));
-
- uploadBlob(BLOB_NAME, BLOB_DATA);
-
- runProcessor();
-
- assertSuccess(BLOB_NAME, Arrays.copyOfRange(BLOB_DATA, start, BLOB_DATA.length), BLOB_DATA.length);
- }
-
- @Test
- public void testFetchBlobRangeWithStartAndLength() throws Exception {
- int start = 3;
- int length = 2;
- runner.setProperty(FetchAzureBlobStorage_v12.RANGE_START, String.format("%d B", start));
- runner.setProperty(FetchAzureBlobStorage_v12.RANGE_LENGTH, String.format("%d B", length));
-
- uploadBlob(BLOB_NAME, BLOB_DATA);
-
- runProcessor();
-
- assertSuccess(BLOB_NAME, Arrays.copyOfRange(BLOB_DATA, start, start + length), BLOB_DATA.length);
- }
-
- @Test
- public void testFetchBlobRangeWithStartAndOverSizeLength() throws Exception {
- int start = 3;
- int length = 10_000;
- runner.setProperty(FetchAzureBlobStorage_v12.RANGE_START, String.format("%d B", start));
- runner.setProperty(FetchAzureBlobStorage_v12.RANGE_LENGTH, String.format("%d B", length));
-
- uploadBlob(BLOB_NAME, BLOB_DATA);
-
- runProcessor();
-
- assertSuccess(BLOB_NAME, Arrays.copyOfRange(BLOB_DATA, start, BLOB_DATA.length), BLOB_DATA.length);
- }
-
- private void runProcessor() {
- runProcessor(Collections.emptyMap());
- }
-
- private void runProcessor(Map<String, String> attributes) {
- runner.assertValid();
- runner.enqueue(EMPTY_CONTENT, attributes);
- runner.run();
- }
-
- private void assertSuccess(String blobName, byte[] blobData) throws Exception {
- assertFlowFile(blobName, blobData, blobData.length);
- assertProvenanceEvents();
- }
-
- private void assertSuccess(String blobName, byte[] blobData, int originalLength) throws Exception {
- assertFlowFile(blobName, blobData, originalLength);
- assertProvenanceEvents();
- }
-
- private void assertFlowFile(String blobName, byte[] blobData, Integer originalLength) throws Exception {
- runner.assertAllFlowFilesTransferred(FetchAzureBlobStorage_v12.REL_SUCCESS, 1);
-
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(FetchAzureBlobStorage_v12.REL_SUCCESS).getFirst();
-
- assertFlowFileCommonBlobAttributes(flowFile, getContainerName(), blobName);
- if(originalLength != null) {
- assertFlowFileResultBlobAttributes(flowFile, originalLength);
- }
- flowFile.assertContentEquals(blobData);
- }
-
- private void assertProvenanceEvents() {
- Set<ProvenanceEventType> expectedEventTypes = Collections.singleton(ProvenanceEventType.FETCH);
-
- Set<ProvenanceEventType> actualEventTypes = runner.getProvenanceEvents().stream()
- .map(ProvenanceEventRecord::getEventType)
- .collect(Collectors.toSet());
- assertEquals(expectedEventTypes, actualEventTypes);
- }
-
- private void assertFailure() throws Exception {
- runner.assertAllFlowFilesTransferred(FetchAzureBlobStorage_v12.REL_FAILURE, 1);
-
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(FetchAzureBlobStorage_v12.REL_FAILURE).getFirst();
- flowFile.assertContentEquals(EMPTY_CONTENT);
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITFetchAzureDataLakeStorage.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITFetchAzureDataLakeStorage.java
deleted file mode 100644
index 767ad58d15..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITFetchAzureDataLakeStorage.java
+++ /dev/null
@@ -1,558 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import com.azure.storage.file.datalake.models.DataLakeStorageException;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
-import org.apache.nifi.provenance.ProvenanceEventRecord;
-import org.apache.nifi.provenance.ProvenanceEventType;
-import org.apache.nifi.reporting.InitializationException;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.Test;
-
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.LinkedHashSet;
-import java.util.Map;
-import java.util.Random;
-import java.util.Set;
-import java.util.stream.Collectors;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public class ITFetchAzureDataLakeStorage extends AbstractAzureDataLakeStorageIT {
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return FetchAzureDataLakeStorage.class;
- }
-
- @Test
- public void testFetchFileFromDirectory() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, inputFlowFileContent, TEST_FILE_CONTENT);
- }
-
- @Test
- public void testFetchFileFromDirectoryUsingProxyConfigurationService() throws InitializationException {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- configureProxyService();
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, inputFlowFileContent, TEST_FILE_CONTENT);
- }
-
- @Test
- public void testFetchFileFromRoot() {
- // GIVEN
- String directory= "";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- uploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, inputFlowFileContent, TEST_FILE_CONTENT);
- }
-
- @Test
- public void testFetchFileFromDirectoryWithWhitespace() {
- // GIVEN
- String directory= "A Test Directory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, inputFlowFileContent, TEST_FILE_CONTENT);
- }
-
- @Test
- public void testFetchFileWithWhitespaceFromDirectory() {
- // GIVEN
- String directory= "TestDirectory";
- String filename = "A test file.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, inputFlowFileContent, TEST_FILE_CONTENT);
- }
-
- @Test
- public void testFetchFileCaseSensitiveFilename() {
- // GIVEN
- String directory = "TestDirectory";
- String filename1 = "testFile.txt";
- String filename2 = "testfile.txt";
- String fileContent1 = "ContentOfFile1";
- String fileContent2 = "ContentOfFile2";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename1, fileContent1);
- uploadFile(directory, filename2, fileContent2);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename1, inputFlowFileContent, fileContent1);
- runner.clearProvenanceEvents();
- runner.clearTransferState();
- testSuccessfulFetch(fileSystemName, directory, filename2, inputFlowFileContent, fileContent2);
- }
-
- @Test
- public void testFetchFileCaseSensitiveDirectoryName() {
- // GIVEN
- String directory1 = "TestDirectory";
- String directory2 = "Testdirectory";
- String filename1 = "testFile1.txt";
- String filename2 = "testFile2.txt";
- String fileContent1 = "ContentOfFile1";
- String fileContent2 = "ContentOfFile2";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory1, filename1, fileContent1);
- createDirectoryAndUploadFile(directory2, filename2, fileContent2);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory1, filename1, inputFlowFileContent, fileContent1);
- runner.clearProvenanceEvents();
- runner.clearTransferState();
- testSuccessfulFetch(fileSystemName, directory2, filename2, inputFlowFileContent, fileContent2);
- }
-
- @Test
- public void testFetchFileFromDeepDirectoryStructure() {
- // GIVEN
- String directory= "Directory01/Directory02/Directory03/Directory04/Directory05/Directory06/Directory07/"
- + "Directory08/Directory09/Directory10/Directory11/Directory12/Directory13/Directory14/Directory15/"
- + "Directory16/Directory17/Directory18/Directory19/Directory20/TestDirectory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, inputFlowFileContent, TEST_FILE_CONTENT);
- }
-
- @Test
- public void testFetchDirectory() {
- // GIVEN
- String parentDirectory = "ParentDirectory";
- String childDirectory = "ChildDirectory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(parentDirectory + "/" + childDirectory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testFailedFetchWithProcessException(fileSystemName, parentDirectory, childDirectory, inputFlowFileContent, inputFlowFileContent);
- }
-
- @Test
- public void testFetchNonExistentFileSystem() {
- // GIVEN
- String fileSystem = "NonExistentFileSystem";
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- // WHEN
- // THEN
- testFailedFetch(fileSystem, directory, filename, inputFlowFileContent, inputFlowFileContent, 400);
- }
-
- @Test
- public void testFetchNonExistentDirectory() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- // WHEN
- // THEN
- testFailedFetch(fileSystemName, directory, filename, inputFlowFileContent, inputFlowFileContent, 404);
- }
-
- @Test
- public void testFetchNonExistentFile() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- fileSystemClient.createDirectory(directory);
-
- // WHEN
- // THEN
- testFailedFetch(fileSystemName, directory, filename, inputFlowFileContent, inputFlowFileContent, 404);
- }
-
- @Test
- public void testFetchLargeFile() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- Random random = new Random();
- byte[] fileContentBytes = new byte[120_000_000];
- random.nextBytes(fileContentBytes);
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, inputFlowFileContent, TEST_FILE_CONTENT);
- }
-
- @Test
- public void testFetchInvalidDirectoryName() {
- // GIVEN
- String directory = "TestDirectory";
- String invalidDirectoryName = "Test/\\Directory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testFailedFetch(fileSystemName, invalidDirectoryName, filename, inputFlowFileContent, inputFlowFileContent, 404);
- }
-
- @Test
- public void testFetchInvalidFilename() {
- // GIVEN
- String directory = "TestDirectory";
- String filename = "testFile.txt";
- String invalidFilename = "test/\\File.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testFailedFetch(fileSystemName, directory, invalidFilename, inputFlowFileContent, inputFlowFileContent, 404);
- }
-
- @Test
- public void testFetchUsingExpressionLanguage() {
- // GIVEN
- String expLangFileSystem = "az.filesystem";
- String expLangDirectory = "az.directory";
- String expLangFilename = "az.filename";
-
- String directory = "TestDirectory";
- String filename = "testFile.txt";
-
- String inputFlowFileContent = "InputFlowFileContent";
-
- Map<String, String> attributes = new HashMap<>();
- attributes.put(expLangFileSystem, fileSystemName);
- attributes.put(expLangDirectory, directory);
- attributes.put(expLangFilename, filename);
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch("${" + expLangFileSystem + "}",
- "${" + expLangDirectory + "}",
- "${" + expLangFilename + "}",
- attributes,
- inputFlowFileContent,
- TEST_FILE_CONTENT);
- }
-
- @Test
- public void testFetchUsingExpressionLanguageFileSystemIsNotSpecified() {
- // GIVEN
- String expLangFileSystem = "az.filesystem";
- String expLangDirectory = "az.directory";
- String expLangFilename = "az.filename";
-
- String directory = "TestDirectory";
- String filename = "testFile.txt";
-
- String inputFlowFileContent = "InputFlowFileContent";
-
- Map<String, String> attributes = new HashMap<>();
- attributes.put(expLangDirectory, directory);
- attributes.put(expLangFilename, filename);
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testFailedFetchWithProcessException("${" + expLangFileSystem + "}",
- "${" + expLangDirectory + "}",
- "${" + expLangFilename + "}",
- attributes,
- inputFlowFileContent,
- inputFlowFileContent);
- }
-
- @Test
- public void testFetchUsingExpressionLanguageFilenameIsNotSpecified() {
- // GIVEN
- String expLangFileSystem = "az.filesystem";
- String expLangDirectory = "az.directory";
- String expLangFilename = "az.filename";
-
- String directory = "TestDirectory";
- String filename = "testFile.txt";
-
- String inputFlowFileContent = "InputFlowFileContent";
-
- Map<String, String> attributes = new HashMap<>();
- attributes.put(expLangFileSystem, fileSystemName);
- attributes.put(expLangDirectory, directory);
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testFailedFetchWithProcessException("${" + expLangFileSystem + "}",
- "${" + expLangDirectory + "}",
- "${" + expLangFilename + "}",
- attributes,
- inputFlowFileContent,
- inputFlowFileContent);
- }
-
- @Test
- public void testFetchWithRangeZeroOne() {
- // GIVEN
- String directory= "A Test Directory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, "0B", "1B", inputFlowFileContent, TEST_FILE_CONTENT.substring(0, 1));
- }
-
- @Test
- public void testFetchWithRangeOneOne() {
- // GIVEN
- String directory= "A Test Directory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, "1B", "1B", inputFlowFileContent, TEST_FILE_CONTENT.substring(1, 2));
- }
-
- @Test
- public void testFetchWithRangeTwentyThreeTwentySix() {
- // GIVEN
- String directory= "A Test Directory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, "23B", "3B", inputFlowFileContent, TEST_FILE_CONTENT.substring(23, 26));
- }
-
- @Test
- public void testFetchWithRangeLengthGreater() {
- // GIVEN
- String directory= "A Test Directory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, "0B", "1KB", inputFlowFileContent, TEST_FILE_CONTENT);
- }
-
- @Test
- public void testFetchWithRangeLengthUnset() {
- // GIVEN
- String directory= "A Test Directory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- // WHEN
- // THEN
- testSuccessfulFetch(fileSystemName, directory, filename, "0B", null, inputFlowFileContent, TEST_FILE_CONTENT);
- }
-
- @Test
- public void testFetchWithRangeStartOutOfRange() {
- // GIVEN
- String directory= "A Test Directory";
- String filename = "testFile.txt";
- String inputFlowFileContent = "InputFlowFileContent";
-
- createDirectoryAndUploadFile(directory, filename, TEST_FILE_CONTENT);
-
- setRunnerProperties(fileSystemName, directory, filename, String.format("%sB", TEST_FILE_CONTENT.length() + 1), "1B");
-
- // WHEN
- startRunner(inputFlowFileContent, Collections.emptyMap());
-
- // THEN
- DataLakeStorageException e = (DataLakeStorageException)runner.getLogger().getErrorMessages().getFirst().getThrowable();
- assertEquals(416, e.getStatusCode());
- }
-
- private void testSuccessfulFetch(String fileSystem, String directory, String filename, String inputFlowFileContent, String expectedFlowFileContent) {
- testSuccessfulFetch(fileSystem, directory, filename, Collections.emptyMap(), inputFlowFileContent, expectedFlowFileContent);
- }
-
- private void testSuccessfulFetch(String fileSystem, String directory, String filename, String rangeStart, String rangeLength, String inputFlowFileContent, String expectedFlowFileContent) {
- testSuccessfulFetch(fileSystem, directory, filename, rangeStart, rangeLength, Collections.emptyMap(), inputFlowFileContent, expectedFlowFileContent);
- }
-
- private void testSuccessfulFetch(String fileSystem, String directory, String filename, Map<String, String> attributes, String inputFlowFileContent, String expectedFlowFileContent) {
- testSuccessfulFetch(fileSystem, directory, filename, null, null, attributes, inputFlowFileContent, expectedFlowFileContent);
- }
-
- private void testSuccessfulFetch(String fileSystem, String directory, String filename, String rangeStart, String rangeLength,
- Map<String, String> attributes, String inputFlowFileContent, String expectedFlowFileContent) {
- // GIVEN
- Set<ProvenanceEventType> expectedEventTypes = new LinkedHashSet<>(Arrays.asList(ProvenanceEventType.CONTENT_MODIFIED, ProvenanceEventType.FETCH));
-
- setRunnerProperties(fileSystem, directory, filename, rangeStart, rangeLength);
-
- // WHEN
- startRunner(inputFlowFileContent, attributes);
-
- // THEN
- assertSuccess(expectedFlowFileContent, expectedEventTypes);
- }
-
- private void testFailedFetch(String fileSystem, String directory, String filename, String inputFlowFileContent, String expectedFlowFileContent, int expectedErrorCode) {
- testFailedFetch(fileSystem, directory, filename, Collections.emptyMap(), inputFlowFileContent, expectedFlowFileContent, expectedErrorCode);
- }
-
- private void testFailedFetch(String fileSystem, String directory, String filename, Map<String, String> attributes,
- String inputFlowFileContent, String expectedFlowFileContent, int expectedErrorCode) {
- // GIVEN
- setRunnerProperties(fileSystem, directory, filename);
-
- // WHEN
- startRunner(inputFlowFileContent, attributes);
-
- // THEN
- DataLakeStorageException e = (DataLakeStorageException)runner.getLogger().getErrorMessages().getFirst().getThrowable();
- assertEquals(expectedErrorCode, e.getStatusCode());
-
- assertFailure(expectedFlowFileContent);
- }
-
- private void testFailedFetchWithProcessException(String fileSystem, String directory, String filename, String inputFlowFileContent, String expectedFlowFileContent) {
- testFailedFetchWithProcessException(fileSystem, directory, filename, Collections.emptyMap(), inputFlowFileContent, expectedFlowFileContent);
- }
-
- private void testFailedFetchWithProcessException(String fileSystem, String directory, String filename, Map<String, String> attributes,
- String inputFlowFileContent, String expectedFlowFileContent) {
- // GIVEN
- setRunnerProperties(fileSystem, directory, filename);
-
- // WHEN
- startRunner(inputFlowFileContent, attributes);
-
- // THEN
- Throwable exception = runner.getLogger().getErrorMessages().getFirst().getThrowable();
- assertEquals(ProcessException.class, exception.getClass());
-
- assertFailure(expectedFlowFileContent);
- }
-
- private void setRunnerProperties(String fileSystem, String directory, String filename) {
- setRunnerProperties(fileSystem, directory, filename, null, null);
- }
-
- private void setRunnerProperties(String fileSystem, String directory, String filename, String rangeStart, String rangeLength) {
- runner.setProperty(AzureStorageUtils.FILESYSTEM, fileSystem);
- runner.setProperty(AzureStorageUtils.DIRECTORY, directory);
- runner.setProperty(AzureStorageUtils.FILE, filename);
-
- if (rangeStart != null) {
- runner.setProperty(FetchAzureDataLakeStorage.RANGE_START, rangeStart);
- }
-
- if (rangeLength != null) {
- runner.setProperty(FetchAzureDataLakeStorage.RANGE_LENGTH, rangeLength);
- }
-
- runner.assertValid();
- }
-
- private void startRunner(String inputFlowFileContent, Map<String, String> attributes) {
- runner.enqueue(inputFlowFileContent, attributes);
- runner.run();
- }
-
- private void assertSuccess(String expectedFlowFileContent, Set<ProvenanceEventType> expectedEventTypes) {
- runner.assertAllFlowFilesTransferred(FetchAzureDataLakeStorage.REL_SUCCESS, 1);
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(FetchAzureDataLakeStorage.REL_SUCCESS).getFirst();
- flowFile.assertContentEquals(expectedFlowFileContent);
-
- Set<ProvenanceEventType> actualEventTypes = runner.getProvenanceEvents().stream()
- .map(ProvenanceEventRecord::getEventType)
- .collect(Collectors.toSet());
- assertEquals(expectedEventTypes, actualEventTypes);
- }
-
- private void assertFailure(String expectedFlowFileContent) {
- runner.assertAllFlowFilesTransferred(FetchAzureDataLakeStorage.REL_FAILURE, 1);
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(FetchAzureDataLakeStorage.REL_FAILURE).getFirst();
- flowFile.assertContentEquals(expectedFlowFileContent);
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITListAzureBlobStorage_v12.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITListAzureBlobStorage_v12.java
deleted file mode 100644
index 5f3a738c10..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITListAzureBlobStorage_v12.java
+++ /dev/null
@@ -1,223 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import org.apache.nifi.flowfile.attributes.CoreAttributes;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
-import org.apache.nifi.processors.azure.storage.utils.BlobAttributes;
-import org.apache.nifi.serialization.record.MockRecordWriter;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.Test;
-
-import java.nio.charset.StandardCharsets;
-import java.util.Arrays;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Set;
-
-import static org.junit.jupiter.api.Assertions.assertFalse;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-public class ITListAzureBlobStorage_v12 extends AbstractAzureBlobStorage_v12IT {
-
- protected static final String BLOB_NAME_1 = "blob1";
- protected static final String BLOB_NAME_2 = "dir1/blob2";
- protected static final String BLOB_NAME_3 = "dir1/dir2/blob3";
- protected static final String BLOB_NAME_4 = "dir2/blob4";
-
- protected static final String EL_BLOB_NAME_PREFIX = "az.blobnameprefix";
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return ListAzureBlobStorage_v12.class;
- }
-
- @Test
- public void testListBlobs() throws Exception {
- uploadBlobs();
-
- runProcessor();
-
- assertSuccess(BLOB_NAME_1, BLOB_NAME_2, BLOB_NAME_3, BLOB_NAME_4);
- }
-
- @Test
- public void testListBlobsUsingProxyConfigurationService() throws Exception {
- uploadBlobs();
-
- configureProxyService();
-
- runProcessor();
-
- assertSuccess(BLOB_NAME_1, BLOB_NAME_2, BLOB_NAME_3, BLOB_NAME_4);
- }
-
- @Test
- public void testListBlobsWithPrefix_1() throws Exception {
- uploadBlobs();
- runner.setProperty(ListAzureBlobStorage_v12.BLOB_NAME_PREFIX, "blob");
-
- runProcessor();
-
- assertSuccess(BLOB_NAME_1);
- }
-
- @Test
- public void testListBlobsWithPrefix_2() throws Exception {
- uploadBlobs();
- runner.setProperty(ListAzureBlobStorage_v12.BLOB_NAME_PREFIX, "dir1/");
-
- runProcessor();
-
- assertSuccess(BLOB_NAME_2, BLOB_NAME_3);
- }
-
- @Test
- public void testListBlobsUsingExpressionLanguage() throws Exception {
- uploadBlobs();
-
- runner.setProperty(ListAzureBlobStorage_v12.CONTAINER, String.format("${%s}", EL_CONTAINER_NAME));
- runner.setProperty(ListAzureBlobStorage_v12.BLOB_NAME_PREFIX, String.format("${%s}", EL_BLOB_NAME_PREFIX));
-
- runner.setEnvironmentVariableValue(EL_CONTAINER_NAME, getContainerName());
- runner.setEnvironmentVariableValue(EL_BLOB_NAME_PREFIX, "dir1/dir");
-
- runProcessor();
-
- assertSuccess(BLOB_NAME_3);
- }
-
- @Test
- public void testListEmptyContainer() throws Exception {
- runProcessor();
-
- assertSuccess();
- }
-
- @Test
- public void testListWithNonExistingContainer() {
- runner.setProperty(AzureStorageUtils.CONTAINER, "dummy");
-
- runProcessor();
-
- assertFailure();
- }
-
- @Test
- public void testListWithRecords() throws Exception {
- uploadBlobs();
-
- MockRecordWriter recordWriter = new MockRecordWriter(null, false);
- runner.addControllerService("record-writer", recordWriter);
- runner.enableControllerService(recordWriter);
- runner.setProperty(ListAzureBlobStorage_v12.RECORD_WRITER, "record-writer");
-
- runner.run();
-
- runner.assertAllFlowFilesTransferred(ListAzureBlobStorage_v12.REL_SUCCESS, 1);
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(ListAzureBlobStorage_v12.REL_SUCCESS).getFirst();
- flowFile.assertAttributeEquals("record.count", "4");
- }
-
- @Test
- public void testListWithMinAge() throws Exception {
- uploadBlobs();
- runner.setProperty(ListAzureBlobStorage_v12.MIN_AGE, "1 hour");
-
- runProcessor();
-
- runner.assertTransferCount(ListAzureBlobStorage_v12.REL_SUCCESS, 0);
- }
-
- @Test
- public void testListWithMaxAge() throws Exception {
- uploadBlobs();
- runner.setProperty(ListAzureBlobStorage_v12.MAX_AGE, "1 hour");
-
- runProcessor();
-
- assertSuccess(BLOB_NAME_1, BLOB_NAME_2, BLOB_NAME_3, BLOB_NAME_4);
- }
-
- @Test
- public void testListWithMinSize() throws Exception {
- uploadBlobs();
- runner.setProperty(ListAzureBlobStorage_v12.MIN_SIZE, "5 B");
- uploadBlob("blob5", "Test".getBytes(StandardCharsets.UTF_8));
-
- runProcessor();
-
- assertSuccess(BLOB_NAME_1, BLOB_NAME_2, BLOB_NAME_3, BLOB_NAME_4);
- }
-
- @Test
- public void testListWithMaxSize() throws Exception {
- uploadBlobs();
- runner.setProperty(ListAzureBlobStorage_v12.MAX_SIZE, "5 B");
- uploadBlob("blob5", "Test".getBytes(StandardCharsets.UTF_8));
-
- runProcessor();
-
- runner.assertAllFlowFilesTransferred(ListAzureBlobStorage_v12.REL_SUCCESS, 1);
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(ListAzureBlobStorage_v12.REL_SUCCESS).getFirst();
- assertFlowFileCommonBlobAttributes(flowFile, getContainerName(), "blob5");
- assertFlowFileResultBlobAttributes(flowFile, "Test".length());
- }
-
- private void uploadBlobs() throws Exception {
- uploadBlob(BLOB_NAME_1, BLOB_DATA);
- uploadBlob(BLOB_NAME_2, BLOB_DATA);
- uploadBlob(BLOB_NAME_3, BLOB_DATA);
- uploadBlob(BLOB_NAME_4, BLOB_DATA);
- }
-
- private void runProcessor() {
- runner.assertValid();
- runner.run();
- }
-
- private void assertSuccess(String... blobNames) throws Exception {
- runner.assertTransferCount(ListAzureBlobStorage_v12.REL_SUCCESS, blobNames.length);
-
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(ListAzureBlobStorage_v12.REL_SUCCESS);
-
- Set<String> expectedBlobNames = new HashSet<>(Arrays.asList(blobNames));
-
- for (MockFlowFile flowFile : flowFiles) {
- String blobName = flowFile.getAttribute(BlobAttributes.ATTR_NAME_BLOBNAME);
- assertTrue(expectedBlobNames.remove(blobName), "Blob should not be listed: " + blobName);
-
- assertFlowFile(flowFile, blobName);
- }
-
- assertTrue(expectedBlobNames.isEmpty(), "Blobs should be listed: " + expectedBlobNames);
- }
-
- private void assertFlowFile(MockFlowFile flowFile, String blobName) throws Exception {
- assertFlowFileCommonBlobAttributes(flowFile, getContainerName(), blobName);
- assertFlowFileResultBlobAttributes(flowFile, BLOB_DATA.length);
- flowFile.assertAttributeEquals(CoreAttributes.FILENAME.key(), blobName.substring(blobName.lastIndexOf('/') + 1));
-
- flowFile.assertContentEquals(EMPTY_CONTENT);
- }
-
- private void assertFailure() {
- assertFalse(runner.getLogger().getErrorMessages().isEmpty());
- runner.assertTransferCount(ListAzureBlobStorage_v12.REL_SUCCESS, 0);
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITListAzureDataLakeStorage.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITListAzureDataLakeStorage.java
deleted file mode 100644
index abfd5f5feb..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITListAzureDataLakeStorage.java
+++ /dev/null
@@ -1,534 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processors.azure.AbstractAzureDataLakeStorageProcessor;
-import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
-import org.apache.nifi.reporting.InitializationException;
-import org.apache.nifi.serialization.record.MockRecordWriter;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.nio.charset.StandardCharsets;
-import java.util.Arrays;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_DIRECTORY;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_ETAG;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_FILENAME;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_FILESYSTEM;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_FILE_PATH;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_LAST_MODIFIED;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_LENGTH;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertFalse;
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-public class ITListAzureDataLakeStorage extends AbstractAzureDataLakeStorageIT {
-
- private Map<String, TestFile> testFiles;
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return ListAzureDataLakeStorage.class;
- }
-
- @BeforeEach
- public void setUp() {
- testFiles = new HashMap<>();
-
- TestFile testFile1 = new TestFile("", "file1");
- uploadFile(testFile1);
- testFiles.put(testFile1.getFilePath(), testFile1);
-
- TestFile testTempFile1 = new TestFile(AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY, "1234file1");
- uploadFile(testTempFile1);
- testFiles.put(testTempFile1.getFilePath(), testTempFile1);
-
- TestFile testFile2 = new TestFile("", "file2");
- uploadFile(testFile2);
- testFiles.put(testFile2.getFilePath(), testFile2);
-
- TestFile testFile11 = new TestFile("dir1", "file11");
- createDirectoryAndUploadFile(testFile11);
- testFiles.put(testFile11.getFilePath(), testFile11);
-
- TestFile testTempFile11 = new TestFile(String.format("dir1/%s", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY), "5678file11");
- uploadFile(testTempFile11);
- testFiles.put(testTempFile11.getFilePath(), testTempFile11);
-
- TestFile testFile12 = new TestFile("dir1", "file12");
- uploadFile(testFile12);
- testFiles.put(testFile12.getFilePath(), testFile12);
-
- TestFile testFile111 = new TestFile("dir1/dir11", "file111");
- createDirectoryAndUploadFile(testFile111);
- testFiles.put(testFile111.getFilePath(), testFile111);
-
- TestFile testTempFile111 = new TestFile(String.format("dir1/dir11/%s", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY), "9010file111");
- uploadFile(testTempFile111);
- testFiles.put(testTempFile111.getFilePath(), testTempFile111);
-
- TestFile testFile21 = new TestFile("dir 2", "file 21", "Test");
- createDirectoryAndUploadFile(testFile21);
- testFiles.put(testFile21.getFilePath(), testFile21);
-
- TestFile testTempFile21 = new TestFile(String.format("dir2/%s", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY), "1112file21", "Test");
- uploadFile(testTempFile21);
- testFiles.put(testTempFile21.getFilePath(), testTempFile21);
-
- createDirectory("dir3");
- }
-
- @Test
- public void testListRootRecursive() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
-
- runProcessor();
-
- assertSuccess("file1", "file2", "dir1/file11", "dir1/file12", "dir1/dir11/file111", "dir 2/file 21");
- }
-
- @Test
- public void testListRootRecursiveWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("file1", "file2", "dir1/file11", "dir1/file12", "dir1/dir11/file111", "dir 2/file 21",
- String.format("%s/1234file1", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/%s/5678file11", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/dir11/%s/9010file111", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir2/%s/1112file21", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- @Test
- public void testListRootRecursiveUsingProxyConfigurationService() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- configureProxyService();
-
- runProcessor();
-
- assertSuccess("file1", "file2", "dir1/file11", "dir1/file12", "dir1/dir11/file111", "dir 2/file 21");
- }
-
- @Test
- public void testListRootNonRecursive() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.RECURSE_SUBDIRECTORIES, "false");
-
- runProcessor();
-
- assertSuccess("file1", "file2");
- }
-
- @Test
- public void testListRootNonRecursiveWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.RECURSE_SUBDIRECTORIES, "false");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("file1", "file2");
- }
-
- @Test
- public void testListSubdirectoryRecursive() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "dir1");
-
- runProcessor();
-
- assertSuccess("dir1/file11", "dir1/file12", "dir1/dir11/file111");
- }
-
- @Test
- public void testListSubdirectoryRecursiveWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "dir1");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("dir1/file11", "dir1/file12", "dir1/dir11/file111",
- String.format("dir1/%s/5678file11", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/dir11/%s/9010file111", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- @Test
- public void testListSubdirectoryNonRecursive() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "dir1");
- runner.setProperty(ListAzureDataLakeStorage.RECURSE_SUBDIRECTORIES, "false");
-
- runProcessor();
-
- assertSuccess("dir1/file11", "dir1/file12");
- }
-
- @Test
- public void testListSubdirectoryNonRecursiveWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "dir1");
- runner.setProperty(ListAzureDataLakeStorage.RECURSE_SUBDIRECTORIES, "false");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("dir1/file11", "dir1/file12");
- }
-
- @Test
- public void testListWithFileFilter() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.FILE_FILTER, ".*file1.*$");
-
- runProcessor();
-
- assertSuccess("file1", "dir1/file11", "dir1/file12", "dir1/dir11/file111");
- }
-
- @Test
- public void testListWithFileFilterWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.FILE_FILTER, ".*file1.*$");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("file1", "dir1/file11", "dir1/file12", "dir1/dir11/file111",
- String.format("%s/1234file1", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/%s/5678file11", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/dir11/%s/9010file111", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- @Test
- public void testListWithFileFilterWithEL() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.FILE_FILTER, ".*file${suffix}$");
- runner.setEnvironmentVariableValue("suffix", "1.*");
-
- runProcessor();
-
- assertSuccess("file1", "dir1/file11", "dir1/file12", "dir1/dir11/file111");
- }
-
- @Test
- public void testListWithFileFilterWithELWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.FILE_FILTER, ".*file${suffix}$");
- runner.setEnvironmentVariableValue("suffix", "1.*");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("file1", "dir1/file11", "dir1/file12", "dir1/dir11/file111",
- String.format("%s/1234file1", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/%s/5678file11", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/dir11/%s/9010file111", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- @Test
- public void testListRootWithPathFilter() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.PATH_FILTER, "^dir1.*$");
-
- runProcessor();
-
- assertSuccess("dir1/file11", "dir1/file12", "dir1/dir11/file111");
- }
-
- @Test
- public void testListRootWithPathFilterWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.PATH_FILTER, "^dir1.*$");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("dir1/file11", "dir1/file12", "dir1/dir11/file111",
- String.format("dir1/%s/5678file11", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/dir11/%s/9010file111", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- @Test
- public void testListRootWithPathFilterWithEL() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.PATH_FILTER, "${prefix}${suffix}");
- runner.setEnvironmentVariableValue("prefix", "^dir");
- runner.setEnvironmentVariableValue("suffix", "1.*$");
-
- runProcessor();
-
- assertSuccess("dir1/file11", "dir1/file12", "dir1/dir11/file111");
- }
-
- @Test
- public void testListRootWithPathFilterWithELWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.PATH_FILTER, "${prefix}${suffix}");
- runner.setEnvironmentVariableValue("prefix", "^dir");
- runner.setEnvironmentVariableValue("suffix", "1.*$");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("dir1/file11", "dir1/file12", "dir1/dir11/file111",
- String.format("dir1/%s/5678file11", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/dir11/%s/9010file111", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- @Test
- public void testListSubdirectoryWithPathFilter() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "dir1");
- runner.setProperty(ListAzureDataLakeStorage.PATH_FILTER, "dir1.*");
-
- runProcessor();
-
- assertSuccess("dir1/dir11/file111");
- }
-
- @Test
- public void testListSubdirectoryWithPathFilterWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "dir1");
- runner.setProperty(ListAzureDataLakeStorage.PATH_FILTER, "dir1.*");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("dir1/dir11/file111", String.format("dir1/dir11/%s/9010file111", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- @Test
- public void testListRootWithFileAndPathFilter() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.FILE_FILTER, ".*11");
- runner.setProperty(ListAzureDataLakeStorage.PATH_FILTER, "dir1.*");
-
- runProcessor();
-
- assertSuccess("dir1/file11", "dir1/dir11/file111");
- }
-
- @Test
- public void testListRootWithFileAndPathFilterWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.FILE_FILTER, ".*11");
- runner.setProperty(ListAzureDataLakeStorage.PATH_FILTER, "dir1.*");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("dir1/file11", "dir1/dir11/file111", String.format("dir1/%s/5678file11", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/dir11/%s/9010file111", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- @Test
- public void testListEmptyDirectory() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "dir3");
-
- runProcessor();
-
- assertSuccess();
- }
-
- @Test
- public void testListNonExistingDirectory() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "dummy");
-
- runProcessor();
-
- assertFailure();
- }
-
- @Test
- public void testListWithNonExistingFileSystem() {
- runner.setProperty(AzureStorageUtils.FILESYSTEM, "dummy");
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
-
- runProcessor();
-
- assertFailure();
- }
-
- @Test
- public void testListWithRecords() throws InitializationException {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "dir1");
-
- MockRecordWriter recordWriter = new MockRecordWriter(null, false);
- runner.addControllerService("record-writer", recordWriter);
- runner.enableControllerService(recordWriter);
- runner.setProperty(ListAzureDataLakeStorage.RECORD_WRITER, "record-writer");
-
- runner.run();
-
- runner.assertAllFlowFilesTransferred(ListAzureDataLakeStorage.REL_SUCCESS, 1);
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(ListAzureDataLakeStorage.REL_SUCCESS).getFirst();
- flowFile.assertAttributeEquals("record.count", "3");
- }
-
- @Test
- public void testListWithRecordsWithTempFiles() throws InitializationException {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "dir1");
-
- MockRecordWriter recordWriter = new MockRecordWriter(null, false);
- runner.addControllerService("record-writer", recordWriter);
- runner.enableControllerService(recordWriter);
- runner.setProperty(ListAzureDataLakeStorage.RECORD_WRITER, "record-writer");
-
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runner.run();
-
- runner.assertAllFlowFilesTransferred(ListAzureDataLakeStorage.REL_SUCCESS, 1);
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(ListAzureDataLakeStorage.REL_SUCCESS).getFirst();
- flowFile.assertAttributeEquals("record.count", "5");
- }
-
- @Test
- public void testListWithMinAge() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.MIN_AGE, "1 hour");
-
- runProcessor();
-
- runner.assertTransferCount(ListAzureDataLakeStorage.REL_SUCCESS, 0);
- }
-
- @Test
- public void testListWithMinAgeWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.MIN_AGE, "1 hour");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- runner.assertTransferCount(ListAzureDataLakeStorage.REL_SUCCESS, 0);
- }
-
- @Test
- public void testListWithMaxAge() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.MAX_AGE, "1 hour");
-
- runProcessor();
-
- assertSuccess("file1", "file2", "dir1/file11", "dir1/file12", "dir1/dir11/file111", "dir 2/file 21");
- }
-
- @Test
- public void testListWithMaxAgeWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.MAX_AGE, "1 hour");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("file1", "file2", "dir1/file11", "dir1/file12", "dir1/dir11/file111", "dir 2/file 21",
- String.format("%s/1234file1", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/%s/5678file11", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/dir11/%s/9010file111", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir2/%s/1112file21", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- @Test
- public void testListWithMinSize() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.MIN_SIZE, "5 B");
-
- runProcessor();
-
- assertSuccess("file1", "file2", "dir1/file11", "dir1/file12", "dir1/dir11/file111");
- }
-
- @Test
- public void testListWithMinSizeWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.MIN_SIZE, "5 B");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("file1", "file2", "dir1/file11", "dir1/file12", "dir1/dir11/file111",
- String.format("%s/1234file1", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/%s/5678file11", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY),
- String.format("dir1/dir11/%s/9010file111", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- @Test
- public void testListWithMaxSize() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.MAX_SIZE, "5 B");
-
- runProcessor();
-
- assertSuccess("dir 2/file 21");
- }
-
- @Test
- public void testListWithMaxSizeWithTempFiles() throws Exception {
- runner.setProperty(AzureStorageUtils.DIRECTORY, "");
- runner.setProperty(ListAzureDataLakeStorage.MAX_SIZE, "5 B");
- runner.setProperty(ListAzureDataLakeStorage.INCLUDE_TEMPORARY_FILES, "true");
-
- runProcessor();
-
- assertSuccess("dir 2/file 21", String.format("dir2/%s/1112file21", AbstractAzureDataLakeStorageProcessor.TEMP_FILE_DIRECTORY));
- }
-
- private void runProcessor() {
- runner.assertValid();
- runner.run();
- }
-
- private void assertSuccess(String... testFilePaths) {
- runner.assertTransferCount(ListAzureDataLakeStorage.REL_SUCCESS, testFilePaths.length);
-
- Map<String, TestFile> expectedFiles = new HashMap<>(testFiles);
- expectedFiles.keySet().retainAll(Arrays.asList(testFilePaths));
-
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(ListAzureDataLakeStorage.REL_SUCCESS);
- assertEquals(expectedFiles.size(), flowFiles.size());
-
- for (MockFlowFile flowFile : flowFiles) {
- String filePath = flowFile.getAttribute("azure.filePath");
- TestFile testFile = expectedFiles.remove(filePath);
- assertNotNull(testFile, "File path not found in the expected map");
- assertFlowFile(testFile, flowFile);
- }
- }
-
- private void assertFlowFile(TestFile testFile, MockFlowFile flowFile) {
- flowFile.assertAttributeEquals(ATTR_NAME_FILESYSTEM, fileSystemName);
- flowFile.assertAttributeEquals(ATTR_NAME_FILE_PATH, testFile.getFilePath());
- flowFile.assertAttributeEquals(ATTR_NAME_DIRECTORY, testFile.getDirectory());
- flowFile.assertAttributeEquals(ATTR_NAME_FILENAME, testFile.getFilename());
- flowFile.assertAttributeEquals(ATTR_NAME_LENGTH, String.valueOf(testFile.getFileContent().getBytes(StandardCharsets.UTF_8).length));
-
- flowFile.assertAttributeExists(ATTR_NAME_LAST_MODIFIED);
- flowFile.assertAttributeExists(ATTR_NAME_ETAG);
-
- flowFile.assertContentEquals("");
- }
-
- private void assertFailure() {
- assertFalse(runner.getLogger().getErrorMessages().isEmpty());
- runner.assertTransferCount(ListAzureDataLakeStorage.REL_SUCCESS, 0);
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITMoveAzureDataLakeStorage.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITMoveAzureDataLakeStorage.java
deleted file mode 100644
index 751ff36f4c..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITMoveAzureDataLakeStorage.java
+++ /dev/null
@@ -1,374 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import com.azure.storage.file.datalake.DataLakeDirectoryClient;
-import com.azure.storage.file.datalake.DataLakeFileClient;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
-import org.apache.nifi.provenance.ProvenanceEventRecord;
-import org.apache.nifi.provenance.ProvenanceEventType;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.nio.charset.StandardCharsets;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Random;
-import java.util.Set;
-import java.util.stream.Collectors;
-
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_DIRECTORY;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_FILENAME;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_FILESYSTEM;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_LENGTH;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_PRIMARY_URI;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_SOURCE_DIRECTORY;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_SOURCE_FILESYSTEM;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-public class ITMoveAzureDataLakeStorage extends AbstractAzureDataLakeStorageIT {
-
- private static final String SOURCE_DIRECTORY = "sourceDir1";
- private static final String DESTINATION_DIRECTORY = "destDir1";
- private static final String FILE_NAME = "file1";
- private static final byte[] FILE_DATA = "0123456789".getBytes(StandardCharsets.UTF_8);
-
- private static final String EL_FILESYSTEM = "az.filesystem";
- private static final String EL_DIRECTORY = "az.directory";
- private static final String EL_FILE_NAME = "az.filename";
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return MoveAzureDataLakeStorage.class;
- }
-
- @BeforeEach
- public void setUp() throws InterruptedException {
- runner.setProperty(MoveAzureDataLakeStorage.SOURCE_FILESYSTEM, fileSystemName);
- runner.setProperty(MoveAzureDataLakeStorage.SOURCE_DIRECTORY, SOURCE_DIRECTORY);
- runner.setProperty(MoveAzureDataLakeStorage.DESTINATION_FILESYSTEM, fileSystemName);
- runner.setProperty(MoveAzureDataLakeStorage.DESTINATION_DIRECTORY, DESTINATION_DIRECTORY);
- runner.setProperty(AzureStorageUtils.FILE, FILE_NAME);
- }
-
- @Test
- public void testMoveFileToExistingDirectory() throws Exception {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
- createDirectory(DESTINATION_DIRECTORY);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(SOURCE_DIRECTORY, DESTINATION_DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @Test
- public void testMoveFileToExistingDirectoryUsingProxyConfigurationService() throws Exception {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
- createDirectory(DESTINATION_DIRECTORY);
- configureProxyService();
-
- runProcessor(FILE_DATA);
-
- assertSuccess(SOURCE_DIRECTORY, DESTINATION_DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @Test
- public void testMoveFileToExistingDirectoryWithReplaceResolution() throws Exception {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
- createDirectory(DESTINATION_DIRECTORY);
-
- runner.setProperty(MoveAzureDataLakeStorage.CONFLICT_RESOLUTION, MoveAzureDataLakeStorage.REPLACE_RESOLUTION);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(SOURCE_DIRECTORY, DESTINATION_DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @Test
- public void testMoveFileToExistingDirectoryWithIgnoreResolution() throws Exception {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
- createDirectory(DESTINATION_DIRECTORY);
-
- runner.setProperty(MoveAzureDataLakeStorage.CONFLICT_RESOLUTION, MoveAzureDataLakeStorage.IGNORE_RESOLUTION);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(SOURCE_DIRECTORY, DESTINATION_DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @Test
- public void testMoveFileToNonExistingDirectory() throws Exception {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(SOURCE_DIRECTORY, DESTINATION_DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @Test
- public void testMoveFileToDeepDirectory() throws Exception {
- String sourceDirectory = "dir1/dir2";
- String destinationDirectory = sourceDirectory + "/dir3/dir4";
- createDirectoryAndUploadFile(sourceDirectory, FILE_NAME, FILE_DATA);
-
- runner.setProperty(MoveAzureDataLakeStorage.SOURCE_DIRECTORY, sourceDirectory);
- runner.setProperty(MoveAzureDataLakeStorage.DESTINATION_DIRECTORY, destinationDirectory);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(sourceDirectory, destinationDirectory, FILE_NAME, FILE_DATA);
- }
-
- @Test
- public void testMoveFileToRootDirectory() throws Exception {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
-
- String rootDirectory = "";
- runner.setProperty(MoveAzureDataLakeStorage.DESTINATION_DIRECTORY, rootDirectory);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(SOURCE_DIRECTORY, rootDirectory, FILE_NAME, FILE_DATA);
- }
-
- @Test
- public void testMoveEmptyFile() throws Exception {
- byte[] fileData = new byte[0];
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, fileData);
-
- runProcessor(fileData);
-
- assertSuccess(SOURCE_DIRECTORY, DESTINATION_DIRECTORY, FILE_NAME, fileData);
- }
-
- @Test
- public void testMoveBigFile() throws Exception {
- Random random = new Random();
- byte[] fileData = new byte[120_000_000];
- random.nextBytes(fileData);
-
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, fileData);
-
- runProcessor(fileData);
-
- assertSuccess(SOURCE_DIRECTORY, DESTINATION_DIRECTORY, FILE_NAME, fileData);
- }
-
- @Test
- public void testMoveFileWithNonExistingFileSystem() {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
-
- runner.setProperty(MoveAzureDataLakeStorage.DESTINATION_FILESYSTEM, "dummy");
-
- runProcessor(FILE_DATA);
-
- assertFailure();
- }
-
- @Test
- public void testMoveFileWithInvalidFileName() {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
-
- runner.setProperty(AzureStorageUtils.FILE, "/file1");
-
- runProcessor(FILE_DATA);
-
- assertFailure();
- }
-
- @Test
- public void testMoveFileWithSpacesInDirectoryAndFileName() throws Exception {
- String sourceDirectory = "dir 1";
- String destinationDirectory = "dest dir1";
- String fileName = "file 1";
- createDirectoryAndUploadFile(sourceDirectory, fileName, FILE_DATA);
-
- runner.setProperty(MoveAzureDataLakeStorage.SOURCE_DIRECTORY, sourceDirectory);
- runner.setProperty(MoveAzureDataLakeStorage.DESTINATION_DIRECTORY, destinationDirectory);
- runner.setProperty(AzureStorageUtils.FILE, fileName);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(sourceDirectory, destinationDirectory, fileName, FILE_DATA);
- }
-
- @Test
- public void testMoveFileToExistingFileWithFailResolution() {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
- fileSystemClient.createFile(String.format("%s/%s", DESTINATION_DIRECTORY, FILE_NAME));
-
- runner.setProperty(MoveAzureDataLakeStorage.CONFLICT_RESOLUTION, MoveAzureDataLakeStorage.FAIL_RESOLUTION);
-
- runProcessor(FILE_DATA);
-
- assertFailure();
- }
-
- @Test
- public void testMoveFileToExistingFileWithReplaceResolution() throws Exception {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
- fileSystemClient.createFile(String.format("%s/%s", DESTINATION_DIRECTORY, FILE_NAME));
-
- runner.setProperty(MoveAzureDataLakeStorage.CONFLICT_RESOLUTION, MoveAzureDataLakeStorage.REPLACE_RESOLUTION);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(SOURCE_DIRECTORY, DESTINATION_DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @Test
- public void testMoveFileToExistingFileWithIgnoreResolution() throws Exception {
- String fileContent = "destination";
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
- createDirectoryAndUploadFile(DESTINATION_DIRECTORY, FILE_NAME, fileContent);
-
- runner.setProperty(MoveAzureDataLakeStorage.CONFLICT_RESOLUTION, MoveAzureDataLakeStorage.IGNORE_RESOLUTION);
-
- runProcessor(FILE_DATA);
-
- assertSuccessWithIgnoreResolution(DESTINATION_DIRECTORY, FILE_NAME, FILE_DATA, fileContent.getBytes(StandardCharsets.UTF_8));
- }
-
- @Test
- public void testMoveFileWithEL() throws Exception {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
- Map<String, String> attributes = createAttributesMap(FILE_DATA);
- setELProperties();
-
- runProcessor(FILE_DATA, attributes);
-
- assertSuccess(SOURCE_DIRECTORY, DESTINATION_DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @Test
- public void testMoveFileWithELButFilesystemIsNotSpecified() {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
- Map<String, String> attributes = createAttributesMap(FILE_DATA);
- attributes.remove(EL_FILESYSTEM);
- setELProperties();
-
- runProcessor(FILE_DATA, attributes);
-
- assertFailure();
- }
-
- @Test
- public void testMoveFileWithELButFileNameIsNotSpecified() {
- createDirectoryAndUploadFile(SOURCE_DIRECTORY, FILE_NAME, FILE_DATA);
- Map<String, String> attributes = createAttributesMap(FILE_DATA);
- attributes.remove(EL_FILE_NAME);
- setELProperties();
-
- runProcessor(FILE_DATA, attributes);
-
- assertFailure();
- }
-
- private Map<String, String> createAttributesMap(byte[] fileData) {
- Map<String, String> attributes = new HashMap<>();
-
- attributes.put(EL_FILESYSTEM, fileSystemName);
- attributes.put(EL_DIRECTORY, SOURCE_DIRECTORY);
- attributes.put(EL_FILE_NAME, FILE_NAME);
- attributes.put(ATTR_NAME_LENGTH, String.valueOf(fileData.length));
-
- return attributes;
- }
-
- private void setELProperties() {
- runner.setProperty(MoveAzureDataLakeStorage.SOURCE_FILESYSTEM, String.format("${%s}", EL_FILESYSTEM));
- runner.setProperty(MoveAzureDataLakeStorage.SOURCE_DIRECTORY, String.format("${%s}", EL_DIRECTORY));
- runner.setProperty(AzureStorageUtils.FILE, String.format("${%s}", EL_FILE_NAME));
- }
-
- private void runProcessor(byte[] fileData) {
- runProcessor(fileData, Collections.singletonMap(ATTR_NAME_LENGTH, String.valueOf(fileData.length)));
- }
-
- private void runProcessor(byte[] testData, Map<String, String> attributes) {
- runner.assertValid();
- runner.enqueue(testData, attributes);
- runner.run();
- }
-
- private void assertSuccess(String sourceDirectory, String destinationDirectory, String fileName, byte[] fileData) throws Exception {
- assertFlowFile(fileData, fileName, sourceDirectory, destinationDirectory);
- assertAzureFile(destinationDirectory, fileName, fileData);
- assertProvenanceEvents();
- }
-
- private void assertSuccessWithIgnoreResolution(String destinationDirectory, String fileName, byte[] fileData, byte[] azureFileData) throws Exception {
- assertFlowFile(fileData);
- assertAzureFile(destinationDirectory, fileName, azureFileData);
- }
-
- private void assertFlowFile(byte[] fileData, String fileName, String sourceDirectory, String destinationDirectory) throws Exception {
- MockFlowFile flowFile = assertFlowFile(fileData);
-
- flowFile.assertAttributeEquals(ATTR_NAME_SOURCE_FILESYSTEM, fileSystemName);
- flowFile.assertAttributeEquals(ATTR_NAME_SOURCE_DIRECTORY, sourceDirectory);
- flowFile.assertAttributeEquals(ATTR_NAME_FILESYSTEM, fileSystemName);
- flowFile.assertAttributeEquals(ATTR_NAME_DIRECTORY, destinationDirectory);
- flowFile.assertAttributeEquals(ATTR_NAME_FILENAME, fileName);
-
- flowFile.assertAttributeExists(ATTR_NAME_PRIMARY_URI);
- flowFile.assertAttributeEquals(ATTR_NAME_LENGTH, Integer.toString(fileData.length));
- }
-
- private MockFlowFile assertFlowFile(byte[] fileData) throws Exception {
- runner.assertAllFlowFilesTransferred(MoveAzureDataLakeStorage.REL_SUCCESS, 1);
-
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(MoveAzureDataLakeStorage.REL_SUCCESS).getFirst();
-
- flowFile.assertContentEquals(fileData);
-
- return flowFile;
- }
-
- private void assertAzureFile(String destinationDirectory, String fileName, byte[] fileData) {
- DataLakeFileClient fileClient;
- if (StringUtils.isNotEmpty(destinationDirectory)) {
- DataLakeDirectoryClient directoryClient = fileSystemClient.getDirectoryClient(destinationDirectory);
- assertTrue(directoryClient.exists());
-
- fileClient = directoryClient.getFileClient(fileName);
- } else {
- fileClient = fileSystemClient.getFileClient(fileName);
- }
-
- assertTrue(fileClient.exists());
- assertEquals(fileData.length, fileClient.getProperties().getFileSize());
- }
-
- private void assertProvenanceEvents() {
- Set<ProvenanceEventType> expectedEventTypes = Collections.singleton(ProvenanceEventType.SEND);
-
- Set<ProvenanceEventType> actualEventTypes = runner.getProvenanceEvents().stream()
- .map(ProvenanceEventRecord::getEventType)
- .collect(Collectors.toSet());
- assertEquals(expectedEventTypes, actualEventTypes);
- }
-
- private void assertFailure() {
- runner.assertAllFlowFilesTransferred(MoveAzureDataLakeStorage.REL_FAILURE, 1);
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITPutAzureBlobStorage_v12.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITPutAzureBlobStorage_v12.java
deleted file mode 100644
index e1ece40360..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITPutAzureBlobStorage_v12.java
+++ /dev/null
@@ -1,385 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import com.azure.storage.blob.BlobClient;
-import com.azure.storage.blob.BlobContainerClient;
-import com.azure.storage.blob.models.BlobErrorCode;
-import org.apache.nifi.fileresource.service.StandardFileResourceService;
-import org.apache.nifi.fileresource.service.api.FileResourceService;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processors.azure.ClientSideEncryptionSupport;
-import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
-import org.apache.nifi.processors.azure.storage.utils.ClientSideEncryptionMethod;
-import org.apache.nifi.processors.transfer.ResourceTransferProperties;
-import org.apache.nifi.processors.transfer.ResourceTransferSource;
-import org.apache.nifi.provenance.ProvenanceEventRecord;
-import org.apache.nifi.provenance.ProvenanceEventType;
-import org.apache.nifi.services.azure.storage.AzureStorageConflictResolutionStrategy;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Random;
-import java.util.Set;
-import java.util.stream.Collectors;
-
-import static org.apache.nifi.processors.azure.storage.utils.BlobAttributes.ATTR_NAME_ERROR_CODE;
-import static org.apache.nifi.processors.azure.storage.utils.BlobAttributes.ATTR_NAME_IGNORED;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertNull;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-public class ITPutAzureBlobStorage_v12 extends AbstractAzureBlobStorage_v12IT {
-
- public static class ITProcessor extends PutAzureBlobStorage_v12 {
- public boolean blobMetadataApplied = false;
-
- @Override
- protected void applyBlobMetadata(Map<String, String> attributes, BlobClient blobClient) {
- super.applyBlobMetadata(attributes, blobClient);
- blobMetadataApplied = true;
- }
- }
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return ITProcessor.class;
- }
-
- @BeforeEach
- public void setUp() {
- runner.setProperty(PutAzureBlobStorage_v12.BLOB_NAME, BLOB_NAME);
- }
-
- @Test
- public void testPutBlobWithSimpleName() throws Exception {
- runProcessor(BLOB_DATA);
-
- assertSuccess(getContainerName(), BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testPutBlobApplyBlobMetadata() throws Exception {
- runProcessor(BLOB_DATA);
-
- assertSuccess(getContainerName(), BLOB_NAME, BLOB_DATA);
- assertTrue(((ITProcessor) runner.getProcessor()).blobMetadataApplied);
- }
-
- @Test
- public void testPutBlobWithSimpleNameUsingProxyConfigurationService() throws Exception {
- configureProxyService();
-
- runProcessor(BLOB_DATA);
-
- assertSuccess(getContainerName(), BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testPutBlobWithCompoundName() throws Exception {
- String blobName = "dir1/dir2/blob1";
- runner.setProperty(PutAzureBlobStorage_v12.BLOB_NAME, blobName);
-
- runProcessor(BLOB_DATA);
-
- assertSuccess(getContainerName(), blobName, BLOB_DATA);
- }
-
- @Test
- public void testPutEmptyBlob() throws Exception {
- byte[] blobData = new byte[0];
-
- runProcessor(blobData);
-
- assertSuccess(getContainerName(), BLOB_NAME, blobData);
- }
-
- @Test
- public void testPutBigBlob() throws Exception {
- Random random = new Random();
- byte[] blobData = new byte[120_000_000];
- random.nextBytes(blobData);
-
- runProcessor(blobData);
-
- assertSuccess(getContainerName(), BLOB_NAME, blobData);
- }
-
- @Test
- public void testPutBlobWithNonExistingContainerAndCreateContainerFalse() throws Exception {
- String containerName = generateContainerName();
- runner.setProperty(AzureStorageUtils.CONTAINER, containerName);
- runner.setProperty(AzureStorageUtils.CREATE_CONTAINER, "false");
-
- runProcessor(BLOB_DATA);
-
- assertFailure(BLOB_DATA, BlobErrorCode.CONTAINER_NOT_FOUND);
- }
-
- @Test
- public void testPutBlobWithNonExistingContainerAndCreateContainerTrue() throws Exception {
- String containerName = generateContainerName();
- runner.setProperty(AzureStorageUtils.CONTAINER, containerName);
- runner.setProperty(AzureStorageUtils.CREATE_CONTAINER, "true");
-
- try {
- runProcessor(BLOB_DATA);
-
- assertSuccess(containerName, BLOB_NAME, BLOB_DATA);
- } finally {
- BlobContainerClient containerClient = getStorageClient().getBlobContainerClient(containerName);
- containerClient.delete();
- }
- }
-
- @Test
- public void testPutBlobWithSpacesInBlobName() throws Exception {
- String blobName = "dir 1/blob 1";
- runner.setProperty(PutAzureBlobStorage_v12.BLOB_NAME, blobName);
-
- runProcessor(BLOB_DATA);
-
- assertSuccess(getContainerName(), blobName, BLOB_DATA);
- }
-
- @Test
- public void testPutBlobToExistingBlob() throws Exception {
- uploadBlob(BLOB_NAME, BLOB_DATA);
-
- runProcessor(BLOB_DATA);
-
- MockFlowFile flowFile = assertFailure(BLOB_DATA, BlobErrorCode.BLOB_ALREADY_EXISTS);
- assertNull(flowFile.getAttribute(ATTR_NAME_IGNORED));
- }
-
- @Test
- public void testPutBlobToExistingBlobConflictStrategyIgnore() throws Exception {
- uploadBlob(BLOB_NAME, BLOB_DATA);
- runner.setProperty(AzureStorageUtils.CONFLICT_RESOLUTION, AzureStorageConflictResolutionStrategy.IGNORE_RESOLUTION);
-
- runProcessor(BLOB_DATA);
-
- MockFlowFile flowFile = assertIgnored(getContainerName(), BLOB_NAME);
- assertEquals(flowFile.getAttribute(ATTR_NAME_IGNORED), "true");
- }
-
- @Test
- public void testPutBlobToExistingBlobConflictStrategyReplace() throws Exception {
- uploadBlob(BLOB_NAME, BLOB_DATA);
- runner.setProperty(AzureStorageUtils.CONFLICT_RESOLUTION, AzureStorageConflictResolutionStrategy.REPLACE_RESOLUTION);
-
- runProcessor(BLOB_DATA);
-
- assertSuccess(getContainerName(), BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testPutBlobUsingExpressionLanguage() throws Exception {
- Map<String, String> attributes = initCommonExpressionLanguageAttributes();
-
- runProcessor(BLOB_DATA, attributes);
-
- assertSuccess(getContainerName(), BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testPutBlob64BLocalCSE() {
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_TYPE, ClientSideEncryptionMethod.LOCAL);
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_ID, KEY_ID_VALUE);
- runner.setProperty(ClientSideEncryptionSupport.CSE_LOCAL_KEY, KEY_64B_VALUE);
- runner.assertNotValid();
- }
-
- @Test
- public void testPutBlob128BLocalCSE() throws Exception {
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_TYPE, ClientSideEncryptionMethod.LOCAL);
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_ID, KEY_ID_VALUE);
- runner.setProperty(ClientSideEncryptionSupport.CSE_LOCAL_KEY, KEY_128B_VALUE);
- runProcessor(BLOB_DATA);
- assertSuccessForCSE(getContainerName(), BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testPutBlob192BLocalCSE() throws Exception {
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_TYPE, ClientSideEncryptionMethod.LOCAL);
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_ID, KEY_ID_VALUE);
- runner.setProperty(ClientSideEncryptionSupport.CSE_LOCAL_KEY, KEY_192B_VALUE);
- runProcessor(BLOB_DATA);
- assertSuccessForCSE(getContainerName(), BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testPutBlob256BLocalCSE() throws Exception {
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_TYPE, ClientSideEncryptionMethod.LOCAL);
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_ID, KEY_ID_VALUE);
- runner.setProperty(ClientSideEncryptionSupport.CSE_LOCAL_KEY, KEY_256B_VALUE);
- runProcessor(BLOB_DATA);
- assertSuccessForCSE(getContainerName(), BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testPutBlob384BLocalCSE() throws Exception {
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_TYPE, ClientSideEncryptionMethod.LOCAL);
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_ID, KEY_ID_VALUE);
- runner.setProperty(ClientSideEncryptionSupport.CSE_LOCAL_KEY, KEY_384B_VALUE);
- runProcessor(BLOB_DATA);
- assertSuccessForCSE(getContainerName(), BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testPutBlob512BLocalCSE() throws Exception {
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_TYPE, ClientSideEncryptionMethod.LOCAL);
- runner.setProperty(ClientSideEncryptionSupport.CSE_KEY_ID, KEY_ID_VALUE);
- runner.setProperty(ClientSideEncryptionSupport.CSE_LOCAL_KEY, KEY_512B_VALUE);
- runProcessor(BLOB_DATA);
- assertSuccessForCSE(getContainerName(), BLOB_NAME, BLOB_DATA);
- }
-
- @Test
- public void testPutBlobFromLocalFile() throws Exception {
- String attributeName = "file.path";
-
- String serviceId = FileResourceService.class.getSimpleName();
- FileResourceService service = new StandardFileResourceService();
- runner.addControllerService(serviceId, service);
- runner.setProperty(service, StandardFileResourceService.FILE_PATH, String.format("${%s}", attributeName));
- runner.enableControllerService(service);
-
- runner.setProperty(ResourceTransferProperties.RESOURCE_TRANSFER_SOURCE, ResourceTransferSource.FILE_RESOURCE_SERVICE);
- runner.setProperty(ResourceTransferProperties.FILE_RESOURCE_SERVICE, serviceId);
-
- Path tempFilePath = Files.createTempFile("ITPutAzureBlobStorage_v12_testPutBlobFromLocalFile_", "");
- Files.write(tempFilePath, BLOB_DATA);
-
- Map<String, String> attributes = new HashMap<>();
- attributes.put(attributeName, tempFilePath.toString());
-
- runProcessor(EMPTY_CONTENT, attributes);
-
- runner.assertAllFlowFilesTransferred(PutAzureBlobStorage_v12.REL_SUCCESS, 1);
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(PutAzureBlobStorage_v12.REL_SUCCESS).getFirst();
- assertFlowFileCommonBlobAttributes(flowFile, getContainerName(), BLOB_NAME);
- assertFlowFileResultBlobAttributes(flowFile, BLOB_DATA.length);
-
- assertAzureBlob(getContainerName(), BLOB_NAME, BLOB_DATA);
- assertProvenanceEvents();
- }
-
- @Test
- public void testPutBlobFromNonExistentLocalFile() throws Exception {
- String attributeName = "file.path";
-
- String serviceId = FileResourceService.class.getSimpleName();
- FileResourceService service = new StandardFileResourceService();
- runner.addControllerService(serviceId, service);
- runner.setProperty(service, StandardFileResourceService.FILE_PATH, String.format("${%s}", attributeName));
- runner.enableControllerService(service);
-
- runner.setProperty(ResourceTransferProperties.RESOURCE_TRANSFER_SOURCE, ResourceTransferSource.FILE_RESOURCE_SERVICE);
- runner.setProperty(ResourceTransferProperties.FILE_RESOURCE_SERVICE, serviceId);
-
- String filePath = "nonexistent.txt";
-
- Map<String, String> attributes = new HashMap<>();
- attributes.put(attributeName, filePath);
-
- runProcessor(EMPTY_CONTENT, attributes);
-
- runner.assertAllFlowFilesTransferred(PutAzureBlobStorage_v12.REL_FAILURE, 1);
-
- assertProvenanceEvents();
- }
-
-
- private void runProcessor(byte[] data) {
- runProcessor(data, Collections.emptyMap());
- }
-
- private void runProcessor(byte[] data, Map<String, String> attributes) {
- runner.assertValid();
- runner.enqueue(data, attributes);
- runner.run();
- }
-
- private void assertSuccess(String containerName, String blobName, byte[] blobData) throws Exception {
- assertFlowFile(containerName, blobName, blobData);
- assertAzureBlob(containerName, blobName, blobData);
- assertProvenanceEvents();
- }
-
- private void assertSuccessForCSE(String containerName, String blobName, byte[] blobData) throws Exception {
- assertFlowFile(containerName, blobName, blobData);
- assertAzureBlobExists(containerName, blobName);
- assertProvenanceEvents();
- }
-
- private MockFlowFile assertIgnored(String containerName, String blobName) throws Exception {
- MockFlowFile flowFile = assertFlowFile(containerName, blobName, null);
- assertProvenanceEvents();
- return flowFile;
- }
-
- private MockFlowFile assertFlowFile(String containerName, String blobName, byte[] blobData) throws Exception {
- runner.assertAllFlowFilesTransferred(PutAzureBlobStorage_v12.REL_SUCCESS, 1);
-
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(PutAzureBlobStorage_v12.REL_SUCCESS).getFirst();
-
- assertFlowFileCommonBlobAttributes(flowFile, containerName, blobName);
- if (blobData != null) {
- assertFlowFileResultBlobAttributes(flowFile, blobData.length);
- flowFile.assertContentEquals(blobData);
- flowFile.assertAttributeEquals("azure.length", String.valueOf(blobData.length));
- }
- return flowFile;
- }
-
- private void assertAzureBlob(String containerName, String blobName, byte[] blobData) {
- BlobClient blobClient = assertAzureBlobExists(containerName, blobName);
- assertEquals(blobData.length, blobClient.getProperties().getBlobSize());
- }
-
- private BlobClient assertAzureBlobExists(String containerName, String blobName) {
- BlobContainerClient containerClient = getStorageClient().getBlobContainerClient(containerName);
- BlobClient blobClient = containerClient.getBlobClient(blobName);
- assertTrue(blobClient.exists());
- return blobClient;
- }
-
- private void assertProvenanceEvents() {
- Set<ProvenanceEventType> expectedEventTypes = Collections.singleton(ProvenanceEventType.SEND);
-
- Set<ProvenanceEventType> actualEventTypes = runner.getProvenanceEvents().stream()
- .map(ProvenanceEventRecord::getEventType)
- .collect(Collectors.toSet());
- assertEquals(expectedEventTypes, actualEventTypes);
- }
-
- private MockFlowFile assertFailure(byte[] blobData, BlobErrorCode errorCode) throws Exception {
- runner.assertAllFlowFilesTransferred(PutAzureBlobStorage_v12.REL_FAILURE, 1);
-
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(DeleteAzureBlobStorage_v12.REL_FAILURE).getFirst();
- flowFile.assertContentEquals(blobData);
- flowFile.assertAttributeEquals(ATTR_NAME_ERROR_CODE, errorCode.toString());
- return flowFile;
- }
-}
diff --git a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITPutAzureDataLakeStorage.java b/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITPutAzureDataLakeStorage.java
deleted file mode 100644
index b4708375b9..0000000000
--- a/nifi-extension-bundles/nifi-azure-bundle/nifi-azure-processors/src/test/java/org/apache/nifi/processors/azure/storage/ITPutAzureDataLakeStorage.java
+++ /dev/null
@@ -1,434 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.azure.storage;
-
-import com.azure.storage.file.datalake.DataLakeDirectoryClient;
-import com.azure.storage.file.datalake.DataLakeFileClient;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.nifi.fileresource.service.StandardFileResourceService;
-import org.apache.nifi.fileresource.service.api.FileResourceService;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processors.azure.storage.utils.AzureStorageUtils;
-import org.apache.nifi.processors.azure.storage.utils.WritingStrategy;
-import org.apache.nifi.processors.transfer.ResourceTransferProperties;
-import org.apache.nifi.processors.transfer.ResourceTransferSource;
-import org.apache.nifi.provenance.ProvenanceEventRecord;
-import org.apache.nifi.provenance.ProvenanceEventType;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.params.ParameterizedTest;
-import org.junit.jupiter.params.provider.EnumSource;
-
-import java.nio.charset.StandardCharsets;
-import java.nio.file.Files;
-import java.nio.file.Path;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Random;
-import java.util.Set;
-import java.util.stream.Collectors;
-
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_DIRECTORY;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_FILENAME;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_FILESYSTEM;
-import static org.apache.nifi.processors.azure.storage.utils.ADLSAttributes.ATTR_NAME_LENGTH;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-public class ITPutAzureDataLakeStorage extends AbstractAzureDataLakeStorageIT {
-
- private static final String DIRECTORY = "dir1";
- private static final String FILE_NAME = "file1";
- private static final byte[] FILE_DATA = "0123456789".getBytes(StandardCharsets.UTF_8);
-
- private static final String EL_FILESYSTEM = "az.filesystem";
- private static final String EL_DIRECTORY = "az.directory";
- private static final String EL_FILE_NAME = "az.filename";
-
- @Override
- protected Class<? extends Processor> getProcessorClass() {
- return PutAzureDataLakeStorage.class;
- }
-
- @BeforeEach
- public void setUp() {
- runner.setProperty(AzureStorageUtils.DIRECTORY, DIRECTORY);
- runner.setProperty(AzureStorageUtils.FILE, FILE_NAME);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileToExistingDirectory(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- fileSystemClient.createDirectory(DIRECTORY);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileToExistingDirectoryWithReplaceResolution(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- fileSystemClient.createDirectory(DIRECTORY);
-
- runner.setProperty(PutAzureDataLakeStorage.CONFLICT_RESOLUTION, PutAzureDataLakeStorage.REPLACE_RESOLUTION);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileToExistingDirectoryWithIgnoreResolution(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- fileSystemClient.createDirectory(DIRECTORY);
-
- runner.setProperty(PutAzureDataLakeStorage.CONFLICT_RESOLUTION, PutAzureDataLakeStorage.IGNORE_RESOLUTION);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileToNonExistingDirectory(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileToDeepDirectory(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- String baseDirectory = "dir1/dir2";
- String fullDirectory = baseDirectory + "/dir3/dir4";
- fileSystemClient.createDirectory(baseDirectory);
- runner.setProperty(AzureStorageUtils.DIRECTORY, fullDirectory);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(fullDirectory, FILE_NAME, FILE_DATA);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileToRootDirectory(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- String rootDirectory = "";
- runner.setProperty(AzureStorageUtils.DIRECTORY, rootDirectory);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(rootDirectory, FILE_NAME, FILE_DATA);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutEmptyFile(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- byte[] fileData = new byte[0];
-
- runProcessor(fileData);
-
- assertSuccess(DIRECTORY, FILE_NAME, fileData);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutBigFile(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- Random random = new Random();
- byte[] fileData = new byte[120_000_000];
- random.nextBytes(fileData);
-
- runProcessor(fileData);
-
- assertSuccess(DIRECTORY, FILE_NAME, fileData);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileWithNonExistingFileSystem(WritingStrategy writingStrategy) {
- setWritingStrategy(writingStrategy);
-
- runner.setProperty(AzureStorageUtils.FILESYSTEM, "dummy");
-
- runProcessor(FILE_DATA);
-
- assertFailure();
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileWithInvalidFileName(WritingStrategy writingStrategy) {
- setWritingStrategy(writingStrategy);
-
- runner.setProperty(AzureStorageUtils.FILE, "/file1");
-
- runProcessor(FILE_DATA);
-
- assertFailure();
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileWithSpacesInDirectoryAndFileName(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- String directory = "dir 1";
- String fileName = "file 1";
- runner.setProperty(AzureStorageUtils.DIRECTORY, directory);
- runner.setProperty(AzureStorageUtils.FILE, fileName);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(directory, fileName, FILE_DATA);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileToExistingFileWithFailResolution(WritingStrategy writingStrategy) {
- setWritingStrategy(writingStrategy);
-
- fileSystemClient.createFile(String.format("%s/%s", DIRECTORY, FILE_NAME));
-
- runProcessor(FILE_DATA);
-
- assertFailure();
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileToExistingFileWithReplaceResolution(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- fileSystemClient.createFile(String.format("%s/%s", DIRECTORY, FILE_NAME));
-
- runner.setProperty(PutAzureDataLakeStorage.CONFLICT_RESOLUTION, PutAzureDataLakeStorage.REPLACE_RESOLUTION);
-
- runProcessor(FILE_DATA);
-
- assertSuccess(DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileToExistingFileWithIgnoreResolution(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- String azureFileContent = "AzureFileContent";
- createDirectoryAndUploadFile(DIRECTORY, FILE_NAME, azureFileContent);
-
- runner.setProperty(PutAzureDataLakeStorage.CONFLICT_RESOLUTION, PutAzureDataLakeStorage.IGNORE_RESOLUTION);
-
- runProcessor(FILE_DATA);
-
- assertSuccessWithIgnoreResolution(DIRECTORY, FILE_NAME, FILE_DATA, azureFileContent.getBytes(StandardCharsets.UTF_8));
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileWithEL(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- Map<String, String> attributes = createAttributesMap();
- setELProperties();
-
- runProcessor(FILE_DATA, attributes);
-
- assertSuccess(DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileWithELButFilesystemIsNotSpecified(WritingStrategy writingStrategy) {
- setWritingStrategy(writingStrategy);
-
- Map<String, String> attributes = createAttributesMap();
- attributes.remove(EL_FILESYSTEM);
- setELProperties();
-
- runProcessor(FILE_DATA, attributes);
-
- assertFailure();
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileWithELButFileNameIsNotSpecified(WritingStrategy writingStrategy) {
- setWritingStrategy(writingStrategy);
-
- Map<String, String> attributes = createAttributesMap();
- attributes.remove(EL_FILE_NAME);
- setELProperties();
-
- runProcessor(FILE_DATA, attributes);
-
- assertFailure();
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileFromLocalFile(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- String attributeName = "file.path";
-
- String serviceId = FileResourceService.class.getSimpleName();
- FileResourceService service = new StandardFileResourceService();
- runner.addControllerService(serviceId, service);
- runner.setProperty(service, StandardFileResourceService.FILE_PATH, String.format("${%s}", attributeName));
- runner.enableControllerService(service);
-
- runner.setProperty(ResourceTransferProperties.RESOURCE_TRANSFER_SOURCE, ResourceTransferSource.FILE_RESOURCE_SERVICE);
- runner.setProperty(ResourceTransferProperties.FILE_RESOURCE_SERVICE, serviceId);
-
- Path tempFilePath = Files.createTempFile("ITPutAzureDataLakeStorage_testPutFileFromLocalFile_", "");
- Files.write(tempFilePath, FILE_DATA);
-
- Map<String, String> attributes = new HashMap<>();
- attributes.put(attributeName, tempFilePath.toString());
-
- runProcessor(EMPTY_CONTENT, attributes);
-
- MockFlowFile flowFile = assertFlowFile(EMPTY_CONTENT);
- assertFlowFileAttributes(flowFile, DIRECTORY, FILE_NAME, FILE_DATA.length);
- assertAzureFile(DIRECTORY, FILE_NAME, FILE_DATA);
- assertProvenanceEvents();
- }
-
- @ParameterizedTest
- @EnumSource(WritingStrategy.class)
- public void testPutFileUsingProxy(WritingStrategy writingStrategy) throws Exception {
- setWritingStrategy(writingStrategy);
-
- fileSystemClient.createDirectory(DIRECTORY);
- configureProxyService();
-
- runProcessor(FILE_DATA);
-
- assertSuccess(DIRECTORY, FILE_NAME, FILE_DATA);
- }
-
- private Map<String, String> createAttributesMap() {
- Map<String, String> attributes = new HashMap<>();
-
- attributes.put(EL_FILESYSTEM, fileSystemName);
- attributes.put(EL_DIRECTORY, DIRECTORY);
- attributes.put(EL_FILE_NAME, FILE_NAME);
-
- return attributes;
- }
-
- private void setWritingStrategy(WritingStrategy writingStrategy) {
- runner.setProperty(PutAzureDataLakeStorage.WRITING_STRATEGY, writingStrategy);
- }
-
- private void setELProperties() {
- runner.setProperty(AzureStorageUtils.FILESYSTEM, String.format("${%s}", EL_FILESYSTEM));
- runner.setProperty(AzureStorageUtils.DIRECTORY, String.format("${%s}", EL_DIRECTORY));
- runner.setProperty(AzureStorageUtils.FILE, String.format("${%s}", EL_FILE_NAME));
- }
-
- private void runProcessor(byte[] fileData) {
- runProcessor(fileData, Collections.emptyMap());
- }
-
- private void runProcessor(byte[] testData, Map<String, String> attributes) {
- runner.assertValid();
- runner.enqueue(testData, attributes);
- runner.run();
- }
-
- private void assertSuccess(String directory, String fileName, byte[] fileData) throws Exception {
- assertFlowFile(fileData, fileName, directory);
- assertAzureFile(directory, fileName, fileData);
- assertProvenanceEvents();
- }
-
- private void assertSuccessWithIgnoreResolution(String directory, String fileName, byte[] fileData, byte[] azureFileData) throws Exception {
- assertFlowFile(fileData);
- assertAzureFile(directory, fileName, azureFileData);
- }
-
- private void assertFlowFile(byte[] fileData, String fileName, String directory) throws Exception {
- MockFlowFile flowFile = assertFlowFile(fileData);
-
- assertFlowFileAttributes(flowFile, directory, fileName, fileData.length);
- }
-
- private MockFlowFile assertFlowFile(byte[] fileData) throws Exception {
- runner.assertAllFlowFilesTransferred(PutAzureDataLakeStorage.REL_SUCCESS, 1);
-
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(PutAzureDataLakeStorage.REL_SUCCESS).getFirst();
-
- flowFile.assertContentEquals(fileData);
-
- return flowFile;
- }
-
- private void assertFlowFileAttributes(MockFlowFile flowFile, String directory, String fileName, int fileLength) {
- flowFile.assertAttributeEquals(ATTR_NAME_FILESYSTEM, fileSystemName);
- flowFile.assertAttributeEquals(ATTR_NAME_DIRECTORY, directory);
- flowFile.assertAttributeEquals(ATTR_NAME_FILENAME, fileName);
-
- flowFile.assertAttributeEquals(ATTR_NAME_LENGTH, Integer.toString(fileLength));
- }
-
- private void assertAzureFile(String directory, String fileName, byte[] fileData) {
- DataLakeFileClient fileClient;
- if (StringUtils.isNotEmpty(directory)) {
- DataLakeDirectoryClient directoryClient = fileSystemClient.getDirectoryClient(directory);
- assertTrue(directoryClient.exists());
-
- fileClient = directoryClient.getFileClient(fileName);
- } else {
- fileClient = fileSystemClient.getFileClient(fileName);
- }
-
- assertTrue(fileClient.exists());
- assertEquals(fileData.length, fileClient.getProperties().getFileSize());
- }
-
- private void assertProvenanceEvents() {
- Set<ProvenanceEventType> expectedEventTypes = Collections.singleton(ProvenanceEventType.SEND);
-
- Set<ProvenanceEventType> actualEventTypes = runner.getProvenanceEvents().stream()
- .map(ProvenanceEventRecord::getEventType)
- .collect(Collectors.toSet());
- assertEquals(expectedEventTypes, actualEventTypes);
- }
-
- private void assertFailure() {
- runner.assertAllFlowFilesTransferred(PutAzureDataLakeStorage.REL_FAILURE, 1);
- }
-}
diff --git a/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/AbstractBoxFileIT.java b/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/AbstractBoxFileIT.java
deleted file mode 100644
index 053083f5c8..0000000000
--- a/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/AbstractBoxFileIT.java
+++ /dev/null
@@ -1,137 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.box;
-
-import static java.nio.charset.StandardCharsets.UTF_8;
-import static org.mockito.Mockito.doReturn;
-import static org.mockito.Mockito.mock;
-
-import com.box.sdk.BoxAPIConnection;
-import com.box.sdk.BoxConfig;
-import com.box.sdk.BoxDeveloperEditionAPIConnection;
-import com.box.sdk.BoxFile;
-import com.box.sdk.BoxFolder;
-import java.io.ByteArrayInputStream;
-import java.io.FileReader;
-import java.io.Reader;
-import java.util.Arrays;
-import java.util.Set;
-import java.util.stream.Collectors;
-import org.apache.nifi.box.controllerservices.BoxClientService;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-
-/**
- * Set the following constants before running:<br />
- * <br />
- * FOLDER_ID - The ID of a Folder the test can use to create files and sub-folders within.<br />
- * ACCOUNT_ID - The ID of the Account owning the Folder.<br />
- * APP_CONFIG_FILE - An App Settings Configuration JSON file.
- * Read nifi-nar-bundles/nifi-box-bundle/nifi-box-services/src/main/resources/docs/org.apache.nifi.box.controllerservices.JsonConfigBasedBoxClientService/additionalDetails.html for details.<br />
- * <br />
- * Created files and folders are cleaned up, but it's advisable to dedicate a folder for this test so that it can be cleaned up easily should the test fail to do so.
- */
-public abstract class AbstractBoxFileIT<T extends Processor> {
- static final String FOLDER_ID = "";
- static final String ACCOUNT_ID = "";
- static final String APP_CONFIG_FILE = "";
-
- public static final String DEFAULT_FILE_CONTENT = "test_content";
- public static final String CHANGED_FILE_CONTENT = "changed_test_content";
- public static final String TEST_FILENAME = "test_filename.txt";
- public static final String MAIN_FOLDER_NAME = "main";
-
- protected T testSubject;
- protected TestRunner testRunner;
-
- protected BoxAPIConnection boxAPIConnection;
-
- protected String targetFolderName;
- protected String mainFolderId;
-
- protected abstract T createTestSubject();
-
- @BeforeEach
- protected void init() throws Exception {
- try (
- Reader reader = new FileReader(APP_CONFIG_FILE);
- ) {
- BoxConfig boxConfig = BoxConfig.readFrom(reader);
- boxAPIConnection = BoxDeveloperEditionAPIConnection.getAppEnterpriseConnection(boxConfig);
- boxAPIConnection.asUser(ACCOUNT_ID);
- }
-
- targetFolderName = new BoxFolder(boxAPIConnection, FOLDER_ID).getInfo("name").getName();
-
- BoxFolder.Info mainFolderInfo = createFolder(MAIN_FOLDER_NAME, FOLDER_ID);
- mainFolderId = mainFolderInfo.getID();
-
- testSubject = createTestSubject();
- testRunner = createTestRunner();
- }
-
- @AfterEach
- protected void tearDown() {
- if (boxAPIConnection != null) {
- BoxFolder folder = new BoxFolder(boxAPIConnection, mainFolderId);
- folder.delete(true);
- }
- }
-
- protected TestRunner createTestRunner() throws Exception{
- TestRunner testRunner = TestRunners.newTestRunner(testSubject);
-
- BoxClientService boxClientService = mock(BoxClientService.class);
- doReturn(boxClientService.toString()).when(boxClientService).getIdentifier();
- doReturn(boxAPIConnection).when(boxClientService).getBoxApiConnection();
-
- testRunner.addControllerService(boxClientService.getIdentifier(), boxClientService);
- testRunner.enableControllerService(boxClientService);
- testRunner.setProperty(BoxClientService.BOX_CLIENT_SERVICE, boxClientService.getIdentifier());
-
- return testRunner;
- }
-
- protected BoxFolder.Info createFolder(String folderName, String parentFolderId) {
- BoxFolder parentFolder = new BoxFolder(boxAPIConnection, parentFolderId);
- BoxFolder.Info folderInfo = parentFolder.createFolder(folderName);
- return folderInfo;
- }
-
- protected BoxFile.Info createFileWithDefaultContent(String name, String folderId) {
- return createFile(name, folderId);
- }
-
- protected BoxFile.Info createFile(String name, String folderId) {
- BoxFolder folder = new BoxFolder(boxAPIConnection, folderId);
-
- BoxFile.Info fileInfo = folder.uploadFile(new ByteArrayInputStream(DEFAULT_FILE_CONTENT.getBytes(UTF_8)), name);
-
- return fileInfo;
- }
-
- protected Set<String> getCheckedAttributeNames() {
- Set<String> checkedAttributeNames = Arrays.stream(BoxFlowFileAttribute.values())
- .map(BoxFlowFileAttribute::getName)
- .collect(Collectors.toSet());
-
- return checkedAttributeNames;
- }
-}
diff --git a/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/FetchBoxFileIT.java b/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/FetchBoxFileIT.java
deleted file mode 100644
index a1773f5fbb..0000000000
--- a/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/FetchBoxFileIT.java
+++ /dev/null
@@ -1,137 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.box;
-
-import static java.util.Collections.singletonList;
-
-import com.box.sdk.BoxFile;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.concurrent.atomic.AtomicBoolean;
-import org.apache.nifi.flowfile.attributes.CoreAttributes;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.Test;
-
-/**
- * See Javadoc {@link AbstractBoxFileIT} for instructions how to run this test.
- */
-public class FetchBoxFileIT extends AbstractBoxFileIT<FetchBoxFile> {
- @Override
- public FetchBoxFile createTestSubject() {
- FetchBoxFile testSubject = new FetchBoxFile();
-
- return testSubject;
- }
-
- @Test
- void testFetchSingleFile() {
- BoxFile.Info file = createFileWithDefaultContent("test_file.txt", mainFolderId);
-
- Map<String, String> inputFlowFileAttributes = new HashMap<>();
- inputFlowFileAttributes.put(BoxFileAttributes.ID, file.getID());
- inputFlowFileAttributes.put(CoreAttributes.FILENAME.key(), file.getName());
-
- HashSet<Map<String, String>> expectedAttributes = new HashSet<>(singletonList(inputFlowFileAttributes));
- List<String> expectedContent = singletonList(DEFAULT_FILE_CONTENT);
-
- testRunner.enqueue("unimportant_data", inputFlowFileAttributes);
- testRunner.run();
-
- testRunner.assertTransferCount(FetchBoxFile.REL_FAILURE, 0);
-
- testRunner.assertAttributes(FetchBoxFile.REL_SUCCESS, getCheckedAttributeNames(), expectedAttributes);
- testRunner.assertContents(FetchBoxFile.REL_SUCCESS, expectedContent);
- }
-
- @Test
- void testInputFlowFileReferencesMissingFile() {
- Map<String, String> inputFlowFileAttributes = new HashMap<>();
- inputFlowFileAttributes.put(BoxFileAttributes.ID, "111");
- inputFlowFileAttributes.put(CoreAttributes.FILENAME.key(), "missing_filename");
-
- Set<Map<String, String>> expectedFailureAttributes = new HashSet<>(singletonList(
- new HashMap<String, String>() {{
- put(BoxFileAttributes.ID, "111");
- put(CoreAttributes.FILENAME.key(), "missing_filename");
- put(BoxFileAttributes.ERROR_CODE, "404");
- }}
- ));
-
-
- testRunner.enqueue("unimportant_data", inputFlowFileAttributes);
- testRunner.run();
-
- testRunner.assertTransferCount(FetchBoxFile.REL_SUCCESS, 0);
-
- testRunner.assertAttributes(FetchBoxFile.REL_FAILURE, getCheckedAttributeNames(), expectedFailureAttributes);
- }
-
- @Test
- void testInputFlowFileThrowsExceptionBeforeFetching() {
- BoxFile.Info file = createFileWithDefaultContent("test_file.txt", mainFolderId);
-
- Map<String, String> inputFlowFileAttributes = new HashMap<>();
- inputFlowFileAttributes.put(BoxFileAttributes.ID, file.getID());
- inputFlowFileAttributes.put(CoreAttributes.FILENAME.key(), file.getName());
-
- MockFlowFile input = new MockFlowFile(1) {
- final AtomicBoolean throwException = new AtomicBoolean(true);
-
- @Override
- public boolean isPenalized() {
- // We want to throw exception only once because the exception handling itself calls this again
- if (throwException.get()) {
- throwException.set(false);
- throw new RuntimeException("Intentional exception");
- } else {
- return super.isPenalized();
- }
- }
-
- @Override
- public Map<String, String> getAttributes() {
- return inputFlowFileAttributes;
- }
- };
-
- Set<Map<String, String>> expectedFailureAttributes = new HashSet<>(singletonList(
- new HashMap<String, String>() {{
- putAll(inputFlowFileAttributes);
- }}
- ));
-
- testRunner.enqueue(input);
- testRunner.run();
-
- testRunner.assertTransferCount(FetchBoxFile.REL_SUCCESS, 0);
-
- testRunner.assertAttributes(FetchBoxFile.REL_FAILURE, getCheckedAttributeNames(), expectedFailureAttributes);
- }
-
- public Set<String> getCheckedAttributeNames() {
- Set<String> checkedAttributeNames = new HashSet<>();
-
- checkedAttributeNames.add(BoxFlowFileAttribute.ID.getName());
- checkedAttributeNames.add(BoxFlowFileAttribute.FILENAME.getName());
- checkedAttributeNames.add(BoxFileAttributes.ERROR_CODE);
-
- return checkedAttributeNames;
- }
-}
diff --git a/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/ListBoxFileIT.java b/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/ListBoxFileIT.java
deleted file mode 100644
index 0692c731e3..0000000000
--- a/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/ListBoxFileIT.java
+++ /dev/null
@@ -1,143 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.box;
-
-import com.box.sdk.BoxFolder;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Set;
-import java.util.stream.Collectors;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-/**
- * See Javadoc {@link AbstractBoxFileIT} for instructions how to run this test.
- */
-public class ListBoxFileIT extends AbstractBoxFileIT<ListBoxFile> {
- @BeforeEach
- public void init() throws Exception {
- super.init();
- testRunner.setProperty(ListBoxFile.FOLDER_ID, mainFolderId);
- }
-
- @Override
- public ListBoxFile createTestSubject() {
- ListBoxFile testSubject = new ListBoxFile();
-
- return testSubject;
- }
-
- @Test
- void listFilesFrom3LayerDeepDirectoryTree() throws Exception {
- BoxFolder.Info main_sub1 = createFolder("main_sub1", mainFolderId);
- BoxFolder.Info main_sub2 = createFolder("main_sub2", mainFolderId);
-
- BoxFolder.Info main_sub1_sub1 = createFolder("main_sub1_sub1", main_sub1.getID());
-
- createFileWithDefaultContent("main_file1", mainFolderId);
- createFileWithDefaultContent("main_file2", mainFolderId);
- createFileWithDefaultContent("main_file3", mainFolderId);
-
- createFileWithDefaultContent("main_sub1_file1", main_sub1.getID());
-
- createFileWithDefaultContent("main_sub2_file1", main_sub2.getID());
- createFileWithDefaultContent("main_sub2_file2", main_sub2.getID());
-
- createFileWithDefaultContent("main_sub1_sub1_file1", main_sub1_sub1.getID());
- createFileWithDefaultContent("main_sub1_sub1_file2", main_sub1_sub1.getID());
- createFileWithDefaultContent("main_sub1_sub1_file3", main_sub1_sub1.getID());
-
- Set<String> expectedFileNames = new HashSet<>(Arrays.asList(
- "main_file1", "main_file2", "main_file3",
- "main_sub1_file1",
- "main_sub2_file1", "main_sub2_file2",
- "main_sub1_sub1_file1", "main_sub1_sub1_file2", "main_sub1_sub1_file3"
- ));
-
- // The creation of the files are not (completely) synchronized.
- Thread.sleep(2000);
-
- testRunner.run();
-
- Set<String> actualFileNames = getActualFileNames();
-
- assertEquals(expectedFileNames, actualFileNames);
-
- // Next, list a sub folder, non-recursively this time. (Changing these properties will clear the Processor state as well
- // so all files are eligible for listing again.)
-
- testRunner.clearTransferState();
-
- expectedFileNames = new HashSet<>(Arrays.asList(
- "main_sub1_file1"
- ));
-
- testRunner.setProperty(ListBoxFile.FOLDER_ID, main_sub1.getID());
- testRunner.setProperty(ListBoxFile.RECURSIVE_SEARCH, "false");
- testRunner.run();
-
- actualFileNames = getActualFileNames();
-
- assertEquals(expectedFileNames, actualFileNames);
- }
-
- @Test
- void doNotListTooYoungFilesWhenMinAgeIsSet() throws Exception {
- testRunner.setProperty(ListBoxFile.MIN_AGE, "15 s");
-
- createFileWithDefaultContent("main_file", mainFolderId);
-
- // Make sure the file 'arrives' and could be listed
- Thread.sleep(5000);
-
- testRunner.run();
-
- Set<String> actualFileNames = getActualFileNames();
-
- assertEquals(Collections.emptySet(), actualFileNames);
-
- // Next, wait for another 10+ seconds for MIN_AGE to expire then list again
-
- Thread.sleep(10000);
-
- Set<String> expectedFileNames = new HashSet<>(Arrays.asList(
- "main_file"
- ));
-
- testRunner.run();
-
- actualFileNames = getActualFileNames();
-
- assertEquals(expectedFileNames, actualFileNames);
- }
-
- private Set<String> getActualFileNames() {
- List<MockFlowFile> successFlowFiles = testRunner.getFlowFilesForRelationship(ListBoxFile.REL_SUCCESS);
-
- Set<String> actualFileNames = successFlowFiles.stream()
- .map(flowFile -> flowFile.getAttribute("filename"))
- .collect(Collectors.toSet());
-
- return actualFileNames;
- }
-}
diff --git a/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/PutBoxFileIT.java b/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/PutBoxFileIT.java
deleted file mode 100644
index 1eea15dbf1..0000000000
--- a/nifi-extension-bundles/nifi-box-bundle/nifi-box-processors/src/test/java/org/apache/nifi/processors/box/PutBoxFileIT.java
+++ /dev/null
@@ -1,120 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.box;
-
-import org.apache.nifi.processors.conflict.resolution.ConflictResolutionStrategy;
-import org.junit.jupiter.api.Test;
-
-/**
- * See Javadoc {@link AbstractBoxFileIT} for instructions how to run this test.
- */
-public class PutBoxFileIT extends AbstractBoxFileIT<PutBoxFile>{
-
- @Override
- public PutBoxFile createTestSubject() {
- return new PutBoxFile();
- }
-
- @Test
- void testUploadFile() {
- testRunner.setProperty(PutBoxFile.FOLDER_ID, mainFolderId);
- testRunner.setProperty(PutBoxFile.FILE_NAME, TEST_FILENAME);
- testRunner.enqueue(DEFAULT_FILE_CONTENT);
- testRunner.run();
- testRunner.assertTransferCount(FetchBoxFile.REL_SUCCESS, 1);
- testRunner.assertTransferCount(FetchBoxFile.REL_FAILURE, 0);
- }
-
- @Test
- void testSubfoldersAreCreated() {
- testRunner.setProperty(PutBoxFile.FOLDER_ID, mainFolderId);
- testRunner.setProperty(PutBoxFile.SUBFOLDER_NAME, "sub1/sub2/sub3");
- testRunner.setProperty(PutBoxFile.CREATE_SUBFOLDER, "true");
- testRunner.setProperty(PutBoxFile.FILE_NAME, TEST_FILENAME);
- testRunner.enqueue(DEFAULT_FILE_CONTENT);
- testRunner.run();
- testRunner.assertTransferCount(FetchBoxFile.REL_SUCCESS, 1);
- testRunner.assertTransferCount(FetchBoxFile.REL_FAILURE, 0);
- }
-
- @Test
- void testSubfolderExists() {
- createFolder("sub1", mainFolderId);
-
- testRunner.setProperty(PutBoxFile.FOLDER_ID, mainFolderId);
- testRunner.setProperty(PutBoxFile.SUBFOLDER_NAME, "sub1/sub2/sub3");
- testRunner.setProperty(PutBoxFile.CREATE_SUBFOLDER, "true");
- testRunner.setProperty(PutBoxFile.FILE_NAME, TEST_FILENAME);
- testRunner.enqueue(DEFAULT_FILE_CONTENT);
- testRunner.run();
- testRunner.assertTransferCount(FetchBoxFile.REL_SUCCESS, 1);
- testRunner.assertTransferCount(FetchBoxFile.REL_FAILURE, 0);
- }
-
- @Test
- void testUploadExistingFileFailResolution() {
- testRunner.setProperty(PutBoxFile.FOLDER_ID, mainFolderId);
- testRunner.setProperty(PutBoxFile.FILE_NAME, TEST_FILENAME);
- testRunner.enqueue(DEFAULT_FILE_CONTENT);
- testRunner.run();
- testRunner.assertTransferCount(PutBoxFile.REL_SUCCESS, 1);
- testRunner.assertTransferCount(PutBoxFile.REL_FAILURE, 0);
- testRunner.clearTransferState();
- testRunner.enqueue(CHANGED_FILE_CONTENT);
- testRunner.run();
- testRunner.assertTransferCount(PutBoxFile.REL_SUCCESS, 0);
- testRunner.assertTransferCount(PutBoxFile.REL_FAILURE, 1);
- }
-
- @Test
- void testUploadExistingFileIgnoreResolution() {
- testRunner.setProperty(PutBoxFile.FOLDER_ID, mainFolderId);
- testRunner.setProperty(PutBoxFile.FILE_NAME, TEST_FILENAME);
- testRunner.setProperty(PutBoxFile.CONFLICT_RESOLUTION, ConflictResolutionStrategy.IGNORE.getValue());
-
- testRunner.enqueue(DEFAULT_FILE_CONTENT);
- testRunner.run();
-
- testRunner.assertTransferCount(PutBoxFile.REL_SUCCESS, 1);
- testRunner.assertTransferCount(PutBoxFile.REL_FAILURE, 0);
- testRunner.clearTransferState();
-
- testRunner.enqueue(CHANGED_FILE_CONTENT);
- testRunner.run();
- testRunner.assertTransferCount(PutBoxFile.REL_SUCCESS, 1);
- testRunner.assertTransferCount(PutBoxFile.REL_FAILURE, 0);
- }
-
- @Test
- void testUploadExistingFileReplaceResolution() {
- testRunner.setProperty(PutBoxFile.FOLDER_ID, mainFolderId);
- testRunner.setProperty(PutBoxFile.FILE_NAME, TEST_FILENAME);
- testRunner.setProperty(PutBoxFile.CONFLICT_RESOLUTION, ConflictResolutionStrategy.REPLACE.getValue());
-
- testRunner.enqueue(DEFAULT_FILE_CONTENT);
- testRunner.run();
-
- testRunner.assertTransferCount(PutBoxFile.REL_SUCCESS, 1);
- testRunner.assertTransferCount(PutBoxFile.REL_FAILURE, 0);
- testRunner.clearTransferState();
-
- testRunner.enqueue(CHANGED_FILE_CONTENT);
- testRunner.run();
- testRunner.assertTransferCount(PutBoxFile.REL_SUCCESS, 1);
- testRunner.assertTransferCount(PutBoxFile.REL_FAILURE, 0);
- }
-}
diff --git a/nifi-extension-bundles/nifi-box-bundle/nifi-box-services/src/test/java/org/apache/nifi/box/controllerservices/JsonConfigBasedBoxClientServiceIT.java b/nifi-extension-bundles/nifi-box-bundle/nifi-box-services/src/test/java/org/apache/nifi/box/controllerservices/JsonConfigBasedBoxClientServiceIT.java
deleted file mode 100644
index 9cd9411b52..0000000000
--- a/nifi-extension-bundles/nifi-box-bundle/nifi-box-services/src/test/java/org/apache/nifi/box/controllerservices/JsonConfigBasedBoxClientServiceIT.java
+++ /dev/null
@@ -1,87 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.box.controllerservices;
-
-import com.box.sdk.BoxAPIConnection;
-import com.box.sdk.BoxFolder;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.nio.file.Files;
-import java.nio.file.Paths;
-
-import static org.mockito.Mockito.mock;
-
-/**
- * Set the following constants before running:<br />
- * <br />
- * FOLDER_ID - The ID of a Folder the test can use to retrieve metadata for.<br />
- * ACCOUNT_ID - The ID of the Account owning the Folder.<br />
- * APP_CONFIG_FILE - An App Settings Configuration JSON file.
- * Read nifi-nar-bundles/nifi-box-bundle/nifi-box-services/src/main/resources/docs/org.apache.nifi.box.controllerservices.JsonConfigBasedBoxClientService/additionalDetails.html for details.<br />
- */
-public class JsonConfigBasedBoxClientServiceIT {
- static final String FOLDER_ID = "";
- static final String ACCOUNT_ID = "";
- static final String APP_CONFIG_FILE = "";
-
- protected JsonConfigBasedBoxClientService testSubject;
- protected TestRunner testRunner;
-
- @BeforeEach
- protected void init() throws Exception {
- testSubject = new JsonConfigBasedBoxClientService();
-
- Processor dummyProcessor = mock(Processor.class);
- testRunner = TestRunners.newTestRunner(dummyProcessor);
-
- testRunner.addControllerService(testSubject.getClass().getSimpleName(), testSubject);
- }
-
- @Test
- void testWithAppConfigJson() throws Exception {
- // GIVEN
- String appConfigJson = new String(Files.readAllBytes(Paths.get(APP_CONFIG_FILE)));
-
- testRunner.setProperty(testSubject, JsonConfigBasedBoxClientService.ACCOUNT_ID, ACCOUNT_ID);
- testRunner.setProperty(testSubject, JsonConfigBasedBoxClientService.APP_CONFIG_JSON, appConfigJson);
- testRunner.enableControllerService(testSubject);
-
- BoxAPIConnection boxAPIConnection = testSubject.getBoxApiConnection();
-
- // WHEN
- // THEN
- new BoxFolder(boxAPIConnection, FOLDER_ID).getInfo("name").getName();
- }
-
- @Test
- void testWithAppConfigFile() throws Exception {
- // GIVEN
- testRunner.setProperty(testSubject, JsonConfigBasedBoxClientService.ACCOUNT_ID, ACCOUNT_ID);
- testRunner.setProperty(testSubject, JsonConfigBasedBoxClientService.APP_CONFIG_FILE, APP_CONFIG_FILE);
- testRunner.enableControllerService(testSubject);
-
- BoxAPIConnection boxAPIConnection = testSubject.getBoxApiConnection();
-
- // WHEN
- // THEN
- new BoxFolder(boxAPIConnection, FOLDER_ID).getInfo("name").getName();
- }
-}
diff --git a/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/AbstractDropboxIT.java b/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/AbstractDropboxIT.java
deleted file mode 100644
index 4dca22f473..0000000000
--- a/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/AbstractDropboxIT.java
+++ /dev/null
@@ -1,128 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.nifi.processors.dropbox;
-
-import com.dropbox.core.DbxRequestConfig;
-import com.dropbox.core.oauth.DbxCredential;
-import com.dropbox.core.v2.DbxClientV2;
-import com.dropbox.core.v2.files.FileMetadata;
-import com.dropbox.core.v2.files.GetMetadataErrorException;
-import java.io.ByteArrayInputStream;
-import java.nio.charset.StandardCharsets;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.services.dropbox.StandardDropboxCredentialService;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-
-/**
- * Set the following constants before running: <br/>
- * <br/> APPLICATION_KEY - App Key of your app
- * <br/> APPLICATION_SECRET - Your App Secret of your app
- * <br/> ACCESS_TOKEN - Access Token generated for your app
- * <br/> REFRESH_TOKEN - Refresh Token generated for your app
- * <br/><br/>
- * For App Key, App Secret, Access Token and Refresh Token generation please check controller service's Additional Details page.<br/><br/>
- * NOTE: Since the integration test creates the test files you need "files.content.write" permission besides the "files.content.read"
- * permission mentioned in controller service's Additional Details page.<br/>
- *
- * <br/> Created files and folders are cleaned up, but it's advisable to dedicate a folder for this test so that it can
- * be cleaned up easily should the test fail to do so. <br/>
- * <br/> WARNING: The creation of a file is not a synchronized operation, may need to adjust tests accordingly!
- */
-public abstract class AbstractDropboxIT<T extends Processor> {
-
- public static final String APP_KEY = "";
- public static final String APP_SECRET = "";
- public static final String ACCESS_TOKEN = "";
- public static final String REFRESH_TOKEN = "";
-
- public static final String MAIN_FOLDER = "/testFolder";
- protected T testSubject;
- protected TestRunner testRunner;
- protected String mainFolderId;
-
- private DbxClientV2 client;
-
- @AfterEach
- public void teardown() throws Exception {
- deleteFolderIfExists(MAIN_FOLDER);
- }
-
- protected abstract T createTestSubject();
-
- @BeforeEach
- protected void init() throws Exception {
- testSubject = createTestSubject();
- testRunner = createTestRunner();
- DbxCredential credential =
- new DbxCredential(ACCESS_TOKEN, -1L, REFRESH_TOKEN, APP_KEY, APP_SECRET);
- DbxRequestConfig config = new DbxRequestConfig("nifi");
- client = new DbxClientV2(config, credential);
- mainFolderId = createFolder(MAIN_FOLDER);
- }
-
- protected TestRunner createTestRunner() throws Exception {
- TestRunner testRunner = TestRunners.newTestRunner(testSubject);
-
- StandardDropboxCredentialService controllerService = new StandardDropboxCredentialService();
- testRunner.addControllerService("dropbox_credential_provider_service", controllerService);
- testRunner.setProperty(controllerService, StandardDropboxCredentialService.APP_KEY, APP_KEY);
- testRunner.setProperty(controllerService, StandardDropboxCredentialService.APP_SECRET, APP_SECRET);
- testRunner.setProperty(controllerService, StandardDropboxCredentialService.ACCESS_TOKEN, ACCESS_TOKEN);
- testRunner.setProperty(controllerService, StandardDropboxCredentialService.REFRESH_TOKEN, REFRESH_TOKEN);
- testRunner.enableControllerService(controllerService);
- testRunner.setProperty(ListDropbox.CREDENTIAL_SERVICE, "dropbox_credential_provider_service");
-
- return testRunner;
- }
-
- protected void deleteFolderIfExists(String path) throws Exception {
- if (folderExists(path)) {
- client.files().deleteV2(path);
- }
- }
-
- protected FileMetadata createFile(String folder, String filename, String fileContent) throws Exception {
- ByteArrayInputStream content = new ByteArrayInputStream(fileContent.getBytes(StandardCharsets.UTF_8));
- return client.files().upload(folder.equals("/") ? "/" + filename : folder + "/" + filename).uploadAndFinish(content);
- }
-
- private String createFolder(String path) throws Exception {
- if (folderExists(path)) {
- deleteFolder(path);
- }
- return client.files().createFolderV2(path).getMetadata().getId();
- }
-
- private void deleteFolder(String path) throws Exception {
- client.files().deleteV2(path);
- }
-
- private boolean folderExists(String path) throws Exception {
- try {
- return client.files().getMetadata(path) != null;
- } catch (GetMetadataErrorException e) {
- if (e.errorValue.isPath() && e.errorValue.getPathValue().isNotFound()) {
- return false;
- }
- throw e;
- }
- }
-}
\ No newline at end of file
diff --git a/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/FetchDropboxIT.java b/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/FetchDropboxIT.java
deleted file mode 100644
index 8f854e4a89..0000000000
--- a/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/FetchDropboxIT.java
+++ /dev/null
@@ -1,132 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.nifi.processors.dropbox;
-
-import com.dropbox.core.v2.files.FileMetadata;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-public class FetchDropboxIT extends AbstractDropboxIT<FetchDropbox> {
-
- private static final String DEFAULT_FILE_CONTENT = "test_file_content1";
-
- @BeforeEach
- public void init() throws Exception {
- super.init();
- }
-
- @Override
- protected FetchDropbox createTestSubject() {
- return new FetchDropbox();
- }
-
- @Test
- void testFetchFileById() throws Exception {
- FileMetadata file = createFile("/testFolder", "test_file1", DEFAULT_FILE_CONTENT);
-
- Map<String, String> inputFlowFileAttributes = new HashMap<>();
- inputFlowFileAttributes.put("dropbox.id", file.getId());
-
- testRunner.enqueue("unimportant_data", inputFlowFileAttributes);
- testRunner.run();
-
- testRunner.assertTransferCount(FetchDropbox.REL_FAILURE, 0);
- testRunner.assertTransferCount(FetchDropbox.REL_SUCCESS, 1);
-
- MockFlowFile flowFile = testRunner.getFlowFilesForRelationship(FetchDropbox.REL_SUCCESS).get(0);
- flowFile.assertContentEquals(DEFAULT_FILE_CONTENT);
- }
-
- @Test
- void testFetchFileByPath() throws Exception {
- createFile("/testFolder", "test_file1", DEFAULT_FILE_CONTENT);
-
- Map<String, String> inputFlowFileAttributes = new HashMap<>();
- inputFlowFileAttributes.put("path", "/testFolder");
- inputFlowFileAttributes.put("filename", "test_file1");
-
- testRunner.setProperty(FetchDropbox.FILE, "${path}/${filename}");
-
- testRunner.enqueue("unimportant_data", inputFlowFileAttributes);
- testRunner.run();
-
- testRunner.assertTransferCount(FetchDropbox.REL_FAILURE, 0);
- testRunner.assertTransferCount(FetchDropbox.REL_SUCCESS, 1);
- List<MockFlowFile> flowFiles = testRunner.getFlowFilesForRelationship(FetchDropbox.REL_SUCCESS);
- MockFlowFile ff0 = flowFiles.get(0);
- ff0.assertContentEquals(DEFAULT_FILE_CONTENT);
- }
-
- @Test
- void testFetchFileFromRootByPath() throws Exception {
- createFile("/", "test_file1", DEFAULT_FILE_CONTENT);
-
- Map<String, String> inputFlowFileAttributes = new HashMap<>();
- inputFlowFileAttributes.put("path", "/");
- inputFlowFileAttributes.put("filename", "test_file1");
-
- testRunner.setProperty(FetchDropbox.FILE, "${path}/${filename}");
-
- testRunner.enqueue("unimportant_data", inputFlowFileAttributes);
- testRunner.run();
-
- testRunner.assertTransferCount(FetchDropbox.REL_FAILURE, 0);
- testRunner.assertTransferCount(FetchDropbox.REL_SUCCESS, 1);
- List<MockFlowFile> flowFiles = testRunner.getFlowFilesForRelationship(FetchDropbox.REL_SUCCESS);
- MockFlowFile ff0 = flowFiles.get(0);
- ff0.assertContentEquals(DEFAULT_FILE_CONTENT);
- }
-
- @Test
- void testFetchingFolderFails() {
- Map<String, String> inputFlowFileAttributes = new HashMap<>();
- inputFlowFileAttributes.put("path", "/testFolder");
-
- testRunner.setProperty(FetchDropbox.FILE, "${path}");
-
- testRunner.enqueue("unimportant_data", inputFlowFileAttributes);
- testRunner.run();
-
- testRunner.assertTransferCount(FetchDropbox.REL_FAILURE, 1);
- testRunner.assertTransferCount(FetchDropbox.REL_SUCCESS, 0);
-
- List<MockFlowFile> flowFiles = testRunner.getFlowFilesForRelationship(FetchDropbox.REL_FAILURE);
- MockFlowFile ff0 = flowFiles.get(0);
- ff0.assertAttributeEquals("error.message", "Exception in 2/files/download: {\".tag\":\"path\",\"path\":\"not_file\"}");
- }
-
- @Test
- void testFetchingFileByWrongIdFails() {
- Map<String, String> inputFlowFileAttributes = new HashMap<>();
- inputFlowFileAttributes.put("dropbox.id", "id:missing");
-
- testRunner.enqueue("unimportant_data", inputFlowFileAttributes);
- testRunner.run();
-
- testRunner.assertTransferCount(FetchDropbox.REL_SUCCESS, 0);
- testRunner.assertTransferCount(FetchDropbox.REL_FAILURE, 1);
-
- List<MockFlowFile> flowFiles = testRunner.getFlowFilesForRelationship(FetchDropbox.REL_FAILURE);
- MockFlowFile ff0 = flowFiles.get(0);
- ff0.assertAttributeEquals("error.message", "Exception in 2/files/download: {\".tag\":\"path\",\"path\":\"not_found\"}");
- }
-}
diff --git a/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/ListDropboxIT.java b/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/ListDropboxIT.java
deleted file mode 100644
index 0d6658e886..0000000000
--- a/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/ListDropboxIT.java
+++ /dev/null
@@ -1,136 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.nifi.processors.dropbox;
-
-import static java.util.Collections.emptyList;
-import static java.util.Collections.singletonList;
-import static java.util.stream.Collectors.toList;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-import java.util.Arrays;
-import java.util.List;
-import org.apache.nifi.util.MockFlowFile;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-public class ListDropboxIT extends AbstractDropboxIT<ListDropbox> {
-
- private static final String NOT_MAIN_FOLDER = "/notMainFolder";
- private static final String YOUNG_FILE_NAME = "just_created";
-
- @BeforeEach
- public void init() throws Exception {
- super.init();
- testRunner.setProperty(ListDropbox.FOLDER, MAIN_FOLDER);
- }
-
- @AfterEach
- public void teardown() throws Exception {
- super.teardown();
- deleteFolderIfExists(NOT_MAIN_FOLDER);
- }
-
- @Override
- protected ListDropbox createTestSubject() {
- return new ListDropbox();
- }
-
- @Test
- void testEmbeddedDirectoriesAreListed() throws Exception {
- createFile(MAIN_FOLDER, "test_file1", "test_file_content1");
- createFile(MAIN_FOLDER, "test_file2", "test_file_content2");
- createFile(MAIN_FOLDER + "/testFolder1", "test_file11", "test_file_content11");
- createFile(MAIN_FOLDER + "/testFolder2", "test_file112", "test_file_content112");
-
- createFile(NOT_MAIN_FOLDER, "test_file_not_in_main_folder", "test_file_content31");
-
- List<String> expectedFileNames = Arrays.asList("test_file1", "test_file2", "test_file11", "test_file112");
-
- waitForFileCreation();
-
- testRunner.run();
-
- List<MockFlowFile> successFlowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS);
-
- List<String> actualFileNames = getFilenames(successFlowFiles);
-
- assertEquals(expectedFileNames, actualFileNames);
- }
-
- @Test
- void testFolderIsListedById() throws Exception {
- testRunner.setProperty(ListDropbox.FOLDER, mainFolderId);
-
- createFile(MAIN_FOLDER, "test_file1", "test_file_content1");
- createFile(MAIN_FOLDER + "/testFolder1", "test_file11", "test_file_content11");
-
- List<String> expectedFileNames = Arrays.asList("test_file1", "test_file11");
-
- waitForFileCreation();
-
- testRunner.run();
-
- List<MockFlowFile> successFlowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS);
-
- List<String> actualFileNames = getFilenames(successFlowFiles);
-
- assertEquals(expectedFileNames, actualFileNames);
- }
-
- @Test
- void testTooYoungFilesNotListedWhenMinAgeIsSet() throws Exception {
- testRunner.setProperty(ListDropbox.MIN_AGE, "15 s");
-
- createFile(MAIN_FOLDER, YOUNG_FILE_NAME, "test_file_content1");
-
- waitForFileCreation();
-
- testRunner.run();
-
- List<MockFlowFile> successFlowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS);
-
- List<String> actualFileNames = getFilenames(successFlowFiles);
-
- assertEquals(emptyList(), actualFileNames);
-
- // Next, wait for another 10+ seconds for MIN_AGE to expire then list again.
- Thread.sleep(10000);
-
- List<String> expectedFileNames = singletonList(YOUNG_FILE_NAME);
-
- testRunner.run();
-
- successFlowFiles = testRunner.getFlowFilesForRelationship(ListDropbox.REL_SUCCESS);
-
- actualFileNames = getFilenames(successFlowFiles);
-
- assertEquals(expectedFileNames, actualFileNames);
- }
-
- private void waitForFileCreation() throws InterruptedException {
- // We need to wait since the creation of the files are not (completely) synchronized.
- Thread.sleep(5000);
- }
-
- private List<String> getFilenames(List<MockFlowFile> flowFiles) {
- return flowFiles.stream()
- .map(flowFile -> flowFile.getAttribute("filename"))
- .collect(toList());
- }
-}
diff --git a/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/PutDropboxIT.java b/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/PutDropboxIT.java
deleted file mode 100644
index e329fd408c..0000000000
--- a/nifi-extension-bundles/nifi-dropbox-bundle/nifi-dropbox-processors/src/test/java/org/apache/nifi/processors/dropbox/PutDropboxIT.java
+++ /dev/null
@@ -1,157 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.nifi.processors.dropbox;
-
-import static org.apache.nifi.processors.conflict.resolution.ConflictResolutionStrategy.FAIL;
-import static org.apache.nifi.processors.conflict.resolution.ConflictResolutionStrategy.IGNORE;
-import static org.apache.nifi.processors.conflict.resolution.ConflictResolutionStrategy.REPLACE;
-
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-public class PutDropboxIT extends AbstractDropboxIT<PutDropbox> {
-
- private static final String CONTENT = "content";
- private static final String CHANGED_CONTENT = "changedContent";
- private static final String NON_EXISTING_FOLDER = "/doesNotExistYet";
-
- @BeforeEach
- public void init() throws Exception {
- super.init();
- testRunner.setProperty(PutDropbox.FILE_NAME, "testFile.json");
- }
-
- @AfterEach
- public void teardown() throws Exception {
- super.teardown();
- deleteFolderIfExists(NON_EXISTING_FOLDER);
- }
-
- @Override
- protected PutDropbox createTestSubject() {
- return new PutDropbox();
- }
-
- @Test
- void testUploadFileToExistingDirectory() {
- testRunner.setProperty(PutDropbox.FOLDER, MAIN_FOLDER);
-
- testRunner.enqueue(CONTENT);
- testRunner.run();
-
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 0);
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 1);
- }
-
- @Test
- void testUploadFileCreateFolderWithSubFolders() {
- testRunner.setProperty(PutDropbox.FOLDER, NON_EXISTING_FOLDER + "/subFolder1/subFolder2");
-
- testRunner.enqueue(CONTENT);
- testRunner.run();
-
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 0);
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 1);
- }
-
- @Test
- void testEmptyFileIsUpladed() {
- testRunner.setProperty(PutDropbox.FOLDER, MAIN_FOLDER);
-
- testRunner.enqueue("");
- testRunner.run();
-
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 0);
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 1);
- }
-
- @Test
- void testUploadExistingFileFailStrategy() {
- testRunner.setProperty(PutDropbox.FOLDER, MAIN_FOLDER);
- testRunner.setProperty(PutDropbox.CONFLICT_RESOLUTION, FAIL.getValue());
-
- testRunner.enqueue(CONTENT);
- testRunner.run();
-
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 1);
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 0);
- testRunner.clearTransferState();
-
- testRunner.enqueue(CHANGED_CONTENT);
- testRunner.run();
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 0);
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 1);
- }
-
- @Test
- void testUploadExistingFileWithSameContentFailStrategy() {
- testRunner.setProperty(PutDropbox.FOLDER, MAIN_FOLDER);
- testRunner.setProperty(PutDropbox.CONFLICT_RESOLUTION, FAIL.getValue());
-
- testRunner.enqueue(CONTENT);
- testRunner.run();
-
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 1);
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 0);
-
- testRunner.clearTransferState();
-
- testRunner.enqueue(CONTENT);
- testRunner.run();
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 0);
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 1);
- }
-
- @Test
- void testUploadExistingFileReplaceStrategy() {
- testRunner.setProperty(PutDropbox.FOLDER, MAIN_FOLDER);
- testRunner.setProperty(PutDropbox.CONFLICT_RESOLUTION, REPLACE.getValue());
-
- testRunner.enqueue(CONTENT);
- testRunner.run();
-
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 0);
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 1);
- testRunner.clearTransferState();
-
- testRunner.enqueue(CHANGED_CONTENT);
- testRunner.run();
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 1);
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 0);
- }
-
- @Test
- void testUploadExistingFileIgnoreStrategy() {
- testRunner.setProperty(PutDropbox.FOLDER, MAIN_FOLDER);
- testRunner.setProperty(PutDropbox.CONFLICT_RESOLUTION, IGNORE.getValue());
-
- testRunner.enqueue(CONTENT);
- testRunner.run();
-
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 0);
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 1);
- testRunner.clearTransferState();
-
- testRunner.enqueue(CHANGED_CONTENT);
- testRunner.run();
-
- testRunner.assertTransferCount(PutDropbox.REL_SUCCESS, 1);
- testRunner.assertTransferCount(PutDropbox.REL_FAILURE, 0);
- }
-}
diff --git a/nifi-extension-bundles/nifi-graph-bundle/nifi-other-graph-services/src/test/java/org/apache/nifi/graph/GremlinClientServiceControllerSettingsIT.java b/nifi-extension-bundles/nifi-graph-bundle/nifi-other-graph-services/src/test/java/org/apache/nifi/graph/GremlinClientServiceControllerSettingsIT.java
deleted file mode 100644
index 92221731e1..0000000000
--- a/nifi-extension-bundles/nifi-graph-bundle/nifi-other-graph-services/src/test/java/org/apache/nifi/graph/GremlinClientServiceControllerSettingsIT.java
+++ /dev/null
@@ -1,81 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.nifi.graph;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.nifi.util.NoOpProcessor;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.util.HashMap;
-import java.util.Map;
-import java.util.concurrent.atomic.AtomicInteger;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-/*
- * As of JanusGraph 0.3.X these tests can be a little inconsistent for a few runs at first.
- */
-public class GremlinClientServiceControllerSettingsIT {
- private TestRunner runner;
- private TestableGremlinClientService clientService;
-
-
- @BeforeEach
- public void setup() throws Exception {
- clientService = new TestableGremlinClientService();
- runner = TestRunners.newTestRunner(NoOpProcessor.class);
- runner.addControllerService("gremlinService", clientService);
- runner.setProperty(clientService, TinkerpopClientService.CONTACT_POINTS, "localhost");
- runner.setProperty(clientService, TinkerpopClientService.PORT, "8182");
- runner.enableControllerService(clientService);
- runner.assertValid();
-
- String teardown = IOUtils.toString(getClass().getResourceAsStream("/teardown.gremlin"), "UTF-8");
- clientService.getCluster().connect().submit(teardown);
- String setup = IOUtils.toString(getClass().getResourceAsStream("/setup.gremlin"), "UTF-8");
- clientService.getCluster().connect().submit(setup);
-
- assertEquals("gremlin://localhost:8182/gremlin", clientService.getTransitUrl());
- }
-
- @AfterEach
- public void tearDown() throws Exception {
- String teardown = IOUtils.toString(getClass().getResourceAsStream("/teardown.gremlin"), "UTF-8");
- clientService.getCluster().connect().submit(teardown);
- }
-
- @Test
- public void testValueMap() {
- String gremlin = "g.V().hasLabel('dog').valueMap()";
- AtomicInteger integer = new AtomicInteger();
- Map<String, String> result = clientService.executeQuery(gremlin, new HashMap<>(), (record, isMore) -> integer.incrementAndGet());
-
- assertEquals(2, integer.get());
- }
-
- @Test
- public void testCount() {
- String gremlin = "g.V().hasLabel('dog').count()";
- AtomicInteger integer = new AtomicInteger();
- Map<String, String> result = clientService.executeQuery(gremlin, new HashMap<>(), (record, isMore) -> integer.incrementAndGet());
- assertEquals(1, integer.get());
- }
-}
\ No newline at end of file
diff --git a/nifi-extension-bundles/nifi-graph-bundle/nifi-other-graph-services/src/test/java/org/apache/nifi/graph/GremlinClientServiceYamlSettingsAndBytecodeIT.java b/nifi-extension-bundles/nifi-graph-bundle/nifi-other-graph-services/src/test/java/org/apache/nifi/graph/GremlinClientServiceYamlSettingsAndBytecodeIT.java
deleted file mode 100644
index 6c6ad4e865..0000000000
--- a/nifi-extension-bundles/nifi-graph-bundle/nifi-other-graph-services/src/test/java/org/apache/nifi/graph/GremlinClientServiceYamlSettingsAndBytecodeIT.java
+++ /dev/null
@@ -1,133 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.nifi.graph;
-
-import org.apache.commons.io.IOUtils;
-import org.apache.nifi.util.NoOpProcessor;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-import org.testcontainers.junit.jupiter.Testcontainers;
-
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicLong;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-/*
- * Note: this integration test does not work out of the box
- * because nifi needs to understand/load specific serializers
- * to use when connecting via yaml. If using the yaml in the
- * resources folder, please update line 54 with the path to a jar
- * That contains org.apache.tinkerpop.gremlin.util.ser.GraphBinaryMessageSerializerV1
- * gremlin-util generally has this available
- */
-@Testcontainers
-public class GremlinClientServiceYamlSettingsAndBytecodeIT {
- private TestRunner runner;
- private TestableGremlinClientService clientService;
-
- @BeforeEach
- public void setup() throws Exception {
-
- String remoteYamlFile = "src/test/resources/gremlin.yml";
- String customJarFile = "src/test/resources/gremlin-util-3.7.0.jar";
- clientService = new TestableGremlinClientService();
- runner = TestRunners.newTestRunner(NoOpProcessor.class);
- runner.addControllerService("gremlinService", clientService);
- runner.setProperty(clientService, TinkerpopClientService.CONNECTION_SETTINGS, "yaml-settings");
- runner.setProperty(clientService, TinkerpopClientService.SUBMISSION_TYPE, "bytecode-submission");
- runner.setProperty(clientService, TinkerpopClientService.REMOTE_OBJECTS_FILE, remoteYamlFile);
- runner.setProperty(clientService, TinkerpopClientService.EXTRA_RESOURCE, customJarFile);
- /*Note: This does not seem to have much of an effect. As long as EXTRA_RESOURCE has the
- classes of interest, both the gremlin driver and ExecuteGraphQueryRecord will properly execute.
- However, something like this will be needed if we ever move away from groovy.
- */
-// runner.setProperty(clientService, TinkerpopClientService.EXTENSION_CLASSES, "org.apache.tinkerpop.gremlin.util.ser.GraphBinaryMessageSerializerV1");
- runner.enableControllerService(clientService);
- runner.assertValid();
-
- String teardown = IOUtils.toString(getClass().getResourceAsStream("/teardown.gremlin"), "UTF-8");
- clientService.getCluster().connect().submit(teardown);
- String setup = IOUtils.toString(getClass().getResourceAsStream("/setup.gremlin"), "UTF-8");
- clientService.getCluster().connect().submit(setup);
-
- }
-
- @AfterEach
- public void tearDown() throws Exception {
- String teardown = IOUtils.toString(getClass().getResourceAsStream("/teardown.gremlin"), "UTF-8");
- clientService.getCluster().connect().submit(teardown);
- }
-
- @Test
- public void testValueMap() {
- String gremlin = "[result: g.V().hasLabel('dog').valueMap().collect()]";
- AtomicInteger integer = new AtomicInteger();
- Map<String, String> result = clientService.executeQuery(gremlin, new HashMap<>(), (record, isMore) -> {
- Assertions.assertTrue(record.containsKey("result"));
- Assertions.assertTrue(record.get("result") instanceof List);
- ((List) record.get("result")).forEach(it -> {
- integer.incrementAndGet();
- });
- });
- }
-
- @Test
- public void testCount() {
- String gremlin = "[result: g.V().hasLabel('dog').count().next()]";
- AtomicLong dogCount = new AtomicLong();
- Map<String, String> result = clientService.executeQuery(gremlin, new HashMap<>(), (record, isMore) -> {
- Assertions.assertTrue(record.containsKey("result"));
- dogCount.set((Long) record.get("result"));
- });
- assertEquals(2, dogCount.get());
- }
-
- @Test
- public void testSubGraph() {
- String gremlin = "[dogInE: g.V().hasLabel('dog').inE().count().next(), dogOutE: g.V().hasLabel('dog').outE().count().next(), " +
- "dogProps: g.V().hasLabel('dog').valueMap().collect()]";
- List<Map<String, Object>> recordSet = new ArrayList<>();
- Map<String, String> result = clientService.executeQuery(gremlin, new HashMap<>(), (record, isMore) -> {
- recordSet.add(record);
- });
- Assertions.assertEquals(3, recordSet.size());
- List<Map<String, Object>> dogInE = recordSet.stream().filter(it -> it.containsKey("dogInE")).toList();
- List<Map<String, Object>> dogOutE = recordSet.stream().filter(it -> it.containsKey("dogOutE")).toList();
- List<Map<String, Object>> dogProps = recordSet.stream().filter(it -> it.containsKey("dogProps")).toList();
-
- Assertions.assertFalse(dogInE.isEmpty());
- Assertions.assertFalse(dogOutE.isEmpty());
- Assertions.assertFalse(dogProps.isEmpty());
-
- Assertions.assertEquals(2, (Long) dogOutE.get(0).get("dogOutE"));
- Assertions.assertEquals(4, (Long) dogInE.get(0).get("dogInE"));
- Assertions.assertTrue(dogProps.get(0).get("dogProps") instanceof List);
- Map<String, Object> dogPropsMap = (Map) ((List<?>) dogProps.get(0).get("dogProps")).get(0);
- Assertions.assertTrue(dogPropsMap.containsKey("name"));
- Assertions.assertTrue(dogPropsMap.containsKey("age"));
- }
-}
\ No newline at end of file
diff --git a/nifi-extension-bundles/nifi-jms-bundle/nifi-jms-processors/src/test/java/org/apache/nifi/jms/processors/JMSPublisherConsumerIT.java b/nifi-extension-bundles/nifi-jms-bundle/nifi-jms-processors/src/test/java/org/apache/nifi/jms/processors/JMSPublisherConsumerIT.java
index 3604bb26dd..6da76332bb 100644
--- a/nifi-extension-bundles/nifi-jms-bundle/nifi-jms-processors/src/test/java/org/apache/nifi/jms/processors/JMSPublisherConsumerIT.java
+++ b/nifi-extension-bundles/nifi-jms-bundle/nifi-jms-processors/src/test/java/org/apache/nifi/jms/processors/JMSPublisherConsumerIT.java
@@ -43,10 +43,8 @@ import java.nio.charset.StandardCharsets;
import java.time.Instant;
import java.util.HashMap;
import java.util.Map;
-import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicInteger;
import java.util.function.Consumer;
import static org.junit.jupiter.api.Assertions.assertArrayEquals;
@@ -363,62 +361,6 @@ public class JMSPublisherConsumerIT {
}
}
- @Test
- @Timeout(value = 20000, unit = TimeUnit.MILLISECONDS)
- public void testMultipleThreads() throws Exception {
- final int threadCount = 4;
- final int totalMessageCount = 1000;
- final int messagesPerThreadCount = totalMessageCount / threadCount;
-
- String destinationName = "testMultipleThreads";
- JmsTemplate publishTemplate = CommonTest.buildJmsTemplateForDestination(false);
- final CountDownLatch consumerTemplateCloseCount = new CountDownLatch(threadCount);
-
- try {
- JMSPublisher publisher = new JMSPublisher((CachingConnectionFactory) publishTemplate.getConnectionFactory(), publishTemplate, mock(ComponentLog.class));
- for (int i = 0; i < totalMessageCount; i++) {
- publisher.publish(destinationName, String.valueOf(i).getBytes(StandardCharsets.UTF_8));
- }
-
- final AtomicInteger msgCount = new AtomicInteger(0);
-
- final Thread[] threads = new Thread[4];
- for (int i = 0; i < 4; i++) {
- final Thread t = new Thread(() -> {
- JmsTemplate consumeTemplate = CommonTest.buildJmsTemplateForDestination(false);
-
- try {
- JMSConsumer consumer = new JMSConsumer((CachingConnectionFactory) consumeTemplate.getConnectionFactory(), consumeTemplate, mock(ComponentLog.class));
-
- for (int j = 0; j < messagesPerThreadCount && msgCount.get() < totalMessageCount; j++) {
- consumer.consumeSingleMessage(destinationName, null, false, false, null, null, "UTF-8",
- response -> msgCount.incrementAndGet());
- }
- } finally {
- ((CachingConnectionFactory) consumeTemplate.getConnectionFactory()).destroy();
- consumerTemplateCloseCount.countDown();
- }
- });
-
- threads[i] = t;
- t.start();
- }
-
- int iterations = 0;
- while (msgCount.get() < totalMessageCount) {
- Thread.sleep(10L);
- if (++iterations % 100 == 0) {
- System.out.println(msgCount.get() + " messages received so far");
- }
- }
- } finally {
- ((CachingConnectionFactory) publishTemplate.getConnectionFactory()).destroy();
-
- consumerTemplateCloseCount.await();
- }
- }
-
-
@Test
@Timeout(value = 10000, unit = TimeUnit.MILLISECONDS)
public void validateMessageRedeliveryWhenNotAcked() {
diff --git a/nifi-extension-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoIT.java b/nifi-extension-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoIT.java
index 8be78b93ac..0aba4ed7b0 100644
--- a/nifi-extension-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoIT.java
+++ b/nifi-extension-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/GetMongoIT.java
@@ -22,14 +22,11 @@ import com.fasterxml.jackson.databind.ObjectMapper;
import com.mongodb.client.MongoClient;
import com.mongodb.client.MongoClients;
import com.mongodb.client.MongoCollection;
-import com.mongodb.client.MongoDatabase;
import org.apache.nifi.components.ValidationResult;
import org.apache.nifi.flowfile.attributes.CoreAttributes;
import org.apache.nifi.mongodb.MongoDBClientService;
import org.apache.nifi.mongodb.MongoDBControllerService;
import org.apache.nifi.processor.ProcessContext;
-import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.reporting.InitializationException;
import org.apache.nifi.util.MockFlowFile;
import org.apache.nifi.util.MockProcessContext;
import org.apache.nifi.util.TestRunner;
@@ -480,79 +477,6 @@ public class GetMongoIT extends AbstractMongoIT {
* End query read behavior tests
*/
- /*
- * Verify that behavior described in NIFI-5305 actually works. This test is to ensure that
- * if a user configures the processor to use EL for the database details (name and collection) that
- * it can work against a flowfile.
- */
- @Test
- public void testDatabaseEL() throws InitializationException {
- runner.clearTransferState();
- runner.setIncomingConnection(true);
-
- String[] collections = new String[] { "a", "b", "c" };
- String[] dbs = new String[] { "el_db_1", "el_db_2", "el_db_3" };
- String query = "{}";
-
- for (int x = 0; x < collections.length; x++) {
- MongoDatabase db = mongoClient.getDatabase(dbs[x]);
- db.getCollection(collections[x])
- .insertOne(new Document().append("msg", "Hello, World"));
-
- Map<String, String> attrs = new HashMap<>();
- attrs.put("db", dbs[x]);
- attrs.put("collection", collections[x]);
- runner.enqueue(query, attrs);
- runner.run();
-
- db.drop();
-
- runner.assertTransferCount(GetMongo.REL_SUCCESS, 1);
- runner.assertTransferCount(GetMongo.REL_ORIGINAL, 1);
- runner.assertTransferCount(GetMongo.REL_FAILURE, 0);
- runner.clearTransferState();
- }
-
- Map<String, Map<String, String>> vals = new HashMap<String, Map<String, String>>(){{
- put("Collection", new HashMap<String, String>(){{
- put("db", "getmongotest");
- put("collection", "");
- }});
- put("Database", new HashMap<String, String>(){{
- put("db", "");
- put("collection", "test");
- }});
- }};
-
- TestRunner tmpRunner;
-
- for (Map.Entry<String, Map<String, String>> entry : vals.entrySet()) {
- // Creating a new runner for each set of attributes map since every subsequent runs will attempt to take the top most enqueued FlowFile
- tmpRunner = TestRunners.newTestRunner(GetMongo.class);
- MongoDBControllerService tempService = new MongoDBControllerService();
- tmpRunner.addControllerService("clientService", tempService);
- tmpRunner.setProperty(tempService, MongoDBControllerService.URI, MONGO_CONTAINER.getConnectionString());
- tmpRunner.setProperty(AbstractMongoProcessor.CLIENT_SERVICE, "clientService");
- tmpRunner.enableControllerService(tempService);
- tmpRunner.setProperty(AbstractMongoProcessor.CLIENT_SERVICE, "clientService");
- tmpRunner.setProperty(AbstractMongoProcessor.DATABASE_NAME, DB_NAME);
- tmpRunner.setProperty(AbstractMongoProcessor.COLLECTION_NAME, COLLECTION_NAME);
- tmpRunner.setIncomingConnection(true);
-
- tmpRunner.enqueue("{ }", entry.getValue());
-
- try {
- tmpRunner.run();
- } catch (Throwable ex) {
- Throwable cause = ex.getCause();
- assertTrue(cause instanceof ProcessException);
- assertTrue(ex.getMessage().contains(entry.getKey()), entry.getKey());
- }
- tmpRunner.clearTransferState();
-
- }
- }
-
@Test
public void testDBAttributes() {
runner.enqueue("{}");
diff --git a/nifi-extension-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/gridfs/PutGridFSIT.java b/nifi-extension-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/gridfs/PutGridFSIT.java
index c17e09da84..1bef78492c 100644
--- a/nifi-extension-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/gridfs/PutGridFSIT.java
+++ b/nifi-extension-bundles/nifi-mongodb-bundle/nifi-mongodb-processors/src/test/java/org/apache/nifi/processors/mongodb/gridfs/PutGridFSIT.java
@@ -120,17 +120,6 @@ public class PutGridFSIT extends GridFSITTestBase {
testUniqueness(attrs, "Hello, world", PutGridFS.UNIQUE_NAME);
}
- @Test
- public void testFileNameAndHashUniqueness() {
- testHashUniqueness(PutGridFS.UNIQUE_BOTH);
- }
-
- @Test
- public void testHashUniqueness() {
- testHashUniqueness(PutGridFS.UNIQUE_HASH);
- }
-
-
@Test
public void testChunkSize() {
String[] chunkSizes = new String[] { "128 KB", "256 KB", "384 KB", "512KB", "768KB", "1024 KB" };
diff --git a/nifi-extension-bundles/nifi-prometheus-bundle/nifi-prometheus-reporting-task/src/test/java/org/apache/nifi/reporting/prometheus/PrometheusReportingTaskIT.java b/nifi-extension-bundles/nifi-prometheus-bundle/nifi-prometheus-reporting-task/src/test/java/org/apache/nifi/reporting/prometheus/PrometheusReportingTaskIT.java
index 596493e4a4..7ac255057b 100644
--- a/nifi-extension-bundles/nifi-prometheus-bundle/nifi-prometheus-reporting-task/src/test/java/org/apache/nifi/reporting/prometheus/PrometheusReportingTaskIT.java
+++ b/nifi-extension-bundles/nifi-prometheus-bundle/nifi-prometheus-reporting-task/src/test/java/org/apache/nifi/reporting/prometheus/PrometheusReportingTaskIT.java
@@ -168,24 +168,6 @@ public class PrometheusReportingTaskIT {
return EntityUtils.toString(entity);
}
- @Test
- public void testNullLabel() throws IOException, InitializationException {
- rootGroupStatus.setName(null);
- testedReportingTask.initialize(reportingInitContextStub);
- testedReportingTask.onScheduled(configurationContextStub);
- reportingContextStub.getEventAccess().setProcessGroupStatus(rootGroupStatus);
- testedReportingTask.onTrigger(reportingContextStub);
-
- String content = getMetrics();
- assertTrue(content.contains("parent_id=\"\""));
-
- try {
- testedReportingTask.OnStopped();
- } catch(Exception e) {
- // Ignore
- }
- }
-
@Test
public void testTwoInstances() throws InitializationException {
testedReportingTask.initialize(reportingInitContextStub);
diff --git a/nifi-extension-bundles/nifi-redis-bundle/nifi-redis-extensions/src/test/java/org/apache/nifi/redis/service/ITRedisDistributedMapCacheClientService.java b/nifi-extension-bundles/nifi-redis-bundle/nifi-redis-extensions/src/test/java/org/apache/nifi/redis/service/ITRedisDistributedMapCacheClientService.java
deleted file mode 100644
index 51a40ce855..0000000000
--- a/nifi-extension-bundles/nifi-redis-bundle/nifi-redis-extensions/src/test/java/org/apache/nifi/redis/service/ITRedisDistributedMapCacheClientService.java
+++ /dev/null
@@ -1,486 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.redis.service;
-
-import org.apache.nifi.components.PropertyDescriptor;
-import org.apache.nifi.distributed.cache.client.AtomicCacheEntry;
-import org.apache.nifi.distributed.cache.client.AtomicDistributedMapCacheClient;
-import org.apache.nifi.distributed.cache.client.Deserializer;
-import org.apache.nifi.distributed.cache.client.Serializer;
-import org.apache.nifi.distributed.cache.client.exception.DeserializationException;
-import org.apache.nifi.distributed.cache.client.exception.SerializationException;
-import org.apache.nifi.flowfile.FlowFile;
-import org.apache.nifi.processor.AbstractProcessor;
-import org.apache.nifi.processor.ProcessContext;
-import org.apache.nifi.processor.ProcessSession;
-import org.apache.nifi.processor.Relationship;
-import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.processor.util.StandardValidators;
-import org.apache.nifi.redis.RedisConnectionPool;
-import org.apache.nifi.redis.testcontainers.RedisContainer;
-import org.apache.nifi.redis.testcontainers.RedisReplicaContainer;
-import org.apache.nifi.redis.testcontainers.RedisSentinelContainer;
-import org.apache.nifi.redis.util.RedisUtils;
-import org.apache.nifi.reporting.InitializationException;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.Test;
-import org.junit.jupiter.api.io.TempDir;
-import org.springframework.lang.NonNull;
-import org.springframework.lang.Nullable;
-
-import java.io.IOException;
-import java.io.OutputStream;
-import java.net.InetSocketAddress;
-import java.net.StandardSocketOptions;
-import java.nio.channels.SocketChannel;
-import java.nio.charset.StandardCharsets;
-import java.nio.file.Path;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Map;
-import java.util.Set;
-import java.util.function.Consumer;
-
-import static org.apache.nifi.redis.util.RedisUtils.REDIS_CONNECTION_POOL;
-import static org.apache.nifi.redis.util.RedisUtils.REDIS_MODE_SENTINEL;
-import static org.junit.jupiter.api.Assertions.assertArrayEquals;
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertFalse;
-import static org.junit.jupiter.api.Assertions.assertNull;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-/**
- * This is an integration test that is meant to be run against a real Redis instance.
- */
-public class ITRedisDistributedMapCacheClientService {
-
- public static final String CONTAINER_IMAGE_TAG = "redis:7.0.12-alpine";
-
- private static final String masterName = "redisLeader";
- @TempDir
-
- private Path testDirectory;
-
- private final TestRedisProcessor proc = new TestRedisProcessor();
- private final TestRunner testRunner = TestRunners.newTestRunner(proc);
-
- private final List<RedisContainer> redisContainers = new ArrayList<>();
- private RedisConnectionPoolService redisConnectionPool;
-
- @Test
- public void testStandaloneRedis() throws InitializationException, IOException {
- int redisPort = setupStandaloneRedis(null,null).port;
- setUpRedisConnectionPool(portsToConnectionString(redisPort), pool -> {
- // uncomment this to test using a different database index than the default 0
- // testRunner.setProperty(pool, RedisUtils.DATABASE, "1");
- });
- setupRedisMapCacheClientService();
-
- executeProcessor();
- }
-
- @Test
- public void testStandaloneRedisWithUsernameAndPasswordAuthentication() throws InitializationException, IOException {
- final String redisUsername = "foo";
- final String redisPassword = "foobared";
- final int redisPort = setupStandaloneRedis(redisUsername, redisPassword).port;
- setUpRedisConnectionPool(portsToConnectionString(redisPort), pool -> {
- testRunner.setProperty(redisConnectionPool, RedisUtils.USERNAME, redisUsername);
- testRunner.setProperty(redisConnectionPool, RedisUtils.PASSWORD, redisPassword);
- });
- setupRedisMapCacheClientService();
-
- executeProcessor();
- }
-
-
- @Test
- public void testStandaloneRedisWithDefaultUserAuthentication() throws InitializationException, IOException {
- final String redisPassword = "foobared";
- final int redisPort = setupStandaloneRedis(null, redisPassword).port;
- setUpRedisConnectionPool(portsToConnectionString(redisPort), pool -> {
- testRunner.setProperty(redisConnectionPool, RedisUtils.PASSWORD, redisPassword);
- });
- setupRedisMapCacheClientService();
-
- executeProcessor();
- }
-
- @Test
- public void testSentinelRedis() throws InitializationException, IOException {
- RedisContainer redisMasterContainer = setupStandaloneRedis(null,null);
- String masterHost = "127.0.0.1";
- int masterPort = redisMasterContainer.port;
- setUpRedisReplica(masterHost, masterPort, null, null);
- setUpRedisReplica(masterHost, masterPort, null, null);
-
- int sentinelAPort = setUpSentinel(masterHost, masterPort, null, null, 2, null, null).port;
- int sentinelBPort = setUpSentinel(masterHost, masterPort, null, null, 2, null, null).port;
- int sentinelCPort = setUpSentinel(masterHost, masterPort, null, null, 2, null, null).port;
-
- setUpRedisConnectionPool(portsToConnectionString(sentinelAPort, sentinelBPort, sentinelCPort), pool -> {
- testRunner.setProperty(redisConnectionPool, RedisUtils.REDIS_MODE, REDIS_MODE_SENTINEL);
- testRunner.setProperty(redisConnectionPool, RedisUtils.SENTINEL_MASTER, masterName);
- });
-
- setupRedisMapCacheClientService();
-
- executeProcessor();
- }
-
- @Test
- public void testSentinelRedisWithDefaultUserAuthentication() throws InitializationException, IOException {
- String redisPassword = "t0p_53cr35";
- String sentinelPassword = "otherPassword";
-
- RedisContainer redisMasterContainer = setupStandaloneRedis(null, redisPassword);
- String masterHost = "127.0.0.1";
- int masterPort = redisMasterContainer.port;
- setUpRedisReplica(masterHost, masterPort, null, redisPassword);
- setUpRedisReplica(masterHost, masterPort, null, redisPassword);
-
- int sentinelAPort = setUpSentinel(masterHost, masterPort, null, redisPassword, 2, null, sentinelPassword).port;
- int sentinelBPort = setUpSentinel(masterHost, masterPort, null, redisPassword, 2, null, sentinelPassword).port;
- int sentinelCPort = setUpSentinel(masterHost, masterPort, null, redisPassword, 2, null, sentinelPassword).port;
-
- setUpRedisConnectionPool(portsToConnectionString(sentinelAPort, sentinelBPort, sentinelCPort), pool -> {
- testRunner.setProperty(redisConnectionPool, RedisUtils.REDIS_MODE, REDIS_MODE_SENTINEL);
- testRunner.setProperty(redisConnectionPool, RedisUtils.SENTINEL_MASTER, masterName);
-
- testRunner.setProperty(redisConnectionPool, RedisUtils.PASSWORD, redisPassword);
- testRunner.setProperty(redisConnectionPool, RedisUtils.SENTINEL_PASSWORD, sentinelPassword);
- });
- setupRedisMapCacheClientService();
-
- executeProcessor();
- }
-
- @Test
- public void testSentinelRedisWithUsernameAndPasswordAuthentication() throws InitializationException, IOException {
- final String redisUser = "redisUser";
- final String redisPassword = "t0p_53cr35";
- final String sentinelUsername = "sentinelUser";
- final String sentinelPassword = "otherPassword";
-
- final RedisContainer redisMasterContainer = setupStandaloneRedis(redisUser, redisPassword);
- final String masterHost = "127.0.0.1";
- final int masterPort = redisMasterContainer.port;
- setUpRedisReplica(masterHost, masterPort, redisUser, redisPassword);
- setUpRedisReplica(masterHost, masterPort, redisUser, redisPassword);
-
- int sentinelAPort = setUpSentinel(masterHost, masterPort, redisUser, redisPassword, 2, sentinelUsername, sentinelPassword).port;
- int sentinelBPort = setUpSentinel(masterHost, masterPort, redisUser, redisPassword, 2, sentinelUsername, sentinelPassword).port;
- int sentinelCPort = setUpSentinel(masterHost, masterPort, redisUser, redisPassword, 2, sentinelUsername, sentinelPassword).port;
-
- setUpRedisConnectionPool(portsToConnectionString(sentinelAPort, sentinelBPort, sentinelCPort), pool -> {
- testRunner.setProperty(redisConnectionPool, RedisUtils.REDIS_MODE, REDIS_MODE_SENTINEL);
- testRunner.setProperty(redisConnectionPool, RedisUtils.SENTINEL_MASTER, masterName);
-
- testRunner.setProperty(redisConnectionPool, RedisUtils.USERNAME, redisUser);
- testRunner.setProperty(redisConnectionPool, RedisUtils.PASSWORD, redisPassword);
- testRunner.setProperty(redisConnectionPool, RedisUtils.SENTINEL_USERNAME, sentinelUsername);
- testRunner.setProperty(redisConnectionPool, RedisUtils.SENTINEL_PASSWORD, sentinelPassword);
- });
- setupRedisMapCacheClientService();
-
- executeProcessor();
- }
-
- @AfterEach
- public void teardown() {
- if (redisConnectionPool != null) {
- redisConnectionPool.onDisabled();
- }
-
- redisContainers.forEach(RedisContainer::stop);
- }
-
- private RedisContainer setupStandaloneRedis(@Nullable final String redisUsername, final @Nullable String redisPassword) throws IOException {
- int redisPort = getAvailablePort();
-
- RedisContainer redisContainer = new RedisContainer(CONTAINER_IMAGE_TAG);
- redisContainer.mountConfigurationFrom(testDirectory);
- redisContainer.setPort(redisPort);
- redisContainer.addPortBinding(redisPort, redisPort);
- redisContainer.setUsername(redisUsername);
- redisContainer.setPassword(redisPassword);
-
- redisContainers.add(redisContainer);
- redisContainer.start();
-
- return redisContainer;
- }
-
- private RedisReplicaContainer setUpRedisReplica(final @NonNull String masterHost,
- final int masterPort,
- final @Nullable String redisUsername,
- final @Nullable String redisPassword) throws IOException {
- int replicaPort = getAvailablePort();
-
- RedisReplicaContainer redisReplicaContainer = new RedisReplicaContainer(CONTAINER_IMAGE_TAG);
- redisReplicaContainer.mountConfigurationFrom(testDirectory);
- redisReplicaContainer.setPort(replicaPort);
- redisReplicaContainer.addPortBinding(replicaPort, replicaPort);
- redisReplicaContainer.setReplicaOf(masterHost, masterPort);
- redisReplicaContainer.setUsername(redisUsername);
- redisReplicaContainer.setPassword(redisPassword);
-
- redisContainers.add(redisReplicaContainer);
- redisReplicaContainer.start();
-
- return redisReplicaContainer;
- }
-
- private RedisSentinelContainer setUpSentinel(final @NonNull String masterHost,
- final int masterPort,
- final @Nullable String redisUsername,
- final @Nullable String redisPassword,
- final int quorumSize,
- final @Nullable String sentinelUsername,
- final @Nullable String sentinelPassword) throws IOException {
- int sentinelPort = getAvailablePort();
-
- RedisSentinelContainer redisSentinelContainer = new RedisSentinelContainer(CONTAINER_IMAGE_TAG);
- redisSentinelContainer.mountConfigurationFrom(testDirectory);
- redisSentinelContainer.setPort(sentinelPort);
- redisSentinelContainer.addPortBinding(sentinelPort, sentinelPort);
- redisSentinelContainer.setMasterHost(masterHost);
- redisSentinelContainer.setMasterPort(masterPort);
- redisSentinelContainer.setMasterName(masterName);
- redisSentinelContainer.setQuorumSize(quorumSize);
- redisSentinelContainer.setUsername(redisUsername);
- redisSentinelContainer.setPassword(redisPassword);
- redisSentinelContainer.setSentinelUsername(sentinelUsername);
- redisSentinelContainer.setSentinelPassword(sentinelPassword);
-
- redisContainers.add(redisSentinelContainer);
- redisSentinelContainer.start();
-
- return redisSentinelContainer;
- }
-
- private int getAvailablePort() throws IOException {
- try (SocketChannel socket = SocketChannel.open()) {
- socket.setOption(StandardSocketOptions.SO_REUSEADDR, true);
- socket.bind(new InetSocketAddress("localhost", 0));
- return socket.socket().getLocalPort();
- }
- }
-
- private String portsToConnectionString(int... ports) {
- StringBuilder stringBuilder = new StringBuilder();
-
- for (int i = 0; i < ports.length; i++) {
- if (i > 0) {
- stringBuilder.append(',');
- }
- stringBuilder.append("localhost");
- stringBuilder.append(':');
- stringBuilder.append(ports[i]);
- }
-
- return stringBuilder.toString();
- }
-
- public void setUpRedisConnectionPool(String connectionString, Consumer<RedisConnectionPool> initializer) throws InitializationException {
- // create, configure, and enable the RedisConnectionPool service
- redisConnectionPool = new RedisConnectionPoolService();
- testRunner.addControllerService("redis-connection-pool", redisConnectionPool);
- testRunner.setProperty(redisConnectionPool, RedisUtils.CONNECTION_STRING, connectionString);
-
- if (initializer != null) {
- initializer.accept(redisConnectionPool);
- }
-
- testRunner.enableControllerService(redisConnectionPool);
- }
-
- private void setupRedisMapCacheClientService() throws InitializationException {
- // create, configure, and enable the RedisDistributedMapCacheClient service
- RedisDistributedMapCacheClientService redisMapCacheClientService = new RedisDistributedMapCacheClientService();
- testRunner.addControllerService("redis-map-cache-client", redisMapCacheClientService);
- testRunner.setProperty(redisMapCacheClientService, REDIS_CONNECTION_POOL, "redis-connection-pool");
- testRunner.enableControllerService(redisMapCacheClientService);
- testRunner.setProperty(TestRedisProcessor.REDIS_MAP_CACHE, "redis-map-cache-client");
- }
-
- private void executeProcessor() {
- // queue a flow file to trigger the processor and executeProcessor it
- testRunner.enqueue("trigger");
- testRunner.run();
- testRunner.assertAllFlowFilesTransferred(TestRedisProcessor.REL_SUCCESS, 1);
- }
-
- /**
- * Test processor that exercises RedisDistributedMapCacheClient.
- */
- private static class TestRedisProcessor extends AbstractProcessor {
-
- public static final PropertyDescriptor REDIS_MAP_CACHE = new PropertyDescriptor.Builder()
- .name("redis-map-cache")
- .displayName("Redis Map Cache")
- .identifiesControllerService(AtomicDistributedMapCacheClient.class)
- .addValidator(StandardValidators.NON_EMPTY_VALIDATOR)
- .required(true)
- .build();
-
- public static final Relationship REL_SUCCESS = new Relationship.Builder().name("success").build();
- public static final Relationship REL_FAILURE = new Relationship.Builder().name("failure").build();
-
- @Override
- protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
- return Collections.singletonList(REDIS_MAP_CACHE);
- }
-
- @Override
- public Set<Relationship> getRelationships() {
- return new HashSet<>(Arrays.asList(REL_SUCCESS, REL_FAILURE));
- }
-
- @Override
- public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException {
- final FlowFile flowFile = session.get();
- if (flowFile == null) {
- return;
- }
-
- final Serializer<String> stringSerializer = new StringSerializer();
- final Deserializer<String> stringDeserializer = new StringDeserializer();
- final Deserializer<String> stringDeserializerWithoutNullCheck = new StringDeserializerWithoutNullCheck();
-
- final AtomicDistributedMapCacheClient cacheClient = context.getProperty(REDIS_MAP_CACHE).asControllerService(AtomicDistributedMapCacheClient.class);
-
- try {
- final long timestamp = System.currentTimeMillis();
- final String key = "test-redis-processor-" + timestamp;
- final String value = "the time is " + timestamp;
-
- // verify the key doesn't exist, put the key/value, then verify it exists
- assertFalse(cacheClient.containsKey(key, stringSerializer));
- cacheClient.put(key, value, stringSerializer, stringSerializer);
- assertTrue(cacheClient.containsKey(key, stringSerializer));
-
- // verify get returns the expected value we set above
- final String retrievedValue = cacheClient.get(key, stringSerializer, stringDeserializer);
- assertEquals(value, retrievedValue);
-
- // verify get returns null for a key that doesn't exist
- final String missingValue = cacheClient.get("does-not-exist", stringSerializer, stringDeserializerWithoutNullCheck);
- assertNull(missingValue);
-
- // verify remove removes the entry and contains key returns false after
- assertTrue(cacheClient.remove(key, stringSerializer));
- assertFalse(cacheClient.containsKey(key, stringSerializer));
-
- // verify putIfAbsent works the first time and returns false the second time
- assertTrue(cacheClient.putIfAbsent(key, value, stringSerializer, stringSerializer));
- assertFalse(cacheClient.putIfAbsent(key, "some other value", stringSerializer, stringSerializer));
- assertEquals(value, cacheClient.get(key, stringSerializer, stringDeserializer));
-
- // verify that getAndPutIfAbsent returns the existing value and doesn't modify it in the cache
- final String getAndPutIfAbsentResult = cacheClient.getAndPutIfAbsent(key, value, stringSerializer, stringSerializer, stringDeserializer);
- assertEquals(value, getAndPutIfAbsentResult);
- assertEquals(value, cacheClient.get(key, stringSerializer, stringDeserializer));
-
- // verify that getAndPutIfAbsent on a key that doesn't exist returns null
- final String keyThatDoesntExist = key + "_DOES_NOT_EXIST";
- assertFalse(cacheClient.containsKey(keyThatDoesntExist, stringSerializer));
- final String getAndPutIfAbsentResultWhenDoesntExist = cacheClient.getAndPutIfAbsent(keyThatDoesntExist, value, stringSerializer, stringSerializer, stringDeserializer);
- assertNull(getAndPutIfAbsentResultWhenDoesntExist);
- assertEquals(value, cacheClient.get(keyThatDoesntExist, stringSerializer, stringDeserializer));
-
- // verify atomic fetch returns the correct entry
- final AtomicCacheEntry<String, String, byte[]> entry = cacheClient.fetch(key, stringSerializer, stringDeserializer);
- assertEquals(key, entry.getKey());
- assertEquals(value, entry.getValue());
- assertArrayEquals(value.getBytes(StandardCharsets.UTF_8), entry.getRevision().orElse(null));
-
- final AtomicCacheEntry<String, String, byte[]> notLatestEntry = new AtomicCacheEntry<>(entry.getKey(), entry.getValue(), "not previous".getBytes(StandardCharsets.UTF_8));
-
- // verify atomic replace does not replace when previous value is not equal
- assertFalse(cacheClient.replace(notLatestEntry, stringSerializer, stringSerializer));
- assertEquals(value, cacheClient.get(key, stringSerializer, stringDeserializer));
-
- // verify atomic replace does replace when previous value is equal
- final String replacementValue = "this value has been replaced";
- entry.setValue(replacementValue);
- assertTrue(cacheClient.replace(entry, stringSerializer, stringSerializer));
- assertEquals(replacementValue, cacheClient.get(key, stringSerializer, stringDeserializer));
-
- // verify atomic replace does replace no value previous existed
- final String replaceKeyDoesntExist = key + "_REPLACE_DOES_NOT_EXIST";
- final AtomicCacheEntry<String, String, byte[]> entryDoesNotExist = new AtomicCacheEntry<>(replaceKeyDoesntExist, replacementValue, null);
- assertTrue(cacheClient.replace(entryDoesNotExist, stringSerializer, stringSerializer));
- assertEquals(replacementValue, cacheClient.get(replaceKeyDoesntExist, stringSerializer, stringDeserializer));
-
- final int numToDelete = 2000;
- for (int i = 0; i < numToDelete; i++) {
- cacheClient.put(key + "-" + i, value, stringSerializer, stringSerializer);
- }
-
- Map<String, String> bulk = new HashMap<>();
- bulk.put("bulk-1", "testing1");
- bulk.put("bulk-2", "testing2");
- bulk.put("bulk-3", "testing3");
- bulk.put("bulk-4", "testing4");
- bulk.put("bulk-5", "testing5");
-
- cacheClient.putAll(bulk, stringSerializer, stringSerializer);
- assertTrue(cacheClient.containsKey("bulk-1", stringSerializer));
- assertTrue(cacheClient.containsKey("bulk-2", stringSerializer));
- assertTrue(cacheClient.containsKey("bulk-3", stringSerializer));
- assertTrue(cacheClient.containsKey("bulk-4", stringSerializer));
- assertTrue(cacheClient.containsKey("bulk-5", stringSerializer));
-
- session.transfer(flowFile, REL_SUCCESS);
- } catch (final Exception e) {
- getLogger().error("Routing to failure due to: " + e.getMessage(), e);
- session.transfer(flowFile, REL_FAILURE);
- }
- }
- }
-
- private static class StringSerializer implements Serializer<String> {
- @Override
- public void serialize(String value, OutputStream output) throws SerializationException, IOException {
- if (value != null) {
- output.write(value.getBytes(StandardCharsets.UTF_8));
- }
- }
- }
-
- private static class StringDeserializer implements Deserializer<String> {
- @Override
- public String deserialize(byte[] input) throws DeserializationException {
- return input == null ? null : new String(input, StandardCharsets.UTF_8);
- }
- }
-
- private static class StringDeserializerWithoutNullCheck implements Deserializer<String> {
- @Override
- public String deserialize(byte[] input) throws DeserializationException {
- return new String(input, StandardCharsets.UTF_8);
- }
- }
-}
-
diff --git a/nifi-extension-bundles/nifi-redis-bundle/nifi-redis-extensions/src/test/java/org/apache/nifi/redis/state/ITRedisStateProvider.java b/nifi-extension-bundles/nifi-redis-bundle/nifi-redis-extensions/src/test/java/org/apache/nifi/redis/state/ITRedisStateProvider.java
index c426b2d0e9..75980340d7 100644
--- a/nifi-extension-bundles/nifi-redis-bundle/nifi-redis-extensions/src/test/java/org/apache/nifi/redis/state/ITRedisStateProvider.java
+++ b/nifi-extension-bundles/nifi-redis-bundle/nifi-redis-extensions/src/test/java/org/apache/nifi/redis/state/ITRedisStateProvider.java
@@ -32,7 +32,6 @@ import org.apache.nifi.components.state.StateMap;
import org.apache.nifi.components.state.StateProvider;
import org.apache.nifi.components.state.StateProviderInitializationContext;
import org.apache.nifi.logging.ComponentLog;
-import org.apache.nifi.redis.service.ITRedisDistributedMapCacheClientService;
import org.apache.nifi.redis.testcontainers.RedisContainer;
import org.apache.nifi.redis.util.RedisUtils;
import org.apache.nifi.util.MockComponentLog;
@@ -57,7 +56,7 @@ public class ITRedisStateProvider {
protected final String componentId = "111111111-1111-1111-1111-111111111111";
@Container
- public RedisContainer redisContainer = new RedisContainer(ITRedisDistributedMapCacheClientService.CONTAINER_IMAGE_TAG)
+ public RedisContainer redisContainer = new RedisContainer("redis:7.0.12-alpine")
.withExposedPorts(6379);
private RedisStateProvider provider;
diff --git a/nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/PutSalesforceObjectIT.java b/nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/PutSalesforceObjectIT.java
deleted file mode 100644
index 96b37d5a35..0000000000
--- a/nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/PutSalesforceObjectIT.java
+++ /dev/null
@@ -1,135 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.salesforce;
-
-import org.apache.nifi.oauth2.StandardOauth2AccessTokenProvider;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processors.salesforce.util.CommonSalesforceProperties;
-import org.apache.nifi.processors.salesforce.util.SalesforceConfigAware;
-import org.apache.nifi.provenance.ProvenanceEventType;
-import org.apache.nifi.reporting.InitializationException;
-import org.apache.nifi.serialization.record.MockRecordParser;
-import org.apache.nifi.serialization.record.RecordFieldType;
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.util.Collections;
-import java.util.List;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-import static org.junit.jupiter.api.Assertions.assertTrue;
-
-class PutSalesforceObjectIT implements SalesforceConfigAware {
-
- private TestRunner runner;
-
- @BeforeEach
- void setUp() throws Exception {
- Processor putSalesforceObject = new PutSalesforceObject() {
- @Override
- int getMaxRecordCount() {
- return 2;
- }
- };
-
- runner = TestRunners.newTestRunner(putSalesforceObject);
-
- StandardOauth2AccessTokenProvider oauth2AccessTokenProvider = initOAuth2AccessTokenProvider(runner);
- runner.setProperty(CommonSalesforceProperties.TOKEN_PROVIDER, oauth2AccessTokenProvider.getIdentifier());
- }
-
- @Test
- void testPutSalesforceObject() throws Exception {
-
- MockRecordParser reader = new MockRecordParser();
- reader.addSchemaField("name", RecordFieldType.STRING);
- reader.addSchemaField("phone", RecordFieldType.STRING);
- reader.addSchemaField("website", RecordFieldType.STRING);
- reader.addSchemaField("numberOfEmployees", RecordFieldType.STRING);
- reader.addSchemaField("industry", RecordFieldType.STRING);
-
- reader.addRecord("SampleAccount1", "111111", "www.salesforce1.com", null, "Banking");
- reader.addRecord("SampleAccount2", null, "www.salesforce2.com", "200", "Banking");
- reader.addRecord("SampleAccount3", "333333", "www.salesforce3.com", "300", "Banking");
- reader.addRecord("SampleAccount4", "444444", null, "400", "Banking");
- reader.addRecord("SampleAccount5", "555555", "www.salesforce5.com", "500", "Banking");
-
- runner.enqueue("", Collections.singletonMap("objectType", "Account"));
-
- configureProcessor(reader);
-
- runner.run();
-
- List<MockFlowFile> results = runner.getFlowFilesForRelationship(PutSalesforceObject.REL_SUCCESS);
-
- assertEquals(1, results.size());
-
- runner.assertProvenanceEvent(ProvenanceEventType.SEND);
- }
-
- @Test
- void testMissingObjectType() throws Exception {
- MockRecordParser reader = new MockRecordParser();
-
- runner.enqueue("");
-
- configureProcessor(reader);
-
- runner.run();
-
- List<MockFlowFile> results = runner.getFlowFilesForRelationship(PutSalesforceObject.REL_FAILURE);
- assertEquals(1, results.size());
- assertTrue(runner.getProvenanceEvents().isEmpty());
-
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutSalesforceObject.REL_FAILURE);
- MockFlowFile ff = flowFiles.get(0);
- ff.assertAttributeExists("error.message");
- }
-
- @Test
- void testErrorForInvalidRecordField() throws Exception {
- MockRecordParser reader = new MockRecordParser();
- reader.addSchemaField("invalidField", RecordFieldType.STRING);
- reader.addRecord("invalidField");
-
- runner.enqueue("", Collections.singletonMap("objectType", "Account"));
-
- configureProcessor(reader);
-
- runner.run();
-
- List<MockFlowFile> results = runner.getFlowFilesForRelationship(PutSalesforceObject.REL_FAILURE);
- assertEquals(1, results.size());
- assertTrue(runner.getProvenanceEvents().isEmpty());
-
- List<MockFlowFile> flowFiles = runner.getFlowFilesForRelationship(PutSalesforceObject.REL_FAILURE);
- MockFlowFile ff = flowFiles.get(0);
- ff.assertAttributeExists("error.message");
- }
-
- private void configureProcessor(final MockRecordParser reader) throws InitializationException {
- runner.addControllerService("reader", reader);
- runner.enableControllerService(reader);
-
- runner.setProperty(CommonSalesforceProperties.API_VERSION, VERSION);
- runner.setProperty(CommonSalesforceProperties.SALESFORCE_INSTANCE_URL, INSTANCE_URL);
- runner.setProperty(PutSalesforceObject.RECORD_READER_FACTORY, reader.getIdentifier());
- }
-}
diff --git a/nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/QuerySalesforceObjectIT.java b/nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/QuerySalesforceObjectIT.java
deleted file mode 100644
index 4da9c137e4..0000000000
--- a/nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/QuerySalesforceObjectIT.java
+++ /dev/null
@@ -1,99 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.salesforce;
-
-import org.apache.nifi.oauth2.StandardOauth2AccessTokenProvider;
-import org.apache.nifi.processor.Processor;
-import org.apache.nifi.processors.salesforce.util.CommonSalesforceProperties;
-import org.apache.nifi.processors.salesforce.util.SalesforceConfigAware;
-import org.apache.nifi.provenance.ProvenanceEventType;
-import org.apache.nifi.serialization.RecordSetWriterFactory;
-import org.apache.nifi.serialization.record.MockRecordWriter;
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.util.List;
-
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-/**
- * Set constants in {@link SalesforceConfigAware}
- */
-class QuerySalesforceObjectIT implements SalesforceConfigAware {
- private TestRunner runner;
-
- @BeforeEach
- void setUp() throws Exception {
- Processor querySObject = new QuerySalesforceObject();
-
- runner = TestRunners.newTestRunner(querySObject);
-
- StandardOauth2AccessTokenProvider oauth2AccessTokenProvider = initOAuth2AccessTokenProvider(runner);
- runner.setProperty(CommonSalesforceProperties.TOKEN_PROVIDER, oauth2AccessTokenProvider.getIdentifier());
- }
-
- @AfterEach
- void tearDown() {
- runner.shutdown();
- }
-
- @Test
- void retrievesAndWritesRecords() throws Exception {
- String sObjectName = "Account";
- String fieldNames = "Id,name,CreatedDate";
-
- RecordSetWriterFactory writer = new MockRecordWriter();
- runner.addControllerService("writer", writer);
- runner.enableControllerService(writer);
-
- runner.setProperty(QuerySalesforceObject.QUERY_TYPE, QuerySalesforceObject.PROPERTY_BASED_QUERY);
- runner.setProperty(QuerySalesforceObject.SOBJECT_NAME, sObjectName);
- runner.setProperty(QuerySalesforceObject.FIELD_NAMES, fieldNames);
- runner.setProperty(CommonSalesforceProperties.API_VERSION, VERSION);
- runner.setProperty(CommonSalesforceProperties.SALESFORCE_INSTANCE_URL, INSTANCE_URL);
- runner.setProperty(QuerySalesforceObject.RECORD_WRITER, writer.getIdentifier());
- runner.setProperty(QuerySalesforceObject.AGE_FIELD, "CreatedDate");
- runner.setProperty(QuerySalesforceObject.INITIAL_AGE_FILTER, "2022-01-06T08:43:24.000+0000");
-
- runner.run();
-
- List<MockFlowFile> results = runner.getFlowFilesForRelationship(QuerySalesforceObject.REL_SUCCESS);
-
- assertNotNull(results.get(0).getContent());
- runner.assertProvenanceEvent(ProvenanceEventType.RECEIVE);
- }
-
- @Test
- void runCustomQuery() {
- String customQuery = "SELECT Id, Name FROM Account";
-
- runner.setProperty(QuerySalesforceObject.QUERY_TYPE, QuerySalesforceObject.CUSTOM_QUERY);
- runner.setProperty(QuerySalesforceObject.CUSTOM_SOQL_QUERY, customQuery);
- runner.setProperty(CommonSalesforceProperties.API_VERSION, VERSION);
- runner.setProperty(CommonSalesforceProperties.SALESFORCE_INSTANCE_URL, INSTANCE_URL);
-
- runner.run();
-
- List<MockFlowFile> results = runner.getFlowFilesForRelationship(QuerySalesforceObject.REL_SUCCESS);
-
- assertNotNull(results.get(0).getContent());
- }
-}
diff --git a/nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/util/SalesforceRestServiceIT.java b/nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/util/SalesforceRestServiceIT.java
deleted file mode 100644
index abbb79282e..0000000000
--- a/nifi-extension-bundles/nifi-salesforce-bundle/nifi-salesforce-processors/src/test/java/org/apache/nifi/processors/salesforce/util/SalesforceRestServiceIT.java
+++ /dev/null
@@ -1,78 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.salesforce.util;
-
-import org.apache.nifi.oauth2.StandardOauth2AccessTokenProvider;
-import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
-import org.apache.nifi.processor.ProcessContext;
-import org.apache.nifi.processor.ProcessSessionFactory;
-import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.processors.salesforce.rest.SalesforceConfiguration;
-import org.apache.nifi.processors.salesforce.rest.SalesforceRestClient;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.AfterEach;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.io.IOException;
-import java.io.InputStream;
-
-import static org.junit.jupiter.api.Assertions.assertNotNull;
-
-/**
- * Set constants in {@link SalesforceConfigAware}
- */
-class SalesforceRestServiceIT implements SalesforceConfigAware {
- private TestRunner runner;
- private SalesforceRestClient testSubject;
-
- @BeforeEach
- void setUp() throws Exception {
- runner = TestRunners.newTestRunner(new AbstractSessionFactoryProcessor() {
- @Override
- public void onTrigger(ProcessContext context, ProcessSessionFactory sessionFactory) throws ProcessException {
- }
- });
-
- StandardOauth2AccessTokenProvider oauth2AccessTokenProvider = initOAuth2AccessTokenProvider(runner);
- SalesforceConfiguration configuration = SalesforceConfiguration.create(INSTANCE_URL, VERSION,
- () -> oauth2AccessTokenProvider.getAccessDetails().getAccessToken(), 5_000);
- testSubject = new SalesforceRestClient(configuration);
- }
-
- @AfterEach
- void tearDown() {
- runner.shutdown();
- }
-
- @Test
- void describeSObjectSucceeds() throws IOException {
- try (InputStream describeSObjectResultJson = testSubject.describeSObject("Account")) {
- assertNotNull(describeSObjectResultJson);
- }
- }
-
- @Test
- void querySucceeds() throws IOException {
- String query = "SELECT id,BillingAddress FROM Account";
-
- try (InputStream querySObjectRecordsResultJson = testSubject.query(query)) {
- assertNotNull(querySObjectRecordsResultJson);
- }
- }
-}
diff --git a/nifi-extension-bundles/nifi-snowflake-bundle/nifi-snowflake-processors/src/test/java/org/apache/nifi/processors/snowflake/SnowflakePipeIT.java b/nifi-extension-bundles/nifi-snowflake-bundle/nifi-snowflake-processors/src/test/java/org/apache/nifi/processors/snowflake/SnowflakePipeIT.java
deleted file mode 100644
index 91eed724b7..0000000000
--- a/nifi-extension-bundles/nifi-snowflake-bundle/nifi-snowflake-processors/src/test/java/org/apache/nifi/processors/snowflake/SnowflakePipeIT.java
+++ /dev/null
@@ -1,122 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.nifi.processors.snowflake;
-
-import net.snowflake.ingest.utils.StagedFileWrapper;
-import org.apache.commons.io.FileUtils;
-import org.apache.nifi.flowfile.attributes.CoreAttributes;
-import org.apache.nifi.processors.snowflake.util.SnowflakeAttributes;
-import org.apache.nifi.processors.snowflake.util.SnowflakeInternalStageType;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.Test;
-
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.Map;
-import java.util.UUID;
-
-class SnowflakePipeIT implements SnowflakeConfigAware {
-
- @Test
- void shouldPutIntoInternalStage() throws Exception {
- final PutSnowflakeInternalStage processor = new PutSnowflakeInternalStage();
-
- final TestRunner runner = TestRunners.newTestRunner(processor);
- final SnowflakeConnectionProviderService connectionProviderService = createConnectionProviderService(runner);
-
- runner.setProperty(PutSnowflakeInternalStage.SNOWFLAKE_CONNECTION_PROVIDER, connectionProviderService.getIdentifier());
- runner.setProperty(PutSnowflakeInternalStage.INTERNAL_STAGE_TYPE, SnowflakeInternalStageType.NAMED);
- runner.setProperty(PutSnowflakeInternalStage.INTERNAL_STAGE, internalStageName);
-
- final String uuid = UUID.randomUUID().toString();
- final String fileName = filePath.getFileName().toString();
-
- final Map<String, String> attributes = new HashMap<>();
- attributes.put(CoreAttributes.FILENAME.key(), fileName);
- attributes.put(CoreAttributes.PATH.key(), uuid + "/");
- runner.enqueue(filePath, attributes);
-
- runner.run();
-
- runner.assertAllFlowFilesTransferred(StartSnowflakeIngest.REL_SUCCESS);
- }
-
- @Test
- void shouldStartFileIngestion() throws Exception {
- final StartSnowflakeIngest processor = new StartSnowflakeIngest();
-
- final TestRunner runner = TestRunners.newTestRunner(processor);
- final SnowflakeConnectionProviderService connectionProviderService = createConnectionProviderService(runner);
- final SnowflakeIngestManagerProviderService ingestManagerProviderService = createIngestManagerProviderService(runner);
-
- final String uuid = UUID.randomUUID().toString();
- final String fileName = filePath.getFileName().toString();
-
- runner.setProperty(StartSnowflakeIngest.INGEST_MANAGER_PROVIDER, ingestManagerProviderService.getIdentifier());
-
- try (final SnowflakeConnectionWrapper snowflakeConnection = connectionProviderService.getSnowflakeConnection()) {
- snowflakeConnection.unwrap().uploadStream(internalStageName,
- uuid,
- FileUtils.openInputStream(filePath.toFile()),
- fileName,
- false);
- }
-
- final Map<String, String> attributes = Collections.singletonMap(SnowflakeAttributes.ATTRIBUTE_STAGED_FILE_PATH, uuid + "/" + fileName);
- runner.enqueue("", attributes);
-
- runner.run();
-
- runner.assertAllFlowFilesTransferred(StartSnowflakeIngest.REL_SUCCESS);
- }
-
- @Test
- void shouldAwaitSnowflakePipeIngestion() throws Exception {
- final GetSnowflakeIngestStatus processor = new GetSnowflakeIngestStatus();
-
- final TestRunner runner = TestRunners.newTestRunner(processor);
- final SnowflakeConnectionProviderService connectionProviderService = createConnectionProviderService(runner);
- final SnowflakeIngestManagerProviderService ingestManagerProviderService = createIngestManagerProviderService(runner);
-
- final String uuid = UUID.randomUUID().toString();
- final String fileName = filePath.getFileName().toString();
-
- runner.setProperty(GetSnowflakeIngestStatus.INGEST_MANAGER_PROVIDER, ingestManagerProviderService.getIdentifier());
-
- try (final SnowflakeConnectionWrapper snowflakeConnection = connectionProviderService.getSnowflakeConnection()) {
- snowflakeConnection.unwrap().uploadStream(internalStageName,
- uuid,
- FileUtils.openInputStream(filePath.toFile()),
- fileName,
- false);
- }
-
- final String stagedFilePathAttribute = uuid + "/" + fileName;
-
- final StagedFileWrapper stagedFile = new StagedFileWrapper(stagedFilePathAttribute);
- ingestManagerProviderService.getIngestManager().ingestFile(stagedFile, null);
-
- final Map<String, String> attributes = Collections.singletonMap(SnowflakeAttributes.ATTRIBUTE_STAGED_FILE_PATH, stagedFilePathAttribute);
- runner.enqueue("", attributes);
-
- runner.run();
-
- runner.assertAllFlowFilesTransferred(GetSnowflakeIngestStatus.REL_RETRY);
- }
-}
diff --git a/nifi-extension-bundles/nifi-snowflake-bundle/nifi-snowflake-services/src/test/java/org/apache/nifi/snowflake/service/SnowflakeConnectionPoolIT.java b/nifi-extension-bundles/nifi-snowflake-bundle/nifi-snowflake-services/src/test/java/org/apache/nifi/snowflake/service/SnowflakeConnectionPoolIT.java
deleted file mode 100644
index 7a2dd8c4f1..0000000000
--- a/nifi-extension-bundles/nifi-snowflake-bundle/nifi-snowflake-services/src/test/java/org/apache/nifi/snowflake/service/SnowflakeConnectionPoolIT.java
+++ /dev/null
@@ -1,87 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.snowflake.service;
-
-import org.apache.nifi.processor.AbstractSessionFactoryProcessor;
-import org.apache.nifi.processor.ProcessContext;
-import org.apache.nifi.processor.ProcessSessionFactory;
-import org.apache.nifi.processor.exception.ProcessException;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.sql.Connection;
-import java.sql.ResultSet;
-import java.sql.Statement;
-
-/**
- * Set the following constants:<p>
- * SNOWFLAKE_URL<p>
- * SNOWFLAKE_USER<p>
- * SNOWFLAKE_PASSWORD<p>
- * TABLE_NAME<p>
- */
-public class SnowflakeConnectionPoolIT {
- public static final String SNOWFLAKE_URL = "account.snowflakecomputing.com";
- public static final String SNOWFLAKE_USER = "???";
- public static final String SNOWFLAKE_PASSWORD = "???";
- public static final String TABLE_NAME = "test_db.public.test_table";
-
- private static final String SERVICE_ID = SnowflakeComputingConnectionPool.class.getName();
-
- private TestRunner runner;
-
- private SnowflakeComputingConnectionPool service;
-
- @BeforeEach
- void setUp() throws Exception {
- service = new SnowflakeComputingConnectionPool();
- runner = TestRunners.newTestRunner(new AbstractSessionFactoryProcessor() {
- @Override
- public void onTrigger(ProcessContext context, ProcessSessionFactory sessionFactory) throws ProcessException {
- }
- });
- runner.addControllerService(SERVICE_ID, service);
-
- runner.setProperty(service, SnowflakeComputingConnectionPool.SNOWFLAKE_URL, SNOWFLAKE_URL);
- runner.setProperty(service, SnowflakeComputingConnectionPool.SNOWFLAKE_USER, SNOWFLAKE_USER);
- runner.setProperty(service, SnowflakeComputingConnectionPool.SNOWFLAKE_PASSWORD, SNOWFLAKE_PASSWORD);
- runner.setProperty(service, "loginTimeout", "5");
- }
-
- @Test
- void testReadSnowflakeTable() throws Exception {
- runner.assertValid(service);
- runner.enableControllerService(service);
- runner.assertValid(service);
-
- try (
- final Connection connection = service.getConnection();
- final Statement st = connection.createStatement()
- ) {
- ResultSet resultSet = st.executeQuery("select * from " + TABLE_NAME);
-
- int nrOfRows = 0;
- while (resultSet.next()) {
- nrOfRows++;
- }
-
- System.out.println("Read " + nrOfRows + " records from " + TABLE_NAME);
- }
- }
-}
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/AbstractConversionIT.java b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/AbstractConversionIT.java
deleted file mode 100644
index 330bd02802..0000000000
--- a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/AbstractConversionIT.java
+++ /dev/null
@@ -1,388 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.standard;
-
-import org.apache.nifi.avro.AvroReader;
-import org.apache.nifi.avro.AvroReaderWithEmbeddedSchema;
-import org.apache.nifi.avro.AvroRecordSetWriter;
-import org.apache.nifi.csv.CSVReader;
-import org.apache.nifi.csv.CSVRecordSetWriter;
-import org.apache.nifi.json.JsonRecordSetWriter;
-import org.apache.nifi.json.JsonTreeReader;
-import org.apache.nifi.reporting.InitializationException;
-import org.apache.nifi.serialization.MalformedRecordException;
-import org.apache.nifi.serialization.RecordReader;
-import org.apache.nifi.serialization.RecordReaderFactory;
-import org.apache.nifi.serialization.RecordSetWriterFactory;
-import org.apache.nifi.serialization.record.Record;
-import org.apache.nifi.util.MockFlowFile;
-import org.apache.nifi.util.TestRunner;
-import org.apache.nifi.util.TestRunners;
-import org.apache.nifi.xml.XMLReader;
-import org.apache.nifi.xml.XMLRecordSetWriter;
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.io.ByteArrayInputStream;
-import java.io.IOException;
-import java.nio.file.Files;
-import java.nio.file.Paths;
-import java.util.ArrayList;
-import java.util.List;
-import java.util.Map;
-import java.util.Optional;
-import java.util.UUID;
-import java.util.function.Consumer;
-
-import static org.junit.jupiter.api.Assertions.assertEquals;
-
-public abstract class AbstractConversionIT {
- protected RecordReaderFactory reader;
- protected Consumer<TestRunner> inputHandler;
- protected Consumer<TestRunner> readerConfigurer;
-
- protected RecordSetWriterFactory writer;
- protected Consumer<MockFlowFile> resultHandler;
- protected Consumer<TestRunner> writerConfigurer;
-
- @BeforeEach
- public void setUp() throws Exception {
- reader = null;
- inputHandler = null;
- readerConfigurer = null;
-
- writer = null;
- resultHandler = null;
- writerConfigurer = null;
- }
-
- @Test
- public void testCsvToJson() throws Exception {
- fromCsv(csvPostfix());
- toJson(jsonPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testCsvToAvro() throws Exception {
- fromCsv(csvPostfix());
- toAvro(avroPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testCsvToAvroToCsv() throws Exception {
- fromCsv(csvPostfix());
-
- AvroRecordSetWriter writer2 = new AvroRecordSetWriter();
- AvroReader reader2 = new AvroReader();
-
- toCsv(csvPostfix());
-
- testChain(writer2, reader2);
- }
-
- @Test
- public void testCsvToXml() throws Exception {
- fromCsv(csvPostfix());
- toXml(xmlPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testJsonToCsv() throws Exception {
- fromJson(jsonPostfix());
- toCsv(csvPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testJsonToAvro() throws Exception {
- fromJson(jsonPostfix());
- toAvro(avroPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testJsonToAvroToJson() throws Exception {
- fromJson(jsonPostfix());
-
- AvroRecordSetWriter writer2 = new AvroRecordSetWriter();
- AvroReader reader2 = new AvroReader();
-
- toJson(jsonPostfix());
-
- testChain(writer2, reader2);
- }
-
- @Test
- public void testAvroToCsv() throws Exception {
- fromAvro(avroPostfix());
- toCsv(csvPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testAvroToJson() throws Exception {
- fromAvro(avroPostfix());
- toJson(jsonPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testAvroToXml() throws Exception {
- fromAvro(avroPostfix());
- toXml(xmlPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testXmlToCsv() throws Exception {
- fromXml(xmlPostfix());
- toCsv(csvPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testXmlToJson() throws Exception {
- fromXml(xmlPostfix());
- toJson(jsonPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testXmlToAvro() throws Exception {
- fromXml(xmlPostfix());
- toAvro(avroPostfix());
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-
- @Test
- public void testXmlToAvroToXml() throws Exception {
- fromXml(xmlPostfix());
-
- AvroRecordSetWriter writer2 = new AvroRecordSetWriter();
- AvroReader reader2 = new AvroReader();
-
- toXml(xmlPostfix());
-
- testChain(writer2, reader2);
- }
-
- abstract protected String csvPostfix();
-
- abstract protected String jsonPostfix();
-
- abstract protected String avroPostfix();
-
- abstract protected String xmlPostfix();
-
- protected void commonReaderConfiguration(TestRunner testRunner) {
- }
-
- protected void commonWriterConfiguration(TestRunner testRunner) {
- }
-
- protected void fromCsv(String postfix) {
- reader = new CSVReader();
- inputHandler = stringInputHandler(getContent(postfix));
-
- readerConfigurer = testRunner -> {
- commonReaderConfiguration(testRunner);
- };
- }
-
- protected void fromJson(String postfix) {
- reader = new JsonTreeReader();
- inputHandler = stringInputHandler(getContent(postfix));
-
- readerConfigurer = testRunner -> {
- commonReaderConfiguration(testRunner);
- };
- }
-
- protected void fromXml(String postfix) {
- reader = new XMLReader();
- inputHandler = stringInputHandler(getContent(postfix));
-
- readerConfigurer = testRunner -> {
- commonReaderConfiguration(testRunner);
- testRunner.setProperty(reader, XMLReader.RECORD_FORMAT, XMLReader.RECORD_ARRAY);
- };
- }
-
- protected void fromAvro(String postfix) {
- reader = new AvroReader();
- inputHandler = byteInputHandler(getByteContent(postfix));
-
- readerConfigurer = testRunner -> {
- commonReaderConfiguration(testRunner);
- };
- }
-
- protected void toCsv(String postfix) {
- writer = new CSVRecordSetWriter();
- resultHandler = stringOutputHandler(getContent(postfix));
-
- writerConfigurer = testRunner -> {
- commonWriterConfiguration(testRunner);
- };
- }
-
- protected void toJson(String postfix) {
- writer = new JsonRecordSetWriter();
- resultHandler = stringOutputHandler(getContent(postfix));
-
- writerConfigurer = testRunner -> {
- commonWriterConfiguration(testRunner);
- testRunner.setProperty(writer, "Pretty Print JSON", "true");
- };
- }
-
- protected void toXml(String postfix) {
- writer = new XMLRecordSetWriter();
- resultHandler = stringOutputHandler(getContent(postfix));
-
- writerConfigurer = testRunner -> {
- commonWriterConfiguration(testRunner);
- testRunner.setProperty(writer, "pretty_print_xml", "true");
- testRunner.setProperty(writer, "root_tag_name", "root");
- testRunner.setProperty(writer, "record_tag_name", "nifiRecord");
- };
- }
-
- protected void toAvro(String postfix) {
- writer = new AvroRecordSetWriter();
- resultHandler = mockFlowFile -> {
- try {
- List<Map<String, Object>> expected = getRecords(getByteContent(postfix));
- List<Map<String, Object>> actual = getRecords(mockFlowFile.toByteArray());
-
- assertEquals(expected, actual);
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- };
-
- writerConfigurer = testRunner -> {
- commonWriterConfiguration(testRunner);
- };
- }
-
- protected Consumer<TestRunner> stringInputHandler(String input) {
- return testRunner -> testRunner.enqueue(input);
- }
-
- protected Consumer<TestRunner> byteInputHandler(byte[] input) {
- return testRunner -> testRunner.enqueue(input);
- }
-
- protected Consumer<MockFlowFile> stringOutputHandler(String expected) {
- return mockFlowFile -> mockFlowFile.assertContentEquals(expected);
- }
-
- protected String getContent(String postfix) {
- return new String(getByteContent(postfix));
- }
-
- protected byte[] getByteContent(String postfix) {
- try {
- return Files.readAllBytes(Paths.get("src/test/resources/TestConversions/data.int_float_string." + postfix));
- } catch (IOException e) {
- throw new RuntimeException(e);
- }
- }
-
- protected List<Map<String, Object>> getRecords(byte[] avroData) throws IOException, MalformedRecordException {
- try (RecordReader reader = new AvroReaderWithEmbeddedSchema(new ByteArrayInputStream(avroData));) {
- return getRecords(reader);
- }
- }
-
- protected List<Map<String, Object>> getRecords(RecordReader reader) throws IOException, MalformedRecordException {
- List<Map<String, Object>> records = new ArrayList<>();
-
- Record record;
- while ((record = reader.nextRecord()) != null) {
- records.add(record.toMap());
- }
-
- return records;
- }
-
- protected void testChain(RecordSetWriterFactory writer2, RecordReaderFactory reader2) throws InitializationException {
- testConversion(reader, readerConfigurer, writer2, null,
- inputHandler,
- mockFlowFile -> {
- try {
- testConversion(reader2, null, writer, writerConfigurer,
- testRunner -> testRunner.enqueue(mockFlowFile),
- resultHandler
- );
- } catch (Exception e) {
- throw new RuntimeException(e);
- }
- });
- }
-
- protected <R extends RecordReaderFactory, W extends RecordSetWriterFactory> void testConversion(
- R reader,
- Consumer<TestRunner> readerConfigurer,
- W writer,
- Consumer<TestRunner> writerConfigurer,
- Consumer<TestRunner> inputHandler,
- Consumer<MockFlowFile> resultHandler
- ) throws InitializationException {
- final TestRunner runner = TestRunners.newTestRunner(ConvertRecord.class);
-
- String readerId = UUID.randomUUID().toString();
- String writerId = UUID.randomUUID().toString();
-
- runner.addControllerService(readerId, reader);
- runner.addControllerService(writerId, writer);
-
- Optional.ofNullable(readerConfigurer).ifPresent(_configurer -> _configurer.accept(runner));
- Optional.ofNullable(writerConfigurer).ifPresent(_configurer -> _configurer.accept(runner));
-
- runner.enableControllerService(reader);
- runner.enableControllerService(writer);
-
- runner.setProperty(ConvertRecord.RECORD_READER, readerId);
- runner.setProperty(ConvertRecord.RECORD_WRITER, writerId);
-
- inputHandler.accept(runner);
-
- runner.run();
-
- runner.assertAllFlowFilesTransferred(ConvertRecord.REL_SUCCESS, 1);
-
- MockFlowFile flowFile = runner.getFlowFilesForRelationship(ConvertRecord.REL_SUCCESS).get(0);
-
- resultHandler.accept(flowFile);
- }
-}
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/ConversionWithExplicitSchemaIT.java b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/ConversionWithExplicitSchemaIT.java
deleted file mode 100644
index fa7595c16f..0000000000
--- a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/ConversionWithExplicitSchemaIT.java
+++ /dev/null
@@ -1,88 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.standard;
-
-import org.apache.avro.Schema;
-import org.apache.nifi.avro.AvroReaderWithExplicitSchema;
-import org.apache.nifi.avro.AvroTypeUtil;
-import org.apache.nifi.csv.CSVUtils;
-import org.apache.nifi.schema.access.SchemaAccessUtils;
-import org.apache.nifi.serialization.MalformedRecordException;
-import org.apache.nifi.serialization.RecordReader;
-import org.apache.nifi.serialization.record.RecordSchema;
-import org.apache.nifi.util.TestRunner;
-import org.junit.jupiter.api.BeforeEach;
-
-import java.io.ByteArrayInputStream;
-import java.io.IOException;
-import java.nio.file.Files;
-import java.nio.file.Paths;
-import java.util.List;
-import java.util.Map;
-
-public class ConversionWithExplicitSchemaIT extends AbstractConversionIT {
- private String schema;
-
- @BeforeEach
- public void setUp() throws Exception {
- super.setUp();
-
- schema = new String(Files.readAllBytes(Paths.get("src/test/resources/TestConversions/explicit.schema.json")));
- }
-
- @Override
- protected String csvPostfix() {
- return "without_header.csv";
- }
-
- @Override
- protected String jsonPostfix() {
- return "json";
- }
-
- @Override
- protected String avroPostfix() {
- return "without_schema.avro";
- }
-
- @Override
- protected String xmlPostfix() {
- return "xml";
- }
-
- @Override
- protected void commonReaderConfiguration(TestRunner testRunner) {
- testRunner.setProperty(reader, SchemaAccessUtils.SCHEMA_ACCESS_STRATEGY, SchemaAccessUtils.SCHEMA_TEXT_PROPERTY);
- testRunner.setProperty(reader, SchemaAccessUtils.SCHEMA_TEXT, schema);
- }
-
- @Override
- protected void commonWriterConfiguration(TestRunner testRunner) {
- testRunner.setProperty(writer, "Schema Write Strategy", "no-schema");
- testRunner.setProperty(writer, CSVUtils.INCLUDE_HEADER_LINE, "false");
- }
-
- @Override
- protected List<Map<String, Object>> getRecords(byte[] avroData) throws IOException, MalformedRecordException {
- Schema avroSchema = new Schema.Parser().parse(schema);
- RecordSchema recordSchema = AvroTypeUtil.createSchema(avroSchema);
-
- try (RecordReader reader = new AvroReaderWithExplicitSchema(new ByteArrayInputStream(avroData), recordSchema, avroSchema);) {
- return getRecords(reader);
- }
- }
-}
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/ConversionWithSchemaInferenceIT.java b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/ConversionWithSchemaInferenceIT.java
deleted file mode 100644
index 24970de6b8..0000000000
--- a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/java/org/apache/nifi/processors/standard/ConversionWithSchemaInferenceIT.java
+++ /dev/null
@@ -1,51 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.nifi.processors.standard;
-
-public class ConversionWithSchemaInferenceIT extends AbstractConversionIT {
- @Override
- protected String csvPostfix() {
- return "with_header.csv";
- }
-
- @Override
- protected String jsonPostfix() {
- return "json";
- }
-
- @Override
- protected String avroPostfix() {
- return "with_schema.avro";
- }
-
- @Override
- protected String xmlPostfix() {
- return "xml";
- }
-
- @Override
- public void testJsonToAvro() throws Exception {
- fromJson(jsonPostfix());
-
- // JSON schema inference doesn't discern FLOAT but uses DOUBLE instead.
- // So the expected avro is a little bit different as the deserialized values also end up in
- // Long and Double objects
- toAvro("with_schema.json.to.avro");
-
- testConversion(reader, readerConfigurer, writer, writerConfigurer, inputHandler, resultHandler);
- }
-}
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.json b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.json
deleted file mode 100644
index 1971dcfd55..0000000000
--- a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.json
+++ /dev/null
@@ -1,22 +0,0 @@
-[ {
- "Id" : 1,
- "Int_Float_String" : 3
-}, {
- "Id" : 2,
- "Int_Float_String" : 3.75
-}, {
- "Id" : 3,
- "Int_Float_String" : 3.85
-}, {
- "Id" : 4,
- "Int_Float_String" : 8
-}, {
- "Id" : 5,
- "Int_Float_String" : 2.0
-}, {
- "Id" : 6,
- "Int_Float_String" : 4.0
-}, {
- "Id" : 7,
- "Int_Float_String" : "some_string"
-} ]
\ No newline at end of file
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_header.csv b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_header.csv
deleted file mode 100644
index d9972884b3..0000000000
--- a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_header.csv
+++ /dev/null
@@ -1,8 +0,0 @@
-Id,Int_Float_String
-1,3
-2,3.75
-3,3.85
-4,8
-5,2.0
-6,4.0
-7,some_string
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_schema.avro b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_schema.avro
deleted file mode 100644
index 3c18077c1e..0000000000
Binary files a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_schema.avro and /dev/null differ
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_schema.json.to.avro b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_schema.json.to.avro
deleted file mode 100644
index 0293c07e4b..0000000000
Binary files a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.with_schema.json.to.avro and /dev/null differ
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.without_header.csv b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.without_header.csv
deleted file mode 100644
index 65cf3657fa..0000000000
--- a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.without_header.csv
+++ /dev/null
@@ -1,7 +0,0 @@
-1,3
-2,3.75
-3,3.85
-4,8
-5,2.0
-6,4.0
-7,some_string
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.without_schema.avro b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.without_schema.avro
deleted file mode 100644
index c800fe6c91..0000000000
Binary files a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.without_schema.avro and /dev/null differ
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.xml b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.xml
deleted file mode 100644
index 36a9b10bdc..0000000000
--- a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/data.int_float_string.xml
+++ /dev/null
@@ -1,31 +0,0 @@
-<?xml version="1.0" ?>
-<root>
- <nifiRecord>
- <Id>1</Id>
- <Int_Float_String>3</Int_Float_String>
- </nifiRecord>
- <nifiRecord>
- <Id>2</Id>
- <Int_Float_String>3.75</Int_Float_String>
- </nifiRecord>
- <nifiRecord>
- <Id>3</Id>
- <Int_Float_String>3.85</Int_Float_String>
- </nifiRecord>
- <nifiRecord>
- <Id>4</Id>
- <Int_Float_String>8</Int_Float_String>
- </nifiRecord>
- <nifiRecord>
- <Id>5</Id>
- <Int_Float_String>2.0</Int_Float_String>
- </nifiRecord>
- <nifiRecord>
- <Id>6</Id>
- <Int_Float_String>4.0</Int_Float_String>
- </nifiRecord>
- <nifiRecord>
- <Id>7</Id>
- <Int_Float_String>some_string</Int_Float_String>
- </nifiRecord>
-</root>
diff --git a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/explicit.schema.json b/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/explicit.schema.json
deleted file mode 100644
index 5c578b06ec..0000000000
--- a/nifi-extension-bundles/nifi-standard-bundle/nifi-standard-processors/src/test/resources/TestConversions/explicit.schema.json
+++ /dev/null
@@ -1,23 +0,0 @@
-{
- "type":"record",
- "name":"nifiRecord",
- "namespace":"org.apache.nifi",
- "fields":[
- {
- "name":"Id",
- "type":[
- "null",
- "int"
- ]
- },
- {
- "name":"Int_Float_String",
- "type":[
- "int",
- "float",
- "string",
- "null"
- ]
- }
- ]
-}
\ No newline at end of file
diff --git a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java
index ef284461dd..392f3c531f 100644
--- a/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java
+++ b/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/test/java/org/apache/nifi/controller/repository/StandardProcessSessionIT.java
@@ -725,6 +725,7 @@ public class StandardProcessSessionIT {
@Test
+ @Disabled("Test should be run manually only - not for automated builds/CI env")
public void testUpdateFlowFileRepoFailsOnSessionCommit() throws IOException {
final ContentClaim contentClaim = contentRepo.create("original".getBytes());
@@ -2686,6 +2687,7 @@ public class StandardProcessSessionIT {
}
@Test
+ @Disabled("Intended for manual performance testing; should not be run in an automated environment")
public void testCloneThenWriteCountsClaimReferencesProperly() throws IOException {
final ContentClaim originalClaim = contentRepo.create(false);
try (final OutputStream out = contentRepo.write(originalClaim)) {
diff --git a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITConnectionAccessControl.java b/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITConnectionAccessControl.java
deleted file mode 100644
index dbbff74331..0000000000
--- a/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/test/java/org/apache/nifi/integration/accesscontrol/ITConnectionAccessControl.java
+++ /dev/null
@@ -1,427 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
... 5460 lines suppressed ...
(nifi) 03/05: NIFI-13216 Replaced deprecated GzipUtils methods
Posted by ex...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
exceptionfactory pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/nifi.git
commit 22109de2ad4516329ba7f7db10758157977646f7
Author: dan-s1 <ds...@gmail.com>
AuthorDate: Fri May 10 21:23:03 2024 +0000
NIFI-13216 Replaced deprecated GzipUtils methods
- Replaced isCompressedFilename and getUncompressedFilename with the API suggested alternatives
This closes #8816
Signed-off-by: David Handermann <ex...@apache.org>
---
.../c2/client/service/operation/TransferDebugOperationHandler.java | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/c2/c2-client-bundle/c2-client-service/src/main/java/org/apache/nifi/c2/client/service/operation/TransferDebugOperationHandler.java b/c2/c2-client-bundle/c2-client-service/src/main/java/org/apache/nifi/c2/client/service/operation/TransferDebugOperationHandler.java
index 98e5703bd9..f44f39246e 100644
--- a/c2/c2-client-bundle/c2-client-service/src/main/java/org/apache/nifi/c2/client/service/operation/TransferDebugOperationHandler.java
+++ b/c2/c2-client-bundle/c2-client-service/src/main/java/org/apache/nifi/c2/client/service/operation/TransferDebugOperationHandler.java
@@ -167,8 +167,8 @@ public class TransferDebugOperationHandler implements C2OperationHandler {
Path tempDirectory = createTempDirectory(operationId);
String fileName = bundleFile.getFileName().toString();
- Path preparedFile = GzipUtils.isCompressedFilename(fileName)
- ? handleGzipFile(bundleFile, Paths.get(tempDirectory.toAbsolutePath().toString(), GzipUtils.getUncompressedFilename(fileName)))
+ Path preparedFile = GzipUtils.isCompressedFileName(fileName)
+ ? handleGzipFile(bundleFile, Paths.get(tempDirectory.toAbsolutePath().toString(), GzipUtils.getUncompressedFileName(fileName)))
: handleUncompressedFile(bundleFile, Paths.get(tempDirectory.toAbsolutePath().toString(), fileName));
preparedFiles.add(preparedFile);
}