You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "balazsgerner (via GitHub)" <gi...@apache.org> on 2024/02/07 09:29:50 UTC
[PR] NIFI-12671 Added AwsFileResourceService [nifi]
balazsgerner opened a new pull request, #8368:
URL: https://github.com/apache/nifi/pull/8368
<!-- Licensed to the Apache Software Foundation (ASF) under one or more -->
<!-- contributor license agreements. See the NOTICE file distributed with -->
<!-- this work for additional information regarding copyright ownership. -->
<!-- The ASF licenses this file to You under the Apache License, Version 2.0 -->
<!-- (the "License"); you may not use this file except in compliance with -->
<!-- the License. You may obtain a copy of the License at -->
<!-- http://www.apache.org/licenses/LICENSE-2.0 -->
<!-- Unless required by applicable law or agreed to in writing, software -->
<!-- distributed under the License is distributed on an "AS IS" BASIS, -->
<!-- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -->
<!-- See the License for the specific language governing permissions and -->
<!-- limitations under the License. -->
# Summary
[NIFI-12671](https://issues.apache.org/jira/browse/NIFI-12671)
Implemented the AWS S3 specific version of FileResourceService to be able to fetch data directly from this service, by-passing the content repository.
# Tracking
Please complete the following tracking steps prior to pull request creation.
### Issue Tracking
- [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created
### Pull Request Tracking
- [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-00000`
- [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-00000`
### Pull Request Formatting
- [x] Pull Request based on current revision of the `main` branch
- [x] Pull Request refers to a feature branch with one commit containing changes
# Verification
Please indicate the verification steps performed prior to pull request creation.
### Build
- [x] Build completed using `mvn clean install -P contrib-check`
- [x] JDK 21
### Licensing
- [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html)
- [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files
### Documentation
- [x] Documentation formatting appears as expected in rendered files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "balazsgerner (via GitHub)" <gi...@apache.org>.
balazsgerner commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1489630403
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/ListS3.java:
##########
@@ -99,6 +99,9 @@
import java.util.function.Function;
import java.util.stream.Collectors;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.REGION;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.S3_REGION;
Review Comment:
Fixed, thanks for spotting that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "balazsgerner (via GitHub)" <gi...@apache.org>.
balazsgerner commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1489620421
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/util/RegionUtilV1.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.util;
+
+import com.amazonaws.regions.Region;
+import com.amazonaws.regions.Regions;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Utility class for AWS region methods. This class uses AWS SDK v1.
+ *
+ */
+public final class RegionUtilV1 {
+
+ private RegionUtilV1() {
+ }
+
+ public static final String S3_REGION_ATTRIBUTE = "s3.region" ;
+ public static final AllowableValue ATTRIBUTE_DEFINED_REGION = new AllowableValue("attribute-defined-region",
+ "Use '" + S3_REGION_ATTRIBUTE + "' Attribute",
+ "Uses '" + S3_REGION_ATTRIBUTE + "' FlowFile attribute as region.");
+
+ public static final PropertyDescriptor REGION = new PropertyDescriptor.Builder()
+ .name("Region")
+ .description("The AWS Region to connect to.")
+ .required(true)
+ .allowableValues(getAvailableRegions())
+ .defaultValue(createAllowableValue(Regions.DEFAULT_REGION).getValue())
+ .build();
+
+ public static final PropertyDescriptor S3_REGION = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(REGION)
+ .allowableValues(getAvailableS3Regions())
+ .build();
+
+ public static Region parseRegionValue(String regionValue) {
Review Comment:
Thanks, I changed the method name to `parseS3RegionValue` and made it private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "mark-bathori (via GitHub)" <gi...@apache.org>.
mark-bathori commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1482928857
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/AwsFileResourceService.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource for other components.")
+@UseCase(
+ description = "Fetch a specific file from S3. " +
+ "The service provides higher performance compared to fetch processors when the data should be moved between different storages without any transformation.",
+ configuration = """
+ "Bucket" = "${s3.bucket}"
+ "Name" = "${filename}"
+
+ The "Region" property must be set to denote the S3 region that the Bucket resides in.
+ If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_REGION}.
+
+ The "AWS Credentials Provider Service" property should specify an instance of the AWSCredentialsProviderService in order to provide credentials for accessing the bucket.
+ """
+)
+public class AwsFileResourceService extends AbstractControllerService implements FileResourceService {
Review Comment:
I was wondering that maybe `S3FileResourceService` could be a better name since the Controller Service is fetching data from S3 and also other processors are using the 'S3' naming eg.: FethcS3, ListS3, PutS3 etc. What is your opinion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "balazsgerner (via GitHub)" <gi...@apache.org>.
balazsgerner commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1482961413
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/AwsFileResourceService.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource for other components.")
+@UseCase(
+ description = "Fetch a specific file from S3. " +
+ "The service provides higher performance compared to fetch processors when the data should be moved between different storages without any transformation.",
+ configuration = """
+ "Bucket" = "${s3.bucket}"
+ "Name" = "${filename}"
+
+ The "Region" property must be set to denote the S3 region that the Bucket resides in.
+ If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_REGION}.
+
+ The "AWS Credentials Provider Service" property should specify an instance of the AWSCredentialsProviderService in order to provide credentials for accessing the bucket.
+ """
+)
+public class AwsFileResourceService extends AbstractControllerService implements FileResourceService {
Review Comment:
Yes, I think it is a better name too, I became too focused on the cloud provider name, but the only service accessed here is S3 so it makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "balazsgerner (via GitHub)" <gi...@apache.org>.
balazsgerner commented on PR #8368:
URL: https://github.com/apache/nifi/pull/8368#issuecomment-1934239454
> Thanks @balazsgerner for implementing this new feature. I've tested the `ControllerService` and it works as expected, I was able to move my data from S3 to Azure. I've left a couple of comments please see them below.
Thanks for the review @mark-bathori, I pushed a commit with the review item fixes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "turcsanyip (via GitHub)" <gi...@apache.org>.
turcsanyip commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1487691256
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/util/RegionUtilV1.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.util;
+
+import com.amazonaws.regions.Region;
+import com.amazonaws.regions.Regions;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Utility class for AWS region methods. This class uses AWS SDK v1.
+ *
+ */
+public final class RegionUtilV1 {
+
+ private RegionUtilV1() {
+ }
+
+ public static final String S3_REGION_ATTRIBUTE = "s3.region" ;
+ public static final AllowableValue ATTRIBUTE_DEFINED_REGION = new AllowableValue("attribute-defined-region",
+ "Use '" + S3_REGION_ATTRIBUTE + "' Attribute",
+ "Uses '" + S3_REGION_ATTRIBUTE + "' FlowFile attribute as region.");
+
+ public static final PropertyDescriptor REGION = new PropertyDescriptor.Builder()
+ .name("Region")
+ .description("The AWS Region to connect to.")
+ .required(true)
+ .allowableValues(getAvailableRegions())
+ .defaultValue(createAllowableValue(Regions.DEFAULT_REGION).getValue())
+ .build();
+
+ public static final PropertyDescriptor S3_REGION = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(REGION)
+ .allowableValues(getAvailableS3Regions())
+ .build();
+
+ public static Region parseRegionValue(String regionValue) {
+ if (regionValue == null) {
+ throw new ProcessException(String.format("[%s] was selected as region source but [%s] attribute does not exist", ATTRIBUTE_DEFINED_REGION, S3_REGION_ATTRIBUTE));
+ }
+
+ try {
+ return Region.getRegion(Regions.fromName(regionValue));
+ } catch (Exception e) {
+ throw new ProcessException(String.format("The [%s] attribute contains an invalid region value [%s]", S3_REGION_ATTRIBUTE, regionValue), e);
+ }
+ }
+
+ public static Region resolveRegion(final PropertyContext context, final Map<String, String> attributes) {
+ String regionValue = context.getProperty(S3_REGION).getValue();
Review Comment:
The method is S3-specific.
```suggestion
public static Region resolveS3Region(final PropertyContext context, final Map<String, String> attributes) {
String regionValue = context.getProperty(S3_REGION).getValue();
```
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/S3FileResourceService.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+import org.apache.nifi.processors.aws.util.RegionUtilV1;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource for other components.")
+@UseCase(
+ description = "Fetch a specific file from S3. " +
+ "The service provides higher performance compared to fetch processors when the data should be moved between different storages without any transformation.",
+ configuration = """
+ "Bucket" = "${s3.bucket}"
+ "Name" = "${filename}"
Review Comment:
Copy-paste error?
```suggestion
"Object Key" = "${filename}"
```
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/S3FileResourceService.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
Review Comment:
I suggest moving `S3FileResourceService` into `...aws.s3.service` sub-package in order to separate processors and services.
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/util/RegionUtilV1.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.util;
+
+import com.amazonaws.regions.Region;
+import com.amazonaws.regions.Regions;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Utility class for AWS region methods. This class uses AWS SDK v1.
+ *
+ */
+public final class RegionUtilV1 {
+
+ private RegionUtilV1() {
+ }
+
+ public static final String S3_REGION_ATTRIBUTE = "s3.region" ;
+ public static final AllowableValue ATTRIBUTE_DEFINED_REGION = new AllowableValue("attribute-defined-region",
+ "Use '" + S3_REGION_ATTRIBUTE + "' Attribute",
+ "Uses '" + S3_REGION_ATTRIBUTE + "' FlowFile attribute as region.");
+
+ public static final PropertyDescriptor REGION = new PropertyDescriptor.Builder()
+ .name("Region")
+ .description("The AWS Region to connect to.")
+ .required(true)
+ .allowableValues(getAvailableRegions())
+ .defaultValue(createAllowableValue(Regions.DEFAULT_REGION).getValue())
+ .build();
+
+ public static final PropertyDescriptor S3_REGION = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(REGION)
+ .allowableValues(getAvailableS3Regions())
+ .build();
+
+ public static Region parseRegionValue(String regionValue) {
Review Comment:
The method should be `private`. It is used in this class only. Also, the error messages are tightly related to the S3 Region attribute resolution, not a generic parse region function.
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/sqs/AbstractSQSIT.java:
##########
@@ -84,7 +85,7 @@ protected TestRunner initRunner(final Class<? extends Processor> processorClass)
TestRunner runner = TestRunners.newTestRunner(processorClass);
AuthUtils.enableAccessKey(runner, localstack.getAccessKey(), localstack.getSecretKey());
- runner.setProperty(AbstractS3Processor.S3_REGION, localstack.getRegion());
+ runner.setProperty(RegionUtilV1.S3_REGION, localstack.getRegion());
runner.setProperty(AbstractS3Processor.ENDPOINT_OVERRIDE, localstack.getEndpointOverride(LocalStackContainer.Service.SQS).toString());
Review Comment:
Not sure why `AbstractS3Processor` was used originally. The SQS processors' properties should have been referenced which come from `AbstractAwsProcessor`.
```suggestion
runner.setProperty(AbstractAwsProcessor.REGION, localstack.getRegion());
runner.setProperty(AbstractAwsProcessor.ENDPOINT_OVERRIDE, localstack.getEndpointOverride(LocalStackContainer.Service.SQS).toString());
```
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/TestListS3.java:
##########
@@ -102,7 +103,7 @@ protected AmazonS3Client createClient(final ProcessContext context, final AWSCre
@Test
public void testList() {
- runner.setProperty(ListS3.REGION, "eu-west-1");
+ runner.setProperty(RegionUtilV1.S3_REGION, "eu-west-1");
Review Comment:
```suggestion
runner.setProperty(RegionUtilV1.REGION, "eu-west-1");
```
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/util/RegionUtilV1.java:
##########
@@ -0,0 +1,97 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.util;
+
+import com.amazonaws.regions.Region;
+import com.amazonaws.regions.Regions;
+import org.apache.commons.lang3.ArrayUtils;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Utility class for AWS region methods. This class uses AWS SDK v1.
+ *
+ */
+public final class RegionUtilV1 {
+
+ private RegionUtilV1() {
+ }
+
+ public static final String S3_REGION_ATTRIBUTE = "s3.region" ;
+ public static final AllowableValue ATTRIBUTE_DEFINED_REGION = new AllowableValue("attribute-defined-region",
+ "Use '" + S3_REGION_ATTRIBUTE + "' Attribute",
+ "Uses '" + S3_REGION_ATTRIBUTE + "' FlowFile attribute as region.");
+
+ public static final PropertyDescriptor REGION = new PropertyDescriptor.Builder()
+ .name("Region")
+ .description("The AWS Region to connect to.")
+ .required(true)
+ .allowableValues(getAvailableRegions())
+ .defaultValue(createAllowableValue(Regions.DEFAULT_REGION).getValue())
+ .build();
+
+ public static final PropertyDescriptor S3_REGION = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(REGION)
+ .allowableValues(getAvailableS3Regions())
+ .build();
+
+ public static Region parseRegionValue(String regionValue) {
+ if (regionValue == null) {
+ throw new ProcessException(String.format("[%s] was selected as region source but [%s] attribute does not exist", ATTRIBUTE_DEFINED_REGION, S3_REGION_ATTRIBUTE));
+ }
+
+ try {
+ return Region.getRegion(Regions.fromName(regionValue));
+ } catch (Exception e) {
+ throw new ProcessException(String.format("The [%s] attribute contains an invalid region value [%s]", S3_REGION_ATTRIBUTE, regionValue), e);
+ }
+ }
+
+ public static Region resolveRegion(final PropertyContext context, final Map<String, String> attributes) {
+ String regionValue = context.getProperty(S3_REGION).getValue();
+
+ if (ATTRIBUTE_DEFINED_REGION.getValue().equals(regionValue)) {
+ regionValue = attributes.get(S3_REGION_ATTRIBUTE);
+ }
+
+ return parseRegionValue(regionValue);
+ }
+
+ public static AllowableValue[] getAvailableS3Regions() {
+ final AllowableValue[] availableRegions = getAvailableRegions();
+ return ArrayUtils.addAll(availableRegions, ATTRIBUTE_DEFINED_REGION);
Review Comment:
Minor:
```suggestion
return ArrayUtils.add(availableRegions, ATTRIBUTE_DEFINED_REGION);
```
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/ListS3.java:
##########
@@ -99,6 +99,9 @@
import java.util.function.Function;
import java.util.stream.Collectors;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.REGION;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.S3_REGION;
Review Comment:
I don't think `S3_REGION` is needed in the List processor. The processor has the simple `REGION` property which does not have the `Use 's3.region' Attribute` option because the List processor has no incoming FlowFile.
Appearing `S3_REGION` in the code seems to be due to improper usage of the properties in former commits.
Could you please modify those references to `REGION` and omit the import?
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/S3FileResourceService.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+import org.apache.nifi.processors.aws.util.RegionUtilV1;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource for other components.")
+@UseCase(
+ description = "Fetch a specific file from S3. " +
+ "The service provides higher performance compared to fetch processors when the data should be moved between different storages without any transformation.",
+ configuration = """
+ "Bucket" = "${s3.bucket}"
+ "Name" = "${filename}"
+
+ The "Region" property must be set to denote the S3 region that the Bucket resides in.
+ If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_REGION}.
Review Comment:
It is not clear for me what you would like to highlight here. Using `#{...}` parameters is quite a generic pattern and it could be mentioned for every properties (e.g. for Bucket too).
Do you mean the `Use 's3.region' Attribute` option? In that case I would mention its name and `${s3.region}` attribute expression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "balazsgerner (via GitHub)" <gi...@apache.org>.
balazsgerner commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1489639335
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/S3FileResourceService.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+import org.apache.nifi.processors.aws.util.RegionUtilV1;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource for other components.")
+@UseCase(
+ description = "Fetch a specific file from S3. " +
+ "The service provides higher performance compared to fetch processors when the data should be moved between different storages without any transformation.",
+ configuration = """
+ "Bucket" = "${s3.bucket}"
+ "Name" = "${filename}"
+
+ The "Region" property must be set to denote the S3 region that the Bucket resides in.
+ If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_REGION}.
Review Comment:
Actually, I was trying to copy this sentence from the `FetchS3` processor's documentation, but it seems to be a bit of an overkill for me too, so I will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "asfgit (via GitHub)" <gi...@apache.org>.
asfgit closed pull request #8368: NIFI-12671 Added AwsFileResourceService
URL: https://github.com/apache/nifi/pull/8368
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "balazsgerner (via GitHub)" <gi...@apache.org>.
balazsgerner commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1489635897
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/S3FileResourceService.java:
##########
@@ -0,0 +1,153 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+import org.apache.nifi.processors.aws.util.RegionUtilV1;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource for other components.")
+@UseCase(
+ description = "Fetch a specific file from S3. " +
+ "The service provides higher performance compared to fetch processors when the data should be moved between different storages without any transformation.",
+ configuration = """
+ "Bucket" = "${s3.bucket}"
+ "Name" = "${filename}"
Review Comment:
Yep
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "mark-bathori (via GitHub)" <gi...@apache.org>.
mark-bathori commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1484280873
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/util/RegionUtilV1.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.util;
+
+import com.amazonaws.regions.Region;
+import com.amazonaws.regions.Regions;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.s3.AbstractS3Processor.S3_REGION;
+
+/**
+ * Utility class for AWS region methods. This class uses AWS SDK v1.
+ *
+ */
+public abstract class RegionUtilV1 {
+
+ public static final String S3_REGION_ATTRIBUTE = "s3.region" ;
+ static final AllowableValue ATTRIBUTE_DEFINED_REGION = new AllowableValue("attribute-defined-region",
Review Comment:
This attribute is still duplicated, region related parts should be removed from `AbstractS3Processor` and moved here.
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/util/RegionUtilV1.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.util;
+
+import com.amazonaws.regions.Region;
+import com.amazonaws.regions.Regions;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.s3.AbstractS3Processor.S3_REGION;
+
+/**
+ * Utility class for AWS region methods. This class uses AWS SDK v1.
+ *
+ */
+public abstract class RegionUtilV1 {
Review Comment:
In my opinion it is better to declare Util classes as final class with private constructor. Abstract class provides the same restriction but it could be misleading since the abstract keyword usually means that the class is intended to be extended or subclassed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] NIFI-12671 Added AwsFileResourceService [nifi]
Posted by "mark-bathori (via GitHub)" <gi...@apache.org>.
mark-bathori commented on code in PR #8368:
URL: https://github.com/apache/nifi/pull/8368#discussion_r1482928857
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/AwsFileResourceService.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource for other components.")
+@UseCase(
+ description = "Fetch a specific file from S3. " +
+ "The service provides higher performance compared to fetch processors when the data should be moved between different storages without any transformation.",
+ configuration = """
+ "Bucket" = "${s3.bucket}"
+ "Name" = "${filename}"
+
+ The "Region" property must be set to denote the S3 region that the Bucket resides in.
+ If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_REGION}.
+
+ The "AWS Credentials Provider Service" property should specify an instance of the AWSCredentialsProviderService in order to provide credentials for accessing the bucket.
+ """
+)
+public class AwsFileResourceService extends AbstractControllerService implements FileResourceService {
Review Comment:
I was wondering that maybe S3FileResourceService could be a better name since the Controller Service is fetching data from S3 and also other processors are using the 'S3' naming eg.: FethcS3, ListS3, PutS3 etc. What is your opinion?
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-abstract-processors/src/main/java/org/apache/nifi/processors/aws/util/RegionUtilV1.java:
##########
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.util;
+
+import com.amazonaws.regions.Region;
+import com.amazonaws.regions.Regions;
+import org.apache.nifi.components.AllowableValue;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.processor.exception.ProcessException;
+
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.s3.AbstractS3Processor.S3_REGION;
+
+/**
+ * Utility class for AWS region methods. This class uses AWS SDK v1.
+ *
+ */
+public abstract class RegionUtilV1 {
Review Comment:
As I can see every method and variables in this class exists already in `AbstractS3Processor`. I think it would be a better approach to extract these into a common place and use them both in `AbstractS3Processor` and `AwsFileResourceService`.
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/AwsFileResourceService.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource for other components.")
+@UseCase(
+ description = "Fetch a specific file from S3. " +
+ "The service provides higher performance compared to fetch processors when the data should be moved between different storages without any transformation.",
+ configuration = """
+ "Bucket" = "${s3.bucket}"
+ "Name" = "${filename}"
+
+ The "Region" property must be set to denote the S3 region that the Bucket resides in.
+ If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_REGION}.
+
+ The "AWS Credentials Provider Service" property should specify an instance of the AWSCredentialsProviderService in order to provide credentials for accessing the bucket.
+ """
+)
+public class AwsFileResourceService extends AbstractControllerService implements FileResourceService {
+
+ public static final PropertyDescriptor BUCKET_WITH_DEFAULT_VALUE = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AbstractS3Processor.BUCKET_WITH_DEFAULT_VALUE)
+ .build();
+
+ public static final PropertyDescriptor KEY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AbstractS3Processor.KEY)
+ .build();
+
+ public static final PropertyDescriptor S3_REGION = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AbstractS3Processor.S3_REGION)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = List.of(
+ BUCKET_WITH_DEFAULT_VALUE,
+ KEY,
+ S3_REGION,
+ AWS_CREDENTIALS_PROVIDER_SERVICE);
+
+ private final Cache<Region, AmazonS3> clientCache = Caffeine.newBuilder().build();
+
+ private volatile PropertyContext context;
+
+ @Override
+ protected List<PropertyDescriptor> getSupportedPropertyDescriptors() {
+ return PROPERTIES;
+ }
+
+ @OnEnabled
+ public void onEnabled(final ConfigurationContext context) {
+ this.context = context;
+ }
+
+ @OnDisabled
+ public void onDisabled() {
+ this.context = null;
+ }
+
+ @Override
+ public FileResource getFileResource(Map<String, String> attributes) {
+ final AWSCredentialsProviderService awsCredentialsProviderService = context.getProperty(AWS_CREDENTIALS_PROVIDER_SERVICE)
+ .asControllerService(AWSCredentialsProviderService.class);
+ final AmazonS3 client = getS3Client(attributes, awsCredentialsProviderService.getCredentialsProvider());
+
+ try {
+ return fetchObject(client, attributes);
+ } catch (final ProcessException | SdkClientException e) {
+ throw new ProcessException("Failed to fetch s3 object", e);
+ }
+ }
+
+ /**
+ * Fetches s3 object from the provided bucket and returns it as FileResource
+ *
+ * @param client amazon s3 client
+ * @param attributes configuration attributes
+ * @return fetched s3 object as FileResource
+ * @throws ProcessException if the object 'bucketName/key' does not exist
+ */
+ private FileResource fetchObject(final AmazonS3 client, final Map<String, String> attributes) throws ProcessException,
+ SdkClientException {
+ final String bucketName = context.getProperty(BUCKET_WITH_DEFAULT_VALUE).evaluateAttributeExpressions(attributes).getValue();
+ final String key = context.getProperty(KEY).evaluateAttributeExpressions(attributes).getValue();
+
+ if (isBlank(bucketName) || isBlank(key)) {
+ throw new ProcessException("Bucket name or key value is missing");
+ }
+
+ if (!client.doesObjectExist(bucketName, key)) {
+ throw new ProcessException(String.format("Object '%s/%s' does not exist in s3", bucketName, key));
+ }
+
+ final S3Object object = client.getObject(bucketName, key);
+ return new FileResource(object.getObjectContent(), object.getObjectMetadata().getContentLength());
+ }
+
+ protected AmazonS3 getS3Client(Map<String, String> attributes, AWSCredentialsProvider credentialsProvider) {
+ Region region = resolveRegion(context, attributes);
Review Comment:
This can be `final`.
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/test/java/org/apache/nifi/processors/aws/s3/AwsFileResourceServiceTest.java:
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.model.ObjectMetadata;
+import com.amazonaws.services.s3.model.S3Object;
+import com.amazonaws.services.s3.model.S3ObjectInputStream;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+import org.apache.nifi.reporting.InitializationException;
+import org.apache.nifi.util.NoOpProcessor;
+import org.apache.nifi.util.TestRunner;
+import org.apache.nifi.util.TestRunners;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.ExtendWith;
+import org.mockito.InjectMocks;
+import org.mockito.Mock;
+import org.mockito.junit.jupiter.MockitoExtension;
+
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.verifyNoInteractions;
+import static org.mockito.Mockito.verifyNoMoreInteractions;
+import static org.mockito.Mockito.when;
+
+@ExtendWith(MockitoExtension.class)
+class AwsFileResourceServiceTest {
+
+ private static final String TEST_NAME = AwsFileResourceServiceTest.class.getSimpleName();
Review Comment:
I think this should be renamed since it is used as a` Service Identifier` and not as as name of the test, probably it could be added directly since this variable is only used at one place.
##########
nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors/src/main/java/org/apache/nifi/processors/aws/s3/AwsFileResourceService.java:
##########
@@ -0,0 +1,149 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.processors.aws.s3;
+
+import com.amazonaws.SdkClientException;
+import com.amazonaws.auth.AWSCredentialsProvider;
+import com.amazonaws.regions.Region;
+import com.amazonaws.services.s3.AmazonS3;
+import com.amazonaws.services.s3.AmazonS3Client;
+import com.amazonaws.services.s3.model.S3Object;
+import com.github.benmanes.caffeine.cache.Cache;
+import com.github.benmanes.caffeine.cache.Caffeine;
+import org.apache.nifi.annotation.documentation.CapabilityDescription;
+import org.apache.nifi.annotation.documentation.SeeAlso;
+import org.apache.nifi.annotation.documentation.Tags;
+import org.apache.nifi.annotation.documentation.UseCase;
+import org.apache.nifi.annotation.lifecycle.OnDisabled;
+import org.apache.nifi.annotation.lifecycle.OnEnabled;
+import org.apache.nifi.components.PropertyDescriptor;
+import org.apache.nifi.context.PropertyContext;
+import org.apache.nifi.controller.AbstractControllerService;
+import org.apache.nifi.controller.ConfigurationContext;
+import org.apache.nifi.fileresource.service.api.FileResource;
+import org.apache.nifi.fileresource.service.api.FileResourceService;
+import org.apache.nifi.processor.exception.ProcessException;
+import org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderService;
+
+import java.util.List;
+import java.util.Map;
+
+import static org.apache.nifi.processors.aws.AbstractAWSCredentialsProviderProcessor.AWS_CREDENTIALS_PROVIDER_SERVICE;
+import static org.apache.nifi.processors.aws.util.RegionUtilV1.resolveRegion;
+import static org.apache.nifi.util.StringUtils.isBlank;
+
+@Tags({"Amazon", "S3", "AWS", "file", "resource"})
+@SeeAlso({FetchS3Object.class})
+@CapabilityDescription("Provides an Amazon Web Services (AWS) S3 file resource for other components.")
+@UseCase(
+ description = "Fetch a specific file from S3. " +
+ "The service provides higher performance compared to fetch processors when the data should be moved between different storages without any transformation.",
+ configuration = """
+ "Bucket" = "${s3.bucket}"
+ "Name" = "${filename}"
+
+ The "Region" property must be set to denote the S3 region that the Bucket resides in.
+ If the flow being built is to be reused elsewhere, it's a good idea to parameterize this property by setting it to something like #{S3_REGION}.
+
+ The "AWS Credentials Provider Service" property should specify an instance of the AWSCredentialsProviderService in order to provide credentials for accessing the bucket.
+ """
+)
+public class AwsFileResourceService extends AbstractControllerService implements FileResourceService {
+
+ public static final PropertyDescriptor BUCKET_WITH_DEFAULT_VALUE = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AbstractS3Processor.BUCKET_WITH_DEFAULT_VALUE)
+ .build();
+
+ public static final PropertyDescriptor KEY = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AbstractS3Processor.KEY)
+ .build();
+
+ public static final PropertyDescriptor S3_REGION = new PropertyDescriptor.Builder()
+ .fromPropertyDescriptor(AbstractS3Processor.S3_REGION)
+ .build();
+
+ private static final List<PropertyDescriptor> PROPERTIES = List.of(
+ BUCKET_WITH_DEFAULT_VALUE,
+ KEY,
+ S3_REGION,
+ AWS_CREDENTIALS_PROVIDER_SERVICE);
+
+ private final Cache<Region, AmazonS3> clientCache = Caffeine.newBuilder().build();
Review Comment:
I think the cache should be emptied when the Controller Service is disabled, so it won't store any unnecessary reference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@nifi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org