additional credential needed in S3 connection

[Glenda Ramos <gl...@hawaii.edu>]

Hi,



We are currently testing Apache NiFi 1.23.0 specifically the "PutS3Object"
processor to automatically transfer data from remote site to an "internal"
S3 bucket. The standard AWSCLI commands work and able to connect the bucket
because we are able to pass in additional credentials like the
AWS_CA_BUNDLE environment variable. That being said, we need AWS_CA_BUNDLE
property added in order to connect successfully to our S3 bucket using
Apache NiFi's "PutS3Object". I have tried passing this as a Parameter and
made it available to the flow but it's not being used by the "PutS3Object"
processor to connect. Is there a way we can add an additional property for
the AWS credentials/config file? It looks like it's only looking for
"accessKey" and "secretKey".



Any help would be greatly appreciated.



Respectfully,

Glenda Ramos

Application Engineer

Re: additional credential needed in S3 connection

[Shane Ardell <sh...@gmail.com>]

Hi Glenda,

Looking at the documentation, an approach you could take is to create an
AWSCredentialsProviderControllerService[1] as your AWS Credentials Provider
Service property value in the PutS3Object processor configuration
properties. You can assign a value to the access key and secret key
properties of the controller service. I have not done this myself, so
hopefully others can chime in to verify if this is the correct approach or
not. Hope that helps.

Best,
Shane

[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.23.0/org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService/index.html

On Tue, Feb 13, 2024 at 3:59 PM Glenda Ramos <gl...@hawaii.edu> wrote:

> Hi,
>
>
>
> We are currently testing Apache NiFi 1.23.0 specifically the "PutS3Object"
> processor to automatically transfer data from remote site to an "internal"
> S3 bucket. The standard AWSCLI commands work and able to connect the bucket
> because we are able to pass in additional credentials like the
> AWS_CA_BUNDLE environment variable. That being said, we need AWS_CA_BUNDLE
> property added in order to connect successfully to our S3 bucket using
> Apache NiFi's "PutS3Object". I have tried passing this as a Parameter and
> made it available to the flow but it's not being used by the "PutS3Object"
> processor to connect. Is there a way we can add an additional property for
> the AWS credentials/config file? It looks like it's only looking for
> "accessKey" and "secretKey".
>
>
>
> Any help would be greatly appreciated.
>
>
>
> Respectfully,
>
> Glenda Ramos
>
> Application Engineer
>

Re: additional credential needed in S3 connection

[Peter Turcsanyi <tu...@apache.org>]

Hi Glenda,

Based on the AWS documentation, AWS_CA_BUNDLE is not a client credential
but it references the SSL certificate of the AWS service.
*AWS_CA_BUNDLE: Specifies the path to a certificate bundle to use for HTTPS
certificate validation.*

If I understand your use case correctly, you are trying to connect to your
"internal" bucket which is hosted by a custom S3 compatible service. Not
the public AWS service with the public certificate. That's why you need to
specify a custom certificate for AWS CLI via the AWS_CA_BUNDLE environment
variable.
If this is the case, you can configure it using the SSL Context Service
property of PutS3Object. You will need to create a keystore with the
certificate first, and set it in the Truststore* properties of SSL Context
Service.

Regards,
Peter Turcsanyi

On Tue, Feb 13, 2024 at 9:57 PM Glenda Ramos <gl...@hawaii.edu> wrote:

> Hi,
>
>
>
> We are currently testing Apache NiFi 1.23.0 specifically the "PutS3Object"
> processor to automatically transfer data from remote site to an "internal"
> S3 bucket. The standard AWSCLI commands work and able to connect the bucket
> because we are able to pass in additional credentials like the
> AWS_CA_BUNDLE environment variable. That being said, we need AWS_CA_BUNDLE
> property added in order to connect successfully to our S3 bucket using
> Apache NiFi's "PutS3Object". I have tried passing this as a Parameter and
> made it available to the flow but it's not being used by the "PutS3Object"
> processor to connect. Is there a way we can add an additional property for
> the AWS credentials/config file? It looks like it's only looking for
> "accessKey" and "secretKey".
>
>
>
> Any help would be greatly appreciated.
>
>
>
> Respectfully,
>
> Glenda Ramos
>
> Application Engineer
>