You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jennifer Kissinger <je...@semanticbits.com> on 2017/11/17 15:10:10 UTC
Using expression language properties in AWSCredentialsProviderControllerService
Good morning,
I'm developing a pipeline that uses the AWSCredentialsProviderControllerService
to establish AWS s3 credentials that include a role ARN for cross-account
access. When I hard-code the values of Access Key and Secret Key, I can
successfully connect. When I use expression language in those fields to
reference custom Nifi properties (i.e. ${my.custom.access_key}), the
connection fails. I've confirmed that these custom properties work when
used directly on a processor like FetchS3Object, ListS3, etc.
I believe that the Access Key and Secret Key fields in the AWS controller
service do not actually evaluate expression language, contrary to the
documentation. However I would welcome any suggestions of possible user
error.
I am using Nifi 1.3.0 locally but will need to deploy this pipeline to Nifi
1.2.0. The error received when using the properties looks like this:
com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
The security token included in the request is invalid. (Service:
AWSSecurityTokenService; Status Code: 403; Error Code:
InvalidClientTokenId; Request ID: ...)
Thanks for your assistance,
~Jenni
Re: Using expression language properties in AWSCredentialsProviderControllerService
Posted by Jennifer Kissinger <je...@semanticbits.com>.
Hi James,
Hmm, looks like you are correct, AWS documentation states the external ID
is not meant to be a secret. Not sure I understand that as they mention the
external ID provides an additional check in case someone figures out your
role arn, but ok.
Either way, the request for expression language on those properties
remains, so we can point them to a custom property with different values
for different environments.
Thanks,
~Jenni
--
Jennifer Kissinger
Senior Data Engineer
SemanticBits, LLC
jennifer.kissinger@semanticbits.com
603-290-1711
On Tue, Nov 21, 2017 at 12:40 AM, James Wing <jv...@gmail.com> wrote:
> I have not understood the Assume Role External ID to be a sensitive
> property. I have most often seen it used to hold a customer ID or
> something similar. Can you add some details of how you are using it and
> why it is sensitive?
>
> Thanks,
>
> James
>
> On Mon, Nov 20, 2017 at 7:05 AM, Jennifer Kissinger <jennifer.kissinger@
> semanticbits.com> wrote:
>
>> Good morning James,
>>
>> Thanks for the reply! I was going crazy trying to figure out why it
>> wasn't working--glad for the confirmation that it's a bug.
>>
>> Yes, we are able to work around, though I'll keep an eye on the Jira
>> issue. It would also be ideal for other fields, particularly "Assume Role
>> External ID", to accept expression language as well. My understanding is
>> that the AWS external id should be treated as sensitive.
>>
>> ~Jenni
>>
>> --
>> Jennifer Kissinger
>> Senior Data Engineer
>> SemanticBits, LLC
>> jennifer.kissinger@semanticbits.com
>> 603-290-1711 <(603)%20290-1711>
>>
>> On Sun, Nov 19, 2017 at 1:18 PM, James Wing <jv...@gmail.com> wrote:
>>
>>> Jenni,
>>>
>>> Thanks for reporting this. I believe you are correct that expression
>>> language is not being applied as expected. There is now a JIRA for this
>>> issue, https://issues.apache.org/jira/browse/NIFI-4619.
>>>
>>> Have you been able to work around the issue? Hopefully, file
>>> credentials or named profiles will work for your deployments.
>>>
>>> Thanks,
>>>
>>> James
>>>
>>>
>>> On Fri, Nov 17, 2017 at 7:10 AM, Jennifer Kissinger <
>>> jennifer.kissinger@semanticbits.com> wrote:
>>>
>>>> Good morning,
>>>>
>>>> I'm developing a pipeline that uses the AWSCredentialsProviderControllerService
>>>> to establish AWS s3 credentials that include a role ARN for cross-account
>>>> access. When I hard-code the values of Access Key and Secret Key, I can
>>>> successfully connect. When I use expression language in those fields to
>>>> reference custom Nifi properties (i.e. ${my.custom.access_key}), the
>>>> connection fails. I've confirmed that these custom properties work when
>>>> used directly on a processor like FetchS3Object, ListS3, etc.
>>>>
>>>> I believe that the Access Key and Secret Key fields in the AWS
>>>> controller service do not actually evaluate expression language, contrary
>>>> to the documentation. However I would welcome any suggestions of possible
>>>> user error.
>>>>
>>>> I am using Nifi 1.3.0 locally but will need to deploy this pipeline to
>>>> Nifi 1.2.0. The error received when using the properties looks like this:
>>>>
>>>> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
>>>> The security token included in the request is invalid. (Service:
>>>> AWSSecurityTokenService; Status Code: 403; Error Code:
>>>> InvalidClientTokenId; Request ID: ...)
>>>>
>>>> Thanks for your assistance,
>>>>
>>>> ~Jenni
>>>>
>>>
>>>
>>
>
Re: Using expression language properties in AWSCredentialsProviderControllerService
Posted by James Wing <jv...@gmail.com>.
I have not understood the Assume Role External ID to be a sensitive
property. I have most often seen it used to hold a customer ID or
something similar. Can you add some details of how you are using it and
why it is sensitive?
Thanks,
James
On Mon, Nov 20, 2017 at 7:05 AM, Jennifer Kissinger <
jennifer.kissinger@semanticbits.com> wrote:
> Good morning James,
>
> Thanks for the reply! I was going crazy trying to figure out why it wasn't
> working--glad for the confirmation that it's a bug.
>
> Yes, we are able to work around, though I'll keep an eye on the Jira
> issue. It would also be ideal for other fields, particularly "Assume Role
> External ID", to accept expression language as well. My understanding is
> that the AWS external id should be treated as sensitive.
>
> ~Jenni
>
> --
> Jennifer Kissinger
> Senior Data Engineer
> SemanticBits, LLC
> jennifer.kissinger@semanticbits.com
> 603-290-1711
>
> On Sun, Nov 19, 2017 at 1:18 PM, James Wing <jv...@gmail.com> wrote:
>
>> Jenni,
>>
>> Thanks for reporting this. I believe you are correct that expression
>> language is not being applied as expected. There is now a JIRA for this
>> issue, https://issues.apache.org/jira/browse/NIFI-4619.
>>
>> Have you been able to work around the issue? Hopefully, file credentials
>> or named profiles will work for your deployments.
>>
>> Thanks,
>>
>> James
>>
>>
>> On Fri, Nov 17, 2017 at 7:10 AM, Jennifer Kissinger <
>> jennifer.kissinger@semanticbits.com> wrote:
>>
>>> Good morning,
>>>
>>> I'm developing a pipeline that uses the AWSCredentialsProviderControllerService
>>> to establish AWS s3 credentials that include a role ARN for cross-account
>>> access. When I hard-code the values of Access Key and Secret Key, I can
>>> successfully connect. When I use expression language in those fields to
>>> reference custom Nifi properties (i.e. ${my.custom.access_key}), the
>>> connection fails. I've confirmed that these custom properties work when
>>> used directly on a processor like FetchS3Object, ListS3, etc.
>>>
>>> I believe that the Access Key and Secret Key fields in the AWS
>>> controller service do not actually evaluate expression language, contrary
>>> to the documentation. However I would welcome any suggestions of possible
>>> user error.
>>>
>>> I am using Nifi 1.3.0 locally but will need to deploy this pipeline to
>>> Nifi 1.2.0. The error received when using the properties looks like this:
>>>
>>> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
>>> The security token included in the request is invalid. (Service:
>>> AWSSecurityTokenService; Status Code: 403; Error Code:
>>> InvalidClientTokenId; Request ID: ...)
>>>
>>> Thanks for your assistance,
>>>
>>> ~Jenni
>>>
>>
>>
>
Re: Using expression language properties in AWSCredentialsProviderControllerService
Posted by Jennifer Kissinger <je...@semanticbits.com>.
Good morning James,
Thanks for the reply! I was going crazy trying to figure out why it wasn't
working--glad for the confirmation that it's a bug.
Yes, we are able to work around, though I'll keep an eye on the Jira issue.
It would also be ideal for other fields, particularly "Assume Role External
ID", to accept expression language as well. My understanding is that the
AWS external id should be treated as sensitive.
~Jenni
--
Jennifer Kissinger
Senior Data Engineer
SemanticBits, LLC
jennifer.kissinger@semanticbits.com
603-290-1711
On Sun, Nov 19, 2017 at 1:18 PM, James Wing <jv...@gmail.com> wrote:
> Jenni,
>
> Thanks for reporting this. I believe you are correct that expression
> language is not being applied as expected. There is now a JIRA for this
> issue, https://issues.apache.org/jira/browse/NIFI-4619.
>
> Have you been able to work around the issue? Hopefully, file credentials
> or named profiles will work for your deployments.
>
> Thanks,
>
> James
>
>
> On Fri, Nov 17, 2017 at 7:10 AM, Jennifer Kissinger <jennifer.kissinger@
> semanticbits.com> wrote:
>
>> Good morning,
>>
>> I'm developing a pipeline that uses the AWSCredentialsProviderControllerService
>> to establish AWS s3 credentials that include a role ARN for cross-account
>> access. When I hard-code the values of Access Key and Secret Key, I can
>> successfully connect. When I use expression language in those fields to
>> reference custom Nifi properties (i.e. ${my.custom.access_key}), the
>> connection fails. I've confirmed that these custom properties work when
>> used directly on a processor like FetchS3Object, ListS3, etc.
>>
>> I believe that the Access Key and Secret Key fields in the AWS controller
>> service do not actually evaluate expression language, contrary to the
>> documentation. However I would welcome any suggestions of possible user
>> error.
>>
>> I am using Nifi 1.3.0 locally but will need to deploy this pipeline to
>> Nifi 1.2.0. The error received when using the properties looks like this:
>>
>> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
>> The security token included in the request is invalid. (Service:
>> AWSSecurityTokenService; Status Code: 403; Error Code:
>> InvalidClientTokenId; Request ID: ...)
>>
>> Thanks for your assistance,
>>
>> ~Jenni
>>
>
>
Re: Using expression language properties in AWSCredentialsProviderControllerService
Posted by James Wing <jv...@gmail.com>.
Jenni,
Thanks for reporting this. I believe you are correct that expression
language is not being applied as expected. There is now a JIRA for this
issue, https://issues.apache.org/jira/browse/NIFI-4619.
Have you been able to work around the issue? Hopefully, file credentials
or named profiles will work for your deployments.
Thanks,
James
On Fri, Nov 17, 2017 at 7:10 AM, Jennifer Kissinger <
jennifer.kissinger@semanticbits.com> wrote:
> Good morning,
>
> I'm developing a pipeline that uses the AWSCredentialsProviderControllerService
> to establish AWS s3 credentials that include a role ARN for cross-account
> access. When I hard-code the values of Access Key and Secret Key, I can
> successfully connect. When I use expression language in those fields to
> reference custom Nifi properties (i.e. ${my.custom.access_key}), the
> connection fails. I've confirmed that these custom properties work when
> used directly on a processor like FetchS3Object, ListS3, etc.
>
> I believe that the Access Key and Secret Key fields in the AWS controller
> service do not actually evaluate expression language, contrary to the
> documentation. However I would welcome any suggestions of possible user
> error.
>
> I am using Nifi 1.3.0 locally but will need to deploy this pipeline to
> Nifi 1.2.0. The error received when using the properties looks like this:
>
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> The security token included in the request is invalid. (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code:
> InvalidClientTokenId; Request ID: ...)
>
> Thanks for your assistance,
>
> ~Jenni
>