You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Andrew McDonald <am...@ccri.com> on 2018/12/12 13:42:09 UTC
Re: How do you use a custom region?
I've created the JIRA https://issues.apache.org/jira/browse/NIFI-5893
I couldn't find one if you had already made it so it could be a dup.
Regards,Andrew
On 11/30/18 11:58, Sivaprasanna wrote:
> I understand where the problem is. The reason why we replaced
> /this.client.setEndpoint(urlstr);/ with
> /this.client.setEndpoint(urlstr, this.client.getServiceName(),
> this.region.getName()); /was that when working VPC enabled services,
> the endpoint URL wasn't properly parsed. Having said that,
> /this.region.getName()/ shouldn't be there. I'll work on a solution.
> Thanks for bringing this up.
>
> -
> Sivaprasanna
>
> On Fri, Nov 30, 2018 at 9:49 PM Andrew McDonald <amcdonald@ccri.com
> <ma...@ccri.com>> wrote:
>
> Setting line 289 in AbstractAWSProcessor.java to
>
> this.client.setEndpoint(urlstr);
>
> worked for me.
>
> Regards,Andrew
>
> On 11/30/18 09:12, Andrew McDonald wrote:
>>
>> This is on an air-gapped system so I'll type in the essential part
>>
>> Failed to receive messages from Amazon SQS due to
>> com.amazonaws.services.sqs.model.AmazonSQSException: Credential
>> should be scoped to a valid region, not 'us-east-1'. (Service:
>> AmazonSQS; Status Code: 403; Error Code: SignatureDoesNotMatch;
>> Request ID: xxxxx-xxxx-xxxx-xxxx-xxxxxxxx).
>>
>> I believe that line 289 in AbstractAWSProcessor.java is setting
>> the wrong regionId
>>
>> this.client.setEndpoint(urlstr, this.client.getServiceName(),
>> this.region.getName());
>>
>> The amazon docs for the third parameter (regionId) state:
>>
>> * @param regionId * The ID of the region in which this service
>> resides AND the * overriding region for signing purposes.
>>
>> This would explain the SignatureDoesNotMatch error.
>>
>> The properties for the sqs processor work for nifi 1.4.0.
>>
>> Regards, Andrew
>>
>>
>> On 11/29/18 23:22, Sivaprasanna wrote:
>>> Andrew, Can you please share the complete error message and also
>>> show us how the properties look for the GetSQS processor?
>>>
>>> Thanks.
>>>
>>> On Fri, Nov 30, 2018 at 3:45 AM Andrew McDonald
>>> <amcdonald@ccri.com <ma...@ccri.com>> wrote:
>>>
>>> So I've installed nifi 1.8.0 and endpoint override didn't help
>>>
>>> The GetSQS is throwing and exception
>>>
>>> com.amazonaws.services.sqs.model.AmazonSQSException:
>>> Credential should be scoped to a valid region, not
>>> 'us-east-1'.
>>>
>>> I'm guessing that the override endpoint is not overriding
>>> the region b/c 'us-east-1' is not my region.
>>>
>>> Regards, Andrew
>>>
>>> On 11/29/18 02:09, Sivaprasanna wrote:
>>>> Yeah. This was added in 1.8.0 for SQS. However, the reason
>>>> why a custom enum was added on the NiFi side[1] was to have
>>>> a proper readable region instead of just the region code
>>>> i.e., Asia Pacific (Singapore) instead of ap-southeast-1.
>>>> However, I raised a request later to the AWS Java SDK team
>>>> to have a readable name. It was added recently, if I
>>>> remember correctly. So ideally this enum has to be removed
>>>> and the one on the official AWS SDK has to be leveraged
>>>> completely. I have created a Jira[2] and started working on
>>>> it. I'll raise a PR soon.
>>>>
>>>> [1] https://issues.apache.org/jira/browse/NIFI-5129
>>>> [2] https://issues.apache.org/jira/browse/NIFI-5850
>>>>
>>>> Thanks,
>>>>
>>>> On Thu, Nov 29, 2018 at 2:20 AM Andrew McDonald
>>>> <amcdonald@ccri.com <ma...@ccri.com>> wrote:
>>>>
>>>> zenfenan added this later to 1.8.0, yay!
>>>>
>>>> On 11/28/18 14:23, Andrew McDonald wrote:
>>>>>
>>>>>
>>>>> This workaround doesn't work for sqs because it
>>>>> doesn't have endpoint override URL property
>>>>>
>>>>>
>>>>> On 11/28/18 12:16, Michael Moser wrote:
>>>>>> Greetings! This JIRA ticket [1] describes the
>>>>>> recommended work around for AWS regions that aren't
>>>>>> in the list.
>>>>>>
>>>>>> -- Mike
>>>>>>
>>>>>> [1] - https://issues.apache.org/jira/browse/NIFI-4523
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 28, 2018 at 11:44 AM Jon Logan
>>>>>> <jmlogan@buffalo.edu <ma...@buffalo.edu>> wrote:
>>>>>>
>>>>>> Andrew,
>>>>>>
>>>>>> I know there's a few regions not in the list. I'm
>>>>>> not sure which region you're targeting, but at
>>>>>> least for the case of one of the new regions, I
>>>>>> submitted a PR for this. I haven't dug into it
>>>>>> deeply, but it seems like a better way to do this
>>>>>> might be to remove the enum entirely and get the
>>>>>> region list via the AWS API, or allow a free-form
>>>>>> entry.
>>>>>>
>>>>>> https://github.com/apache/nifi/pull/3187
>>>>>>
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> On Wed, Nov 28, 2018 at 10:35 AM Andrew McDonald
>>>>>> <amcdonald@ccri.com <ma...@ccri.com>>
>>>>>> wrote:
>>>>>>
>>>>>> I'm trying to upgrade from 1.4.0 to 1.7.1 but
>>>>>> the s3 processors can not
>>>>>> be initialized.
>>>>>>
>>>>>> Nifi 1.7.1 uses 1.11.319 and is throwing an
>>>>>> IllegalArgumentException: no
>>>>>> region provided
>>>>>>
>>>>>> The region I'm using isn't in the enum, so is
>>>>>> it possible to use a
>>>>>> custom region?
>>>>>>
>>>>>> Regards,Andrew
>>>>>>
>>>>>>
Re: How do you use a custom region?
Posted by Sivaprasanna <si...@gmail.com>.
Extremely sorry, Andrew. Was super busy for the past few days so wasn't
able to reply on time. Thanks for creating the Jira. I'll take a look.
Thanks once again.
-
Sivaprasanna
On Wed, Dec 12, 2018 at 7:18 PM Andrew McDonald <am...@ccri.com> wrote:
> I've created the JIRA https://issues.apache.org/jira/browse/NIFI-5893
>
> I couldn't find one if you had already made it so it could be a dup.
>
> Regards,Andrew
> On 11/30/18 11:58, Sivaprasanna wrote:
>
> I understand where the problem is. The reason why we replaced
> *this.client.setEndpoint(urlstr);* with *this.client.setEndpoint(urlstr,
> this.client.getServiceName(), this.region.getName()); *was that when
> working VPC enabled services, the endpoint URL wasn't properly parsed.
> Having said that, *this.region.getName()* shouldn't be there. I'll work
> on a solution. Thanks for bringing this up.
>
> -
> Sivaprasanna
>
> On Fri, Nov 30, 2018 at 9:49 PM Andrew McDonald <am...@ccri.com>
> wrote:
>
>> Setting line 289 in AbstractAWSProcessor.java to
>>
>> this.client.setEndpoint(urlstr);
>>
>> worked for me.
>>
>> Regards,Andrew
>> On 11/30/18 09:12, Andrew McDonald wrote:
>>
>> This is on an air-gapped system so I'll type in the essential part
>>
>> Failed to receive messages from Amazon SQS due to
>> com.amazonaws.services.sqs.model.AmazonSQSException: Credential should be
>> scoped to a valid region, not 'us-east-1'. (Service: AmazonSQS; Status
>> Code: 403; Error Code: SignatureDoesNotMatch; Request ID:
>> xxxxx-xxxx-xxxx-xxxx-xxxxxxxx).
>>
>> I believe that line 289 in AbstractAWSProcessor.java is setting the wrong
>> regionId
>>
>> this.client.setEndpoint(urlstr, this.client.getServiceName(), this.region.getName());
>>
>> The amazon docs for the third parameter (regionId) state:
>>
>> * @param regionId* The ID of the region in which this service resides AND the* overriding region for signing purposes.
>>
>> This would explain the SignatureDoesNotMatch error.
>>
>> The properties for the sqs processor work for nifi 1.4.0.
>>
>> Regards, Andrew
>>
>>
>> On 11/29/18 23:22, Sivaprasanna wrote:
>>
>> Andrew, Can you please share the complete error message and also show us
>> how the properties look for the GetSQS processor?
>>
>> Thanks.
>>
>> On Fri, Nov 30, 2018 at 3:45 AM Andrew McDonald <am...@ccri.com>
>> wrote:
>>
>>> So I've installed nifi 1.8.0 and endpoint override didn't help
>>>
>>> The GetSQS is throwing and exception
>>>
>>> com.amazonaws.services.sqs.model.AmazonSQSException: Credential should
>>> be scoped to a valid region, not 'us-east-1'.
>>>
>>> I'm guessing that the override endpoint is not overriding the region b/c
>>> 'us-east-1' is not my region.
>>>
>>> Regards, Andrew
>>> On 11/29/18 02:09, Sivaprasanna wrote:
>>>
>>> Yeah. This was added in 1.8.0 for SQS. However, the reason why a custom
>>> enum was added on the NiFi side[1] was to have a proper readable region
>>> instead of just the region code i.e., Asia Pacific (Singapore) instead
>>> of ap-southeast-1. However, I raised a request later to the AWS Java
>>> SDK team to have a readable name. It was added recently, if I remember
>>> correctly. So ideally this enum has to be removed and the one on the
>>> official AWS SDK has to be leveraged completely. I have created a Jira[2]
>>> and started working on it. I'll raise a PR soon.
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-5129
>>> [2] https://issues.apache.org/jira/browse/NIFI-5850
>>>
>>> Thanks,
>>>
>>> On Thu, Nov 29, 2018 at 2:20 AM Andrew McDonald <am...@ccri.com>
>>> wrote:
>>>
>>>> zenfenan added this later to 1.8.0, yay!
>>>> On 11/28/18 14:23, Andrew McDonald wrote:
>>>>
>>>>
>>>> This workaround doesn't work for sqs because it doesn't have endpoint
>>>> override URL property
>>>>
>>>>
>>>> On 11/28/18 12:16, Michael Moser wrote:
>>>>
>>>> Greetings! This JIRA ticket [1] describes the recommended work around
>>>> for AWS regions that aren't in the list.
>>>>
>>>> -- Mike
>>>>
>>>> [1] - https://issues.apache.org/jira/browse/NIFI-4523
>>>>
>>>>
>>>>
>>>> On Wed, Nov 28, 2018 at 11:44 AM Jon Logan <jm...@buffalo.edu> wrote:
>>>>
>>>>> Andrew,
>>>>>
>>>>> I know there's a few regions not in the list. I'm not sure which
>>>>> region you're targeting, but at least for the case of one of the new
>>>>> regions, I submitted a PR for this. I haven't dug into it deeply, but it
>>>>> seems like a better way to do this might be to remove the enum entirely and
>>>>> get the region list via the AWS API, or allow a free-form entry.
>>>>>
>>>>> https://github.com/apache/nifi/pull/3187
>>>>>
>>>>>
>>>>> Jon
>>>>>
>>>>> On Wed, Nov 28, 2018 at 10:35 AM Andrew McDonald <am...@ccri.com>
>>>>> wrote:
>>>>>
>>>>>> I'm trying to upgrade from 1.4.0 to 1.7.1 but the s3 processors can
>>>>>> not
>>>>>> be initialized.
>>>>>>
>>>>>> Nifi 1.7.1 uses 1.11.319 and is throwing an IllegalArgumentException:
>>>>>> no
>>>>>> region provided
>>>>>>
>>>>>> The region I'm using isn't in the enum, so is it possible to use a
>>>>>> custom region?
>>>>>>
>>>>>> Regards,Andrew
>>>>>>
>>>>>>
>>>>>>