You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Peter Turcsanyi (Jira)" <ji...@apache.org> on 2021/06/04 18:40:00 UTC
[jira] [Updated] (NIFI-8662) Failed to parse AWS region from VPCE
endpoint URL in AbstractAWSProcessor
[ https://issues.apache.org/jira/browse/NIFI-8662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Turcsanyi updated NIFI-8662:
----------------------------------
Description:
The AWS client library cannot parse the region from custom endpoint URLs properly.
NIFI-5456 fixed this issue via passing the region configured on the processor to {{AmazonWebServiceClient.setEndpoint()}} directly (no parsing needed in the client library, neither in NiFi).
NIFI-5893 implemented the fix in another way: parsing the region from the endpoint URL on the NiFi side. It is not clear for me what special use case it wanted to solve but a regular VPCE endpoint does not work with it now.
Endpoint URL: {{https://vpce-*****************-********.sqs.us-west-2.vpce.amazonaws.com}}
Error:
{code:java}
2021-06-04 18:25:57,101 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.processors.aws.sqs.PutSQS PutSQS[id=c4714170-c2cb-39e9-a36c-c43e4604f64a] Failed to send messages to Amazon SQS due to com.amazonaws.services.sqs.model.AmazonSQSException: Credential should be scoped to a valid region, not 'us-east-1'. (Service: AmazonSQS; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 63ea72ad-a856-5eca-8c00-2b99da238d07)
{code}
It seems the "sqs" part does not match the regex used for parsing: {{^(?:.+[vpce-][a-z0-9-]+\.)?([a-z0-9-]+)$}}.
However, the endpoint properly works with NIFI-5456 only.
To support both fixes, I will implement the following logic:
- use the parse method from NIFI-5893 first and use that region if the parse successful (no change here)
- if the parsing fails, then fall back to NIFI-5456 (that is using the configured region) instead of the hard coded "us-east-1"
was:
The AWS client library cannot parse the region from custom endpoint URLs properly.
NIFI-5456 fixed this issue via passing the region configured on the processor to {{AmazonWebServiceClient.setEndpoint()}} directly (no parsing needed in the client library, neither in NiFi).
NIFI-5893 implemented the fix in another way: parsing the region from the endpoint URL on the NiFi side. It is not clear for me what special use case it wanted to solve but a regular VPCE endpoint does not work with it now.
Endpoint URL: {{https://vpce-*****************-********.sqs.us-west-2.vpce.amazonaws.com}}
Error:
{code:java}
2021-06-04 18:25:57,101 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.processors.aws.sqs.PutSQS PutSQS[id=c4714170-c2cb-39e9-a36c-c43e4604f64a] Failed to send messages to Amazon SQS due to com.amazonaws.services.sqs.model.AmazonSQSException: Credential should be scoped to a valid region, not 'us-east-1'. (Service: AmazonSQS; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 63ea72ad-a856-5eca-8c00-2b99da238d07)
{code}
It seems the "sqs" part does not match the regex used for parsing ({{^(?:.+[vpce-][a-z0-9-]+\.)?([a-z0-9-]+)$}}).
However, the endpoint properly works with NIFI-5456 only.
To support both fixes, I will implement the following logic:
- use the parse method from NIFI-5893 first and use that region if the parse successful (no change here)
- if the parsing fails, then fall back to NIFI-5456 (that is using the configured region) instead of the hard coded "us-west-1"
> Failed to parse AWS region from VPCE endpoint URL in AbstractAWSProcessor
> -------------------------------------------------------------------------
>
> Key: NIFI-8662
> URL: https://issues.apache.org/jira/browse/NIFI-8662
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Peter Turcsanyi
> Assignee: Peter Turcsanyi
> Priority: Major
>
> The AWS client library cannot parse the region from custom endpoint URLs properly.
> NIFI-5456 fixed this issue via passing the region configured on the processor to {{AmazonWebServiceClient.setEndpoint()}} directly (no parsing needed in the client library, neither in NiFi).
> NIFI-5893 implemented the fix in another way: parsing the region from the endpoint URL on the NiFi side. It is not clear for me what special use case it wanted to solve but a regular VPCE endpoint does not work with it now.
> Endpoint URL: {{https://vpce-*****************-********.sqs.us-west-2.vpce.amazonaws.com}}
> Error:
> {code:java}
> 2021-06-04 18:25:57,101 ERROR [Timer-Driven Process Thread-5] o.apache.nifi.processors.aws.sqs.PutSQS PutSQS[id=c4714170-c2cb-39e9-a36c-c43e4604f64a] Failed to send messages to Amazon SQS due to com.amazonaws.services.sqs.model.AmazonSQSException: Credential should be scoped to a valid region, not 'us-east-1'. (Service: AmazonSQS; Status Code: 403; Error Code: SignatureDoesNotMatch; Request ID: 63ea72ad-a856-5eca-8c00-2b99da238d07)
> {code}
> It seems the "sqs" part does not match the regex used for parsing: {{^(?:.+[vpce-][a-z0-9-]+\.)?([a-z0-9-]+)$}}.
> However, the endpoint properly works with NIFI-5456 only.
> To support both fixes, I will implement the following logic:
> - use the parse method from NIFI-5893 first and use that region if the parse successful (no change here)
> - if the parsing fails, then fall back to NIFI-5456 (that is using the configured region) instead of the hard coded "us-east-1"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)