You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by tc...@tutanota.com on 2020/09/24 23:23:52 UTC
Setting AWS endponts per transform
We're currently working on getting our Beam app working with localstack (and potentially other AWS regions). We're using SqsIO and S3 as part of our pipeline (with other AWS components likely to come into the mix). While I could cast the PipelineOptions to AwsOptions and then call AwsOptions.setAwsServiceEndpoint() prior to pipeline construction, that won't work as different AWS services make use of different endpoints -- e.g. the endpoint for SqsIO isn't going to work for S3.
What I'd really like to do is provide a different set of AwsOptions per AWS service. What's the best means of accomplishing this?
Tim.
Re: Setting AWS endponts per transform
Posted by Reuven Lax <re...@google.com>.
It would also be ideal if someone extended the SqsIO transform to allow
specifying endpoints via a builder method.
On Fri, Sep 25, 2020 at 9:50 AM Luke Cwik <lc...@google.com> wrote:
> You can create a PipelineOptions interface with a unique endpoint
> identifier for each service you want to use like:
> public interface AwsEndpointOptions extends PipelineOptions {
> @Default.InstanceFactory(FallbackToAwsOptions.class);
> String getSnsEndpoint();
> void setSnsEndpoint();
>
> @Default.InstanceFactory(FallbackToAwsOptions.class);
> String getS3Endpoint();
> void setS3Endpoint();
> ...
>
> class FallbackToAwsOptions implements DefaultValueFactory<String> {
> @Override
> public String create(PipelineOptions options) {
> return options.as(AwsOptions.class).getAwsServiceEndpoint();
> }
> }
> }
>
> It looks like AWS IOs support passing in a provider[1] and/or configuring
> the endpoint during pipeline construction. Then during pipeline
> creation/execution you can configure each AWS IO instance with the specific
> endpoint from this new PipelineOptions interface.
>
> 1:
> https://github.com/apache/beam/blob/6fdde4f4eab72b49b10a8bb1cb3be263c5c416b5/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java#L273
>
> On Thu, Sep 24, 2020 at 4:24 PM <tc...@tutanota.com> wrote:
>
>> We're currently working on getting our Beam app working with localstack
>> (and potentially other AWS regions). We're using SqsIO and S3 as part of
>> our pipeline (with other AWS components likely to come into the mix).
>> While I could cast the PipelineOptions to AwsOptions and then call
>> AwsOptions.setAwsServiceEndpoint() prior to pipeline construction, that
>> won't work as different AWS services make use of different endpoints --
>> e.g. the endpoint for SqsIO isn't going to work for S3.
>>
>> What I'd really like to do is provide a different set of AwsOptions per
>> AWS service. What's the best means of accomplishing this?
>>
>> Tim.
>>
>>
Re: Setting AWS endponts per transform
Posted by Luke Cwik <lc...@google.com>.
You can create a PipelineOptions interface with a unique endpoint
identifier for each service you want to use like:
public interface AwsEndpointOptions extends PipelineOptions {
@Default.InstanceFactory(FallbackToAwsOptions.class);
String getSnsEndpoint();
void setSnsEndpoint();
@Default.InstanceFactory(FallbackToAwsOptions.class);
String getS3Endpoint();
void setS3Endpoint();
...
class FallbackToAwsOptions implements DefaultValueFactory<String> {
@Override
public String create(PipelineOptions options) {
return options.as(AwsOptions.class).getAwsServiceEndpoint();
}
}
}
It looks like AWS IOs support passing in a provider[1] and/or configuring
the endpoint during pipeline construction. Then during pipeline
creation/execution you can configure each AWS IO instance with the specific
endpoint from this new PipelineOptions interface.
1:
https://github.com/apache/beam/blob/6fdde4f4eab72b49b10a8bb1cb3be263c5c416b5/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java#L273
On Thu, Sep 24, 2020 at 4:24 PM <tc...@tutanota.com> wrote:
> We're currently working on getting our Beam app working with localstack
> (and potentially other AWS regions). We're using SqsIO and S3 as part of
> our pipeline (with other AWS components likely to come into the mix).
> While I could cast the PipelineOptions to AwsOptions and then call
> AwsOptions.setAwsServiceEndpoint() prior to pipeline construction, that
> won't work as different AWS services make use of different endpoints --
> e.g. the endpoint for SqsIO isn't going to work for S3.
>
> What I'd really like to do is provide a different set of AwsOptions per
> AWS service. What's the best means of accomplishing this?
>
> Tim.
>
>