You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Robert Kubis (Jira)" <ji...@apache.org> on 2020/11/12 20:33:00 UTC
[jira] [Commented] (BEAM-11257) multiple buckets and regions
support in S3FileSystem
[ https://issues.apache.org/jira/browse/BEAM-11257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17230917#comment-17230917 ]
Robert Kubis commented on BEAM-11257:
-------------------------------------
```
public void processElement(ProcessContext c) {
AwsOptions options = c.getPipelineOptions().as(AwsOptions.class);
...
options.setAwsCredentialsProvider(<bucket01_auth>);
options.setAwsRegion(<bucket01_region>);
...
ResourceId resourceId = FileSystems.matchNewResource(<bucket01_filename>, false);
try (WritableByteChannel out = FileSystems.create(resourceId, MimeTypes.TEXT)) {
out.write(contents);
}
...
options.setAwsCredentialsProvider(<bucket02_auth>);
options.setAwsRegion(<bucket02_region>);
ResourceId resourceId = FileSystems.matchNewResource(<bucket02_filename>, false);
try (WritableByteChannel out = FileSystems.create(resourceId, MimeTypes.TEXT)) {
out.write(contents);
}
}
> multiple buckets and regions support in S3FileSystem
> ----------------------------------------------------
>
> Key: BEAM-11257
> URL: https://issues.apache.org/jira/browse/BEAM-11257
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
> Affects Versions: 2.25.0
> Reporter: Robert Kubis
> Priority: P2
>
> S3FileSystem picks up the bucket region from AwsOptions at pipeline.run() disallowing the change of the region and thus support to write to Buckets in different regions in one pipeline.
> Changing Auth to multiple different buckets is supported though
--
This message was sent by Atlassian Jira
(v8.3.4#803005)