You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/29 07:55:04 UTC
[GitHub] [beam] mosche opened a new pull request, #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
mosche opened a new pull request, #22093:
URL: https://github.com/apache/beam/pull/22093
Currently there's two different versions of AWS IO connectors in Beam for the Java SDK:
* [amazon-web-services](https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services) and [kinesis](https://github.com/apache/beam/tree/master/sdks/java/io/kinesis) for the AWS Java SDK v1
* [amazon-web-services2](https://github.com/apache/beam/tree/master/sdks/java/io/amazon-web-services2) (including kinesis) for the AWS Java SDK v2
This deprecates IOs in both, the `amazon-web-services` and `kinesis` module in favor of the `amazon-web-services2` module and closes #21249.
After a few Beam releases both, the `amazon-web-services` and `kinesis` module will be removed.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #22093:
URL: https://github.com/apache/beam/pull/22093#issuecomment-1169657053
R: @aromanenko-dev
R: @aaltay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #22093:
URL: https://github.com/apache/beam/pull/22093#issuecomment-1169697354
Interesting that no anyJenkins jobs were triggered for this PR....
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #22093:
URL: https://github.com/apache/beam/pull/22093#issuecomment-1178625000
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a diff in pull request #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
Posted by GitBox <gi...@apache.org>.
mosche commented on code in PR #22093:
URL: https://github.com/apache/beam/pull/22093#discussion_r913839397
##########
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java:
##########
@@ -120,6 +124,10 @@ class S3FileSystem extends FileSystem<S3ResourceId> {
MoreExecutors.listeningDecorator(
Executors.newFixedThreadPool(
config.getS3ThreadPoolSize(), new ThreadFactoryBuilder().setDaemon(true).build()));
+
+ LOG.warn(
Review Comment:
✔️ done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a diff in pull request #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on code in PR #22093:
URL: https://github.com/apache/beam/pull/22093#discussion_r910203929
##########
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java:
##########
@@ -120,6 +124,10 @@ class S3FileSystem extends FileSystem<S3ResourceId> {
MoreExecutors.listeningDecorator(
Executors.newFixedThreadPool(
config.getS3ThreadPoolSize(), new ThreadFactoryBuilder().setDaemon(true).build()));
+
+ LOG.warn(
Review Comment:
Yes but it would be useful to have it in runtime as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on a diff in pull request #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on code in PR #22093:
URL: https://github.com/apache/beam/pull/22093#discussion_r909347302
##########
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java:
##########
@@ -120,6 +124,10 @@ class S3FileSystem extends FileSystem<S3ResourceId> {
MoreExecutors.listeningDecorator(
Executors.newFixedThreadPool(
config.getS3ThreadPoolSize(), new ThreadFactoryBuilder().setDaemon(true).build()));
+
+ LOG.warn(
Review Comment:
I'd suggest to add similar warning log message to all other deprecated IO `expand` or read/write methods for more visibility of this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #22093:
URL: https://github.com/apache/beam/pull/22093#issuecomment-1169880191
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #22093:
URL: https://github.com/apache/beam/pull/22093#issuecomment-1169697510
retest this please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on a diff in pull request #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
Posted by GitBox <gi...@apache.org>.
mosche commented on code in PR #22093:
URL: https://github.com/apache/beam/pull/22093#discussion_r909368789
##########
sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystem.java:
##########
@@ -120,6 +124,10 @@ class S3FileSystem extends FileSystem<S3ResourceId> {
MoreExecutors.listeningDecorator(
Executors.newFixedThreadPool(
config.getS3ThreadPoolSize(), new ThreadFactoryBuilder().setDaemon(true).build()));
+
+ LOG.warn(
Review Comment:
The reason I added it here is that filesystems are loaded automatically... for all other IOs users should see a warning in their IDE / when compiling right? That feels a lot more noticeable than the log messages.
BTW, here's the reason the logs for S3FileSystem will be extremely verbose :/
Looks like there's a severe problem regarding initialisation of file systems with respect to `SerializablePipelineOptions`, though didn't get any feedback
https://lists.apache.org/thread/sp9g643lfyj6fzg9m2pym23kjv6zkbmq
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche merged pull request #22093: Deprecate AWS IOs (Java) using AWS SDK v1 in favor of IOs in amazon-web-services2
Posted by GitBox <gi...@apache.org>.
mosche merged PR #22093:
URL: https://github.com/apache/beam/pull/22093
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org