You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/04/06 00:06:54 UTC
[GitHub] [beam] jfarr opened a new pull request #11318: [BEAM-9702] Update
Java KinesisIO to support AWS SDK v2
jfarr opened a new pull request #11318: [BEAM-9702] Update Java KinesisIO to support AWS SDK v2
URL: https://github.com/apache/beam/pull/11318
Hi @aromanenko-dev, I thought it would be a good idea to separate #9899 into two PRs so that users can potentially benefit from the update to use the v2 AWS SDK while enhanced fanout support is still being worked on. lmkwyt
FYI, I found these commands helpful for comparing v1 vs. v2:
```
diff -u -r sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/ sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/ > diff-main.txt
diff -u -r sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/ sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/kinesis/ > diff-test.txt
diff -u sdks/java/io/kinesis/build.gradle sdks/java/io/amazon-web-services2/build.gradle
```
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11318: [BEAM-9702] Update
Java KinesisIO to support AWS SDK v2
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11318: [BEAM-9702] Update Java KinesisIO to support AWS SDK v2
URL: https://github.com/apache/beam/pull/11318#issuecomment-615369782
@jfarr I agree with your point but I'm afraid that once we will have two versions of KinesisIO (for AWS SDK 1 & 2), that are based on the large amount of the same code but existing in two different packages , then we will need to always keep in mind to do code changes in two places. I'd definitively want to avoid it since these KinesisIO will diverge quickly and I'm not sure that we will deprecate and remove the AWS SDK V1 version soon.
Well, maybe it will be a good point to discuss this on dev@ mailing list and to hear what the community think about it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11318: [BEAM-9702] Update
Java KinesisIO to support AWS SDK v2
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11318: [BEAM-9702] Update Java KinesisIO to support AWS SDK v2
URL: https://github.com/apache/beam/pull/11318#issuecomment-611037916
@jfarr Agree, but I don't have a perfect solution except to extract and to wrap all AWS depended code into own interfaces and then use them to work with AWS. All what I'm afraid is to lose all latest fixes and improvements that we done for old KinesisIO and probably will be done in the future since people actively use it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] jfarr commented on issue #11318: [BEAM-9702] Update Java
KinesisIO to support AWS SDK v2
Posted by GitBox <gi...@apache.org>.
jfarr commented on issue #11318: [BEAM-9702] Update Java KinesisIO to support AWS SDK v2
URL: https://github.com/apache/beam/pull/11318#issuecomment-612511129
Hi, @aromanenko-dev. That sounds like a significant rewrite of the existing KinesisIO. Should the other AWS IOs be redesigned to fit that model too, or where does that stop? I actually like the idea, just trying to clarify what you have in mind.
And don't you think that the same clearance logic applies here? A rewrite is going to take some time, probably a significant amount. Why not get this PR into the hands of the community so they can start using it now and then do the refactor in a separate PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] jfarr commented on issue #11318: [BEAM-9702] Update Java
KinesisIO to support AWS SDK v2
Posted by GitBox <gi...@apache.org>.
jfarr commented on issue #11318: [BEAM-9702] Update Java KinesisIO to support AWS SDK v2
URL: https://github.com/apache/beam/pull/11318#issuecomment-610452905
Hi @aromanenko-dev, I share your concern. However, that was always going to be the case, splitting it into multiple PRs doesn’t change that, and it’s already the case for other AWS IOs such as SNS and DynamoDB. What do you propose for a solution?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] jfarr commented on issue #11318: [BEAM-9702] Update Java
KinesisIO to support AWS SDK v2
Posted by GitBox <gi...@apache.org>.
jfarr commented on issue #11318: [BEAM-9702] Update Java KinesisIO to support AWS SDK v2
URL: https://github.com/apache/beam/pull/11318#issuecomment-615535538
@aromanenko-dev Good idea, I started a new thread.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] aromanenko-dev commented on issue #11318: [BEAM-9702] Update
Java KinesisIO to support AWS SDK v2
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #11318: [BEAM-9702] Update Java KinesisIO to support AWS SDK v2
URL: https://github.com/apache/beam/pull/11318#issuecomment-609894337
Yes, I agree that it will make sense to split it into several PR for clearance. In the same time, I'm more concerned that with two packages we will have a lot of duplicated code there.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services