You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/07/13 12:22:31 UTC
[GitHub] [beam] piotr-szuberski opened a new pull request #12235: [BEAM-10138] Python wrapper for Java's KinesisIO.Write with integration test
piotr-szuberski opened a new pull request #12235:
URL: https://github.com/apache/beam/pull/12235
Add a python wrapper for KinesisIO.Write from [#12234](https://github.com/apache/beam/pull/12234)
This test cannot be run on Jenkins because Beam does not have an AWS account. It has been tested locally on DirectRunner, FlinkRunner and SparkRunner.
R: @TheNeuralBit The same as with Java KinesisIO.Write external registrar - if you wanted to do this review I'd appreciate it :)
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | --- | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit closed pull request #12235: [BEAM-10138] Python wrapper for Cross-language Java's KinesisIO
Posted by GitBox <gi...@apache.org>.
TheNeuralBit closed pull request #12235:
URL: https://github.com/apache/beam/pull/12235
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #12235: [BEAM-10138] Python wrapper for Java's KinesisIO.Write with integration test
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #12235:
URL: https://github.com/apache/beam/pull/12235#issuecomment-662898441
> Sorry for the delay @piotr-szuberski! I'll try to take a look at these today and/or tomorrow.
No problem at all!
> One high-level question - do we have a jira about not running KinesisIO tests continuously? If not, please file one and make sure it mentions both direct Java tests as well as xlang tests.
Yes, there is [BEAM-601](https://issues.apache.org/jira/browse/BEAM-601) - I've added a comment there and I'll mention it in the integration tests comment
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12235: [BEAM-10138] Python wrapper for Java's KinesisIO.Write with integration test
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #12235:
URL: https://github.com/apache/beam/pull/12235#issuecomment-663077738
@TheNeuralBit I've just figured out that I could use kinesis SDK which is already included in AWS_REQUIREMENTS (boto3) - It will probably make it possible to separate the read and write test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski edited a comment on pull request #12235: [BEAM-10138] Python wrapper for Cross-language Java's KinesisIO
Posted by GitBox <gi...@apache.org>.
piotr-szuberski edited a comment on pull request #12235:
URL: https://github.com/apache/beam/pull/12235#issuecomment-663077738
@TheNeuralBit I've just figured out that I could use kinesis SDK which is already included in AWS_REQUIREMENTS (boto3) - It will probably make it possible to separate the read and write test.
EDIT: I followed the suggestion from Jira to use localstack with testcontainers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12235: [BEAM-10138] Python wrapper for Cross-language Java's KinesisIO
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12235:
URL: https://github.com/apache/beam/pull/12235#issuecomment-666766546
Closing in favor of #12297
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] piotr-szuberski commented on pull request #12235: [BEAM-10138] Python wrapper for Java's KinesisIO.Write with integration test
Posted by GitBox <gi...@apache.org>.
piotr-szuberski commented on pull request #12235:
URL: https://github.com/apache/beam/pull/12235#issuecomment-662898441
> Sorry for the delay @piotr-szuberski! I'll try to take a look at these today and/or tomorrow.
No problem at all!
> One high-level question - do we have a jira about not running KinesisIO tests continuously? If not, please file one and make sure it mentions both direct Java tests as well as xlang tests.
Yes, there is [BEAM-601](https://issues.apache.org/jira/browse/BEAM-601) - I've added a comment there and I'll mention it in the integration tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] TheNeuralBit commented on pull request #12235: [BEAM-10138] Python wrapper for Java's KinesisIO.Write with integration test
Posted by GitBox <gi...@apache.org>.
TheNeuralBit commented on pull request #12235:
URL: https://github.com/apache/beam/pull/12235#issuecomment-662691235
Sorry for the delay @piotr-szuberski! I'll try to take a look at these today and/or tomorrow.
One high-level question - do we have a jira about not running KinesisIO tests continuously? If not, please file one and make sure it mentions both direct Java tests as well as xlang tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org