You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/07/14 21:49:34 UTC
[GitHub] [beam] allenpradeep opened a new pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
allenpradeep opened a new pull request #12255:
URL: https://github.com/apache/beam/pull/12255
Adding mutation byte counters for all mutations written to Spanner. This will help monitor job progress.
@nielm @chamikaramj
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [x] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-665768931
@allenpradeep given that this seems to be a per-element change, have we done a performance analysis to confirm that there's won't be a significant performance degradation due to this ?
LGTM other than that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] allenpradeep commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
allenpradeep commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-670204460
> @allenpradeep - any progress on this?
@aaltay
I had similar thoughts on this as @chamikaramj had with respect to performance. I'll would want to hear feedback from Niel on this and so I'll work with him once he is back.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #12255:
URL: https://github.com/apache/beam/pull/12255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] nielm commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
nielm commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-680931499
LGTM from me :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-670160224
@allenpradeep - any progress on this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-689895432
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-744944339
Allen, any updates ? Seems like there's a conflict to be resolved now.
Feel free to close if this is no longer needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-1032573489
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] allenpradeep commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
allenpradeep commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-689895773
> Allen, are you doing further updates here ? If not we can merge.
Hi Cham,
I'll a few more tests and rebase this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] stale[bot] commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
stale[bot] commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-727715793
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-1041455142
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aaltay commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
aaltay commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-663159341
@nielm - Could you do the first review on this one?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] chamikaramj commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-689895246
Allen, are you doing further updates here ? If not we can merge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] allenpradeep commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
allenpradeep commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-728161911
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] allenpradeep commented on pull request #12255: [BEAM-9910]Add mutation byte counters to SpannerIO.write
Posted by GitBox <gi...@apache.org>.
allenpradeep commented on pull request #12255:
URL: https://github.com/apache/beam/pull/12255#issuecomment-670202696
> @allenpradeep given that this seems to be a per-element change, have we done a performance analysis to confirm that there's won't be a significant performance degradation due to this ?
>
> LGTM other than that.
Hi Cham,
While dealing with a batch of mutations the size of the batch is already calculated. We are just passing this size as a KV pair through the pipeline and so there would be no cost here. But if a batch fails, then we end doing a recalculation of individual mutation as we commit it separately which is the added cost here. This may not be significant compared to the cost of making a RPC call per mutation.
However, I'm planning to work with Niel on this once he is back.
In my initial tests, I did not find performance degradation when the batches did not fail.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org