You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/30 15:03:13 UTC
[GitHub] [beam] johnjcasey opened a new pull request, #22113: papercut fix: change unique id generation to be during the pipeline, …
johnjcasey opened a new pull request, #22113:
URL: https://github.com/apache/beam/pull/22113
…preventing re-use between jobs
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #22113:
URL: https://github.com/apache/beam/pull/22113#issuecomment-1171333745
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
URL: https://github.com/apache/beam/pull/22113
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a diff in pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
pabloem commented on code in PR #22113:
URL: https://github.com/apache/beam/pull/22113#discussion_r911428194
##########
sdks/python/apache_beam/io/gcp/bigquery_test.py:
##########
@@ -589,6 +596,8 @@ def test_read_export_exception(self, exception_type, error_message):
table='project:dataset.table',
gcs_location="gs://temp_location")
+ p.run().wait_until_finish(duration=30)
Review Comment:
these will cause the pipeline to be ran twice (see https://stackoverflow.com/a/52282001/1255356)
##########
sdks/python/apache_beam/io/gcp/bigquery_test.py:
##########
@@ -508,21 +508,26 @@ def test_temp_dataset_is_configurable(
])
def test_create_temp_dataset_exception(self, exception_type, error_message):
- with mock.patch.object(bigquery_v2_client.BigqueryV2.JobsService,
+ with mock.patch.object(beam.io.gcp.bigquery._CustomBigQuerySource,
+ 'estimate_size') as mock_estimate,\
+ mock.patch.object(bigquery_v2_client.BigqueryV2.JobsService,
'Insert'),\
mock.patch.object(BigQueryWrapper,
'get_or_create_dataset') as mock_insert, \
mock.patch('time.sleep'), \
self.assertRaises(Exception) as exc,\
beam.Pipeline() as p:
+ mock_estimate.return_value = None
mock_insert.side_effect = exception_type(error_message)
_ = p | ReadFromBigQuery(
project='apache-beam-testing',
query='SELECT * FROM `project.dataset.table`',
gcs_location='gs://temp_location')
+ p.run().wait_until_finish(duration=30)
Review Comment:
these will cause the pipeline to be ran twice (see https://stackoverflow.com/a/52282001/1255356)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #22113:
URL: https://github.com/apache/beam/pull/22113#issuecomment-1171333751
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #22113:
URL: https://github.com/apache/beam/pull/22113#issuecomment-1451814800
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
robertwb commented on PR #22113:
URL: https://github.com/apache/beam/pull/22113#issuecomment-1185837596
Is this ready for another round of review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on a diff in pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on code in PR #22113:
URL: https://github.com/apache/beam/pull/22113#discussion_r912067547
##########
sdks/python/apache_beam/io/gcp/bigquery_test.py:
##########
@@ -508,21 +508,26 @@ def test_temp_dataset_is_configurable(
])
def test_create_temp_dataset_exception(self, exception_type, error_message):
- with mock.patch.object(bigquery_v2_client.BigqueryV2.JobsService,
+ with mock.patch.object(beam.io.gcp.bigquery._CustomBigQuerySource,
+ 'estimate_size') as mock_estimate,\
+ mock.patch.object(bigquery_v2_client.BigqueryV2.JobsService,
'Insert'),\
mock.patch.object(BigQueryWrapper,
'get_or_create_dataset') as mock_insert, \
mock.patch('time.sleep'), \
self.assertRaises(Exception) as exc,\
beam.Pipeline() as p:
+ mock_estimate.return_value = None
mock_insert.side_effect = exception_type(error_message)
_ = p | ReadFromBigQuery(
project='apache-beam-testing',
query='SELECT * FROM `project.dataset.table`',
gcs_location='gs://temp_location')
+ p.run().wait_until_finish(duration=30)
Review Comment:
That is likely causing my mocking problem. What is strange then is why the exception that is getting mocked here isn't getting thrown
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on PR #22113:
URL: https://github.com/apache/beam/pull/22113#issuecomment-1185840603
No, I'll convert it back to draft. I haven't hade time to fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a diff in pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
pabloem commented on code in PR #22113:
URL: https://github.com/apache/beam/pull/22113#discussion_r911428194
##########
sdks/python/apache_beam/io/gcp/bigquery_test.py:
##########
@@ -589,6 +596,8 @@ def test_read_export_exception(self, exception_type, error_message):
table='project:dataset.table',
gcs_location="gs://temp_location")
+ p.run().wait_until_finish(duration=30)
Review Comment:
this will cause the pipeline to be ran twice (see https://stackoverflow.com/a/52282001/1255356)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #22113:
URL: https://github.com/apache/beam/pull/22113#issuecomment-1171333748
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #22113:
URL: https://github.com/apache/beam/pull/22113#issuecomment-1171333749
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] asf-ci commented on pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
asf-ci commented on PR #22113:
URL: https://github.com/apache/beam/pull/22113#issuecomment-1171333746
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] johnjcasey commented on pull request #22113: papercut fix: change unique id generation to be during the pipeline, …
Posted by GitBox <gi...@apache.org>.
johnjcasey commented on PR #22113:
URL: https://github.com/apache/beam/pull/22113#issuecomment-1171360039
run python postcommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org