You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "liferoad (via GitHub)" <gi...@apache.org> on 2023/09/22 17:27:29 UTC
[GitHub] [beam] liferoad opened a new pull request, #28621: Automatically Switch to upload_graph when the graph is large
liferoad opened a new pull request, #28621:
URL: https://github.com/apache/beam/pull/28621
We might need to update the CHANGES.md since the behavior is changed.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1368990729
##########
CHANGES.md:
##########
@@ -72,6 +72,8 @@ should handle this. ([#25252](https://github.com/apache/beam/issues/25252)).
## Breaking Changes
+* `upload_graph` is not needed since the graph now is uploaded automatically when it is larger than 10MB for Java SDK ([PR#28621](https://github.com/apache/beam/pull/28621).
Review Comment:
This does not sound like a `breaking change` instead of an improvement, i.e., in which scenario this would break existing user?
If so, consider phrasing like: `upload_graph` experiments option is no longer required in the case of large pipeline graphs ...
##########
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java:
##########
@@ -256,6 +256,22 @@ private Pipeline buildDataflowPipeline(DataflowPipelineOptions options) {
return p;
}
+ private Pipeline buildDataflowPipelineWithLargeGraph(DataflowPipelineOptions options) {
Review Comment:
minor: this method and the previous one (buildDataflowPipeline) could be static
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically switch to upload_graph when the graph is large [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1781712301
Once it is green again (for the record) will merge. Or any committer can merge.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically switch to upload_graph when the graph is large [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1782108615
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically switch to upload_graph when the graph is large [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1782138488
Note `"org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest.testActiveThreadMetric[*]"` seems flaky. I tried several times to get all greens.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on pull request #28621: Automatically Switch to upload_graph when the graph is large
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1741802110
R: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically switch to upload_graph when the graph is large [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1781899823
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically switch to upload_graph when the graph is large [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1373622952
##########
CHANGES.md:
##########
@@ -72,6 +72,8 @@ should handle this. ([#25252](https://github.com/apache/beam/issues/25252)).
## Breaking Changes
+* `upload_graph` is not needed since the graph now is uploaded automatically when it is larger than 10MB for Java SDK ([PR#28621](https://github.com/apache/beam/pull/28621).
Review Comment:
Done. Thanks.
##########
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java:
##########
@@ -256,6 +256,22 @@ private Pipeline buildDataflowPipeline(DataflowPipelineOptions options) {
return p;
}
+ private Pipeline buildDataflowPipelineWithLargeGraph(DataflowPipelineOptions options) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on pull request #28621: Automatically Switch to upload_graph when the graph is large
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1741802072
Tested these changes internally. The test passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically switch to upload_graph when the graph is large [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1782180824
> Note `"org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest.testActiveThreadMetric[*]"` seems flaky. I tried several times to get all greens.
Yes this is unrelated. Tracking bug: #28957 caused by #28513 there is a pending PR include a fix: https://github.com/apache/beam/pull/29041#issuecomment-1782121904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1367467116
##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java:
##########
@@ -1331,6 +1331,14 @@ public DataflowPipelineJob run(Pipeline pipeline) {
hooks.modifyEnvironmentBeforeSubmission(newJob.getEnvironment());
}
+ // enable upload_graph when the graph is too large
+ if ((Utf8.encodedLength(newJob.toString()) >= CREATE_JOB_REQUEST_LIMIT_BYTES)
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1368991768
##########
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java:
##########
@@ -824,6 +840,24 @@ public void testUploadGraph() throws IOException {
.startsWith("gs://valid-bucket/temp/staging/dataflow_graph"));
}
+ /** Test for automatically using upload_graph when the job graph is too large (>10MB). */
+ @Test
+ public void testUploadGraphWithAutoUpload() throws IOException {
+ DataflowPipelineOptions options = buildPipelineOptions();
+ Pipeline p = buildDataflowPipelineWithLargeGraph(options);
+ p.run();
+
+ ArgumentCaptor<Job> jobCaptor = ArgumentCaptor.forClass(Job.class);
+ Mockito.verify(mockJobs).create(eq(PROJECT_ID), eq(REGION_ID), jobCaptor.capture());
+ assertValidJob(jobCaptor.getValue());
+ assertTrue(jobCaptor.getValue().getSteps().isEmpty());
+ assertTrue(
+ jobCaptor
+ .getValue()
+ .getStepsLocation()
+ .startsWith("gs://valid-bucket/temp/staging/dataflow_graph"));
Review Comment:
nit: here and in the other tests `gs://valid-bucket/temp/staging/` should be part of the test case. It is not clear from the test if it is:
1. Part of the spec for this feature
2. A constant behavior of the code under test (in which case it should be a reference to that)
3. A value pass in from global test set up (this is an anti-pattern)
Just noting this - the whole file has this problem a lot and you don't have to fix it.
##########
CHANGES.md:
##########
@@ -72,6 +72,8 @@ should handle this. ([#25252](https://github.com/apache/beam/issues/25252)).
## Breaking Changes
+* `upload_graph` is not needed since the graph now is uploaded automatically when it is larger than 10MB for Java SDK ([PR#28621](https://github.com/apache/beam/pull/28621).
Review Comment:
It won't break anyone though. it is more like a new cool feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1773366164
> Let us get a test externally. It should be very easy - just a single small test method with a big graph.
Added one unit test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically switch to upload_graph when the graph is large [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1782818716
I think it is OK to disable the test or even better to roll back the change so we do not have untested stuff in the codebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] liferoad commented on pull request #28621: Automatically Switch to upload_graph when the graph is large
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1734277175
I plan to manually test this later (might add one integration test). So no need to merge this now. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #28621: Automatically Switch to upload_graph when the graph is large
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1741802631
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1743099675
Let us get a test externally. It should be very easy - just a single small test method with a big graph.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1344439659
##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java:
##########
@@ -1331,6 +1331,14 @@ public DataflowPipelineJob run(Pipeline pipeline) {
hooks.modifyEnvironmentBeforeSubmission(newJob.getEnvironment());
}
+ // enable upload_graph when the graph is too large
+ if ((Utf8.encodedLength(newJob.toString()) >= CREATE_JOB_REQUEST_LIMIT_BYTES)
Review Comment:
The calculation of `Utf8.encodedLength(newJob.toString())` could be expensive especially when the graph is large. Similar calculation include
- L1349: DataflowPipelineTranslator.jobToString(newJob).getBytes(UTF_8) when upload the graph
- L1410: if (Utf8.encodedLength(newJob.toString()) >= CREATE_JOB_REQUEST_LIMIT_BYTES) when decide the error message
We should be able to just run the conversion of `Job->utf8 encoded bytes` once
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically switch to upload_graph when the graph is large [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1373620864
##########
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java:
##########
@@ -824,6 +840,24 @@ public void testUploadGraph() throws IOException {
.startsWith("gs://valid-bucket/temp/staging/dataflow_graph"));
}
+ /** Test for automatically using upload_graph when the job graph is too large (>10MB). */
+ @Test
+ public void testUploadGraphWithAutoUpload() throws IOException {
+ DataflowPipelineOptions options = buildPipelineOptions();
+ Pipeline p = buildDataflowPipelineWithLargeGraph(options);
+ p.run();
+
+ ArgumentCaptor<Job> jobCaptor = ArgumentCaptor.forClass(Job.class);
+ Mockito.verify(mockJobs).create(eq(PROJECT_ID), eq(REGION_ID), jobCaptor.capture());
+ assertValidJob(jobCaptor.getValue());
+ assertTrue(jobCaptor.getValue().getSteps().isEmpty());
+ assertTrue(
+ jobCaptor
+ .getValue()
+ .getStepsLocation()
+ .startsWith("gs://valid-bucket/temp/staging/dataflow_graph"));
Review Comment:
Noted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Automatically switch to upload_graph when the graph is large [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #28621:
URL: https://github.com/apache/beam/pull/28621
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org