You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "liferoad (via GitHub)" <gi...@apache.org> on 2023/09/22 17:27:29 UTC

[GitHub] [beam] liferoad opened a new pull request, #28621: Automatically Switch to upload_graph when the graph is large

liferoad opened a new pull request, #28621:
URL: https://github.com/apache/beam/pull/28621

   We might need to update the CHANGES.md since the behavior is changed.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1368990729


##########
CHANGES.md:
##########
@@ -72,6 +72,8 @@ should handle this. ([#25252](https://github.com/apache/beam/issues/25252)).
 
 ## Breaking Changes
 
+* `upload_graph` is not needed since the graph now is uploaded automatically when it is larger than 10MB for Java SDK ([PR#28621](https://github.com/apache/beam/pull/28621).

Review Comment:
   This does not sound like a `breaking change` instead of an improvement, i.e., in which scenario this would break existing user?
   
   If so, consider phrasing like: `upload_graph` experiments option is no longer required in the case of large pipeline graphs ...
   



##########
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java:
##########
@@ -256,6 +256,22 @@ private Pipeline buildDataflowPipeline(DataflowPipelineOptions options) {
     return p;
   }
 
+  private Pipeline buildDataflowPipelineWithLargeGraph(DataflowPipelineOptions options) {

Review Comment:
   minor: this method and the previous one (buildDataflowPipeline) could be static



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically switch to upload_graph when the graph is large [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1781712301

   Once it is green again (for the record) will merge. Or any committer can merge.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically switch to upload_graph when the graph is large [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1782108615

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically switch to upload_graph when the graph is large [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1782138488

   Note `"org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest.testActiveThreadMetric[*]"` seems flaky. I tried several times to get all greens.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] liferoad commented on pull request #28621: Automatically Switch to upload_graph when the graph is large

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1741802110

   R: @Abacn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically switch to upload_graph when the graph is large [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1781899823

   Run Java PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically switch to upload_graph when the graph is large [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1373622952


##########
CHANGES.md:
##########
@@ -72,6 +72,8 @@ should handle this. ([#25252](https://github.com/apache/beam/issues/25252)).
 
 ## Breaking Changes
 
+* `upload_graph` is not needed since the graph now is uploaded automatically when it is larger than 10MB for Java SDK ([PR#28621](https://github.com/apache/beam/pull/28621).

Review Comment:
   Done. Thanks.



##########
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java:
##########
@@ -256,6 +256,22 @@ private Pipeline buildDataflowPipeline(DataflowPipelineOptions options) {
     return p;
   }
 
+  private Pipeline buildDataflowPipelineWithLargeGraph(DataflowPipelineOptions options) {

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] liferoad commented on pull request #28621: Automatically Switch to upload_graph when the graph is large

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1741802072

   Tested these changes internally. The test passed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically switch to upload_graph when the graph is large [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1782180824

   > Note `"org.apache.beam.runners.dataflow.worker.StreamingDataflowWorkerTest.testActiveThreadMetric[*]"` seems flaky. I tried several times to get all greens.
   
   Yes this is unrelated. Tracking bug: #28957 caused by #28513 there is a pending PR include a fix: https://github.com/apache/beam/pull/29041#issuecomment-1782121904


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1367467116


##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java:
##########
@@ -1331,6 +1331,14 @@ public DataflowPipelineJob run(Pipeline pipeline) {
       hooks.modifyEnvironmentBeforeSubmission(newJob.getEnvironment());
     }
 
+    // enable upload_graph when the graph is too large
+    if ((Utf8.encodedLength(newJob.toString()) >= CREATE_JOB_REQUEST_LIMIT_BYTES)

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1368991768


##########
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java:
##########
@@ -824,6 +840,24 @@ public void testUploadGraph() throws IOException {
             .startsWith("gs://valid-bucket/temp/staging/dataflow_graph"));
   }
 
+  /** Test for automatically using upload_graph when the job graph is too large (>10MB). */
+  @Test
+  public void testUploadGraphWithAutoUpload() throws IOException {
+    DataflowPipelineOptions options = buildPipelineOptions();
+    Pipeline p = buildDataflowPipelineWithLargeGraph(options);
+    p.run();
+
+    ArgumentCaptor<Job> jobCaptor = ArgumentCaptor.forClass(Job.class);
+    Mockito.verify(mockJobs).create(eq(PROJECT_ID), eq(REGION_ID), jobCaptor.capture());
+    assertValidJob(jobCaptor.getValue());
+    assertTrue(jobCaptor.getValue().getSteps().isEmpty());
+    assertTrue(
+        jobCaptor
+            .getValue()
+            .getStepsLocation()
+            .startsWith("gs://valid-bucket/temp/staging/dataflow_graph"));

Review Comment:
   nit: here and in the other tests `gs://valid-bucket/temp/staging/` should be part of the test case. It is not clear from the test if it is:
   
   1. Part of the spec for this feature
   2. A constant  behavior of the code under test (in which case it should be a reference to that)
   3. A value pass in from global test set up (this is an anti-pattern)
   
   Just noting this - the whole file has this problem a lot and you don't have to fix it.



##########
CHANGES.md:
##########
@@ -72,6 +72,8 @@ should handle this. ([#25252](https://github.com/apache/beam/issues/25252)).
 
 ## Breaking Changes
 
+* `upload_graph` is not needed since the graph now is uploaded automatically when it is larger than 10MB for Java SDK ([PR#28621](https://github.com/apache/beam/pull/28621).

Review Comment:
   It won't break anyone though. it is more like a new cool feature.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1773366164

   > Let us get a test externally. It should be very easy - just a single small test method with a big graph.
   
   Added one unit test.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically switch to upload_graph when the graph is large [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1782818716

   I think it is OK to disable the test or even better to roll back the change so we do not have untested stuff in the codebase.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] liferoad commented on pull request #28621: Automatically Switch to upload_graph when the graph is large

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1734277175

   I plan to manually test this later (might add one integration test). So no need to merge this now. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [beam] github-actions[bot] commented on pull request #28621: Automatically Switch to upload_graph when the graph is large

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1741802631

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on PR #28621:
URL: https://github.com/apache/beam/pull/28621#issuecomment-1743099675

   Let us get a test externally. It should be very easy - just a single small test method with a big graph.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically Switch to upload_graph when the graph is large [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1344439659


##########
runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java:
##########
@@ -1331,6 +1331,14 @@ public DataflowPipelineJob run(Pipeline pipeline) {
       hooks.modifyEnvironmentBeforeSubmission(newJob.getEnvironment());
     }
 
+    // enable upload_graph when the graph is too large
+    if ((Utf8.encodedLength(newJob.toString()) >= CREATE_JOB_REQUEST_LIMIT_BYTES)

Review Comment:
   The calculation of `Utf8.encodedLength(newJob.toString())` could be expensive especially when the graph is large. Similar calculation include
   
   - L1349: DataflowPipelineTranslator.jobToString(newJob).getBytes(UTF_8) when upload the graph
   - L1410: if (Utf8.encodedLength(newJob.toString()) >= CREATE_JOB_REQUEST_LIMIT_BYTES) when decide the error message
   
   We should be able to just run the conversion of `Job->utf8 encoded bytes` once



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically switch to upload_graph when the graph is large [beam]

Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on code in PR #28621:
URL: https://github.com/apache/beam/pull/28621#discussion_r1373620864


##########
runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java:
##########
@@ -824,6 +840,24 @@ public void testUploadGraph() throws IOException {
             .startsWith("gs://valid-bucket/temp/staging/dataflow_graph"));
   }
 
+  /** Test for automatically using upload_graph when the job graph is too large (>10MB). */
+  @Test
+  public void testUploadGraphWithAutoUpload() throws IOException {
+    DataflowPipelineOptions options = buildPipelineOptions();
+    Pipeline p = buildDataflowPipelineWithLargeGraph(options);
+    p.run();
+
+    ArgumentCaptor<Job> jobCaptor = ArgumentCaptor.forClass(Job.class);
+    Mockito.verify(mockJobs).create(eq(PROJECT_ID), eq(REGION_ID), jobCaptor.capture());
+    assertValidJob(jobCaptor.getValue());
+    assertTrue(jobCaptor.getValue().getSteps().isEmpty());
+    assertTrue(
+        jobCaptor
+            .getValue()
+            .getStepsLocation()
+            .startsWith("gs://valid-bucket/temp/staging/dataflow_graph"));

Review Comment:
   Noted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Automatically switch to upload_graph when the graph is large [beam]

Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #28621:
URL: https://github.com/apache/beam/pull/28621


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org