You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/03/01 21:37:23 UTC
[GitHub] [beam] Fokko opened a new pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Fokko opened a new pull request #14122:
URL: https://github.com/apache/beam/pull/14122
**Please** add a meaningful description for your change here. I would like to remove the PrettyPrint from the output. This decreases the size of the job definition:
```
With the PR:
-rw-r--r-- 1 fdriesprong wheel 45M Mar 1 22:27 /tmp/pubsub-to-bigquery.json
With 2.28.0:
-rw-r--r-- 1 fdriesprong wheel 47M Mar 1 22:30 /tmp/pubsub-to-bigquery.json
```
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [x] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment.
- [x] Format the pull request title like `[BEAM-11904] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Dataflow | Flink | Samza | Spark | Twister2
--- | --- | --- | --- | --- | --- | ---
Go | [](https://ci-beam.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) | ---
Java | [](https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://ci-beam
.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Twister2/lastCompletedBuild/)
Python | [](https://ci-beam.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python38/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) | ---
XLang | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Direct/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | [](https://ci-beam.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/) | ---
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website | Whitespace | Typescript
--- | --- | --- | --- | --- | --- | ---
Non-portable | [](https://ci-beam.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/)<br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Cron/lastCompletedBuild/) <br>[](https://ci-beam.apache.org/job/beam_PreCommit_PythonDocs_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/lastCompletedBuild/) | [](https://ci-beam.apache.org/job/beam_PreCommit_Typescript_Cron/lastCompletedBuild/)
Portable | --- | [](https://ci-beam.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-794301999
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-788342791
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-788359120
hm right, because the UI also passes the job graph in an API call...
I guess a good next-step is to change how we include schemas in the graph to have a dictionary of some sort...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-790147525
Run Java PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem edited a comment on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
pabloem edited a comment on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-788339926
Can you try using `--experiments=upload_graph`? This should upload your graph to GCS instead of an HTTP request
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Fokko commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
Fokko commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-788767476
@pabloem Seems like I'm unable to see the stacktrace of the failed test, can you confirm?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Fokko commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
Fokko commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-790465036
@pabloem Do you think this is something that we can move forward?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] dpcollins-google commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
dpcollins-google commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-788329463
R: @pabloem
I don't know if this would affect anything in the dataflow backend? But AFAICT this would result in more compressed JSON and fewer user jobs over the limit.
@Fokko, I don't know that this solves your issue. Even after this change, your job graph size is still over the limit (20 MiB IIUC?) https://cloud.google.com/dataflow/docs/guides/common-errors#job-graph-too-large
It might be good to look at what is in the serialized JSON payload for your job, see if there's any multi-MiB blobs of data that you shouldn't be serializing?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Fokko commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
Fokko commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-788351247
@pabloem Thanks for the pointer. It looks good at first glance. The UI seems to become unresponsive: _Graph monitoring data exceeded maximum allowable size. Some monitoring data including counters may be incomplete._ But if it works, that would be okay for now. :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Fokko commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
Fokko commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-788335682
We're writing data from PubSub to BigQuery, and there are Avro messages that we know upfront, and that we decode from PubSub. I don't see anything huge, but the `serialized_source` and the `serialized_fn` are rather large (not big). I think the schema is represented multiple times in the graph, each time it is being passed to the next stage ((de)serializing).
@dpcollins-google Thanks for the quick reply. I'm iterating towards a <20MB job, which is indeed the limit. I noticed that it was pretty printed, so that was a quick win of around 5%.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #14122:
URL: https://github.com/apache/beam/pull/14122
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-788341591
let me know if that works, and if so, I can get it on stackoverflow / documentation for others
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-788339926
Can you try using `--experiments=upload_graph`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-788333007
right - often closures with larger-than-intended context can explode the job graph size
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-789115677
that's correct - though the failed test is broken on the main branch: https://ci-beam.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/testReport/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Fokko commented on pull request #14122: [BEAM-11904] Remove PrettyPrint from job definition
Posted by GitBox <gi...@apache.org>.
Fokko commented on pull request #14122:
URL: https://github.com/apache/beam/pull/14122#issuecomment-796025142
Thanks @pabloem
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org