You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/23 15:35:08 UTC
[GitHub] [beam] kamilwu opened a new pull request #11193: [BEAM-9550]
Increase JVM Metaspace size for the TaskExecutors.
kamilwu opened a new pull request #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193
Additionally, replace deprecated `taskmanager.heap.mb` config key to `taskmanager.memory.process.mb`, as suggested in Apache Flink 1.10 Migration Guide.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11193:
[BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#discussion_r405579807
##########
File path: .test-infra/dataproc/init-actions/flink.sh
##########
@@ -155,7 +155,9 @@ function configure_flink() {
# Settings applied by Cloud Dataproc initialization action
jobmanager.rpc.address: ${master_hostname}
jobmanager.heap.mb: ${flink_jobmanager_memory}
-taskmanager.heap.mb: ${flink_taskmanager_memory}
+taskmanager.memory.process.size: "${flink_taskmanager_memory} mb"
+taskmanager.memory.jvm-metaspace.size: 512 mb
+taskmanager.memory.managed.fraction: 0.5
Review comment:
Should we set a fixed amount of memory here? 0.5 could be a lot.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-602678595
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604877826
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-611423066
https://github.com/apache/beam/pull/11291 closed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on a change in pull request #11193:
[BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on a change in pull request #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#discussion_r405643248
##########
File path: .test-infra/dataproc/init-actions/flink.sh
##########
@@ -155,7 +155,9 @@ function configure_flink() {
# Settings applied by Cloud Dataproc initialization action
jobmanager.rpc.address: ${master_hostname}
jobmanager.heap.mb: ${flink_jobmanager_memory}
-taskmanager.heap.mb: ${flink_taskmanager_memory}
+taskmanager.memory.process.size: "${flink_taskmanager_memory} mb"
+taskmanager.memory.jvm-metaspace.size: 512 mb
+taskmanager.memory.managed.fraction: 0.5
Review comment:
Yes, the jobs started working after setting the fraction to 0.5. The default value is 0.4, so the difference might not be that huge.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-602688453
Run Python Load Tests ParDo Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604929753
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-609875720
Run Python Load Tests ParDo Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11193:
[BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#discussion_r397443327
##########
File path: .test-infra/dataproc/init-actions/flink.sh
##########
@@ -155,7 +155,8 @@ function configure_flink() {
# Settings applied by Cloud Dataproc initialization action
jobmanager.rpc.address: ${master_hostname}
jobmanager.heap.mb: ${flink_jobmanager_memory}
-taskmanager.heap.mb: ${flink_taskmanager_memory}
+taskmanager.memory.process.mb: ${flink_taskmanager_memory}
Review comment:
Also this one is `taskmanager.memory.process.size`: https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#taskmanager-memory-process-size
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-603841123
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-611001909
> @mxm Could you take a look at the current changes and tell me if it's fine?
>
> I've tried recently to downgrade to 1.9 [(link)](https://github.com/apache/beam/pull/11291), but that didn't work. Jobs were failing with an error message: `org.apache.beam.runners.core.metrics.MetricsContainerStepMap cannot be cast to org.apache.beam.runners.core.metrics.MetricsContainerStepMap`.
This looks like the Flink version used for submission is a different one than the cluster Flink version.
> I think I'm gonna stay with Flink 1.10 and merge this fix as it is now, so that ParDo and Combine tests would work at least. After that, I'd create a separate JIRA ticket to fix GBK and coGBK tests.
Thanks, I'll have a look.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-603841228
Run Python Load Tests ParDo Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-611054122
> Is it about `--flink_version` parameter? Indeed I forgot to set it to `1.9`
Yes, that seems likely since the job server jar will not match the cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu removed a comment on issue #11193: [BEAM-9550]
Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu removed a comment on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606018706
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu removed a comment on issue #11193: [BEAM-9550]
Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu removed a comment on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604906681
Run Python Load Tests Combine Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu merged pull request #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu merged pull request #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-605928155
Run Load Tests Python CoGBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606475180
> Unfortunately, I have no stack traces or error messages. I had to cancel the jobs manually after a long period of inactivity (up to 4 hours, while normally it takes at most 1h 30 min to complete one test)
Do you have a log from those runs to see in which test it got stuck?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-603366145
@mxm Could you take a look at this? Our python tests, running on Flink cluster on YARN, are flaky since we updated Flink version to 1.10. Most of the time the pipeline is being aborted by this exception: `OutOfMemoryError: Metaspace`, for example here [1] and here [2]. I tried to increase JVM metaspace size (as suggested here: [3]), but it didn't work. Do you have ideas what's going wrong?
[1] https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch_PR/61/console
[2] https://builds.apache.org/job/beam_LoadTests_Python_ParDo_Flink_Batch_PR/17/console
[3] https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/memory/mem_trouble.html
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-609897927
Run Chicago Taxi on Flink
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604908152
Run Python Load Tests Combine Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-611410235
@mxm any other comments? I might merge if it looks fine
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-602692562
Run Load Tests Python CoGBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11193:
[BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#discussion_r397443529
##########
File path: .test-infra/dataproc/init-actions/flink.sh
##########
@@ -46,7 +46,7 @@ readonly FLINK_NETWORK_NUM_BUFFERS=2048
readonly FLINK_JOBMANAGER_MEMORY_FRACTION='1.0'
# Heap memory used by the task managers (slaves) determined by the physical (free) memory of the servers.
-# Flink config entry: taskmanager.heap.mb.
+# Flink config entry: taskmanager.memory.process.mb.
Review comment:
```suggestion
# Flink config entry: taskmanager.memory.process.size
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606054405
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-603808862
Run Chicago Taxi on Flink
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-605009385
@mxm Thanks, the option names are now correct and properly recognized by Flink.
I had one more problem. GroupByKey (as well as coGroupByKey) tests got stuck at some point and their progress didn't change despite the fact that the job was running. It seems changing the execution mode to `BATCH_FORCED` solved the problem. It is fine to keep this mode enabled permanently?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-608492373
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604537182
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11193:
[BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#discussion_r405648025
##########
File path: .test-infra/dataproc/init-actions/flink.sh
##########
@@ -155,7 +155,9 @@ function configure_flink() {
# Settings applied by Cloud Dataproc initialization action
jobmanager.rpc.address: ${master_hostname}
jobmanager.heap.mb: ${flink_jobmanager_memory}
-taskmanager.heap.mb: ${flink_taskmanager_memory}
+taskmanager.memory.process.size: "${flink_taskmanager_memory} mb"
+taskmanager.memory.jvm-metaspace.size: 512 mb
+taskmanager.memory.managed.fraction: 0.5
Review comment:
Sounds good!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11193:
[BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#discussion_r397443001
##########
File path: .test-infra/dataproc/init-actions/flink.sh
##########
@@ -155,7 +155,8 @@ function configure_flink() {
# Settings applied by Cloud Dataproc initialization action
jobmanager.rpc.address: ${master_hostname}
jobmanager.heap.mb: ${flink_jobmanager_memory}
-taskmanager.heap.mb: ${flink_taskmanager_memory}
+taskmanager.memory.process.mb: ${flink_taskmanager_memory}
+taskmanager.memory.jvm-metaspace.mb: 512
Review comment:
The correct option name is: `taskmanager.memory.jvm-metaspace.size`: https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/config.html#taskmanager-memory-jvm-metaspace-size
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-607236502
Increasing the memory is fine. The memory configuration has changed in Flink 1.10, so we need to tune these anyways. Downgrading to 1.9 could also be an option if that does not work.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-611424574
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-611421088
Feel free to merge. Looks good.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606587808
At the moment, I'm considering three options:
1. Downgrade Flink version to 1.9 for all load tests
2. Set execution mode to `BATCH_FORCED` for GBK and coGBK tests
3. Increase the size of Managed Memory/optimize test scenarios
3) is probably the best, but for now, because of other priorities that I have on my to do list, I'd rather go 1) and get back to this problem in the future.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606035132
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-605011286
> @mxm Thanks, the option names are now correct and properly recognized by Flink.
Nice.
> I had one more problem. GroupByKey (as well as coGroupByKey) tests got stuck at some point and their progress didn't change despite the fact that the job was running. It seems changing the execution mode to `BATCH_FORCED` solved the problem. It is fine to keep this mode enabled permanently?
I wouldn't keep it enabled permanently. The pipelined mode generally provides better performance. Do you have a stack trace or error message? Looks like something to take a look at. The support for Flink 1.10 is still early and we need to iron out these issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-603840955
Run Load Tests Python CoGBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-605928022
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu removed a comment on issue #11193: [BEAM-9550]
Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu removed a comment on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606035132
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606018706
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-605016809
Unfortunately, I have no stack traces or error messages. I had to cancel the jobs manually after a long period of inactivity (up to 4 hours, while normally it takes at most 1h 30 min to complete one test)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-602688839
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604906637
Run Python Load Tests ParDo Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604540494
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606581424
@mxm I checked two tests from [GBK suite](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_LoadTests_GBK_Flink_Python.groovy) and one test from [coGBK suite](https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_LoadTests_coGBK_Flink_Python.groovy). Each of them got stuck at some point. It's highly possible that every test from these suites is affected, because the size of input is the same in all tests.
I observed that the problem might be connected with a lack of Managed Memory. After increasing `memory.managed.fraction` by `0.1`, the first two tests from GBK suite passed, and a pipeline got stuck at the third test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-609875962
Run Load Tests Python Combine Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu edited a comment on issue #11193: [BEAM-9550]
Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu edited a comment on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606582028
This is a log from today's attempt: https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch_PR/70/consoleFull
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606582028
This this is a log from today's attempt: https://builds.apache.org/job/beam_LoadTests_Python_GBK_Flink_Batch_PR/70/consoleFull
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu removed a comment on issue #11193: [BEAM-9550]
Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu removed a comment on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604908152
Run Python Load Tests Combine Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604905471
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-606458176
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-611047675
Thanks @mxm .
> This looks like the Flink version used for submission is a different one than the cluster Flink version.
Is it about `--flink_version` parameter? Indeed I forgot to set it to `1.9`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604545009
Run Load Tests Python CoGBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-607237113
Please consider increasing the managed memory globally, we haven't settled for good values there yet; this was just an initial guess when merging #10945.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on a change in pull request #11193:
[BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on a change in pull request #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#discussion_r405581569
##########
File path: .test-infra/dataproc/init-actions/flink.sh
##########
@@ -155,7 +155,9 @@ function configure_flink() {
# Settings applied by Cloud Dataproc initialization action
jobmanager.rpc.address: ${master_hostname}
jobmanager.heap.mb: ${flink_jobmanager_memory}
-taskmanager.heap.mb: ${flink_taskmanager_memory}
+taskmanager.memory.process.size: "${flink_taskmanager_memory} mb"
+taskmanager.memory.jvm-metaspace.size: 512 mb
+taskmanager.memory.managed.fraction: 0.5
Review comment:
If you needed 0.5, then that's fine.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-610306012
@mxm Could you take a look at the current changes and tell me if it's fine?
I've tried recently to downgrade to 1.9 [(link)](https://github.com/apache/beam/pull/11291), but that didn't work. Jobs were failing with an error message: `org.apache.beam.runners.core.metrics.MetricsContainerStepMap cannot be cast to org.apache.beam.runners.core.metrics.MetricsContainerStepMap`.
I think I'm gonna stay with Flink 1.10 and merge this fix as it is now, so that ParDo and Combine tests would work at least. After that, I'd create a separate JIRA ticket to fix GBK and coGBK tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-608492571
Run Chicago Taxi on Flink
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604544950
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-602692837
Run Chicago Taxi on Flink
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604906681
Run Python Load Tests Combine Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604934554
Run Load Tests Python GBK Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-609875556
Run Chicago Taxi on Flink
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604702445
Run Seed Job
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] mxm commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
mxm commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-611421666
Btw, can we close https://github.com/apache/beam/pull/11291?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] kamilwu commented on issue #11193: [BEAM-9550] Increase JVM
Metaspace size for the TaskExecutors.
Posted by GitBox <gi...@apache.org>.
kamilwu commented on issue #11193: [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors.
URL: https://github.com/apache/beam/pull/11193#issuecomment-604934972
Run Load Tests Python Combine Flink Batch
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services