You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2020/01/15 09:37:23 UTC
[GitHub] [flink] xintongsong opened a new pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
xintongsong opened a new pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860
## What is the purpose of the change
This PR updates FLIP-49 TM memory sizes configuration default values according to the outcome of tuning with real jobs.
## Brief change log
- JVM overhead min: 128MB -> 192MB
- JVM metaspace: 128MB -> 96MB
- Total process size (in default flink-conf.yaml): 1024MB -> 1568MB
## Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
- The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no)
- The S3 file system connector: (no)
## Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574591065
<!--
Meta data
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144485357 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144496504 TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4366 TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
Hash:3ac114c589d40aed1ecd0267174941e50a423fc2 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144659236 TriggerType:PUSH TriggerID:3ac114c589d40aed1ecd0267174941e50a423fc2
Hash:3ac114c589d40aed1ecd0267174941e50a423fc2 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4381 TriggerType:PUSH TriggerID:3ac114c589d40aed1ecd0267174941e50a423fc2
-->
## CI report:
* 5f9cd2224ef458e5068f3179aa5be025848f8723 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144485357) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363)
* f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144496504) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4366)
* 3ac114c589d40aed1ecd0267174941e50a423fc2 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144659236) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4381)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366795773
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,9 +42,12 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskManager.
+#
+# Note this accounts for all memory usages of a TaskManager process, including JVM metaspace and other overheads.
+# To exclude JVM metaspace and other overheads, please use total flink memory size (taskmanager.memory.flink.size) instead.
-taskmanager.memory.process.size: 1024m
+taskmanager.memory.process.size: 1568m
Review comment:
```suggestion
# To exclude JVM metaspace and overhead, please, use total Flink memory size instead of 'taskmanager.memory.process.size'.
# It is not recommended to set both 'taskmanager.memory.process.size' and Flink memory.
# This option is similar to the deprecated option 'taskmanager.heap.size' for standalone environment:
#
# taskmanager.memory.flink.size: 1280
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574591065
<!--
Meta data
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144485357 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144496504 TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4366 TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
Hash:3ac114c589d40aed1ecd0267174941e50a423fc2 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144659236 TriggerType:PUSH TriggerID:3ac114c589d40aed1ecd0267174941e50a423fc2
Hash:3ac114c589d40aed1ecd0267174941e50a423fc2 Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4381 TriggerType:PUSH TriggerID:3ac114c589d40aed1ecd0267174941e50a423fc2
-->
## CI report:
* 5f9cd2224ef458e5068f3179aa5be025848f8723 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144485357) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363)
* f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144496504) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4366)
* 3ac114c589d40aed1ecd0267174941e50a423fc2 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144659236) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4381)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574591065
<!--
Meta data
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144485357 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144496504 TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4366 TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
-->
## CI report:
* 5f9cd2224ef458e5068f3179aa5be025848f8723 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144485357) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363)
* f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144496504) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4366)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574591065
<!--
Meta data
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:PENDING URL:https://travis-ci.com/flink-ci/flink/builds/144485357 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
-->
## CI report:
* 5f9cd2224ef458e5068f3179aa5be025848f8723 Travis: [PENDING](https://travis-ci.com/flink-ci/flink/builds/144485357) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366795773
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,9 +42,12 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskManager.
+#
+# Note this accounts for all memory usages of a TaskManager process, including JVM metaspace and other overheads.
+# To exclude JVM metaspace and other overheads, please use total flink memory size (taskmanager.memory.flink.size) instead.
-taskmanager.memory.process.size: 1024m
+taskmanager.memory.process.size: 1568m
Review comment:
```suggestion
# To exclude JVM metaspace and overhead, please, use total Flink memory size instead of 'taskmanager.memory.process.size'.
# It is not recommended to set both 'taskmanager.memory.process.size' and Flink memory.
#
# taskmanager.memory.flink.size: 1280
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366991374
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,9 +42,12 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskManager.
+#
+# Note this accounts for all memory usages of a TaskManager process, including JVM metaspace and other overheads.
Review comment:
Same here with `TaskExecutor` -> `TaskManager`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366787995
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,9 +42,12 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskManager.
+#
+# Note this accounts for all memory usages of a TaskManager process, including JVM metaspace and other overheads.
Review comment:
```suggestion
# Note this accounts for all memory usage within the TaskExecutor process, including JVM metaspace and other overhead.
# For containerised environment (Yarn/Mesos), it is similar to the deprecated option
# 'taskmanager.heap.size' which included the deprecated 'containerized.heap-cutoff*'.
taskmanager.memory.process.size: 1568m
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on issue #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-575030417
Thanks @xintongsong for addressing my comments so quickly. Merging this PR now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574591065
<!--
Meta data
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144485357 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144496504 TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4366 TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
-->
## CI report:
* 5f9cd2224ef458e5068f3179aa5be025848f8723 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144485357) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363)
* f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144496504) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4366)
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366787228
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,9 +42,12 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskManager.
+#
+# Note this accounts for all memory usages of a TaskManager process, including JVM metaspace and other overheads.
+# To exclude JVM metaspace and other overheads, please use total flink memory size (taskmanager.memory.flink.size) instead.
Review comment:
```suggestion
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10860: [FLINK-15145][config]
Change TM memory configuration default values for FLIP-49.
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574591065
<!--
Meta data
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
-->
## CI report:
* 5f9cd2224ef458e5068f3179aa5be025848f8723 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366991253
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,9 +42,12 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskManager.
Review comment:
In the code base it is correct that we call it `TaskExecutor`. However, we never fully transitioned from `TaskManager` to `TaskExecutor` in our documentation. Given that the memory config options are prefixed with `taskmanager`, I would suggest to stick with `TaskManager` for the time being.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366787685
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,9 +42,12 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskManager.
Review comment:
```suggestion
# The total process memory size for the TaskExecutor.
```
I think this the most recent term for task manager after FLIP-6
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on issue #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
xintongsong commented on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574942430
Thank you @tillrohrmann.
I've addressed your comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366993379
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,10 +42,16 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskExecutor.
+#
+# Note this accounts for all memory usage within the TaskExecutor process, including JVM metaspace and other overhead.
Review comment:
Same here with `TaskExecutor` and `TaskManager`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574591065
<!--
Meta data
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144485357 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144496504 TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:FAILURE URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4366 TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
Hash:3ac114c589d40aed1ecd0267174941e50a423fc2 Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:3ac114c589d40aed1ecd0267174941e50a423fc2
-->
## CI report:
* 5f9cd2224ef458e5068f3179aa5be025848f8723 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144485357) Azure: [SUCCESS](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363)
* f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144496504) Azure: [FAILURE](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4366)
* 3ac114c589d40aed1ecd0267174941e50a423fc2 UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366795773
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,9 +42,12 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskManager.
+#
+# Note this accounts for all memory usages of a TaskManager process, including JVM metaspace and other overheads.
+# To exclude JVM metaspace and other overheads, please use total flink memory size (taskmanager.memory.flink.size) instead.
-taskmanager.memory.process.size: 1024m
+taskmanager.memory.process.size: 1568m
Review comment:
```suggestion
# To exclude JVM metaspace and overhead, please, use total Flink memory size instead of 'taskmanager.memory.process.size'.
# It is not recommended to set both 'taskmanager.memory.process.size' and Flink memory.
#
# taskmanager.memory.flink.size: 1280
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10860: [FLINK-15145][config]
Change TM memory configuration default values for FLIP-49.
Posted by GitBox <gi...@apache.org>.
flinkbot commented on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574577044
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community
to review your pull request. We will use this comment to track the progress of the review.
## Automated Checks
Last check on commit 5f9cd2224ef458e5068f3179aa5be025848f8723 (Wed Jan 15 09:40:04 UTC 2020)
**Warnings:**
* No documentation files were touched! Remember to keep the Flink docs up to date!
<sub>Mention the bot in a comment to re-run the automated checks.</sub>
## Review Progress
* ❓ 1. The [description] looks good.
* ❓ 2. There is [consensus] that the contribution should go into to Flink.
* ❓ 3. Needs [attention] from.
* ❓ 4. The change fits into the overall [architecture].
* ❓ 5. Overall code [quality] is good.
Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process.<details>
The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required <summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`)
- `@flinkbot approve all` to approve all aspects
- `@flinkbot approve-until architecture` to approve everything until `architecture`
- `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention
- `@flinkbot disapprove architecture` to remove an approval you gave earlier
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
azagrebin commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366787995
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,9 +42,12 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskManager.
+#
+# Note this accounts for all memory usages of a TaskManager process, including JVM metaspace and other overheads.
Review comment:
```suggestion
# Note this accounts for all memory usage within the TaskExecutor process, including JVM metaspace and other overhead.
taskmanager.memory.process.size: 1568m
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] xintongsong commented on issue #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
xintongsong commented on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574613727
Thanks for the review, @azagrebin.
Comments addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann commented on a change in pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
tillrohrmann commented on a change in pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#discussion_r366993092
##########
File path: flink-dist/src/main/resources/flink-conf.yaml
##########
@@ -42,10 +42,16 @@ jobmanager.rpc.port: 6123
jobmanager.heap.size: 1024m
-# The heap size for the TaskManager JVM
+# The total process memory size for the TaskExecutor.
Review comment:
I would suggest to use the term `TaskExecutor` for time being because we have never really changed the name in the documetation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] tillrohrmann closed pull request #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
tillrohrmann closed pull request #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10860:
[FLINK-15145][config] Change TM memory configuration default values for
FLIP-49.
Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on issue #10860: [FLINK-15145][config] Change TM memory configuration default values for FLIP-49.
URL: https://github.com/apache/flink/pull/10860#issuecomment-574591065
<!--
Meta data
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:SUCCESS URL:https://travis-ci.com/flink-ci/flink/builds/144485357 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:5f9cd2224ef458e5068f3179aa5be025848f8723 Status:PENDING URL:https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363 TriggerType:PUSH TriggerID:5f9cd2224ef458e5068f3179aa5be025848f8723
Hash:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d Status:UNKNOWN URL:TBD TriggerType:PUSH TriggerID:f57ca417ecb8c6206a0f7b8617776d922ba9cd7d
-->
## CI report:
* 5f9cd2224ef458e5068f3179aa5be025848f8723 Travis: [SUCCESS](https://travis-ci.com/flink-ci/flink/builds/144485357) Azure: [PENDING](https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_build/results?buildId=4363)
* f57ca417ecb8c6206a0f7b8617776d922ba9cd7d UNKNOWN
<details>
<summary>Bot commands</summary>
The @flinkbot bot supports the following commands:
- `@flinkbot run travis` re-run the last Travis build
- `@flinkbot run azure` re-run the last Azure build
</details>
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services