You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@samza.apache.org by "mynameborat (via GitHub)" <gi...@apache.org> on 2023/11/01 17:07:49 UTC
[PR] SAMZA-2763: Support worker JVM opts for Samza Beam portable mode [samza]
mynameborat opened a new pull request, #1689:
URL: https://github.com/apache/samza/pull/1689
**Summary**: Support JVM options for worker process in Samza Beam portable mode
Description: With portable mode support for Samza Beam, we want to tune and configure the JVM options for worker process. In this PR, we add support by introducing worker.opts configuration and autosizing integration support.
**Changes**:
- Added worker.opts configuration
- Add autosizing integration support for Xmx
- Updated configuration table and website
**API Changes**: None
**Usage Instructions**: worker.opts can be used similar to other samza application configuration although it only applies to Samza Beam portable execution mode and is ignored otherwise.
**Upgrade Instructions**: None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@samza.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SAMZA-2763: Support worker JVM opts for Samza Beam portable mode [samza]
Posted by "shanthoosh (via GitHub)" <gi...@apache.org>.
shanthoosh commented on code in PR #1689:
URL: https://github.com/apache/samza/pull/1689#discussion_r1391913021
##########
samza-core/src/main/java/org/apache/samza/config/ShellCommandConfig.java:
##########
@@ -97,20 +100,16 @@ public String getCommand() {
}
public Optional<String> getTaskOpts() {
- Optional<String> jvmOpts = Optional.ofNullable(get(ShellCommandConfig.TASK_JVM_OPTS));
- Optional<String> maxHeapMbOptional = Optional.ofNullable(get(JobConfig.JOB_AUTOSIZING_CONTAINER_MAX_HEAP_MB));
- if (new JobConfig(this).getAutosizingEnabled() && maxHeapMbOptional.isPresent()) {
- String maxHeapMb = maxHeapMbOptional.get();
- String xmxSetting = "-Xmx" + maxHeapMb + "m";
- if (jvmOpts.isPresent() && jvmOpts.get().contains("-Xmx")) {
- jvmOpts = Optional.of(jvmOpts.get().replaceAll("-Xmx\\S+", xmxSetting));
- } else if (jvmOpts.isPresent()) {
- jvmOpts = Optional.of(jvmOpts.get().concat(" " + xmxSetting));
- } else {
- jvmOpts = Optional.of(xmxSetting);
- }
- }
- return jvmOpts;
+ String taskOpts = get(ShellCommandConfig.TASK_JVM_OPTS);
+ String autosizingContainerMaxHeap = get(JobConfig.JOB_AUTOSIZING_CONTAINER_MAX_HEAP_MB);
+
+ return Optional.ofNullable(getFinalJvmOptions(taskOpts, autosizingContainerMaxHeap));
+ }
+
+ public Optional<String> getWorkerOpts() {
Review Comment:
Minor: Please add java docs to these methods.
##########
samza-core/src/main/java/org/apache/samza/config/ShellCommandConfig.java:
##########
@@ -97,20 +100,16 @@ public String getCommand() {
}
public Optional<String> getTaskOpts() {
- Optional<String> jvmOpts = Optional.ofNullable(get(ShellCommandConfig.TASK_JVM_OPTS));
- Optional<String> maxHeapMbOptional = Optional.ofNullable(get(JobConfig.JOB_AUTOSIZING_CONTAINER_MAX_HEAP_MB));
- if (new JobConfig(this).getAutosizingEnabled() && maxHeapMbOptional.isPresent()) {
- String maxHeapMb = maxHeapMbOptional.get();
- String xmxSetting = "-Xmx" + maxHeapMb + "m";
- if (jvmOpts.isPresent() && jvmOpts.get().contains("-Xmx")) {
- jvmOpts = Optional.of(jvmOpts.get().replaceAll("-Xmx\\S+", xmxSetting));
- } else if (jvmOpts.isPresent()) {
- jvmOpts = Optional.of(jvmOpts.get().concat(" " + xmxSetting));
- } else {
- jvmOpts = Optional.of(xmxSetting);
- }
- }
- return jvmOpts;
+ String taskOpts = get(ShellCommandConfig.TASK_JVM_OPTS);
+ String autosizingContainerMaxHeap = get(JobConfig.JOB_AUTOSIZING_CONTAINER_MAX_HEAP_MB);
+
+ return Optional.ofNullable(getFinalJvmOptions(taskOpts, autosizingContainerMaxHeap));
+ }
+
+ public Optional<String> getWorkerOpts() {
+ String autosizingWorkerHeapMb = get(JobConfig.JOB_AUTOSIZING_WORKER_MAX_HEAP_MB);
Review Comment:
Do we want to use default value for worker-heap mb if not defined?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@samza.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SAMZA-2763: Support worker JVM opts for Samza Beam portable mode [samza]
Posted by "amansinghoriginal (via GitHub)" <gi...@apache.org>.
amansinghoriginal commented on PR #1689:
URL: https://github.com/apache/samza/pull/1689#issuecomment-1810962790
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@samza.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SAMZA-2763: Support worker JVM opts for Samza Beam portable mode [samza]
Posted by "mynameborat (via GitHub)" <gi...@apache.org>.
mynameborat commented on code in PR #1689:
URL: https://github.com/apache/samza/pull/1689#discussion_r1399802665
##########
samza-core/src/main/java/org/apache/samza/config/ShellCommandConfig.java:
##########
@@ -97,20 +100,16 @@ public String getCommand() {
}
public Optional<String> getTaskOpts() {
- Optional<String> jvmOpts = Optional.ofNullable(get(ShellCommandConfig.TASK_JVM_OPTS));
- Optional<String> maxHeapMbOptional = Optional.ofNullable(get(JobConfig.JOB_AUTOSIZING_CONTAINER_MAX_HEAP_MB));
- if (new JobConfig(this).getAutosizingEnabled() && maxHeapMbOptional.isPresent()) {
- String maxHeapMb = maxHeapMbOptional.get();
- String xmxSetting = "-Xmx" + maxHeapMb + "m";
- if (jvmOpts.isPresent() && jvmOpts.get().contains("-Xmx")) {
- jvmOpts = Optional.of(jvmOpts.get().replaceAll("-Xmx\\S+", xmxSetting));
- } else if (jvmOpts.isPresent()) {
- jvmOpts = Optional.of(jvmOpts.get().concat(" " + xmxSetting));
- } else {
- jvmOpts = Optional.of(xmxSetting);
- }
- }
- return jvmOpts;
+ String taskOpts = get(ShellCommandConfig.TASK_JVM_OPTS);
+ String autosizingContainerMaxHeap = get(JobConfig.JOB_AUTOSIZING_CONTAINER_MAX_HEAP_MB);
+
+ return Optional.ofNullable(getFinalJvmOptions(taskOpts, autosizingContainerMaxHeap));
+ }
+
+ public Optional<String> getWorkerOpts() {
+ String autosizingWorkerHeapMb = get(JobConfig.JOB_AUTOSIZING_WORKER_MAX_HEAP_MB);
Review Comment:
We don't know what would be the ideal default. Leaving it to the JVM to infer this or use its own default for now.
##########
samza-core/src/main/java/org/apache/samza/config/ShellCommandConfig.java:
##########
@@ -97,20 +100,16 @@ public String getCommand() {
}
public Optional<String> getTaskOpts() {
- Optional<String> jvmOpts = Optional.ofNullable(get(ShellCommandConfig.TASK_JVM_OPTS));
- Optional<String> maxHeapMbOptional = Optional.ofNullable(get(JobConfig.JOB_AUTOSIZING_CONTAINER_MAX_HEAP_MB));
- if (new JobConfig(this).getAutosizingEnabled() && maxHeapMbOptional.isPresent()) {
- String maxHeapMb = maxHeapMbOptional.get();
- String xmxSetting = "-Xmx" + maxHeapMb + "m";
- if (jvmOpts.isPresent() && jvmOpts.get().contains("-Xmx")) {
- jvmOpts = Optional.of(jvmOpts.get().replaceAll("-Xmx\\S+", xmxSetting));
- } else if (jvmOpts.isPresent()) {
- jvmOpts = Optional.of(jvmOpts.get().concat(" " + xmxSetting));
- } else {
- jvmOpts = Optional.of(xmxSetting);
- }
- }
- return jvmOpts;
+ String taskOpts = get(ShellCommandConfig.TASK_JVM_OPTS);
+ String autosizingContainerMaxHeap = get(JobConfig.JOB_AUTOSIZING_CONTAINER_MAX_HEAP_MB);
+
+ return Optional.ofNullable(getFinalJvmOptions(taskOpts, autosizingContainerMaxHeap));
+ }
+
+ public Optional<String> getWorkerOpts() {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@samza.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] SAMZA-2763: Support worker JVM opts for Samza Beam portable mode [samza]
Posted by "mynameborat (via GitHub)" <gi...@apache.org>.
mynameborat merged PR #1689:
URL: https://github.com/apache/samza/pull/1689
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@samza.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org