You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/05/19 05:27:33 UTC
[GitHub] [incubator-doris] hf200012 opened a new pull request, #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
hf200012 opened a new pull request, #9683:
URL: https://github.com/apache/incubator-doris/pull/9683
When many users start fe in the production environment, they do not modify the JAVA_OPTS configuration according to the document, and there is an OOM problem. The modification here will judge the available memory of the user's machine when the user starts fe. If it is greater than 8G, directly set the JAVA_OPTS option in the The memory is set to 8G, and when it is less than 8G, it is set to 4G
Set the memory size used when FE starts according to the user's available memory at startup
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
hf200012 commented on PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#issuecomment-1133522546
Here is the main solution to the problem of not modifying the default 4G memory in the user's reproduction environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
hf200012 commented on PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#issuecomment-1131570860
> will it work if I have set fe jvm config? what if user need memory for 36G ?
Modify the JAVA_OPTS_8G or JAVA_OPTS_FOR_JDK_9_8G parameter in fe.conf
This is just to avoid the OOM problem that may be caused by users using the default 4G memory
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] adonis0147 commented on a diff in pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
adonis0147 commented on code in PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#discussion_r877847296
##########
conf/fe.conf:
##########
@@ -25,11 +25,14 @@
LOG_DIR = ${DORIS_HOME}/log
DATE = `date +%Y%m%d-%H%M%S`
-JAVA_OPTS="-Xmx4096m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"
+JAVA_OPTS_8G="-Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"
+
+JAVA_OPTS_4G="-Xmx4096m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"
# For jdk 9+, this JAVA_OPTS will be used as default JVM options
-JAVA_OPTS_FOR_JDK_9="-Xmx4096m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$DATE:time"
+JAVA_OPTS_FOR_JDK_9_8G="-Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$DATE:time"
+JAVA_OPTS_FOR_JDK_9_4G="-Xmx4096m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$DATE:time"
Review Comment:
Looks verbose. It is better to use a variable to set the size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
caiconghui commented on PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#issuecomment-1131565834
will it work if I have set fe jvm config? what if user need memory for 36G ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] adonis0147 commented on a diff in pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
adonis0147 commented on code in PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#discussion_r876687681
##########
bin/start_fe.sh:
##########
@@ -132,13 +132,21 @@ fi
# check java version and choose correct JAVA_OPTS
java_version=$(jdk_version)
-final_java_opt=$JAVA_OPTS
+final_java_opt=$JAVA_OPTS_4G
+MemTotal=`free -g | grep "Mem" | awk '{print $7}'`
Review Comment:
The 7th item indicates the size of cached memory which may not be suitable.
```shell
MemTotalInGB="$(cat /proc/meminfo | grep 'MemTotal' | awk '{print int($2 / 1024 / 1024)}')"
MemAvailableInGB="$(cat /proc/meminfo | grep 'MemAvailable' | awk '{print int($2 / 1024 / 1024)}')"
```
These metrics may be useful.
MemTotalInGB = total
MemAvailableInGB = free + buffers/cache
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
morningman commented on PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#issuecomment-1145552785
> why modify the default Xmx from 4G to 8G can solve oom problem?
No, just because many new user does not modify the default value. And 4GB is too small for default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morrySnow commented on a diff in pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#discussion_r876625332
##########
bin/start_fe.sh:
##########
@@ -132,13 +132,21 @@ fi
# check java version and choose correct JAVA_OPTS
java_version=$(jdk_version)
-final_java_opt=$JAVA_OPTS
+final_java_opt=$JAVA_OPTS_4G
+MemTotal=`free -g | grep "Mem" | awk '{print $7}'`
Review Comment:
free is not available on all Linux Distribution. we'd better to do `which free` before use it. BTW, extra space aflter "Mem"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
morningman commented on PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#issuecomment-1140399058
It is too complex, I think we can just modify the default Xmx to 8G to resolve this problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#issuecomment-1145497676
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morrySnow commented on a diff in pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
morrySnow commented on code in PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#discussion_r876625332
##########
bin/start_fe.sh:
##########
@@ -132,13 +132,21 @@ fi
# check java version and choose correct JAVA_OPTS
java_version=$(jdk_version)
-final_java_opt=$JAVA_OPTS
+final_java_opt=$JAVA_OPTS_4G
+MemTotal=`free -g | grep "Mem" | awk '{print $7}'`
Review Comment:
free is not available on all Linux Distribution. we'd better to do `which free` before use it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on a diff in pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
hf200012 commented on code in PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#discussion_r876907925
##########
bin/start_fe.sh:
##########
@@ -132,13 +132,21 @@ fi
# check java version and choose correct JAVA_OPTS
java_version=$(jdk_version)
-final_java_opt=$JAVA_OPTS
+final_java_opt=$JAVA_OPTS_4G
+MemTotal=`free -g | grep "Mem" | awk '{print $7}'`
Review Comment:
thx, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] hf200012 commented on pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
hf200012 commented on PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#issuecomment-1133522582
> I think we should forbid FE from starting when the machine lacks memory. In such way, users will modify the conf by themselves.
Here is the main solution to the problem of not modifying the default 4G memory in the user's reproduction environment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] caiconghui commented on pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
caiconghui commented on PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#issuecomment-1145529002
why modify the default Xmx from 4G to 8G can solve oom problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] adonis0147 commented on pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
adonis0147 commented on PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#issuecomment-1132593294
I think we should forbid FE from starting when the machine lacks memory. In such way, users will modify the conf by themselves.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683#issuecomment-1145497693
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #9683: [fix]Set the memory size used when FE starts according to the user's available memory at startup
Posted by GitBox <gi...@apache.org>.
morningman merged PR #9683:
URL: https://github.com/apache/incubator-doris/pull/9683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org