You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "sarvekshayr (via GitHub)" <gi...@apache.org> on 2024/07/22 04:03:01 UTC
[PR] HDDS-10382. Optimize Netty memory allocation by avoiding zero assignment in Java 9+ [ozone]
sarvekshayr opened a new pull request, #6975:
URL: https://github.com/apache/ozone/pull/6975
## What changes were proposed in this pull request?
Added `-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true` to the ozone script to allow Netty unsafe memory allocation and avoid the zeroing cost for Java 9+.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-10382
## How was this patch tested?
Tested the patch in docker by changing the Java version.
```
recon-1 | Java version string: openjdk version "17.0.2" 2022-01-18
recon-1 | OpenJDK Runtime Environment (build 17.0.2+8-86)
recon-1 | OpenJDK 64-Bit Server VM (build 17.0.2+8-86, mixed mode, sharing)
recon-1 | Java major version: 17
recon-1 | NETTY_OPTS: -Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10382. Optimize Netty memory allocation by avoiding zero assignment in Java 9+ [ozone]
Posted by "tanvipenumudy (via GitHub)" <gi...@apache.org>.
tanvipenumudy commented on code in PR #6975:
URL: https://github.com/apache/ozone/pull/6975#discussion_r1688336668
##########
hadoop-ozone/dist/src/shell/ozone/ozone:
##########
@@ -81,6 +81,21 @@ function ozonecmd_case
# Corresponding Ratis issue https://issues.apache.org/jira/browse/RATIS-534.
RATIS_OPTS="-Dorg.apache.ratis.thirdparty.io.netty.allocator.useCacheForAllThreads=false ${RATIS_OPTS}"
+ # Add JVM parameter for Java 9+
+ # (org.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true) to allow Netty unsafe memory allocation.
+ # Corresponding issue https://issues.apache.org/jira/browse/HDDS-10382.
+ NETTY_OPTS=""
+
+ # Get the version string
+ JAVA_VERSION_STRING=$(java -version 2>&1)
+
+ # Extract the major version number
+ JAVA_MAJOR_VERSION=$(echo "$JAVA_VERSION_STRING" | grep -oE '[0-9]+(\.[0-9]+)*' | head -n 1 | awk -F. '{print ($1 == 1 ? $2 : $1)}')
+
+ if [[ "${JAVA_MAJOR_VERSION}" -ge "9" ]]; then
+ NETTY_OPTS="-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true"
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10382. Optimize Netty memory allocation by avoiding zero assignment in Java 9+ [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #6975:
URL: https://github.com/apache/ozone/pull/6975
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
Re: [PR] HDDS-10382. Optimize Netty memory allocation by avoiding zero assignment in Java 9+ [ozone]
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #6975:
URL: https://github.com/apache/ozone/pull/6975#discussion_r1687681250
##########
hadoop-ozone/dist/src/shell/ozone/ozone:
##########
@@ -81,6 +81,21 @@ function ozonecmd_case
# Corresponding Ratis issue https://issues.apache.org/jira/browse/RATIS-534.
RATIS_OPTS="-Dorg.apache.ratis.thirdparty.io.netty.allocator.useCacheForAllThreads=false ${RATIS_OPTS}"
+ # Add JVM parameter for Java 9+
+ # (org.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true) to allow Netty unsafe memory allocation.
+ # Corresponding issue https://issues.apache.org/jira/browse/HDDS-10382.
+ NETTY_OPTS=""
+
+ # Get the version string
+ JAVA_VERSION_STRING=$(java -version 2>&1)
+
+ # Extract the major version number
+ JAVA_MAJOR_VERSION=$(echo "$JAVA_VERSION_STRING" | grep -oE '[0-9]+(\.[0-9]+)*' | head -n 1 | awk -F. '{print ($1 == 1 ? $2 : $1)}')
+
+ if [[ "${JAVA_MAJOR_VERSION}" -ge "9" ]]; then
+ NETTY_OPTS="-Dorg.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true"
Review Comment:
this can be like NETTY_OPTS="..... ${NETTY_OPTS}" to have more changes when needed
##########
hadoop-ozone/dist/src/shell/ozone/ozone:
##########
@@ -81,6 +81,21 @@ function ozonecmd_case
# Corresponding Ratis issue https://issues.apache.org/jira/browse/RATIS-534.
RATIS_OPTS="-Dorg.apache.ratis.thirdparty.io.netty.allocator.useCacheForAllThreads=false ${RATIS_OPTS}"
+ # Add JVM parameter for Java 9+
+ # (org.apache.ratis.thirdparty.io.netty.tryReflectionSetAccessible=true) to allow Netty unsafe memory allocation.
+ # Corresponding issue https://issues.apache.org/jira/browse/HDDS-10382.
+ NETTY_OPTS=""
Review Comment:
We should not reset value to empty, as it can be set via env also. comments can be moved down
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org