You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "deagrawa (via GitHub)" <gi...@apache.org> on 2025/05/09 15:02:30 UTC
[PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
deagrawa opened a new pull request, #5809:
URL: https://github.com/apache/hive/pull/5809
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
The "add-open" JVM options have been incorporated to facilitate access to internal packages, a requirement for Java Development Kit (JDK) versions 9 and higher.
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
To support JDK 9 or higher.
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
No impact on existing code functionality; changes take effect when switching from JDK version 8 to 9 or higher.
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
Tested by running test cases locally.
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2088346345
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded();
+ conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
Review Comment:
can we keep the `if (!amJavaOpts.isEmpty())` check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2870986343
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081974924
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java:
##########
@@ -708,7 +709,9 @@ Map<String, String> getContainerEnvironment(Configuration conf, boolean isMap) {
* are set
*/
private static String getContainerJavaOpts(Configuration conf) {
- String javaOpts = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS);
+ String baseJavaOpts = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS);
Review Comment:
````
String javaOpts = StringUtils.defaultString(HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS));
javaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084400804
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/util/JavaVersionUtils.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.util;
+
+public class JavaVersionUtils {
Review Comment:
We should add a private constructor.
https://sonarcloud.io/organizations/apache/rules?open=java%3AS1118&rule_key=java%3AS1118&tab=how_to_fix
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
+ return String.join(" ", amJavaOpts, getJavaAddOpenOptions());
+ } else {
+ return amJavaOpts;
+ }
+ }
+ private String getJavaAddOpenOptions() {
+ return String.join(" ",
+ "-XX:+IgnoreUnrecognizedVMOptions",
+ "--add-opens=java.base/java.net=ALL-UNNAMED",
+ "--add-opens=java.base/java.util=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang=ALL-UNNAMED",
+ "--add-opens=java.base/java.io=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED",
+ "--add-opens=java.base/java.math=ALL-UNNAMED",
+ "--add-opens=java.base/java.nio=ALL-UNNAMED",
+ "--add-opens=java.base/java.text=ALL-UNNAMED",
+ "--add-opens=java.base/java.time=ALL-UNNAMED",
+ "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED",
+ "--add-opens=java.base/jdk.internal.reflect=ALL-UNNAMED",
+ "--add-opens=java.sql/java.sql=ALL-UNNAMED",
+ "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
+ "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.regex=ALL-UNNAMED"
Review Comment:
Just for your information, we are adding `java.base/java.security=ALL-UNNAMED` and `java.base/sun.security.provider=ALL-UNNAMED` as well in our JDK 11 environment. We don't have to add them unless we really need them, though.
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
Review Comment:
```suggestion
if (javaSpecVer >= 9) {
```
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
Review Comment:
```suggestion
final int javaSpecVer =
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084408578
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/util/JavaVersionUtils.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.util;
+
+public class JavaVersionUtils {
Review Comment:
updated the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2087215148
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
Review Comment:
[this](https://github.com/apache/hive/blob/03d8f4d4b2791fc1fa49c2b2969996458c194577/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L726-L729) didn't handle spaces either.
Why not simply use
````
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
add-opens, if required, should start with a space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2083937732
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
+ return String.join(" ", amJavaOpts, getJavaAddOpenOptions());
+ } else {
+ return amJavaOpts;
Review Comment:
Ok, but current code base is not migrated to JDK9 or higher yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2083933577
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -123,7 +123,7 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
String amJavaOpts = appConf.controllerAMChildOpts();
if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
+ conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, addJavaOpensPackages(amJavaOpts));
Review Comment:
Thanks incorporated the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081950699
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -123,7 +123,7 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
String amJavaOpts = appConf.controllerAMChildOpts();
if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
+ conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, addJavaOpensPackages(amJavaOpts));
Review Comment:
should we add `add-opens` only when `appConf.controllerAMChildOpts()` isn't empty? maybe
````
String amJavaOpts = appConf.controllerAMChildOpts();
amJavaOpts += getJavaAddOpenOptions()
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081950699
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -123,7 +123,7 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
String amJavaOpts = appConf.controllerAMChildOpts();
if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
+ conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, addJavaOpensPackages(amJavaOpts));
Review Comment:
should we add `add-opens` only when `appConf.controllerAMChildOpts()` isn't empty? maybe
````
String amJavaOpts = StringUtils.defaultIfEmpty(appConf.controllerAMChildOpts(), "");
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081974924
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java:
##########
@@ -708,7 +709,9 @@ Map<String, String> getContainerEnvironment(Configuration conf, boolean isMap) {
* are set
*/
private static String getContainerJavaOpts(Configuration conf) {
- String javaOpts = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS);
+ String baseJavaOpts = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS);
Review Comment:
````
String javaOpts = StringUtils.defaultString(HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS));
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "tanishq-chugh (via GitHub)" <gi...@apache.org>.
tanishq-chugh commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2876823693
Hi @zhangbutao
Yes, this PR is specifically to add the required add-opens for yarn jobs or tez/mapreduce tasks for jdk 9+ . This is a pre-requisite for [Hive-JDK17-PR](https://github.com/apache/hive/pull/5404) . The intention here was to place these add-opens configs in the codebase itself, and not in the xml config files, as it could cause these necessary values to be removed/overridden by user leading to potentially crashing the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2087215148
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
Review Comment:
[this](https://github.com/apache/hive/blob/03d8f4d4b2791fc1fa49c2b2969996458c194577/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L726-L729) didn't handle spaces either.
Why not simply use
````
String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
add-opens, if required, should start with a space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2086411408
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded());
+ conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
Review Comment:
Thanks, updated the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2086770857
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
Review Comment:
It's my advice. It cleans up the final output when either LHS or RHS is empty, and also it's more robust when there is another validation similar to [this](https://github.com/apache/hive/blob/03d8f4d4b2791fc1fa49c2b2969996458c194577/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L726-L729).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2867061535
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2262219128
##########
common/src/java/org/apache/hadoop/hive/common/JavaVersionUtils.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common;
+
+public class JavaVersionUtils {
+ private JavaVersionUtils() {
+ throw new IllegalStateException("Utility class");
+ }
+
+ /**
+ * Returns the detected Java major version (e.g., 8, 11, 17).
+ */
+ public static int getJavaMajorVersion() {
+ return Math.max(8, Integer.parseInt(
+ System.getProperty("java.specification.version").split("\\.")[0]));
+ }
+
+ /**
+ * Returns the full --add-opens string if Java 9, or an empty string if Java 8 or below.
+ */
+ public static String getAddOpensFlagsIfNeeded() {
+ //TODO: Please remove this once the codebase has been migrated to JDK 9 or a higher version.
+ int javaVersion = getJavaMajorVersion();
+ if (javaVersion >= 9) {
+ return " --add-opens=java.base/java.net=ALL-UNNAMED" +
+ " --add-opens=java.base/java.util=ALL-UNNAMED" +
+ " --add-opens=java.base/java.util.concurrent=ALL-UNNAMED" +
+ " --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" +
+ " --add-opens=java.base/java.lang=ALL-UNNAMED" +
+ " --add-opens=java.base/java.io=ALL-UNNAMED" +
+ " --add-opens java.base/java.lang=ALL-UNNAMED" +
Review Comment:
@Chaho12 Good catch. Could you please file a PR to fix this?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2888443564
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2087215148
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
Review Comment:
[this](https://github.com/apache/hive/blob/03d8f4d4b2791fc1fa49c2b2969996458c194577/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L726-L729) didn't handle spaces either.
Why not simply use
````
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
add-opens, if required, should start with a space and it already does, so @deagrawa why do you need to add 1 more space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081974924
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java:
##########
@@ -708,7 +709,9 @@ Map<String, String> getContainerEnvironment(Configuration conf, boolean isMap) {
* are set
*/
private static String getContainerJavaOpts(Configuration conf) {
- String javaOpts = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS);
+ String baseJavaOpts = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS);
Review Comment:
String javaOpts = StringUtils.defaultString(HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS));
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084327838
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
+ return String.join(" ", amJavaOpts, getJavaAddOpenOptions());
+ } else {
+ return amJavaOpts;
Review Comment:
Thanks, updated the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081950699
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -123,7 +123,7 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
String amJavaOpts = appConf.controllerAMChildOpts();
if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
+ conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, addJavaOpensPackages(amJavaOpts));
Review Comment:
should we add `add-opens` only when `appConf.controllerAMChildOpts()` isn't empty? maybe
````
String amJavaOpts = ObjectUtils.defaultIfNull(appConf.controllerAMChildOpts(), "");
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "Chaho12 (via GitHub)" <gi...@apache.org>.
Chaho12 commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2262122299
##########
common/src/java/org/apache/hadoop/hive/common/JavaVersionUtils.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common;
+
+public class JavaVersionUtils {
+ private JavaVersionUtils() {
+ throw new IllegalStateException("Utility class");
+ }
+
+ /**
+ * Returns the detected Java major version (e.g., 8, 11, 17).
+ */
+ public static int getJavaMajorVersion() {
+ return Math.max(8, Integer.parseInt(
+ System.getProperty("java.specification.version").split("\\.")[0]));
+ }
+
+ /**
+ * Returns the full --add-opens string if Java 9, or an empty string if Java 8 or below.
+ */
+ public static String getAddOpensFlagsIfNeeded() {
+ //TODO: Please remove this once the codebase has been migrated to JDK 9 or a higher version.
+ int javaVersion = getJavaMajorVersion();
+ if (javaVersion >= 9) {
+ return " --add-opens=java.base/java.net=ALL-UNNAMED" +
+ " --add-opens=java.base/java.util=ALL-UNNAMED" +
+ " --add-opens=java.base/java.util.concurrent=ALL-UNNAMED" +
+ " --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" +
+ " --add-opens=java.base/java.lang=ALL-UNNAMED" +
+ " --add-opens=java.base/java.io=ALL-UNNAMED" +
+ " --add-opens java.base/java.lang=ALL-UNNAMED" +
Review Comment:
missing `=` after --add-opens
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2875902623
@deagrawa This PR is specifically for the Tez task launched by HS2, not the Hive service, right?
https://github.com/apache/hive/pull/5404 is mainly for Hive service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2088358939
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded();
+ conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
Review Comment:
What would be the benefits of adding empty check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2888564903
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2888134891
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081973640
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
+ return String.join(" ", amJavaOpts, getJavaAddOpenOptions());
+ } else {
+ return amJavaOpts;
+ }
+ }
+ private String getJavaAddOpenOptions() {
Review Comment:
could this be moved to common to avoid copy&paste?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084645866
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java:
##########
@@ -548,8 +550,9 @@ private void setupTezParamsBasedOnMR(TezConfiguration conf) {
conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_ENV, env);
}
- conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS,
- org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf));
+ String mrAmJavaOpts = StringUtils.defaultString(org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf));
+ mrAmJavaOpts = String.join(" ", mrAmJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded());
+ conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS, mrAmJavaOpts);
Review Comment:
I am feeling this might not work when TEZ_AM_LAUNCH_CMD_OPTS is explicitly set. The decision table is like this? I guess we don't update the third pattern.
| JDK | TEZ_AM_LAUNCH_CMD_OPTS is set | Result |
|-|-|-|
| 8 | Yes | Value of TEZ_AM_LAUNCH_CMD_OPTS |
| 8 | No | Value of getJavaOptsForMRAM |
| 17 | Yes | Value of TEZ_AM_LAUNCH_CMD_OPTS + getAddOpensFlagsIfNeeded |
| 17 | No | Value of getJavaOptsForMRAM + getAddOpensFlagsIfNeeded |
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java:
##########
@@ -489,6 +491,15 @@ public int execute() {
return (returnVal);
}
+ private void addJavaOpts(JobConf job) {
+ String commonAddOpens = JavaVersionUtils.getAddOpensFlagsIfNeeded();
+ if(!commonAddOpens.isEmpty()){
+ job.set("yarn.app.mapreduce.am.command-opts", commonAddOpens);
+ job.set("mapreduce.map.java.opts", commonAddOpens);
+ job.set("mapreduce.reduce.java.opts", commonAddOpens);
Review Comment:
Probably, when we add getAddOpensFlagsIfNeeded to `xxx.yyy`, it needs to be like this.
```
String original = job.get("xxx.yyy", "");
job.set("xxx.yyy", String.join(" ", original, commonAddOpens).trim());
```
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java:
##########
@@ -708,7 +709,8 @@ Map<String, String> getContainerEnvironment(Configuration conf, boolean isMap) {
* are set
*/
private static String getContainerJavaOpts(Configuration conf) {
- String javaOpts = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS);
+ String javaOpts = StringUtils.defaultString(HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS));
+ javaOpts = String.join(" ", javaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded());
Review Comment:
I found this disables the warning from L728. Probably, we should update not `javaOpts` but `finalOpts`.
```
// Probablly, better to trim it because some later steps may do empty-checks
finalOpts = String.join(" ", finalOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
LOG.debug("Tez container final opts: {}", finalOpts);
return finalOpts;
```
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java:
##########
@@ -548,8 +550,9 @@ private void setupTezParamsBasedOnMR(TezConfiguration conf) {
conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_ENV, env);
}
- conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS,
- org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf));
+ String mrAmJavaOpts = StringUtils.defaultString(org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf));
+ mrAmJavaOpts = String.join(" ", mrAmJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded());
+ conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS, mrAmJavaOpts);
Review Comment:
I'd write like this.
```
String opts = conf.get(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS);
if (opts == null) {
// The MR-related param is a fallback
opts = StringUtils.defaultString(org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf));
}
// Add additional flags if it's JDK 17. Probably, better to trim because someone does an empty-check
opts = String.join(" ", opts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
conf.set(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS, opts);
```
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded());
+ conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
Review Comment:
`AppConfig.HADOOP_MR_AM_JAVA_OPTS` is `yarn.app.mapreduce.am.command-opts`, and `yarn.app.mapreduce.am.command-opts` has the default value, `-Xmx1024m`. I guess the default value can be gone.
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java:
##########
@@ -489,6 +491,15 @@ public int execute() {
return (returnVal);
}
+ private void addJavaOpts(JobConf job) {
+ String commonAddOpens = JavaVersionUtils.getAddOpensFlagsIfNeeded();
+ if(!commonAddOpens.isEmpty()){
+ job.set("yarn.app.mapreduce.am.command-opts", commonAddOpens);
+ job.set("mapreduce.map.java.opts", commonAddOpens);
+ job.set("mapreduce.reduce.java.opts", commonAddOpens);
Review Comment:
I could be wrong. Is it possible that this overwrites the originally configured value?
For example, `yarn.app.mapreduce.am.command-opts` is probably configured as below now.
| JDK | yarn-site.xml | Result | Note |
|-|-|-|-|
| 8 | N/A | -Xmx1024m | Default value |
| 8 | -Xmx2048m | -Xmx2048m | |
With this patch, the table changes as follows.
| JDK | yarn-site.xml | Result | Note |
|-|-|-|-|
| 8 | N/A | -Xmx1024m | Default value |
| 8 | -Xmx2048m | -Xmx2048m | |
| 17 | N/A | -XX:IgnoreUnrecognizedVMOptions --add-opens=... | |
| 17 | -Xmx2048m | -XX:IgnoreUnrecognizedVMOptions --add-opens=... | |
We may be able to point similar things out for other files, such as Tez configurations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2083934605
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java:
##########
@@ -708,7 +709,9 @@ Map<String, String> getContainerEnvironment(Configuration conf, boolean isMap) {
* are set
*/
private static String getContainerJavaOpts(Configuration conf) {
- String javaOpts = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS);
+ String baseJavaOpts = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS);
Review Comment:
Thanks incorporated the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2087215148
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
Review Comment:
[this](https://github.com/apache/hive/blob/03d8f4d4b2791fc1fa49c2b2969996458c194577/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L726-L729) didn't handle spaces either.
Why not simply use
````
String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
add-opens, if required, should start with a space
````
return String.join(" ",
" -XX:IgnoreUnrecognizedVMOptions",
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2086410152
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java:
##########
@@ -548,8 +550,9 @@ private void setupTezParamsBasedOnMR(TezConfiguration conf) {
conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_ENV, env);
}
- conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS,
- org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf));
+ String mrAmJavaOpts = StringUtils.defaultString(org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf));
+ mrAmJavaOpts = String.join(" ", mrAmJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded());
+ conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS, mrAmJavaOpts);
Review Comment:
Thanks, updated the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2086770857
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
Review Comment:
It's my advice. It cleans up the final output when either LHS or RHS is empty, and also I thought there might be another place to do a null or empty check in addition to [this validation](https://github.com/apache/hive/blob/03d8f4d4b2791fc1fa49c2b2969996458c194577/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L726-L729).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "tanishq-chugh (via GitHub)" <gi...@apache.org>.
tanishq-chugh commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084201580
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
+ return String.join(" ", amJavaOpts, getJavaAddOpenOptions());
+ } else {
+ return amJavaOpts;
+ }
+ }
+ private String getJavaAddOpenOptions() {
Review Comment:
+1
@deagrawa it would be better if we can move this to hive-common util and refer it from there at all the required places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084326843
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
+ return String.join(" ", amJavaOpts, getJavaAddOpenOptions());
+ } else {
+ return amJavaOpts;
+ }
+ }
+ private String getJavaAddOpenOptions() {
Review Comment:
Thanks, updated the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081980018
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/util/JavaVersionUtils.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.util;
+
+public class JavaVersionUtils {
+ /**
+ * Returns the detected Java major version (e.g., 8, 11, 17).
+ */
+ public static int getJavaMajorVersion() {
+ return Math.max(8, Integer.parseInt(
Review Comment:
8 is no longer supported after the move
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "Chaho12 (via GitHub)" <gi...@apache.org>.
Chaho12 commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2262322624
##########
common/src/java/org/apache/hadoop/hive/common/JavaVersionUtils.java:
##########
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.common;
+
+public class JavaVersionUtils {
+ private JavaVersionUtils() {
+ throw new IllegalStateException("Utility class");
+ }
+
+ /**
+ * Returns the detected Java major version (e.g., 8, 11, 17).
+ */
+ public static int getJavaMajorVersion() {
+ return Math.max(8, Integer.parseInt(
+ System.getProperty("java.specification.version").split("\\.")[0]));
+ }
+
+ /**
+ * Returns the full --add-opens string if Java 9, or an empty string if Java 8 or below.
+ */
+ public static String getAddOpensFlagsIfNeeded() {
+ //TODO: Please remove this once the codebase has been migrated to JDK 9 or a higher version.
+ int javaVersion = getJavaMajorVersion();
+ if (javaVersion >= 9) {
+ return " --add-opens=java.base/java.net=ALL-UNNAMED" +
+ " --add-opens=java.base/java.util=ALL-UNNAMED" +
+ " --add-opens=java.base/java.util.concurrent=ALL-UNNAMED" +
+ " --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" +
+ " --add-opens=java.base/java.lang=ALL-UNNAMED" +
+ " --add-opens=java.base/java.io=ALL-UNNAMED" +
+ " --add-opens java.base/java.lang=ALL-UNNAMED" +
Review Comment:
https://issues.apache.org/jira/browse/HIVE-29126 Sure here it is
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2868660229
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2869006936
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2086711615
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
Review Comment:
Why do you need `trim()` here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2888698351
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084079063
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/util/JavaVersionUtils.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.util;
+
+public class JavaVersionUtils {
+ /**
+ * Returns the detected Java major version (e.g., 8, 11, 17).
+ */
+ public static int getJavaMajorVersion() {
+ return Math.max(8, Integer.parseInt(
Review Comment:
true, could we add TODO to remove it once migrated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2086409674
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java:
##########
@@ -708,7 +709,8 @@ Map<String, String> getContainerEnvironment(Configuration conf, boolean isMap) {
* are set
*/
private static String getContainerJavaOpts(Configuration conf) {
- String javaOpts = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS);
+ String javaOpts = StringUtils.defaultString(HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_TEZ_JAVA_OPTS));
+ javaOpts = String.join(" ", javaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded());
Review Comment:
Thanks, updated the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2086410987
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java:
##########
@@ -489,6 +491,15 @@ public int execute() {
return (returnVal);
}
+ private void addJavaOpts(JobConf job) {
+ String commonAddOpens = JavaVersionUtils.getAddOpensFlagsIfNeeded();
+ if(!commonAddOpens.isEmpty()){
+ job.set("yarn.app.mapreduce.am.command-opts", commonAddOpens);
+ job.set("mapreduce.map.java.opts", commonAddOpens);
+ job.set("mapreduce.reduce.java.opts", commonAddOpens);
Review Comment:
Thanks, updated the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084326563
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/util/JavaVersionUtils.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.util;
+
+public class JavaVersionUtils {
+ /**
+ * Returns the detected Java major version (e.g., 8, 11, 17).
+ */
+ public static int getJavaMajorVersion() {
+ return Math.max(8, Integer.parseInt(
Review Comment:
Thanks, updated the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2083936667
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/util/JavaVersionUtils.java:
##########
@@ -0,0 +1,62 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.util;
+
+public class JavaVersionUtils {
+ /**
+ * Returns the detected Java major version (e.g., 8, 11, 17).
+ */
+ public static int getJavaMajorVersion() {
+ return Math.max(8, Integer.parseInt(
Review Comment:
Ok, but current code base is not migrated to JDK17 yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2087215148
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
Review Comment:
[this](https://github.com/apache/hive/blob/03d8f4d4b2791fc1fa49c2b2969996458c194577/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L726-L729) didn't handle spaces either.
Why not simply use
````
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
add-opens should start with a space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2087215148
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
Review Comment:
[this](https://github.com/apache/hive/blob/03d8f4d4b2791fc1fa49c2b2969996458c194577/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L726-L729) didn't handle spaces either.
Why not simply use
````
String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
add-opens, if required, should start with a space
````
return String.join(" ",
" -XX:IgnoreUnrecognizedVMOptions",
````
in my opinion it's much more readable than current impl
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2876904075
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2087215148
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -121,10 +123,9 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
if (amMemoryMB != null && !amMemoryMB.isEmpty()) {
conf.set(AppConfig.HADOOP_MR_AM_MEMORY_MB, amMemoryMB);
}
- String amJavaOpts = appConf.controllerAMChildOpts();
- if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
- }
+ String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
+ amJavaOpts = String.join(" ", amJavaOpts, JavaVersionUtils.getAddOpensFlagsIfNeeded()).trim();
Review Comment:
[this](https://github.com/apache/hive/blob/03d8f4d4b2791fc1fa49c2b2969996458c194577/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DagUtils.java#L726-L729) didn't handle spaces either.
Why not simply use
````
String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
add-opens, if required, should start with a space and it already does, so @deagrawa why do you need to add 1 more space
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081952230
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
+ return String.join(" ", amJavaOpts, getJavaAddOpenOptions());
+ } else {
+ return amJavaOpts;
Review Comment:
we won't support anything below JDK9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn merged PR #5809:
URL: https://github.com/apache/hive/pull/5809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2887734241
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2876294218
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [4 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2887261943
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [3 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081958597
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java:
##########
@@ -489,6 +491,15 @@ public int execute() {
return (returnVal);
}
+ private void setJobOptions(JobConf job) {
Review Comment:
maybe `addJavaOpts`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081950699
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -123,7 +123,7 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
String amJavaOpts = appConf.controllerAMChildOpts();
if (amJavaOpts != null && !amJavaOpts.isEmpty()) {
- conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, amJavaOpts);
+ conf.set(AppConfig.HADOOP_MR_AM_JAVA_OPTS, addJavaOpensPackages(amJavaOpts));
Review Comment:
should we add `add-opens` only when `appConf.controllerAMChildOpts()` isn't empty? maybe
````
String amJavaOpts = StringUtils.defaultString(appConf.controllerAMChildOpts());
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2081978131
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java:
##########
@@ -548,8 +549,9 @@ private void setupTezParamsBasedOnMR(TezConfiguration conf) {
conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_ENV, env);
}
- conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS,
- org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf));
+ String mrAmJavaOpts = org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf);
Review Comment:
````
String amJavaOpts = StringUtils.defaultString(org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf));
amJavaOpts += JavaVersionUtils.getAddOpensFlagsIfNeeded()
conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS, amJavaOpts);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2083935385
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java:
##########
@@ -548,8 +549,9 @@ private void setupTezParamsBasedOnMR(TezConfiguration conf) {
conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_ENV, env);
}
- conf.setIfUnset(TezConfiguration.TEZ_AM_LAUNCH_CMD_OPTS,
- org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf));
+ String mrAmJavaOpts = org.apache.tez.mapreduce.hadoop.MRHelpers.getJavaOptsForMRAM(conf);
Review Comment:
Thanks incorporated the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2083933875
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java:
##########
@@ -489,6 +491,15 @@ public int execute() {
return (returnVal);
}
+ private void setJobOptions(JobConf job) {
Review Comment:
Thanks incorporated the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5809:
URL: https://github.com/apache/hive/pull/5809#issuecomment-2872555181
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5809&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5809&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5809&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5809)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084393979
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
+ return String.join(" ", amJavaOpts, getJavaAddOpenOptions());
+ } else {
+ return amJavaOpts;
+ }
+ }
+ private String getJavaAddOpenOptions() {
+ return String.join(" ",
+ "-XX:+IgnoreUnrecognizedVMOptions",
+ "--add-opens=java.base/java.net=ALL-UNNAMED",
+ "--add-opens=java.base/java.util=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang=ALL-UNNAMED",
+ "--add-opens=java.base/java.io=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED",
+ "--add-opens=java.base/java.math=ALL-UNNAMED",
+ "--add-opens=java.base/java.nio=ALL-UNNAMED",
+ "--add-opens=java.base/java.text=ALL-UNNAMED",
+ "--add-opens=java.base/java.time=ALL-UNNAMED",
+ "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED",
+ "--add-opens=java.base/jdk.internal.reflect=ALL-UNNAMED",
+ "--add-opens=java.sql/java.sql=ALL-UNNAMED",
+ "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
+ "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.regex=ALL-UNNAMED"
Review Comment:
Just for your information, we are adding `java.base/java.security=ALL-UNNAMED` and `java.base/sun.security.provider=ALL-UNNAMED` as well in our JDK 11 environment. I'm not confident that we're too defensive, or we really needed them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084408033
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
Review Comment:
Thanks its taken care.
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
Review Comment:
Thanks its taken care.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28869: added --add-opens JVM options to support JDK 9 or higher. [hive]
Posted by "deagrawa (via GitHub)" <gi...@apache.org>.
deagrawa commented on code in PR #5809:
URL: https://github.com/apache/hive/pull/5809#discussion_r2084532139
##########
hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/tool/TempletonControllerJob.java:
##########
@@ -167,6 +167,39 @@ public int run(String[] args) throws IOException, InterruptedException, ClassNot
}
return 0;
}
+ private String addJavaOpensPackages(String amJavaOpts) {
+ final int JAVA_SPEC_VER =
+ Math.max(8, Integer.parseInt(System.getProperty("java.specification.version").split("\\.")[0]));
+ if (JAVA_SPEC_VER >= 9) {
+ return String.join(" ", amJavaOpts, getJavaAddOpenOptions());
+ } else {
+ return amJavaOpts;
+ }
+ }
+ private String getJavaAddOpenOptions() {
+ return String.join(" ",
+ "-XX:+IgnoreUnrecognizedVMOptions",
+ "--add-opens=java.base/java.net=ALL-UNNAMED",
+ "--add-opens=java.base/java.util=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang=ALL-UNNAMED",
+ "--add-opens=java.base/java.io=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED",
+ "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED",
+ "--add-opens=java.base/java.math=ALL-UNNAMED",
+ "--add-opens=java.base/java.nio=ALL-UNNAMED",
+ "--add-opens=java.base/java.text=ALL-UNNAMED",
+ "--add-opens=java.base/java.time=ALL-UNNAMED",
+ "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED",
+ "--add-opens=java.base/jdk.internal.reflect=ALL-UNNAMED",
+ "--add-opens=java.sql/java.sql=ALL-UNNAMED",
+ "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED",
+ "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED",
+ "--add-opens=java.base/java.util.regex=ALL-UNNAMED"
Review Comment:
added java.base/java.security=ALL-UNNAMED and java.base/sun.security.provider=ALL-UNNAMED as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org