You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "shmilygkd (via GitHub)" <gi...@apache.org> on 2026/09/03 07:20:13 UTC
[PR] HIVE-30011: MR jobs submitted outside ExecDriver miss the JDK 17 --add-opens flags [hive]
shmilygkd opened a new pull request, #6755:
URL: https://github.com/apache/hive/pull/6755
### What changes were proposed in this pull request?
Move the `--add-opens` injection for MR jobs from a private method in `ExecDriver` into `JavaVersionUtils.addOpensFlags(Configuration)` and call it from every code path that submits an MR job through `JobClient`:
- `ExecDriver.execute` (unchanged behaviour, now delegates to the helper)
- `MergeFileTask.execute` (`ALTER TABLE ... CONCATENATE` on the MR engine)
- `ColumnTruncateTask.execute` (`TRUNCATE TABLE ... COLUMNS` on RCFile tables)
- `MRCompactor.launchCompactionJob` (MR-based ACID compaction launched by the metastore Worker)
The helper appends the flags to `mapreduce.map.java.opts`, `mapreduce.reduce.java.opts` and `yarn.app.mapreduce.am.command-opts`, keeping whatever is already configured. `TestJavaVersionUtils` is added to cover the helper.
### Why are the changes needed?
HIVE-28869 only injects the flags in `ExecDriver`. The three paths above submit MR jobs directly, so on a JDK 17 cluster whose `mapred-site.xml` does not carry the flags their task JVMs fail on the first reflective access into `java.base`, for example:
```
Error: java.lang.reflect.InaccessibleObjectException: Unable to make field private volatile java.lang.String java.net.URI.string accessible:
module java.base does not "opens java.net" to unnamed module
at org.apache.hadoop.hive.common.StringInternUtils.<clinit>
at org.apache.hadoop.hive.ql.io.HiveInputFormat.init
```
`MRCompactor` is still the default compactor for full-CRUD tables (`hive.compactor.crud.query.based=false`), so this affects regular ACID maintenance, not only the MR engine.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
- New unit test `TestJavaVersionUtils#testAddOpensFlagsAppendedToMRJobOpts` (existing `-Xmx` value preserved, all three keys end with the flag set).
- The qtest MiniMR drivers run tasks in the local job runner, where `*.java.opts` is ignored, so the container command line cannot be exercised in CI. An equivalent patch on a 3.1.x-based build was verified on a 3-node YARN cluster (Hadoop 3.4.3, JDK 17 on submitter and containers, no cluster-side `--add-opens`): before the patch `ALTER TABLE ... CONCATENATE` failed as above; after it the AM and task `launch_container.sh` of the merge job, the RCFile column-truncate job and a metastore-launched major compaction all carry the flags and the jobs succeed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30011: MR jobs submitted outside ExecDriver miss the JDK 17 --add-opens flags [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6755:
URL: https://github.com/apache/hive/pull/6755#issuecomment-5552876846
cc @kuczoram
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30011: MR jobs submitted outside ExecDriver miss the JDK 17 --add-opens flags [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6755:
URL: https://github.com/apache/hive/pull/6755#issuecomment-5522954581
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6755) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6755&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6755&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6755&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6755&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6755&metric=new_duplicated_lines_density&view=list)
<!-- sqra-placement-anchor -->
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6755)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-30011: MR jobs submitted outside ExecDriver miss the JDK 17 --add-opens flags [hive]
Posted by "shmilygkd (via GitHub)" <gi...@apache.org>.
shmilygkd commented on PR #6755:
URL: https://github.com/apache/hive/pull/6755#issuecomment-5707786423
Hi @kuczoram, friendly ping on this one when you have a moment. Happy to address any comments.
For context on the Jenkins precommit red mark: the only failure in that run was `TestBeeLineDriver[udf_unix_timestamp]` (1 test out of 109,790). The log shows the query being cancelled by the beeline harness mid-execution and the local job interrupted while writing files (`interrupted after 2.193 seconds` in `Shell.runCommand`). There is no module-access or class-loading error anywhere, and the failing job runs on the local job runner, where `*.java.opts` is ignored, so it is unrelated to this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org