You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org> on 2025/06/21 08:38:33 UTC
[PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Aggarwal-Raghav opened a new pull request, #5884:
URL: https://github.com/apache/hive/pull/5884
### What changes were proposed in this pull request?
Log the HS2 and HMS PID in their respective logs and give sample hms and hs2 HADOOP_OPTS in hive-env.sh.template.
### Why are the changes needed?
During debugging prod issue, I saw the need for this.
1. If HMS/HS2 is crashing periodically then the heapdump will have the name in default format i.e. `java_pid14784.hprof`, it becomes difficult to identify whether it belongs to HMS or HS2 without loading it in any analyzer tool.
2. To get the gc logs for a particular HS2 or HMS crash requires extraction of timestamp from logs using text manipulation commands etc.
### Does this PR introduce _any_ user-facing change?
Yes, in HS2 and HMS logs PID will be logged
### How was this patch tested?
On local setup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5884:
URL: https://github.com/apache/hive/pull/5884#discussion_r2160836776
##########
service/src/java/org/apache/hive/service/server/HiveServer2.java:
##########
@@ -1206,7 +1206,8 @@ public void startPrivilegeSynchronizer(HiveConf hiveConf) throws Exception {
private static void startHiveServer2() throws Throwable {
long attempts = 0, maxAttempts = 1;
while (true) {
- LOG.info("Starting HiveServer2");
+ long pid = ProcessHandle.current().pid();
Review Comment:
not sure, I am just used to inline when the var is not used in multiple places
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5884:
URL: https://github.com/apache/hive/pull/5884#issuecomment-2997098331
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5884) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5884&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5884&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5884&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5884&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5884&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5884)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "okumin (via GitHub)" <gi...@apache.org>.
okumin commented on code in PR #5884:
URL: https://github.com/apache/hive/pull/5884#discussion_r2160230839
##########
conf/hive-env.sh.template:
##########
@@ -18,18 +20,29 @@
# to control the execution of Hive. It should be used by admins to configure
# the Hive installation (so that users do not have to set environment variables
# or set command line parameters to get correct behavior).
-#
-# The hive service being invoked (CLI etc.) is available via the environment
-# variable SERVICE
+# Set HADOOP_HOME to point to a specific hadoop install directory
+# HADOOP_HOME=${bin}/../../hadoop
+
+# Hive Configuration Directory can be controlled by:
+# export HIVE_CONF_DIR=
+
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+# export HIVE_AUX_JARS_PATH=
+
+# HIVE Log Directory can be controlled by and ensure it exists on the filesystem:
+# export HIVE_LOG_DIR=
+
+# The hive service being invoked (beeline etc.) is available via the environment
+# variable SERVICE
# Hive Client memory usage can be an issue if a large number of clients
-# are running at the same time. The flags below have been useful in
+# are running at the same time. The flags below have been useful in
# reducing memory usage:
-#
-# if [ "$SERVICE" = "cli" ]; then
+
+# if [ "$SERVICE" = "beeline" ]; then
# if [ -z "$DEBUG" ]; then
-# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
+# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+G1GC -XX:-UseGCOverheadLimit"
Review Comment:
`-XX:+UseG1GC` is the correct one?
##########
conf/hive-env.sh.template:
##########
@@ -18,18 +20,29 @@
# to control the execution of Hive. It should be used by admins to configure
# the Hive installation (so that users do not have to set environment variables
# or set command line parameters to get correct behavior).
-#
-# The hive service being invoked (CLI etc.) is available via the environment
-# variable SERVICE
+# Set HADOOP_HOME to point to a specific hadoop install directory
+# HADOOP_HOME=${bin}/../../hadoop
+
+# Hive Configuration Directory can be controlled by:
+# export HIVE_CONF_DIR=
+
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+# export HIVE_AUX_JARS_PATH=
+
+# HIVE Log Directory can be controlled by and ensure it exists on the filesystem:
+# export HIVE_LOG_DIR=
+
+# The hive service being invoked (beeline etc.) is available via the environment
+# variable SERVICE
# Hive Client memory usage can be an issue if a large number of clients
-# are running at the same time. The flags below have been useful in
+# are running at the same time. The flags below have been useful in
# reducing memory usage:
-#
-# if [ "$SERVICE" = "cli" ]; then
+
+# if [ "$SERVICE" = "beeline" ]; then
# if [ -z "$DEBUG" ]; then
-# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
+# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+G1GC -XX:-UseGCOverheadLimit"
Review Comment:
Can we remove `-XX:+G1GC` since it's the default with JDK 17+? I also see [-XX:NewRatio is not recommended](https://docs.oracle.com/en/java/javase/21/gctuning/garbage-first-garbage-collector-tuning.html#GUID-0BB3B742-A985-4D5E-A9C5-433A127FE0F6). However, as I don't know the intention of the DEBUG option, we may keep it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5884:
URL: https://github.com/apache/hive/pull/5884#issuecomment-2993464545
For example:
<img width="473" alt="Screenshot 2025-06-21 at 1 45 54 PM" src="https://github.com/user-attachments/assets/63bae5aa-1fd6-4ef8-8225-ed3629e2a97d" />
I can easily tell that PID 14784 belongs to HS2 heap dump and can easily fetch gc logs for that PID easily.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #5884:
URL: https://github.com/apache/hive/pull/5884#issuecomment-3045482018
@okumin @deniskuzZ Thanks for the reivew!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5884:
URL: https://github.com/apache/hive/pull/5884#discussion_r2160237406
##########
conf/hive-env.sh.template:
##########
@@ -18,18 +20,29 @@
# to control the execution of Hive. It should be used by admins to configure
# the Hive installation (so that users do not have to set environment variables
# or set command line parameters to get correct behavior).
-#
-# The hive service being invoked (CLI etc.) is available via the environment
-# variable SERVICE
+# Set HADOOP_HOME to point to a specific hadoop install directory
+# HADOOP_HOME=${bin}/../../hadoop
+
+# Hive Configuration Directory can be controlled by:
+# export HIVE_CONF_DIR=
+
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+# export HIVE_AUX_JARS_PATH=
+
+# HIVE Log Directory can be controlled by and ensure it exists on the filesystem:
+# export HIVE_LOG_DIR=
+
+# The hive service being invoked (beeline etc.) is available via the environment
+# variable SERVICE
# Hive Client memory usage can be an issue if a large number of clients
-# are running at the same time. The flags below have been useful in
+# are running at the same time. The flags below have been useful in
# reducing memory usage:
-#
-# if [ "$SERVICE" = "cli" ]; then
+
+# if [ "$SERVICE" = "beeline" ]; then
# if [ -z "$DEBUG" ]; then
-# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
+# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+G1GC -XX:-UseGCOverheadLimit"
Review Comment:
Thanks for the review @okumin. Few points:
1. Yes -`XX:+UseG1GC` is the correct one. I doubled confirmed using:
```java -XX:+UnlockExperimentalVMOptions -XX:+PrintFlagsFinal -version | grep -i "G1GC"```
2. Regarding removing the GC flags and NewRatio, I am inclined towards removing every flag other than Xmx and Xms. Reason being, G1GC and ZGC doesn't require much tuning and tuning should be system specific + workload specifc which varies cluster to cluster.
3. I "want to believe" that the DEBUG option is there to showcase user the possibilities / kind of tuning users can do.
Let me know your thoughts based on that I will update the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5884:
URL: https://github.com/apache/hive/pull/5884#issuecomment-2993513482
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5884) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5884&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5884&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5884&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5884&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5884&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5884)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #5884:
URL: https://github.com/apache/hive/pull/5884#issuecomment-3015093205
@Aggarwal-Raghav I have just figured out the reason why the `HADOOP_OPTS` in my **hive-env.sh** file is ineffective.
Because in order for `Hadoop 3.4.1` to run on JDK 17, I configured the `HADOOP_OPTS` parameter in the **hadoop-env.sh** file:
`export HADOOP_OPTS="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.net.util=ALL-UNNAMED --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.lang.reflect=ALL-UNNAMED"`
So when Hive starts, the value of the `HADOOP_OPTS` parameter in **hive-env.sh** is always overwritten by the parameter value in **hadoop-env.sh**.
Therefore, in order to prevent the parameters in **hive-env.sh** from being overridden by those in **hadoop-env.sh**, we can specify a clean **hadoop-env.sh** file for hive (a separate hadoop client for hive)
Back to this current PR, regarding whether we should change the `HADOOP_OPTS `in the **hive-env.sh template** to `HADOOP_CLIENT_OPTS`. My answer is yes.
See HIVE-19886
https://github.com/apache/hive/blob/925df927248c1be11b84df3e8a0c3e43955e133e/service/src/java/org/apache/hive/service/server/HiveServer2.java#L1469-L1480
But I also want to hear the opinions of the other folks. :)
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5884:
URL: https://github.com/apache/hive/pull/5884#discussion_r2167361528
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -262,7 +262,11 @@ public static void main(String[] args) throws Throwable {
startupShutdownMessage(HiveMetaStore.class, args, LOG);
try {
- String msg = "Starting hive metastore on port " + cli.getPort();
+ String msg =
Review Comment:
> why do we need to add this into System.err?
when HMS starts with verbose mode i.e. `bin/hive --service metastore --verbose`, the verbose info/logs are printed via `System.err`. It was done in [HIVE-2139](https://issues.apache.org/jira/browse/HIVE-2139). After doing some research (googling) found this:
`
In Java, System.err is typically used for error messages and verbose output because it's generally not buffered, unlike System.out, which is often buffered. This means System.err sends output directly to the console or error stream, ensuring that error messages are displayed immediately, even if System.out's buffer is not yet full. This behavior is designed to prioritize the display of error information, even if it means potentially interrupting or appearing before the buffered output from System.out.
`
Also, "Starting hive metastore" is printed 3 times (attaching screenshot for the same)
<img width="1308" alt="Screenshot 2025-06-25 at 11 31 54 PM" src="https://github.com/user-attachments/assets/e0a727f3-fceb-4d9a-b597-0614323f35d0" />
<img width="1508" alt="Screenshot 2025-06-25 at 11 37 17 PM" src="https://github.com/user-attachments/assets/62f70ae9-2e7a-4717-b7de-b16926080465" />
> i think it's better to use logger template "{}"
As its not a LOG line but something like this can be done:
```
String msg =
"Starting hive metastore on port %d. PID is %d"
.formatted(cli.getPort(), ProcessHandle.current().pid());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5884:
URL: https://github.com/apache/hive/pull/5884#discussion_r2161781888
##########
conf/hive-env.sh.template:
##########
@@ -18,18 +20,29 @@
# to control the execution of Hive. It should be used by admins to configure
# the Hive installation (so that users do not have to set environment variables
# or set command line parameters to get correct behavior).
-#
-# The hive service being invoked (CLI etc.) is available via the environment
-# variable SERVICE
+# Set HADOOP_HOME to point to a specific hadoop install directory
+# HADOOP_HOME=${bin}/../../hadoop
+
+# Hive Configuration Directory can be controlled by:
+# export HIVE_CONF_DIR=
+
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+# export HIVE_AUX_JARS_PATH=
+
+# HIVE Log Directory can be controlled by and ensure it exists on the filesystem:
+# export HIVE_LOG_DIR=
+
+# The hive service being invoked (beeline etc.) is available via the environment
+# variable SERVICE
# Hive Client memory usage can be an issue if a large number of clients
-# are running at the same time. The flags below have been useful in
+# are running at the same time. The flags below have been useful in
# reducing memory usage:
-#
-# if [ "$SERVICE" = "cli" ]; then
+
+# if [ "$SERVICE" = "beeline" ]; then
# if [ -z "$DEBUG" ]; then
-# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
+# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+G1GC -XX:-UseGCOverheadLimit"
Review Comment:
Addressed in ba2c9caf135d1449ff11d061a6dbe7a5ead1fb47
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5884:
URL: https://github.com/apache/hive/pull/5884#issuecomment-3015036118
> I just don't understand why HADOOP_OPTS no longer takes effect after [HIVE-26473](https://issues.apache.org/jira/browse/HIVE-26473) was merged.
Just to clarify, i think it was not working way before HIVE-26473.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5884:
URL: https://github.com/apache/hive/pull/5884#issuecomment-2993466718
HMS log will look like
```
2025-06-21 13:00:24,689 INFO metastore.HiveMetaStore: Starting hive metastore on port 9083. PID is 7951
```
HS2 log will look like
```
2025-06-21 13:01:04,646 INFO server.HiveServer2: Starting HiveServer2. PID is: 8083
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5884:
URL: https://github.com/apache/hive/pull/5884#discussion_r2164840424
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java:
##########
@@ -262,7 +262,11 @@ public static void main(String[] args) throws Throwable {
startupShutdownMessage(HiveMetaStore.class, args, LOG);
try {
- String msg = "Starting hive metastore on port " + cli.getPort();
+ String msg =
Review Comment:
i think it's better to use logger template "{}"
why do we need to add this into `System.err`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #5884:
URL: https://github.com/apache/hive/pull/5884#issuecomment-3015039035
> > I just don't understand why HADOOP_OPTS no longer takes effect after [HIVE-26473](https://issues.apache.org/jira/browse/HIVE-26473) was merged.
>
> Just to clarify, i think it was not working way before [HIVE-26473](https://issues.apache.org/jira/browse/HIVE-26473).
OK. It might also be a issue caused by other PR. Because previously, when I conducted local tests, using `HAODOOP_OPTS` was always effective.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5884:
URL: https://github.com/apache/hive/pull/5884#discussion_r2159987338
##########
service/src/java/org/apache/hive/service/server/HiveServer2.java:
##########
@@ -1206,7 +1206,8 @@ public void startPrivilegeSynchronizer(HiveConf hiveConf) throws Exception {
private static void startHiveServer2() throws Throwable {
long attempts = 0, maxAttempts = 1;
while (true) {
- LOG.info("Starting HiveServer2");
+ long pid = ProcessHandle.current().pid();
Review Comment:
Do we need a separate var for pid extraction?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5884:
URL: https://github.com/apache/hive/pull/5884#discussion_r2160842761
##########
conf/hive-env.sh.template:
##########
@@ -18,18 +20,29 @@
# to control the execution of Hive. It should be used by admins to configure
# the Hive installation (so that users do not have to set environment variables
# or set command line parameters to get correct behavior).
-#
-# The hive service being invoked (CLI etc.) is available via the environment
-# variable SERVICE
+# Set HADOOP_HOME to point to a specific hadoop install directory
+# HADOOP_HOME=${bin}/../../hadoop
+
+# Hive Configuration Directory can be controlled by:
+# export HIVE_CONF_DIR=
+
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+# export HIVE_AUX_JARS_PATH=
+
+# HIVE Log Directory can be controlled by and ensure it exists on the filesystem:
+# export HIVE_LOG_DIR=
+
+# The hive service being invoked (beeline etc.) is available via the environment
+# variable SERVICE
# Hive Client memory usage can be an issue if a large number of clients
-# are running at the same time. The flags below have been useful in
+# are running at the same time. The flags below have been useful in
# reducing memory usage:
-#
-# if [ "$SERVICE" = "cli" ]; then
+
+# if [ "$SERVICE" = "beeline" ]; then
# if [ -z "$DEBUG" ]; then
-# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
+# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+G1GC -XX:-UseGCOverheadLimit"
Review Comment:
sorry, just to confirm, `-XX:+G1GC` was replaced with `XX:+UseG1GC` ? if yes, we still have `-XX:+G1GC` in beeline HADOOP_OPTS
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #5884:
URL: https://github.com/apache/hive/pull/5884#discussion_r2159992478
##########
service/src/java/org/apache/hive/service/server/HiveServer2.java:
##########
@@ -1206,7 +1206,8 @@ public void startPrivilegeSynchronizer(HiveConf hiveConf) throws Exception {
private static void startHiveServer2() throws Throwable {
long attempts = 0, maxAttempts = 1;
while (true) {
- LOG.info("Starting HiveServer2");
+ long pid = ProcessHandle.current().pid();
Review Comment:
We can inline it in LOG message in both HS2 and HMS, if you think that's best.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #5884:
URL: https://github.com/apache/hive/pull/5884#issuecomment-2993465441
@ayushtkn @deniskuzZ , I would like to get you opinion on this small improvement patch. I have done this in our internal hive distribution and has helped a lot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao commented on PR #5884:
URL: https://github.com/apache/hive/pull/5884#issuecomment-3015035125
> @zhangbutao , the patch that you have sent above is not what I meant regarding add-opens. Let me rephrase, It seems HADOOP_OPTS are not picked up for starting hive services no matter where you set them i.e. `bin/hive.sh`, `bin/hive-config.sh`,`bin/ext/hiveserver2.sh`, etc or atleast I couldn't make them work. Confirmed from `ps -ef | grep -i hiveserver2` output.
>
> In [HIVE-26473](https://issues.apache.org/jira/browse/HIVE-26473), all the add-opens were appended in HADOOP_CLIENT_OPTS, that's why I changed from HADOOP_OPTS => HADOOP_CLIENT_OPTS in `hive-env.sh.template` file.
Yes, I understand what you mean.
I just don't understand why `HADOOP_OPTS` no longer takes effect after HIVE-26473 was merged.
For me, using `HADOOP_CLIENT_OPTS` at present is acceptable, but I think, in the future, there might be users who are confused about the issue of `HADOOP_OPTS` being ineffective.
Maybe we can address this issue later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5884:
URL: https://github.com/apache/hive/pull/5884#discussion_r2160842761
##########
conf/hive-env.sh.template:
##########
@@ -18,18 +20,29 @@
# to control the execution of Hive. It should be used by admins to configure
# the Hive installation (so that users do not have to set environment variables
# or set command line parameters to get correct behavior).
-#
-# The hive service being invoked (CLI etc.) is available via the environment
-# variable SERVICE
+# Set HADOOP_HOME to point to a specific hadoop install directory
+# HADOOP_HOME=${bin}/../../hadoop
+
+# Hive Configuration Directory can be controlled by:
+# export HIVE_CONF_DIR=
+
+# Folder containing extra libraries required for hive compilation/execution can be controlled by:
+# export HIVE_AUX_JARS_PATH=
+
+# HIVE Log Directory can be controlled by and ensure it exists on the filesystem:
+# export HIVE_LOG_DIR=
+
+# The hive service being invoked (beeline etc.) is available via the environment
+# variable SERVICE
# Hive Client memory usage can be an issue if a large number of clients
-# are running at the same time. The flags below have been useful in
+# are running at the same time. The flags below have been useful in
# reducing memory usage:
-#
-# if [ "$SERVICE" = "cli" ]; then
+
+# if [ "$SERVICE" = "beeline" ]; then
# if [ -z "$DEBUG" ]; then
-# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+UseParNewGC -XX:-UseGCOverheadLimit"
+# export HADOOP_OPTS="$HADOOP_OPTS -XX:NewRatio=12 -Xms10m -XX:MaxHeapFreeRatio=40 -XX:MinHeapFreeRatio=15 -XX:+G1GC -XX:-UseGCOverheadLimit"
Review Comment:
sorry, just to confirm, `-XX:+G1GC` was replaced with `XX:+UseG1GC` ? if yes, we still have `-XX:+G1GC` in beeline HADOOP_OPTS and need to replace
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28983: Log HS2 and HMS PID and update hive-env.sh template [hive]
Posted by "zhangbutao (via GitHub)" <gi...@apache.org>.
zhangbutao merged PR #5884:
URL: https://github.com/apache/hive/pull/5884
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org