You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "ayushtkn (via GitHub)" <gi...@apache.org> on 2026/04/07 18:00:37 UTC
[PR] HIVE-29552: Add docker scripts for Ozone [hive]
ayushtkn opened a new pull request, #6414:
URL: https://github.com/apache/hive/pull/6414
### What changes were proposed in this pull request?
Add a way to run Hive with Ozone as backend storage
### Why are the changes needed?
For using Ozone in Hive in Docker env
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Deployed locally
```
export HIVE_VERSION=4.2.0
mvn dependency:copy -Dartifact="org.postgresql:postgresql:42.5.1" && \
export POSTGRES_LOCAL_PATH=`mvn help:evaluate -Dexpression=settings.localRepository -q -DforceStdout`/org/postgresql/postgresql/42.5.1/postgresql-42.5.1.jar
./start-hive.sh --ozone
```
<img width="1728" height="297" alt="image" src="https://github.com/user-attachments/assets/06b2e6bf-9787-49f3-acfe-e94941940193" />
```
docker exec -it hiveserver2 beeline -u 'jdbc:hive2://localhost:10000/'
```
Created Tables
```
create table emp(id int) stored by iceberg;
insert into emp values (1),(2),(3),(4)
```
Verified data is in Ozone
<img width="3456" height="1274" alt="image" src="https://github.com/user-attachments/assets/d666a9c6-80a1-40e7-b20e-722bc6309822" />
Ozone Shell
<img width="3456" height="626" alt="image" src="https://github.com/user-attachments/assets/6fbe9092-5497-4b43-8cdc-933ddcd8a34f" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn merged PR #6414:
URL: https://github.com/apache/hive/pull/6414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6414:
URL: https://github.com/apache/hive/pull/6414#issuecomment-4210567763
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6414) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6414&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6414&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6414&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6414&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6414&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6414)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #6414:
URL: https://github.com/apache/hive/pull/6414#issuecomment-4208474421
Thanx @abstractdog for the review. I have added changes for LLAP
<img width="3454" height="1166" alt="image" src="https://github.com/user-attachments/assets/2a174938-868e-4898-acda-641d10c7db5f" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6414:
URL: https://github.com/apache/hive/pull/6414#issuecomment-4202869715
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6414) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6414&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6414&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6414&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6414&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6414&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6414)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6414:
URL: https://github.com/apache/hive/pull/6414#issuecomment-4206657617
thanks @ayushtkn, looks very good so far!
it worked with:
```
./start-hive.sh --ozone
```
also tried with:
```
./start-hive.sh --ozone --llap
```
but I hit this below:
[ozone_llap_failure.log](https://github.com/user-attachments/files/26570050/ozone_llap_failure.log)
is there a chance you can make this work easily? having ozone in the fully distributed (--llap) would be awesome
I don't necessarily want to block this initiative for that (thinking of a follow-up ticket), but there is hope that you can make it work 💪 🚀
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6414:
URL: https://github.com/apache/hive/pull/6414#issuecomment-4214195531
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6414) **Quality Gate passed**
Issues
 [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6414&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6414&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6414&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6414&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6414&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6414)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6414:
URL: https://github.com/apache/hive/pull/6414#issuecomment-4211605001
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6414) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6414&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6414&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6414&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6414&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6414&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6414)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6414:
URL: https://github.com/apache/hive/pull/6414#discussion_r3057192927
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -155,6 +155,12 @@ services:
LLAP_WEB_PORT: '15001'
LLAP_MANAGEMENT_RPC_PORT: '15004'
LLAP_SHUFFLE_PORT: '15551'
+
+ DEFAULT_FS: "${DEFAULT_FS}"
+ HADOOP_CLASSPATH: /opt/hadoop/share/hadoop/tools/lib/*
Review Comment:
`HADOOP_CLASSPATH` is not needed here as in the entrypoint, classpath is defined as `$HADOOP_HOME/share/hadoop/tools/lib/*`
I tried it without, worked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6414:
URL: https://github.com/apache/hive/pull/6414#discussion_r3051462205
##########
packaging/src/docker/storage/ozone/docker-compose.yml:
##########
@@ -0,0 +1,119 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+version: "3"
Review Comment:
version is deprecated as far as I know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6414:
URL: https://github.com/apache/hive/pull/6414#discussion_r3057192927
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -155,6 +155,12 @@ services:
LLAP_WEB_PORT: '15001'
LLAP_MANAGEMENT_RPC_PORT: '15004'
LLAP_SHUFFLE_PORT: '15551'
+
+ DEFAULT_FS: "${DEFAULT_FS}"
+ HADOOP_CLASSPATH: /opt/hadoop/share/hadoop/tools/lib/*
Review Comment:
`HADOOP_CLASSPATH` is not needed here as in the entrypoint, classpath is defined as `$HADOOP_HOME/share/hadoop/tools/lib/*`
I tried it without this, worked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6414:
URL: https://github.com/apache/hive/pull/6414#discussion_r3051754575
##########
packaging/src/docker/stop-hive.sh:
##########
@@ -21,19 +21,25 @@ cd "$SCRIPT_DIR"
PROFILE="--profile llap" # delete all containers regardless of profile
CLEANUP_FLAG=""
+COMPOSE_FILES="docker-compose.yml"
for arg in "$@"; do
case "$arg" in
--cleanup)
CLEANUP_FLAG="--volumes"
;;
+ --ozone)
Review Comment:
I feel that we don't need `--ozone` flag for stop script, simply delete ozone containers all the time
while working on the LLAP image, we made it work by `stop-hive.sh` (no `--llap`) regardless of the execution mode, so similarly, `stop-hive.sh` should delete all the containers regardless of the storage, as in the context of dockerized hive testing, it makes no sense to keep the storage containers running beyond the hive containers: we should not let the user to achieve this by accidentally calling `stop-hive.sh` without `--ozone`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29552: Add docker scripts for Ozone [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6414:
URL: https://github.com/apache/hive/pull/6414#discussion_r3051754575
##########
packaging/src/docker/stop-hive.sh:
##########
@@ -21,19 +21,25 @@ cd "$SCRIPT_DIR"
PROFILE="--profile llap" # delete all containers regardless of profile
CLEANUP_FLAG=""
+COMPOSE_FILES="docker-compose.yml"
for arg in "$@"; do
case "$arg" in
--cleanup)
CLEANUP_FLAG="--volumes"
;;
+ --ozone)
Review Comment:
I feel that we don't need `--ozone` flag for stop script, simply delete ozone containers all the time
while working on the LLAP image, we made it work by `stop-hive.sh` (no `--llap`) regardless of the execution mode, so similarly, `stop-hive.sh` should delete all the containers regardless of the storage, as in the context of dockerized hive testing, it makes no sense to keep the storage containers running beyond the hive containers: we should not let the user to achieve this by accidentally calling `stop-hive.sh` without `--ozone`
later we will have more traits to be added to this cluster (even concurrently: like `start-hive.sh --ozone --another_storage --some_amazing_thirdparty_catalog`, I wish the user can simply stop the cluster without recalling all the arguments used upon creation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org