You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "abstractdog (via GitHub)" <gi...@apache.org> on 2026/04/15 15:52:04 UTC
[PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
abstractdog opened a new pull request, #6435:
URL: https://github.com/apache/hive/pull/6435
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099527093
##########
packaging/src/docker/conf/hive-site.xml.template:
##########
@@ -76,4 +76,25 @@
<name>hive.query.results.cache.directory</name>
<value>${HIVE_QUERY_RESULTS_CACHE_DIRECTORY}</value>
</property>
+ <property>
+ <name>hive.server2.tez.initialize.default.sessions</name>
+ <value>false</value>
Review Comment:
it's a yarn-based thing that HS2 initializes a few sessions on startup
in the standalone mode, HS2 has no control over them, just discovers them via ZK
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3100429152
##########
packaging/src/docker/Dockerfile:
##########
@@ -67,19 +67,25 @@ RUN tar -xzv \
tar -xzv \
--exclude="apache-tez-$TEZ_VERSION-bin/share" \
-f /opt/apache-tez-$TEZ_VERSION-bin.tar.gz \
- -C /opt
+ -C /opt; \
+ mkdir -p /opt/tez-snapshot;
FROM eclipse-temurin:21-jdk-ubi9-minimal AS run
ARG UID=1000
ARG HADOOP_VERSION
ARG HIVE_VERSION
ARG TEZ_VERSION
+ARG TEZ_SNAPSHOT_VERSION=
+ARG TEZ_SNAPSHOT_REPO_URL=https://repository.apache.org/content/repositories/snapshots
+
+# When snapshot jars are included, client version must match the snapshot version.
+ENV TEZ_CLIENT_VERSION=${TEZ_SNAPSHOT_VERSION:-$TEZ_VERSION}
Review Comment:
this is needed or another config to disable client version check in TezAM, otherwise we get:
```
tezam | 2026-04-17T12:50:58,952 INFO DAGAppMaster - Comparing client version with AM version, clientVersion=Unknown, AMVersion=1.0.0-SNAPSHOT
tezam | 2026-04-17T12:50:58,953 ERROR DAGAppMaster - Incompatible versions found, clientVersion=Unknown, AMVersion=1.0.0-SNAPSHOT
```
I'm fine with disabling the check:
https://github.com/apache/tez/blob/2ca44b4ba3839ff4c2c2ab2ec95e34d687f61c09/tez-dag/src/main/java/org/apache/tez/dag/app/DAGAppMaster.java#L455-L476
in which case tez site xml has to contain:
```
<property>
<name>tez.am.disable.client-version-check</name>
<value>true</value>
</property>
```
it's fine to keep TEZ_CLIENT_VERSION, but the comment should be changed then to:
```
Client version check is enabled by default in Tez AM, which is picked up from TEZ_CLIENT_VERSION env var.
```
to reflect that this is not because of the optional snapshot override
wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6435:
URL: https://github.com/apache/hive/pull/6435#issuecomment-4267150054
FYI, there is a docker profile to create new Hive image: -Pdocker
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3100300911
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -124,14 +124,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
+ image: apache/hive:${HIVE_VERSION}
+ container_name: tezam
+ hostname: tezam
+ depends_on:
+ - zookeeper
+ restart: on-failure:3
+ environment:
+ USER: hive
+ SERVICE_NAME: 'tezam'
+
+ TEZ_FRAMEWORK_MODE: STANDALONE_ZOOKEEPER
+ TEZ_AM_ZOOKEEPER_QUORUM: zookeeper:2181
+
+ # LLAP daemon discovery
+ HIVE_ZOOKEEPER_QUORUM: zookeeper:2181
+ LLAP_SERVICE_HOSTS: '@llap0'
Review Comment:
good catch, it's not correct, the only reason why it works is that I take care of it one level deeper in entrypoint:
https://github.com/apache/hive/pull/6435/changes#diff-b7d5fbeab2c6af92616ab371fa3f237867d60d7c81072045342cb44a8981bf90R45
fixing this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3130721059
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -132,14 +132,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
Review Comment:
what would this standalone mode look like? you mean: hs2 + tezam + tezchild (class single task executor)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099695834
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -124,14 +124,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
+ image: apache/hive:${HIVE_VERSION}
+ container_name: tezam
+ hostname: tezam
+ depends_on:
+ - zookeeper
+ restart: on-failure:3
+ environment:
+ USER: hive
+ SERVICE_NAME: 'tezam'
+
+ TEZ_FRAMEWORK_MODE: STANDALONE_ZOOKEEPER
+ TEZ_AM_ZOOKEEPER_QUORUM: zookeeper:2181
+
+ # LLAP daemon discovery
+ HIVE_ZOOKEEPER_QUORUM: zookeeper:2181
+ LLAP_SERVICE_HOSTS: '@llap0'
Review Comment:
not `HIVE_LLAP_DAEMON_SERVICE_HOSTS` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6435:
URL: https://github.com/apache/hive/pull/6435#issuecomment-4306028404
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6435) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6435&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6435&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6435&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6435&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6435&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6435)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3130257871
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -132,14 +132,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
Review Comment:
should we add standalone support later? without llap daemons -> Tez containers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3131925233
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -132,14 +132,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
Review Comment:
yea, something like that. not useful?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099646778
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -124,14 +124,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
+ image: apache/hive:${HIVE_VERSION}
Review Comment:
is it Ok to reuse hive image for TezAM, shouldn't we build Tez specific one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3101304811
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
short term, **yes**, otherwise the whole TezAM initiative cannot be tried out as here
Tez 1.0.0 could be months (or more) away, and I would like to be able to proceed with cloud-native initiative in the meantime
released/published Docker images must not contain SNAPSHOT jars, I agree, but a temporary Hive image built with Tez 1.0.0-SNAPSHOT can be utilized for distributed/perf test even in the next 1-2 months
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3146371844
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -132,14 +132,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
Review Comment:
sorry, forgot to respond: short term, it's not worth the effort from Hive project's point of view, given that we adapted at least one of our main execution modes (LLAP vs. Tez container mode), LLAP, to cloud, but long term, it's a good-to-have and not only for Hive, but Tez also, that's what [TEZ-4662](https://issues.apache.org/jira/browse/TEZ-4662) is about
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3102542702
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
would it make sense splitting this PR in 2 parts so we could revert the SNAPSHOT workaround once Tez 1.0.0 is available?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099646778
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -124,14 +124,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
+ image: apache/hive:${HIVE_VERSION}
Review Comment:
is it Ok to reuse hive image for TezAM, shouldn't we build Tez specific one? is is the same downstream?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099554506
##########
packaging/src/docker/conf/hive-site.xml.template:
##########
@@ -76,4 +76,25 @@
<name>hive.query.results.cache.directory</name>
<value>${HIVE_QUERY_RESULTS_CACHE_DIRECTORY}</value>
</property>
+ <property>
+ <name>hive.server2.tez.initialize.default.sessions</name>
+ <value>false</value>
+ </property>
+ <property>
+ <name>hive.server2.tez.use.external.sessions</name>
+ <value>${HIVE_SERVER2_TEZ_USE_EXTERNAL_SESSIONS}</value>
+ </property>
+ <!--
+ A registry namespace prefix is a hardcoded prefix for Tez external sessions.
+ The actual tez.am.registry.namespace value is appended to this prefix.
+ Once hive can use the registry client that Tez provides (ZkAMRegistryClient), this property will be removed.
Review Comment:
for the release we also have: https://issues.apache.org/jira/browse/TEZ-4701
now I created: https://issues.apache.org/jira/browse/HIVE-29573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099549449
##########
packaging/src/docker/conf/tezam-log4j.properties:
##########
@@ -0,0 +1,5 @@
+log4j.rootLogger=INFO, console
Review Comment:
maybe, simply `[tez-log4j.properties]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3100316889
##########
packaging/src/docker/conf/tez-site.xml.template:
##########
@@ -0,0 +1,74 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+ <property>
+ <name>tez.am.mode.session</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>tez.am.framework.mode</name>
+ <value>${TEZ_FRAMEWORK_MODE}</value>
+ </property>
+ <property>
+ <name>tez.am.zookeeper.quorum</name>
+ <value>${TEZ_AM_ZOOKEEPER_QUORUM}</value>
+ </property>
+ <property>
+ <name>tez.am.registry.namespace</name>
+ <value>${TEZ_AM_REGISTRY_NAMESPACE}</value>
+ </property>
+ <property>
+ <name>tez.local.mode</name>
+ <value>false</value>
+ </property>
+ <property>
+ <name>tez.am.tez-ui.webservice.enable</name>
+ <value>false</value>
+ </property>
+ <!-- Tez AM should not timeout in ZK Mode -->
+ <property>
+ <name>tez.session.am.dag.submit.timeout.secs</name>
+ <value>-1</value>
+ </property>
+ <property>
+ <name>tez.ignore.lib.uris</name>
+ <value>true</value>
+ </property>
+
+ <!--
+ In standalone AM mode with LLAP, the task scheduler (LlapTaskSchedulerService) and task
+ communicator (LlapTaskCommunicator) are instantiated at AM startup from service_plugins_descriptor.json
+ on the classpath — not from the DAG payload as in YARN-submitted mode. These plugins read their
+ configuration from TezConfiguration, which is built exclusively from tez-site.xml on the classpath.
+ Hive-specific properties must therefore appear here so the AM can find LLAP daemons and connect
+ to them, even though they are conceptually Hive settings.
+ -->
+ <property>
+ <name>hive.zookeeper.quorum</name>
+ <value>${HIVE_ZOOKEEPER_QUORUM}</value>
+ </property>
+ <property>
+ <name>hive.llap.daemon.service.hosts</name>
+ <value>${HIVE_LLAP_DAEMON_SERVICE_HOSTS}</value>
+ </property>
+ <property>
+ <name>hive.llap.daemon.umbilical.port</name>
+ <value>33333</value>
Review Comment:
right, it works but needs a minor fix in LlapTaskCommunicator to handle the default properly:
```
- conf.get(HiveConf.ConfVars.LLAP_TASK_UMBILICAL_SERVER_PORT.varname)
+ HiveConf.getVar(conf, HiveConf.ConfVars.LLAP_TASK_UMBILICAL_SERVER_PORT)
```
otherwise I get:
```
tezam | 2026-04-17T12:23:28,727 INFO AbstractService - Service org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator failed in state STARTED
tezam | java.lang.NullPointerException: Cannot invoke "String.split(String)" because the return value of "org.apache.hadoop.conf.Configuration.get(String)" is null
tezam | at org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.startRpcServer(LlapTaskCommunicator.java:261)
tezam | at org.apache.tez.dag.app.TezTaskCommunicatorImpl.start(TezTaskCommunicatorImpl.java:140)
tezam | at org.apache.hadoop.hive.llap.tezplugins.LlapTaskCommunicator.start(LlapTaskCommunicator.java:237)
tezam | at org.apache.tez.dag.app.ServicePluginLifecycleAbstractService.serviceStart(ServicePluginLifecycleAbstractService.java:41)
tezam | at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
tezam | at org.apache.tez.dag.app.TaskCommunicatorManager.serviceStart(TaskCommunicatorManager.java:165)
tezam | at org.apache.hadoop.service.AbstractService.start(AbstractService.java:195)
tezam | at org.apache.tez.dag.app.DAGAppMaster$ServiceWithDependency.start(DAGAppMaster.java:1857)
tezam | at org.apache.tez.dag.app.DAGAppMaster$ServiceThread.run(DAGAppMaster.java:1878)
```
good catch, fixing it now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099569359
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
can we combine and use just TEZ_VERSION? check local mvn and if not present go with tarball?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099756277
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -124,14 +124,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
+ image: apache/hive:${HIVE_VERSION}
Review Comment:
I think it's ok to reuse, actually, we build both (Hive specific TezAM image + Tez AM image in Tez project), see the motivation on [HIVE-29419](https://issues.apache.org/jira/browse/HIVE-29419)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099554506
##########
packaging/src/docker/conf/hive-site.xml.template:
##########
@@ -76,4 +76,25 @@
<name>hive.query.results.cache.directory</name>
<value>${HIVE_QUERY_RESULTS_CACHE_DIRECTORY}</value>
</property>
+ <property>
+ <name>hive.server2.tez.initialize.default.sessions</name>
+ <value>false</value>
+ </property>
+ <property>
+ <name>hive.server2.tez.use.external.sessions</name>
+ <value>${HIVE_SERVER2_TEZ_USE_EXTERNAL_SESSIONS}</value>
+ </property>
+ <!--
+ A registry namespace prefix is a hardcoded prefix for Tez external sessions.
+ The actual tez.am.registry.namespace value is appended to this prefix.
+ Once hive can use the registry client that Tez provides (ZkAMRegistryClient), this property will be removed.
Review Comment:
for the release we already have: https://issues.apache.org/jira/browse/TEZ-4701
now I created: https://issues.apache.org/jira/browse/HIVE-29573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099569359
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
can we combine and use just TEZ_VERSION? check local mvn and if not present go with tarball?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3102556350
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
maybe a cleaner option might be using Tez nightly docker image and patch it with Hive jars?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099756277
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -124,14 +124,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
+ image: apache/hive:${HIVE_VERSION}
Review Comment:
I think it's ok to reuse, actually, we build both (Hive specific TezAM image + Tez AM image in Tez project), see the motivation on [HIVE-29419](https://issues.apache.org/jira/browse/HIVE-29419)
downstream we have only a hive image for TezAM, but upstream, the Tez project needs to have its own, [TEZ-4682](https://issues.apache.org/jira/browse/TEZ-4682), most probably mimics the Tez Container mode (DAGAppMaster + TezChiild), not the LLAP
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099646778
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -124,14 +124,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
+ image: apache/hive:${HIVE_VERSION}
Review Comment:
is it Ok to reuse hive image for TezAM, IDK just asking? is is the same downstream?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3101233888
##########
packaging/src/docker/Dockerfile:
##########
@@ -67,19 +67,25 @@ RUN tar -xzv \
tar -xzv \
--exclude="apache-tez-$TEZ_VERSION-bin/share" \
-f /opt/apache-tez-$TEZ_VERSION-bin.tar.gz \
- -C /opt
+ -C /opt; \
+ mkdir -p /opt/tez-snapshot;
FROM eclipse-temurin:21-jdk-ubi9-minimal AS run
ARG UID=1000
ARG HADOOP_VERSION
ARG HIVE_VERSION
ARG TEZ_VERSION
+ARG TEZ_SNAPSHOT_VERSION=
+ARG TEZ_SNAPSHOT_REPO_URL=https://repository.apache.org/content/repositories/snapshots
+
+# When snapshot jars are included, client version must match the snapshot version.
+ENV TEZ_CLIENT_VERSION=${TEZ_SNAPSHOT_VERSION:-$TEZ_VERSION}
Review Comment:
if version check makes sense, let's keep the env var
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3130721059
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -132,14 +132,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
Review Comment:
what would this standalone mode look like? you mean: hs2 + tezam + tezchild (classic single task executor)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3129736268
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
I'm open to splitting this into 2 PRs
regarding tez base image: yeah, theoretically, hive image can depend upon the Tez one, but I still feel that in order to be able to move fast later, hive has to build its standalone TezAM image without actually depending on the Tez image, similarly to Tez doesn't depend on the Hadoop, and also Hive image so far doesn't depend on the Hadoop image (https://hub.docker.com/r/apache/hadoop/tags)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3130257871
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -132,14 +132,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
Review Comment:
should we add standalone support later? without llap daemons - Tez containers?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #6435:
URL: https://github.com/apache/hive/pull/6435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3101259971
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
Do we need to support SNAPSHOT overlays? Currently, the Docker image only allows released versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3101304811
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
short term, **yes**, otherwise the whole TezAM initiative cannot be tried out as here
Tez 1.0.0 could be months (or more) away, and I would like to be able to proceed with cloud-native initiative in the meantime
released/published Docker images must contain SNAPSHOT-only jars, I agree, but the a temporary Hive image built with Tez 1.0.0-SNAPSHOT can be utilized for distributed/perf test even in 1-2 months
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #6435:
URL: https://github.com/apache/hive/pull/6435#issuecomment-4267283130
why not create and publish Tez docker image on Tez side and reuse it Hive? building it here with snapshot jars patching looks not very strait-forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099695834
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -124,14 +124,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
+ image: apache/hive:${HIVE_VERSION}
+ container_name: tezam
+ hostname: tezam
+ depends_on:
+ - zookeeper
+ restart: on-failure:3
+ environment:
+ USER: hive
+ SERVICE_NAME: 'tezam'
+
+ TEZ_FRAMEWORK_MODE: STANDALONE_ZOOKEEPER
+ TEZ_AM_ZOOKEEPER_QUORUM: zookeeper:2181
+
+ # LLAP daemon discovery
+ HIVE_ZOOKEEPER_QUORUM: zookeeper:2181
+ LLAP_SERVICE_HOSTS: '@llap0'
Review Comment:
is the env var correct? not `HIVE_LLAP_DAEMON_SERVICE_HOSTS` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099560629
##########
packaging/src/docker/conf/tez-site.xml.template:
##########
@@ -0,0 +1,74 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+ <property>
+ <name>tez.am.mode.session</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>tez.am.framework.mode</name>
+ <value>${TEZ_FRAMEWORK_MODE}</value>
+ </property>
+ <property>
+ <name>tez.am.zookeeper.quorum</name>
+ <value>${TEZ_AM_ZOOKEEPER_QUORUM}</value>
+ </property>
+ <property>
+ <name>tez.am.registry.namespace</name>
+ <value>${TEZ_AM_REGISTRY_NAMESPACE}</value>
+ </property>
+ <property>
+ <name>tez.local.mode</name>
+ <value>false</value>
+ </property>
+ <property>
+ <name>tez.am.tez-ui.webservice.enable</name>
+ <value>false</value>
+ </property>
+ <!-- Tez AM should not timeout in ZK Mode -->
+ <property>
+ <name>tez.session.am.dag.submit.timeout.secs</name>
+ <value>-1</value>
+ </property>
+ <property>
+ <name>tez.ignore.lib.uris</name>
+ <value>true</value>
+ </property>
+
+ <!--
+ In standalone AM mode with LLAP, the task scheduler (LlapTaskSchedulerService) and task
+ communicator (LlapTaskCommunicator) are instantiated at AM startup from service_plugins_descriptor.json
+ on the classpath — not from the DAG payload as in YARN-submitted mode. These plugins read their
+ configuration from TezConfiguration, which is built exclusively from tez-site.xml on the classpath.
+ Hive-specific properties must therefore appear here so the AM can find LLAP daemons and connect
+ to them, even though they are conceptually Hive settings.
+ -->
+ <property>
+ <name>hive.zookeeper.quorum</name>
+ <value>${HIVE_ZOOKEEPER_QUORUM}</value>
+ </property>
+ <property>
+ <name>hive.llap.daemon.service.hosts</name>
+ <value>${HIVE_LLAP_DAEMON_SERVICE_HOSTS}</value>
+ </property>
+ <property>
+ <name>hive.llap.daemon.umbilical.port</name>
+ <value>33333</value>
Review Comment:
default is 0, do we even need to touch it
##########
packaging/src/docker/conf/tez-site.xml.template:
##########
@@ -0,0 +1,74 @@
+<?xml version="1.0"?>
+<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+<configuration>
+ <property>
+ <name>tez.am.mode.session</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>tez.am.framework.mode</name>
+ <value>${TEZ_FRAMEWORK_MODE}</value>
+ </property>
+ <property>
+ <name>tez.am.zookeeper.quorum</name>
+ <value>${TEZ_AM_ZOOKEEPER_QUORUM}</value>
+ </property>
+ <property>
+ <name>tez.am.registry.namespace</name>
+ <value>${TEZ_AM_REGISTRY_NAMESPACE}</value>
+ </property>
+ <property>
+ <name>tez.local.mode</name>
+ <value>false</value>
+ </property>
+ <property>
+ <name>tez.am.tez-ui.webservice.enable</name>
+ <value>false</value>
+ </property>
+ <!-- Tez AM should not timeout in ZK Mode -->
+ <property>
+ <name>tez.session.am.dag.submit.timeout.secs</name>
+ <value>-1</value>
+ </property>
+ <property>
+ <name>tez.ignore.lib.uris</name>
+ <value>true</value>
+ </property>
+
+ <!--
+ In standalone AM mode with LLAP, the task scheduler (LlapTaskSchedulerService) and task
+ communicator (LlapTaskCommunicator) are instantiated at AM startup from service_plugins_descriptor.json
+ on the classpath — not from the DAG payload as in YARN-submitted mode. These plugins read their
+ configuration from TezConfiguration, which is built exclusively from tez-site.xml on the classpath.
+ Hive-specific properties must therefore appear here so the AM can find LLAP daemons and connect
+ to them, even though they are conceptually Hive settings.
+ -->
+ <property>
+ <name>hive.zookeeper.quorum</name>
+ <value>${HIVE_ZOOKEEPER_QUORUM}</value>
+ </property>
+ <property>
+ <name>hive.llap.daemon.service.hosts</name>
+ <value>${HIVE_LLAP_DAEMON_SERVICE_HOSTS}</value>
+ </property>
+ <property>
+ <name>hive.llap.daemon.umbilical.port</name>
+ <value>33333</value>
Review Comment:
default is 0, do we even need to touch it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099695834
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -124,14 +124,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
+ image: apache/hive:${HIVE_VERSION}
+ container_name: tezam
+ hostname: tezam
+ depends_on:
+ - zookeeper
+ restart: on-failure:3
+ environment:
+ USER: hive
+ SERVICE_NAME: 'tezam'
+
+ TEZ_FRAMEWORK_MODE: STANDALONE_ZOOKEEPER
+ TEZ_AM_ZOOKEEPER_QUORUM: zookeeper:2181
+
+ # LLAP daemon discovery
+ HIVE_ZOOKEEPER_QUORUM: zookeeper:2181
+ LLAP_SERVICE_HOSTS: '@llap0'
Review Comment:
is the env var correct? not `HIVE_LLAP_DAEMON_SERVICE_HOSTS` ? see https://github.com/apache/hive/pull/6435/changes#diff-d389e28124c68afe85354e9f947d6d9ec8f7cfb9d49c46b39891ab5da7a7be7cR68
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099646778
##########
packaging/src/docker/docker-compose.yml:
##########
@@ -124,14 +124,35 @@ services:
- zookeeper_datalog:/datalog
- zookeeper_logs:/logs
- #TODO Tez AM container (in the meantime, the HS2(with local Tez AM) + LLAP daemon setup is working properly)
- # 1. Define and use a Tez AM image from HIVE-29419 or TEZ-4682
- # 2. Configure TezAM to use Zookeeper Llap Registry to discover the LLAP daemon
- # 3. Configure HiveServer2 to use the Tez AM Zookeeper Registry to discover the Tez AM
- # Prerequisites:
- # - tez-api 1.0.0-SNAPSHOT jar injected into HiveSever2 until Tez 1.0.0 is released
- # - make HIVE-29477 happen to let HiveServer2 use Tez external sessions
- # 4. Define hadoop components here to be used by all the containers (working example can be found at TEZ-4682), currently a local volume
+ tezam:
+ profiles:
+ - llap
+ image: apache/hive:${HIVE_VERSION}
Review Comment:
is it Ok to reuse hive image for TezAM? IDK just asking. is is the same downstream?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099504925
##########
packaging/src/docker/conf/hive-site.xml.template:
##########
@@ -76,4 +76,25 @@
<name>hive.query.results.cache.directory</name>
<value>${HIVE_QUERY_RESULTS_CACHE_DIRECTORY}</value>
</property>
+ <property>
+ <name>hive.server2.tez.initialize.default.sessions</name>
+ <value>false</value>
+ </property>
+ <property>
+ <name>hive.server2.tez.use.external.sessions</name>
+ <value>${HIVE_SERVER2_TEZ_USE_EXTERNAL_SESSIONS}</value>
+ </property>
+ <!--
+ A registry namespace prefix is a hardcoded prefix for Tez external sessions.
+ The actual tez.am.registry.namespace value is appended to this prefix.
+ Once hive can use the registry client that Tez provides (ZkAMRegistryClient), this property will be removed.
Review Comment:
is there a ticket for that? needs Tez upgrade?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099718182
##########
packaging/src/docker/Dockerfile:
##########
@@ -67,19 +67,25 @@ RUN tar -xzv \
tar -xzv \
--exclude="apache-tez-$TEZ_VERSION-bin/share" \
-f /opt/apache-tez-$TEZ_VERSION-bin.tar.gz \
- -C /opt
+ -C /opt; \
+ mkdir -p /opt/tez-snapshot;
FROM eclipse-temurin:21-jdk-ubi9-minimal AS run
ARG UID=1000
ARG HADOOP_VERSION
ARG HIVE_VERSION
ARG TEZ_VERSION
+ARG TEZ_SNAPSHOT_VERSION=
+ARG TEZ_SNAPSHOT_REPO_URL=https://repository.apache.org/content/repositories/snapshots
+
+# When snapshot jars are included, client version must match the snapshot version.
+ENV TEZ_CLIENT_VERSION=${TEZ_SNAPSHOT_VERSION:-$TEZ_VERSION}
Review Comment:
is this used somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3102542702
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
would it make sense splitting this PR in 2 parts so we could revert the SNAPSHOT workaround once Tez 1.0.0 is released?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3102556350
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
maybe a cleaner approach could be to build a custom Tez Docker image on top of the nightly Tez image, extending it with the required Hive jars.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3102556350
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
maybe a cleaner option might be creating a custom Tez docker image based on Tez nightly image and patched with Hive jars?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099490882
##########
packaging/src/docker/conf/hive-site.xml.template:
##########
@@ -76,4 +76,25 @@
<name>hive.query.results.cache.directory</name>
<value>${HIVE_QUERY_RESULTS_CACHE_DIRECTORY}</value>
</property>
+ <property>
+ <name>hive.server2.tez.initialize.default.sessions</name>
+ <value>false</value>
Review Comment:
why not initialize ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099569359
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
can we combine and use just TEZ_VERSION?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099718182
##########
packaging/src/docker/Dockerfile:
##########
@@ -67,19 +67,25 @@ RUN tar -xzv \
tar -xzv \
--exclude="apache-tez-$TEZ_VERSION-bin/share" \
-f /opt/apache-tez-$TEZ_VERSION-bin.tar.gz \
- -C /opt
+ -C /opt; \
+ mkdir -p /opt/tez-snapshot;
FROM eclipse-temurin:21-jdk-ubi9-minimal AS run
ARG UID=1000
ARG HADOOP_VERSION
ARG HIVE_VERSION
ARG TEZ_VERSION
+ARG TEZ_SNAPSHOT_VERSION=
+ARG TEZ_SNAPSHOT_REPO_URL=https://repository.apache.org/content/repositories/snapshots
+
+# When snapshot jars are included, client version must match the snapshot version.
+ENV TEZ_CLIENT_VERSION=${TEZ_SNAPSHOT_VERSION:-$TEZ_VERSION}
Review Comment:
is this used somewhere? can't find any usages here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3099743181
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
could we keep one required input in [build.sh]: -tez <version>
If it ends with -SNAPSHOT, treat it as snapshot channel automatically - otherwise release (for example 0.10.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6435:
URL: https://github.com/apache/hive/pull/6435#discussion_r3100948813
##########
packaging/src/docker/build.sh:
##########
@@ -20,13 +20,15 @@ set -eux
HIVE_VERSION=
HADOOP_VERSION=
TEZ_VERSION=
+TEZ_SNAPSHOT_VERSION=
Review Comment:
a release tarball must always be fetched (as the structural foundation, mostly because it contains all the dependencies for Tez), and snapshot jars are an optional overlay, placed on top to override the release-version tez jars while leaving lib/ intact
a snapshot version alone cannot form a self-contained, runnable Tez installation, as we don't have SNAPSHOT tez tarballs downloadable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6435:
URL: https://github.com/apache/hive/pull/6435#issuecomment-4254118846
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6435) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6435&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6435&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6435&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6435&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6435&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6435)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6435:
URL: https://github.com/apache/hive/pull/6435#issuecomment-4258203581
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6435) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6435&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6435&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6435&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6435&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6435&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6435)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29419: Provide a Hive-specific docker image for Tez AM [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6435:
URL: https://github.com/apache/hive/pull/6435#issuecomment-4303589525
@deniskuzZ: pushed the latest commit without snapshot changes, validated that it can run properly without the tez snapshots (hs2 + llap):
```
export HIVE_VERSION=4.2.0
./stop-hive.sh --cleanup
export POSTGRES_LOCAL_PATH=~/.m2/repository/org/postgresql/postgresql/42.7.3/postgresql-42.7.3.jar
./build.sh -hive 4.2.0 -hadoop 3.4.1 -tez 0.10.5
./start-hive.sh --llap
docker compose --profile llap logs -f
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org