You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2022/07/18 02:24:17 UTC
[GitHub] [hive] dengzhhu653 opened a new pull request, #3448: [WIP] HIVE-26400: Provide a self-contained docker
dengzhhu653 opened a new pull request, #3448:
URL: https://github.com/apache/hive/pull/3448
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
abstractdog commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1193615641
is there a chance that we can build this docker image + run simple queries in precommit time? otherwise, I'm afraid we cannot guarantee the stability of this feature
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1378437752
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1458842098
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1035785502
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
+```
+
+- Launch a single standalone HiveServer2 for a quick start
+
+The HiveServer2 will be started with an embedded Metastore by initiating:
+```shell
+docker run --name hiveserver2-standalone apachehive/hiveserver2:$HIVE_VERSION
+```
+The data of the HiveServer2 would be lost between container restarts.
+
+- Launch a cluster with HiveServer2, Metastore and MySQL as its back database.
+
+To save data between container restarts, Volumes is used to persist data generated by and used by Hive. Just by executing:
+```shell
+export HIVESERVER2_IMAGE=apachehive/hiveserver2:$HIVE_VERSION
+export METASTORE_IMAGE=apachehive/metastore:$HIVE_VERSION
+docker network create hive && docker-compose -f docker-compose.yml up -d
+```
+
+#### Usage
+
+---
+- Show the containers
+```shell
+docker ps
+```
+- Check HiveServer2 web ui
+ - Accessed on browser at http://localhost:10002/
+- Run Beeline:
+```shell
+docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1297096210
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1357261284
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
abstractdog commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1320055622
> @zabetak @abstractdog what is to be done for this here. This seems like very useful and easier way to run hive services out of the box than dev-box setup though less powerful. the dev-box setup is not part of hive codebase as I understand it. What is it we want to complete before this can be merged? Tahnk you
I haven't picked up the context here yet
need to check what we have at the moment, I'll try out what was implemented here
my expectation is that I can run different hive components very conveniently on my local machine according to a readme (also included in the repo), and if so, this PR is good to be merged in, let me check next week
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1396511930
> I had some comments earlier, if you can confirm you addressed those @dengzhhu653 , I'll take a second look and approve, it's time to merge this I believe please create an umbrella ticket for hive docker improvements and add this one as the first sub-jira, let's track further improvements there
I've merged the two images into only one(apache/hive), and create a parent jira to track the improvements, please take a look if have time, thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] TuroczyX commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "TuroczyX (via GitHub)" <gi...@apache.org>.
TuroczyX commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1509108262
Any update on this? :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zratkai commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "zratkai (via GitHub)" <gi...@apache.org>.
zratkai commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1511018569
I think it would be a good idea to add ports for development / debugging like this ones:
-p9866:9866 -p10000:10000 -p10001:10001 -p9000:9000 -p8000:8000 -p3306:3306 -p50070:50070 -p50030:50030
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1332020803
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1036995252
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
Review Comment:
we should also consider using a single hive image:
```
apache/hive
```
and according to an docker parameter, it can start as hs2 or metastore driven from docker-compose (btw, this would be quite similar to Cloudera's CDW)
this might be important and convenient, as a single image is easier to maintain, and looks much better than having apache/hive-hiveserver2 and apache/hive-metastore images separately later when we really distribute the image(s) to public repo
I know it changes the current implementation here, would you consider trying it @dengzhhu653 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1487920458
> > Should be included in this initiative also create an docker image for the hive metastore standalone?
> > Something like this: https://techjogging.com/standalone-hive-metastore-presto-docker.html
> > https://github.com/arempter/hive-metastore-docker/blob/master/Dockerfile
> > https://github.com/aws-samples/hive-emr-on-eks/blob/main/docker/Dockerfile
> > Thanks
>
> It is a good point. @dengzhhu653 @deniskuzZ @ayushtkn @abstractdog What do you think about it?
The image can serve both HS2 and Metastore, as you can see in the README: https://github.com/apache/hive/pull/3448/files#diff-75345b4702a737ff955983bea3daeac9243e26ef1d2dc0398a31ef28380da9cb. Separating them needs another build, makes it a bit hard to maintain in the public repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 closed pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
dengzhhu653 closed pull request #3448: HIVE-26400: Provide docker images for Hive
URL: https://github.com/apache/hive/pull/3448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1483681136
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] TuroczyX commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "TuroczyX (via GitHub)" <gi...@apache.org>.
TuroczyX commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1488936926
> > > Should be included in this initiative also create an docker image for the hive metastore standalone?
> > > Something like this: https://techjogging.com/standalone-hive-metastore-presto-docker.html
> > > https://github.com/arempter/hive-metastore-docker/blob/master/Dockerfile
> > > https://github.com/aws-samples/hive-emr-on-eks/blob/main/docker/Dockerfile
> > > Thanks
> >
> >
> > It is a good point. @dengzhhu653 @deniskuzZ @ayushtkn @abstractdog What do you think about it?
>
> The image can serve both HS2 and Metastore, as you can see in the README: https://github.com/apache/hive/pull/3448/files#diff-75345b4702a737ff955983bea3daeac9243e26ef1d2dc0398a31ef28380da9cb. Separating them needs another build, makes it a bit hard to maintain in the public repo.
Understandable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] achennagiri commented on a diff in pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
achennagiri commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r930507758
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,125 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions to build images.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- deploy.sh Entrance to build images and run them.
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself.
+Providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the versions are not given during build, then it will read the version info from project top `pom.xml`: project.version, hadoop.version, tez.version,
+these are assigned to hive version, hadoop version, tez version accordingly. There are two different ways to build the image, the key difference is whether we
+have specified the Hive version or not.
+- Build remotely
+
+The Hive version is picked up by `--hive <hive version>`, for example:
+```shell
+sh deploy.sh --hive 3.1.3
+```
+This command will pull the Hive tar ball from Apache to local, together with Hadoop and Tez, while those two versions are defined in top `pom.xml`
+to build the image.
+
+- Build locally
+
+If the Hive version is not specified, then it will search the file: `packaging/target/apache-hive-${project.version}-bin.tar.gz` to make sure it exists, otherwise it will
+stop building.
+```shell
+sh deploy.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above example will use the local `apache-hive-${project.version}-bin.tar.gz`, Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you want to just test Metastore or play around with it, execute the following:
+```shell
+sh deploy.sh --metastore
+```
+or run with docker if Metastore image is already here.
+```shell
+docker run --name metastore-standalone hive:metastore-$HIVE_VERSION
+```
+
+- Launch a single standalone HiveServer2 for a quick start
+
+The HiveServer2 will be started with an embedded Metastore. To launch it, execute the following:
+```shell
+sh deploy.sh --hiveserver2
+```
+Or if the image for HiveServer2 has been built successfully, simply running with:
+```shell
+docker run --name hiveserver2-standalone hive:hiveserver2-$HIVE_VERSION
+```
+Please pay attention to that the data of the HiveServer2 would be lost between container restarts.
+In order to save the data, try to bring up the Hive with the following way.
+
+- Launch a cluster with HiveServer2, Metastore and MySQL as its back database.
+
+To save data between container restarts, we use Docker's volume to persist data to the local disk. Just by executing:
Review Comment:
@dengzhhu653 I didn't get this part. Where are we creating the docker volume?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1157465223
##########
packaging/src/docker/README.md:
##########
@@ -0,0 +1,131 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+
+
+### Quickstart
+
+---
+#### Build image
+Apache Hive relies on Hadoop, Tez and some others to facilitate reading, writing, and managing large datasets.
+The `build.sh` provides ways to build the image against specified version of the dependent, as well as build from source.
+
+##### Build from source
+```shell
+mvn clean package -pl packaging -DskipTests -Pdocker
+```
+##### Build with specified version
+There are some arguments to specify the component version:
+```shell
+-hadoop <hadoop version>
+-tez <tez version>
+-hive <hive version>
+```
+If the version is not provided, it will read the version from current `pom.xml`:
+`project.version`, `hadoop.version` and `tez.version` for Hive, Hadoop and Tez respectively.
+
+For example, the following command uses Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` to build the image,
+```shell
+./build.sh -hive 3.1.3
+```
+If the command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`(will trigger a build if it doesn't exist),
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the image,
+```shell
+./build.sh -hadoop 3.1.0 -tez 0.10.1
+```
+After building successfully, we can get a Docker image named `apache/hive` by default, the image is tagged by the provided Hive version.
+
+#### Run services
+
+Before going further, we should define the environment variable `HIVE_VERSION` first.
+For example, if `-hive 3.1.3` is specified to build the image,
+```shell
+export HIVE_VERSION=3.1.3
+```
+or assuming that you're relying on current `project.version` from pom.xml,
+```shell
+export HIVE_VERSION=$(mvn -f pom.xml -q help:evaluate -Dexpression=project.version -DforceStdout)
+```
+- Metastore
+
+For a quick start, launch the Metastore with Derby,
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore --name metastore-standalone apache/hive:${HIVE_VERSION}
+ ```
+ Everything would be lost when the service is down. In order to save the Hive table's schema and data, start the container with an external Postgres and Volume to keep them,
+
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore \
+ --env DB_DRIVER=postgres \
Review Comment:
@dengzhhu653 where is the driver (jar) coming from?
I think would be great to have a separate docker image for HMS as it's an independent component (Data Catalog) that could be used by external apps. Note: that could be done in another PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1490001374
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1491241293
> Seems like the build is broken. @deniskuzZ Could you please re-start?
A fork of this gets a green run: https://github.com/apache/hive/pull/4133. I think the broken build is may due to run some time consuming tests in the same split.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1495118727
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1089995000
##########
dev-support/docker/Dockerfile:
##########
@@ -0,0 +1,53 @@
+#
Review Comment:
I agree, this looks better in the packaging project
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1505067290
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1353666413
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r930543306
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,125 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions to build images.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- deploy.sh Entrance to build images and run them.
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself.
+Providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the versions are not given during build, then it will read the version info from project top `pom.xml`: project.version, hadoop.version, tez.version,
+these are assigned to hive version, hadoop version, tez version accordingly. There are two different ways to build the image, the key difference is whether we
+have specified the Hive version or not.
+- Build remotely
+
+The Hive version is picked up by `--hive <hive version>`, for example:
+```shell
+sh deploy.sh --hive 3.1.3
+```
+This command will pull the Hive tar ball from Apache to local, together with Hadoop and Tez, while those two versions are defined in top `pom.xml`
+to build the image.
+
+- Build locally
+
+If the Hive version is not specified, then it will search the file: `packaging/target/apache-hive-${project.version}-bin.tar.gz` to make sure it exists, otherwise it will
+stop building.
+```shell
+sh deploy.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above example will use the local `apache-hive-${project.version}-bin.tar.gz`, Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you want to just test Metastore or play around with it, execute the following:
+```shell
+sh deploy.sh --metastore
+```
+or run with docker if Metastore image is already here.
+```shell
+docker run --name metastore-standalone hive:metastore-$HIVE_VERSION
+```
+
+- Launch a single standalone HiveServer2 for a quick start
+
+The HiveServer2 will be started with an embedded Metastore. To launch it, execute the following:
+```shell
+sh deploy.sh --hiveserver2
+```
+Or if the image for HiveServer2 has been built successfully, simply running with:
+```shell
+docker run --name hiveserver2-standalone hive:hiveserver2-$HIVE_VERSION
+```
+Please pay attention to that the data of the HiveServer2 would be lost between container restarts.
+In order to save the data, try to bring up the Hive with the following way.
+
+- Launch a cluster with HiveServer2, Metastore and MySQL as its back database.
+
+To save data between container restarts, we use Docker's volume to persist data to the local disk. Just by executing:
Review Comment:
Hi, @achennagiri, thank you for your feedback. The volume is defined here:
https://github.com/apache/hive/blob/9cbe6d2a5081c7a72910ac337990bf3defea3101/dev-support/docker/docker-compose.yml#L17-L18
The docker will create it if not exits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r930543306
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,125 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions to build images.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- deploy.sh Entrance to build images and run them.
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself.
+Providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the versions are not given during build, then it will read the version info from project top `pom.xml`: project.version, hadoop.version, tez.version,
+these are assigned to hive version, hadoop version, tez version accordingly. There are two different ways to build the image, the key difference is whether we
+have specified the Hive version or not.
+- Build remotely
+
+The Hive version is picked up by `--hive <hive version>`, for example:
+```shell
+sh deploy.sh --hive 3.1.3
+```
+This command will pull the Hive tar ball from Apache to local, together with Hadoop and Tez, while those two versions are defined in top `pom.xml`
+to build the image.
+
+- Build locally
+
+If the Hive version is not specified, then it will search the file: `packaging/target/apache-hive-${project.version}-bin.tar.gz` to make sure it exists, otherwise it will
+stop building.
+```shell
+sh deploy.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above example will use the local `apache-hive-${project.version}-bin.tar.gz`, Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you want to just test Metastore or play around with it, execute the following:
+```shell
+sh deploy.sh --metastore
+```
+or run with docker if Metastore image is already here.
+```shell
+docker run --name metastore-standalone hive:metastore-$HIVE_VERSION
+```
+
+- Launch a single standalone HiveServer2 for a quick start
+
+The HiveServer2 will be started with an embedded Metastore. To launch it, execute the following:
+```shell
+sh deploy.sh --hiveserver2
+```
+Or if the image for HiveServer2 has been built successfully, simply running with:
+```shell
+docker run --name hiveserver2-standalone hive:hiveserver2-$HIVE_VERSION
+```
+Please pay attention to that the data of the HiveServer2 would be lost between container restarts.
+In order to save the data, try to bring up the Hive with the following way.
+
+- Launch a cluster with HiveServer2, Metastore and MySQL as its back database.
+
+To save data between container restarts, we use Docker's volume to persist data to the local disk. Just by executing:
Review Comment:
Hi, @achennagiri, thank you for your feedback. The volume is defined here:
https://github.com/apache/hive/blob/9cbe6d2a5081c7a72910ac337990bf3defea3101/dev-support/docker/docker-compose.yml#L64-L66
The docker will create it if not exits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r930545582
##########
dev-support/docker/deploy.sh:
##########
@@ -0,0 +1,156 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+usage() {
+ cat <<EOF 1>&2
+Usage: $0 [--help] [--hadoop <hadoop version>] [--tez <tez version>] [--hive <hive version>] [--repo <docker repo>]
+ [--hiveserver2 <start standalone HiveServer2>] [--metastore <start standalone Metastore>]
+Build the Hive Docker image and Run services
+--help Help
+--hadoop Hadoop version for building image
+--tez Tez version for building image
+--hive Hive version for building image
+--repo Docker repository
Review Comment:
It's used to tag and push the image to remote hub repository
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] cdmikechen commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "cdmikechen (via GitHub)" <gi...@apache.org>.
cdmikechen commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1146075254
##########
packaging/src/docker/Dockerfile:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jre
+
+ARG HADOOP_VERSION
+ARG HIVE_VERSION
+ARG TEZ_VERSION
+
+# Install dependencies
+RUN set -ex; \
+ apt-get update; \
+ apt-get -y install curl; \
+ apt-get -y install procps; \
+ rm -rf /var/lib/apt/lists/*
+
+COPY hadoop-$HADOOP_VERSION.tar.gz /opt
+COPY apache-hive-$HIVE_VERSION-bin.tar.gz /opt
+COPY apache-tez-$TEZ_VERSION-bin.tar.gz /opt
+
+RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ && \
Review Comment:
How big is your final docker image? Maybe it's better to describe it like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] TuroczyX commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "TuroczyX (via GitHub)" <gi...@apache.org>.
TuroczyX commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1485015881
I love this initiative. Can we get more eyes on it?
I have 2 comments about it:
1. HMS can work together with MySQL but to many times we found bugs with MySQL which gave us a lot of headache. Is it possible to change for Postgre?
2. I think we should ask a docker account to push the image to the repository as we have a new build or new release.
What is the remaining part of this task to make it happens?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1495231356
> > > Seems like the build is broken. @deniskuzZ Could you please re-start?
> >
> >
> > A fork of this gets a green run: #4133. I think the broken build is may due to some time consuming tests running in the same split.
>
> Do you need any help?
Thank you @TuroczyX.
The build would get a green run if I open another jira, I think the fix doesn't change any codes, nor build, so it's safe to go into master if the change itself looks fine, we can ignore the build failure in such case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1037715468
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
Review Comment:
sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] achennagiri commented on a diff in pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
achennagiri commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r930548578
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,125 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions to build images.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- deploy.sh Entrance to build images and run them.
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself.
+Providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the versions are not given during build, then it will read the version info from project top `pom.xml`: project.version, hadoop.version, tez.version,
+these are assigned to hive version, hadoop version, tez version accordingly. There are two different ways to build the image, the key difference is whether we
+have specified the Hive version or not.
+- Build remotely
+
+The Hive version is picked up by `--hive <hive version>`, for example:
+```shell
+sh deploy.sh --hive 3.1.3
+```
+This command will pull the Hive tar ball from Apache to local, together with Hadoop and Tez, while those two versions are defined in top `pom.xml`
+to build the image.
+
+- Build locally
+
+If the Hive version is not specified, then it will search the file: `packaging/target/apache-hive-${project.version}-bin.tar.gz` to make sure it exists, otherwise it will
+stop building.
+```shell
+sh deploy.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above example will use the local `apache-hive-${project.version}-bin.tar.gz`, Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you want to just test Metastore or play around with it, execute the following:
+```shell
+sh deploy.sh --metastore
+```
+or run with docker if Metastore image is already here.
+```shell
+docker run --name metastore-standalone hive:metastore-$HIVE_VERSION
+```
+
+- Launch a single standalone HiveServer2 for a quick start
+
+The HiveServer2 will be started with an embedded Metastore. To launch it, execute the following:
+```shell
+sh deploy.sh --hiveserver2
+```
+Or if the image for HiveServer2 has been built successfully, simply running with:
+```shell
+docker run --name hiveserver2-standalone hive:hiveserver2-$HIVE_VERSION
+```
+Please pay attention to that the data of the HiveServer2 would be lost between container restarts.
+In order to save the data, try to bring up the Hive with the following way.
+
+- Launch a cluster with HiveServer2, Metastore and MySQL as its back database.
+
+To save data between container restarts, we use Docker's volume to persist data to the local disk. Just by executing:
Review Comment:
Thank you! This work is really cool and helpful. Thank you for working on this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1511175822
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1146107280
##########
packaging/src/docker/Dockerfile:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jre
+
+ARG HADOOP_VERSION
+ARG HIVE_VERSION
+ARG TEZ_VERSION
+
+# Install dependencies
+RUN set -ex; \
+ apt-get update; \
+ apt-get -y install curl; \
+ apt-get -y install procps; \
+ rm -rf /var/lib/apt/lists/*
+
+COPY hadoop-$HADOOP_VERSION.tar.gz /opt
+COPY apache-hive-$HIVE_VERSION-bin.tar.gz /opt
+COPY apache-tez-$TEZ_VERSION-bin.tar.gz /opt
+
+RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ && \
Review Comment:
It's about 3gb.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] jtvmatos commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "jtvmatos (via GitHub)" <gi...@apache.org>.
jtvmatos commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1486844798
Should be included in this initiative also create an docker image for the hive metastore standalone?
Something like this: https://techjogging.com/standalone-hive-metastore-presto-docker.html
https://github.com/arempter/hive-metastore-docker/blob/master/Dockerfile
https://github.com/aws-samples/hive-emr-on-eks/blob/main/docker/Dockerfile
Thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1485078827
> I love this initiative. Can we get more eyes on it?
>
> I have 2 comments about it:
>
> 1. HMS can work together with MySQL but to many times we found bugs with MySQL which gave us a lot of headache. Is it possible to change for Postgre?
> 2. I think we should ask a docker account to push the image to the repository as we have a new build or new release.
>
> What is the remaining part of this task to make it happens?
Thank you @TuroczyX for the comments.
1. Have changed the back db to Postgres or embedded Derby;
2. This is the remaining part, want to track it in the future after this task has finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] TuroczyX commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "TuroczyX (via GitHub)" <gi...@apache.org>.
TuroczyX commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1495644501
Yes, it should be unrelated. But it would be also great if we have a green build to follow the industrial standard. (Even logically does not have relation)
Btw we need to think about as a community to push hive images into docker hub. That would be super cool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1384751312
@deniskuzZ any thoughts about the PR? Thank you in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] pvary commented on pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1193604488
Very nice work @dengzhhu653!
Could we add BeeLine to the examples, so the user could start to run queries immediately?
Will we officially provide an already built image for the Apache Hive versions? It would be good to update https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=27362090#content/view/27362090 for using these images.
Thanks, Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
abstractdog commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1387132348
I had some comments earlier, if you can confirm you addressed those @dengzhhu653 , I'll take a second look and approve, it's time to merge this I believe
please create an umbrella ticket for hive docker improvements and add this one as the first sub-jira, let's track further improvements there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1035785789
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
+```
+
+- Launch a single standalone HiveServer2 for a quick start
+
+The HiveServer2 will be started with an embedded Metastore by initiating:
+```shell
+docker run --name hiveserver2-standalone apachehive/hiveserver2:$HIVE_VERSION
+```
+The data of the HiveServer2 would be lost between container restarts.
+
+- Launch a cluster with HiveServer2, Metastore and MySQL as its back database.
+
+To save data between container restarts, Volumes is used to persist data generated by and used by Hive. Just by executing:
+```shell
+export HIVESERVER2_IMAGE=apachehive/hiveserver2:$HIVE_VERSION
+export METASTORE_IMAGE=apachehive/metastore:$HIVE_VERSION
+docker network create hive && docker-compose -f docker-compose.yml up -d
+```
+
+#### Usage
+
+---
+- Show the containers
+```shell
+docker ps
+```
+- Check HiveServer2 web ui
+ - Accessed on browser at http://localhost:10002/
+- Run Beeline:
+```shell
+docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'
+```
+- Test running some queries
+```sql
+show tables;
+create table hive_example(a string, b int) partitioned by(c int);
+alter table hive_example add partition(c=1);
+insert into hive_example partition(c=1) values('a', 1), ('a', 2),('b',3);
+select count(distinct a) from hive_example;
+set hive.execution.engine=tez;
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] TuroczyX commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "TuroczyX (via GitHub)" <gi...@apache.org>.
TuroczyX commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1494781322
> > Seems like the build is broken. @deniskuzZ Could you please re-start?
>
> A fork of this gets a green run: #4133. I think the broken build is may due to some time consuming tests running in the same split.
Do you need any help?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1157915578
##########
packaging/src/docker/README.md:
##########
@@ -0,0 +1,131 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+
+
+### Quickstart
+
+---
+#### Build image
+Apache Hive relies on Hadoop, Tez and some others to facilitate reading, writing, and managing large datasets.
+The `build.sh` provides ways to build the image against specified version of the dependent, as well as build from source.
+
+##### Build from source
+```shell
+mvn clean package -pl packaging -DskipTests -Pdocker
+```
+##### Build with specified version
+There are some arguments to specify the component version:
+```shell
+-hadoop <hadoop version>
+-tez <tez version>
+-hive <hive version>
+```
+If the version is not provided, it will read the version from current `pom.xml`:
+`project.version`, `hadoop.version` and `tez.version` for Hive, Hadoop and Tez respectively.
+
+For example, the following command uses Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` to build the image,
+```shell
+./build.sh -hive 3.1.3
+```
+If the command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`(will trigger a build if it doesn't exist),
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the image,
+```shell
+./build.sh -hadoop 3.1.0 -tez 0.10.1
+```
+After building successfully, we can get a Docker image named `apache/hive` by default, the image is tagged by the provided Hive version.
+
+#### Run services
+
+Before going further, we should define the environment variable `HIVE_VERSION` first.
+For example, if `-hive 3.1.3` is specified to build the image,
+```shell
+export HIVE_VERSION=3.1.3
+```
+or assuming that you're relying on current `project.version` from pom.xml,
+```shell
+export HIVE_VERSION=$(mvn -f pom.xml -q help:evaluate -Dexpression=project.version -DforceStdout)
+```
+- Metastore
+
+For a quick start, launch the Metastore with Derby,
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore --name metastore-standalone apache/hive:${HIVE_VERSION}
+ ```
+ Everything would be lost when the service is down. In order to save the Hive table's schema and data, start the container with an external Postgres and Volume to keep them,
+
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore \
+ --env DB_DRIVER=postgres \
Review Comment:
This jar is already here in the hive tar ball, so I skip to fetch it from external
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] TuroczyX commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "TuroczyX (via GitHub)" <gi...@apache.org>.
TuroczyX commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1486852027
> Should be included in this initiative also create an docker image for the hive metastore standalone?
>
> Something like this: https://techjogging.com/standalone-hive-metastore-presto-docker.html
>
> https://github.com/arempter/hive-metastore-docker/blob/master/Dockerfile
>
> https://github.com/aws-samples/hive-emr-on-eks/blob/main/docker/Dockerfile
>
> Thanks
It is a good point. @dengzhhu653 @deniskuzZ @ayushtkn @abstractdog What do you think about it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1146124731
##########
packaging/src/docker/Dockerfile:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jre
+
+ARG HADOOP_VERSION
+ARG HIVE_VERSION
+ARG TEZ_VERSION
+
+# Install dependencies
+RUN set -ex; \
+ apt-get update; \
+ apt-get -y install curl; \
+ apt-get -y install procps; \
+ rm -rf /var/lib/apt/lists/*
+
+COPY hadoop-$HADOOP_VERSION.tar.gz /opt
+COPY apache-hive-$HIVE_VERSION-bin.tar.gz /opt
+COPY apache-tez-$TEZ_VERSION-bin.tar.gz /opt
+
+RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ && \
Review Comment:
you don't need the archives in the image, you can just copy the final files
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1146124731
##########
packaging/src/docker/Dockerfile:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jre
+
+ARG HADOOP_VERSION
+ARG HIVE_VERSION
+ARG TEZ_VERSION
+
+# Install dependencies
+RUN set -ex; \
+ apt-get update; \
+ apt-get -y install curl; \
+ apt-get -y install procps; \
+ rm -rf /var/lib/apt/lists/*
+
+COPY hadoop-$HADOOP_VERSION.tar.gz /opt
+COPY apache-hive-$HIVE_VERSION-bin.tar.gz /opt
+COPY apache-tez-$TEZ_VERSION-bin.tar.gz /opt
+
+RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ && \
Review Comment:
you don't need the archives in the image:
````
find . -name *.tar.gz | xargs rm
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1147134656
##########
packaging/src/docker/Dockerfile:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jre
+
+ARG HADOOP_VERSION
+ARG HIVE_VERSION
+ARG TEZ_VERSION
+
+# Install dependencies
+RUN set -ex; \
+ apt-get update; \
+ apt-get -y install curl; \
+ apt-get -y install procps; \
+ rm -rf /var/lib/apt/lists/*
+
+COPY hadoop-$HADOOP_VERSION.tar.gz /opt
+COPY apache-hive-$HIVE_VERSION-bin.tar.gz /opt
+COPY apache-tez-$TEZ_VERSION-bin.tar.gz /opt
+
+RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ && \
Review Comment:
The archives are removed from the image, now the image size is about 1.4G
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1347985020
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1094458438
##########
dev-support/docker/Dockerfile:
##########
@@ -0,0 +1,53 @@
+#
Review Comment:
Done, and added a maven command to build from source, cc @ayushtkn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
zabetak commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1034624562
##########
dev-support/docker/docker-compose.yml:
##########
@@ -0,0 +1,71 @@
+version: '3.9'
+services:
Review Comment:
We could have HS2, Metastore, DBMS in a single image; wouldn't this be much easier to use? It would also drop the need for `docker-compose`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
zabetak commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1034626727
##########
dev-support/docker/docker-compose.yml:
##########
@@ -0,0 +1,71 @@
+version: '3.9'
+services:
Review Comment:
OK I've seen that there is an intention to also launch HMS as standalone so ignore my previous comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zabetak commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
zabetak commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1034613441
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
Review Comment:
Note that there is already an apache namespace in `dockerhub` (e.g., https://hub.docker.com/r/apache/flink) for hosting ASF projects so if we aim to provide official images we should create something like apache/hive there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] cdmikechen commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "cdmikechen (via GitHub)" <gi...@apache.org>.
cdmikechen commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1136361572
##########
packaging/src/docker/Dockerfile:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jre
+
+ARG HADOOP_VERSION
+ARG HIVE_VERSION
+ARG TEZ_VERSION
+
+# Install dependencies
+RUN set -ex; \
+ apt-get update; \
+ apt-get -y install curl; \
+ apt-get -y install procps; \
+ rm -rf /var/lib/apt/lists/*
+
+COPY hadoop-$HADOOP_VERSION.tar.gz /opt
+COPY apache-hive-$HIVE_VERSION-bin.tar.gz /opt
+COPY apache-tez-$TEZ_VERSION-bin.tar.gz /opt
+
+RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ && \
Review Comment:
If we just copy and decompress these files, I think the docker image will be too large.
My suggestion is whether we can do a multi-phase build and then copy the final files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
abstractdog commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1265068519
why is this closed? I cannot see the patch on master
I would be sad to see this forgotten, how can we proceed with this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] achennagiri commented on a diff in pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
achennagiri commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r930531218
##########
dev-support/docker/deploy.sh:
##########
@@ -0,0 +1,156 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+usage() {
+ cat <<EOF 1>&2
+Usage: $0 [--help] [--hadoop <hadoop version>] [--tez <tez version>] [--hive <hive version>] [--repo <docker repo>]
+ [--hiveserver2 <start standalone HiveServer2>] [--metastore <start standalone Metastore>]
+Build the Hive Docker image and Run services
+--help Help
+--hadoop Hadoop version for building image
+--tez Tez version for building image
+--hive Hive version for building image
+--repo Docker repository
Review Comment:
What does repo signify here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1301850083
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] TuroczyX commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "TuroczyX (via GitHub)" <gi...@apache.org>.
TuroczyX commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1511410250
Excellent! Thank you all of your effort @dengzhhu653!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ closed pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ closed pull request #3448: HIVE-26400: Provide docker images for Hive
URL: https://github.com/apache/hive/pull/3448
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] TuroczyX commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "TuroczyX (via GitHub)" <gi...@apache.org>.
TuroczyX commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1488941527
Seems like the build is broken. @deniskuzZ Could you please re-start?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1193793273
> Very nice work @dengzhhu653!
>
> Could we add BeeLine to the examples, so the user could start to run queries immediately?
>
> Will we officially provide an already built image for the Apache Hive versions? It would be good to update https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=27362090#content/view/27362090 for using these images.
>
> Thanks, Peter
Seems we should create the remote offical repository [first](https://issues.apache.org/jira/projects/INFRA/issues/INFRA-22097), any suggestions on the repository name?
such as apache/hive-hiveserver2, apache/hive-metastore, or just apache/hive? also cc @nrg4878, @abstractdog, @ayushtkn
Thanks,
Zhihua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1191134925
> Is there any scope to run it with local version of Hive/Hadoop/Tez or do we need a released version always for this?
The quick answer is yes, but there are some places to modify in order to run specified version:
- change version in docker-compose.yml
https://github.com/apache/hive/blob/213208570d1efa0f7a41d5a742edd0439b99163b/dev-support/docker/docker-compose.yml#L51-L52
- change the download url in Dockerfile
https://github.com/apache/hive/blob/213208570d1efa0f7a41d5a742edd0439b99163b/dev-support/docker/Dockerfile#L37-L43
I'm wondering if we can build hive from source directly, but need some feedback and investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] achennagiri commented on a diff in pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
achennagiri commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r930532023
##########
dev-support/docker/deploy.sh:
##########
@@ -0,0 +1,156 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+usage() {
+ cat <<EOF 1>&2
+Usage: $0 [--help] [--hadoop <hadoop version>] [--tez <tez version>] [--hive <hive version>] [--repo <docker repo>]
+ [--hiveserver2 <start standalone HiveServer2>] [--metastore <start standalone Metastore>]
+Build the Hive Docker image and Run services
+--help Help
+--hadoop Hadoop version for building image
+--tez Tez version for building image
+--hive Hive version for building image
+--repo Docker repository
+--hiveserver2 Start HiveServer2 only, with embedded Metastore
+--metastore Start Metastore only, with embedded derby
+EOF
+}
+
+# components for building image
+MYSQL_VERSION=8.0.27
Review Comment:
As I understand, we can create hive docker images currently with derby and Mysql as options for backend DB right? We could probably extend this to the other flavors like Oracle, MSSQL, Postgres in the future right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1035786020
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
+```
+
+- Launch a single standalone HiveServer2 for a quick start
+
+The HiveServer2 will be started with an embedded Metastore by initiating:
+```shell
+docker run --name hiveserver2-standalone apachehive/hiveserver2:$HIVE_VERSION
+```
+The data of the HiveServer2 would be lost between container restarts.
+
+- Launch a cluster with HiveServer2, Metastore and MySQL as its back database.
+
+To save data between container restarts, Volumes is used to persist data generated by and used by Hive. Just by executing:
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1036995252
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
Review Comment:
we should also consider using a single hive image:
```
apache/hive
```
and according to an docker parameter, it can start as hs2 or metastore (driven from docker-compose)
this might be important and convenient, as a single image is easier to maintain, and looks much better than having apache/hive-hiveserver2 and apache/hive-metastore images later when we really distritbute the image to public repo
I know it changes the current implementation here, would you consider trying it @dengzhhu653 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1142999778
##########
packaging/src/docker/Dockerfile:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jre
+
+ARG HADOOP_VERSION
+ARG HIVE_VERSION
+ARG TEZ_VERSION
+
+# Install dependencies
+RUN set -ex; \
+ apt-get update; \
+ apt-get -y install curl; \
+ apt-get -y install procps; \
+ rm -rf /var/lib/apt/lists/*
+
+COPY hadoop-$HADOOP_VERSION.tar.gz /opt
+COPY apache-hive-$HIVE_VERSION-bin.tar.gz /opt
+COPY apache-tez-$TEZ_VERSION-bin.tar.gz /opt
+
+RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ && \
Review Comment:
Not fully understand, but will leave it open for further improvements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1146124731
##########
packaging/src/docker/Dockerfile:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jre
+
+ARG HADOOP_VERSION
+ARG HIVE_VERSION
+ARG TEZ_VERSION
+
+# Install dependencies
+RUN set -ex; \
+ apt-get update; \
+ apt-get -y install curl; \
+ apt-get -y install procps; \
+ rm -rf /var/lib/apt/lists/*
+
+COPY hadoop-$HADOOP_VERSION.tar.gz /opt
+COPY apache-hive-$HIVE_VERSION-bin.tar.gz /opt
+COPY apache-tez-$TEZ_VERSION-bin.tar.gz /opt
+
+RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ && \
Review Comment:
you don't need the archives in the image
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1482310775
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1486586664
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1157465223
##########
packaging/src/docker/README.md:
##########
@@ -0,0 +1,131 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+
+
+### Quickstart
+
+---
+#### Build image
+Apache Hive relies on Hadoop, Tez and some others to facilitate reading, writing, and managing large datasets.
+The `build.sh` provides ways to build the image against specified version of the dependent, as well as build from source.
+
+##### Build from source
+```shell
+mvn clean package -pl packaging -DskipTests -Pdocker
+```
+##### Build with specified version
+There are some arguments to specify the component version:
+```shell
+-hadoop <hadoop version>
+-tez <tez version>
+-hive <hive version>
+```
+If the version is not provided, it will read the version from current `pom.xml`:
+`project.version`, `hadoop.version` and `tez.version` for Hive, Hadoop and Tez respectively.
+
+For example, the following command uses Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` to build the image,
+```shell
+./build.sh -hive 3.1.3
+```
+If the command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`(will trigger a build if it doesn't exist),
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the image,
+```shell
+./build.sh -hadoop 3.1.0 -tez 0.10.1
+```
+After building successfully, we can get a Docker image named `apache/hive` by default, the image is tagged by the provided Hive version.
+
+#### Run services
+
+Before going further, we should define the environment variable `HIVE_VERSION` first.
+For example, if `-hive 3.1.3` is specified to build the image,
+```shell
+export HIVE_VERSION=3.1.3
+```
+or assuming that you're relying on current `project.version` from pom.xml,
+```shell
+export HIVE_VERSION=$(mvn -f pom.xml -q help:evaluate -Dexpression=project.version -DforceStdout)
+```
+- Metastore
+
+For a quick start, launch the Metastore with Derby,
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore --name metastore-standalone apache/hive:${HIVE_VERSION}
+ ```
+ Everything would be lost when the service is down. In order to save the Hive table's schema and data, start the container with an external Postgres and Volume to keep them,
+
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore \
+ --env DB_DRIVER=postgres \
Review Comment:
where is the driver coming from?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1192186831
The new changes add support for running with local version of Hive:
```
sh deploy.sh --hadoop <hadoop version> --tez <tez version>
```
this command will build the image with given Hadoop and Tez version, and the local packaging/target/apache-hive-${project.version}-bin.tar.gz that built from source, a cluster with HiveServer2, Metastore and MySQL would be started.
We can also build the image with a specified Hive version, just append `--hive <hive version>` to the above command.
By default, the command will read the version info from project `pom.xml`: `project.version`, `hadoop.version`, `tez.version`, these properties are read as hive version, hadoop.version, tez.version and used for `deploy.sh` to build the image.
Besides, we can start a standalone HiveServer2 only with embedded Metastore,
```
sh deploy.sh --hiveserver2
```
or just start a standalone Metastore with derby,
```
sh deploy.sh --metastore
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1193455935
@kgyrtkirk @abstractdog @pvary any ideas or suggestions? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
abstractdog commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1193886325
> is there a chance that we can build this docker image + run simple queries in precommit time? otherwise, I'm afraid we cannot guarantee the stability of this feature
I've just realized that the docker image isn't built from the latest master, but instead from an already released hive version, so after thinking this over again, I guess we don't need a docker image build + test in precommit time (my original idea was to automatically check for every single hive commit if it breaks the image...like a smoke test)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide a self-contained docker
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r930546009
##########
dev-support/docker/deploy.sh:
##########
@@ -0,0 +1,156 @@
+#!/bin/bash
+
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+usage() {
+ cat <<EOF 1>&2
+Usage: $0 [--help] [--hadoop <hadoop version>] [--tez <tez version>] [--hive <hive version>] [--repo <docker repo>]
+ [--hiveserver2 <start standalone HiveServer2>] [--metastore <start standalone Metastore>]
+Build the Hive Docker image and Run services
+--help Help
+--hadoop Hadoop version for building image
+--tez Tez version for building image
+--hive Hive version for building image
+--repo Docker repository
+--hiveserver2 Start HiveServer2 only, with embedded Metastore
+--metastore Start Metastore only, with embedded derby
+EOF
+}
+
+# components for building image
+MYSQL_VERSION=8.0.27
Review Comment:
yeah, that's right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
abstractdog commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1033497364
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
+```
+
+- Launch a single standalone HiveServer2 for a quick start
+
+The HiveServer2 will be started with an embedded Metastore by initiating:
+```shell
+docker run --name hiveserver2-standalone apachehive/hiveserver2:$HIVE_VERSION
+```
+The data of the HiveServer2 would be lost between container restarts.
+
+- Launch a cluster with HiveServer2, Metastore and MySQL as its back database.
+
+To save data between container restarts, Volumes is used to persist data generated by and used by Hive. Just by executing:
+```shell
+export HIVESERVER2_IMAGE=apachehive/hiveserver2:$HIVE_VERSION
+export METASTORE_IMAGE=apachehive/metastore:$HIVE_VERSION
+docker network create hive && docker-compose -f docker-compose.yml up -d
+```
+
+#### Usage
+
+---
+- Show the containers
+```shell
+docker ps
+```
+- Check HiveServer2 web ui
+ - Accessed on browser at http://localhost:10002/
+- Run Beeline:
+```shell
+docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'
+```
+- Test running some queries
+```sql
+show tables;
+create table hive_example(a string, b int) partitioned by(c int);
+alter table hive_example add partition(c=1);
+insert into hive_example partition(c=1) values('a', 1), ('a', 2),('b',3);
+select count(distinct a) from hive_example;
+set hive.execution.engine=tez;
Review Comment:
please remove this clause: we're building this image to run on tez local mode, so let's not give the users the opportunity to think "whether it can run on MR or Spark if I configure so?"
also, the data loading commands should already run on tez: we should not rely on MR in any way (I tried it out, worked with tez out of the box, so I guess you should just have to add hive.execution.engine to the hiveserver2-site.xml
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
+```
+
+- Launch a single standalone HiveServer2 for a quick start
+
+The HiveServer2 will be started with an embedded Metastore by initiating:
+```shell
+docker run --name hiveserver2-standalone apachehive/hiveserver2:$HIVE_VERSION
+```
+The data of the HiveServer2 would be lost between container restarts.
+
+- Launch a cluster with HiveServer2, Metastore and MySQL as its back database.
+
+To save data between container restarts, Volumes is used to persist data generated by and used by Hive. Just by executing:
Review Comment:
nit: "volumes are"
also, this line emphasizes used volumes, if it refers to hive-db and warehouse, please mention them specifically
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
+```
+
+- Launch a single standalone HiveServer2 for a quick start
+
+The HiveServer2 will be started with an embedded Metastore by initiating:
+```shell
+docker run --name hiveserver2-standalone apachehive/hiveserver2:$HIVE_VERSION
+```
+The data of the HiveServer2 would be lost between container restarts.
+
+- Launch a cluster with HiveServer2, Metastore and MySQL as its back database.
+
+To save data between container restarts, Volumes is used to persist data generated by and used by Hive. Just by executing:
+```shell
+export HIVESERVER2_IMAGE=apachehive/hiveserver2:$HIVE_VERSION
+export METASTORE_IMAGE=apachehive/metastore:$HIVE_VERSION
+docker network create hive && docker-compose -f docker-compose.yml up -d
+```
+
+#### Usage
+
+---
+- Show the containers
+```shell
+docker ps
+```
+- Check HiveServer2 web ui
+ - Accessed on browser at http://localhost:10002/
+- Run Beeline:
+```shell
+docker exec -it hiveserver2 beeline -u 'jdbc:hive2://hiveserver2:10000/'
Review Comment:
please mention that if we have beeline installed on host machine, hs2 can be simply reached as:
```
beeline -u 'jdbc:hive2://localhost:10000/'
```
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
Review Comment:
1. what is "apachehive" here? do we have a docker hub account?
at the moment I guess it's just a common namespace (to tag images), please clarify this in a comment somewhere:
```
docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
```
above command makes me think we already have the images pushed somewhere, but instead, this command just relies on the fact that we built and tagged the images by build.sh to have this namespace, so it's a prerequisite
2. in build.sh, we made the $HIVE_VERSION be automagically resolved...so a README cannot contain usage that refers to this, instead, let's mention a way to resolve it, etc.:
```
#assuming that you're relying on current hive.version from pom.xml
export HIVE_VERSION=$(...example command to resolve hive.version from pom.xml, it should work from hive root folder and dev-support/docker too...)
...rest of the docker run commands...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] nrg4878 commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
nrg4878 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1313796744
@zabetak @abstractdog what is to be done for this here. This seems like very useful and easier way to run hive services out of the box than dev-box setup though less powerful.
the dev-box setup is not part of hive codebase as I understand it. What is it we want to complete before this can be merged? Tahnk you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by GitBox <gi...@apache.org>.
dengzhhu653 commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1035787603
##########
dev-support/docker/README.md:
##########
@@ -0,0 +1,93 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, with MySQL as its back database.
+Provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+- Images can be used as the basis for the Kubernetes operator
+
+### Overview
+
+---
+#### Files
+- docker-compose.yml: Docker compose file
+- Dockerfile-*, scripts/docker-entrypoint.sh: Instructions of the image.
+- conf/hiveserver2-site.xml: Configuration for HiveServer2
+- conf/metastore-site.xml: Configuration for Hive Metastore
+- build.sh Scripts for build images
+
+### Quickstart
+
+---
+#### Build images
+Hive relies on Hadoop, Tez and MySQL to work correctly. Up to now, there are so many versions that these dependents have been released, including Hive itself,
+providing a way to build Hive against a specified version of the dependent sounds reasonable. There are some build args for this purpose, as listed below:
+```shell
+--hadoop <hadoop version>
+--tez <tez version>
+--hive <hive version>
+```
+If the version is not provided, then it will read the version from the properties in project top `pom.xml`,
+that is, `project.version` for Hive, `hadoop.version` for Hadoop and `tez.version` for Tez. For example:
+
+```shell
+./build.sh --hive 3.1.3
+```
+The command will pull the tarballs of Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` from apache repository
+to build the target image.
+
+```shell
+./build.sh --hadoop 3.1.0 --tez 0.10.1
+```
+The above command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`,
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the target image.
+
+#### Run services
+
+- Launch a single standalone Metastore
+
+If you just want to play around with Metastore, run:
+```shell
+docker run --name metastore-standalone apachehive/metastore:$HIVE_VERSION
Review Comment:
ack, agreed if we can put the images to the apache namespace in `dockerhub`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1088324666
##########
dev-support/docker/Dockerfile:
##########
@@ -0,0 +1,53 @@
+#
Review Comment:
i think, it should be under `packaging/src/docker`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1415180462
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=3448)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=3448&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL) [1 Code Smell](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=3448&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=3448&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1163861025
##########
packaging/src/docker/README.md:
##########
@@ -0,0 +1,131 @@
+### Introduction
+
+---
+Run Apache Hive inside docker container in pseudo-distributed mode, provide the following
+- Quick-start/Debugging/Prepare a test env for Hive
+
+
+### Quickstart
+
+---
+#### Build image
+Apache Hive relies on Hadoop, Tez and some others to facilitate reading, writing, and managing large datasets.
+The `build.sh` provides ways to build the image against specified version of the dependent, as well as build from source.
+
+##### Build from source
+```shell
+mvn clean package -pl packaging -DskipTests -Pdocker
+```
+##### Build with specified version
+There are some arguments to specify the component version:
+```shell
+-hadoop <hadoop version>
+-tez <tez version>
+-hive <hive version>
+```
+If the version is not provided, it will read the version from current `pom.xml`:
+`project.version`, `hadoop.version` and `tez.version` for Hive, Hadoop and Tez respectively.
+
+For example, the following command uses Hive 3.1.3, Hadoop `hadoop.version` and Tez `tez.version` to build the image,
+```shell
+./build.sh -hive 3.1.3
+```
+If the command does not specify the Hive version, it will use the local `apache-hive-${project.version}-bin.tar.gz`(will trigger a build if it doesn't exist),
+together with Hadoop 3.1.0 and Tez 0.10.1 to build the image,
+```shell
+./build.sh -hadoop 3.1.0 -tez 0.10.1
+```
+After building successfully, we can get a Docker image named `apache/hive` by default, the image is tagged by the provided Hive version.
+
+#### Run services
+
+Before going further, we should define the environment variable `HIVE_VERSION` first.
+For example, if `-hive 3.1.3` is specified to build the image,
+```shell
+export HIVE_VERSION=3.1.3
+```
+or assuming that you're relying on current `project.version` from pom.xml,
+```shell
+export HIVE_VERSION=$(mvn -f pom.xml -q help:evaluate -Dexpression=project.version -DforceStdout)
+```
+- Metastore
+
+For a quick start, launch the Metastore with Derby,
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore --name metastore-standalone apache/hive:${HIVE_VERSION}
+ ```
+ Everything would be lost when the service is down. In order to save the Hive table's schema and data, start the container with an external Postgres and Volume to keep them,
+
+ ```shell
+ docker run -d -p 9083:9083 --env SERVICE_NAME=metastore \
+ --env DB_DRIVER=postgres \
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] cdmikechen commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "cdmikechen (via GitHub)" <gi...@apache.org>.
cdmikechen commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1136363369
##########
packaging/src/docker/Dockerfile:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jre
+
+ARG HADOOP_VERSION
+ARG HIVE_VERSION
+ARG TEZ_VERSION
+
+# Install dependencies
+RUN set -ex; \
+ apt-get update; \
+ apt-get -y install curl; \
+ apt-get -y install procps; \
+ rm -rf /var/lib/apt/lists/*
+
+COPY hadoop-$HADOOP_VERSION.tar.gz /opt
+COPY apache-hive-$HIVE_VERSION-bin.tar.gz /opt
+COPY apache-tez-$TEZ_VERSION-bin.tar.gz /opt
+
+RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ && \
Review Comment:
I think we can refer to mr3-hive project:
https://github.com/mr3project/mr3-run-k8s/blob/master/kubernetes/hive/Dockerfile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] TuroczyX commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "TuroczyX (via GitHub)" <gi...@apache.org>.
TuroczyX commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1486475328
>
Excellent! We have started a conversation today how we can regularly publish the hive into the docker hub. It would be cool if the daily build or the release version could be playable easily.
Can't wait to play with it :)
-Attila
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] cdmikechen commented on a diff in pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "cdmikechen (via GitHub)" <gi...@apache.org>.
cdmikechen commented on code in PR #3448:
URL: https://github.com/apache/hive/pull/3448#discussion_r1147042933
##########
packaging/src/docker/Dockerfile:
##########
@@ -0,0 +1,54 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+FROM openjdk:8-jre
+
+ARG HADOOP_VERSION
+ARG HIVE_VERSION
+ARG TEZ_VERSION
+
+# Install dependencies
+RUN set -ex; \
+ apt-get update; \
+ apt-get -y install curl; \
+ apt-get -y install procps; \
+ rm -rf /var/lib/apt/lists/*
+
+COPY hadoop-$HADOOP_VERSION.tar.gz /opt
+COPY apache-hive-$HIVE_VERSION-bin.tar.gz /opt
+COPY apache-tez-$TEZ_VERSION-bin.tar.gz /opt
+
+RUN tar -xzvf /opt/hadoop-$HADOOP_VERSION.tar.gz -C /opt/ && \
Review Comment:
This image is too big, a good java image should be limited to 500M-1G, too big an image will affect the service starting on different nodes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] dengzhhu653 commented on pull request #3448: HIVE-26400: Provide docker images for Hive
Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.
dengzhhu653 commented on PR #3448:
URL: https://github.com/apache/hive/pull/3448#issuecomment-1511357356
> I think it would be a good idea to add ports for development / debugging like this ones: -p9866:9866 -p10000:10000 -p10001:10001 -p9000:9000 -p8000:8000 -p3306:3306 -p50070:50070 -p50030:50030
For debug purpose, you can launch the container with:
```shell
docker run -d -p 9083:9083 -p 8009:8009 --env SERVICE_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8009" --env SERVICE_NAME=metastore --name metastore-standalone apache/hive:4.0.0-SNAPSHOT
```
SERVICE_OPTS will finally propagate to the JVM args of the service.
I opened a new PR for testing it out, https://github.com/apache/hive/pull/4240 cc @TuroczyX @deniskuzZ.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org