You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "zabetak (via GitHub)" <gi...@apache.org> on 2024/07/02 17:11:19 UTC
[PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
zabetak opened a new pull request, #5331:
URL: https://github.com/apache/hive/pull/5331
### What changes were proposed in this pull request?
1. Dockerfile for building the apache/hive-ci-jenkins image (to be used by http://ci.hive.apache.org/)
2. GitHub action for building and deploying the image to production on demand
3. Instructions/Suggestions for building, testing, and upgrading the Jenkins instance in CI.
### Why are the changes needed?
1. Upgrade the Jenkins CI instance to the latest version
2. Host and maintain the Jenkins CI images under the apache/hive namespace
3. Provide guidelines/reference points for future updates
### Does this PR introduce _any_ user-facing change?
No
### Is the change a dependency upgrade?
No
### How was this patch tested?
Followed roughly the steps described in the new `README.md` file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #5331:
URL: https://github.com/apache/hive/pull/5331#issuecomment-2362473026
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #5331:
URL: https://github.com/apache/hive/pull/5331#discussion_r1680929160
##########
ci/jenkins/README.md:
##########
@@ -0,0 +1,92 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+# Hive CI Jenkins
+
+This directory contains all the necessary resources to build the Jenkins image that is used in http://ci.hive.apache.org/
+and is hosted under https://hub.docker.com/r/apache/hive-ci-jenkins/.
+
+The Jenkins container runs in a Kubernetes cluster powered by Google Cloud.
+
+## Build
+
+### Local
+To build a new image with the latest Jenkins version and the respective plugins run.
+```
+docker build . -t apache/hive-ci-jenkins:lts-jdk21
+```
+The Dockerfile does not contain fixed versions for Jenkins and the plugins used which means that on every build
+everything will be upgraded based on the latest available.
+
+### CI
+
+There is a GitHub action that can be used to build and publish the latest Jenkins image on https://hub.docker.com/r/apache/hive-ci-jenkins/.
+
+## Run & test
+To start the container and test that everything is working run.
+```
+docker run -p 35000:8080 apache/hive-ci-jenkins:lts-jdk21
+```
+The Jenkins UI can be accessed via http://0.0.0.0:35000/.
+
+## Upgrade guide
+
+The bulk of stateful content that is maintained by Jenkins is located under the "/var/jenkins_home" directory and the
+latter is mounted to a persistent volume (see `kubectl describe pod/JENKINS_POD`).
+
+For the upgrade, we need to ensure that the current state and configurations in the persistent volume are not in conflict
+with the new Jenkins version before pushing the new image to production.
+
+1. Obtain a backup of the jenkins_home directory from the persistent volume.
Review Comment:
You can test various aspects but indeed launching jobs or connecting to the K8 nodes will not work.
The process outlined above indeed makes sense but wasn't very comfortable doing that in the actual cluster cause I was afraid of potential side-effects. If a second fully-working Jenkins instance comes up running it may interfere with the main one.
Anyways given that very few people have access to the K8 cluster it may be better to remove these instructions altogether for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5331:
URL: https://github.com/apache/hive/pull/5331#issuecomment-2232928420
I can see that Zoltan's repo contains an entrypoint script: https://github.com/kgyrtkirk/hive-test-kube/blob/master/htk-jenkins/entrypoint
isn't that missing from here? I guess we want to track it together with the Dockerfile
thanks for this effort so far, I totally agree with this initiative to include hive's ci infra code to the main repo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #5331:
URL: https://github.com/apache/hive/pull/5331#issuecomment-2232961550
@abstractdog The https://github.com/kgyrtkirk/hive-test-kube/blob/master/htk-jenkins/entrypoint does not appear necessary.
The JVM options are now passed using `ENV JENKINS_JAVA_OPTS` and the default configuration in [jenkins_home](https://github.com/kgyrtkirk/hive-test-kube/tree/master/htk-jenkins/jenkins_home) is anyways overridden by the content of the persistent volume that is attached to the pod.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #5331:
URL: https://github.com/apache/hive/pull/5331#issuecomment-2233124973
I agree with Zoltan that a separate repo would be beneficial for tracking CI and product changes separately. However, I am willing to sacrifice this separation for facilitating the discovery and contributions to the CI part.
At the same time, I feel that the testability aspect for Jenkins changes is independent of where the code is hosted. Even if this code is hosted in another repo we would still need a way to prove that changes in the Jenkins image do not break production. This is definitely an area with lots of room for improvements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5331:
URL: https://github.com/apache/hive/pull/5331#issuecomment-2233323900
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5331) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5331&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5331&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5331&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5331&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5331&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5331)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on code in PR #5331:
URL: https://github.com/apache/hive/pull/5331#discussion_r1680809620
##########
ci/jenkins/README.md:
##########
@@ -0,0 +1,92 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+# Hive CI Jenkins
+
+This directory contains all the necessary resources to build the Jenkins image that is used in http://ci.hive.apache.org/
+and is hosted under https://hub.docker.com/r/apache/hive-ci-jenkins/.
+
+The Jenkins container runs in a Kubernetes cluster powered by Google Cloud.
+
+## Build
+
+### Local
+To build a new image with the latest Jenkins version and the respective plugins run.
+```
+docker build . -t apache/hive-ci-jenkins:lts-jdk21
+```
+The Dockerfile does not contain fixed versions for Jenkins and the plugins used which means that on every build
+everything will be upgraded based on the latest available.
+
+### CI
+
+There is a GitHub action that can be used to build and publish the latest Jenkins image on https://hub.docker.com/r/apache/hive-ci-jenkins/.
+
+## Run & test
+To start the container and test that everything is working run.
+```
+docker run -p 35000:8080 apache/hive-ci-jenkins:lts-jdk21
+```
+The Jenkins UI can be accessed via http://0.0.0.0:35000/.
+
+## Upgrade guide
+
+The bulk of stateful content that is maintained by Jenkins is located under the "/var/jenkins_home" directory and the
+latter is mounted to a persistent volume (see `kubectl describe pod/JENKINS_POD`).
+
+For the upgrade, we need to ensure that the current state and configurations in the persistent volume are not in conflict
+with the new Jenkins version before pushing the new image to production.
+
+1. Obtain a backup of the jenkins_home directory from the persistent volume.
Review Comment:
these instructions work outside k8s; as it is I guess you can't launch a single job during the validation...
the right process should be:
* copy k8s volume
* create a new pod with the copied data
* use that to check things out
instead of launching with `docker`: I think you should document that (or remove these instructions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #5331:
URL: https://github.com/apache/hive/pull/5331#issuecomment-2208716616
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5331) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5331&resolved=false&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5331&resolutions=WONTFIX)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5331&resolved=false&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5331&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5331&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5331)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #5331:
URL: https://github.com/apache/hive/pull/5331#discussion_r1680942133
##########
ci/jenkins/README.md:
##########
@@ -0,0 +1,92 @@
+<!--
+{% comment %}
+Licensed to the Apache Software Foundation (ASF) under one or more
+contributor license agreements. See the NOTICE file distributed with
+this work for additional information regarding copyright ownership.
+The ASF licenses this file to you under the Apache License, Version 2.0
+(the "License"); you may not use this file except in compliance with
+the License. You may obtain a copy of the License at
+
+http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+{% endcomment %}
+-->
+# Hive CI Jenkins
+
+This directory contains all the necessary resources to build the Jenkins image that is used in http://ci.hive.apache.org/
+and is hosted under https://hub.docker.com/r/apache/hive-ci-jenkins/.
+
+The Jenkins container runs in a Kubernetes cluster powered by Google Cloud.
+
+## Build
+
+### Local
+To build a new image with the latest Jenkins version and the respective plugins run.
+```
+docker build . -t apache/hive-ci-jenkins:lts-jdk21
+```
+The Dockerfile does not contain fixed versions for Jenkins and the plugins used which means that on every build
+everything will be upgraded based on the latest available.
+
+### CI
+
+There is a GitHub action that can be used to build and publish the latest Jenkins image on https://hub.docker.com/r/apache/hive-ci-jenkins/.
+
+## Run & test
+To start the container and test that everything is working run.
+```
+docker run -p 35000:8080 apache/hive-ci-jenkins:lts-jdk21
+```
+The Jenkins UI can be accessed via http://0.0.0.0:35000/.
+
+## Upgrade guide
+
+The bulk of stateful content that is maintained by Jenkins is located under the "/var/jenkins_home" directory and the
+latter is mounted to a persistent volume (see `kubectl describe pod/JENKINS_POD`).
+
+For the upgrade, we need to ensure that the current state and configurations in the persistent volume are not in conflict
+with the new Jenkins version before pushing the new image to production.
+
+1. Obtain a backup of the jenkins_home directory from the persistent volume.
Review Comment:
Removed upgrade instructions in https://github.com/apache/hive/pull/5331/commits/9870120be7b26c9af44f65ba4f134d2ec8ebbf20
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5331:
URL: https://github.com/apache/hive/pull/5331#issuecomment-2233034293
> yes - it was a surprise to me as well that k8s doesn't support `VOLUME`-s; the default config copy was a courtesy to make it usable right away if it gets deployed at a new location
>
> I think putting this into the main repo is not the best - as you will need to deploy it and push the image to see what happens...so you will at most will be committing to the repo as an after-tought....keeping a separate un-confined repo will make it easier to track what and when was changed - but I might be alone with this view...
I feel that keeping this image in the main repo is the way to go
how to manage the challenges that come with this approach is a different question, regarding which I have no idea now, honestly :)
this approach is not strictly "after-tought", only if it's proven to be impossible to test the Dockerfile changes in advance, I hope it's not the case, thinking about a github action to create and push an image with a custom tag from a PR like this (automatically or driven by a label)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "kgyrtkirk (via GitHub)" <gi...@apache.org>.
kgyrtkirk commented on PR #5331:
URL: https://github.com/apache/hive/pull/5331#issuecomment-2232969152
yes - it was a surprise to me as well that k8s doesn't support `VOLUME`-s; the default config copy was a courtesy to make it usable right away if it gets deployed at a new location
I think putting this into the main repo is not the best - as you will need to deploy it and push the image to see what happens...so you will at most will be committing to the repo as an after-tought....keeping a separate un-confined repo will make it easier to track what and when was changed - but I might be alone with this view...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #5331:
URL: https://github.com/apache/hive/pull/5331#issuecomment-2238848407
@abstractdog @kgyrtkirk Is there anything else apart from the repo debate holding off this PR? I want to get a feeling what is pending to finalize this work to see how to advance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #5331: HIVE-28339: Dockerfile and GitHub action for hive-ci-jenkins image
URL: https://github.com/apache/hive/pull/5331
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org