You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "pan3793 (via GitHub)" <gi...@apache.org> on 2026/01/04 06:50:19 UTC
[PR] Upgrade components of play ground to recent versions [kyuubi]
pan3793 opened a new pull request, #7295:
URL: https://github.com/apache/kyuubi/pull/7295
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/master/contributing/code/index.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug, and what versions are affected.
-->
Upgrade components of play ground to recent versions, in addition, Kyuubi and Spark switch to use JDK 17, while other components like Hadoop and Hive, remamin using JDK 8.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Tested locally by building images and run demo, the updated docker images will be available on DockerHub soon, reviewer can test it too.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If a generative AI tooling has been used in the process of authoring this patch, please include
phrase 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #7295: Upgrade components of play ground to recent versions
URL: https://github.com/apache/kyuubi/pull/7295
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#discussion_r2660636569
##########
docker/playground/image/kyuubi-playground-base.Dockerfile:
##########
@@ -10,13 +10,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM eclipse-temurin:8-focal
+FROM ubuntu:focal
Review Comment:
FYI: Filed https://issues.apache.org/jira/browse/HADOOP-19774 to use Ubuntu 24.04 in Hadoop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#discussion_r2659455560
##########
docker/playground/README.md:
##########
@@ -10,24 +10,35 @@ Playground
### Play
-1. Connect using `beeline`
+1. Connect using `kyuubi-beeline`
-`docker exec -it kyuubi /opt/kyuubi/bin/beeline -u 'jdbc:hive2://0.0.0.0:10009/tpcds/tiny'`;
+```
+docker exec -it kyuubi /opt/kyuubi/bin/kyuubi-beeline -u 'jdbc:hive2://0.0.0.0:10009/tpcds/tiny'
Review Comment:
we recommend using `kyuubi-beeline` instead of `beeline`, to distinguish from Hive/Spark's `beeline`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#discussion_r2659455237
##########
docker/playground/.env:
##########
@@ -18,15 +18,15 @@
AWS_JAVA_SDK_VERSION=1.12.367
HADOOP_VERSION=3.3.6
HIVE_VERSION=2.3.9
-ICEBERG_VERSION=1.6.1
-KYUUBI_VERSION=1.9.0
+ICEBERG_VERSION=1.10.1
+KYUUBI_VERSION=1.11.0
KYUUBI_HADOOP_VERSION=3.3.6
POSTGRES_VERSION=12
POSTGRES_JDBC_VERSION=42.3.4
SCALA_BINARY_VERSION=2.12
-SPARK_VERSION=3.4.4
-SPARK_BINARY_VERSION=3.4
+SPARK_VERSION=3.5.7
+SPARK_BINARY_VERSION=3.5
SPARK_HADOOP_VERSION=3.3.4
Review Comment:
Spark 4 uses Hadoop client 3.4, which switches to AWS SDK 2.x, requires more work, so let's keep using Spark 3.5 for now. this also matches the current state of Kyuubi project - default Spakr version is 3.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#discussion_r2660114922
##########
docker/playground/image/kyuubi-playground-base.Dockerfile:
##########
@@ -10,13 +10,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM eclipse-temurin:8-focal
+FROM ubuntu:focal
Review Comment:
yes, we should move forward.
one additional consideration, we'd better align it with hadoop dev container, otherwise there might be some issues when using hadoop native libs, especially when users play with security configs. e.g., ubuntu focal is the latest version that provides openssl 1.x, the hadoop native libs shipped by official release compile against ubuntu focal with openssl 1.x, a runtime linkage issue will be thrown if we try to enable kerberos on ubuntu jammy or nobel.
but I think it has no issues for SIMPLE mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#discussion_r2659454478
##########
docker/playground/image/kyuubi-playground-base.Dockerfile:
##########
@@ -10,13 +10,23 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM eclipse-temurin:8-focal
+FROM ubuntu:focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV DEBCONF_TERSE=true
RUN set -x && \
+ echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/10disableextras && \
+ echo 'APT::Install-Suggests "0";' >> /etc/apt/apt.conf.d/10disableextras && \
ln -snf /usr/bin/bash /usr/bin/sh && \
apt-get update -q && \
- apt-get install -yq retry busybox && \
+ apt-get install -yq \
+ retry \
+ busybox \
+ openjdk-8-jdk-headless \
+ openjdk-17-jdk-headless && \
rm -rf /var/lib/apt/lists/* && \
+ update-java-alternatives --set $(update-java-alternatives --list | grep java-1.8.0-openjdk | awk '{print $NF}') || \
Review Comment:
use `||` to ignore the error code returned by `update-java-alternatives` command, as JDK 8 lacks some commands provided by mordern JDKs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#issuecomment-3707853968
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/7295?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 0.00%. Comparing base ([`eecdaaa`](https://app.codecov.io/gh/apache/kyuubi/commit/eecdaaa4d78fef90d6acac9ab36cc1b9d3babe1c?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a4b643d`](https://app.codecov.io/gh/apache/kyuubi/commit/a4b643d34b70780f03181f1a1bd3a88f220552ec?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #7295 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 698 698
Lines 43636 43636
Branches 5893 5893
======================================
Misses 43636 43636
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/7295?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#discussion_r2659460000
##########
docker/playground/image/kyuubi-playground-hadoop.Dockerfile:
##########
@@ -29,9 +28,7 @@ RUN set -x && \
tar -xzf ${HADOOP_TAR_NAME}.tar.gz -C /opt && \
ln -s /opt/hadoop-${HADOOP_VERSION} ${HADOOP_HOME} && \
rm ${HADOOP_TAR_NAME}.tar.gz && \
- HADOOP_CLOUD_STORAGE_JAR_NAME=hadoop-cloud-storage && \
- wget -q ${MAVEN_MIRROR}/org/apache/hadoop/${HADOOP_CLOUD_STORAGE_JAR_NAME}/${HADOOP_VERSION}/${HADOOP_CLOUD_STORAGE_JAR_NAME}-${HADOOP_VERSION}.jar -P ${HADOOP_HOME}/share/hadoop/hdfs/lib && \
Review Comment:
`hadoop-cloud-storage` is a package for assembling, has no classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#discussion_r2660132680
##########
docker/playground/image/kyuubi-playground-base.Dockerfile:
##########
@@ -10,13 +10,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM eclipse-temurin:8-focal
+FROM ubuntu:focal
Review Comment:
not related to this PR, another issue related to Hadoop and Ubuntu, the APT repo's `jsvc` is too old to support modern JDK, as Hadoop trunk is moving to JDK 17+, this could be another noisy for users to run kerberized Hadoop with JDK 17+ on Ubuntu, maybe we should contact Debian or Ubuntu Java team to upgrade it ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#issuecomment-3709086742
thanks, merged to master.
for ubuntu upgrading, I think hadoop is preparing to switch to ubuntu noble in the upcoming 3.5.0, let's wait for a little bit more time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#discussion_r2660064244
##########
docker/playground/image/kyuubi-playground-base.Dockerfile:
##########
@@ -10,13 +10,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM eclipse-temurin:8-focal
+FROM ubuntu:focal
Review Comment:
Ubuntu 20.04 (Focal Fossa) is already end of standard support.
https://ubuntu.com/blog/ubuntu-20-04-lts-end-of-life-standard-support-is-coming-to-an-end-heres-how-to-prepare
Let's upgrade the OS version in this PR or a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#discussion_r2660114922
##########
docker/playground/image/kyuubi-playground-base.Dockerfile:
##########
@@ -10,13 +10,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM eclipse-temurin:8-focal
+FROM ubuntu:focal
Review Comment:
yes, we should move forward.
one additional consideration, we'd better align it with hadoop dev container, otherwise there might be some issues when using hadoop native libs, especially when users play with security configs. e.g., ubuntu focal is the latest version that provides openssl 1.x, the hadoop native libs shipped by official release compile against ubuntu focal with openssl 1.x, a runtime linkage issue will be thrown if we try to enable kerberos on ubuntu jammy or noble.
but I think it has no issues for SIMPLE mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] Upgrade components of play ground to recent versions [kyuubi]
Posted by "aajisaka (via GitHub)" <gi...@apache.org>.
aajisaka commented on code in PR #7295:
URL: https://github.com/apache/kyuubi/pull/7295#discussion_r2660239997
##########
docker/playground/image/kyuubi-playground-base.Dockerfile:
##########
@@ -10,13 +10,25 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-FROM eclipse-temurin:8-focal
+FROM ubuntu:focal
Review Comment:
> APT repo's jsvc is too old
Found very old issue https://bugs.launchpad.net/ubuntu/+source/commons-daemon/+bug/1788154
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org