You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "Z1Wu (via GitHub)" <gi...@apache.org> on 2025/09/01 14:58:40 UTC
[PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Z1Wu opened a new pull request, #7182:
URL: https://github.com/apache/kyuubi/pull/7182
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/master/contributing/code/index.html
2. If the PR is related to an issue in https://github.com/apache/kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug, and what versions are affected.
-->
- There are some SQLs need to access hbase such as SQL read from Hive-Hbase Storage Handler tables. And for proxy-user basd SparkSQLEngine, it is unable to renew delegation token by itself. It would be convenient if kyuubi can renew hbase delegation token.
- Currently kyuubi server only supports updating hdfs and hms delegation token.
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
Add a unit test which fetching delegation token from a hbase cluster created by in docker containter.
### Was this patch authored or co-authored using generative AI tooling?
<!--
If a generative AI tooling has been used in the process of authoring this patch, please include
phrase 'Generated-by: ' followed by the name of the tool and its version.
If no, write 'No'.
Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details.
-->
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324222324
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
+ wget \
+ supervisor \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///tmp/hbase-data</value>
+ </property>
+ <property>
+ <name>hbase.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hbase.master.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.master.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2181</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.stream.capability.enforce</name>
+ <value>false</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.region.classes</name>
+ <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
Review Comment:
keep unified style for indentation, blank lines
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#issuecomment-3275317629
The failed unit tests seems irrelevant with this patch. Please review again at your convenience. Thank you.
@cxzl25 @pan3793 @zhouyifan279
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2330347805
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired = hbaseConf.get("hbase.security.authentication") == "kerberos"
Review Comment:
Yes, you are right. This config `hbase.security.authentication` is case insensitive.
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
+ wget \
+ supervisor \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///tmp/hbase-data</value>
+ </property>
+ <property>
+ <name>hbase.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hbase.master.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.master.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2181</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.stream.capability.enforce</name>
+ <value>false</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.region.classes</name>
+ <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2330299210
##########
kyuubi-server/pom.xml:
##########
@@ -123,6 +123,33 @@
<version>${kyuubi-relocated.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-client</artifactId>
+ <version>2.5.12-hadoop3</version>
Review Comment:
We use 2.5.12 and 1.4.9 in our production environment. And 2.5.12 client seems able to fetch token from servers of those two versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324157900
##########
dev/dependencyList:
##########
@@ -155,6 +157,9 @@ netty-transport/4.1.108.Final//netty-transport-4.1.108.Final.jar
okhttp-urlconnection/3.14.9//okhttp-urlconnection-3.14.9.jar
okhttp/3.12.12//okhttp-3.12.12.jar
okio/1.15.0//okio-1.15.0.jar
+opentelemetry-api/1.49.0//opentelemetry-api-1.49.0.jar
+opentelemetry-context/1.49.0//opentelemetry-context-1.49.0.jar
+opentelemetry-semconv/1.29.0-alpha//opentelemetry-semconv-1.29.0-alpha.jar
Review Comment:
are they necessary at runtime?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324157537
##########
dev/dependencyList:
##########
@@ -27,6 +27,7 @@ arrow-memory-core/16.0.0//arrow-memory-core-16.0.0.jar
arrow-memory-netty-buffer-patch/16.0.0//arrow-memory-netty-buffer-patch-16.0.0.jar
arrow-memory-netty/16.0.0//arrow-memory-netty-16.0.0.jar
arrow-vector/16.0.0//arrow-vector-16.0.0.jar
+audience-annotations/0.13.0//audience-annotations-0.13.0.jar
Review Comment:
this does not seem to be used at runtime, can we exclude it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2338434140
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,322 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ wget \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///tmp/hbase-data</value>
+ </property>
+ <property>
+ <name>hbase.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.security.authorization</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>hbase.master.hostname</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.unsafe.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+ <property>
+ <name>hbase.master.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2181</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.client.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.client.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.unsafe.stream.capability.enforce</name>
+ <value>false</value>
+ </property>
+ <property>
+ <name>hbase.coprocessor.region.classes</name>
+ <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+</configuration>
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/core-site.xml
+
+<configuration>
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authorization</name>
+ <value>true</value>
+ </property>
+
Review Comment:
again, remove the blank line to keep the consistent style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2330370085
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
+ wget \
+ supervisor \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///tmp/hbase-data</value>
+ </property>
+ <property>
+ <name>hbase.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hbase.master.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.master.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2181</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.stream.capability.enforce</name>
+ <value>false</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.region.classes</name>
+ <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+</configuration>
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/core-site.xml
+
+<configuration>
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>fs.defaultFS</name>
+ <value>hdfs://localhost:9000</value>
Review Comment:
fixed
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
+ wget \
+ supervisor \
Review Comment:
fixed
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2330307037
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired = hbaseConf.get("hbase.security.authentication") == "kerberos"
+ }
+
+ override def obtainDelegationTokens(
+ owner: String,
+ creds: Credentials): Unit = {
+ doAsProxyUser(owner) {
+ try {
+ info(s"Getting hbase token for ${owner} ...")
+ val conn = ConnectionFactory.createConnection(hbaseConf)
Review Comment:
Sounds reasonable, I use property name `hbase.deps.scope` to align with spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2314223664
##########
kyuubi-server/pom.xml:
##########
@@ -123,6 +123,33 @@
<version>${kyuubi-relocated.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-client</artifactId>
+ <version>2.5.12-hadoop3</version>
Review Comment:
I remember HBase has shaded-client, is it sufficient for requesting DT? I am afraid pulling Hive/HBase deps into the server classpath is the beginning of a nightmare ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2316068495
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired = hbaseConf.get("hbase.security.authentication") == "kerberos"
+ }
+
+ override def obtainDelegationTokens(
+ owner: String,
+ creds: Credentials): Unit = {
+ doAsProxyUser(owner) {
+ try {
+ info(s"Getting hbase token for ${owner} ...")
+ val conn = ConnectionFactory.createConnection(hbaseConf)
Review Comment:
If we use reflection to obtain tokens similar to Spark, this will avoid direct introduction of dependencies.
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/security/HBaseDelegationTokenProvider.scala#L45-L54
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2314292564
##########
kyuubi-server/pom.xml:
##########
@@ -123,6 +123,33 @@
<version>${kyuubi-relocated.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-client</artifactId>
+ <version>2.5.12-hadoop3</version>
Review Comment:
It seems ok to use this shaded hbase client.
```
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-shaded-client</artifactId>
<version>2.5.12-hadoop3</version>
</dependency>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324165054
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired = hbaseConf.get("hbase.security.authentication") == "kerberos"
+ }
+
+ override def obtainDelegationTokens(
+ owner: String,
+ creds: Credentials): Unit = {
+ doAsProxyUser(owner) {
+ try {
+ info(s"Getting hbase token for ${owner} ...")
+ val conn = ConnectionFactory.createConnection(hbaseConf)
Review Comment:
@Z1Wu I think we'd better align with Spark, we can add a maven property `hbase.scope`, default value is `provided`, then the official release does not ship HBase jars, and users who use HBase can build with `-Dhbase.scope=compile`.
If this approach is adopted, we must ensure it won't throw linkage issues without HBase jars under the Kyuubi server's classpath
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2348457680
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired =
+ hbaseConf.get("hbase.security.authentication", "simple").toLowerCase() == "kerberos"
+ }
+
+ override def obtainDelegationTokens(
+ owner: String,
+ creds: Credentials): Unit = {
+ doAsProxyUser(owner) {
+ try {
+ info(s"Getting HBase delegation token for ${owner} ...")
+ val conn = ConnectionFactory.createConnection(hbaseConf)
Review Comment:
Nice catch, fixed by referring to Spark's implementation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324173749
##########
kyuubi-server/pom.xml:
##########
@@ -123,6 +123,33 @@
<version>${kyuubi-relocated.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-client</artifactId>
+ <version>2.5.12-hadoop3</version>
Review Comment:
BTW, why is version 2.5.12 chosen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2338432716
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,322 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ wget \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
Review Comment:
convert TAB to spaces
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2338441112
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,322 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
Review Comment:
I will upload the image after you fix the style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2341109667
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,322 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ wget \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
Review Comment:
fixed
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,322 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ wget \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///tmp/hbase-data</value>
+ </property>
+ <property>
+ <name>hbase.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.security.authorization</name>
+ <value>true</value>
+ </property>
+ <property>
+ <name>hbase.master.hostname</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.unsafe.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+ <property>
+ <name>hbase.master.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2181</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.client.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.client.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.unsafe.stream.capability.enforce</name>
+ <value>false</value>
+ </property>
+ <property>
+ <name>hbase.coprocessor.region.classes</name>
+ <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+</configuration>
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/core-site.xml
+
+<configuration>
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authorization</name>
+ <value>true</value>
+ </property>
+
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2330348997
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
+ wget \
+ supervisor \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///tmp/hbase-data</value>
+ </property>
+ <property>
+ <name>hbase.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hbase.master.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.master.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2181</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.stream.capability.enforce</name>
+ <value>false</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.region.classes</name>
+ <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+</configuration>
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/core-site.xml
+
+<configuration>
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>fs.defaultFS</name>
+ <value>hdfs://localhost:9000</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
Review Comment:
fixed
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
+ wget \
+ supervisor \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///tmp/hbase-data</value>
+ </property>
+ <property>
+ <name>hbase.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hbase.master.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.master.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2181</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.stream.capability.enforce</name>
+ <value>false</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.region.classes</name>
+ <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+</configuration>
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/core-site.xml
+
+<configuration>
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>fs.defaultFS</name>
+ <value>hdfs://localhost:9000</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.proxyuser.hbase.hosts</name>
+ <value>*</value>
+ </property>
+
+ <property>
+ <name>hadoop.proxyuser.hbase.groups</name>
+ <value>*</value>
+ </property>
+
+</configuration>
+
+EOF
+
+COPY <<"EOF" ${ZK_HOME}/conf/zk-jaas.conf
+Server {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ storeKey=true
+ useTicketCache=false
+ keyTab="/opt/zookeeper/conf/zookeeper.keytab"
+ principal="zookeeper/localhost@TEST.ORG";
+};
+
+Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true;
+};
+EOF
+
+COPY <<"EOF" ${ZK_HOME}/conf/zoo.cfg
+tickTime=2000
+dataDir=/opt/zookeeper/data
+clientPort=2181
+# Kerberos settings
+authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
+kerberos.keytab.file=/opt/zookeeper/conf/zk.keytab
+kerberos.principal=zookeeper/localhost@TEST.ORG
+requireClientAuthScheme=sasl
+EOF
+
+COPY <<"EOF" /entrypoint.sh
+#!/bin/bash
+set -e
+
+# Function: Wait for a service to be available on specified ports
+# Parameters:
+# $1 - Service name
+# $2 - Timeout in seconds, default 60 seconds
+# $3 - Check interval in seconds, default 1 second
+# Remaining parameters - List of ports to check
+wait_for_service() {
+ local service_name="$1"
+ local timeout="${2:-60}"
+ local interval="${3:-1}"
+ shift 3 # Remove the first three parameters, remaining are ports
+
+ local end_time=$((SECONDS + timeout))
+ local ports_available=false
+ echo "Starting to wait for $service_name service to be available on ports $@..."
+
+ while [ $SECONDS -lt $end_time ]; do
+ ports_available=true
+
+ # Check all provided ports
+ for port in "$@"; do
+ if ! /opt/busybox/nc localhost "$port" -e true; then
+ ports_available=false
+ break
+ fi
+ done
+
+ if $ports_available; then
+ echo "$service_name service is now available"
+ return 0
+ else
+ echo "$service_name service is not available, retrying in $interval seconds..."
+ sleep $interval
+ fi
+ done
+
+ echo "Timeout reached. $service_name service is still not available."
+ exit 1
+}
+
+export REALM="TEST.ORG"
+export KADMIN_PASS="admin"
+export HBASE_CONF_DIR="${HBASE_HOME}/conf"
+export ZK_CONF_DIR="${ZK_HOME}/conf"
+
+echo "Creating KDC database..."
+kdb5_util create -r ${REALM} -s -P ${KADMIN_PASS}
+
+echo "Creating principals and keytabs..."
+kadmin.local -q "addprinc -randkey hbase/localhost@${REALM}"
+kadmin.local -q "ktadd -k ${HBASE_CONF_DIR}/hbase.keytab hbase/localhost@${REALM}"
+
+kadmin.local -q "addprinc -randkey zookeeper/localhost@${REALM}"
+kadmin.local -q "ktadd -k ${ZK_CONF_DIR}/zookeeper.keytab zookeeper/localhost@${REALM}"
+
+mkdir -p ${HBASE_HOME}/logs/
+mkdir -p ${ZK_HOME}/logs/
+
+echo "Principals created. Starting supervisor..."
+# create kdc service
+echo "Start kdc ..."
+/etc/init.d/krb5-kdc start
+wait_for_service "kdc" 15 1 88
+
+# start zookeeper server
+export JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper/conf/zk-jaas.conf"
+/opt/zookeeper/bin/zkServer.sh start
+unset JVMFLAGS
+wait_for_service "zookeeper" 15 1 2181
+
+kinit -kt ${HBASE_CONF_DIR}/hbase.keytab hbase/localhost@${REALM}
+
+# # start hbase master server
+nohup /opt/hbase/bin/hbase master start > "/opt/hbase/logs/master.log" 2>&1 < /dev/null &
+wait_for_service "master" 15 1 16000
+
+# # start hbase region server
+nohup /opt/hbase/bin/hbase regionserver start > "/opt/hbase/logs/regionserver.log" 2>&1 < /dev/null &
+wait_for_service "master" 15 1 16020
+
+tail -F --follow=name --retry -v /opt/zookeeper/logs/* /opt/hbase/logs/*
+
+
+EOF
+
+RUN chmod +x /entrypoint.sh
+
+# HBase Master UI
+EXPOSE 16010
+# Kerberos
+EXPOSE 88/udp
+# zookeeper
+EXPOSE 2181
+# HBase Master
+EXPOSE 16000
+# HBase RegionServer
+EXPOSE 16020
+
+ENTRYPOINT ["/entrypoint.sh"]
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2330320455
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired = hbaseConf.get("hbase.security.authentication") == "kerberos"
+ }
+
+ override def obtainDelegationTokens(
+ owner: String,
+ creds: Credentials): Unit = {
+ doAsProxyUser(owner) {
+ try {
+ info(s"Getting hbase token for ${owner} ...")
+ val conn = ConnectionFactory.createConnection(hbaseConf)
+ val token = ClientTokenUtil.obtainToken(conn)
+ info(s"Get hbase token ${token}")
+ creds.addToken(token.getService, token)
+ } catch {
+ case e: Throwable =>
+ error(createFailedToGetTokenMessage(serviceName, e), e)
+ throw new KyuubiException(s"Failed to get hbase token owned by $owner", e)
+ }
+ }
+ }
+
+ override def delegationTokensRequired(): Boolean = {
+ tokenRequired
+ }
+
+ private def createFailedToGetTokenMessage(serviceName: String, e: scala.Throwable): String = {
+ val message = "Failed to get token from service %s due to %s. "
+ message.format(serviceName, e, serviceName, serviceName)
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2347877574
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired =
+ hbaseConf.get("hbase.security.authentication", "simple").toLowerCase() == "kerberos"
+ }
+
+ override def obtainDelegationTokens(
+ owner: String,
+ creds: Credentials): Unit = {
+ doAsProxyUser(owner) {
+ try {
+ info(s"Getting HBase delegation token for ${owner} ...")
+ val conn = ConnectionFactory.createConnection(hbaseConf)
Review Comment:
Nice catch!
I have re-read Spark's implementation of calling Hbase, and there is a close connection.
org.apache.hadoop.hbase.security.token.TokenUtil#obtainToken(org.apache.hadoop.conf.Configuration)
```java
try (Connection connection = ConnectionFactory.createConnection(conf)) {
return obtainToken(connection);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324159042
##########
kyuubi-server/pom.xml:
##########
@@ -123,6 +123,33 @@
<version>${kyuubi-relocated.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-client</artifactId>
+ <version>2.5.12-hadoop3</version>
Review Comment:
define version at root `pom.xml`'s `dependencyManagement`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324173749
##########
kyuubi-server/pom.xml:
##########
@@ -123,6 +123,33 @@
<version>${kyuubi-relocated.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-client</artifactId>
+ <version>2.5.12-hadoop3</version>
Review Comment:
BTW, why version 2.5.12 is chosen?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2342956352
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,322 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
Review Comment:
@pan3793 All style related comment have been resolved, PTAL. And the failed unit tests seem irrelevant with the modification.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "zhouyifan279 (via GitHub)" <gi...@apache.org>.
zhouyifan279 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2347104477
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired =
+ hbaseConf.get("hbase.security.authentication", "simple").toLowerCase() == "kerberos"
+ }
+
+ override def obtainDelegationTokens(
+ owner: String,
+ creds: Credentials): Unit = {
+ doAsProxyUser(owner) {
+ try {
+ info(s"Getting HBase delegation token for ${owner} ...")
+ val conn = ConnectionFactory.createConnection(hbaseConf)
Review Comment:
Connection should be closed after obtaining token.
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
Review Comment:
`hadoopConf` does not contain HBase properties. HBase properties is in hbase-site.xml and needs to be loaded using class `HBaseConfiguration`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2348440850
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
Review Comment:
Currently I enable this feature by adding hbase related configuration in `kyuubi-default.conf`. So that conf is initialized as below:
``` scala
// org.apache.kyuubi.credentials.HadoopCredentialsManager#initialize
override def initialize(conf: KyuubiConf): Unit = {
hadoopConf = KyuubiHadoopUtils.newHadoopConf(conf)
providers = HadoopCredentialsManager.loadProviders(conf)
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2314292564
##########
kyuubi-server/pom.xml:
##########
@@ -123,6 +123,33 @@
<version>${kyuubi-relocated.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
+ <artifactId>hbase-client</artifactId>
+ <version>2.5.12-hadoop3</version>
Review Comment:
It seems ok to using this shaded hbase client.
```
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-shaded-client</artifactId>
<version>2.5.12-hadoop3</version>
</dependency>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2338438209
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired = hbaseConf.get("hbase.security.authentication") == "kerberos"
+ }
+
+ override def obtainDelegationTokens(
+ owner: String,
+ creds: Credentials): Unit = {
+ doAsProxyUser(owner) {
+ try {
+ info(s"Getting hbase token for ${owner} ...")
+ val conn = ConnectionFactory.createConnection(hbaseConf)
Review Comment:
Have you tested running Kyuubi server without HBase deps? Will it cause linkage error on initializing `HBaseDelegationTokenProvider`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#issuecomment-3247913033
## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/7182?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `0%` with `21 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 0.00%. Comparing base ([`ea75fa8`](https://app.codecov.io/gh/apache/kyuubi/commit/ea75fa8f13e02be4bc608a65adbfe5f6b6045ea1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`9244dac`](https://app.codecov.io/gh/apache/kyuubi/commit/9244dac1904049dba2926707e01503ccffc707a4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/kyuubi/pull/7182?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...ubi/credentials/HBaseDelegationTokenProvider.scala](https://app.codecov.io/gh/apache/kyuubi/pull/7182?src=pr&el=tree&filepath=kyuubi-server%2Fsrc%2Fmain%2Fscala%2Forg%2Fapache%2Fkyuubi%2Fcredentials%2FHBaseDelegationTokenProvider.scala&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jcmVkZW50aWFscy9IQmFzZURlbGVnYXRpb25Ub2tlblByb3ZpZGVyLnNjYWxh) | 0.00% | [21 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/7182?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #7182 +/- ##
======================================
Coverage 0.00% 0.00%
======================================
Files 695 696 +1
Lines 43433 43454 +21
Branches 5887 5888 +1
======================================
- Misses 43433 43454 +21
```
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/7182?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324223620
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
+ wget \
+ supervisor \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///tmp/hbase-data</value>
+ </property>
+ <property>
+ <name>hbase.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hbase.master.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.master.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2181</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.stream.capability.enforce</name>
+ <value>false</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.region.classes</name>
+ <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+</configuration>
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/core-site.xml
+
+<configuration>
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>fs.defaultFS</name>
+ <value>hdfs://localhost:9000</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.proxyuser.hbase.hosts</name>
+ <value>*</value>
+ </property>
+
+ <property>
+ <name>hadoop.proxyuser.hbase.groups</name>
+ <value>*</value>
+ </property>
+
+</configuration>
+
+EOF
+
+COPY <<"EOF" ${ZK_HOME}/conf/zk-jaas.conf
+Server {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useKeyTab=true
+ storeKey=true
+ useTicketCache=false
+ keyTab="/opt/zookeeper/conf/zookeeper.keytab"
+ principal="zookeeper/localhost@TEST.ORG";
+};
+
+Client {
+ com.sun.security.auth.module.Krb5LoginModule required
+ useTicketCache=true;
+};
+EOF
+
+COPY <<"EOF" ${ZK_HOME}/conf/zoo.cfg
+tickTime=2000
+dataDir=/opt/zookeeper/data
+clientPort=2181
+# Kerberos settings
+authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider
+kerberos.keytab.file=/opt/zookeeper/conf/zk.keytab
+kerberos.principal=zookeeper/localhost@TEST.ORG
+requireClientAuthScheme=sasl
+EOF
+
+COPY <<"EOF" /entrypoint.sh
+#!/bin/bash
+set -e
+
+# Function: Wait for a service to be available on specified ports
+# Parameters:
+# $1 - Service name
+# $2 - Timeout in seconds, default 60 seconds
+# $3 - Check interval in seconds, default 1 second
+# Remaining parameters - List of ports to check
+wait_for_service() {
+ local service_name="$1"
+ local timeout="${2:-60}"
+ local interval="${3:-1}"
+ shift 3 # Remove the first three parameters, remaining are ports
+
+ local end_time=$((SECONDS + timeout))
+ local ports_available=false
+ echo "Starting to wait for $service_name service to be available on ports $@..."
+
+ while [ $SECONDS -lt $end_time ]; do
+ ports_available=true
+
+ # Check all provided ports
+ for port in "$@"; do
+ if ! /opt/busybox/nc localhost "$port" -e true; then
+ ports_available=false
+ break
+ fi
+ done
+
+ if $ports_available; then
+ echo "$service_name service is now available"
+ return 0
+ else
+ echo "$service_name service is not available, retrying in $interval seconds..."
+ sleep $interval
+ fi
+ done
+
+ echo "Timeout reached. $service_name service is still not available."
+ exit 1
+}
+
+export REALM="TEST.ORG"
+export KADMIN_PASS="admin"
+export HBASE_CONF_DIR="${HBASE_HOME}/conf"
+export ZK_CONF_DIR="${ZK_HOME}/conf"
+
+echo "Creating KDC database..."
+kdb5_util create -r ${REALM} -s -P ${KADMIN_PASS}
+
+echo "Creating principals and keytabs..."
+kadmin.local -q "addprinc -randkey hbase/localhost@${REALM}"
+kadmin.local -q "ktadd -k ${HBASE_CONF_DIR}/hbase.keytab hbase/localhost@${REALM}"
+
+kadmin.local -q "addprinc -randkey zookeeper/localhost@${REALM}"
+kadmin.local -q "ktadd -k ${ZK_CONF_DIR}/zookeeper.keytab zookeeper/localhost@${REALM}"
+
+mkdir -p ${HBASE_HOME}/logs/
+mkdir -p ${ZK_HOME}/logs/
+
+echo "Principals created. Starting supervisor..."
+# create kdc service
+echo "Start kdc ..."
+/etc/init.d/krb5-kdc start
+wait_for_service "kdc" 15 1 88
+
+# start zookeeper server
+export JVMFLAGS="-Djava.security.auth.login.config=/opt/zookeeper/conf/zk-jaas.conf"
+/opt/zookeeper/bin/zkServer.sh start
+unset JVMFLAGS
+wait_for_service "zookeeper" 15 1 2181
+
+kinit -kt ${HBASE_CONF_DIR}/hbase.keytab hbase/localhost@${REALM}
+
+# # start hbase master server
+nohup /opt/hbase/bin/hbase master start > "/opt/hbase/logs/master.log" 2>&1 < /dev/null &
+wait_for_service "master" 15 1 16000
+
+# # start hbase region server
+nohup /opt/hbase/bin/hbase regionserver start > "/opt/hbase/logs/regionserver.log" 2>&1 < /dev/null &
+wait_for_service "master" 15 1 16020
+
+tail -F --follow=name --retry -v /opt/zookeeper/logs/* /opt/hbase/logs/*
+
+
+EOF
+
+RUN chmod +x /entrypoint.sh
+
+# HBase Master UI
+EXPOSE 16010
+# Kerberos
+EXPOSE 88/udp
+# zookeeper
+EXPOSE 2181
+# HBase Master
+EXPOSE 16000
+# HBase RegionServer
+EXPOSE 16020
+
+ENTRYPOINT ["/entrypoint.sh"]
Review Comment:
nit: new line at end
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#issuecomment-3300944477
Thanks all, merged to master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2343042820
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,322 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
Review Comment:
`nekyuubi/kyuubi-hbase-cluster:latest` is available on dockerhub
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324222690
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
+ wget \
+ supervisor \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///tmp/hbase-data</value>
+ </property>
+ <property>
+ <name>hbase.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hbase.master.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.master.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2181</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.stream.capability.enforce</name>
+ <value>false</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.region.classes</name>
+ <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+</configuration>
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/core-site.xml
+
+<configuration>
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>fs.defaultFS</name>
+ <value>hdfs://localhost:9000</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
Review Comment:
duplicated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2316049549
##########
kyuubi-server/src/main/resources/META-INF/services/org.apache.kyuubi.credentials.HadoopDelegationTokenProvider:
##########
@@ -17,3 +17,5 @@
org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider
org.apache.kyuubi.credentials.HiveDelegationTokenProvider
+org.apache.kyuubi.credentials.YarnRMDelegationTokenProvider
Review Comment:
What is this `org.apache.kyuubi.credentials.YarnRMDelegationTokenProvider`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2316235782
##########
kyuubi-server/src/main/resources/META-INF/services/org.apache.kyuubi.credentials.HadoopDelegationTokenProvider:
##########
@@ -17,3 +17,5 @@
org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider
org.apache.kyuubi.credentials.HiveDelegationTokenProvider
+org.apache.kyuubi.credentials.YarnRMDelegationTokenProvider
Review Comment:
fixed, this line is irrelevant to this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "zhouyifan279 (via GitHub)" <gi...@apache.org>.
zhouyifan279 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2354001133
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
Review Comment:
This reminds me that we need to add the new token provider in the doc [here](https://github.com/apache/kyuubi/blob/master/docs/security/hadoop_credentials_manager.md).
@Z1Wu would you like to write the doc in a following pr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #7182: [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine
URL: https://github.com/apache/kyuubi/pull/7182
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#issuecomment-3245578192
Failed unit tests in CI look confusing. I run that unit test locally without failure. How could I debug the problem in CI.
<img width="2052" height="1323" alt="image" src="https://github.com/user-attachments/assets/8087c0ae-3b04-4e26-a7b8-69a3c1e5951f" />
<img width="2543" height="923" alt="image" src="https://github.com/user-attachments/assets/6d0906f1-ec5e-4a91-af48-475d22ef5f0b" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324155663
##########
kyuubi-server/src/test/scala/org/apache/kyuubi/WithSecuredHBaseContainer.scala:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi
+
+import java.io.File
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Paths}
+
+import com.dimafeng.testcontainers.{ContainerDef, FixedHostPortGenericContainer}
+import com.dimafeng.testcontainers.scalatest.TestContainerForAll
+import com.github.dockerjava.api.model.{ExposedPort, Ports}
+import org.apache.hadoop.conf.Configuration
+import org.testcontainers.containers.wait.strategy.HostPortWaitStrategy
+
+trait WithSecuredHBaseContainer extends KerberizedTestHelper with TestContainerForAll {
+
+ final val HADOOP_SECURITY_AUTHENTICATION = "kerberos"
+ final val HBASE_KERBEROS_REALM = "TEST.ORG"
+ final val HBASE_KERBEROS_PRINCIPAL = "hbase/localhost"
+ final val HBASE_KERBEROS_KEYTAB = "/opt/hbase/conf/hbase.keytab"
+ final val DOCKER_IMAGE_NAME = "z1wu97/kyuubi-hbase-cluster:latest"
Review Comment:
Does this docker have a github repository
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "zhouyifan279 (via GitHub)" <gi...@apache.org>.
zhouyifan279 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2328533552
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
+ wget \
+ supervisor \
Review Comment:
Supervisor should be removed if it is not used.
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
+ wget \
+ supervisor \
+ busybox \
+ && mkdir /opt/busybox && \
+ busybox --install /opt/busybox
+
+# download hbase
+RUN wget -q ${APACHE_MIRROR}/hbase/${HBASE_VERSION}/hbase-${HBASE_VERSION}-bin.tar.gz && \
+ tar -xzf hbase-${HBASE_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/hbase-${HBASE_VERSION} ${HBASE_HOME} && \
+ rm hbase-${HBASE_VERSION}-bin.tar.gz
+
+# download zookeeper
+RUN wget -q ${APACHE_MIRROR}/zookeeper/zookeeper-${ZK_VERSION}/apache-zookeeper-${ZK_VERSION}-bin.tar.gz && \
+ tar -xzf apache-zookeeper-${ZK_VERSION}-bin.tar.gz -C /opt && \
+ mv /opt/apache-zookeeper-${ZK_VERSION}-bin ${ZK_HOME} && \
+ rm apache-zookeeper-${ZK_VERSION}-bin.tar.gz
+
+# add config files
+COPY <<"EOF" /etc/krb5.conf
+
+[libdefaults]
+ default_realm = TEST.ORG
+ dns_lookup_realm = false
+ dns_lookup_kdc = false
+
+[realms]
+ TEST.ORG = {
+ kdc = localhost:88
+ }
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/hbase-site.xml
+
+<configuration>
+ <property>
+ <name>hbase.cluster.distributed</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hbase.rootdir</name>
+ <value>file:///tmp/hbase-data</value>
+ </property>
+ <property>
+ <name>hbase.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+ <property>
+ <name>hbase.master.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.master.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+ <property>
+ <name>hbase.regionserver.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.security.authorization</name>
+ <value>true</value>
+ </property>
+
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>hbase.master.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.regionserver.hostname</name>
+ <value>localhost</value>
+ </property>
+
+ <property>
+ <name>hbase.regionserver.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.master.ipc.address</name>
+ <value>0.0.0.0</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.quorum</name>
+ <value>localhost</value>
+ </property>
+ <property>
+ <name>hbase.zookeeper.property.clientPort</name>
+ <value>2181</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.keytab.file</name>
+ <value>/opt/hbase/conf/hbase.keytab</value>
+ </property>
+
+ <property>
+ <name>hbase.zookeeper.client.kerberos.principal</name>
+ <value>hbase/localhost@TEST.ORG</value>
+ </property>
+
+ <property>
+ <name>hbase.unsafe.stream.capability.enforce</name>
+ <value>false</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.region.classes</name>
+ <value>org.apache.hadoop.hbase.security.token.TokenProvider,org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+
+ <property>
+ <name>hbase.coprocessor.master.classes</name>
+ <value>org.apache.hadoop.hbase.security.access.AccessController</value>
+ </property>
+</configuration>
+
+EOF
+
+COPY <<"EOF" ${HBASE_HOME}/conf/core-site.xml
+
+<configuration>
+ <property>
+ <name>hadoop.security.authentication</name>
+ <value>kerberos</value>
+ </property>
+
+ <property>
+ <name>fs.defaultFS</name>
+ <value>hdfs://localhost:9000</value>
Review Comment:
HDFS is not used. Better to remove this property to avoid confusion.
##########
build/Dockerfile.HBase:
##########
@@ -0,0 +1,355 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+# This Dockerfile is for building a HBase Cluster used in testing
+
+# Usage:
+# Run the docker command below
+# docker build \
+# --build-arg APACHE_MIRROR="https://archive.apache.org/dist" \
+# --build-arg HBASE_VERSION="2.5.12" \
+# --build-arg ZK_VERSION="3.8.4" \
+# --file build/Dockerfile.HBase \
+# --tag nekyuubi/kyuubi-hbase-cluster:<tag> \
+# .
+# Options:
+# -f, --file this docker file
+# -t, --tag the target repo and tag name
+# more options can be found with -h, --help
+
+FROM eclipse-temurin:8-jdk-focal
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+ARG HBASE_VERSION=2.5.12
+ARG APACHE_MIRROR=https://archive.apache.org/dist
+ARG ZK_VERSION=3.8.1
+
+ENV HBASE_HOME=/opt/hbase
+ENV ZK_HOME=/opt/zookeeper
+ENV JAVA_HOME=/opt/java/openjdk
+ENV PATH=$PATH:$HBASE_HOME/bin
+
+RUN apt-get update && apt-get install -y \
+ krb5-kdc \
+ krb5-admin-server \
+ openjdk-8-jdk \
Review Comment:
Base image already has a JDK 8 installed. Why install another one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324226771
##########
kyuubi-server/src/test/scala/org/apache/kyuubi/WithSecuredHBaseContainer.scala:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi
+
+import java.io.File
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Paths}
+
+import com.dimafeng.testcontainers.{ContainerDef, FixedHostPortGenericContainer}
+import com.dimafeng.testcontainers.scalatest.TestContainerForAll
+import com.github.dockerjava.api.model.{ExposedPort, Ports}
+import org.apache.hadoop.conf.Configuration
+import org.testcontainers.containers.wait.strategy.HostPortWaitStrategy
+
+trait WithSecuredHBaseContainer extends KerberizedTestHelper with TestContainerForAll {
+
+ final val HADOOP_SECURITY_AUTHENTICATION = "kerberos"
+ final val HBASE_KERBEROS_REALM = "TEST.ORG"
+ final val HBASE_KERBEROS_PRINCIPAL = "hbase/localhost"
+ final val HBASE_KERBEROS_KEYTAB = "/opt/hbase/conf/hbase.keytab"
+ final val DOCKER_IMAGE_NAME = "z1wu97/kyuubi-hbase-cluster:latest"
Review Comment:
This must be either built in the CI workflow, or managed by Kyuubi PMC. I can help create an image after you freeze the Dockerfile
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324220670
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired = hbaseConf.get("hbase.security.authentication") == "kerberos"
Review Comment:
is the config value case sensitive? IIRC, `hadoop.security.authentication` is case insensitive
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2330340251
##########
dev/dependencyList:
##########
@@ -27,6 +27,7 @@ arrow-memory-core/16.0.0//arrow-memory-core-16.0.0.jar
arrow-memory-netty-buffer-patch/16.0.0//arrow-memory-netty-buffer-patch-16.0.0.jar
arrow-memory-netty/16.0.0//arrow-memory-netty-16.0.0.jar
arrow-vector/16.0.0//arrow-vector-16.0.0.jar
+audience-annotations/0.13.0//audience-annotations-0.13.0.jar
Review Comment:
It is ok to exclude this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2330344821
##########
dev/dependencyList:
##########
@@ -155,6 +157,9 @@ netty-transport/4.1.108.Final//netty-transport-4.1.108.Final.jar
okhttp-urlconnection/3.14.9//okhttp-urlconnection-3.14.9.jar
okhttp/3.12.12//okhttp-3.12.12.jar
okio/1.15.0//okio-1.15.0.jar
+opentelemetry-api/1.49.0//opentelemetry-api-1.49.0.jar
+opentelemetry-context/1.49.0//opentelemetry-context-1.49.0.jar
+opentelemetry-semconv/1.29.0-alpha//opentelemetry-semconv-1.29.0-alpha.jar
Review Comment:
Yes , excluding these will result in ClassNotFound Exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2343043390
##########
kyuubi-server/src/test/scala/org/apache/kyuubi/WithSecuredHBaseContainer.scala:
##########
@@ -0,0 +1,150 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi
+
+import java.io.File
+import java.nio.charset.StandardCharsets
+import java.nio.file.{Files, Paths}
+
+import com.dimafeng.testcontainers.{ContainerDef, FixedHostPortGenericContainer}
+import com.dimafeng.testcontainers.scalatest.TestContainerForAll
+import com.github.dockerjava.api.model.{ExposedPort, Ports}
+import org.apache.hadoop.conf.Configuration
+import org.testcontainers.containers.wait.strategy.HostPortWaitStrategy
+
+trait WithSecuredHBaseContainer extends KerberizedTestHelper with TestContainerForAll {
+
+ final val HADOOP_SECURITY_AUTHENTICATION = "kerberos"
+ final val HBASE_KERBEROS_REALM = "TEST.ORG"
+ final val HBASE_KERBEROS_PRINCIPAL = "hbase/localhost"
+ final val HBASE_KERBEROS_KEYTAB = "/opt/hbase/conf/hbase.keytab"
+ final val DOCKER_IMAGE_NAME = "z1wu97/kyuubi-hbase-cluster:latest"
Review Comment:
```suggestion
final val DOCKER_IMAGE_NAME = "nekyuubi/kyuubi-hbase-cluster:latest"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2322504255
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired = hbaseConf.get("hbase.security.authentication") == "kerberos"
+ }
+
+ override def obtainDelegationTokens(
+ owner: String,
+ creds: Credentials): Unit = {
+ doAsProxyUser(owner) {
+ try {
+ info(s"Getting hbase token for ${owner} ...")
+ val conn = ConnectionFactory.createConnection(hbaseConf)
Review Comment:
Introducing HBase's dependencies directly is based on the following considerations :
- Using reflection may reduce code readability and make unit testing more difficult to implement.
- Furthermore, it increases the deployment complexity of the Kyuubi Server, as users need to place related HBase jars into the classpath of the Kyuubi Server.
- The shaded hbase client package seems to have clean depdency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#issuecomment-3254176953
All failed unit tests have been fixed. Please review this patch at your convenience. Thank you.
@cxzl25 @pan3793 @zhouyifan279
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update hbase delegation token for Spark SQL Engine [kyuubi]
Posted by "cxzl25 (via GitHub)" <gi...@apache.org>.
cxzl25 commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2324151329
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,69 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
+ this.tokenRequired = hbaseConf.get("hbase.security.authentication") == "kerberos"
+ }
+
+ override def obtainDelegationTokens(
+ owner: String,
+ creds: Credentials): Unit = {
+ doAsProxyUser(owner) {
+ try {
+ info(s"Getting hbase token for ${owner} ...")
+ val conn = ConnectionFactory.createConnection(hbaseConf)
+ val token = ClientTokenUtil.obtainToken(conn)
+ info(s"Get hbase token ${token}")
+ creds.addToken(token.getService, token)
+ } catch {
+ case e: Throwable =>
+ error(createFailedToGetTokenMessage(serviceName, e), e)
+ throw new KyuubiException(s"Failed to get hbase token owned by $owner", e)
+ }
+ }
+ }
+
+ override def delegationTokensRequired(): Boolean = {
+ tokenRequired
+ }
+
+ private def createFailedToGetTokenMessage(serviceName: String, e: scala.Throwable): String = {
+ val message = "Failed to get token from service %s due to %s. "
+ message.format(serviceName, e, serviceName, serviceName)
Review Comment:
extra parameters
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "zhouyifan279 (via GitHub)" <gi...@apache.org>.
zhouyifan279 commented on PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#issuecomment-3300881102
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
Re: [PR] [KYUUBI #7156] Support update HBase delegation token for Spark SQL Engine [kyuubi]
Posted by "Z1Wu (via GitHub)" <gi...@apache.org>.
Z1Wu commented on code in PR #7182:
URL: https://github.com/apache/kyuubi/pull/7182#discussion_r2348440850
##########
kyuubi-server/src/main/scala/org/apache/kyuubi/credentials/HBaseDelegationTokenProvider.scala:
##########
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.credentials
+
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.hbase.client.ConnectionFactory
+import org.apache.hadoop.hbase.security.token.ClientTokenUtil
+import org.apache.hadoop.security.Credentials
+
+import org.apache.kyuubi.{KyuubiException, Logging}
+import org.apache.kyuubi.config.KyuubiConf
+import org.apache.kyuubi.credentials.HadoopFsDelegationTokenProvider.doAsProxyUser
+
+private class HBaseDelegationTokenProvider
+ extends HadoopDelegationTokenProvider with Logging {
+
+ override def serviceName: String = "hbase"
+ private var tokenRequired: Boolean = _
+ private var hbaseConf: Configuration = _
+ private var kyuubiConf: KyuubiConf = _
+
+ override def initialize(hadoopConf: Configuration, kyuubiConf: KyuubiConf): Unit = {
+ this.kyuubiConf = kyuubiConf
+ this.hbaseConf = hadoopConf
Review Comment:
Currently I enable this feature by adding hbase related configurations in `kyuubi-default.conf`. And hadoop conf is initialized as below:
``` scala
// org.apache.kyuubi.credentials.HadoopCredentialsManager#initialize
override def initialize(conf: KyuubiConf): Unit = {
hadoopConf = KyuubiHadoopUtils.newHadoopConf(conf)
providers = HadoopCredentialsManager.loadProviders(conf)
.filter { case (_, provider) =>
provider.initialize(hadoopConf, conf)
val required = provider.delegationTokensRequired()
if (!required) {
warn(s"Service ${provider.serviceName} does not require a token." +
s" Check your configuration to see if security is disabled or not." +
s" If security is enabled, some configurations of ${provider.serviceName} " +
s" might be missing, please check the configurations in " +
s" https://kyuubi.readthedocs.io/en/master/security" +
s"/hadoop_credentials_manager.html#required-security-configs")
provider.close()
}
required
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org