You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by "zhaohehuhu (via GitHub)" <gi...@apache.org> on 2024/04/26 07:45:10 UTC

[PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

zhaohehuhu opened a new pull request, #6339:
URL: https://github.com/apache/kyuubi/pull/6339

   # :mag: Description
   ## Issue References ๐Ÿ”—
   <!-- Append the issue number after #. If there is no issue for you to link create one or -->
   <!-- If there are no issues to link, please provide details here. -->
   as tile
   This pull request close  #6255
   
   ## Describe Your Solution ๐Ÿ”ง
   
   Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
   
   
   ## Types of changes :bookmark:
   <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
   - [ ] Bugfix (non-breaking change which fixes an issue)
   - [ ] New feature (non-breaking change which adds functionality)
   - [ ] Breaking change (fix or feature that would cause existing functionality to change)
   
   ## Test Plan ๐Ÿงช
   
   #### Behavior Without This Pull Request :coffin:
   
   
   #### Behavior With This Pull Request :tada:
   
   
   #### Related Unit Tests
   
   
   ---
   
   # Checklist ๐Ÿ“
   <!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
   <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
   
   - [ ] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html)
   
   **Be nice. Be informative.**
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1580720726


##########
kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParser.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.hs2connection;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+import java.util.Map.Entry;
+import java.util.stream.Collectors;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class implements HS2ConnectionFileParser for the user-specific connection configuration file
+ * named kyuubi-defaults.conf. The class looks for this file in ${KYUUBI_CONF_DIR} in that order and
+ * uses the first file found in the above locations.
+ */
+public class KyuubiConfFileParser implements HS2ConnectionFileParser {
+
+  public static final String DEFAULT_CONNECTION_CONFIG_FILE_NAME = "kyuubi-defaults.conf";
+
+  private final List<String> locations = new ArrayList<>();
+  private static final Logger log = LoggerFactory.getLogger(KyuubiConfFileParser.class);
+
+  public KyuubiConfFileParser() {
+    if (System.getenv("KYUUBI_CONF_DIR") != null) {
+      locations.add(
+          System.getenv("KYUUBI_CONF_DIR") + File.separator + DEFAULT_CONNECTION_CONFIG_FILE_NAME);
+    }
+  }
+
+  @VisibleForTesting
+  KyuubiConfFileParser(List<String> testLocations) {
+    if (testLocations == null) {
+      return;
+    }
+    locations.addAll(testLocations);
+  }
+
+  @Override
+  public Properties getConnectionProperties() throws KyuubiConfFileParseException {
+    Properties props = new Properties();
+    String fileLocation = getFileLocation();
+    if (fileLocation == null) {
+      log.debug("Kyuubi default configuration file not found");
+      return props;
+    }
+    log.info("Using Kyuubi default configuration file at " + fileLocation);
+    File propsFile = new File(fileLocation);
+    try {
+      Map<String, String> propsMap = getPropertiesFromFile(propsFile);
+      StringBuilder hiveConf = new StringBuilder();
+      for (Entry<String, String> kv : propsMap.entrySet()) {

Review Comment:
   Currently, most of configurations in `kyuubi-defaults.conf` are server-side configurations. We can define a prefix to determine whether it is a client config, and then add them to hive_conf of jdbc url.
   
   In addition to inferring `HIVE_CONF_PROPERTY_KEY`, we also need to infer `HOST_PROPERTY_KEY` based on server configuration.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#issuecomment-2078973479

   we need the design before implementation.
   
   what's the priority of those configuration files? should we merge that?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1580682343


##########
kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/BeeLine.java:
##########
@@ -1012,6 +1010,14 @@ private String getDefaultConnectionUrl(CommandLine cl) throws BeelineConfFilePar
       userConnectionProperties = userHS2ConnFileParser.getConnectionProperties();
     }
 
+    if (kyuubiConfFileParser.configExists()) {
+      // get the connection properties from user specific config file
+      Properties properties = kyuubiConfFileParser.getConnectionProperties();
+      for (String key : properties.stringPropertyNames()) {
+        userConnectionProperties.setProperty(key, properties.getProperty(key));

Review Comment:
   We should merge `HIVE_CONF_PROPERTY_KEY` instead of overwriting.



##########
kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParser.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.hs2connection;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+import java.util.Map.Entry;
+import java.util.stream.Collectors;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class implements HS2ConnectionFileParser for the user-specific connection configuration file
+ * named kyuubi-defaults.conf. The class looks for this file in ${KYUUBI_CONF_DIR} in that order and
+ * uses the first file found in the above locations.
+ */
+public class KyuubiConfFileParser implements HS2ConnectionFileParser {
+
+  public static final String DEFAULT_CONNECTION_CONFIG_FILE_NAME = "kyuubi-defaults.conf";
+
+  private final List<String> locations = new ArrayList<>();
+  private static final Logger log = LoggerFactory.getLogger(KyuubiConfFileParser.class);
+
+  public KyuubiConfFileParser() {
+    if (System.getenv("KYUUBI_CONF_DIR") != null) {
+      locations.add(
+          System.getenv("KYUUBI_CONF_DIR") + File.separator + DEFAULT_CONNECTION_CONFIG_FILE_NAME);
+    }
+  }
+
+  @VisibleForTesting
+  KyuubiConfFileParser(List<String> testLocations) {
+    if (testLocations == null) {
+      return;
+    }
+    locations.addAll(testLocations);
+  }
+
+  @Override
+  public Properties getConnectionProperties() throws KyuubiConfFileParseException {
+    Properties props = new Properties();
+    String fileLocation = getFileLocation();
+    if (fileLocation == null) {
+      log.debug("Kyuubi default configuration file not found");
+      return props;
+    }
+    log.info("Using Kyuubi default configuration file at " + fileLocation);
+    File propsFile = new File(fileLocation);
+    try {
+      Map<String, String> propsMap = getPropertiesFromFile(propsFile);
+      StringBuilder hiveConf = new StringBuilder();
+      for (Entry<String, String> kv : propsMap.entrySet()) {

Review Comment:
   Load all configurations? Should we use a fixed prefix like `BEELINE_CONNECTION_PROPERTY_PREFIX`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1626856792


##########
docs/client/cli/kyuubi_beeline.md:
##########
@@ -0,0 +1,77 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements.  See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License.  You may obtain a copy of the License at
+-
+-   http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Kyuubi Beeline
+
+## What is Kyuubi Beeline
+
+Kyuubi Beeline is a Command Line Shell that uses JDBC driver to connect to Kyuubi server, HiveServer2 and other
+HiveServer2 compatible servers, like Impala, Flink SQL Gateway, to execute queries. Kyuubi Beeline is derived from
+[Hive Beeline](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Beeline%E2%80%93CommandLineShell),
+almost functionalities of Hive Beeline should be applicable to Kyuubi Beeline as well.
+
+Note: Kyuubi Beeline removes the support of "embedded mode" because it is coupled with Apache Hive implementation details,
+thus only "remote mode" can be used in Kyuubi Beeline. 
+
+##  Kyuubi Beeline properties files
+
+Kyuubi Beeline automatically constructs a connection URL based on some configuration properties in below four configuration files.
+
+- `beeline-site.xml`: By configuring named URLs within the `beeline-site.xml` file, users can employ the command `beeline -c <named_url>` to connect to predefined Kyuubi servers.
+
+- `beeline-hs2-connection.xml`: Users are able to set various properties in this file to change Kyuubi Beeline's behavior when connecting to Kyuubi servers. It provides user-specific connection properties, such as username, password, and other connection properties, such as SSL configuration properties, etc.
+
+- `kyuubi-defaults.conf`: The main configuration file for Apache Kyuubi. It contains global configuration properties in Kyuubi server like hostname and port information, Kerberos configuration properties.
+
+- `hive-site.xml`: It is one of the main configuration files for Apache Hive. It contains global configuration properties in Hive services like HS2 hostname and port information, Kerberos configuration properties, SSL properties, etc.
+
+## How users automatically connect to Kyuubi Servers via properties files
+
+Kyuubi Beeline looks for all configuration files sequentially once it gets started.
+
+1. `beeline-site.xml`: Kyuubi Beeline starts by retrieving the named URL from this file.
+
+2. `beeline-hs2-connection.xml`: This file is checked next, and any properties in this file take precedence over those in `beeline-site.xml`.
+
+3. `kyuubi-defaults.conf`: Properties from this file are loaded next. Values in this file may merge with the values of hiveconf from `beeline-hs2-connection.xml`.

Review Comment:
    They are both appended to the URL, but the latter one will take effect in case of duplicated properties. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1626857149


##########
docs/client/cli/kyuubi_beeline.md:
##########
@@ -0,0 +1,77 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements.  See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License.  You may obtain a copy of the License at
+-
+-   http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Kyuubi Beeline
+
+## What is Kyuubi Beeline
+
+Kyuubi Beeline is a Command Line Shell that uses JDBC driver to connect to Kyuubi server, HiveServer2 and other
+HiveServer2 compatible servers, like Impala, Flink SQL Gateway, to execute queries. Kyuubi Beeline is derived from
+[Hive Beeline](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Beeline%E2%80%93CommandLineShell),
+almost functionalities of Hive Beeline should be applicable to Kyuubi Beeline as well.
+
+Note: Kyuubi Beeline removes the support of "embedded mode" because it is coupled with Apache Hive implementation details,
+thus only "remote mode" can be used in Kyuubi Beeline. 
+
+##  Kyuubi Beeline properties files
+
+Kyuubi Beeline automatically constructs a connection URL based on some configuration properties in below four configuration files.
+
+- `beeline-site.xml`: By configuring named URLs within the `beeline-site.xml` file, users can employ the command `beeline -c <named_url>` to connect to predefined Kyuubi servers.
+
+- `beeline-hs2-connection.xml`: Users are able to set various properties in this file to change Kyuubi Beeline's behavior when connecting to Kyuubi servers. It provides user-specific connection properties, such as username, password, and other connection properties, such as SSL configuration properties, etc.
+
+- `kyuubi-defaults.conf`: The main configuration file for Apache Kyuubi. It contains global configuration properties in Kyuubi server like hostname and port information, Kerberos configuration properties.
+
+- `hive-site.xml`: It is one of the main configuration files for Apache Hive. It contains global configuration properties in Hive services like HS2 hostname and port information, Kerberos configuration properties, SSL properties, etc.
+
+## How users automatically connect to Kyuubi Servers via properties files
+
+Kyuubi Beeline looks for all configuration files sequentially once it gets started.
+
+1. `beeline-site.xml`: Kyuubi Beeline starts by retrieving the named URL from this file.
+
+2. `beeline-hs2-connection.xml`: This file is checked next, and any properties in this file take precedence over those in `beeline-site.xml`.
+
+3. `kyuubi-defaults.conf`: Properties from this file are loaded next. Values in this file may merge with the values of hiveconf from `beeline-hs2-connection.xml`.
+
+4. `hive-site.xml`: Any remaining connection properties are picked up from this file. If properties are present in multiple configuration files, the value from other files (e.g. `beeline-site.xml`, `beeline-hs2-connection.xml`, and `kyuubi-defaults.conf`) takes priority over `hive-site.xml`.
+
+Finally, Kyuubi Beeline generates the complete JDBC connection URL to be used when connecting to Kyuubi servers.
+
+### Tips
+
+Some settings may not be supported by the Kyuubi Beeline client due to compatibility issues.
+The configurations defined in `kyuubi-defaults.conf` might break compatibility with Kyuubi Beeline.
+Therefore, it is recommended to verify the Kyuubi configuration properties before starting Kyuubi Beeline to avoid potential issues.
+
+| Key                                                    | Comment                                           |
+|--------------------------------------------------------|---------------------------------------------------|
+| kyuubi.authentication.ldap.baseDN                      | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.binddn                      | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.customLDAPQuery             | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.groupDNPattern              | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.groupFilter                 | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.userDNPattern               | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.userFilter                  | comma-separated values are not supported          |
+| kyuubi.batch.conf.ignore.list                          | comma-separated values are not supported          |
+| kyuubi.frontend.protocols                              | comma-separated values are not supported          |
+| kyuubi.frontend.thrift.binary.ssl.disallowed.protocols | The value(REST) is an existing keyword in Beeline |

Review Comment:
   Got it. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "wForget (via GitHub)" <gi...@apache.org>.
wForget commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1580741158


##########
kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParser.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.hs2connection;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+import java.util.Map.Entry;
+import java.util.stream.Collectors;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class implements HS2ConnectionFileParser for the user-specific connection configuration file
+ * named kyuubi-defaults.conf. The class looks for this file in ${KYUUBI_CONF_DIR} in that order and
+ * uses the first file found in the above locations.
+ */
+public class KyuubiConfFileParser implements HS2ConnectionFileParser {
+
+  public static final String DEFAULT_CONNECTION_CONFIG_FILE_NAME = "kyuubi-defaults.conf";
+
+  private final List<String> locations = new ArrayList<>();
+  private static final Logger log = LoggerFactory.getLogger(KyuubiConfFileParser.class);
+
+  public KyuubiConfFileParser() {
+    if (System.getenv("KYUUBI_CONF_DIR") != null) {
+      locations.add(
+          System.getenv("KYUUBI_CONF_DIR") + File.separator + DEFAULT_CONNECTION_CONFIG_FILE_NAME);
+    }
+  }
+
+  @VisibleForTesting
+  KyuubiConfFileParser(List<String> testLocations) {
+    if (testLocations == null) {
+      return;
+    }
+    locations.addAll(testLocations);
+  }
+
+  @Override
+  public Properties getConnectionProperties() throws KyuubiConfFileParseException {
+    Properties props = new Properties();
+    String fileLocation = getFileLocation();
+    if (fileLocation == null) {
+      log.debug("Kyuubi default configuration file not found");
+      return props;
+    }
+    log.info("Using Kyuubi default configuration file at " + fileLocation);
+    File propsFile = new File(fileLocation);
+    try {
+      Map<String, String> propsMap = getPropertiesFromFile(propsFile);
+      StringBuilder hiveConf = new StringBuilder();
+      for (Entry<String, String> kv : propsMap.entrySet()) {

Review Comment:
   > beeline-site.xml takes responsibility for that, I don't think we should propagate it to kyuubi-defaults.conf
   
   In addition to kyuubi server `host` and `port`, what other valid configurations can we infer in the current `kyuubi-defaults.conf`?
   
   > Why do we need HOST_PROPERTY_KEY?
   
   Pass server host and port configured in kyuubi-defaults.conf to jdbc url.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1626063393


##########
docs/client/cli/kyuubi_beeline.md:
##########
@@ -0,0 +1,77 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements.  See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License.  You may obtain a copy of the License at
+-
+-   http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Kyuubi Beeline
+
+## What is Kyuubi Beeline
+
+Kyuubi Beeline is a Command Line Shell that uses JDBC driver to connect to Kyuubi server, HiveServer2 and other
+HiveServer2 compatible servers, like Impala, Flink SQL Gateway, to execute queries. Kyuubi Beeline is derived from
+[Hive Beeline](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Beeline%E2%80%93CommandLineShell),
+almost functionalities of Hive Beeline should be applicable to Kyuubi Beeline as well.
+
+Note: Kyuubi Beeline removes the support of "embedded mode" because it is coupled with Apache Hive implementation details,
+thus only "remote mode" can be used in Kyuubi Beeline. 
+
+##  Kyuubi Beeline properties files
+
+Kyuubi Beeline automatically constructs a connection URL based on some configuration properties in below four configuration files.
+
+- `beeline-site.xml`: By configuring named URLs within the `beeline-site.xml` file, users can employ the command `beeline -c <named_url>` to connect to predefined Kyuubi servers.
+
+- `beeline-hs2-connection.xml`: Users are able to set various properties in this file to change Kyuubi Beeline's behavior when connecting to Kyuubi servers. It provides user-specific connection properties, such as username, password, and other connection properties, such as SSL configuration properties, etc.
+
+- `kyuubi-defaults.conf`: The main configuration file for Apache Kyuubi. It contains global configuration properties in Kyuubi server like hostname and port information, Kerberos configuration properties.
+
+- `hive-site.xml`: It is one of the main configuration files for Apache Hive. It contains global configuration properties in Hive services like HS2 hostname and port information, Kerberos configuration properties, SSL properties, etc.
+
+## How users automatically connect to Kyuubi Servers via properties files
+
+Kyuubi Beeline looks for all configuration files sequentially once it gets started.
+
+1. `beeline-site.xml`: Kyuubi Beeline starts by retrieving the named URL from this file.
+
+2. `beeline-hs2-connection.xml`: This file is checked next, and any properties in this file take precedence over those in `beeline-site.xml`.
+
+3. `kyuubi-defaults.conf`: Properties from this file are loaded next. Values in this file may merge with the values of hiveconf from `beeline-hs2-connection.xml`.
+
+4. `hive-site.xml`: Any remaining connection properties are picked up from this file. If properties are present in multiple configuration files, the value from other files (e.g. `beeline-site.xml`, `beeline-hs2-connection.xml`, and `kyuubi-defaults.conf`) takes priority over `hive-site.xml`.
+
+Finally, Kyuubi Beeline generates the complete JDBC connection URL to be used when connecting to Kyuubi servers.
+
+### Tips
+
+Some settings may not be supported by the Kyuubi Beeline client due to compatibility issues.
+The configurations defined in `kyuubi-defaults.conf` might break compatibility with Kyuubi Beeline.
+Therefore, it is recommended to verify the Kyuubi configuration properties before starting Kyuubi Beeline to avoid potential issues.
+
+| Key                                                    | Comment                                           |
+|--------------------------------------------------------|---------------------------------------------------|
+| kyuubi.authentication.ldap.baseDN                      | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.binddn                      | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.customLDAPQuery             | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.groupDNPattern              | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.groupFilter                 | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.userDNPattern               | comma-separated values are not supported          |
+| kyuubi.authentication.ldap.userFilter                  | comma-separated values are not supported          |
+| kyuubi.batch.conf.ignore.list                          | comma-separated values are not supported          |
+| kyuubi.frontend.protocols                              | comma-separated values are not supported          |
+| kyuubi.frontend.thrift.binary.ssl.disallowed.protocols | The value(REST) is an existing keyword in Beeline |

Review Comment:
   let's use word "Kyuubi Beeline" to distinguish with Hive Beeline
   ```suggestion
   | kyuubi.frontend.thrift.binary.ssl.disallowed.protocols | The value(REST) is an existing keyword in Kyuubi Beeline |
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#issuecomment-2104200216

   ## [Codecov](https://app.codecov.io/gh/apache/kyuubi/pull/6339?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `40.86022%` with `55 lines` in your changes are missing coverage. Please review.
   > Project coverage is 58.41%. Comparing base [(`8edcb00`)](https://app.codecov.io/gh/apache/kyuubi/commit/8edcb005ee8c863d5fd070e276a05d9c0c30d018?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`b9854a6`)](https://app.codecov.io/gh/apache/kyuubi/pull/6339?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 23 commits behind head on master.
   
   | [Files](https://app.codecov.io/gh/apache/kyuubi/pull/6339?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [.../beeline/hs2connection/HS2ConnectionFileUtils.java](https://app.codecov.io/gh/apache/kyuubi/pull/6339?src=pr&el=tree&filepath=kyuubi-hive-beeline%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhive%2Fbeeline%2Fhs2connection%2FHS2ConnectionFileUtils.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLWhpdmUtYmVlbGluZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGl2ZS9iZWVsaW5lL2hzMmNvbm5lY3Rpb24vSFMyQ29ubmVjdGlvbkZpbGVVdGlscy5qYXZh) | 0.00% | [27 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6339?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ve/beeline/hs2connection/KyuubiConfFileParser.java](https://app.codecov.io/gh/apache/kyuubi/pull/6339?src=pr&el=tree&filepath=kyuubi-hive-beeline%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhive%2Fbeeline%2Fhs2connection%2FKyuubiConfFileParser.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLWhpdmUtYmVlbGluZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGl2ZS9iZWVsaW5lL2hzMmNvbm5lY3Rpb24vS3l1dWJpQ29uZkZpbGVQYXJzZXIuamF2YQ==) | 64.28% | [13 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6339?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...src/main/java/org/apache/hive/beeline/BeeLine.java](https://app.codecov.io/gh/apache/kyuubi/pull/6339?src=pr&el=tree&filepath=kyuubi-hive-beeline%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhive%2Fbeeline%2FBeeLine.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLWhpdmUtYmVlbGluZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGl2ZS9iZWVsaW5lL0JlZUxpbmUuamF2YQ==) | 33.33% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6339?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...ne/hs2connection/KyuubiConfFileParseException.java](https://app.codecov.io/gh/apache/kyuubi/pull/6339?src=pr&el=tree&filepath=kyuubi-hive-beeline%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhive%2Fbeeline%2Fhs2connection%2FKyuubiConfFileParseException.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-a3l1dWJpLWhpdmUtYmVlbGluZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvaGl2ZS9iZWVsaW5lL2hzMmNvbm5lY3Rpb24vS3l1dWJpQ29uZkZpbGVQYXJzZUV4Y2VwdGlvbi5qYXZh) | 0.00% | [4 Missing :warning: ](https://app.codecov.io/gh/apache/kyuubi/pull/6339?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #6339      +/-   ##
   ============================================
   - Coverage     58.50%   58.41%   -0.09%     
     Complexity       24       24              
   ============================================
     Files           653      655       +2     
     Lines         39775    40052     +277     
     Branches       5473     5505      +32     
   ============================================
   + Hits          23269    23397     +128     
   - Misses        14022    14146     +124     
   - Partials       2484     2509      +25     
   ```
   
   
   
   </details>
   
   [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/kyuubi/pull/6339?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1626057776


##########
docs/client/cli/kyuubi_beeline.md:
##########
@@ -0,0 +1,77 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements.  See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License.  You may obtain a copy of the License at
+-
+-   http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Kyuubi Beeline
+
+## What is Kyuubi Beeline
+
+Kyuubi Beeline is a Command Line Shell that uses JDBC driver to connect to Kyuubi server, HiveServer2 and other
+HiveServer2 compatible servers, like Impala, Flink SQL Gateway, to execute queries. Kyuubi Beeline is derived from
+[Hive Beeline](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Beeline%E2%80%93CommandLineShell),
+almost functionalities of Hive Beeline should be applicable to Kyuubi Beeline as well.
+
+Note: Kyuubi Beeline removes the support of "embedded mode" because it is coupled with Apache Hive implementation details,
+thus only "remote mode" can be used in Kyuubi Beeline. 
+
+##  Kyuubi Beeline properties files
+
+Kyuubi Beeline automatically constructs a connection URL based on some configuration properties in below four configuration files.
+
+- `beeline-site.xml`: By configuring named URLs within the `beeline-site.xml` file, users can employ the command `beeline -c <named_url>` to connect to predefined Kyuubi servers.
+
+- `beeline-hs2-connection.xml`: Users are able to set various properties in this file to change Kyuubi Beeline's behavior when connecting to Kyuubi servers. It provides user-specific connection properties, such as username, password, and other connection properties, such as SSL configuration properties, etc.
+
+- `kyuubi-defaults.conf`: The main configuration file for Apache Kyuubi. It contains global configuration properties in Kyuubi server like hostname and port information, Kerberos configuration properties.
+
+- `hive-site.xml`: It is one of the main configuration files for Apache Hive. It contains global configuration properties in Hive services like HS2 hostname and port information, Kerberos configuration properties, SSL properties, etc.
+
+## How users automatically connect to Kyuubi Servers via properties files
+
+Kyuubi Beeline looks for all configuration files sequentially once it gets started.
+
+1. `beeline-site.xml`: Kyuubi Beeline starts by retrieving the named URL from this file.
+
+2. `beeline-hs2-connection.xml`: This file is checked next, and any properties in this file take precedence over those in `beeline-site.xml`.
+
+3. `kyuubi-defaults.conf`: Properties from this file are loaded next. Values in this file may merge with the values of hiveconf from `beeline-hs2-connection.xml`.

Review Comment:
   the "merge" behavior is confusing here, which one will take effect for duplicated properties?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1596376060


##########
kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/BeeLine.java:
##########
@@ -1012,6 +1010,14 @@ private String getDefaultConnectionUrl(CommandLine cl) throws BeelineConfFilePar
       userConnectionProperties = userHS2ConnFileParser.getConnectionProperties();
     }
 
+    if (kyuubiConfFileParser.configExists()) {
+      // get the connection properties from user specific config file
+      Properties properties = kyuubiConfFileParser.getConnectionProperties();
+      for (String key : properties.stringPropertyNames()) {
+        userConnectionProperties.setProperty(key, properties.getProperty(key));

Review Comment:
   done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#issuecomment-2146455713

   thanks for update, will take a look soon


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1626065333


##########
docs/client/cli/kyuubi_beeline.md:
##########
@@ -0,0 +1,77 @@
+<!--
+- Licensed to the Apache Software Foundation (ASF) under one or more
+- contributor license agreements.  See the NOTICE file distributed with
+- this work for additional information regarding copyright ownership.
+- The ASF licenses this file to You under the Apache License, Version 2.0
+- (the "License"); you may not use this file except in compliance with
+- the License.  You may obtain a copy of the License at
+-
+-   http://www.apache.org/licenses/LICENSE-2.0
+-
+- Unless required by applicable law or agreed to in writing, software
+- distributed under the License is distributed on an "AS IS" BASIS,
+- WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+- See the License for the specific language governing permissions and
+- limitations under the License.
+-->
+
+# Kyuubi Beeline
+
+## What is Kyuubi Beeline
+
+Kyuubi Beeline is a Command Line Shell that uses JDBC driver to connect to Kyuubi server, HiveServer2 and other
+HiveServer2 compatible servers, like Impala, Flink SQL Gateway, to execute queries. Kyuubi Beeline is derived from
+[Hive Beeline](https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-Beeline%E2%80%93CommandLineShell),
+almost functionalities of Hive Beeline should be applicable to Kyuubi Beeline as well.
+
+Note: Kyuubi Beeline removes the support of "embedded mode" because it is coupled with Apache Hive implementation details,
+thus only "remote mode" can be used in Kyuubi Beeline. 
+
+##  Kyuubi Beeline properties files

Review Comment:
   ```suggestion
   ## Kyuubi Beeline properties files
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#issuecomment-2147595658

   @zhaohehuhu before looking into the code, I read and modified `kyuubi_beeline.md`, almost good, there is still one question on the docs, could you please clarify it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1580690271


##########
kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParser.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.hs2connection;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+import java.util.Map.Entry;
+import java.util.stream.Collectors;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class implements HS2ConnectionFileParser for the user-specific connection configuration file
+ * named kyuubi-defaults.conf. The class looks for this file in ${KYUUBI_CONF_DIR} in that order and
+ * uses the first file found in the above locations.
+ */
+public class KyuubiConfFileParser implements HS2ConnectionFileParser {
+
+  public static final String DEFAULT_CONNECTION_CONFIG_FILE_NAME = "kyuubi-defaults.conf";
+
+  private final List<String> locations = new ArrayList<>();
+  private static final Logger log = LoggerFactory.getLogger(KyuubiConfFileParser.class);
+
+  public KyuubiConfFileParser() {
+    if (System.getenv("KYUUBI_CONF_DIR") != null) {
+      locations.add(
+          System.getenv("KYUUBI_CONF_DIR") + File.separator + DEFAULT_CONNECTION_CONFIG_FILE_NAME);
+    }
+  }
+
+  @VisibleForTesting
+  KyuubiConfFileParser(List<String> testLocations) {
+    if (testLocations == null) {
+      return;
+    }
+    locations.addAll(testLocations);
+  }
+
+  @Override
+  public Properties getConnectionProperties() throws KyuubiConfFileParseException {
+    Properties props = new Properties();
+    String fileLocation = getFileLocation();
+    if (fileLocation == null) {
+      log.debug("Kyuubi default configuration file not found");
+      return props;
+    }
+    log.info("Using Kyuubi default configuration file at " + fileLocation);
+    File propsFile = new File(fileLocation);
+    try {
+      Map<String, String> propsMap = getPropertiesFromFile(propsFile);
+      StringBuilder hiveConf = new StringBuilder();
+      for (Entry<String, String> kv : propsMap.entrySet()) {

Review Comment:
   the intention is infer Kyuubi connection info from the existing `kyuubi-defaults.conf`, instead supporting configuring beeline specific configurations inside `kyuubi-defaults.conf` 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#discussion_r1580726210


##########
kyuubi-hive-beeline/src/main/java/org/apache/hive/beeline/hs2connection/KyuubiConfFileParser.java:
##########
@@ -0,0 +1,125 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hive.beeline.hs2connection;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStreamReader;
+import java.nio.charset.StandardCharsets;
+import java.util.*;
+import java.util.Map.Entry;
+import java.util.stream.Collectors;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * This class implements HS2ConnectionFileParser for the user-specific connection configuration file
+ * named kyuubi-defaults.conf. The class looks for this file in ${KYUUBI_CONF_DIR} in that order and
+ * uses the first file found in the above locations.
+ */
+public class KyuubiConfFileParser implements HS2ConnectionFileParser {
+
+  public static final String DEFAULT_CONNECTION_CONFIG_FILE_NAME = "kyuubi-defaults.conf";
+
+  private final List<String> locations = new ArrayList<>();
+  private static final Logger log = LoggerFactory.getLogger(KyuubiConfFileParser.class);
+
+  public KyuubiConfFileParser() {
+    if (System.getenv("KYUUBI_CONF_DIR") != null) {
+      locations.add(
+          System.getenv("KYUUBI_CONF_DIR") + File.separator + DEFAULT_CONNECTION_CONFIG_FILE_NAME);
+    }
+  }
+
+  @VisibleForTesting
+  KyuubiConfFileParser(List<String> testLocations) {
+    if (testLocations == null) {
+      return;
+    }
+    locations.addAll(testLocations);
+  }
+
+  @Override
+  public Properties getConnectionProperties() throws KyuubiConfFileParseException {
+    Properties props = new Properties();
+    String fileLocation = getFileLocation();
+    if (fileLocation == null) {
+      log.debug("Kyuubi default configuration file not found");
+      return props;
+    }
+    log.info("Using Kyuubi default configuration file at " + fileLocation);
+    File propsFile = new File(fileLocation);
+    try {
+      Map<String, String> propsMap = getPropertiesFromFile(propsFile);
+      StringBuilder hiveConf = new StringBuilder();
+      for (Entry<String, String> kv : propsMap.entrySet()) {

Review Comment:
   > We can define a prefix to determine whether it is a client config
   
   `beeline-site.xml` takes responsibility for that, I don't think we should propagate it to `kyuubi-defaults.conf`
   
   Why do we need `HOST_PROPERTY_KEY`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#issuecomment-2147597434

   Also cc @wForget, do you think the behavior described in the docs make sense?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#issuecomment-2146454355

   plz help review again @pan3793 @wForget. Thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


Re: [PR] kyuubi-beeline auto constructs JDBC URL from kyuubi-defaults.conf [kyuubi]

Posted by "zhaohehuhu (via GitHub)" <gi...@apache.org>.
zhaohehuhu commented on PR #6339:
URL: https://github.com/apache/kyuubi/pull/6339#issuecomment-2146483430

   > thanks for update, will take a look soon
   
   Appreciate it. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org