You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/09/05 08:41:39 UTC

[GitHub] [incubator-kyuubi] bowenliang123 opened a new pull request, #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

bowenliang123 opened a new pull request, #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407

   <!--
   Thanks for sending a pull request!
   
   Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/CONTRIBUTING.html
     2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
   -->
   
   ### _Why are the changes needed?_
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you add a feature, you can talk about the use case of it.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   close #3406.
   
   Add PySpark client docs.
   
   ### _How was this patch tested?_
   - [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
   
   - [ ] Add screenshots for manual tests if appropriate
   
   - [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#discussion_r979263326


##########
docs/client/python/pyspark.md:
##########
@@ -0,0 +1,122 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements.  See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License.  You may obtain a copy of the License at
+ -
+ -   http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+
+# PySpark
+
+[PySpark](https://spark.apache.org/docs/latest/api/python/index.html) is an interface for Apache Spark in Python. Kyuubi can be used as JDBC source in PySpark.
+
+## Requirements
+PySpark works with Python 3.7 and above.
+
+Install PySpark with Spark SQL and optional pandas on Spark using PyPI as follows:
+
+```shell
+pip install pyspark 'pyspark[sql]' 'pyspark[pandas_on_spark]'
+```
+
+For installation using Conda or manually downloading, please refer to [PySpark installation](https://spark.apache.org/docs/latest/api/python/getting_started/install.html).
+
+## Preperation
+
+
+### Prepare JDBC driver 
+Prepare JDBC driver jar file. Supported Hive compatible JDBC Driver as below:
+
+| Driver     | Driver Class Name | Remarks|
+| ---------- | ----------------- | ----- |
+| Kyuubi Hive Driver ([doc](../jdbc/kyuubi_jdbc.html))| org.apache.kyuubi.jdbc.KyuubiHiveDriver |  Compile for the driver on master branch, as [KYUUBI #3484](https://github.com/apache/incubator-kyuubi/pull/3485) required by Spark JDBC source not yet included in released version.
+| Hive Driver ([doc](../jdbc/hive_jdbc.html))| org.apache.hive.jdbc.HiveDriver | 
+
+Refer to docs of the driver and prepare the JDBC driver jar file.
+
+### Prepare JDBC Hive Dialect extension
+
+Hive Dialect support is requried by Spark for wraping sql correctly and sending to JDBC driver. Kyuubi provides a JDBC dialect extension with auto regiested Hive Daliect support for Spark. Follow the instrunctions in [Hive Dialect Support](../../engines/spark/jdbc-dialect.html) to prepare the plugin jar file `kyuubi-extension-spark-jdbc-dialect_-*.jar`.
+
+### Including jars of JDBC driver and Hive Dialect extention
+
+Choose one of following ways to include jar files to Spark.
+
+- Put the jar file of JDBC driver and Hive Dialect to `$SPARK_HOME/jars` directory to make it visible for the classpath of PySpark. And adding `spark.sql.extensions = org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension` to `$SPARK_HOME/conf/spark_defaults.conf.`
+
+- With spark's start shell, include JDBC driver when you submit the application with `--packages`, and the Hive Dialect plugins with `--jars`
+
+```
+$SPARK_HOME/bin/pyspark --py-files PY_FILES \
+--packages org.apache.hive:hive-jdbc:x.y.z \
+--jars /path/kyuubi-extension-spark-jdbc-dialect_-*.jar 
+```
+
+- Setting jars and config with SparkSession builder
+
+```python
+from pyspark.sql import SparkSession
+
+spark = SparkSession.builder \
+        .config("spark.jars", "/path/hive-jdbc-x.y.z.jar,/path/kyuubi-extension-spark-jdbc-dialect_-*.jar") \
+        .config("spark.sql.extensions", "org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension") \
+        .getOrCreate()
+```
+
+
+
+## Usage
+
+For further information about PySpark JDBC usage and options, please refer to Spark's [JDBC To Other Databases](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html).
+
+### Reading and Writing via JDBC data source
+
+```python
+# Loading data from Kyuubi via HiveDriver as JDBC source
+jdbcDF = spark.read \
+  .format("jdbc") \
+  .options(driver="org.apache.hive.jdbc.HiveDriver",

Review Comment:
   same obstacle as above.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#issuecomment-1236785502

   # [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/3407?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#3407](https://codecov.io/gh/apache/incubator-kyuubi/pull/3407?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (692197e) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/b3ecaef35e92058ffb3febf8ffbdbff3f9b4d76d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b3ecaef) will **decrease** coverage by `0.01%`.
   > The diff coverage is `n/a`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #3407      +/-   ##
   ============================================
   - Coverage     51.47%   51.45%   -0.02%     
     Complexity       13       13              
   ============================================
     Files           480      480              
     Lines         26664    26664              
     Branches       3728     3728              
   ============================================
   - Hits          13724    13719       -5     
   - Misses        11598    11601       +3     
   - Partials       1342     1344       +2     
   ```
   
   
   | [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/3407?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...pache/kyuubi/engine/YarnApplicationOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvWWFybkFwcGxpY2F0aW9uT3BlcmF0aW9uLnNjYWxh) | `66.66% <0.00%> (-1.86%)` | :arrow_down: |
   | [...ache/kyuubi/operation/KyuubiOperationManager.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uTWFuYWdlci5zY2FsYQ==) | `80.82% <0.00%> (-1.37%)` | :arrow_down: |
   | [...ain/scala/org/apache/kyuubi/engine/EngineRef.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvRW5naW5lUmVmLnNjYWxh) | `72.26% <0.00%> (-0.85%)` | :arrow_down: |
   | [...yuubi/server/metadata/jdbc/JDBCMetadataStore.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvbWV0YWRhdGEvamRiYy9KREJDTWV0YWRhdGFTdG9yZS5zY2FsYQ==) | `89.27% <0.00%> (-0.70%)` | :arrow_down: |
   | [...a/org/apache/kyuubi/service/TFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2aWNlL1RGcm9udGVuZFNlcnZpY2Uuc2NhbGE=) | `91.17% <0.00%> (-0.30%)` | :arrow_down: |
   | [...g/apache/kyuubi/operation/BatchJobSubmission.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vQmF0Y2hKb2JTdWJtaXNzaW9uLnNjYWxh) | `74.64% <0.00%> (ø)` | |
   | [...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3407/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9lbmdpbmUvUHJvY0J1aWxkZXIuc2NhbGE=) | `83.22% <0.00%> (+0.62%)` | :arrow_up: |
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#discussion_r979263131


##########
docs/client/python/pyspark.md:
##########
@@ -0,0 +1,122 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements.  See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License.  You may obtain a copy of the License at
+ -
+ -   http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+
+# PySpark
+
+[PySpark](https://spark.apache.org/docs/latest/api/python/index.html) is an interface for Apache Spark in Python. Kyuubi can be used as JDBC source in PySpark.
+
+## Requirements
+PySpark works with Python 3.7 and above.
+
+Install PySpark with Spark SQL and optional pandas on Spark using PyPI as follows:
+
+```shell
+pip install pyspark 'pyspark[sql]' 'pyspark[pandas_on_spark]'
+```
+
+For installation using Conda or manually downloading, please refer to [PySpark installation](https://spark.apache.org/docs/latest/api/python/getting_started/install.html).
+
+## Preperation
+
+
+### Prepare JDBC driver 
+Prepare JDBC driver jar file. Supported Hive compatible JDBC Driver as below:
+
+| Driver     | Driver Class Name | Remarks|
+| ---------- | ----------------- | ----- |
+| Kyuubi Hive Driver ([doc](../jdbc/kyuubi_jdbc.html))| org.apache.kyuubi.jdbc.KyuubiHiveDriver |  Compile for the driver on master branch, as [KYUUBI #3484](https://github.com/apache/incubator-kyuubi/pull/3485) required by Spark JDBC source not yet included in released version.
+| Hive Driver ([doc](../jdbc/hive_jdbc.html))| org.apache.hive.jdbc.HiveDriver | 
+
+Refer to docs of the driver and prepare the JDBC driver jar file.
+
+### Prepare JDBC Hive Dialect extension
+
+Hive Dialect support is requried by Spark for wraping sql correctly and sending to JDBC driver. Kyuubi provides a JDBC dialect extension with auto regiested Hive Daliect support for Spark. Follow the instrunctions in [Hive Dialect Support](../../engines/spark/jdbc-dialect.html) to prepare the plugin jar file `kyuubi-extension-spark-jdbc-dialect_-*.jar`.
+
+### Including jars of JDBC driver and Hive Dialect extention
+
+Choose one of following ways to include jar files to Spark.
+
+- Put the jar file of JDBC driver and Hive Dialect to `$SPARK_HOME/jars` directory to make it visible for the classpath of PySpark. And adding `spark.sql.extensions = org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension` to `$SPARK_HOME/conf/spark_defaults.conf.`
+
+- With spark's start shell, include JDBC driver when you submit the application with `--packages`, and the Hive Dialect plugins with `--jars`
+
+```
+$SPARK_HOME/bin/pyspark --py-files PY_FILES \
+--packages org.apache.hive:hive-jdbc:x.y.z \

Review Comment:
   Negative for right now. As `isSigned` implementation of KyuubiHiveDriver #3485 not yet released in latest 1.6.0 release.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#issuecomment-1256279500

   cc @pan3793 , please have a check if some time available.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on a diff in pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on code in PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#discussion_r979263627


##########
docs/client/python/pyspark.md:
##########
@@ -0,0 +1,122 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements.  See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License.  You may obtain a copy of the License at
+ -
+ -   http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+
+# PySpark
+
+[PySpark](https://spark.apache.org/docs/latest/api/python/index.html) is an interface for Apache Spark in Python. Kyuubi can be used as JDBC source in PySpark.
+
+## Requirements
+PySpark works with Python 3.7 and above.
+
+Install PySpark with Spark SQL and optional pandas on Spark using PyPI as follows:
+
+```shell
+pip install pyspark 'pyspark[sql]' 'pyspark[pandas_on_spark]'
+```
+
+For installation using Conda or manually downloading, please refer to [PySpark installation](https://spark.apache.org/docs/latest/api/python/getting_started/install.html).
+
+## Preperation
+
+
+### Prepare JDBC driver 
+Prepare JDBC driver jar file. Supported Hive compatible JDBC Driver as below:
+
+| Driver     | Driver Class Name | Remarks|
+| ---------- | ----------------- | ----- |
+| Kyuubi Hive Driver ([doc](../jdbc/kyuubi_jdbc.html))| org.apache.kyuubi.jdbc.KyuubiHiveDriver |  Compile for the driver on master branch, as [KYUUBI #3484](https://github.com/apache/incubator-kyuubi/pull/3485) required by Spark JDBC source not yet included in released version.
+| Hive Driver ([doc](../jdbc/hive_jdbc.html))| org.apache.hive.jdbc.HiveDriver | 
+
+Refer to docs of the driver and prepare the JDBC driver jar file.
+
+### Prepare JDBC Hive Dialect extension
+
+Hive Dialect support is requried by Spark for wraping sql correctly and sending to JDBC driver. Kyuubi provides a JDBC dialect extension with auto regiested Hive Daliect support for Spark. Follow the instrunctions in [Hive Dialect Support](../../engines/spark/jdbc-dialect.html) to prepare the plugin jar file `kyuubi-extension-spark-jdbc-dialect_-*.jar`.
+
+### Including jars of JDBC driver and Hive Dialect extention
+
+Choose one of following ways to include jar files to Spark.
+
+- Put the jar file of JDBC driver and Hive Dialect to `$SPARK_HOME/jars` directory to make it visible for the classpath of PySpark. And adding `spark.sql.extensions = org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension` to `$SPARK_HOME/conf/spark_defaults.conf.`
+
+- With spark's start shell, include JDBC driver when you submit the application with `--packages`, and the Hive Dialect plugins with `--jars`
+
+```
+$SPARK_HOME/bin/pyspark --py-files PY_FILES \
+--packages org.apache.hive:hive-jdbc:x.y.z \
+--jars /path/kyuubi-extension-spark-jdbc-dialect_-*.jar 
+```
+
+- Setting jars and config with SparkSession builder
+
+```python
+from pyspark.sql import SparkSession
+
+spark = SparkSession.builder \
+        .config("spark.jars", "/path/hive-jdbc-x.y.z.jar,/path/kyuubi-extension-spark-jdbc-dialect_-*.jar") \
+        .config("spark.sql.extensions", "org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension") \
+        .getOrCreate()
+```
+
+
+
+## Usage
+
+For further information about PySpark JDBC usage and options, please refer to Spark's [JDBC To Other Databases](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html).
+
+### Reading and Writing via JDBC data source
+
+```python
+# Loading data from Kyuubi via HiveDriver as JDBC source
+jdbcDF = spark.read \
+  .format("jdbc") \
+  .options(driver="org.apache.hive.jdbc.HiveDriver",
+           url="jdbc:hive2://kyuubi_server_ip:port",
+           user="user",
+           password="password",
+           query="select * from testdb.src_table"
+           ) \
+  .load()
+
+
+# Saving data to Kyuubi via HiveDriver as JDBC source
+jdbcDF.write \
+    .format("jdbc") \
+    .options(driver="org.apache.hive.jdbc.HiveDriver",
+             url="jdbc:hive2://kyuubi_server_ip:port",
+           user="user",
+           password="password",
+           dbtable="testdb.tgt_table"
+           ) \
+    .save()
+```
+
+
+### Use PySpark with Pandas
+From PySpark 3.2.0, PySpark supports pandas API on Spark which allows you to scale your pandas workload out.
+
+Pandas-on-Spark DataFrame and Spark DataFrame are virtually interchangeable.More instructions in [From/to pandas and PySpark DataFrames](https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/pandas_pyspark.html#pyspark).

Review Comment:
   sorry for Misdirection. A blank is now added before `More`. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#issuecomment-1256995971

   Thanks, merging to master/1.6


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] pan3793 closed pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
pan3793 closed pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs
URL: https://github.com/apache/incubator-kyuubi/pull/3407


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#discussion_r979193408


##########
docs/client/python/pyspark.md:
##########
@@ -0,0 +1,122 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements.  See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License.  You may obtain a copy of the License at
+ -
+ -   http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+
+# PySpark
+
+[PySpark](https://spark.apache.org/docs/latest/api/python/index.html) is an interface for Apache Spark in Python. Kyuubi can be used as JDBC source in PySpark.
+
+## Requirements
+PySpark works with Python 3.7 and above.
+
+Install PySpark with Spark SQL and optional pandas on Spark using PyPI as follows:
+
+```shell
+pip install pyspark 'pyspark[sql]' 'pyspark[pandas_on_spark]'
+```
+
+For installation using Conda or manually downloading, please refer to [PySpark installation](https://spark.apache.org/docs/latest/api/python/getting_started/install.html).
+
+## Preperation
+
+
+### Prepare JDBC driver 
+Prepare JDBC driver jar file. Supported Hive compatible JDBC Driver as below:
+
+| Driver     | Driver Class Name | Remarks|
+| ---------- | ----------------- | ----- |
+| Kyuubi Hive Driver ([doc](../jdbc/kyuubi_jdbc.html))| org.apache.kyuubi.jdbc.KyuubiHiveDriver |  Compile for the driver on master branch, as [KYUUBI #3484](https://github.com/apache/incubator-kyuubi/pull/3485) required by Spark JDBC source not yet included in released version.
+| Hive Driver ([doc](../jdbc/hive_jdbc.html))| org.apache.hive.jdbc.HiveDriver | 
+
+Refer to docs of the driver and prepare the JDBC driver jar file.
+
+### Prepare JDBC Hive Dialect extension
+
+Hive Dialect support is requried by Spark for wraping sql correctly and sending to JDBC driver. Kyuubi provides a JDBC dialect extension with auto regiested Hive Daliect support for Spark. Follow the instrunctions in [Hive Dialect Support](../../engines/spark/jdbc-dialect.html) to prepare the plugin jar file `kyuubi-extension-spark-jdbc-dialect_-*.jar`.
+
+### Including jars of JDBC driver and Hive Dialect extention
+
+Choose one of following ways to include jar files to Spark.
+
+- Put the jar file of JDBC driver and Hive Dialect to `$SPARK_HOME/jars` directory to make it visible for the classpath of PySpark. And adding `spark.sql.extensions = org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension` to `$SPARK_HOME/conf/spark_defaults.conf.`
+
+- With spark's start shell, include JDBC driver when you submit the application with `--packages`, and the Hive Dialect plugins with `--jars`
+
+```
+$SPARK_HOME/bin/pyspark --py-files PY_FILES \
+--packages org.apache.hive:hive-jdbc:x.y.z \
+--jars /path/kyuubi-extension-spark-jdbc-dialect_-*.jar 
+```
+
+- Setting jars and config with SparkSession builder
+
+```python
+from pyspark.sql import SparkSession
+
+spark = SparkSession.builder \
+        .config("spark.jars", "/path/hive-jdbc-x.y.z.jar,/path/kyuubi-extension-spark-jdbc-dialect_-*.jar") \
+        .config("spark.sql.extensions", "org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension") \
+        .getOrCreate()
+```
+
+
+
+## Usage
+
+For further information about PySpark JDBC usage and options, please refer to Spark's [JDBC To Other Databases](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html).
+
+### Reading and Writing via JDBC data source
+
+```python
+# Loading data from Kyuubi via HiveDriver as JDBC source
+jdbcDF = spark.read \
+  .format("jdbc") \
+  .options(driver="org.apache.hive.jdbc.HiveDriver",
+           url="jdbc:hive2://kyuubi_server_ip:port",
+           user="user",
+           password="password",
+           query="select * from testdb.src_table"
+           ) \
+  .load()
+
+
+# Saving data to Kyuubi via HiveDriver as JDBC source
+jdbcDF.write \
+    .format("jdbc") \
+    .options(driver="org.apache.hive.jdbc.HiveDriver",
+             url="jdbc:hive2://kyuubi_server_ip:port",
+           user="user",
+           password="password",
+           dbtable="testdb.tgt_table"
+           ) \
+    .save()
+```
+
+
+### Use PySpark with Pandas
+From PySpark 3.2.0, PySpark supports pandas API on Spark which allows you to scale your pandas workload out.
+
+Pandas-on-Spark DataFrame and Spark DataFrame are virtually interchangeable.More instructions in [From/to pandas and PySpark DataFrames](https://spark.apache.org/docs/latest/api/python/user_guide/pandas_on_spark/pandas_pyspark.html#pyspark).

Review Comment:
   quote interchangeable.More by `



##########
docs/client/python/pyspark.md:
##########
@@ -0,0 +1,122 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements.  See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License.  You may obtain a copy of the License at
+ -
+ -   http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+
+# PySpark
+
+[PySpark](https://spark.apache.org/docs/latest/api/python/index.html) is an interface for Apache Spark in Python. Kyuubi can be used as JDBC source in PySpark.
+
+## Requirements
+PySpark works with Python 3.7 and above.
+
+Install PySpark with Spark SQL and optional pandas on Spark using PyPI as follows:
+
+```shell
+pip install pyspark 'pyspark[sql]' 'pyspark[pandas_on_spark]'
+```
+
+For installation using Conda or manually downloading, please refer to [PySpark installation](https://spark.apache.org/docs/latest/api/python/getting_started/install.html).
+
+## Preperation
+
+
+### Prepare JDBC driver 
+Prepare JDBC driver jar file. Supported Hive compatible JDBC Driver as below:
+
+| Driver     | Driver Class Name | Remarks|
+| ---------- | ----------------- | ----- |
+| Kyuubi Hive Driver ([doc](../jdbc/kyuubi_jdbc.html))| org.apache.kyuubi.jdbc.KyuubiHiveDriver |  Compile for the driver on master branch, as [KYUUBI #3484](https://github.com/apache/incubator-kyuubi/pull/3485) required by Spark JDBC source not yet included in released version.
+| Hive Driver ([doc](../jdbc/hive_jdbc.html))| org.apache.hive.jdbc.HiveDriver | 
+
+Refer to docs of the driver and prepare the JDBC driver jar file.
+
+### Prepare JDBC Hive Dialect extension
+
+Hive Dialect support is requried by Spark for wraping sql correctly and sending to JDBC driver. Kyuubi provides a JDBC dialect extension with auto regiested Hive Daliect support for Spark. Follow the instrunctions in [Hive Dialect Support](../../engines/spark/jdbc-dialect.html) to prepare the plugin jar file `kyuubi-extension-spark-jdbc-dialect_-*.jar`.

Review Comment:
   sql => SQL



##########
docs/client/python/pyspark.md:
##########
@@ -0,0 +1,122 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements.  See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License.  You may obtain a copy of the License at
+ -
+ -   http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+
+# PySpark
+
+[PySpark](https://spark.apache.org/docs/latest/api/python/index.html) is an interface for Apache Spark in Python. Kyuubi can be used as JDBC source in PySpark.
+
+## Requirements
+PySpark works with Python 3.7 and above.
+
+Install PySpark with Spark SQL and optional pandas on Spark using PyPI as follows:
+
+```shell
+pip install pyspark 'pyspark[sql]' 'pyspark[pandas_on_spark]'
+```
+
+For installation using Conda or manually downloading, please refer to [PySpark installation](https://spark.apache.org/docs/latest/api/python/getting_started/install.html).
+
+## Preperation
+
+
+### Prepare JDBC driver 
+Prepare JDBC driver jar file. Supported Hive compatible JDBC Driver as below:
+
+| Driver     | Driver Class Name | Remarks|
+| ---------- | ----------------- | ----- |
+| Kyuubi Hive Driver ([doc](../jdbc/kyuubi_jdbc.html))| org.apache.kyuubi.jdbc.KyuubiHiveDriver |  Compile for the driver on master branch, as [KYUUBI #3484](https://github.com/apache/incubator-kyuubi/pull/3485) required by Spark JDBC source not yet included in released version.
+| Hive Driver ([doc](../jdbc/hive_jdbc.html))| org.apache.hive.jdbc.HiveDriver | 
+
+Refer to docs of the driver and prepare the JDBC driver jar file.
+
+### Prepare JDBC Hive Dialect extension
+
+Hive Dialect support is requried by Spark for wraping sql correctly and sending to JDBC driver. Kyuubi provides a JDBC dialect extension with auto regiested Hive Daliect support for Spark. Follow the instrunctions in [Hive Dialect Support](../../engines/spark/jdbc-dialect.html) to prepare the plugin jar file `kyuubi-extension-spark-jdbc-dialect_-*.jar`.
+
+### Including jars of JDBC driver and Hive Dialect extention
+
+Choose one of following ways to include jar files to Spark.
+
+- Put the jar file of JDBC driver and Hive Dialect to `$SPARK_HOME/jars` directory to make it visible for the classpath of PySpark. And adding `spark.sql.extensions = org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension` to `$SPARK_HOME/conf/spark_defaults.conf.`
+
+- With spark's start shell, include JDBC driver when you submit the application with `--packages`, and the Hive Dialect plugins with `--jars`
+
+```
+$SPARK_HOME/bin/pyspark --py-files PY_FILES \
+--packages org.apache.hive:hive-jdbc:x.y.z \
+--jars /path/kyuubi-extension-spark-jdbc-dialect_-*.jar 
+```
+
+- Setting jars and config with SparkSession builder
+
+```python
+from pyspark.sql import SparkSession
+
+spark = SparkSession.builder \
+        .config("spark.jars", "/path/hive-jdbc-x.y.z.jar,/path/kyuubi-extension-spark-jdbc-dialect_-*.jar") \
+        .config("spark.sql.extensions", "org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension") \
+        .getOrCreate()
+```
+
+
+
+## Usage
+
+For further information about PySpark JDBC usage and options, please refer to Spark's [JDBC To Other Databases](https://spark.apache.org/docs/latest/sql-data-sources-jdbc.html).
+
+### Reading and Writing via JDBC data source
+
+```python
+# Loading data from Kyuubi via HiveDriver as JDBC source
+jdbcDF = spark.read \
+  .format("jdbc") \
+  .options(driver="org.apache.hive.jdbc.HiveDriver",

Review Comment:
   ditto, prefer to use KyuubiHiveDriver



##########
docs/client/python/pyspark.md:
##########
@@ -0,0 +1,122 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements.  See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License.  You may obtain a copy of the License at
+ -
+ -   http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+
+# PySpark
+
+[PySpark](https://spark.apache.org/docs/latest/api/python/index.html) is an interface for Apache Spark in Python. Kyuubi can be used as JDBC source in PySpark.
+
+## Requirements
+PySpark works with Python 3.7 and above.
+
+Install PySpark with Spark SQL and optional pandas on Spark using PyPI as follows:
+
+```shell
+pip install pyspark 'pyspark[sql]' 'pyspark[pandas_on_spark]'
+```
+
+For installation using Conda or manually downloading, please refer to [PySpark installation](https://spark.apache.org/docs/latest/api/python/getting_started/install.html).
+
+## Preperation
+
+
+### Prepare JDBC driver 
+Prepare JDBC driver jar file. Supported Hive compatible JDBC Driver as below:
+
+| Driver     | Driver Class Name | Remarks|
+| ---------- | ----------------- | ----- |
+| Kyuubi Hive Driver ([doc](../jdbc/kyuubi_jdbc.html))| org.apache.kyuubi.jdbc.KyuubiHiveDriver |  Compile for the driver on master branch, as [KYUUBI #3484](https://github.com/apache/incubator-kyuubi/pull/3485) required by Spark JDBC source not yet included in released version.
+| Hive Driver ([doc](../jdbc/hive_jdbc.html))| org.apache.hive.jdbc.HiveDriver | 
+
+Refer to docs of the driver and prepare the JDBC driver jar file.
+
+### Prepare JDBC Hive Dialect extension
+
+Hive Dialect support is requried by Spark for wraping sql correctly and sending to JDBC driver. Kyuubi provides a JDBC dialect extension with auto regiested Hive Daliect support for Spark. Follow the instrunctions in [Hive Dialect Support](../../engines/spark/jdbc-dialect.html) to prepare the plugin jar file `kyuubi-extension-spark-jdbc-dialect_-*.jar`.
+
+### Including jars of JDBC driver and Hive Dialect extention
+
+Choose one of following ways to include jar files to Spark.
+
+- Put the jar file of JDBC driver and Hive Dialect to `$SPARK_HOME/jars` directory to make it visible for the classpath of PySpark. And adding `spark.sql.extensions = org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension` to `$SPARK_HOME/conf/spark_defaults.conf.`
+
+- With spark's start shell, include JDBC driver when you submit the application with `--packages`, and the Hive Dialect plugins with `--jars`
+
+```
+$SPARK_HOME/bin/pyspark --py-files PY_FILES \
+--packages org.apache.hive:hive-jdbc:x.y.z \

Review Comment:
   Can we use kyuubi hive driver in demo code?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] pan3793 commented on a diff in pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
pan3793 commented on code in PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#discussion_r979263827


##########
docs/client/python/pyspark.md:
##########
@@ -0,0 +1,122 @@
+<!--
+ - Licensed to the Apache Software Foundation (ASF) under one or more
+ - contributor license agreements.  See the NOTICE file distributed with
+ - this work for additional information regarding copyright ownership.
+ - The ASF licenses this file to You under the Apache License, Version 2.0
+ - (the "License"); you may not use this file except in compliance with
+ - the License.  You may obtain a copy of the License at
+ -
+ -   http://www.apache.org/licenses/LICENSE-2.0
+ -
+ - Unless required by applicable law or agreed to in writing, software
+ - distributed under the License is distributed on an "AS IS" BASIS,
+ - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ - See the License for the specific language governing permissions and
+ - limitations under the License.
+ -->
+
+
+# PySpark
+
+[PySpark](https://spark.apache.org/docs/latest/api/python/index.html) is an interface for Apache Spark in Python. Kyuubi can be used as JDBC source in PySpark.
+
+## Requirements
+PySpark works with Python 3.7 and above.
+
+Install PySpark with Spark SQL and optional pandas on Spark using PyPI as follows:
+
+```shell
+pip install pyspark 'pyspark[sql]' 'pyspark[pandas_on_spark]'
+```
+
+For installation using Conda or manually downloading, please refer to [PySpark installation](https://spark.apache.org/docs/latest/api/python/getting_started/install.html).
+
+## Preperation
+
+
+### Prepare JDBC driver 
+Prepare JDBC driver jar file. Supported Hive compatible JDBC Driver as below:
+
+| Driver     | Driver Class Name | Remarks|
+| ---------- | ----------------- | ----- |
+| Kyuubi Hive Driver ([doc](../jdbc/kyuubi_jdbc.html))| org.apache.kyuubi.jdbc.KyuubiHiveDriver |  Compile for the driver on master branch, as [KYUUBI #3484](https://github.com/apache/incubator-kyuubi/pull/3485) required by Spark JDBC source not yet included in released version.
+| Hive Driver ([doc](../jdbc/hive_jdbc.html))| org.apache.hive.jdbc.HiveDriver | 
+
+Refer to docs of the driver and prepare the JDBC driver jar file.
+
+### Prepare JDBC Hive Dialect extension
+
+Hive Dialect support is requried by Spark for wraping sql correctly and sending to JDBC driver. Kyuubi provides a JDBC dialect extension with auto regiested Hive Daliect support for Spark. Follow the instrunctions in [Hive Dialect Support](../../engines/spark/jdbc-dialect.html) to prepare the plugin jar file `kyuubi-extension-spark-jdbc-dialect_-*.jar`.
+
+### Including jars of JDBC driver and Hive Dialect extention
+
+Choose one of following ways to include jar files to Spark.
+
+- Put the jar file of JDBC driver and Hive Dialect to `$SPARK_HOME/jars` directory to make it visible for the classpath of PySpark. And adding `spark.sql.extensions = org.apache.spark.sql.dialect.KyuubiSparkJdbcDialectExtension` to `$SPARK_HOME/conf/spark_defaults.conf.`
+
+- With spark's start shell, include JDBC driver when you submit the application with `--packages`, and the Hive Dialect plugins with `--jars`
+
+```
+$SPARK_HOME/bin/pyspark --py-files PY_FILES \
+--packages org.apache.hive:hive-jdbc:x.y.z \

Review Comment:
   Emm... 1.6.1 is scheduled mid-Oct. then let's update doc after that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#issuecomment-1256295235

   Thanks for adding this document, have some high-level thoughts.
   
   We should not narrow the Kyuubi JDBC data source use cases into PySpark, it's also suitable for `spark-shell`, `spark-sql` and the normal spark jar jobs, right? So, I think maybe we can split the doc into two parts, the 1st part is "How to use Kyuubi as a Spark JDBC data source", and the 2nd is "How to use PySpark to access Kyuubi JDBC data source".


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] pan3793 commented on pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
pan3793 commented on PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#issuecomment-1256300061

   Also, to unconfusing users, we should emphasize there are TWO Spark applications in such use architects, and the most benefit is security but not performance, the arch also suitable for `spark-shell`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org


[GitHub] [incubator-kyuubi] bowenliang123 commented on pull request #3407: [KYUUBI #3406] [Subtask] [Doc] Add PySpark client docs

Posted by GitBox <gi...@apache.org>.
bowenliang123 commented on PR #3407:
URL: https://github.com/apache/incubator-kyuubi/pull/3407#issuecomment-1256313498

   Yes. But so far this docs is purposed for python user (like AI team and etc.) accessing to Kyuubi and it is put under `/client/python` rather than discussing all avaialble accessing way in Spark which could be enhanced in future docs solely for Spark under `/client`.  And they are not confusing with two spark jobs as discussed in first paragraph it is about connect to Kyuubi but not the any realted to Spark engine.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org