You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@kyuubi.apache.org by GitBox <gi...@apache.org> on 2022/07/20 07:06:49 UTC
[GitHub] [incubator-kyuubi] deadwind4 opened a new pull request, #3099: [KYUUBI #3068][DOC] Add the Hudi connector doc for Spark SQL Query Engine
deadwind4 opened a new pull request, #3099:
URL: https://github.com/apache/incubator-kyuubi/pull/3099
### _Why are the changes needed?_
Add the Hudi connector doc for Spark SQL Query Engine
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] deadwind4 commented on pull request #3099: [KYUUBI #3068][DOC] Add the Hudi connector doc for Spark SQL Query Engine
Posted by GitBox <gi...@apache.org>.
deadwind4 commented on PR #3099:
URL: https://github.com/apache/incubator-kyuubi/pull/3099#issuecomment-1190159605
> thanks @deadwind4 for helping improve the doc.
>
> BTW, the doc now has only two main parts - integration & operations. I know this doc structure is derived from the delta lake page. Because I mainly focus on building the skeleton of the doc not the content inside it. So, do we need to put more information for users to understand better? I want to hear your thoughts.
We can only write a ‘quick start' in the 1.6.0 release. We can await feedback from community members before further optimizing the document about these connectors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #3099: [KYUUBI #3068][DOC] Add the Hudi connector doc for Spark SQL Query Engine
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on PR #3099:
URL: https://github.com/apache/incubator-kyuubi/pull/3099#issuecomment-1189917415
thanks @deadwind4 for helping improve the doc.
BTW, the doc now has only two main parts - integration & operations. I know this doc structure is derived from the delta lake page. But do we need to put more information for users to understand better? I want to hear your thoughts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3099: [KYUUBI #3068][DOC] Add the Hudi connector doc for Spark SQL Query Engine
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3099:
URL: https://github.com/apache/incubator-kyuubi/pull/3099#discussion_r925597893
##########
docs/connector/spark/hudi.rst:
##########
@@ -16,21 +16,70 @@
`Hudi`_
========
+Apache Hudi (pronounced “hoodie”) is the next generation streaming data lake platform.
+Apache Hudi brings core warehouse and database functionality directly to a data lake.
+
+.. tip::
+ This article assumes that you have mastered the basic knowledge and operation of `Hudi`_.
+ For the knowledge about Hudi not mentioned in this article,
+ you can obtain it from its `Official Documentation`_.
+
+By using Kyuubi, we can run SQL queries towards Hudi which is more convenient, easy to understand,
+and easy to expand than directly using Spark to manipulate Hudi.
+
Hudi Integration
----------------
+To enable the integration of kyuubi spark sql engine and Hudi through
+Catalog APIs, you need to:
+
+- Referencing the Hudi :ref:`dependencies`
+- Setting the Spark extension and catalog :ref:`configurations`
+
.. _dependencies:
Dependencies
************
+The **classpath** of kyuubi spark sql engine with Hudi supported consists of
+
+1. kyuubi-spark-sql-engine-|release|.jar, the engine jar deployed with Kyuubi distributions
+2. a copy of spark distribution
+3. hudi-spark<spark.version>-bundle_<scala.version>-<hudi.version>.jar, which can be found in the `Maven Central`_
+
+In order to make the Hudi packages visible for the runtime classpath of engines, we can use one of these methods:
+
+1. Put the Hudi packages into ``$SPARK_HOME/jars`` directly
+2. Set ``spark.jars=/path/to/hudi-spark-bundle``
+
.. _configurations:
Configurations
**************
+To activate functionality of Hudi, we can set the following configurations:
+
+.. code-block:: properties
+ # Spark 3.2
+ spark.serializer=org.apache.spark.serializer.KryoSerializer
+ spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension
+ spark.sql.catalog.spark_catalog=org.apache.spark.sql.hudi.catalog.HoodieCatalog
+
+ # Spark 3.1
+ spark.serializer=org.apache.spark.serializer.KryoSerializer
+ spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension
Hudi Operations
---------------
-.. _Hudi: https://hudi.apache.org/
\ No newline at end of file
+.. code-block:: sql
+
+ CREATE TABLE hudi_cow_nonpcf_tbl (
Review Comment:
how about we add 4 basic examples including `create/insert/select/delete`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a diff in pull request #3099: [KYUUBI #3068][DOC] Add the Hudi connector doc for Spark SQL Query Engine
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on code in PR #3099:
URL: https://github.com/apache/incubator-kyuubi/pull/3099#discussion_r925632789
##########
docs/connector/spark/hudi.rst:
##########
@@ -16,21 +16,70 @@
`Hudi`_
========
+Apache Hudi (pronounced “hoodie”) is the next generation streaming data lake platform.
+Apache Hudi brings core warehouse and database functionality directly to a data lake.
+
+.. tip::
+ This article assumes that you have mastered the basic knowledge and operation of `Hudi`_.
+ For the knowledge about Hudi not mentioned in this article,
+ you can obtain it from its `Official Documentation`_.
+
+By using Kyuubi, we can run SQL queries towards Hudi which is more convenient, easy to understand,
+and easy to expand than directly using Spark to manipulate Hudi.
+
Hudi Integration
----------------
+To enable the integration of kyuubi spark sql engine and Hudi through
+Catalog APIs, you need to:
+
+- Referencing the Hudi :ref:`dependencies`
+- Setting the Spark extension and catalog :ref:`configurations`
+
.. _dependencies:
Dependencies
************
+The **classpath** of kyuubi spark sql engine with Hudi supported consists of
+
+1. kyuubi-spark-sql-engine-|release|.jar, the engine jar deployed with Kyuubi distributions
+2. a copy of spark distribution
+3. hudi-spark<spark.version>-bundle_<scala.version>-<hudi.version>.jar, which can be found in the `Maven Central`_
Review Comment:
spark.version and scala.version are usaully ternary, while here might be binary? this may not quite meet the reader's expectation, how about giving an example?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed pull request #3099: [KYUUBI #3068][DOC] Add the Hudi connector doc for Spark SQL Query Engine
Posted by GitBox <gi...@apache.org>.
yaooqinn closed pull request #3099: [KYUUBI #3068][DOC] Add the Hudi connector doc for Spark SQL Query Engine
URL: https://github.com/apache/incubator-kyuubi/pull/3099
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #3099: [KYUUBI #3068][DOC] Add the Hudi connector doc for Spark SQL Query Engine
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #3099:
URL: https://github.com/apache/incubator-kyuubi/pull/3099#issuecomment-1189962354
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#3099](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0ee870d) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/8f3d789821484a6941dfaa9cec203077c0c2d091?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8f3d789) will **increase** coverage by `0.02%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #3099 +/- ##
============================================
+ Coverage 51.30% 51.32% +0.02%
Complexity 6 6
============================================
Files 458 458
Lines 25381 25389 +8
Branches 3541 3536 -5
============================================
+ Hits 13021 13032 +11
+ Misses 11119 11114 -5
- Partials 1241 1243 +2
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...er/http/authentication/AuthenticationHandler.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvaHR0cC9hdXRoZW50aWNhdGlvbi9BdXRoZW50aWNhdGlvbkhhbmRsZXIuc2NhbGE=) | `52.94% <0.00%> (-11.77%)` | :arrow_down: |
| [...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWN0bC9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jdGwvY21kL2xvZy9Mb2dCYXRjaENvbW1hbmQuc2NhbGE=) | `78.00% <0.00%> (-2.00%)` | :arrow_down: |
| [.../org/apache/kyuubi/operation/KyuubiOperation.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vS3l1dWJpT3BlcmF0aW9uLnNjYWxh) | `68.91% <0.00%> (-1.36%)` | :arrow_down: |
| [...ver/http/authentication/AuthenticationFilter.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvaHR0cC9hdXRoZW50aWNhdGlvbi9BdXRoZW50aWNhdGlvbkZpbHRlci5zY2FsYQ==) | `90.76% <0.00%> (-0.90%)` | :arrow_down: |
| [.../apache/kyuubi/server/http/ThriftHttpServlet.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvaHR0cC9UaHJpZnRIdHRwU2VydmxldC5zY2FsYQ==) | `68.93% <0.00%> (-0.83%)` | :arrow_down: |
| [...a/org/apache/kyuubi/service/TFrontendService.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2aWNlL1RGcm9udGVuZFNlcnZpY2Uuc2NhbGE=) | `91.17% <0.00%> (-0.30%)` | :arrow_down: |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `97.25% <0.00%> (ø)` | |
| [...rg/apache/kyuubi/server/api/v1/AdminResource.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9zZXJ2ZXIvYXBpL3YxL0FkbWluUmVzb3VyY2Uuc2NhbGE=) | `90.90% <0.00%> (ø)` | |
| [...g/apache/kyuubi/operation/BatchJobSubmission.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXNlcnZlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9vcGVyYXRpb24vQmF0Y2hKb2JTdWJtaXNzaW9uLnNjYWxh) | `77.63% <0.00%> (ø)` | |
| [...rg/apache/kyuubi/jdbc/hive/auth/HttpAuthUtils.java](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWhpdmUtamRiYy9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3l1dWJpL2pkYmMvaGl2ZS9hdXRoL0h0dHBBdXRoVXRpbHMuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| ... and [22 more](https://codecov.io/gh/apache/incubator-kyuubi/pull/3099/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@kyuubi.apache.org
For additional commands, e-mail: notifications-help@kyuubi.apache.org