You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/08/27 08:44:54 UTC
[GitHub] [incubator-kyuubi] simon824 opened a new pull request #994: [KYUUBI #703] support hudi-0.9.0
simon824 opened a new pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994
### _Why are the changes needed?_
Hudi has add Spark Sql Support in version 0.9.0.
This PR add Hudi-0.9.0 dependency and Unit Testing #703
### _How was this patch tested?_
- [x] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [ ] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r701611005
##########
File path: .github/workflows/master.yml
##########
@@ -90,6 +90,7 @@ jobs:
if: ${{ matrix.codecov == 'true' }}
run: bash <(curl -s https://codecov.io/bash)
- name: Detected Dependency List Change
+ if: ${{ matrix.java == 8 }}
Review comment:
is this already in master？ we can restore this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909086522
```
at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createHiveDataSourceTable(CreateHoodieTableCommand.scala:214)
at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createTableInCatalog(CreateHoodieTableCommand.scala:177)
at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.run(CreateHoodieTableCommand.scala:73)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
```
cc @pan3793, the data lakes are bound to hive metastore sometimes, are we good to use `in-memory` for them all?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 removed a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 removed a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909820127
> It say Hudi 0.9.0 doesn't support Spark 3.1 [apache/hudi#3554](https://github.com/apache/hudi/issues/3554)
support Spark-2.4.3+ & Spark 3.x, https://hudi.apache.org/docs/quick-start-guide/
Maybe still have some bugs in some aspects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r701527631
##########
File path: kyuubi-server/pom.xml
##########
@@ -133,6 +133,42 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-avro_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark3_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
Review comment:
ah thanks, it's a good news ..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909157748
> ```
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createHiveDataSourceTable(CreateHoodieTableCommand.scala:214)
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createTableInCatalog(CreateHoodieTableCommand.scala:177)
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.run(CreateHoodieTableCommand.scala:73)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
> at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
> at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
> ```
>
> cc @pan3793, the data lakes are bound to hive metastore sometimes, are we good to use `in-memory` for them all?
Make sense.
For Iceberg, use neither hive catalog nor in-memory but a separate v2 catalog.
For DeltaLake, both hive catalog and in-memory should be ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r699551313
##########
File path: pom.xml
##########
@@ -1064,6 +1065,32 @@
<version>${iceberg.version}</version>
</dependency>
+ <!-- Hudi dependency
+ TODO: replace this 4 dependency with hudi-spark3-bundle_2.12 -->
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark3_${scala.binary.version}</artifactId>
Review comment:
IMO, this package is broken, the `hudi-spark3_2.12` depends on `hudi-spark-common_2.11`, which will cause the Scala library conflict, you can check at
https://mvnrepository.com/artifact/org.apache.hudi/hudi-spark3_2.12/0.9.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909912750
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#994](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1313398) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/b382ed0dd87e9e19cab40f3e6b596c042ecdc066?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b382ed0) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head 1313398 differs from pull request most recent head a04b2e4. Consider uploading reports for the commit a04b2e4 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #994 +/- ##
============================================
- Coverage 80.06% 80.06% -0.01%
Complexity 11 11
============================================
Files 152 152
Lines 5553 5552 -1
Branches 644 644
============================================
- Hits 4446 4445 -1
Misses 725 725
Partials 382 382
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...la/org/apache/kyuubi/zookeeper/ZookeeperConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXpvb2tlZXBlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS96b29rZWVwZXIvWm9va2VlcGVyQ29uZi5zY2FsYQ==) | `100.00% <100.00%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b382ed0...a04b2e4](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912791182
Stand on the Spark side, HUDI is a data source implementation, or just call it a plugin.
Thus, for the transitive dependencies of HUDI, like Hadoop, Curator, Parquet, Avro, ORC, Hive, we must **RESPECT** the Spark bundled versions, as Kyuubi support multi Spark versions, we need to separate those deps version by different Spark versions using maven profiles. e.g. use parquet 1.10.0 for Spark 3.0/3.1, but use parquet 1.12.0 for Spark 3.2(coming soon).
Besides on Spark, we should also respect the Kyuubi deps version, assume you are using IDEA, please open the `External Libraries`, there are lots of jars which has multi versions exist in the classpath, which means a high risk of class conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909912750
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#994](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a04b2e4) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/b382ed0dd87e9e19cab40f3e6b596c042ecdc066?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b382ed0) will **increase** coverage by `0.70%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #994 +/- ##
============================================
+ Coverage 80.06% 80.76% +0.70%
- Complexity 11 14 +3
============================================
Files 152 154 +2
Lines 5553 5771 +218
Branches 644 659 +15
============================================
+ Hits 4446 4661 +215
- Misses 725 729 +4
+ Partials 382 381 -1
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...la/org/apache/kyuubi/zookeeper/ZookeeperConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXpvb2tlZXBlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS96b29rZWVwZXIvWm9va2VlcGVyQ29uZi5zY2FsYQ==) | `100.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC9LeXV1YmlTUUxDb25mLnNjYWxh) | `100.00% <0.00%> (ø)` | |
| [...rg/apache/kyuubi/sql/KyuubiSparkSQLExtension.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC9LeXV1YmlTcGFya1NRTEV4dGVuc2lvbi5zY2FsYQ==) | `100.00% <0.00%> (ø)` | |
| [...ql/sqlclassification/KyuubiSqlClassification.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC9zcWxjbGFzc2lmaWNhdGlvbi9LeXV1YmlTcWxDbGFzc2lmaWNhdGlvbi5zY2FsYQ==) | `66.66% <0.00%> (ø)` | |
| [...sqlclassification/KyuubiGetSqlClassification.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC9zcWxjbGFzc2lmaWNhdGlvbi9LeXV1YmlHZXRTcWxDbGFzc2lmaWNhdGlvbi5zY2FsYQ==) | `73.33% <0.00%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `96.92% <0.00%> (+2.08%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b382ed0...a04b2e4](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909912750
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#994](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1313398) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/b382ed0dd87e9e19cab40f3e6b596c042ecdc066?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b382ed0) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head 1313398 differs from pull request most recent head f33ebdf. Consider uploading reports for the commit f33ebdf to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #994 +/- ##
============================================
- Coverage 80.06% 80.06% -0.01%
Complexity 11 11
============================================
Files 152 152
Lines 5553 5552 -1
Branches 644 644
============================================
- Hits 4446 4445 -1
Misses 725 725
Partials 382 382
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...la/org/apache/kyuubi/zookeeper/ZookeeperConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXpvb2tlZXBlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS96b29rZWVwZXIvWm9va2VlcGVyQ29uZi5zY2FsYQ==) | `100.00% <100.00%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b382ed0...f33ebdf](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-913277867
> Hudi uses `HFileBootstrapIndex` by default, I don't think it's required for our ut, can we overwrite it and test if it can work without hbase deps?
>
> https://github.com/apache/hudi/blob/release-0.9.0/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L144-L147
ok，i will test and fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r699551313
##########
File path: pom.xml
##########
@@ -1064,6 +1065,32 @@
<version>${iceberg.version}</version>
</dependency>
+ <!-- Hudi dependency
+ TODO: replace this 4 dependency with hudi-spark3-bundle_2.12 -->
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark3_${scala.binary.version}</artifactId>
Review comment:
IMO, this package is broken, the `hudi-spark3_2.12` depends on `hudi-spark-common_2.11`, which will cause the Scala library conflict, you can check at
https://mvnrepository.com/artifact/org.apache.hudi/hudi-spark3_2.12/0.9.0
##########
File path: pom.xml
##########
@@ -1064,6 +1065,32 @@
<version>${iceberg.version}</version>
</dependency>
+ <!-- Hudi dependency
+ TODO: replace this 4 dependency with hudi-spark3-bundle_2.12 -->
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark3_${scala.binary.version}</artifactId>
Review comment:
cc @yanghua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-907056808
> Does hudi Spark SQL Extension only support Spark 3.1? Where can I find the support matrix or docs?
Support Spark-2.4.3+ & Spark 3.0, I have tested locally, spark 3.1 does not work
0.9.0 was released just a few days ago, I haven't yet found the release notes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912758022
Hudi uses `HFileBootstrapIndex` by default, I don't think it's required for our ut, can we overwrite it and test if it can work without hbase deps?
https://github.com/apache/hudi/blob/release-0.9.0/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L144-L147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-907037771
Thanks for contributing!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-907274371
CI failed because of `error file=/home/runner/work/incubator-kyuubi/incubator-kyuubi/kyuubi-common/src/test/scala/org/apache/kyuubi/HudiSuiteMixin.scala message=File must end with newline character`
Would you please make CI happy first?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912358794
> thanks, @simon824, this change LGTM, leave it to @pan3793 for the last comfirmation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909157748
> ```
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createHiveDataSourceTable(CreateHoodieTableCommand.scala:214)
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createTableInCatalog(CreateHoodieTableCommand.scala:177)
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.run(CreateHoodieTableCommand.scala:73)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
> at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
> at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
> ```
>
> cc @pan3793, the data lakes are bound to hive metastore sometimes, are we good to use `in-memory` for them all?
Make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-907162685
cc @yanghua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912758022
Hudi use `HFileBootstrapIndex` by default, I think it's not required for our ut, can we overwrite it and test if it can work without hbase deps?
https://github.com/apache/hudi/blob/release-0.9.0/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L144-L147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912360041
OK, thanks a lot for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912200813
Congratulation! The CI passed. Thanks to @simon824 for good work. But I need some time to have a deep look at the transitive deps pulled by Hudi before merging it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909157748
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 closed pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 closed pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r697524613
##########
File path: kyuubi-common/src/test/scala/org/apache/kyuubi/HudiSuiteMixin.scala
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi
+
+import java.nio.file.Path
+
+trait HudiSuiteMixin extends DataLakeSuiteMixin {
+
+ override protected def format: String = "hudi"
+
+ override protected def catalog: String = "spark_catalog"
+
+ override protected def warehouse: Path = Utils.createTempDir()
+
+ override protected def extraJars: String = {
+ System.getProperty("java.class.path")
+ .split(":")
+ .filter(_.contains("hudi-spark")).head
+ }
+
+ override protected def extraConfigs = Map(
+ "spark.sql.defaultCatalog" -> catalog,
+ "spark.sql.extensions" -> "org.apache.spark.sql.hudi.HoodieSparkSessionExtension",
+ "spark.sql.serializer" -> "org.apache.spark.serializer.KryoSerializer",
+ "spark.jars" -> extraJars)
+}
Review comment:
new line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-907040010
Does hudi Spark SQL Extension only support Spark 3.1? Where can I find the support matrix or docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r697528636
##########
File path: kyuubi-common/src/test/scala/org/apache/kyuubi/HudiSuiteMixin.scala
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi
+
+import java.nio.file.Path
+
+trait HudiSuiteMixin extends DataLakeSuiteMixin {
+
+ override protected def format: String = "hudi"
+
+ override protected def catalog: String = "spark_catalog"
+
+ override protected def warehouse: Path = Utils.createTempDir()
+
+ override protected def extraJars: String = {
+ System.getProperty("java.class.path")
+ .split(":")
+ .filter(_.contains("hudi-spark")).head
+ }
+
+ override protected def extraConfigs = Map(
+ "spark.sql.defaultCatalog" -> catalog,
+ "spark.sql.extensions" -> "org.apache.spark.sql.hudi.HoodieSparkSessionExtension",
+ "spark.sql.serializer" -> "org.apache.spark.serializer.KryoSerializer",
Review comment:
Do you mean `spark.serializer`? There is no conf name `spark.sql.serializer` in spark conf list, you can check it at http://spark.apache.org/docs/latest/configuration.html.
BTW, can Hudi work without `KryoSerializer`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912778048
I still see `hudi-spark_2.12` pulls `jackson-module-scala_2.11` into classpath
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r700788528
##########
File path: kyuubi-server/pom.xml
##########
@@ -133,6 +133,42 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-avro_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark3_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
Review comment:
the whole project
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909086522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 removed a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 removed a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909820127
> It say Hudi 0.9.0 doesn't support Spark 3.1 [apache/hudi#3554](https://github.com/apache/hudi/issues/3554)
support Spark-2.4.3+ & Spark 3.x, https://hudi.apache.org/docs/quick-start-guide/
Maybe still have some bugs in some aspects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r699553714
##########
File path: pom.xml
##########
@@ -1064,6 +1065,32 @@
<version>${iceberg.version}</version>
</dependency>
+ <!-- Hudi dependency
+ TODO: replace this 4 dependency with hudi-spark3-bundle_2.12 -->
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark3_${scala.binary.version}</artifactId>
Review comment:
cc @yanghua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r700784576
##########
File path: kyuubi-server/pom.xml
##########
@@ -133,6 +133,42 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-avro_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark3_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
Review comment:
Hudi needs to depend on hbase.
You mean to exclude it in kyuubi-server module or the whole project?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909216243
```
*** RUN ABORTED ***
java.lang.NoSuchMethodError: org.apache.hadoop.yarn.server.utils.BuilderUtils.newApplicationResourceUsageReport(IILorg/apache/hadoop/yarn/api/records/Resource;Lorg/apache/hadoop/yarn/api/records/Resource;Lorg/apache/hadoop/yarn/api/records/Resource;Ljava/util/Map;Ljava/util/Map;)Lorg/apache/hadoop/yarn/api/records/ApplicationResourceUsageReport;
at org.apache.hadoop.yarn.server.resourcemanager.RMServerUtils.<clinit>(RMServerUtils.java:493)
at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:277)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:348)
at org.apache.hadoop.yarn.server.MiniYARNCluster.access$200(MiniYARNCluster.java:128)
at org.apache.hadoop.yarn.server.MiniYARNCluster$ResourceManagerWrapper.serviceInit(MiniYARNCluster.java:497)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
at org.apache.hadoop.yarn.server.MiniYARNCluster.serviceInit(MiniYARNCluster.java:316)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
...
```
YARN dep conflicts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r701612040
##########
File path: .github/workflows/master.yml
##########
@@ -90,6 +90,7 @@ jobs:
if: ${{ matrix.codecov == 'true' }}
run: bash <(curl -s https://codecov.io/bash)
- name: Detected Dependency List Change
+ if: ${{ matrix.java == 8 }}
Review comment:
I think the `git merge` will auto-resolve this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909473658
It say Hudi 0.9.0 doesn't support Spark 3.1 https://github.com/apache/hudi/issues/3554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909912750
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#994](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1313398) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/b382ed0dd87e9e19cab40f3e6b596c042ecdc066?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b382ed0) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head 1313398 differs from pull request most recent head ff2585c. Consider uploading reports for the commit ff2585c to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #994 +/- ##
============================================
- Coverage 80.06% 80.06% -0.01%
Complexity 11 11
============================================
Files 152 152
Lines 5553 5552 -1
Branches 644 644
============================================
- Hits 4446 4445 -1
Misses 725 725
Partials 382 382
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...la/org/apache/kyuubi/zookeeper/ZookeeperConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXpvb2tlZXBlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS96b29rZWVwZXIvWm9va2VlcGVyQ29uZi5zY2FsYQ==) | `100.00% <100.00%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b382ed0...ff2585c](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] ulysses-you commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
ulysses-you commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r701526908
##########
File path: kyuubi-server/pom.xml
##########
@@ -133,6 +133,42 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-avro_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark3_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
Review comment:
> Hudi needs to depend on hbase.
just curious, why hudi depend on hbase ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yanghua commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yanghua commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r697787023
##########
File path: kyuubi-common/src/test/scala/org/apache/kyuubi/operation/BasicHudiJDBCTests.scala
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import org.apache.kyuubi.HudiSuiteMixin
+import org.apache.kyuubi.operation.meta.ResultSetSchemaConstant._
+
+
+trait BasicHudiJDBCTests extends JDBCTestUtils with HudiSuiteMixin {
Review comment:
`HudiBasicJDBCTests` sounds better?
##########
File path: pom.xml
##########
@@ -1064,6 +1066,12 @@
<version>${iceberg.version}</version>
</dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>${hudi.name}</artifactId>
Review comment:
ditto
##########
File path: kyuubi-server/pom.xml
##########
@@ -133,6 +133,12 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>${hudi.name}</artifactId>
Review comment:
IMO, we do not need to follow `${iceberg.name}`, it can be written here directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909157748
> ```
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createHiveDataSourceTable(CreateHoodieTableCommand.scala:214)
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createTableInCatalog(CreateHoodieTableCommand.scala:177)
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.run(CreateHoodieTableCommand.scala:73)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
> at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
> at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
> ```
>
> cc @pan3793, the data lakes are bound to hive metastore sometimes, are we good to use `in-memory` for them all?
Make sense.
For Iceberg, use neither hive catalog nor in-memory but a separate v2 catalog.
For DeltaLake, both hive catalog and in-memory should be ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r697524173
##########
File path: pom.xml
##########
@@ -1618,7 +1626,7 @@
<properties>
<spark.version>3.0.3</spark.version>
<delta.version>0.8.0</delta.version>
- <maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest</maven.plugin.scalatest.exclude.tags>
+ <maven.plugin.scalatest.exclude.tags>org.apache.kyuubi.tags.ExtendedSQLTest,org.apache.kyuubi.tags.HudiTest</maven.plugin.scalatest.exclude.tags>
Review comment:
So, hudi incompatible with spark 3.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-908869920
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-907056808
> Does hudi Spark SQL Extension only support Spark 3.1? Where can I find the support matrix or docs?
Support Spark-2.4.3+ & Spark 3.0 & spark 3.1, I have tested locally.
0.9.0 was released just a few days ago, I haven't yet found the release notes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909820127
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r700734023
##########
File path: kyuubi-server/pom.xml
##########
@@ -133,6 +133,42 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-avro_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark3_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
Review comment:
the test failure is related to this dep? can we exclude it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909157748
> ```
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createHiveDataSourceTable(CreateHoodieTableCommand.scala:214)
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createTableInCatalog(CreateHoodieTableCommand.scala:177)
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.run(CreateHoodieTableCommand.scala:73)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
> at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
> at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
> ```
>
> cc @pan3793, the data lakes are bound to hive metastore sometimes, are we good to use `in-memory` for them all?
Make sense.
For Iceberg, use neither hive catalog nor in-memory but a separate v2 catalog.
For DeltaLake, both hive catalog and in-memory should be ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912292723
thanks, @simon824, this change LGTM, leave it to @pan3793 for the last comfirmation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r697528636
##########
File path: kyuubi-common/src/test/scala/org/apache/kyuubi/HudiSuiteMixin.scala
##########
@@ -0,0 +1,41 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi
+
+import java.nio.file.Path
+
+trait HudiSuiteMixin extends DataLakeSuiteMixin {
+
+ override protected def format: String = "hudi"
+
+ override protected def catalog: String = "spark_catalog"
+
+ override protected def warehouse: Path = Utils.createTempDir()
+
+ override protected def extraJars: String = {
+ System.getProperty("java.class.path")
+ .split(":")
+ .filter(_.contains("hudi-spark")).head
+ }
+
+ override protected def extraConfigs = Map(
+ "spark.sql.defaultCatalog" -> catalog,
+ "spark.sql.extensions" -> "org.apache.spark.sql.hudi.HoodieSparkSessionExtension",
+ "spark.sql.serializer" -> "org.apache.spark.serializer.KryoSerializer",
Review comment:
Do you mean `spark.serializer`? There is no conf named `spark.sql.serializer` in spark conf list, you can check it at http://spark.apache.org/docs/latest/configuration.html.
BTW, can Hudi work without `KryoSerializer`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r697522725
##########
File path: pom.xml
##########
@@ -1749,4 +1757,4 @@
</build>
</profile>
</profiles>
-</project>
+</project>
Review comment:
pls insert a newline at end of file
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912895157
it worths another PR to solve, this PR has met its original goal. thanks, merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909912750
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#994](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1313398) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/b382ed0dd87e9e19cab40f3e6b596c042ecdc066?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b382ed0) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head 1313398 differs from pull request most recent head ff2585c. Consider uploading reports for the commit ff2585c to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #994 +/- ##
============================================
- Coverage 80.06% 80.06% -0.01%
Complexity 11 11
============================================
Files 152 152
Lines 5553 5552 -1
Branches 644 644
============================================
- Hits 4446 4445 -1
Misses 725 725
Partials 382 382
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...la/org/apache/kyuubi/zookeeper/ZookeeperConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXpvb2tlZXBlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS96b29rZWVwZXIvWm9va2VlcGVyQ29uZi5zY2FsYQ==) | `100.00% <100.00%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b382ed0...ff2585c](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909825474
> Do we need to test spark 3.0 only for now?
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-908869920
Hi @simon824, can you take a look at the test failure?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909157748
> ```
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createHiveDataSourceTable(CreateHoodieTableCommand.scala:214)
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.createTableInCatalog(CreateHoodieTableCommand.scala:177)
> at org.apache.spark.sql.hudi.command.CreateHoodieTableCommand.run(CreateHoodieTableCommand.scala:73)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:70)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:68)
> at org.apache.spark.sql.execution.command.ExecutedCommandExec.executeCollect(commands.scala:79)
> at org.apache.spark.sql.Dataset.$anonfun$logicalPlan$1(Dataset.scala:228)
> at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3687)
> ```
>
> cc @pan3793, the data lakes are bound to hive metastore sometimes, are we good to use `in-memory` for them all?
Make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912758022
Hudi uses `HFileBootstrapIndex` by default, I think it's not required for our ut, can we overwrite it and test if it can work without hbase deps?
https://github.com/apache/hudi/blob/release-0.9.0/hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java#L144-L147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] cfmcgrady commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
cfmcgrady commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r698130713
##########
File path: kyuubi-common/src/test/scala/org/apache/kyuubi/operation/BasicHudiJDBCTests.scala
##########
@@ -0,0 +1,104 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kyuubi.operation
+
+import org.apache.kyuubi.HudiSuiteMixin
+import org.apache.kyuubi.operation.meta.ResultSetSchemaConstant._
+
+
+trait BasicHudiJDBCTests extends JDBCTestUtils with HudiSuiteMixin {
+
+ test("get catalogs") {
+ withJdbcStatement() { statement =>
+ val metaData = statement.getConnection.getMetaData
+ val catalogs = metaData.getCatalogs
+ catalogs.next()
+ assert(catalogs.getString(TABLE_CAT) === "spark_catalog")
+ assert(!catalogs.next())
+ }
+ }
+
+ test("get schemas") {
+ val dbs = Seq("db1", "db2", "db33", "db44")
+ val dbDflts = Seq("default", "global_temp")
+
+ val catalog = "spark_catalog"
+ withDatabases(dbs: _*) { statement =>
+ dbs.foreach(db => statement.execute(s"CREATE DATABASE IF NOT EXISTS $db"))
+ val metaData = statement.getConnection.getMetaData
+
+ Seq("", "*", "%", null, ".*", "_*", "_%", ".%") foreach { pattern =>
+ checkGetSchemas(metaData.getSchemas(catalog, pattern), dbs ++ dbDflts, catalog)
+ }
+
+ Seq("db%", "db.*") foreach { pattern =>
+ checkGetSchemas(metaData.getSchemas(catalog, pattern), dbs, catalog)
+ }
+
+ Seq("db_", "db.") foreach { pattern =>
+ checkGetSchemas(metaData.getSchemas(catalog, pattern), dbs.take(2), catalog)
+ }
+
+ checkGetSchemas(metaData.getSchemas(catalog, "db1"), Seq("db1"), catalog)
+ checkGetSchemas(metaData.getSchemas(catalog, "db_not_exist"), Seq.empty, catalog)
+ }
+ }
+
+ test("get tables") {
+ val table = "table_1_test"
+ val schema = "default"
+ val tableType = "TABLE"
+ withJdbcStatement(table) { statement =>
+ statement.execute(
+ s"""
+ | create table $table (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ | ) using $format
+ | options (
+ | primaryKey = 'id',
+ | preCombineField = 'ts'
+ | )
+ """.stripMargin)
+
+ val metaData = statement.getConnection.getMetaData
+ val rs1 = metaData.getTables(null, null, null, null)
+
+ assert(rs1.next())
+ val catalogName = rs1.getString(TABLE_CAT)
+ assert(catalogName === "spark_catalog" || catalogName === null)
+ assert(rs1.getString(TABLE_SCHEM) === schema)
+ assert(rs1.getString(TABLE_NAME) == table)
+ assert(rs1.getString(TABLE_TYPE) == tableType)
+ assert(rs1.getString(REMARKS) === table)
+ assert(!rs1.next())
+
+ val rs2 = metaData.getTables(null, null, "table%", Array("TABLE"))
+ assert(rs2.next())
+ assert(rs2.getString(TABLE_NAME) == table)
+ assert(!rs2.next())
+
+ val rs3 = metaData.getTables(null, "default", "*", Array("VIEW"))
+ assert(!rs3.next())
+ }
+ }
+
+
+}
Review comment:
```suggestion
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn closed pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r697521556
##########
File path: pom.xml
##########
@@ -167,6 +167,8 @@
<hive.version>2.3.7</hive.version>
<iceberg.name>iceberg-spark3-runtime</iceberg.name>
<iceberg.version>0.12.0</iceberg.version>
+ <hudi.name>hudi-spark-bundle_2.12</hudi.name>
Review comment:
Let's just inline the value and remove the property `hudi.name`, and also replace `_2.12` by `_${scala.binary.version}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909824968
Do we need to test spark 3.0 only for now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r697522296
##########
File path: pom.xml
##########
@@ -167,6 +167,8 @@
<hive.version>2.3.7</hive.version>
<iceberg.name>iceberg-spark3-runtime</iceberg.name>
<iceberg.version>0.12.0</iceberg.version>
+ <hudi.name>hudi-spark-bundle_2.12</hudi.name>
+ <hudi.version>0.9.0</hudi.version>
Review comment:
Sort alphabetically, move it to line 167
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909820127
> It say Hudi 0.9.0 doesn't support Spark 3.1 [apache/hudi#3554](https://github.com/apache/hudi/issues/3554)
support Spark-2.4.3+ & Spark 3.x, https://hudi.apache.org/docs/quick-start-guide/
Maybe still have some bugs in some aspects.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909912750
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#994](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (1313398) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/b382ed0dd87e9e19cab40f3e6b596c042ecdc066?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b382ed0) will **decrease** coverage by `0.00%`.
> The diff coverage is `100.00%`.
> :exclamation: Current head 1313398 differs from pull request most recent head 50e1ab0. Consider uploading reports for the commit 50e1ab0 to get more accurate results
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #994 +/- ##
============================================
- Coverage 80.06% 80.06% -0.01%
Complexity 11 11
============================================
Files 152 152
Lines 5553 5552 -1
Branches 644 644
============================================
- Hits 4446 4445 -1
Misses 725 725
Partials 382 382
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...la/org/apache/kyuubi/zookeeper/ZookeeperConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXpvb2tlZXBlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS96b29rZWVwZXIvWm9va2VlcGVyQ29uZi5zY2FsYQ==) | `100.00% <100.00%> (ø)` | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b382ed0...50e1ab0](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912791182
Stand on the Spark side, HUDI is a data source implementation, or just call it a plugin.
Thus, for the transitive dependencies of HUDI, like Hadoop, Curator, Parquet, Avro, ORC, Hive, we must **RESPECT** the Spark bundled versions, as Kyuubi support multi Spark versions, we need to separate those deps version by different Spark versions using maven profiles. e.g. use parquet 1.10.0 for Spark 3.0/3.1, but use parquet 1.12.0 for Spark 3.2(coming soon).
Besides on Spark, we should also respect the Kyuubi deps version, assume you are using IDEA, please open the `External Libraries`, there are lots of jars which has multi versions that exist in the classpath, which means a high risk of class conflicts.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] codecov-commenter edited a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909912750
# [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#994](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a04b2e4) into [master](https://codecov.io/gh/apache/incubator-kyuubi/commit/b382ed0dd87e9e19cab40f3e6b596c042ecdc066?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b382ed0) will **increase** coverage by `0.00%`.
> The diff coverage is `n/a`.
[](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #994 +/- ##
=========================================
Coverage 80.06% 80.06%
- Complexity 11 14 +3
=========================================
Files 152 154 +2
Lines 5553 5584 +31
Branches 644 647 +3
=========================================
+ Hits 4446 4471 +25
- Misses 725 729 +4
- Partials 382 384 +2
```
| [Impacted Files](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...la/org/apache/kyuubi/zookeeper/ZookeeperConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLXpvb2tlZXBlci9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS96b29rZWVwZXIvWm9va2VlcGVyQ29uZi5zY2FsYQ==) | `100.00% <0.00%> (ø)` | |
| [...in/scala/org/apache/kyuubi/sql/KyuubiSQLConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC9LeXV1YmlTUUxDb25mLnNjYWxh) | `100.00% <0.00%> (ø)` | |
| [...rg/apache/kyuubi/sql/KyuubiSparkSQLExtension.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC9LeXV1YmlTcGFya1NRTEV4dGVuc2lvbi5zY2FsYQ==) | `100.00% <0.00%> (ø)` | |
| [...ql/sqlclassification/KyuubiSqlClassification.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC9zcWxjbGFzc2lmaWNhdGlvbi9LeXV1YmlTcWxDbGFzc2lmaWNhdGlvbi5zY2FsYQ==) | `66.66% <0.00%> (ø)` | |
| [...sqlclassification/KyuubiGetSqlClassification.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-ZGV2L2t5dXViaS1leHRlbnNpb24tc3BhcmstMy0xL3NyYy9tYWluL3NjYWxhL29yZy9hcGFjaGUva3l1dWJpL3NxbC9zcWxjbGFzc2lmaWNhdGlvbi9LeXV1YmlHZXRTcWxDbGFzc2lmaWNhdGlvbi5zY2FsYQ==) | `73.33% <0.00%> (ø)` | |
| [...in/scala/org/apache/kyuubi/config/KyuubiConf.scala](https://codecov.io/gh/apache/incubator-kyuubi/pull/994/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-a3l1dWJpLWNvbW1vbi9zcmMvbWFpbi9zY2FsYS9vcmcvYXBhY2hlL2t5dXViaS9jb25maWcvS3l1dWJpQ29uZi5zY2FsYQ==) | `94.89% <0.00%> (+0.05%)` | :arrow_up: |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b382ed0...a04b2e4](https://codecov.io/gh/apache/incubator-kyuubi/pull/994?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on a change in pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on a change in pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#discussion_r701527235
##########
File path: kyuubi-server/pom.xml
##########
@@ -133,6 +133,42 @@
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.spark</groupId>
+ <artifactId>spark-avro_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hudi</groupId>
+ <artifactId>hudi-spark3_${scala.binary.version}</artifactId>
+ <scope>test</scope>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.hbase</groupId>
Review comment:
For index
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] pan3793 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
pan3793 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-909469702
Have a look at `hudi-spark-bundle_2.12`, I'm curious why hudi only relocate part of deps but left others(e.g. curator, prometheus, hbase, rocksdb) with the original package name in the fat jar? It leads to a very high probability of class conflict.
https://github.com/apache/hudi/blob/master/packaging/hudi-spark-bundle/pom.xml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 commented on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 commented on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-907056808
> Does hudi Spark SQL Extension only support Spark 3.1? Where can I find the support matrix or docs?
Support Spark-2.4.3+ & Spark 3.0, I have tested locally, spark 3.1 does not work
0.9.0 was released just a few days ago, I haven't yet found the release notes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] simon824 removed a comment on pull request #994: [KYUUBI #703] support hudi-0.9.0
Posted by GitBox <gi...@apache.org>.
simon824 removed a comment on pull request #994:
URL: https://github.com/apache/incubator-kyuubi/pull/994#issuecomment-912358794
> thanks, @simon824, this change LGTM, leave it to @pan3793 for the last comfirmation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org