You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "zhangyue19921010 (via GitHub)" <gi...@apache.org> on 2023/05/15 13:39:47 UTC
[GitHub] [hudi] zhangyue19921010 opened a new pull request, #8714: [Do Not Merge][WIP][Test] Hudi spark 3.0.x adoption
zhangyue19921010 opened a new pull request, #8714:
URL: https://github.com/apache/hudi/pull/8714
### Change Logs
Hudi spark 3.0.x adoption
### Impact
Hudi spark 3.0.x adoption
### Risk level (write none, low medium or high below)
low
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1198708847
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/avro/TestAvroSerDe.scala:
##########
@@ -20,11 +20,12 @@ package org.apache.spark.sql.avro
import org.apache.avro.generic.GenericData
import org.apache.hudi.SparkAdapterSupport
import org.apache.hudi.avro.model.{HoodieMetadataColumnStats, IntWrapper}
+import org.apache.spark.internal.Logging
import org.apache.spark.sql.avro.SchemaConverters.SchemaType
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Test
-class TestAvroSerDe extends SparkAdapterSupport {
+class TestAvroSerDe extends SparkAdapterSupport with Logging {
Review Comment:
Removed
##########
.github/workflows/bot.yml:
##########
@@ -63,6 +63,10 @@ jobs:
sparkProfile: "spark3.1"
sparkModules: "hudi-spark-datasource/hudi-spark3.1.x"
+ - scalaProfile: "scala-2.12"
Review Comment:
Sure. Base image for this new bundle validation is not ready. Will add this validation as next step.
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable.scala:
##########
@@ -114,58 +114,65 @@ class TestMergeIntoTable extends HoodieSparkSqlTestBase with ScalaAssertionSuppo
})
}
- test("Test MergeInto with more than once update actions") {
- withRecordType()(withTempDir {tmp =>
- val targetTable = generateTableName
- spark.sql(
- s"""
- |create table ${targetTable} (
- | id int,
- | name string,
- | data int,
- | country string,
- | ts bigint
- |) using hudi
- |tblproperties (
- | type = 'cow',
- | primaryKey = 'id',
- | preCombineField = 'ts'
- | )
- |partitioned by (country)
- |location '${tmp.getCanonicalPath}/$targetTable'
- |""".stripMargin)
- spark.sql(
- s"""
- |merge into ${targetTable} as target
- |using (
- |select 1 as id, 'lb' as name, 6 as data, 'shu' as country, 1646643193 as ts
- |) source
- |on source.id = target.id
- |when matched then
- |update set *
- |when not matched then
- |insert *
- |""".stripMargin)
- spark.sql(
- s"""
- |merge into ${targetTable} as target
- |using (
- |select 1 as id, 'lb' as name, 5 as data, 'shu' as country, 1646643196 as ts
- |) source
- |on source.id = target.id
- |when matched and source.data > target.data then
- |update set target.data = source.data, target.ts = source.ts
- |when matched and source.data = 5 then
- |update set target.data = source.data, target.ts = source.ts
- |when not matched then
- |insert *
- |""".stripMargin)
-
- checkAnswer(s"select id, name, data, country, ts from $targetTable")(
- Seq(1, "lb", 5, "shu", 1646643196L)
- )
+ /**
+ * For spark3.0.x didn't support 'UPDATE and DELETE can appear at most once in MATCHED clauses in a MERGE statement'
+ * details: org.apache.spark.sql.catalyst.parser.AstBuilder#visitMergeIntoTable
Review Comment:
Changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1554715795
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
}, {
"hash" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187",
"triggerID" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"triggerType" : "PUSH"
}, {
"hash" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17210",
"triggerID" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"triggerType" : "PUSH"
}, {
"hash" : "c4a5675de364a34774e3cecc0e625f92639382b4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17212",
"triggerID" : "c4a5675de364a34774e3cecc0e625f92639382b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* c4a5675de364a34774e3cecc0e625f92639382b4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1550856886
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 0c8facb84a9a4362986eafa17854faad647f4fde Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110)
* a315e98dc2039f19275943f7024035816cbf4448 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1549090116
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* b43956687e916bb0d0259f82c7846af74c018d1e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105)
* 0c8facb84a9a4362986eafa17854faad647f4fde Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212] Hudi Spark 3.0.x integration
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1199898331
##########
packaging/bundle-validation/ci_run.sh:
##########
@@ -41,6 +41,16 @@ if [[ ${SPARK_RUNTIME} == 'spark2.4' ]]; then
CONFLUENT_VERSION=5.5.12
KAFKA_CONNECT_HDFS_VERSION=10.1.13
IMAGE_TAG=flink1136hive239spark248
+elif [[ ${SPARK_RUNTIME} == 'spark3.0.2' ]]; then
+ HADOOP_VERSION=2.7.7
+ HIVE_VERSION=3.1.3
+ DERBY_VERSION=10.14.1.0
+ FLINK_VERSION=1.14.6
+ SPARK_VERSION=3.0.2
+ SPARK_HADOOP_VERSION=2.7
+ CONFLUENT_VERSION=5.5.12
+ KAFKA_CONNECT_HDFS_VERSION=10.1.13
+ IMAGE_TAG=flink1146hive313spark302
elif [[ ${SPARK_RUNTIME} == 'spark3.1.3' ]]; then
Review Comment:
Please refer this PR
https://github.com/apache/hudi/pull/8767
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1551093980
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* a315e98dc2039f19275943f7024035816cbf4448 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141)
* 172e9f2ab16b6e7358d4970e8b7da4cf22cc0396 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1548628486
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* d3f558c7c1ff3ab0169ff27d11955df48c3b3459 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1552483947
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 172e9f2ab16b6e7358d4970e8b7da4cf22cc0396 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151)
* cd96d898ad095e994bf3d548914e4febb13a1e8c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8714: [HUDI-6212] Hudi Spark 3.0.x integration
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1199297595
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.avro
+
+import org.apache.avro.Conversions.DecimalConversion
+import org.apache.avro.LogicalTypes.{TimestampMicros, TimestampMillis}
+import org.apache.avro.Schema.Type._
+import org.apache.avro.generic._
+import org.apache.avro.util.Utf8
+import org.apache.avro.{LogicalTypes, Schema, SchemaBuilder}
+import org.apache.spark.sql.avro.AvroDeserializer.{createDateRebaseFuncInRead, createTimestampRebaseFuncInRead}
+import org.apache.spark.sql.catalyst.expressions.{SpecificInternalRow, UnsafeArrayData}
+import org.apache.spark.sql.catalyst.util.DateTimeConstants.MILLIS_PER_DAY
+import org.apache.spark.sql.catalyst.util._
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources.DataSourceUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+import java.math.BigDecimal
+import java.nio.ByteBuffer
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * A deserializer to deserialize data in avro format to data in catalyst format.
+ *
+ * NOTE: This code is borrowed from Spark 3.1.2
+ * This code is borrowed, so that we can better control compatibility w/in Spark minor
+ * branches (3.2.x, 3.1.x, etc)
+ *
+ * PLEASE REFRAIN MAKING ANY CHANGES TO THIS CODE UNLESS ABSOLUTELY NECESSARY
+ */
+private[sql] class AvroDeserializer(rootAvroType: Schema,
Review Comment:
Sg.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1547909254
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1198728500
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/adapter/Spark3_0Adapter.scala:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.adapter
+
+import org.apache.avro.Schema
+import org.apache.hudi.Spark30HoodieFileScanRDD
+import org.apache.spark.sql.avro.{HoodieAvroDeserializer, HoodieAvroSerializer, HoodieSpark3_0AvroDeserializer, HoodieSpark3_0AvroSerializer}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression}
+import org.apache.spark.sql.catalyst.parser.ParserInterface
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.execution.datasources.parquet.{ParquetFileFormat, Spark30HoodieParquetFileFormat}
+import org.apache.spark.sql.execution.datasources.{FilePartition, FileScanRDD, PartitionedFile}
+import org.apache.spark.sql.hudi.SparkAdapter
+import org.apache.spark.sql.parser.{HoodieExtendedParserInterface, HoodieSpark3_0ExtendedSqlParser}
+import org.apache.spark.sql.types.{DataType, Metadata, MetadataBuilder, StructType}
+import org.apache.spark.sql.vectorized.ColumnarUtils
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.storage.StorageLevel._
+
+/**
+ * Implementation of [[SparkAdapter]] for Spark 3.0.x
+ */
+class Spark3_0Adapter extends BaseSpark3Adapter {
+
+ override def resolveHoodieTable(plan: LogicalPlan): Option[CatalogTable] = {
+ super.resolveHoodieTable(plan).orElse {
+ EliminateSubqueryAliases(plan) match {
+ // First, we need to weed out unresolved plans
+ case plan if !plan.resolved => None
+ // NOTE: When resolving Hudi table we allow [[Filter]]s and [[Project]]s be applied
+ // on top of it
+ case PhysicalOperation(_, _, DataSourceV2Relation(table: CatalogTable, _, _, _, _)) if isHoodieTable(table) =>
+ Some(table)
+ case _ => None
Review Comment:
But as far as I know, both hudi write and read in spark3.0 are not use V2, so it doesn't matter to do fallback here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1551105452
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* a315e98dc2039f19275943f7024035816cbf4448 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141)
* 172e9f2ab16b6e7358d4970e8b7da4cf22cc0396 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1198414998
##########
.github/workflows/bot.yml:
##########
@@ -63,6 +63,10 @@ jobs:
sparkProfile: "spark3.1"
sparkModules: "hudi-spark-datasource/hudi-spark3.1.x"
+ - scalaProfile: "scala-2.12"
Review Comment:
Would be good to add bundle validation on Spark 3.0.x too in `validate-bundles` section
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable.scala:
##########
@@ -114,58 +114,65 @@ class TestMergeIntoTable extends HoodieSparkSqlTestBase with ScalaAssertionSuppo
})
}
- test("Test MergeInto with more than once update actions") {
- withRecordType()(withTempDir {tmp =>
- val targetTable = generateTableName
- spark.sql(
- s"""
- |create table ${targetTable} (
- | id int,
- | name string,
- | data int,
- | country string,
- | ts bigint
- |) using hudi
- |tblproperties (
- | type = 'cow',
- | primaryKey = 'id',
- | preCombineField = 'ts'
- | )
- |partitioned by (country)
- |location '${tmp.getCanonicalPath}/$targetTable'
- |""".stripMargin)
- spark.sql(
- s"""
- |merge into ${targetTable} as target
- |using (
- |select 1 as id, 'lb' as name, 6 as data, 'shu' as country, 1646643193 as ts
- |) source
- |on source.id = target.id
- |when matched then
- |update set *
- |when not matched then
- |insert *
- |""".stripMargin)
- spark.sql(
- s"""
- |merge into ${targetTable} as target
- |using (
- |select 1 as id, 'lb' as name, 5 as data, 'shu' as country, 1646643196 as ts
- |) source
- |on source.id = target.id
- |when matched and source.data > target.data then
- |update set target.data = source.data, target.ts = source.ts
- |when matched and source.data = 5 then
- |update set target.data = source.data, target.ts = source.ts
- |when not matched then
- |insert *
- |""".stripMargin)
-
- checkAnswer(s"select id, name, data, country, ts from $targetTable")(
- Seq(1, "lb", 5, "shu", 1646643196L)
- )
+ /**
+ * For spark3.0.x didn't support 'UPDATE and DELETE can appear at most once in MATCHED clauses in a MERGE statement'
+ * details: org.apache.spark.sql.catalyst.parser.AstBuilder#visitMergeIntoTable
Review Comment:
```suggestion
* In Spark 3.0.x, UPDATE and DELETE can appear at most once in MATCHED clauses in a MERGE INTO statement.
* Refer to: `org.apache.spark.sql.catalyst.parser.AstBuilder#visitMergeIntoTable`
```
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/avro/TestAvroSerDe.scala:
##########
@@ -20,11 +20,12 @@ package org.apache.spark.sql.avro
import org.apache.avro.generic.GenericData
import org.apache.hudi.SparkAdapterSupport
import org.apache.hudi.avro.model.{HoodieMetadataColumnStats, IntWrapper}
+import org.apache.spark.internal.Logging
import org.apache.spark.sql.avro.SchemaConverters.SchemaType
import org.junit.jupiter.api.Assertions.assertEquals
import org.junit.jupiter.api.Test
-class TestAvroSerDe extends SparkAdapterSupport {
+class TestAvroSerDe extends SparkAdapterSupport with Logging {
Review Comment:
Is this necessary for testing?
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystExpressionUtils.scala:
##########
@@ -17,16 +17,9 @@
package org.apache.spark.sql
-import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression, Predicate, PredicateHelper}
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-
-trait HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils
- with PredicateHelper {
-
- override def normalizeExprs(exprs: Seq[Expression], attributes: Seq[Attribute]): Seq[Expression] =
- DataSourceStrategy.normalizeExprs(exprs, attributes)
-
- override def extractPredicatesWithinOutputSet(condition: Expression,
- outputSet: AttributeSet): Option[Expression] =
- super[PredicateHelper].extractPredicatesWithinOutputSet(condition, outputSet)
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression}
+abstract class HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils {
Review Comment:
could we keep `with PredicateHelper`?
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/java/org/apache/spark/sql/execution/datasources/parquet/Spark30HoodieVectorizedParquetRecordReader.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.hadoop.mapreduce.InputSplit;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hudi.client.utils.SparkInternalSchemaConverter;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.HashMap;
+import java.util.Map;
+
+public class Spark30HoodieVectorizedParquetRecordReader extends VectorizedParquetRecordReader {
Review Comment:
I assume these cases are mostly copied from Hudi Spark 3.1.x module.
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/hudi/Spark30ResolveHudiAlterTableCommand.scala:
##########
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.hudi
+
+import org.apache.hudi.common.config.HoodieCommonConfig
+import org.apache.hudi.internal.schema.action.TableChange.ColumnChangeID
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.TableChange._
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.hudi.command.Spark30AlterTableCommand
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, NullType, StructType}
+import org.apache.spark.sql.{AnalysisException, SparkSession}
+
+import java.util.Locale
+import scala.collection.mutable
+
+/**
+ * Rule to mostly resolve, normalize and rewrite column names based on case sensitivity
+ * for alter table column commands.
+ * TODO: we should remove this file when we support datasourceV2 for hoodie on spark3.0x
Review Comment:
```suggestion
* TODO: we should remove this file when we support datasourceV2 for hoodie on spark3.0.x
```
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.avro
+
+import org.apache.avro.Conversions.DecimalConversion
+import org.apache.avro.LogicalTypes.{TimestampMicros, TimestampMillis}
+import org.apache.avro.Schema.Type._
+import org.apache.avro.generic._
+import org.apache.avro.util.Utf8
+import org.apache.avro.{LogicalTypes, Schema, SchemaBuilder}
+import org.apache.spark.sql.avro.AvroDeserializer.{createDateRebaseFuncInRead, createTimestampRebaseFuncInRead}
+import org.apache.spark.sql.catalyst.expressions.{SpecificInternalRow, UnsafeArrayData}
+import org.apache.spark.sql.catalyst.util.DateTimeConstants.MILLIS_PER_DAY
+import org.apache.spark.sql.catalyst.util._
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources.DataSourceUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+import java.math.BigDecimal
+import java.nio.ByteBuffer
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * A deserializer to deserialize data in avro format to data in catalyst format.
+ *
+ * NOTE: This code is borrowed from Spark 3.1.2
+ * This code is borrowed, so that we can better control compatibility w/in Spark minor
+ * branches (3.2.x, 3.1.x, etc)
+ *
+ * PLEASE REFRAIN MAKING ANY CHANGES TO THIS CODE UNLESS ABSOLUTELY NECESSARY
+ */
+private[sql] class AvroDeserializer(rootAvroType: Schema,
Review Comment:
Should the code of the class from Spark 3.0.x be copied to here, instead of using Spark 3.1.2 version?
##########
pom.xml:
##########
@@ -2238,6 +2239,44 @@
</activation>
</profile>
+ <profile>
+ <id>spark3.0</id>
+ <properties>
+ <spark3.version>${spark30.version}</spark3.version>
+ <spark.version>${spark3.version}</spark.version>
+ <sparkbundle.version>3.0</sparkbundle.version>
+ <scala.version>${scala12.version}</scala.version>
+ <scala.binary.version>2.12</scala.binary.version>
+ <hudi.spark.module>hudi-spark3.0.x</hudi.spark.module>
+ <hudi.spark.common.modules.1>hudi-spark3-common</hudi.spark.common.modules.1>
+ <kafka.version>${kafka.spark3.version}</kafka.version>
Review Comment:
Do we need `<scalatest.version>${scalatest.spark3.version}</scalatest.version>`?
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/adapter/Spark3_0Adapter.scala:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.adapter
+
+import org.apache.avro.Schema
+import org.apache.hudi.Spark30HoodieFileScanRDD
+import org.apache.spark.sql.avro.{HoodieAvroDeserializer, HoodieAvroSerializer, HoodieSpark3_0AvroDeserializer, HoodieSpark3_0AvroSerializer}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression}
+import org.apache.spark.sql.catalyst.parser.ParserInterface
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.execution.datasources.parquet.{ParquetFileFormat, Spark30HoodieParquetFileFormat}
+import org.apache.spark.sql.execution.datasources.{FilePartition, FileScanRDD, PartitionedFile}
+import org.apache.spark.sql.hudi.SparkAdapter
+import org.apache.spark.sql.parser.{HoodieExtendedParserInterface, HoodieSpark3_0ExtendedSqlParser}
+import org.apache.spark.sql.types.{DataType, Metadata, MetadataBuilder, StructType}
+import org.apache.spark.sql.vectorized.ColumnarUtils
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.storage.StorageLevel._
+
+/**
+ * Implementation of [[SparkAdapter]] for Spark 3.0.x
+ */
+class Spark3_0Adapter extends BaseSpark3Adapter {
+
+ override def resolveHoodieTable(plan: LogicalPlan): Option[CatalogTable] = {
+ super.resolveHoodieTable(plan).orElse {
+ EliminateSubqueryAliases(plan) match {
+ // First, we need to weed out unresolved plans
+ case plan if !plan.resolved => None
+ // NOTE: When resolving Hudi table we allow [[Filter]]s and [[Project]]s be applied
+ // on top of it
+ case PhysicalOperation(_, _, DataSourceV2Relation(table: CatalogTable, _, _, _, _)) if isHoodieTable(table) =>
+ Some(table)
+ case _ => None
Review Comment:
Is there a way to achieve fallback from V2 to V1 in Spark 3.0? Without the fallback, the performance is affected.
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/hudi/command/Spark30AlterTableCommand.scala:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command
+
+import java.net.URI
+import java.nio.charset.StandardCharsets
+import java.util
+import java.util.concurrent.atomic.AtomicInteger
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.avro.AvroSchemaUtils.getAvroRecordQualifiedName
+import org.apache.hudi.client.utils.SparkInternalSchemaConverter
+import org.apache.hudi.common.model.{HoodieCommitMetadata, WriteOperationType}
+import org.apache.hudi.{AvroConversionUtils, DataSourceOptionsHelper, DataSourceUtils}
+import org.apache.hudi.common.table.timeline.{HoodieActiveTimeline, HoodieInstant}
+import org.apache.hudi.common.table.timeline.HoodieInstant.State
+import org.apache.hudi.common.table.{HoodieTableMetaClient, TableSchemaResolver}
+import org.apache.hudi.common.util.{CommitUtils, Option}
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.internal.schema.InternalSchema
+import org.apache.hudi.internal.schema.action.TableChange.ColumnChangeID
+import org.apache.hudi.internal.schema.action.TableChanges
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter
+import org.apache.hudi.internal.schema.utils.{SchemaChangeUtils, SerDeHelper}
+import org.apache.hudi.internal.schema.io.FileBasedInternalSchemaStorageManager
+import org.apache.hudi.table.HoodieSparkTable
+import org.apache.spark.api.java.JavaSparkContext
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.catalog.{CatalogTable, CatalogTableType}
+import org.apache.spark.sql.connector.catalog.{TableCatalog, TableChange}
+import org.apache.spark.sql.connector.catalog.TableChange.{AddColumn, DeleteColumn, RemoveProperty, SetProperty}
+import org.apache.spark.sql.execution.command.RunnableCommand
+import org.apache.spark.sql.types.StructType
+
+import scala.collection.JavaConverters._
+import scala.util.control.NonFatal
+
+// TODO: we should remove this file when we support datasourceV2 for hoodie on spark3.0x
Review Comment:
```suggestion
// TODO: we should remove this file when we support datasourceV2 for hoodie on spark3.0.x
```
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/HoodieSpark30CatalogUtils.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.connector.expressions.{BucketTransform, NamedReference, Transform}
+
+object HoodieSpark30CatalogUtils extends HoodieSpark3CatalogUtils {
+
+ override def unapplyBucketTransform(t: Transform): Option[(Int, Seq[NamedReference], Seq[NamedReference])] =
+ t match {
+ case BucketTransform(numBuckets, ref) => Some(numBuckets, Seq(ref), Seq.empty)
+ case _ => None
+ }
+
Review Comment:
nit: remove new line
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/Spark30NestedSchemaPruning.scala:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import org.apache.hudi.{HoodieBaseRelation, SparkAdapterSupport}
+import org.apache.spark.sql.HoodieSpark3CatalystPlanUtils
+import org.apache.spark.sql.catalyst.expressions.{Alias, And, Attribute, AttributeReference, AttributeSet, Expression, NamedExpression, ProjectionOverSchema}
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.sources.BaseRelation
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructField, StructType}
+
+/**
+ * Prunes unnecessary physical columns given a [[PhysicalOperation]] over a data source relation.
+ * By "physical column", we mean a column as defined in the data source format like Parquet format
+ * or ORC format. For example, in Spark SQL, a root-level Parquet column corresponds to a SQL
+ * column, and a nested Parquet column corresponds to a [[StructField]].
+ *
+ * NOTE: This class is borrowed from Spark 3.2.1, with modifications adapting it to handle [[HoodieBaseRelation]],
+ * instead of [[HadoopFsRelation]]
+ */
+class Spark30NestedSchemaPruning extends Rule[LogicalPlan] {
+ import org.apache.spark.sql.catalyst.expressions.SchemaPruning._
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ if (SQLConf.get.nestedSchemaPruningEnabled) {
+ apply0(plan)
+ } else {
+ plan
+ }
+
+ private def apply0(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ // NOTE: This is modified to accommodate for Hudi's custom relations, given that original
+ // [[NestedSchemaPruning]] rule is tightly coupled w/ [[HadoopFsRelation]]
+ // TODO generalize to any file-based relation
+ l @ LogicalRelation(relation: HoodieBaseRelation, _, _, _))
+ if relation.canPruneRelationSchema =>
+
+ prunePhysicalColumns(l.output, projects, filters, relation.dataSchema,
+ prunedDataSchema => {
+ val prunedRelation =
+ relation.updatePrunedDataSchema(prunedSchema = prunedDataSchema)
+ buildPrunedRelation(l, prunedRelation)
+ }).getOrElse(op)
+ }
+
+ /**
+ * This method returns optional logical plan. `None` is returned if no nested field is required or
+ * all nested fields are required.
+ */
+ private def prunePhysicalColumns(output: Seq[AttributeReference],
+ projects: Seq[NamedExpression],
+ filters: Seq[Expression],
+ dataSchema: StructType,
+ outputRelationBuilder: StructType => LogicalRelation): Option[LogicalPlan] = {
+ val (normalizedProjects, normalizedFilters) =
+ normalizeAttributeRefNames(output, projects, filters)
+ val requestedRootFields = identifyRootFields(normalizedProjects, normalizedFilters)
+
+ // If requestedRootFields includes a nested field, continue. Otherwise,
+ // return op
+ if (requestedRootFields.exists { root: RootField => !root.derivedFromAtt }) {
+ val prunedDataSchema = pruneDataSchema(dataSchema, requestedRootFields)
+
+ // If the data schema is different from the pruned data schema, continue. Otherwise,
+ // return op. We effect this comparison by counting the number of "leaf" fields in
+ // each schemata, assuming the fields in prunedDataSchema are a subset of the fields
+ // in dataSchema.
+ if (countLeaves(dataSchema) > countLeaves(prunedDataSchema)) {
+ val planUtils = SparkAdapterSupport.sparkAdapter.getCatalystPlanUtils.asInstanceOf[HoodieSpark3CatalystPlanUtils]
+
+ val prunedRelation = outputRelationBuilder(prunedDataSchema)
+ val projectionOverSchema = planUtils.projectOverSchema(prunedDataSchema, AttributeSet(output))
+
+ Some(buildNewProjection(projects, normalizedProjects, normalizedFilters,
+ prunedRelation, projectionOverSchema))
+ } else {
+ None
+ }
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes the names of the attribute references in the given projects and filters to reflect
+ * the names in the given logical relation. This makes it possible to compare attributes and
+ * fields by name. Returns a tuple with the normalized projects and filters, respectively.
+ */
+ private def normalizeAttributeRefNames(output: Seq[AttributeReference],
+ projects: Seq[NamedExpression],
+ filters: Seq[Expression]): (Seq[NamedExpression], Seq[Expression]) = {
+ val normalizedAttNameMap = output.map(att => (att.exprId, att.name)).toMap
+ val normalizedProjects = projects.map(_.transform {
+ case att: AttributeReference if normalizedAttNameMap.contains(att.exprId) =>
+ att.withName(normalizedAttNameMap(att.exprId))
+ }).map { case expr: NamedExpression => expr }
+ val normalizedFilters = filters.map(_.transform {
+ case att: AttributeReference if normalizedAttNameMap.contains(att.exprId) =>
+ att.withName(normalizedAttNameMap(att.exprId))
+ })
+ (normalizedProjects, normalizedFilters)
+ }
+
+ /**
+ * Builds the new output [[Project]] Spark SQL operator that has the `leafNode`.
+ */
+ private def buildNewProjection(projects: Seq[NamedExpression],
+ normalizedProjects: Seq[NamedExpression],
+ filters: Seq[Expression],
+ prunedRelation: LogicalRelation,
+ projectionOverSchema: ProjectionOverSchema): Project = {
+ // Construct a new target for our projection by rewriting and
+ // including the original filters where available
+ val projectionChild =
+ if (filters.nonEmpty) {
+ val projectedFilters = filters.map(_.transformDown {
+ case projectionOverSchema(expr) => expr
+ })
+ val newFilterCondition = projectedFilters.reduce(And)
+ Filter(newFilterCondition, prunedRelation)
+ } else {
+ prunedRelation
+ }
+
+ // Construct the new projections of our Project by
+ // rewriting the original projections
+ val newProjects = normalizedProjects.map(_.transformDown {
+ case projectionOverSchema(expr) => expr
+ }).map { case expr: NamedExpression => expr }
+
+ if (log.isDebugEnabled) {
+ logDebug(s"New projects:\n${newProjects.map(_.treeString).mkString("\n")}")
+ }
+
+ Project(restoreOriginalOutputNames(newProjects, projects.map(_.name)), projectionChild)
+ }
+
+ /**
+ * Builds a pruned logical relation from the output of the output relation and the schema of the
+ * pruned base relation.
+ */
+ private def buildPrunedRelation(outputRelation: LogicalRelation,
+ prunedBaseRelation: BaseRelation): LogicalRelation = {
+ val prunedOutput = getPrunedOutput(outputRelation.output, prunedBaseRelation.schema)
+ outputRelation.copy(relation = prunedBaseRelation, output = prunedOutput)
+ }
+
+ // Prune the given output to make it consistent with `requiredSchema`.
+ private def getPrunedOutput(output: Seq[AttributeReference],
+ requiredSchema: StructType): Seq[AttributeReference] = {
+ // We need to replace the expression ids of the pruned relation output attributes
+ // with the expression ids of the original relation output attributes so that
+ // references to the original relation's output are not broken
+ val outputIdMap = output.map(att => (att.name, att.exprId)).toMap
+ requiredSchema
+ .toAttributes
+ .map {
+ case att if outputIdMap.contains(att.name) =>
+ att.withExprId(outputIdMap(att.name))
+ case att => att
+ }
+ }
+
+ /**
+ * Counts the "leaf" fields of the given dataType. Informally, this is the
+ * number of fields of non-complex data type in the tree representation of
+ * [[DataType]].
+ */
+ private def countLeaves(dataType: DataType): Int = {
+ dataType match {
+ case array: ArrayType => countLeaves(array.elementType)
+ case map: MapType => countLeaves(map.keyType) + countLeaves(map.valueType)
+ case struct: StructType =>
+ struct.map(field => countLeaves(field.dataType)).sum
+ case _ => 1
+ }
+ }
+
+ private def restoreOriginalOutputNames(
+ projectList: Seq[NamedExpression],
+ originalNames: Seq[String]): Seq[NamedExpression] = {
+ projectList.zip(originalNames).map {
+ case (attr: Attribute, name) => attr.withName(name)
+ case (alias: Alias, name) => if (name == alias.name) {
+ alias
+ } else {
+ AttributeReference(name, alias.dataType, alias.nullable, alias.metadata)(alias.exprId, alias.qualifier)
+ }
+ case (other, _) => other
+ }
+ }
+
+
Review Comment:
```suggestion
// NOTE: `pruneDataSchema` and `sortLeftFieldsByRight` functions are copied from Spark 3.1.2,
// as these functions in `SchemaPruning` have bugs in Spark 3.0.2 (see SPARK-35096,
// https://github.com/apache/spark/commit/2bbe0a4151f2af00f1105489d5757be28ff278d6)
```
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystExpressionUtils.scala:
##########
@@ -17,16 +17,9 @@
package org.apache.spark.sql
-import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression, Predicate, PredicateHelper}
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-
-trait HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils
- with PredicateHelper {
-
- override def normalizeExprs(exprs: Seq[Expression], attributes: Seq[Attribute]): Seq[Expression] =
- DataSourceStrategy.normalizeExprs(exprs, attributes)
-
- override def extractPredicatesWithinOutputSet(condition: Expression,
- outputSet: AttributeSet): Option[Expression] =
- super[PredicateHelper].extractPredicatesWithinOutputSet(condition, outputSet)
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression}
+abstract class HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils {
+ def normalizeExprs(exprs: Seq[Expression], attributes: Seq[Attribute]): Seq[Expression]
+ def extractPredicatesWithinOutputSet(condition: Expression,
+ outputSet: AttributeSet): Option[Expression]
Review Comment:
Scala docs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1198715332
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/avro/AvroDeserializer.scala:
##########
@@ -0,0 +1,492 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.avro
+
+import org.apache.avro.Conversions.DecimalConversion
+import org.apache.avro.LogicalTypes.{TimestampMicros, TimestampMillis}
+import org.apache.avro.Schema.Type._
+import org.apache.avro.generic._
+import org.apache.avro.util.Utf8
+import org.apache.avro.{LogicalTypes, Schema, SchemaBuilder}
+import org.apache.spark.sql.avro.AvroDeserializer.{createDateRebaseFuncInRead, createTimestampRebaseFuncInRead}
+import org.apache.spark.sql.catalyst.expressions.{SpecificInternalRow, UnsafeArrayData}
+import org.apache.spark.sql.catalyst.util.DateTimeConstants.MILLIS_PER_DAY
+import org.apache.spark.sql.catalyst.util._
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.execution.datasources.DataSourceUtils
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.internal.SQLConf.LegacyBehaviorPolicy
+import org.apache.spark.sql.types._
+import org.apache.spark.unsafe.types.UTF8String
+
+import java.math.BigDecimal
+import java.nio.ByteBuffer
+import scala.collection.JavaConverters._
+import scala.collection.mutable.ArrayBuffer
+
+/**
+ * A deserializer to deserialize data in avro format to data in catalyst format.
+ *
+ * NOTE: This code is borrowed from Spark 3.1.2
+ * This code is borrowed, so that we can better control compatibility w/in Spark minor
+ * branches (3.2.x, 3.1.x, etc)
+ *
+ * PLEASE REFRAIN MAKING ANY CHANGES TO THIS CODE UNLESS ABSOLUTELY NECESSARY
+ */
+private[sql] class AvroDeserializer(rootAvroType: Schema,
Review Comment:
we need to make AvroDeserializer, AvroSerializer and SchemaConverters consistent.
For example make UT `TestAvroSerDe#testAvroUnionSerDe` happy.
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/execution/datasources/Spark30NestedSchemaPruning.scala:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources
+
+import org.apache.hudi.{HoodieBaseRelation, SparkAdapterSupport}
+import org.apache.spark.sql.HoodieSpark3CatalystPlanUtils
+import org.apache.spark.sql.catalyst.expressions.{Alias, And, Attribute, AttributeReference, AttributeSet, Expression, NamedExpression, ProjectionOverSchema}
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.catalyst.plans.logical.{Filter, LogicalPlan, Project}
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.sources.BaseRelation
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructField, StructType}
+
+/**
+ * Prunes unnecessary physical columns given a [[PhysicalOperation]] over a data source relation.
+ * By "physical column", we mean a column as defined in the data source format like Parquet format
+ * or ORC format. For example, in Spark SQL, a root-level Parquet column corresponds to a SQL
+ * column, and a nested Parquet column corresponds to a [[StructField]].
+ *
+ * NOTE: This class is borrowed from Spark 3.2.1, with modifications adapting it to handle [[HoodieBaseRelation]],
+ * instead of [[HadoopFsRelation]]
+ */
+class Spark30NestedSchemaPruning extends Rule[LogicalPlan] {
+ import org.apache.spark.sql.catalyst.expressions.SchemaPruning._
+ override def apply(plan: LogicalPlan): LogicalPlan =
+ if (SQLConf.get.nestedSchemaPruningEnabled) {
+ apply0(plan)
+ } else {
+ plan
+ }
+
+ private def apply0(plan: LogicalPlan): LogicalPlan =
+ plan transformDown {
+ case op @ PhysicalOperation(projects, filters,
+ // NOTE: This is modified to accommodate for Hudi's custom relations, given that original
+ // [[NestedSchemaPruning]] rule is tightly coupled w/ [[HadoopFsRelation]]
+ // TODO generalize to any file-based relation
+ l @ LogicalRelation(relation: HoodieBaseRelation, _, _, _))
+ if relation.canPruneRelationSchema =>
+
+ prunePhysicalColumns(l.output, projects, filters, relation.dataSchema,
+ prunedDataSchema => {
+ val prunedRelation =
+ relation.updatePrunedDataSchema(prunedSchema = prunedDataSchema)
+ buildPrunedRelation(l, prunedRelation)
+ }).getOrElse(op)
+ }
+
+ /**
+ * This method returns optional logical plan. `None` is returned if no nested field is required or
+ * all nested fields are required.
+ */
+ private def prunePhysicalColumns(output: Seq[AttributeReference],
+ projects: Seq[NamedExpression],
+ filters: Seq[Expression],
+ dataSchema: StructType,
+ outputRelationBuilder: StructType => LogicalRelation): Option[LogicalPlan] = {
+ val (normalizedProjects, normalizedFilters) =
+ normalizeAttributeRefNames(output, projects, filters)
+ val requestedRootFields = identifyRootFields(normalizedProjects, normalizedFilters)
+
+ // If requestedRootFields includes a nested field, continue. Otherwise,
+ // return op
+ if (requestedRootFields.exists { root: RootField => !root.derivedFromAtt }) {
+ val prunedDataSchema = pruneDataSchema(dataSchema, requestedRootFields)
+
+ // If the data schema is different from the pruned data schema, continue. Otherwise,
+ // return op. We effect this comparison by counting the number of "leaf" fields in
+ // each schemata, assuming the fields in prunedDataSchema are a subset of the fields
+ // in dataSchema.
+ if (countLeaves(dataSchema) > countLeaves(prunedDataSchema)) {
+ val planUtils = SparkAdapterSupport.sparkAdapter.getCatalystPlanUtils.asInstanceOf[HoodieSpark3CatalystPlanUtils]
+
+ val prunedRelation = outputRelationBuilder(prunedDataSchema)
+ val projectionOverSchema = planUtils.projectOverSchema(prunedDataSchema, AttributeSet(output))
+
+ Some(buildNewProjection(projects, normalizedProjects, normalizedFilters,
+ prunedRelation, projectionOverSchema))
+ } else {
+ None
+ }
+ } else {
+ None
+ }
+ }
+
+ /**
+ * Normalizes the names of the attribute references in the given projects and filters to reflect
+ * the names in the given logical relation. This makes it possible to compare attributes and
+ * fields by name. Returns a tuple with the normalized projects and filters, respectively.
+ */
+ private def normalizeAttributeRefNames(output: Seq[AttributeReference],
+ projects: Seq[NamedExpression],
+ filters: Seq[Expression]): (Seq[NamedExpression], Seq[Expression]) = {
+ val normalizedAttNameMap = output.map(att => (att.exprId, att.name)).toMap
+ val normalizedProjects = projects.map(_.transform {
+ case att: AttributeReference if normalizedAttNameMap.contains(att.exprId) =>
+ att.withName(normalizedAttNameMap(att.exprId))
+ }).map { case expr: NamedExpression => expr }
+ val normalizedFilters = filters.map(_.transform {
+ case att: AttributeReference if normalizedAttNameMap.contains(att.exprId) =>
+ att.withName(normalizedAttNameMap(att.exprId))
+ })
+ (normalizedProjects, normalizedFilters)
+ }
+
+ /**
+ * Builds the new output [[Project]] Spark SQL operator that has the `leafNode`.
+ */
+ private def buildNewProjection(projects: Seq[NamedExpression],
+ normalizedProjects: Seq[NamedExpression],
+ filters: Seq[Expression],
+ prunedRelation: LogicalRelation,
+ projectionOverSchema: ProjectionOverSchema): Project = {
+ // Construct a new target for our projection by rewriting and
+ // including the original filters where available
+ val projectionChild =
+ if (filters.nonEmpty) {
+ val projectedFilters = filters.map(_.transformDown {
+ case projectionOverSchema(expr) => expr
+ })
+ val newFilterCondition = projectedFilters.reduce(And)
+ Filter(newFilterCondition, prunedRelation)
+ } else {
+ prunedRelation
+ }
+
+ // Construct the new projections of our Project by
+ // rewriting the original projections
+ val newProjects = normalizedProjects.map(_.transformDown {
+ case projectionOverSchema(expr) => expr
+ }).map { case expr: NamedExpression => expr }
+
+ if (log.isDebugEnabled) {
+ logDebug(s"New projects:\n${newProjects.map(_.treeString).mkString("\n")}")
+ }
+
+ Project(restoreOriginalOutputNames(newProjects, projects.map(_.name)), projectionChild)
+ }
+
+ /**
+ * Builds a pruned logical relation from the output of the output relation and the schema of the
+ * pruned base relation.
+ */
+ private def buildPrunedRelation(outputRelation: LogicalRelation,
+ prunedBaseRelation: BaseRelation): LogicalRelation = {
+ val prunedOutput = getPrunedOutput(outputRelation.output, prunedBaseRelation.schema)
+ outputRelation.copy(relation = prunedBaseRelation, output = prunedOutput)
+ }
+
+ // Prune the given output to make it consistent with `requiredSchema`.
+ private def getPrunedOutput(output: Seq[AttributeReference],
+ requiredSchema: StructType): Seq[AttributeReference] = {
+ // We need to replace the expression ids of the pruned relation output attributes
+ // with the expression ids of the original relation output attributes so that
+ // references to the original relation's output are not broken
+ val outputIdMap = output.map(att => (att.name, att.exprId)).toMap
+ requiredSchema
+ .toAttributes
+ .map {
+ case att if outputIdMap.contains(att.name) =>
+ att.withExprId(outputIdMap(att.name))
+ case att => att
+ }
+ }
+
+ /**
+ * Counts the "leaf" fields of the given dataType. Informally, this is the
+ * number of fields of non-complex data type in the tree representation of
+ * [[DataType]].
+ */
+ private def countLeaves(dataType: DataType): Int = {
+ dataType match {
+ case array: ArrayType => countLeaves(array.elementType)
+ case map: MapType => countLeaves(map.keyType) + countLeaves(map.valueType)
+ case struct: StructType =>
+ struct.map(field => countLeaves(field.dataType)).sum
+ case _ => 1
+ }
+ }
+
+ private def restoreOriginalOutputNames(
+ projectList: Seq[NamedExpression],
+ originalNames: Seq[String]): Seq[NamedExpression] = {
+ projectList.zip(originalNames).map {
+ case (attr: Attribute, name) => attr.withName(name)
+ case (alias: Alias, name) => if (name == alias.name) {
+ alias
+ } else {
+ AttributeReference(name, alias.dataType, alias.nullable, alias.metadata)(alias.exprId, alias.qualifier)
+ }
+ case (other, _) => other
+ }
+ }
+
+
Review Comment:
changed.
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/hudi/Spark30ResolveHudiAlterTableCommand.scala:
##########
@@ -0,0 +1,297 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.spark.sql.hudi
+
+import org.apache.hudi.common.config.HoodieCommonConfig
+import org.apache.hudi.internal.schema.action.TableChange.ColumnChangeID
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.TableChange._
+import org.apache.spark.sql.connector.catalog.TableChange
+import org.apache.spark.sql.hudi.command.Spark30AlterTableCommand
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{ArrayType, DataType, MapType, NullType, StructType}
+import org.apache.spark.sql.{AnalysisException, SparkSession}
+
+import java.util.Locale
+import scala.collection.mutable
+
+/**
+ * Rule to mostly resolve, normalize and rewrite column names based on case sensitivity
+ * for alter table column commands.
+ * TODO: we should remove this file when we support datasourceV2 for hoodie on spark3.0x
Review Comment:
changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1198721220
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/adapter/Spark3_0Adapter.scala:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.adapter
+
+import org.apache.avro.Schema
+import org.apache.hudi.Spark30HoodieFileScanRDD
+import org.apache.spark.sql.avro.{HoodieAvroDeserializer, HoodieAvroSerializer, HoodieSpark3_0AvroDeserializer, HoodieSpark3_0AvroSerializer}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression}
+import org.apache.spark.sql.catalyst.parser.ParserInterface
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.execution.datasources.parquet.{ParquetFileFormat, Spark30HoodieParquetFileFormat}
+import org.apache.spark.sql.execution.datasources.{FilePartition, FileScanRDD, PartitionedFile}
+import org.apache.spark.sql.hudi.SparkAdapter
+import org.apache.spark.sql.parser.{HoodieExtendedParserInterface, HoodieSpark3_0ExtendedSqlParser}
+import org.apache.spark.sql.types.{DataType, Metadata, MetadataBuilder, StructType}
+import org.apache.spark.sql.vectorized.ColumnarUtils
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.storage.StorageLevel._
+
+/**
+ * Implementation of [[SparkAdapter]] for Spark 3.0.x
+ */
+class Spark3_0Adapter extends BaseSpark3Adapter {
+
+ override def resolveHoodieTable(plan: LogicalPlan): Option[CatalogTable] = {
+ super.resolveHoodieTable(plan).orElse {
+ EliminateSubqueryAliases(plan) match {
+ // First, we need to weed out unresolved plans
+ case plan if !plan.resolved => None
+ // NOTE: When resolving Hudi table we allow [[Filter]]s and [[Project]]s be applied
+ // on top of it
+ case PhysicalOperation(_, _, DataSourceV2Relation(table: CatalogTable, _, _, _, _)) if isHoodieTable(table) =>
+ Some(table)
+ case _ => None
Review Comment:
Sure, will have a deep look and test for this performance issue.
As step one maybe we can focus on functionality :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1554274848
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
}, {
"hash" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187",
"triggerID" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"triggerType" : "PUSH"
}, {
"hash" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17210",
"triggerID" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 722fae1d1a8717873ed89b87fb08b7c74fa3ccf5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187)
* d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17210)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212] Hudi Spark 3.0.x integration
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1555122644
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
}, {
"hash" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187",
"triggerID" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"triggerType" : "PUSH"
}, {
"hash" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17210",
"triggerID" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"triggerType" : "PUSH"
}, {
"hash" : "c4a5675de364a34774e3cecc0e625f92639382b4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17212",
"triggerID" : "c4a5675de364a34774e3cecc0e625f92639382b4",
"triggerType" : "PUSH"
}, {
"hash" : "f284403cfa1fdd79b77936d545b24da02b4e93e9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17223",
"triggerID" : "f284403cfa1fdd79b77936d545b24da02b4e93e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* c4a5675de364a34774e3cecc0e625f92639382b4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17212)
* f284403cfa1fdd79b77936d545b24da02b4e93e9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17223)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1550847289
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 0c8facb84a9a4362986eafa17854faad647f4fde Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110)
* a315e98dc2039f19275943f7024035816cbf4448 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1198712315
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/HoodieSpark30CatalogUtils.scala:
##########
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.spark.sql.connector.expressions.{BucketTransform, NamedReference, Transform}
+
+object HoodieSpark30CatalogUtils extends HoodieSpark3CatalogUtils {
+
+ override def unapplyBucketTransform(t: Transform): Option[(Int, Seq[NamedReference], Seq[NamedReference])] =
+ t match {
+ case BucketTransform(numBuckets, ref) => Some(numBuckets, Seq(ref), Seq.empty)
+ case _ => None
+ }
+
Review Comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1198715597
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/hudi/command/Spark30AlterTableCommand.scala:
##########
@@ -0,0 +1,325 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command
+
+import java.net.URI
+import java.nio.charset.StandardCharsets
+import java.util
+import java.util.concurrent.atomic.AtomicInteger
+import org.apache.hadoop.conf.Configuration
+import org.apache.hadoop.fs.Path
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.avro.AvroSchemaUtils.getAvroRecordQualifiedName
+import org.apache.hudi.client.utils.SparkInternalSchemaConverter
+import org.apache.hudi.common.model.{HoodieCommitMetadata, WriteOperationType}
+import org.apache.hudi.{AvroConversionUtils, DataSourceOptionsHelper, DataSourceUtils}
+import org.apache.hudi.common.table.timeline.{HoodieActiveTimeline, HoodieInstant}
+import org.apache.hudi.common.table.timeline.HoodieInstant.State
+import org.apache.hudi.common.table.{HoodieTableMetaClient, TableSchemaResolver}
+import org.apache.hudi.common.util.{CommitUtils, Option}
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.internal.schema.InternalSchema
+import org.apache.hudi.internal.schema.action.TableChange.ColumnChangeID
+import org.apache.hudi.internal.schema.action.TableChanges
+import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter
+import org.apache.hudi.internal.schema.utils.{SchemaChangeUtils, SerDeHelper}
+import org.apache.hudi.internal.schema.io.FileBasedInternalSchemaStorageManager
+import org.apache.hudi.table.HoodieSparkTable
+import org.apache.spark.api.java.JavaSparkContext
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.{Row, SparkSession}
+import org.apache.spark.sql.catalyst.catalog.{CatalogTable, CatalogTableType}
+import org.apache.spark.sql.connector.catalog.{TableCatalog, TableChange}
+import org.apache.spark.sql.connector.catalog.TableChange.{AddColumn, DeleteColumn, RemoveProperty, SetProperty}
+import org.apache.spark.sql.execution.command.RunnableCommand
+import org.apache.spark.sql.types.StructType
+
+import scala.collection.JavaConverters._
+import scala.util.control.NonFatal
+
+// TODO: we should remove this file when we support datasourceV2 for hoodie on spark3.0x
Review Comment:
changed.
##########
pom.xml:
##########
@@ -2238,6 +2239,44 @@
</activation>
</profile>
+ <profile>
+ <id>spark3.0</id>
+ <properties>
+ <spark3.version>${spark30.version}</spark3.version>
+ <spark.version>${spark3.version}</spark.version>
+ <sparkbundle.version>3.0</sparkbundle.version>
+ <scala.version>${scala12.version}</scala.version>
+ <scala.binary.version>2.12</scala.binary.version>
+ <hudi.spark.module>hudi-spark3.0.x</hudi.spark.module>
+ <hudi.spark.common.modules.1>hudi-spark3-common</hudi.spark.common.modules.1>
+ <kafka.version>${kafka.spark3.version}</kafka.version>
Review Comment:
changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1551819625
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 172e9f2ab16b6e7358d4970e8b7da4cf22cc0396 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1552836021
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
}, {
"hash" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187",
"triggerID" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* cd96d898ad095e994bf3d548914e4febb13a1e8c Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177)
* 722fae1d1a8717873ed89b87fb08b7c74fa3ccf5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1552468763
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 172e9f2ab16b6e7358d4970e8b7da4cf22cc0396 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151)
* cd96d898ad095e994bf3d548914e4febb13a1e8c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1552825503
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
}, {
"hash" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 172e9f2ab16b6e7358d4970e8b7da4cf22cc0396 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151)
* cd96d898ad095e994bf3d548914e4febb13a1e8c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177)
* 722fae1d1a8717873ed89b87fb08b7c74fa3ccf5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1548000449
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* d3f558c7c1ff3ab0169ff27d11955df48c3b3459 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1198712050
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystExpressionUtils.scala:
##########
@@ -17,16 +17,9 @@
package org.apache.spark.sql
-import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression, Predicate, PredicateHelper}
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-
-trait HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils
- with PredicateHelper {
-
- override def normalizeExprs(exprs: Seq[Expression], attributes: Seq[Attribute]): Seq[Expression] =
- DataSourceStrategy.normalizeExprs(exprs, attributes)
-
- override def extractPredicatesWithinOutputSet(condition: Expression,
- outputSet: AttributeSet): Option[Expression] =
- super[PredicateHelper].extractPredicatesWithinOutputSet(condition, outputSet)
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression}
+abstract class HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils {
+ def normalizeExprs(exprs: Seq[Expression], attributes: Seq[Attribute]): Seq[Expression]
+ def extractPredicatesWithinOutputSet(condition: Expression,
+ outputSet: AttributeSet): Option[Expression]
Review Comment:
Added.
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/java/org/apache/spark/sql/execution/datasources/parquet/Spark30HoodieVectorizedParquetRecordReader.java:
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet;
+
+import org.apache.hadoop.mapreduce.InputSplit;
+import org.apache.hadoop.mapreduce.TaskAttemptContext;
+import org.apache.hudi.client.utils.SparkInternalSchemaConverter;
+import org.apache.hudi.common.util.collection.Pair;
+import org.apache.spark.memory.MemoryMode;
+import org.apache.spark.sql.catalyst.InternalRow;
+import org.apache.spark.sql.execution.vectorized.OffHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.OnHeapColumnVector;
+import org.apache.spark.sql.execution.vectorized.WritableColumnVector;
+import org.apache.spark.sql.types.DataType;
+import org.apache.spark.sql.types.StructType;
+import org.apache.spark.sql.vectorized.ColumnarBatch;
+
+import java.io.IOException;
+import java.time.ZoneId;
+import java.util.HashMap;
+import java.util.Map;
+
+public class Spark30HoodieVectorizedParquetRecordReader extends VectorizedParquetRecordReader {
Review Comment:
Exactly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1549033541
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* d3f558c7c1ff3ab0169ff27d11955df48c3b3459 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085)
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* b43956687e916bb0d0259f82c7846af74c018d1e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105)
* 0c8facb84a9a4362986eafa17854faad647f4fde UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1554263623
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
}, {
"hash" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187",
"triggerID" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"triggerType" : "PUSH"
}, {
"hash" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 722fae1d1a8717873ed89b87fb08b7c74fa3ccf5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187)
* d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1198728500
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/adapter/Spark3_0Adapter.scala:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.adapter
+
+import org.apache.avro.Schema
+import org.apache.hudi.Spark30HoodieFileScanRDD
+import org.apache.spark.sql.avro.{HoodieAvroDeserializer, HoodieAvroSerializer, HoodieSpark3_0AvroDeserializer, HoodieSpark3_0AvroSerializer}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression}
+import org.apache.spark.sql.catalyst.parser.ParserInterface
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.execution.datasources.parquet.{ParquetFileFormat, Spark30HoodieParquetFileFormat}
+import org.apache.spark.sql.execution.datasources.{FilePartition, FileScanRDD, PartitionedFile}
+import org.apache.spark.sql.hudi.SparkAdapter
+import org.apache.spark.sql.parser.{HoodieExtendedParserInterface, HoodieSpark3_0ExtendedSqlParser}
+import org.apache.spark.sql.types.{DataType, Metadata, MetadataBuilder, StructType}
+import org.apache.spark.sql.vectorized.ColumnarUtils
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.storage.StorageLevel._
+
+/**
+ * Implementation of [[SparkAdapter]] for Spark 3.0.x
+ */
+class Spark3_0Adapter extends BaseSpark3Adapter {
+
+ override def resolveHoodieTable(plan: LogicalPlan): Option[CatalogTable] = {
+ super.resolveHoodieTable(plan).orElse {
+ EliminateSubqueryAliases(plan) match {
+ // First, we need to weed out unresolved plans
+ case plan if !plan.resolved => None
+ // NOTE: When resolving Hudi table we allow [[Filter]]s and [[Project]]s be applied
+ // on top of it
+ case PhysicalOperation(_, _, DataSourceV2Relation(table: CatalogTable, _, _, _, _)) if isHoodieTable(table) =>
+ Some(table)
+ case _ => None
Review Comment:
But as far as I know, both hudi write and read in spark3.0 not use V2, so it doesn't matter to do fallback here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1554344740
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
}, {
"hash" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187",
"triggerID" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"triggerType" : "PUSH"
}, {
"hash" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17210",
"triggerID" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"triggerType" : "PUSH"
}, {
"hash" : "c4a5675de364a34774e3cecc0e625f92639382b4",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17212",
"triggerID" : "c4a5675de364a34774e3cecc0e625f92639382b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17210)
* c4a5675de364a34774e3cecc0e625f92639382b4 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8714: [HUDI-6212] Hudi Spark 3.0.x integration
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1199294944
##########
.github/workflows/bot.yml:
##########
@@ -63,6 +63,10 @@ jobs:
sparkProfile: "spark3.1"
sparkModules: "hudi-spark-datasource/hudi-spark3.1.x"
+ - scalaProfile: "scala-2.12"
Review Comment:
Makes sense. I've uploaded the bundle validation image for Spark 3.0: [`apachehudi/hudi-ci-bundle-validation-base:flink1146hive313spark302`](https://hub.docker.com/layers/apachehudi/hudi-ci-bundle-validation-base/flink1146hive313spark302/images/sha256-647120133e1a4aa4a48b2050c8b7a7fe678049c5368a479f1071e432e7cf84ba?context=explore). Added the config of bundle validation on Spark 3.0.
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystExpressionUtils.scala:
##########
@@ -17,16 +17,9 @@
package org.apache.spark.sql
-import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression, Predicate, PredicateHelper}
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-
-trait HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils
- with PredicateHelper {
-
- override def normalizeExprs(exprs: Seq[Expression], attributes: Seq[Attribute]): Seq[Expression] =
- DataSourceStrategy.normalizeExprs(exprs, attributes)
-
- override def extractPredicatesWithinOutputSet(condition: Expression,
- outputSet: AttributeSet): Option[Expression] =
- super[PredicateHelper].extractPredicatesWithinOutputSet(condition, outputSet)
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression}
+abstract class HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils {
Review Comment:
Makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #8714: [HUDI-6212] Hudi Spark 3.0.x integration
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #8714:
URL: https://github.com/apache/hudi/pull/8714
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1548939394
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* d3f558c7c1ff3ab0169ff27d11955df48c3b3459 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085)
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* b43956687e916bb0d0259f82c7846af74c018d1e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1548934249
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* d3f558c7c1ff3ab0169ff27d11955df48c3b3459 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085)
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* b43956687e916bb0d0259f82c7846af74c018d1e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1194548389
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystExpressionUtils.scala:
##########
@@ -17,16 +17,9 @@
package org.apache.spark.sql
-import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression, Predicate, PredicateHelper}
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-
-trait HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils
- with PredicateHelper {
-
- override def normalizeExprs(exprs: Seq[Expression], attributes: Seq[Attribute]): Seq[Expression] =
Review Comment:
`DataSourceStrategy.normalizeExprs(exprs, attributes)` are not included in spark3.0. So that we need to remove this specific implement in spark3-common space.
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystExpressionUtils.scala:
##########
@@ -17,16 +17,9 @@
package org.apache.spark.sql
-import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression, Predicate, PredicateHelper}
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-
-trait HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils
- with PredicateHelper {
-
- override def normalizeExprs(exprs: Seq[Expression], attributes: Seq[Attribute]): Seq[Expression] =
- DataSourceStrategy.normalizeExprs(exprs, attributes)
-
- override def extractPredicatesWithinOutputSet(condition: Expression,
Review Comment:
`extractPredicatesWithinOutputSet` are not included in spark3.0. So that we need to remove this specific implement in spark3-common space.
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark3Adapter.scala:
##########
@@ -108,23 +87,5 @@ abstract class BaseSpark3Adapter extends SparkAdapter with Logging {
DefaultSource.createRelation(sqlContext, metaClient, dataSchema, globPaths, parameters.asScala.toMap)
}
- /**
- * Converts instance of [[StorageLevel]] to a corresponding string
- */
- override def convertStorageLevelToString(level: StorageLevel): String = level match {
- case NONE => "NONE"
- case DISK_ONLY => "DISK_ONLY"
- case DISK_ONLY_2 => "DISK_ONLY_2"
- case DISK_ONLY_3 => "DISK_ONLY_3"
Review Comment:
`DISK_ONLY_3` are not included in spark3.0. So that we need to remove this specific implement in spark3-common space.
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/adapter/BaseSpark3Adapter.scala:
##########
@@ -63,20 +56,6 @@ abstract class BaseSpark3Adapter extends SparkAdapter with Logging {
new Spark3RowSerDe(encoder)
}
- override def resolveHoodieTable(plan: LogicalPlan): Option[CatalogTable] = {
- super.resolveHoodieTable(plan).orElse {
- EliminateSubqueryAliases(plan) match {
- // First, we need to weed out unresolved plans
- case plan if !plan.resolved => None
- // NOTE: When resolving Hudi table we allow [[Filter]]s and [[Project]]s be applied
- // on top of it
- case PhysicalOperation(_, _, DataSourceV2Relation(v2: V2TableWithV1Fallback, _, _, _, _)) if isHoodieTable(v2.v1Table) =>
Review Comment:
`V2TableWithV1Fallback` are not included in spark3.0. So that we need to remove this specific implement in spark3-common space.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1548013414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* d3f558c7c1ff3ab0169ff27d11955df48c3b3459 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1551082869
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* a315e98dc2039f19275943f7024035816cbf4448 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #8714: [HUDI-6212] Hudi Spark 3.0.x integration
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1555314918
Azure CI is green.
<img width="1618" alt="Screenshot 2023-05-19 at 15 08 27" src="https://github.com/apache/hudi/assets/2497195/c501a046-e0f3-441d-a521-dc8ea5745032">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1554333743
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
}, {
"hash" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187",
"triggerID" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"triggerType" : "PUSH"
}, {
"hash" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17210",
"triggerID" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"triggerType" : "PUSH"
}, {
"hash" : "c4a5675de364a34774e3cecc0e625f92639382b4",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c4a5675de364a34774e3cecc0e625f92639382b4",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 722fae1d1a8717873ed89b87fb08b7c74fa3ccf5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187)
* d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17210)
* c4a5675de364a34774e3cecc0e625f92639382b4 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8714: [HUDI-6212] Hudi Spark 3.0.x integration
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1199893197
##########
packaging/bundle-validation/ci_run.sh:
##########
@@ -41,6 +41,16 @@ if [[ ${SPARK_RUNTIME} == 'spark2.4' ]]; then
CONFLUENT_VERSION=5.5.12
KAFKA_CONNECT_HDFS_VERSION=10.1.13
IMAGE_TAG=flink1136hive239spark248
+elif [[ ${SPARK_RUNTIME} == 'spark3.0.2' ]]; then
+ HADOOP_VERSION=2.7.7
+ HIVE_VERSION=3.1.3
+ DERBY_VERSION=10.14.1.0
+ FLINK_VERSION=1.14.6
+ SPARK_VERSION=3.0.2
+ SPARK_HADOOP_VERSION=2.7
+ CONFLUENT_VERSION=5.5.12
+ KAFKA_CONNECT_HDFS_VERSION=10.1.13
+ IMAGE_TAG=flink1146hive313spark302
elif [[ ${SPARK_RUNTIME} == 'spark3.1.3' ]]; then
Review Comment:
Does the tag `flink1146hive313spark302` needs to be prefabricated, seems no script really build this image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1548894570
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* d3f558c7c1ff3ab0169ff27d11955df48c3b3459 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085)
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1549419989
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 0c8facb84a9a4362986eafa17854faad647f4fde Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212][Do Not Merge][Test] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1549073761
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* b43956687e916bb0d0259f82c7846af74c018d1e Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105)
* 0c8facb84a9a4362986eafa17854faad647f4fde UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1553710804
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
}, {
"hash" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187",
"triggerID" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* 722fae1d1a8717873ed89b87fb08b7c74fa3ccf5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] zhangyue19921010 commented on a diff in pull request #8714: [HUDI-6212] Hudi spark 3.0.x adoption
Posted by "zhangyue19921010 (via GitHub)" <gi...@apache.org>.
zhangyue19921010 commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1198710376
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystExpressionUtils.scala:
##########
@@ -17,16 +17,9 @@
package org.apache.spark.sql
-import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression, Predicate, PredicateHelper}
-import org.apache.spark.sql.execution.datasources.DataSourceStrategy
-
-trait HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils
- with PredicateHelper {
-
- override def normalizeExprs(exprs: Seq[Expression], attributes: Seq[Attribute]): Seq[Expression] =
- DataSourceStrategy.normalizeExprs(exprs, attributes)
-
- override def extractPredicatesWithinOutputSet(condition: Expression,
- outputSet: AttributeSet): Option[Expression] =
- super[PredicateHelper].extractPredicatesWithinOutputSet(condition, outputSet)
+import org.apache.spark.sql.catalyst.expressions.{Attribute, AttributeSet, Expression}
+abstract class HoodieSpark3CatalystExpressionUtils extends HoodieCatalystExpressionUtils {
Review Comment:
Sure, this PredicateHelper is moved to specific implement as there is some compatibility issue if keep it in spark3-common module.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8714: [HUDI-6212] Hudi Spark 3.0.x integration
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8714:
URL: https://github.com/apache/hudi/pull/8714#discussion_r1199299441
##########
pom.xml:
##########
@@ -2238,6 +2239,44 @@
</activation>
</profile>
+ <profile>
+ <id>spark3.0</id>
+ <properties>
+ <spark3.version>${spark30.version}</spark3.version>
+ <spark.version>${spark3.version}</spark.version>
+ <sparkbundle.version>3.0</sparkbundle.version>
+ <scala.version>${scala12.version}</scala.version>
+ <scala.binary.version>2.12</scala.binary.version>
+ <hudi.spark.module>hudi-spark3.0.x</hudi.spark.module>
+ <hudi.spark.common.modules.1>hudi-spark3-common</hudi.spark.common.modules.1>
+ <kafka.version>${kafka.spark3.version}</kafka.version>
Review Comment:
Seems it's not added. But I assume it's not really necessary here.
##########
hudi-spark-datasource/hudi-spark3.0.x/src/main/scala/org/apache/spark/sql/adapter/Spark3_0Adapter.scala:
##########
@@ -0,0 +1,117 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.adapter
+
+import org.apache.avro.Schema
+import org.apache.hudi.Spark30HoodieFileScanRDD
+import org.apache.spark.sql.avro.{HoodieAvroDeserializer, HoodieAvroSerializer, HoodieSpark3_0AvroDeserializer, HoodieSpark3_0AvroSerializer}
+import org.apache.spark.sql.catalyst.InternalRow
+import org.apache.spark.sql.catalyst.expressions.{AttributeReference, Expression}
+import org.apache.spark.sql.catalyst.parser.ParserInterface
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.execution.datasources.parquet.{ParquetFileFormat, Spark30HoodieParquetFileFormat}
+import org.apache.spark.sql.execution.datasources.{FilePartition, FileScanRDD, PartitionedFile}
+import org.apache.spark.sql.hudi.SparkAdapter
+import org.apache.spark.sql.parser.{HoodieExtendedParserInterface, HoodieSpark3_0ExtendedSqlParser}
+import org.apache.spark.sql.types.{DataType, Metadata, MetadataBuilder, StructType}
+import org.apache.spark.sql.vectorized.ColumnarUtils
+import org.apache.spark.sql._
+import org.apache.spark.sql.catalyst.analysis.EliminateSubqueryAliases
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
+import org.apache.spark.sql.catalyst.planning.PhysicalOperation
+import org.apache.spark.sql.execution.datasources.v2.DataSourceV2Relation
+import org.apache.spark.storage.StorageLevel
+import org.apache.spark.storage.StorageLevel._
+
+/**
+ * Implementation of [[SparkAdapter]] for Spark 3.0.x
+ */
+class Spark3_0Adapter extends BaseSpark3Adapter {
+
+ override def resolveHoodieTable(plan: LogicalPlan): Option[CatalogTable] = {
+ super.resolveHoodieTable(plan).orElse {
+ EliminateSubqueryAliases(plan) match {
+ // First, we need to weed out unresolved plans
+ case plan if !plan.resolved => None
+ // NOTE: When resolving Hudi table we allow [[Filter]]s and [[Project]]s be applied
+ // on top of it
+ case PhysicalOperation(_, _, DataSourceV2Relation(table: CatalogTable, _, _, _, _)) if isHoodieTable(table) =>
+ Some(table)
+ case _ => None
Review Comment:
Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8714: [HUDI-6212] Hudi Spark 3.0.x integration
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8714:
URL: https://github.com/apache/hudi/pull/8714#issuecomment-1555114021
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b3da8cccadddc1cc95c08ad0643a763726a9a010",
"triggerType" : "PUSH"
}, {
"hash" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17085",
"triggerID" : "d3f558c7c1ff3ab0169ff27d11955df48c3b3459",
"triggerType" : "PUSH"
}, {
"hash" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "8dbee823426fe3a74d68084e1c47aedc90939a7a",
"triggerType" : "PUSH"
}, {
"hash" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17105",
"triggerID" : "b43956687e916bb0d0259f82c7846af74c018d1e",
"triggerType" : "PUSH"
}, {
"hash" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17110",
"triggerID" : "0c8facb84a9a4362986eafa17854faad647f4fde",
"triggerType" : "PUSH"
}, {
"hash" : "a315e98dc2039f19275943f7024035816cbf4448",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17141",
"triggerID" : "a315e98dc2039f19275943f7024035816cbf4448",
"triggerType" : "PUSH"
}, {
"hash" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17151",
"triggerID" : "172e9f2ab16b6e7358d4970e8b7da4cf22cc0396",
"triggerType" : "PUSH"
}, {
"hash" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17177",
"triggerID" : "cd96d898ad095e994bf3d548914e4febb13a1e8c",
"triggerType" : "PUSH"
}, {
"hash" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17187",
"triggerID" : "722fae1d1a8717873ed89b87fb08b7c74fa3ccf5",
"triggerType" : "PUSH"
}, {
"hash" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17210",
"triggerID" : "d6b11cbb3722ce087ce1e77bf63cfdfbab1c1393",
"triggerType" : "PUSH"
}, {
"hash" : "c4a5675de364a34774e3cecc0e625f92639382b4",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17212",
"triggerID" : "c4a5675de364a34774e3cecc0e625f92639382b4",
"triggerType" : "PUSH"
}, {
"hash" : "f284403cfa1fdd79b77936d545b24da02b4e93e9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f284403cfa1fdd79b77936d545b24da02b4e93e9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* b3da8cccadddc1cc95c08ad0643a763726a9a010 UNKNOWN
* 8dbee823426fe3a74d68084e1c47aedc90939a7a UNKNOWN
* c4a5675de364a34774e3cecc0e625f92639382b4 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17212)
* f284403cfa1fdd79b77936d545b24da02b4e93e9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org