You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/01/16 09:10:58 UTC
[GitHub] [hudi] leesf opened a new pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
leesf opened a new pull request #4611:
URL: https://github.com/apache/hudi/pull/4611
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
Introduce HoodieCatalog to manage Hudi tables for Spark Datasource V2.
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017400948
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* b6b74ca2553aab849511a37f0efd411fef068285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370)
* addb9f9925913e522852b2efcb0a80fccc4153f2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013868788
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
* a5fecdc15707afd659887b025d1e1094543468e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282)
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013877803
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a5fecdc15707afd659887b025d1e1094543468e7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282)
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r790364397
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
Review comment:
try to keep name aligned for clarity; applies to the rest of the file too
```suggestion
val preCombineField = Option(tableConfig.getPreCombineField).getOrElse("")
```
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
+ assert(hoodieCatalogTable.primaryKeys.nonEmpty,
+ s"There are no primary key in table ${hoodieCatalogTable.table.identifier}, cannot execute update operator")
+ val enableHive = isEnableHive(sparkSession)
+
+ withSparkConf(sparkSession, catalogProperties) {
+ Map(
+ "path" -> hoodieCatalogTable.tableLocation,
+ RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
+ PRECOMBINE_FIELD.key -> preCombineColumn,
+ TBL_NAME.key -> hoodieCatalogTable.tableName,
+ HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
+ URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
+ KEYGENERATOR_CLASS_NAME.key -> classOf[SqlKeyGenerator].getCanonicalName,
+ SqlKeyGenerator.ORIGIN_KEYGEN_CLASS_NAME -> tableConfig.getKeyGeneratorClassName,
+ OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+ PARTITIONPATH_FIELD.key -> tableConfig.getPartitionFieldProp,
+ META_SYNC_ENABLED.key -> enableHive.toString,
+ HIVE_SYNC_MODE.key -> HiveSyncMode.HMS.name(),
+ HIVE_USE_JDBC.key -> "false",
+ HIVE_DATABASE.key -> tableId.database.getOrElse("default"),
+ HIVE_TABLE.key -> tableId.table,
+ HIVE_PARTITION_FIELDS.key -> tableConfig.getPartitionFieldProp,
+ HIVE_PARTITION_EXTRACTOR_CLASS.key -> classOf[MultiPartKeysValueExtractor].getCanonicalName,
+ HIVE_SUPPORT_TIMESTAMP_TYPE.key -> "true",
+ HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key -> "200",
+ SqlKeyGenerator.PARTITION_SCHEMA -> hoodieCatalogTable.partitionSchema.toDDL
+ )
+ }
+ }
+
+ def cast(exp: Expression, field: StructField, sqlConf: SQLConf): Expression = {
+ castIfNeeded(exp, field.dataType, sqlConf)
+ }
+
+ /**
+ * Build the default config for insert.
+ * @return
+ */
+ def buildHoodieInsertConfig(hoodieCatalogTable: HoodieCatalogTable,
+ sparkSession: SparkSession,
+ isOverwrite: Boolean,
+ insertPartitions: Map[String, Option[String]] = Map.empty,
+ extraOptions: Map[String, String]): Map[String, String] = {
+
+ if (insertPartitions.nonEmpty &&
+ (insertPartitions.keys.toSet != hoodieCatalogTable.partitionFields.toSet)) {
+ throw new IllegalArgumentException(s"Insert partition fields" +
+ s"[${insertPartitions.keys.mkString(" " )}]" +
+ s" not equal to the defined partition in table[${hoodieCatalogTable.partitionFields.mkString(",")}]")
+ }
Review comment:
can be replaced with `require`
##########
File path: hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HudiSpark3SqlUtils.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
+import org.apache.spark.sql.connector.expressions.{BucketTransform, FieldReference, IdentityTransform, Transform}
+
+import scala.collection.mutable
+
+object HudiSpark3SqlUtils {
Review comment:
```suggestion
object HoodieSpark3SqlUtils {
```
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlCommonUtils.scala
##########
@@ -54,6 +54,10 @@ object HoodieSqlCommonUtils extends SparkAdapterSupport {
override def get() = new SimpleDateFormat("yyyy-MM-dd")
})
+ def isHoodieTable(properties: Map[String, String]): Boolean = {
+ properties.getOrElse("provider", "").toLowerCase(Locale.ROOT) == "hudi"
+ }
+
Review comment:
this and other overloaded `isHoodieTable()` duplicate the ones in spark adapter?
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
+ assert(hoodieCatalogTable.primaryKeys.nonEmpty,
+ s"There are no primary key in table ${hoodieCatalogTable.table.identifier}, cannot execute update operator")
+ val enableHive = isEnableHive(sparkSession)
+
+ withSparkConf(sparkSession, catalogProperties) {
+ Map(
+ "path" -> hoodieCatalogTable.tableLocation,
+ RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
+ PRECOMBINE_FIELD.key -> preCombineColumn,
+ TBL_NAME.key -> hoodieCatalogTable.tableName,
+ HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
+ URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
+ KEYGENERATOR_CLASS_NAME.key -> classOf[SqlKeyGenerator].getCanonicalName,
+ SqlKeyGenerator.ORIGIN_KEYGEN_CLASS_NAME -> tableConfig.getKeyGeneratorClassName,
+ OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+ PARTITIONPATH_FIELD.key -> tableConfig.getPartitionFieldProp,
+ META_SYNC_ENABLED.key -> enableHive.toString,
+ HIVE_SYNC_MODE.key -> HiveSyncMode.HMS.name(),
+ HIVE_USE_JDBC.key -> "false",
+ HIVE_DATABASE.key -> tableId.database.getOrElse("default"),
+ HIVE_TABLE.key -> tableId.table,
+ HIVE_PARTITION_FIELDS.key -> tableConfig.getPartitionFieldProp,
+ HIVE_PARTITION_EXTRACTOR_CLASS.key -> classOf[MultiPartKeysValueExtractor].getCanonicalName,
+ HIVE_SUPPORT_TIMESTAMP_TYPE.key -> "true",
+ HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key -> "200",
+ SqlKeyGenerator.PARTITION_SCHEMA -> hoodieCatalogTable.partitionSchema.toDDL
+ )
+ }
+ }
+
+ def cast(exp: Expression, field: StructField, sqlConf: SQLConf): Expression = {
+ castIfNeeded(exp, field.dataType, sqlConf)
+ }
Review comment:
is this needed? just wrapping another method
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
Review comment:
~better use a singleton object instead of a trait ?~
Saw that HoodieCatalog implementing this, so it reads better with `ProvidesHoodieConfig` as a trait ?
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/AlterHoodieTableAddColumnsCommand.scala
##########
@@ -57,7 +57,8 @@ case class AlterHoodieTableAddColumnsCommand(
s" table columns is: [${hoodieCatalogTable.tableSchemaWithoutMetaFields.fieldNames.mkString(",")}]")
}
// Get the new schema
- val newSqlSchema = StructType(tableSchema.fields ++ colsToAdd)
+ val rearrangeSchema = hoodieCatalogTable.dataSchema ++ colsToAdd ++ hoodieCatalogTable.partitionSchema
Review comment:
```suggestion
val rearrangedSchema = hoodieCatalogTable.dataSchema ++ colsToAdd ++ hoodieCatalogTable.partitionSchema
```
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/BaseStagedTable.scala
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.connector.catalog._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder}
+import org.apache.spark.sql.types.StructType
+
+import java.util
+
+case class BaseStagedTable(ident: Identifier,
Review comment:
some docs for this class ?
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieInternalV2Table.scala
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.common.table.{HoodieTableConfig, HoodieTableMetaClient}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogTable, HoodieCatalogTable}
+import org.apache.spark.sql.connector.catalog.TableCapability._
+import org.apache.spark.sql.connector.catalog.{SupportsWrite, Table, TableCapability, V2TableWithV1Fallback}
+import org.apache.spark.sql.connector.expressions.{FieldReference, IdentityTransform, Transform}
+import org.apache.spark.sql.connector.write._
+import org.apache.spark.sql.sources.{Filter, InsertableRelation}
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
+
+import java.util
+import scala.collection.JavaConverters.{mapAsJavaMapConverter, setAsJavaSetConverter}
+
+case class HoodieInternalV2Table(spark: SparkSession,
+ path: String,
+ catalogTable: Option[CatalogTable] = None,
+ tableIdentifier: Option[String] = None,
+ options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty())
+ extends Table with SupportsWrite with V2TableWithV1Fallback {
+
+ lazy val hoodieCatalogTable: HoodieCatalogTable = if (catalogTable.isDefined) {
+ HoodieCatalogTable(spark, catalogTable.get)
+ } else {
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient.builder()
+ .setBasePath(path)
+ .setConf(SparkSession.active.sessionState.newHadoopConf)
+ .build()
+
+ val tableConfig: HoodieTableConfig = metaClient.getTableConfig
+ val tableName: String = tableConfig.getTableName
+
+ HoodieCatalogTable(spark, TableIdentifier(tableName))
+ }
+
+ private lazy val tableSchema: StructType = hoodieCatalogTable.tableSchema
+
+ override def name(): String = hoodieCatalogTable.table.identifier.unquotedString
+
+ override def schema(): StructType = tableSchema
+
+ override def capabilities(): util.Set[TableCapability] = Set(
+ BATCH_READ, V1_BATCH_WRITE, OVERWRITE_BY_FILTER, TRUNCATE, ACCEPT_ANY_SCHEMA
+ ).asJava
+
+ override def properties(): util.Map[String, String] = {
+ val map = new util.HashMap[String, String]()
+ map.put("provider", "hudi")
Review comment:
why not make this part of catalog properties ?
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieInternalV2Table.scala
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.common.table.{HoodieTableConfig, HoodieTableMetaClient}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogTable, HoodieCatalogTable}
+import org.apache.spark.sql.connector.catalog.TableCapability._
+import org.apache.spark.sql.connector.catalog.{SupportsWrite, Table, TableCapability, V2TableWithV1Fallback}
+import org.apache.spark.sql.connector.expressions.{FieldReference, IdentityTransform, Transform}
+import org.apache.spark.sql.connector.write._
+import org.apache.spark.sql.sources.{Filter, InsertableRelation}
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
+
+import java.util
+import scala.collection.JavaConverters.{mapAsJavaMapConverter, setAsJavaSetConverter}
+
+case class HoodieInternalV2Table(spark: SparkSession,
+ path: String,
+ catalogTable: Option[CatalogTable] = None,
+ tableIdentifier: Option[String] = None,
+ options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty())
+ extends Table with SupportsWrite with V2TableWithV1Fallback {
+
+ lazy val hoodieCatalogTable: HoodieCatalogTable = if (catalogTable.isDefined) {
+ HoodieCatalogTable(spark, catalogTable.get)
+ } else {
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient.builder()
+ .setBasePath(path)
+ .setConf(SparkSession.active.sessionState.newHadoopConf)
+ .build()
+
+ val tableConfig: HoodieTableConfig = metaClient.getTableConfig
+ val tableName: String = tableConfig.getTableName
+
+ HoodieCatalogTable(spark, TableIdentifier(tableName))
+ }
+
+ private lazy val tableSchema: StructType = hoodieCatalogTable.tableSchema
+
+ override def name(): String = hoodieCatalogTable.table.identifier.unquotedString
+
+ override def schema(): StructType = tableSchema
+
+ override def capabilities(): util.Set[TableCapability] = Set(
+ BATCH_READ, V1_BATCH_WRITE, OVERWRITE_BY_FILTER, TRUNCATE, ACCEPT_ANY_SCHEMA
+ ).asJava
+
+ override def properties(): util.Map[String, String] = {
+ val map = new util.HashMap[String, String]()
+ map.put("provider", "hudi")
+ map.putAll(hoodieCatalogTable.catalogProperties.asJava)
+ map
+ }
+
+ override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
+ new WriteIntoHoodieBuilder(info.options, hoodieCatalogTable, spark)
+ }
+
+ override def v1Table: CatalogTable = hoodieCatalogTable.table
+
+ override def partitioning(): Array[Transform] = {
+ hoodieCatalogTable.partitionFields.map { col =>
+ new IdentityTransform(new FieldReference(Seq(col)))
+ }.toArray
+ }
+
+}
+
+private class WriteIntoHoodieBuilder(writeOptions: CaseInsensitiveStringMap,
Review comment:
to be explicit, better call `HoodieV1WriteBuilder` ?
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
+ assert(hoodieCatalogTable.primaryKeys.nonEmpty,
Review comment:
better use `require` instead of `assert`
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala
##########
@@ -42,12 +43,39 @@ object HoodieAnalysis {
Seq(
session => HoodieResolveReferences(session),
session => HoodieAnalysis(session)
- )
+ ) ++ extraResolutionRules()
def customPostHocResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] =
Seq(
session => HoodiePostAnalysisRule(session)
- )
+ ) ++ extraPostHocResolutionRules()
+
+ def extraResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] = {
+ if (HoodieSparkUtils.isSpark3_2) {
+ val spark3AnalysisClass = "org.apache.spark.sql.hudi.analysis.HoodieSpark3Analysis"
+ val spark3Analysis: SparkSession => Rule[LogicalPlan] =
+ session => ReflectionUtils.loadClass(spark3AnalysisClass, session).asInstanceOf[Rule[LogicalPlan]]
+
+ val spark3ResolveReferences = "org.apache.spark.sql.hudi.analysis.HoodieSpark3ResolveReferences"
+ val spark3References: SparkSession => Rule[LogicalPlan] =
+ session => ReflectionUtils.loadClass(spark3ResolveReferences, session).asInstanceOf[Rule[LogicalPlan]]
+
+ Seq(spark3Analysis, spark3References)
+ } else {
+ Seq.empty
+ }
+ }
+
+ def extraPostHocResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] =
+ if (HoodieSparkUtils.isSpark3_2) {
Review comment:
ditto
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala
##########
@@ -42,12 +43,39 @@ object HoodieAnalysis {
Seq(
session => HoodieResolveReferences(session),
session => HoodieAnalysis(session)
- )
+ ) ++ extraResolutionRules()
def customPostHocResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] =
Seq(
session => HoodiePostAnalysisRule(session)
- )
+ ) ++ extraPostHocResolutionRules()
+
+ def extraResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] = {
+ if (HoodieSparkUtils.isSpark3_2) {
Review comment:
assuming in future spark 3 versions like 3.3 this case also applies, then we should have something like `HoodieSparkutils.beforeSpark3_2` and negate the condition
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala
##########
@@ -94,11 +94,17 @@ class TestHoodieSparkSqlWriter {
* Utility method for initializing the spark context.
*/
def initSparkContext(): Unit = {
+ val sparkConf = new SparkConf()
+ if (HoodieSparkUtils.isSpark3_2) {
+ sparkConf.set("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
Review comment:
can you help me understand why only set this for spark 3.2 pls?
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala
##########
@@ -92,4 +95,31 @@ trait SparkAdapter extends Serializable {
* ParserInterface#parseMultipartIdentifier is supported since spark3, for spark2 this should not be called.
*/
def parseMultipartIdentifier(parser: ParserInterface, sqlText: String): Seq[String]
+
+ def isHoodieTable(table: LogicalPlan, spark: SparkSession): Boolean = {
+ tripAlias(table) match {
+ case LogicalRelation(_, _, Some(tbl), _) => isHoodieTable(tbl)
+ case relation: UnresolvedRelation =>
+ isHoodieTable(toTableIdentifier(relation), spark)
+ case _=> false
+ }
+ }
+
+ def tripAlias(plan: LogicalPlan): LogicalPlan = {
+ plan match {
+ case SubqueryAlias(_, relation: LogicalPlan) =>
+ tripAlias(relation)
+ case other =>
+ other
+ }
+ }
+
+ def isHoodieTable(table: CatalogTable): Boolean = {
+ table.provider.map(_.toLowerCase(Locale.ROOT)).orNull == "hudi"
+ }
+
+ def isHoodieTable(tableId: TableIdentifier, spark: SparkSession): Boolean = {
+ val table = spark.sessionState.catalog.getTableMetadata(tableId)
+ isHoodieTable(table)
+ }
Review comment:
it'll be nice to move these 2 up closer to their overloading method above `tripAlias()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038542507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960)
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1034547393
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 0abd457fb40ee1af67da5d283cd6c09e6e07dac9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490)
* e93af4603cc620726fc34a7d4aa482ee0acb7c41 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r805451853
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/hudi/Spark3DefaultSource.scala
##########
@@ -17,8 +17,30 @@
package org.apache.hudi
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.connector.catalog.{Table, TableProvider}
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.hudi.catalog.HoodieInternalV2Table
import org.apache.spark.sql.sources.DataSourceRegister
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+
+class Spark3DefaultSource extends DefaultSource with DataSourceRegister with TableProvider {
-class Spark3DefaultSource extends DefaultSource with DataSourceRegister {
override def shortName(): String = "hudi"
+
+ def inferSchema: StructType = new StructType()
+
+ override def inferSchema(options: CaseInsensitiveStringMap): StructType = inferSchema
+
+ override def getTable(schema: StructType,
+ partitioning: Array[Transform],
+ properties: java.util.Map[String, String]): Table = {
+ val options = new CaseInsensitiveStringMap(properties)
+ val path = options.get("path")
+ if (path == null) throw new HoodieException("path is null")
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038743897
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
}, {
"hash" : "22f6e2f42db71974b19aa30809420ed7afa50cc5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22f6e2f42db71974b19aa30809420ed7afa50cc5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965)
* 22f6e2f42db71974b19aa30809420ed7afa50cc5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1039149130
verified via spark shell basic read write are working cc @leesf
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038584770
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1034558510
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 0abd457fb40ee1af67da5d283cd6c09e6e07dac9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490)
* e93af4603cc620726fc34a7d4aa482ee0acb7c41 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038164254
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* e93af4603cc620726fc34a7d4aa482ee0acb7c41 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865)
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r805452236
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/BaseStagedTable.scala
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.connector.catalog._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder}
+import org.apache.spark.sql.types.StructType
+
+import java.util
+
+/**
+ * Basic implementation that represents a table which is staged for being committed.
+ * @param ident table ident
+ * @param table table
+ * @param catalog table catalog
+ */
+case class BaseStagedTable(ident: Identifier,
+ table: Table,
+ catalog: TableCatalog) extends SupportsWrite with StagedTable {
+ override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
+ info match {
+ case supportsWrite: SupportsWrite => supportsWrite.newWriteBuilder(info)
+ case _ => throw new HoodieException(s"Table implementation does not support writes: $ident.name")
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013842745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5e6315181eb95abc55bc6b1ee4f3488a60d65d0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280)
* 27cba00d5daf148f99be78e72dade08229491d18 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013855071
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013867974
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
* a5fecdc15707afd659887b025d1e1094543468e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013867484
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
* a5fecdc15707afd659887b025d1e1094543468e7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017499750
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1034547393
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 0abd457fb40ee1af67da5d283cd6c09e6e07dac9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490)
* e93af4603cc620726fc34a7d4aa482ee0acb7c41 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038584770
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038763912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
}, {
"hash" : "22f6e2f42db71974b19aa30809420ed7afa50cc5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5975",
"triggerID" : "22f6e2f42db71974b19aa30809420ed7afa50cc5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965)
* 22f6e2f42db71974b19aa30809420ed7afa50cc5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5975)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1019440860
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017280671
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365)
* b6b74ca2553aab849511a37f0efd411fef068285 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1019440860
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
xushiyan commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r795138030
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/BaseStagedTable.scala
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.connector.catalog._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder}
+import org.apache.spark.sql.types.StructType
+
+import java.util
+
+/**
+ * Basic implementation that represents a table which is staged for being committed.
+ * @param ident table ident
+ * @param table table
+ * @param catalog table catalog
+ */
+case class BaseStagedTable(ident: Identifier,
+ table: Table,
+ catalog: TableCatalog) extends SupportsWrite with StagedTable {
+ override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
+ info match {
+ case supportsWrite: SupportsWrite => supportsWrite.newWriteBuilder(info)
+ case _ => throw new HoodieException(s"Table implementation does not support writes: $ident.name")
Review comment:
should it still be `HoodieException` given this is not HoodieStagedTable? also suggested a clearer message.
```suggestion
case _ => throw new HoodieException(s"Table `$ident.name` does not support writes.")
```
##########
File path: hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3SqlUtils.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
+import org.apache.spark.sql.connector.expressions.{BucketTransform, FieldReference, IdentityTransform, Transform}
+
+import scala.collection.mutable
+
+object HoodieSpark3SqlUtils {
+ def convertTransforms(partitions: Seq[Transform]): (Seq[String], Option[BucketSpec]) = {
+ val identityCols = new mutable.ArrayBuffer[String]
+ var bucketSpec = Option.empty[BucketSpec]
+
+ partitions.map {
+ case IdentityTransform(FieldReference(Seq(col))) =>
+ identityCols += col
+
+
+ case BucketTransform(numBuckets, FieldReference(Seq(col))) =>
+ bucketSpec = Some(BucketSpec(numBuckets, col :: Nil, Nil))
+
+ case _ =>
+ throw new HoodieException(s"Partitioning by expressions")
Review comment:
is this exception message user-facing ? then we need to clarify it further.
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala
##########
@@ -171,7 +171,7 @@ class IncrementalRelation(val sqlContext: SQLContext,
if (metaBootstrapFileIdToFullPath.nonEmpty) {
df = sqlContext.sparkSession.read
- .format("hudi")
+ .format("hudi_v1")
Review comment:
so incremental read still go through v1. is there any follow-up plan to use v2 for this as well ?
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/hudi/Spark3DefaultSource.scala
##########
@@ -17,8 +17,30 @@
package org.apache.hudi
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.connector.catalog.{Table, TableProvider}
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.hudi.catalog.HoodieInternalV2Table
import org.apache.spark.sql.sources.DataSourceRegister
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+
+class Spark3DefaultSource extends DefaultSource with DataSourceRegister with TableProvider {
-class Spark3DefaultSource extends DefaultSource with DataSourceRegister {
override def shortName(): String = "hudi"
+
+ def inferSchema: StructType = new StructType()
+
+ override def inferSchema(options: CaseInsensitiveStringMap): StructType = inferSchema
+
+ override def getTable(schema: StructType,
+ partitioning: Array[Transform],
+ properties: java.util.Map[String, String]): Table = {
+ val options = new CaseInsensitiveStringMap(properties)
+ val path = options.get("path")
+ if (path == null) throw new HoodieException("path is null")
Review comment:
is this also user-facing message? how about "'path' cannot be null; missing 'path' from table properties?" just an example.
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/BaseStagedTable.scala
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.connector.catalog._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder}
+import org.apache.spark.sql.types.StructType
+
+import java.util
+
+/**
+ * Basic implementation that represents a table which is staged for being committed.
+ * @param ident table ident
+ * @param table table
+ * @param catalog table catalog
+ */
+case class BaseStagedTable(ident: Identifier,
Review comment:
I was confused about the name and now think would it be better named `BasicStagedTable` or `SimpleStagedTable` ? `BaseXXX` implies to-be-extended but it's actually parallel to `HoodieStagedTable`
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlUtils.scala
##########
@@ -19,10 +19,8 @@ package org.apache.spark.sql.hudi
import org.apache.hudi.SparkAdapterSupport
import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.expressions.{And, Cast, Expression, Literal}
+import org.apache.spark.sql.catalyst.expressions.{And, Expression}
import org.apache.spark.sql.catalyst.plans.logical.{MergeIntoTable, SubqueryAlias}
-import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.{DataType, NullType}
object HoodieSqlUtils extends SparkAdapterSupport {
Review comment:
just wonder if this can be merged into `HoodieSqlCommonUtils` to have fewer classes for clarity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017239081
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365)
* b6b74ca2553aab849511a37f0efd411fef068285 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017499750
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r791287818
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
+ assert(hoodieCatalogTable.primaryKeys.nonEmpty,
+ s"There are no primary key in table ${hoodieCatalogTable.table.identifier}, cannot execute update operator")
+ val enableHive = isEnableHive(sparkSession)
+
+ withSparkConf(sparkSession, catalogProperties) {
+ Map(
+ "path" -> hoodieCatalogTable.tableLocation,
+ RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
+ PRECOMBINE_FIELD.key -> preCombineColumn,
+ TBL_NAME.key -> hoodieCatalogTable.tableName,
+ HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
+ URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
+ KEYGENERATOR_CLASS_NAME.key -> classOf[SqlKeyGenerator].getCanonicalName,
+ SqlKeyGenerator.ORIGIN_KEYGEN_CLASS_NAME -> tableConfig.getKeyGeneratorClassName,
+ OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+ PARTITIONPATH_FIELD.key -> tableConfig.getPartitionFieldProp,
+ META_SYNC_ENABLED.key -> enableHive.toString,
+ HIVE_SYNC_MODE.key -> HiveSyncMode.HMS.name(),
+ HIVE_USE_JDBC.key -> "false",
+ HIVE_DATABASE.key -> tableId.database.getOrElse("default"),
+ HIVE_TABLE.key -> tableId.table,
+ HIVE_PARTITION_FIELDS.key -> tableConfig.getPartitionFieldProp,
+ HIVE_PARTITION_EXTRACTOR_CLASS.key -> classOf[MultiPartKeysValueExtractor].getCanonicalName,
+ HIVE_SUPPORT_TIMESTAMP_TYPE.key -> "true",
+ HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key -> "200",
+ SqlKeyGenerator.PARTITION_SCHEMA -> hoodieCatalogTable.partitionSchema.toDDL
+ )
+ }
+ }
+
+ def cast(exp: Expression, field: StructField, sqlConf: SQLConf): Expression = {
+ castIfNeeded(exp, field.dataType, sqlConf)
+ }
+
+ /**
+ * Build the default config for insert.
+ * @return
+ */
+ def buildHoodieInsertConfig(hoodieCatalogTable: HoodieCatalogTable,
+ sparkSession: SparkSession,
+ isOverwrite: Boolean,
+ insertPartitions: Map[String, Option[String]] = Map.empty,
+ extraOptions: Map[String, String]): Map[String, String] = {
+
+ if (insertPartitions.nonEmpty &&
+ (insertPartitions.keys.toSet != hoodieCatalogTable.partitionFields.toSet)) {
+ throw new IllegalArgumentException(s"Insert partition fields" +
+ s"[${insertPartitions.keys.mkString(" " )}]" +
+ s" not equal to the defined partition in table[${hoodieCatalogTable.partitionFields.mkString(",")}]")
+ }
Review comment:
here if replace it with require, the condition will be very complicated, so i just left it as it is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013839138
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5e6315181eb95abc55bc6b1ee4f3488a60d65d0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013838711
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5e6315181eb95abc55bc6b1ee4f3488a60d65d0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013843123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5e6315181eb95abc55bc6b1ee4f3488a60d65d0 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280)
* 27cba00d5daf148f99be78e72dade08229491d18 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1034600357
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038542507
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960)
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038538563
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960)
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r805451359
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlUtils.scala
##########
@@ -19,10 +19,8 @@ package org.apache.spark.sql.hudi
import org.apache.hudi.SparkAdapterSupport
import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.expressions.{And, Cast, Expression, Literal}
+import org.apache.spark.sql.catalyst.expressions.{And, Expression}
import org.apache.spark.sql.catalyst.plans.logical.{MergeIntoTable, SubqueryAlias}
-import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.{DataType, NullType}
object HoodieSqlUtils extends SparkAdapterSupport {
Review comment:
because the methods in the object are only used in hudi-spark module, so i just make it still locate in the object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013869592
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a5fecdc15707afd659887b025d1e1094543468e7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282)
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013869592
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a5fecdc15707afd659887b025d1e1094543468e7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282)
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013855071
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
YannByron commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r790287307
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark3Analysis.scala
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.analysis
+
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.{DefaultSource, SparkAdapterSupport}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.{ResolvedTable, UnresolvedPartitionSpec}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, V2SessionCatalog}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, getTableLocation, isHoodieTable, removeMetaFields, tableExistsInPath}
+import org.apache.spark.sql.hudi.catalog.{HoodieCatalog, HoodieConfigHelper, HoodieInternalV2Table}
+import org.apache.spark.sql.hudi.command.{AlterHoodieTableDropPartitionCommand, ShowHoodieTablePartitionsCommand, TruncateHoodieTableCommand}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{AnalysisException, SQLContext, SparkSession}
+
+/**
+ * Rule for convert the logical plan to command.
+ * @param sparkSession
+ */
+case class HoodieSpark3Analysis(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsDown {
+ case dsv2 @ DataSourceV2Relation(d: HoodieInternalV2Table, _, _, _, _) =>
+ val output = dsv2.output
+ val catalogTable = if (d.catalogTable.isDefined) {
+ Some(d.v1Table)
+ } else {
+ None
+ }
+ val relation = new DefaultSource().createRelation(new SQLContext(sparkSession),
+ buildHoodieConfig(d.hoodieCatalogTable))
+ LogicalRelation(relation, output, catalogTable, isStreaming = false)
+ case a @ InsertIntoStatement(r: DataSourceV2Relation, partitionSpec, _, _, _, _) if a.query.resolved &&
+ r.table.isInstanceOf[HoodieInternalV2Table] &&
+ needsSchemaAdjustment(a.query, r.table.asInstanceOf[HoodieInternalV2Table], partitionSpec, r.schema) =>
+ val projection = resolveQueryColumnsByOrdinal(a.query, r.output)
+ if (projection != a.query) {
+ a.copy(query = projection)
+ } else {
+ a
+ }
+ }
+
+ private def needsSchemaAdjustment(query: LogicalPlan,
+ hoodieTable: HoodieInternalV2Table,
+ partitionSpec: Map[String, Option[String]],
+ schema: StructType): Boolean = {
+ val output = query.output
+ val queryOutputWithoutMetaFields = removeMetaFields(output)
+ val partitionFields = hoodieTable.hoodieCatalogTable.partitionFields
+ val partitionSchema = hoodieTable.hoodieCatalogTable.partitionSchema
+ val staticPartitionValues = partitionSpec.filter(p => p._2.isDefined).mapValues(_.get)
+
+ assert(staticPartitionValues.isEmpty ||
+ staticPartitionValues.size == partitionSchema.size,
+ s"Required partition columns is: ${partitionSchema.json}, Current static partitions " +
+ s"is: ${staticPartitionValues.mkString("," + "")}")
+
+ assert(staticPartitionValues.size + queryOutputWithoutMetaFields.size
+ == hoodieTable.hoodieCatalogTable.tableSchemaWithoutMetaFields.size,
+ s"Required select columns count: ${hoodieTable.hoodieCatalogTable.tableSchemaWithoutMetaFields.size}, " +
+ s"Current select columns(including static partition column) count: " +
+ s"${staticPartitionValues.size + queryOutputWithoutMetaFields.size}，columns: " +
+ s"(${(queryOutputWithoutMetaFields.map(_.name) ++ staticPartitionValues.keys).mkString(",")})")
+
+ // static partition insert.
+ if (staticPartitionValues.nonEmpty) {
+ // drop partition fields in origin schema to align fields.
+ schema.dropWhile(p => partitionFields.contains(p.name))
+ }
+
+ val existingSchemaOutput = output.take(schema.length)
+ existingSchemaOutput.map(_.name) != schema.map(_.name) ||
+ existingSchemaOutput.map(_.dataType) != schema.map(_.dataType)
+ }
+
+ private def resolveQueryColumnsByOrdinal(query: LogicalPlan,
+ targetAttrs: Seq[Attribute]): LogicalPlan = {
+ // always add a Cast. it will be removed in the optimizer if it is unnecessary.
+ val project = query.output.zipWithIndex.map { case (attr, i) =>
+ if (i < targetAttrs.length) {
+ val targetAttr = targetAttrs(i)
+ val castAttr = castIfNeeded(attr.withNullability(targetAttr.nullable), targetAttr.dataType, conf)
+ Alias(castAttr, targetAttr.name)()
+ } else {
+ attr
+ }
+ }
+ Project(project, query)
+ }
+}
+
+/**
+ * Rule for resolve hoodie's extended syntax or rewrite some logical plan.
+ * @param sparkSession
+ */
+case class HoodieSpark3ResolveReferences(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+ // Fill schema for Create Table without specify schema info
+ case c @ CreateV2Table(tableCatalog, tableName, schema, _, properties, _)
+ if isHoodieTable(properties) =>
+
+ val hoodieCatalog = tableCatalog match {
+ case catalog: HoodieCatalog => catalog
+ case _ => tableCatalog.asInstanceOf[V2SessionCatalog]
+ }
+ val tablePath = getTableLocation(properties,
+ TableIdentifier(tableName.name(), tableName.namespace().lastOption), sparkSession)
+
+ val tableExistInCatalog = hoodieCatalog.tableExists(tableName)
+ // Only when the table has not exist in catalog, we need to fill the schema info for creating table.
+ if (!tableExistInCatalog && tableExistsInPath(tablePath, sparkSession.sessionState.newHadoopConf())) {
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(tablePath)
+ .setConf(sparkSession.sessionState.newHadoopConf())
+ .build()
+ val tableSchema = HoodieSqlCommonUtils.getTableSqlSchema(metaClient)
+ if (tableSchema.isDefined && schema.isEmpty) {
+ // Fill the schema with the schema from the table
+ c.copy(tableSchema = tableSchema.get)
+ } else if (tableSchema.isDefined && schema != tableSchema.get) {
+ throw new AnalysisException(s"Specified schema in create table statement is not equal to the table schema." +
+ s"You should not specify the schema for an exist table: $tableName ")
+ } else {
+ c
+ }
+ } else {
+ c
+ }
+ case DropPartitions(child, specs, ifExists, purge)
Review comment:
can we deal with `DropParitions` like `ShowPartitions` or `TruncateTable` as follows. Transform the hoodie command in `PostAnalysisRule`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017400948
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* b6b74ca2553aab849511a37f0efd411fef068285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370)
* addb9f9925913e522852b2efcb0a80fccc4153f2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017232114
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038164254
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* e93af4603cc620726fc34a7d4aa482ee0acb7c41 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865)
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1034558510
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 0abd457fb40ee1af67da5d283cd6c09e6e07dac9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490)
* e93af4603cc620726fc34a7d4aa482ee0acb7c41 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1020807520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 0abd457fb40ee1af67da5d283cd6c09e6e07dac9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038218350
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038141572
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r791287041
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala
##########
@@ -94,11 +94,17 @@ class TestHoodieSparkSqlWriter {
* Utility method for initializing the spark context.
*/
def initSparkContext(): Unit = {
+ val sparkConf = new SparkConf()
+ if (HoodieSparkUtils.isSpark3_2) {
+ sparkConf.set("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
Review comment:
here is because in the class we use Spark SQL to create hudi table, so we need to set HoodieCatalog
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala
##########
@@ -42,12 +43,39 @@ object HoodieAnalysis {
Seq(
session => HoodieResolveReferences(session),
session => HoodieAnalysis(session)
- )
+ ) ++ extraResolutionRules()
def customPostHocResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] =
Seq(
session => HoodiePostAnalysisRule(session)
- )
+ ) ++ extraPostHocResolutionRules()
+
+ def extraResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] = {
+ if (HoodieSparkUtils.isSpark3_2) {
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala
##########
@@ -42,12 +43,39 @@ object HoodieAnalysis {
Seq(
session => HoodieResolveReferences(session),
session => HoodieAnalysis(session)
- )
+ ) ++ extraResolutionRules()
def customPostHocResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] =
Seq(
session => HoodiePostAnalysisRule(session)
- )
+ ) ++ extraPostHocResolutionRules()
+
+ def extraResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] = {
+ if (HoodieSparkUtils.isSpark3_2) {
+ val spark3AnalysisClass = "org.apache.spark.sql.hudi.analysis.HoodieSpark3Analysis"
+ val spark3Analysis: SparkSession => Rule[LogicalPlan] =
+ session => ReflectionUtils.loadClass(spark3AnalysisClass, session).asInstanceOf[Rule[LogicalPlan]]
+
+ val spark3ResolveReferences = "org.apache.spark.sql.hudi.analysis.HoodieSpark3ResolveReferences"
+ val spark3References: SparkSession => Rule[LogicalPlan] =
+ session => ReflectionUtils.loadClass(spark3ResolveReferences, session).asInstanceOf[Rule[LogicalPlan]]
+
+ Seq(spark3Analysis, spark3References)
+ } else {
+ Seq.empty
+ }
+ }
+
+ def extraPostHocResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] =
+ if (HoodieSparkUtils.isSpark3_2) {
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HudiSpark3SqlUtils.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
+import org.apache.spark.sql.connector.expressions.{BucketTransform, FieldReference, IdentityTransform, Transform}
+
+import scala.collection.mutable
+
+object HudiSpark3SqlUtils {
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
+ assert(hoodieCatalogTable.primaryKeys.nonEmpty,
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
+ assert(hoodieCatalogTable.primaryKeys.nonEmpty,
+ s"There are no primary key in table ${hoodieCatalogTable.table.identifier}, cannot execute update operator")
+ val enableHive = isEnableHive(sparkSession)
+
+ withSparkConf(sparkSession, catalogProperties) {
+ Map(
+ "path" -> hoodieCatalogTable.tableLocation,
+ RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
+ PRECOMBINE_FIELD.key -> preCombineColumn,
+ TBL_NAME.key -> hoodieCatalogTable.tableName,
+ HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
+ URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
+ KEYGENERATOR_CLASS_NAME.key -> classOf[SqlKeyGenerator].getCanonicalName,
+ SqlKeyGenerator.ORIGIN_KEYGEN_CLASS_NAME -> tableConfig.getKeyGeneratorClassName,
+ OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+ PARTITIONPATH_FIELD.key -> tableConfig.getPartitionFieldProp,
+ META_SYNC_ENABLED.key -> enableHive.toString,
+ HIVE_SYNC_MODE.key -> HiveSyncMode.HMS.name(),
+ HIVE_USE_JDBC.key -> "false",
+ HIVE_DATABASE.key -> tableId.database.getOrElse("default"),
+ HIVE_TABLE.key -> tableId.table,
+ HIVE_PARTITION_FIELDS.key -> tableConfig.getPartitionFieldProp,
+ HIVE_PARTITION_EXTRACTOR_CLASS.key -> classOf[MultiPartKeysValueExtractor].getCanonicalName,
+ HIVE_SUPPORT_TIMESTAMP_TYPE.key -> "true",
+ HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key -> "200",
+ SqlKeyGenerator.PARTITION_SCHEMA -> hoodieCatalogTable.partitionSchema.toDDL
+ )
+ }
+ }
+
+ def cast(exp: Expression, field: StructField, sqlConf: SQLConf): Expression = {
+ castIfNeeded(exp, field.dataType, sqlConf)
+ }
Review comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r791288079
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieInternalV2Table.scala
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.common.table.{HoodieTableConfig, HoodieTableMetaClient}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogTable, HoodieCatalogTable}
+import org.apache.spark.sql.connector.catalog.TableCapability._
+import org.apache.spark.sql.connector.catalog.{SupportsWrite, Table, TableCapability, V2TableWithV1Fallback}
+import org.apache.spark.sql.connector.expressions.{FieldReference, IdentityTransform, Transform}
+import org.apache.spark.sql.connector.write._
+import org.apache.spark.sql.sources.{Filter, InsertableRelation}
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
+
+import java.util
+import scala.collection.JavaConverters.{mapAsJavaMapConverter, setAsJavaSetConverter}
+
+case class HoodieInternalV2Table(spark: SparkSession,
+ path: String,
+ catalogTable: Option[CatalogTable] = None,
+ tableIdentifier: Option[String] = None,
+ options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty())
+ extends Table with SupportsWrite with V2TableWithV1Fallback {
+
+ lazy val hoodieCatalogTable: HoodieCatalogTable = if (catalogTable.isDefined) {
+ HoodieCatalogTable(spark, catalogTable.get)
+ } else {
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient.builder()
+ .setBasePath(path)
+ .setConf(SparkSession.active.sessionState.newHadoopConf)
+ .build()
+
+ val tableConfig: HoodieTableConfig = metaClient.getTableConfig
+ val tableName: String = tableConfig.getTableName
+
+ HoodieCatalogTable(spark, TableIdentifier(tableName))
+ }
+
+ private lazy val tableSchema: StructType = hoodieCatalogTable.tableSchema
+
+ override def name(): String = hoodieCatalogTable.table.identifier.unquotedString
+
+ override def schema(): StructType = tableSchema
+
+ override def capabilities(): util.Set[TableCapability] = Set(
+ BATCH_READ, V1_BATCH_WRITE, OVERWRITE_BY_FILTER, TRUNCATE, ACCEPT_ANY_SCHEMA
+ ).asJava
+
+ override def properties(): util.Map[String, String] = {
+ val map = new util.HashMap[String, String]()
+ map.put("provider", "hudi")
Review comment:
here because the catalog not only handle hudi tables, but would also handle other types of tables, such as parquet.
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieInternalV2Table.scala
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.common.table.{HoodieTableConfig, HoodieTableMetaClient}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogTable, HoodieCatalogTable}
+import org.apache.spark.sql.connector.catalog.TableCapability._
+import org.apache.spark.sql.connector.catalog.{SupportsWrite, Table, TableCapability, V2TableWithV1Fallback}
+import org.apache.spark.sql.connector.expressions.{FieldReference, IdentityTransform, Transform}
+import org.apache.spark.sql.connector.write._
+import org.apache.spark.sql.sources.{Filter, InsertableRelation}
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
+
+import java.util
+import scala.collection.JavaConverters.{mapAsJavaMapConverter, setAsJavaSetConverter}
+
+case class HoodieInternalV2Table(spark: SparkSession,
+ path: String,
+ catalogTable: Option[CatalogTable] = None,
+ tableIdentifier: Option[String] = None,
+ options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty())
+ extends Table with SupportsWrite with V2TableWithV1Fallback {
+
+ lazy val hoodieCatalogTable: HoodieCatalogTable = if (catalogTable.isDefined) {
+ HoodieCatalogTable(spark, catalogTable.get)
+ } else {
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient.builder()
+ .setBasePath(path)
+ .setConf(SparkSession.active.sessionState.newHadoopConf)
+ .build()
+
+ val tableConfig: HoodieTableConfig = metaClient.getTableConfig
+ val tableName: String = tableConfig.getTableName
+
+ HoodieCatalogTable(spark, TableIdentifier(tableName))
+ }
+
+ private lazy val tableSchema: StructType = hoodieCatalogTable.tableSchema
+
+ override def name(): String = hoodieCatalogTable.table.identifier.unquotedString
+
+ override def schema(): StructType = tableSchema
+
+ override def capabilities(): util.Set[TableCapability] = Set(
+ BATCH_READ, V1_BATCH_WRITE, OVERWRITE_BY_FILTER, TRUNCATE, ACCEPT_ANY_SCHEMA
+ ).asJava
+
+ override def properties(): util.Map[String, String] = {
+ val map = new util.HashMap[String, String]()
+ map.put("provider", "hudi")
+ map.putAll(hoodieCatalogTable.catalogProperties.asJava)
+ map
+ }
+
+ override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
+ new WriteIntoHoodieBuilder(info.options, hoodieCatalogTable, spark)
+ }
+
+ override def v1Table: CatalogTable = hoodieCatalogTable.table
+
+ override def partitioning(): Array[Transform] = {
+ hoodieCatalogTable.partitionFields.map { col =>
+ new IdentityTransform(new FieldReference(Seq(col)))
+ }.toArray
+ }
+
+}
+
+private class WriteIntoHoodieBuilder(writeOptions: CaseInsensitiveStringMap,
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/BaseStagedTable.scala
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.connector.catalog._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder}
+import org.apache.spark.sql.types.StructType
+
+import java.util
+
+case class BaseStagedTable(ident: Identifier,
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013848003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5e6315181eb95abc55bc6b1ee4f3488a60d65d0 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280)
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013867484
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
* a5fecdc15707afd659887b025d1e1094543468e7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013842745
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5e6315181eb95abc55bc6b1ee4f3488a60d65d0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280)
* 27cba00d5daf148f99be78e72dade08229491d18 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013843123
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5e6315181eb95abc55bc6b1ee4f3488a60d65d0 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280)
* 27cba00d5daf148f99be78e72dade08229491d18 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
YannByron commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r790285210
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/connector/catalog/HoodieIdentifierHelper.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.catalog
+
+import java.util
+import java.util.Objects
+
+object HoodieIdentifierHelper {
+ def of(namespace: Array[String], name: String): Identifier = {
+ HoodieIdentifier(namespace, name)
Review comment:
maybe using the companion object of `HoodieIdentifier` and it's `apply` method is better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
xushiyan commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017219746
@leesf is this to replace https://github.com/apache/hudi/pull/4350 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017386448
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* b6b74ca2553aab849511a37f0efd411fef068285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017232114
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013887306
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013869160
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
* a5fecdc15707afd659887b025d1e1094543468e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282)
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1020724538
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439)
* 0abd457fb40ee1af67da5d283cd6c09e6e07dac9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1019440860
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1034600357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* e93af4603cc620726fc34a7d4aa482ee0acb7c41 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038763912
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
}, {
"hash" : "22f6e2f42db71974b19aa30809420ed7afa50cc5",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5975",
"triggerID" : "22f6e2f42db71974b19aa30809420ed7afa50cc5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965)
* 22f6e2f42db71974b19aa30809420ed7afa50cc5 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5975)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r805450982
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala
##########
@@ -171,7 +171,7 @@ class IncrementalRelation(val sqlContext: SQLContext,
if (metaBootstrapFileIdToFullPath.nonEmpty) {
df = sqlContext.sparkSession.read
- .format("hudi")
+ .format("hudi_v1")
Review comment:
yes, currently we use hudi_v1, and will use v2 as well after finding a solution, filing a jira to track this work. https://issues.apache.org/jira/projects/HUDI/issues/HUDI-3416
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/HoodieSqlUtils.scala
##########
@@ -19,10 +19,8 @@ package org.apache.spark.sql.hudi
import org.apache.hudi.SparkAdapterSupport
import org.apache.spark.sql.catalyst.TableIdentifier
-import org.apache.spark.sql.catalyst.expressions.{And, Cast, Expression, Literal}
+import org.apache.spark.sql.catalyst.expressions.{And, Expression}
import org.apache.spark.sql.catalyst.plans.logical.{MergeIntoTable, SubqueryAlias}
-import org.apache.spark.sql.internal.SQLConf
-import org.apache.spark.sql.types.{DataType, NullType}
object HoodieSqlUtils extends SparkAdapterSupport {
Review comment:
because the methods in the object are only used in hudi-spark module, so i just make it still locate in the object.
##########
File path: hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3SqlUtils.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
+import org.apache.spark.sql.connector.expressions.{BucketTransform, FieldReference, IdentityTransform, Transform}
+
+import scala.collection.mutable
+
+object HoodieSpark3SqlUtils {
+ def convertTransforms(partitions: Seq[Transform]): (Seq[String], Option[BucketSpec]) = {
+ val identityCols = new mutable.ArrayBuffer[String]
+ var bucketSpec = Option.empty[BucketSpec]
+
+ partitions.map {
+ case IdentityTransform(FieldReference(Seq(col))) =>
+ identityCols += col
+
+
+ case BucketTransform(numBuckets, FieldReference(Seq(col))) =>
+ bucketSpec = Some(BucketSpec(numBuckets, col :: Nil, Nil))
+
+ case _ =>
+ throw new HoodieException(s"Partitioning by expressions")
Review comment:
changed to `Partitioning by expressions is not supported.`
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/hudi/Spark3DefaultSource.scala
##########
@@ -17,8 +17,30 @@
package org.apache.hudi
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.connector.catalog.{Table, TableProvider}
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.hudi.catalog.HoodieInternalV2Table
import org.apache.spark.sql.sources.DataSourceRegister
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+
+class Spark3DefaultSource extends DefaultSource with DataSourceRegister with TableProvider {
-class Spark3DefaultSource extends DefaultSource with DataSourceRegister {
override def shortName(): String = "hudi"
+
+ def inferSchema: StructType = new StructType()
+
+ override def inferSchema(options: CaseInsensitiveStringMap): StructType = inferSchema
+
+ override def getTable(schema: StructType,
+ partitioning: Array[Transform],
+ properties: java.util.Map[String, String]): Table = {
+ val options = new CaseInsensitiveStringMap(properties)
+ val path = options.get("path")
+ if (path == null) throw new HoodieException("path is null")
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/BaseStagedTable.scala
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.connector.catalog._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder}
+import org.apache.spark.sql.types.StructType
+
+import java.util
+
+/**
+ * Basic implementation that represents a table which is staged for being committed.
+ * @param ident table ident
+ * @param table table
+ * @param catalog table catalog
+ */
+case class BaseStagedTable(ident: Identifier,
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/BaseStagedTable.scala
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.connector.catalog._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder}
+import org.apache.spark.sql.types.StructType
+
+import java.util
+
+/**
+ * Basic implementation that represents a table which is staged for being committed.
+ * @param ident table ident
+ * @param table table
+ * @param catalog table catalog
+ */
+case class BaseStagedTable(ident: Identifier,
+ table: Table,
+ catalog: TableCatalog) extends SupportsWrite with StagedTable {
+ override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
+ info match {
+ case supportsWrite: SupportsWrite => supportsWrite.newWriteBuilder(info)
+ case _ => throw new HoodieException(s"Table implementation does not support writes: $ident.name")
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1034600357
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* e93af4603cc620726fc34a7d4aa482ee0acb7c41 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038820835
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
}, {
"hash" : "22f6e2f42db71974b19aa30809420ed7afa50cc5",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5975",
"triggerID" : "22f6e2f42db71974b19aa30809420ed7afa50cc5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* 22f6e2f42db71974b19aa30809420ed7afa50cc5 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5975)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017405909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* b6b74ca2553aab849511a37f0efd411fef068285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370)
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1019433731
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
YannByron commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r790287042
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark3Analysis.scala
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.analysis
+
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.{DefaultSource, SparkAdapterSupport}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.{ResolvedTable, UnresolvedPartitionSpec}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, V2SessionCatalog}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, getTableLocation, isHoodieTable, removeMetaFields, tableExistsInPath}
+import org.apache.spark.sql.hudi.catalog.{HoodieCatalog, HoodieConfigHelper, HoodieInternalV2Table}
+import org.apache.spark.sql.hudi.command.{AlterHoodieTableDropPartitionCommand, ShowHoodieTablePartitionsCommand, TruncateHoodieTableCommand}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{AnalysisException, SQLContext, SparkSession}
+
+/**
+ * Rule for convert the logical plan to command.
+ * @param sparkSession
+ */
+case class HoodieSpark3Analysis(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsDown {
+ case dsv2 @ DataSourceV2Relation(d: HoodieInternalV2Table, _, _, _, _) =>
+ val output = dsv2.output
+ val catalogTable = if (d.catalogTable.isDefined) {
+ Some(d.v1Table)
+ } else {
+ None
+ }
+ val relation = new DefaultSource().createRelation(new SQLContext(sparkSession),
+ buildHoodieConfig(d.hoodieCatalogTable))
+ LogicalRelation(relation, output, catalogTable, isStreaming = false)
+ case a @ InsertIntoStatement(r: DataSourceV2Relation, partitionSpec, _, _, _, _) if a.query.resolved &&
+ r.table.isInstanceOf[HoodieInternalV2Table] &&
+ needsSchemaAdjustment(a.query, r.table.asInstanceOf[HoodieInternalV2Table], partitionSpec, r.schema) =>
+ val projection = resolveQueryColumnsByOrdinal(a.query, r.output)
+ if (projection != a.query) {
+ a.copy(query = projection)
+ } else {
+ a
+ }
+ }
+
+ private def needsSchemaAdjustment(query: LogicalPlan,
+ hoodieTable: HoodieInternalV2Table,
+ partitionSpec: Map[String, Option[String]],
+ schema: StructType): Boolean = {
+ val output = query.output
+ val queryOutputWithoutMetaFields = removeMetaFields(output)
+ val partitionFields = hoodieTable.hoodieCatalogTable.partitionFields
+ val partitionSchema = hoodieTable.hoodieCatalogTable.partitionSchema
+ val staticPartitionValues = partitionSpec.filter(p => p._2.isDefined).mapValues(_.get)
+
+ assert(staticPartitionValues.isEmpty ||
+ staticPartitionValues.size == partitionSchema.size,
+ s"Required partition columns is: ${partitionSchema.json}, Current static partitions " +
+ s"is: ${staticPartitionValues.mkString("," + "")}")
+
+ assert(staticPartitionValues.size + queryOutputWithoutMetaFields.size
+ == hoodieTable.hoodieCatalogTable.tableSchemaWithoutMetaFields.size,
+ s"Required select columns count: ${hoodieTable.hoodieCatalogTable.tableSchemaWithoutMetaFields.size}, " +
+ s"Current select columns(including static partition column) count: " +
+ s"${staticPartitionValues.size + queryOutputWithoutMetaFields.size}，columns: " +
+ s"(${(queryOutputWithoutMetaFields.map(_.name) ++ staticPartitionValues.keys).mkString(",")})")
+
+ // static partition insert.
+ if (staticPartitionValues.nonEmpty) {
+ // drop partition fields in origin schema to align fields.
+ schema.dropWhile(p => partitionFields.contains(p.name))
+ }
+
+ val existingSchemaOutput = output.take(schema.length)
+ existingSchemaOutput.map(_.name) != schema.map(_.name) ||
+ existingSchemaOutput.map(_.dataType) != schema.map(_.dataType)
+ }
+
+ private def resolveQueryColumnsByOrdinal(query: LogicalPlan,
+ targetAttrs: Seq[Attribute]): LogicalPlan = {
+ // always add a Cast. it will be removed in the optimizer if it is unnecessary.
+ val project = query.output.zipWithIndex.map { case (attr, i) =>
+ if (i < targetAttrs.length) {
+ val targetAttr = targetAttrs(i)
+ val castAttr = castIfNeeded(attr.withNullability(targetAttr.nullable), targetAttr.dataType, conf)
+ Alias(castAttr, targetAttr.name)()
+ } else {
+ attr
+ }
+ }
+ Project(project, query)
+ }
+}
+
+/**
+ * Rule for resolve hoodie's extended syntax or rewrite some logical plan.
+ * @param sparkSession
+ */
+case class HoodieSpark3ResolveReferences(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+ // Fill schema for Create Table without specify schema info
+ case c @ CreateV2Table(tableCatalog, tableName, schema, _, properties, _)
+ if isHoodieTable(properties) =>
+
+ val hoodieCatalog = tableCatalog match {
+ case catalog: HoodieCatalog => catalog
+ case _ => tableCatalog.asInstanceOf[V2SessionCatalog]
+ }
+ val tablePath = getTableLocation(properties,
+ TableIdentifier(tableName.name(), tableName.namespace().lastOption), sparkSession)
+
+ val tableExistInCatalog = hoodieCatalog.tableExists(tableName)
+ // Only when the table has not exist in catalog, we need to fill the schema info for creating table.
+ if (!tableExistInCatalog && tableExistsInPath(tablePath, sparkSession.sessionState.newHadoopConf())) {
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(tablePath)
+ .setConf(sparkSession.sessionState.newHadoopConf())
+ .build()
+ val tableSchema = HoodieSqlCommonUtils.getTableSqlSchema(metaClient)
+ if (tableSchema.isDefined && schema.isEmpty) {
+ // Fill the schema with the schema from the table
+ c.copy(tableSchema = tableSchema.get)
Review comment:
i think we don't need to fill the table schema in here. We can move this process to `HoodieCatalog.createHoodieTable`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
YannByron commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r790286075
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark3Analysis.scala
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.analysis
+
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.{DefaultSource, SparkAdapterSupport}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.{ResolvedTable, UnresolvedPartitionSpec}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, V2SessionCatalog}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, getTableLocation, isHoodieTable, removeMetaFields, tableExistsInPath}
+import org.apache.spark.sql.hudi.catalog.{HoodieCatalog, HoodieConfigHelper, HoodieInternalV2Table}
+import org.apache.spark.sql.hudi.command.{AlterHoodieTableDropPartitionCommand, ShowHoodieTablePartitionsCommand, TruncateHoodieTableCommand}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{AnalysisException, SQLContext, SparkSession}
+
+/**
+ * Rule for convert the logical plan to command.
+ * @param sparkSession
+ */
+case class HoodieSpark3Analysis(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsDown {
+ case dsv2 @ DataSourceV2Relation(d: HoodieInternalV2Table, _, _, _, _) =>
+ val output = dsv2.output
+ val catalogTable = if (d.catalogTable.isDefined) {
+ Some(d.v1Table)
+ } else {
+ None
+ }
+ val relation = new DefaultSource().createRelation(new SQLContext(sparkSession),
+ buildHoodieConfig(d.hoodieCatalogTable))
+ LogicalRelation(relation, output, catalogTable, isStreaming = false)
+ case a @ InsertIntoStatement(r: DataSourceV2Relation, partitionSpec, _, _, _, _) if a.query.resolved &&
+ r.table.isInstanceOf[HoodieInternalV2Table] &&
+ needsSchemaAdjustment(a.query, r.table.asInstanceOf[HoodieInternalV2Table], partitionSpec, r.schema) =>
+ val projection = resolveQueryColumnsByOrdinal(a.query, r.output)
+ if (projection != a.query) {
+ a.copy(query = projection)
+ } else {
+ a
+ }
+ }
+
+ private def needsSchemaAdjustment(query: LogicalPlan,
Review comment:
add some description for this method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017221717
> @leesf is this to replace #4350 ?
@xushiyan yes, #4350 would be closed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017229766
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283)
* 1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013869160
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
* a5fecdc15707afd659887b025d1e1094543468e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282)
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r791286706
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/connector/catalog/HoodieIdentifierHelper.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.catalog
+
+import java.util
+import java.util.Objects
+
+object HoodieIdentifierHelper {
+ def of(namespace: Array[String], name: String): Identifier = {
+ HoodieIdentifier(namespace, name)
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark3Analysis.scala
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.analysis
+
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.{DefaultSource, SparkAdapterSupport}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.{ResolvedTable, UnresolvedPartitionSpec}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, V2SessionCatalog}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, getTableLocation, isHoodieTable, removeMetaFields, tableExistsInPath}
+import org.apache.spark.sql.hudi.catalog.{HoodieCatalog, HoodieConfigHelper, HoodieInternalV2Table}
+import org.apache.spark.sql.hudi.command.{AlterHoodieTableDropPartitionCommand, ShowHoodieTablePartitionsCommand, TruncateHoodieTableCommand}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{AnalysisException, SQLContext, SparkSession}
+
+/**
+ * Rule for convert the logical plan to command.
+ * @param sparkSession
+ */
+case class HoodieSpark3Analysis(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsDown {
+ case dsv2 @ DataSourceV2Relation(d: HoodieInternalV2Table, _, _, _, _) =>
+ val output = dsv2.output
+ val catalogTable = if (d.catalogTable.isDefined) {
+ Some(d.v1Table)
+ } else {
+ None
+ }
+ val relation = new DefaultSource().createRelation(new SQLContext(sparkSession),
+ buildHoodieConfig(d.hoodieCatalogTable))
+ LogicalRelation(relation, output, catalogTable, isStreaming = false)
+ case a @ InsertIntoStatement(r: DataSourceV2Relation, partitionSpec, _, _, _, _) if a.query.resolved &&
+ r.table.isInstanceOf[HoodieInternalV2Table] &&
+ needsSchemaAdjustment(a.query, r.table.asInstanceOf[HoodieInternalV2Table], partitionSpec, r.schema) =>
+ val projection = resolveQueryColumnsByOrdinal(a.query, r.output)
+ if (projection != a.query) {
+ a.copy(query = projection)
+ } else {
+ a
+ }
+ }
+
+ private def needsSchemaAdjustment(query: LogicalPlan,
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark3Analysis.scala
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.analysis
+
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.{DefaultSource, SparkAdapterSupport}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.{ResolvedTable, UnresolvedPartitionSpec}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, V2SessionCatalog}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, getTableLocation, isHoodieTable, removeMetaFields, tableExistsInPath}
+import org.apache.spark.sql.hudi.catalog.{HoodieCatalog, HoodieConfigHelper, HoodieInternalV2Table}
+import org.apache.spark.sql.hudi.command.{AlterHoodieTableDropPartitionCommand, ShowHoodieTablePartitionsCommand, TruncateHoodieTableCommand}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{AnalysisException, SQLContext, SparkSession}
+
+/**
+ * Rule for convert the logical plan to command.
+ * @param sparkSession
+ */
+case class HoodieSpark3Analysis(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsDown {
+ case dsv2 @ DataSourceV2Relation(d: HoodieInternalV2Table, _, _, _, _) =>
+ val output = dsv2.output
+ val catalogTable = if (d.catalogTable.isDefined) {
+ Some(d.v1Table)
+ } else {
+ None
+ }
+ val relation = new DefaultSource().createRelation(new SQLContext(sparkSession),
+ buildHoodieConfig(d.hoodieCatalogTable))
+ LogicalRelation(relation, output, catalogTable, isStreaming = false)
+ case a @ InsertIntoStatement(r: DataSourceV2Relation, partitionSpec, _, _, _, _) if a.query.resolved &&
+ r.table.isInstanceOf[HoodieInternalV2Table] &&
+ needsSchemaAdjustment(a.query, r.table.asInstanceOf[HoodieInternalV2Table], partitionSpec, r.schema) =>
+ val projection = resolveQueryColumnsByOrdinal(a.query, r.output)
+ if (projection != a.query) {
+ a.copy(query = projection)
+ } else {
+ a
+ }
+ }
+
+ private def needsSchemaAdjustment(query: LogicalPlan,
+ hoodieTable: HoodieInternalV2Table,
+ partitionSpec: Map[String, Option[String]],
+ schema: StructType): Boolean = {
+ val output = query.output
+ val queryOutputWithoutMetaFields = removeMetaFields(output)
+ val partitionFields = hoodieTable.hoodieCatalogTable.partitionFields
+ val partitionSchema = hoodieTable.hoodieCatalogTable.partitionSchema
+ val staticPartitionValues = partitionSpec.filter(p => p._2.isDefined).mapValues(_.get)
+
+ assert(staticPartitionValues.isEmpty ||
+ staticPartitionValues.size == partitionSchema.size,
+ s"Required partition columns is: ${partitionSchema.json}, Current static partitions " +
+ s"is: ${staticPartitionValues.mkString("," + "")}")
+
+ assert(staticPartitionValues.size + queryOutputWithoutMetaFields.size
+ == hoodieTable.hoodieCatalogTable.tableSchemaWithoutMetaFields.size,
+ s"Required select columns count: ${hoodieTable.hoodieCatalogTable.tableSchemaWithoutMetaFields.size}, " +
+ s"Current select columns(including static partition column) count: " +
+ s"${staticPartitionValues.size + queryOutputWithoutMetaFields.size}，columns: " +
+ s"(${(queryOutputWithoutMetaFields.map(_.name) ++ staticPartitionValues.keys).mkString(",")})")
+
+ // static partition insert.
+ if (staticPartitionValues.nonEmpty) {
+ // drop partition fields in origin schema to align fields.
+ schema.dropWhile(p => partitionFields.contains(p.name))
+ }
+
+ val existingSchemaOutput = output.take(schema.length)
+ existingSchemaOutput.map(_.name) != schema.map(_.name) ||
+ existingSchemaOutput.map(_.dataType) != schema.map(_.dataType)
+ }
+
+ private def resolveQueryColumnsByOrdinal(query: LogicalPlan,
+ targetAttrs: Seq[Attribute]): LogicalPlan = {
+ // always add a Cast. it will be removed in the optimizer if it is unnecessary.
+ val project = query.output.zipWithIndex.map { case (attr, i) =>
+ if (i < targetAttrs.length) {
+ val targetAttr = targetAttrs(i)
+ val castAttr = castIfNeeded(attr.withNullability(targetAttr.nullable), targetAttr.dataType, conf)
+ Alias(castAttr, targetAttr.name)()
+ } else {
+ attr
+ }
+ }
+ Project(project, query)
+ }
+}
+
+/**
+ * Rule for resolve hoodie's extended syntax or rewrite some logical plan.
+ * @param sparkSession
+ */
+case class HoodieSpark3ResolveReferences(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+ // Fill schema for Create Table without specify schema info
+ case c @ CreateV2Table(tableCatalog, tableName, schema, _, properties, _)
+ if isHoodieTable(properties) =>
+
+ val hoodieCatalog = tableCatalog match {
+ case catalog: HoodieCatalog => catalog
+ case _ => tableCatalog.asInstanceOf[V2SessionCatalog]
+ }
+ val tablePath = getTableLocation(properties,
+ TableIdentifier(tableName.name(), tableName.namespace().lastOption), sparkSession)
+
+ val tableExistInCatalog = hoodieCatalog.tableExists(tableName)
+ // Only when the table has not exist in catalog, we need to fill the schema info for creating table.
+ if (!tableExistInCatalog && tableExistsInPath(tablePath, sparkSession.sessionState.newHadoopConf())) {
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(tablePath)
+ .setConf(sparkSession.sessionState.newHadoopConf())
+ .build()
+ val tableSchema = HoodieSqlCommonUtils.getTableSqlSchema(metaClient)
+ if (tableSchema.isDefined && schema.isEmpty) {
+ // Fill the schema with the schema from the table
+ c.copy(tableSchema = tableSchema.get)
+ } else if (tableSchema.isDefined && schema != tableSchema.get) {
+ throw new AnalysisException(s"Specified schema in create table statement is not equal to the table schema." +
+ s"You should not specify the schema for an exist table: $tableName ")
+ } else {
+ c
+ }
+ } else {
+ c
+ }
+ case DropPartitions(child, specs, ifExists, purge)
Review comment:
done
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala
##########
@@ -92,4 +95,31 @@ trait SparkAdapter extends Serializable {
* ParserInterface#parseMultipartIdentifier is supported since spark3, for spark2 this should not be called.
*/
def parseMultipartIdentifier(parser: ParserInterface, sqlText: String): Seq[String]
+
+ def isHoodieTable(table: LogicalPlan, spark: SparkSession): Boolean = {
+ tripAlias(table) match {
+ case LogicalRelation(_, _, Some(tbl), _) => isHoodieTable(tbl)
+ case relation: UnresolvedRelation =>
+ isHoodieTable(toTableIdentifier(relation), spark)
+ case _=> false
+ }
+ }
+
+ def tripAlias(plan: LogicalPlan): LogicalPlan = {
+ plan match {
+ case SubqueryAlias(_, relation: LogicalPlan) =>
+ tripAlias(relation)
+ case other =>
+ other
+ }
+ }
+
+ def isHoodieTable(table: CatalogTable): Boolean = {
+ table.provider.map(_.toLowerCase(Locale.ROOT)).orNull == "hudi"
+ }
+
+ def isHoodieTable(tableId: TableIdentifier, spark: SparkSession): Boolean = {
+ val table = spark.sessionState.catalog.getTableMetadata(tableId)
+ isHoodieTable(table)
+ }
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1020807520
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 0abd457fb40ee1af67da5d283cd6c09e6e07dac9 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038743897
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
}, {
"hash" : "22f6e2f42db71974b19aa30809420ed7afa50cc5",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "22f6e2f42db71974b19aa30809420ed7afa50cc5",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5965)
* 22f6e2f42db71974b19aa30809420ed7afa50cc5 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] xushiyan merged pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
xushiyan merged pull request #4611:
URL: https://github.com/apache/hudi/pull/4611
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038218350
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r805450982
##########
File path: hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/IncrementalRelation.scala
##########
@@ -171,7 +171,7 @@ class IncrementalRelation(val sqlContext: SQLContext,
if (metaBootstrapFileIdToFullPath.nonEmpty) {
df = sqlContext.sparkSession.read
- .format("hudi")
+ .format("hudi_v1")
Review comment:
yes, currently we use hudi_v1, and will use v2 as well after finding a solution, filing a jira to track this work. https://issues.apache.org/jira/projects/HUDI/issues/HUDI-3416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038141572
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* e93af4603cc620726fc34a7d4aa482ee0acb7c41 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865)
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038538563
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
}, {
"hash" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960)
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
* b67b0c1eb2bc28fc9e610e6e8ec08b22f5eed83e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r805451660
##########
File path: hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3SqlUtils.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
+import org.apache.spark.sql.connector.expressions.{BucketTransform, FieldReference, IdentityTransform, Transform}
+
+import scala.collection.mutable
+
+object HoodieSpark3SqlUtils {
+ def convertTransforms(partitions: Seq[Transform]): (Seq[String], Option[BucketSpec]) = {
+ val identityCols = new mutable.ArrayBuffer[String]
+ var bucketSpec = Option.empty[BucketSpec]
+
+ partitions.map {
+ case IdentityTransform(FieldReference(Seq(col))) =>
+ identityCols += col
+
+
+ case BucketTransform(numBuckets, FieldReference(Seq(col))) =>
+ bucketSpec = Some(BucketSpec(numBuckets, col :: Nil, Nil))
+
+ case _ =>
+ throw new HoodieException(s"Partitioning by expressions")
Review comment:
changed to `Partitioning by expressions is not supported.`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r805452034
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/BaseStagedTable.scala
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.connector.catalog._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder}
+import org.apache.spark.sql.types.StructType
+
+import java.util
+
+/**
+ * Basic implementation that represents a table which is staged for being committed.
+ * @param ident table ident
+ * @param table table
+ * @param catalog table catalog
+ */
+case class BaseStagedTable(ident: Identifier,
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017227414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283)
* 1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017386448
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* b6b74ca2553aab849511a37f0efd411fef068285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1019433731
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013877803
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a5fecdc15707afd659887b025d1e1094543468e7 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282)
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013868788
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
* a5fecdc15707afd659887b025d1e1094543468e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282)
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013839138
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5e6315181eb95abc55bc6b1ee4f3488a60d65d0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013867974
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
* a5fecdc15707afd659887b025d1e1094543468e7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013887306
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a change in pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on a change in pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#discussion_r791286706
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/connector/catalog/HoodieIdentifierHelper.scala
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.catalog
+
+import java.util
+import java.util.Objects
+
+object HoodieIdentifierHelper {
+ def of(namespace: Array[String], name: String): Identifier = {
+ HoodieIdentifier(namespace, name)
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark3Analysis.scala
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.analysis
+
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.{DefaultSource, SparkAdapterSupport}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.{ResolvedTable, UnresolvedPartitionSpec}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, V2SessionCatalog}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, getTableLocation, isHoodieTable, removeMetaFields, tableExistsInPath}
+import org.apache.spark.sql.hudi.catalog.{HoodieCatalog, HoodieConfigHelper, HoodieInternalV2Table}
+import org.apache.spark.sql.hudi.command.{AlterHoodieTableDropPartitionCommand, ShowHoodieTablePartitionsCommand, TruncateHoodieTableCommand}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{AnalysisException, SQLContext, SparkSession}
+
+/**
+ * Rule for convert the logical plan to command.
+ * @param sparkSession
+ */
+case class HoodieSpark3Analysis(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsDown {
+ case dsv2 @ DataSourceV2Relation(d: HoodieInternalV2Table, _, _, _, _) =>
+ val output = dsv2.output
+ val catalogTable = if (d.catalogTable.isDefined) {
+ Some(d.v1Table)
+ } else {
+ None
+ }
+ val relation = new DefaultSource().createRelation(new SQLContext(sparkSession),
+ buildHoodieConfig(d.hoodieCatalogTable))
+ LogicalRelation(relation, output, catalogTable, isStreaming = false)
+ case a @ InsertIntoStatement(r: DataSourceV2Relation, partitionSpec, _, _, _, _) if a.query.resolved &&
+ r.table.isInstanceOf[HoodieInternalV2Table] &&
+ needsSchemaAdjustment(a.query, r.table.asInstanceOf[HoodieInternalV2Table], partitionSpec, r.schema) =>
+ val projection = resolveQueryColumnsByOrdinal(a.query, r.output)
+ if (projection != a.query) {
+ a.copy(query = projection)
+ } else {
+ a
+ }
+ }
+
+ private def needsSchemaAdjustment(query: LogicalPlan,
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark3Analysis.scala
##########
@@ -0,0 +1,187 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.analysis
+
+import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.{DefaultSource, SparkAdapterSupport}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.analysis.{ResolvedTable, UnresolvedPartitionSpec}
+import org.apache.spark.sql.catalyst.expressions.{Alias, Attribute}
+import org.apache.spark.sql.catalyst.plans.logical._
+import org.apache.spark.sql.catalyst.rules.Rule
+import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.IdentifierHelper
+import org.apache.spark.sql.execution.datasources.LogicalRelation
+import org.apache.spark.sql.execution.datasources.v2.{DataSourceV2Relation, V2SessionCatalog}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, getTableLocation, isHoodieTable, removeMetaFields, tableExistsInPath}
+import org.apache.spark.sql.hudi.catalog.{HoodieCatalog, HoodieConfigHelper, HoodieInternalV2Table}
+import org.apache.spark.sql.hudi.command.{AlterHoodieTableDropPartitionCommand, ShowHoodieTablePartitionsCommand, TruncateHoodieTableCommand}
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.{AnalysisException, SQLContext, SparkSession}
+
+/**
+ * Rule for convert the logical plan to command.
+ * @param sparkSession
+ */
+case class HoodieSpark3Analysis(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ override def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsDown {
+ case dsv2 @ DataSourceV2Relation(d: HoodieInternalV2Table, _, _, _, _) =>
+ val output = dsv2.output
+ val catalogTable = if (d.catalogTable.isDefined) {
+ Some(d.v1Table)
+ } else {
+ None
+ }
+ val relation = new DefaultSource().createRelation(new SQLContext(sparkSession),
+ buildHoodieConfig(d.hoodieCatalogTable))
+ LogicalRelation(relation, output, catalogTable, isStreaming = false)
+ case a @ InsertIntoStatement(r: DataSourceV2Relation, partitionSpec, _, _, _, _) if a.query.resolved &&
+ r.table.isInstanceOf[HoodieInternalV2Table] &&
+ needsSchemaAdjustment(a.query, r.table.asInstanceOf[HoodieInternalV2Table], partitionSpec, r.schema) =>
+ val projection = resolveQueryColumnsByOrdinal(a.query, r.output)
+ if (projection != a.query) {
+ a.copy(query = projection)
+ } else {
+ a
+ }
+ }
+
+ private def needsSchemaAdjustment(query: LogicalPlan,
+ hoodieTable: HoodieInternalV2Table,
+ partitionSpec: Map[String, Option[String]],
+ schema: StructType): Boolean = {
+ val output = query.output
+ val queryOutputWithoutMetaFields = removeMetaFields(output)
+ val partitionFields = hoodieTable.hoodieCatalogTable.partitionFields
+ val partitionSchema = hoodieTable.hoodieCatalogTable.partitionSchema
+ val staticPartitionValues = partitionSpec.filter(p => p._2.isDefined).mapValues(_.get)
+
+ assert(staticPartitionValues.isEmpty ||
+ staticPartitionValues.size == partitionSchema.size,
+ s"Required partition columns is: ${partitionSchema.json}, Current static partitions " +
+ s"is: ${staticPartitionValues.mkString("," + "")}")
+
+ assert(staticPartitionValues.size + queryOutputWithoutMetaFields.size
+ == hoodieTable.hoodieCatalogTable.tableSchemaWithoutMetaFields.size,
+ s"Required select columns count: ${hoodieTable.hoodieCatalogTable.tableSchemaWithoutMetaFields.size}, " +
+ s"Current select columns(including static partition column) count: " +
+ s"${staticPartitionValues.size + queryOutputWithoutMetaFields.size}，columns: " +
+ s"(${(queryOutputWithoutMetaFields.map(_.name) ++ staticPartitionValues.keys).mkString(",")})")
+
+ // static partition insert.
+ if (staticPartitionValues.nonEmpty) {
+ // drop partition fields in origin schema to align fields.
+ schema.dropWhile(p => partitionFields.contains(p.name))
+ }
+
+ val existingSchemaOutput = output.take(schema.length)
+ existingSchemaOutput.map(_.name) != schema.map(_.name) ||
+ existingSchemaOutput.map(_.dataType) != schema.map(_.dataType)
+ }
+
+ private def resolveQueryColumnsByOrdinal(query: LogicalPlan,
+ targetAttrs: Seq[Attribute]): LogicalPlan = {
+ // always add a Cast. it will be removed in the optimizer if it is unnecessary.
+ val project = query.output.zipWithIndex.map { case (attr, i) =>
+ if (i < targetAttrs.length) {
+ val targetAttr = targetAttrs(i)
+ val castAttr = castIfNeeded(attr.withNullability(targetAttr.nullable), targetAttr.dataType, conf)
+ Alias(castAttr, targetAttr.name)()
+ } else {
+ attr
+ }
+ }
+ Project(project, query)
+ }
+}
+
+/**
+ * Rule for resolve hoodie's extended syntax or rewrite some logical plan.
+ * @param sparkSession
+ */
+case class HoodieSpark3ResolveReferences(sparkSession: SparkSession) extends Rule[LogicalPlan]
+ with SparkAdapterSupport with HoodieConfigHelper {
+
+ def apply(plan: LogicalPlan): LogicalPlan = plan resolveOperatorsUp {
+ // Fill schema for Create Table without specify schema info
+ case c @ CreateV2Table(tableCatalog, tableName, schema, _, properties, _)
+ if isHoodieTable(properties) =>
+
+ val hoodieCatalog = tableCatalog match {
+ case catalog: HoodieCatalog => catalog
+ case _ => tableCatalog.asInstanceOf[V2SessionCatalog]
+ }
+ val tablePath = getTableLocation(properties,
+ TableIdentifier(tableName.name(), tableName.namespace().lastOption), sparkSession)
+
+ val tableExistInCatalog = hoodieCatalog.tableExists(tableName)
+ // Only when the table has not exist in catalog, we need to fill the schema info for creating table.
+ if (!tableExistInCatalog && tableExistsInPath(tablePath, sparkSession.sessionState.newHadoopConf())) {
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(tablePath)
+ .setConf(sparkSession.sessionState.newHadoopConf())
+ .build()
+ val tableSchema = HoodieSqlCommonUtils.getTableSqlSchema(metaClient)
+ if (tableSchema.isDefined && schema.isEmpty) {
+ // Fill the schema with the schema from the table
+ c.copy(tableSchema = tableSchema.get)
+ } else if (tableSchema.isDefined && schema != tableSchema.get) {
+ throw new AnalysisException(s"Specified schema in create table statement is not equal to the table schema." +
+ s"You should not specify the schema for an exist table: $tableName ")
+ } else {
+ c
+ }
+ } else {
+ c
+ }
+ case DropPartitions(child, specs, ifExists, purge)
Review comment:
done
##########
File path: hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/hudi/SparkAdapter.scala
##########
@@ -92,4 +95,31 @@ trait SparkAdapter extends Serializable {
* ParserInterface#parseMultipartIdentifier is supported since spark3, for spark2 this should not be called.
*/
def parseMultipartIdentifier(parser: ParserInterface, sqlText: String): Seq[String]
+
+ def isHoodieTable(table: LogicalPlan, spark: SparkSession): Boolean = {
+ tripAlias(table) match {
+ case LogicalRelation(_, _, Some(tbl), _) => isHoodieTable(tbl)
+ case relation: UnresolvedRelation =>
+ isHoodieTable(toTableIdentifier(relation), spark)
+ case _=> false
+ }
+ }
+
+ def tripAlias(plan: LogicalPlan): LogicalPlan = {
+ plan match {
+ case SubqueryAlias(_, relation: LogicalPlan) =>
+ tripAlias(relation)
+ case other =>
+ other
+ }
+ }
+
+ def isHoodieTable(table: CatalogTable): Boolean = {
+ table.provider.map(_.toLowerCase(Locale.ROOT)).orNull == "hudi"
+ }
+
+ def isHoodieTable(tableId: TableIdentifier, spark: SparkSession): Boolean = {
+ val table = spark.sessionState.catalog.getTableMetadata(tableId)
+ isHoodieTable(table)
+ }
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/TestHoodieSparkSqlWriter.scala
##########
@@ -94,11 +94,17 @@ class TestHoodieSparkSqlWriter {
* Utility method for initializing the spark context.
*/
def initSparkContext(): Unit = {
+ val sparkConf = new SparkConf()
+ if (HoodieSparkUtils.isSpark3_2) {
+ sparkConf.set("spark.sql.catalog.spark_catalog",
+ "org.apache.spark.sql.hudi.catalog.HoodieCatalog")
Review comment:
here is because in the class we use Spark SQL to create hudi table, so we need to set HoodieCatalog
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala
##########
@@ -42,12 +43,39 @@ object HoodieAnalysis {
Seq(
session => HoodieResolveReferences(session),
session => HoodieAnalysis(session)
- )
+ ) ++ extraResolutionRules()
def customPostHocResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] =
Seq(
session => HoodiePostAnalysisRule(session)
- )
+ ) ++ extraPostHocResolutionRules()
+
+ def extraResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] = {
+ if (HoodieSparkUtils.isSpark3_2) {
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala
##########
@@ -42,12 +43,39 @@ object HoodieAnalysis {
Seq(
session => HoodieResolveReferences(session),
session => HoodieAnalysis(session)
- )
+ ) ++ extraResolutionRules()
def customPostHocResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] =
Seq(
session => HoodiePostAnalysisRule(session)
- )
+ ) ++ extraPostHocResolutionRules()
+
+ def extraResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] = {
+ if (HoodieSparkUtils.isSpark3_2) {
+ val spark3AnalysisClass = "org.apache.spark.sql.hudi.analysis.HoodieSpark3Analysis"
+ val spark3Analysis: SparkSession => Rule[LogicalPlan] =
+ session => ReflectionUtils.loadClass(spark3AnalysisClass, session).asInstanceOf[Rule[LogicalPlan]]
+
+ val spark3ResolveReferences = "org.apache.spark.sql.hudi.analysis.HoodieSpark3ResolveReferences"
+ val spark3References: SparkSession => Rule[LogicalPlan] =
+ session => ReflectionUtils.loadClass(spark3ResolveReferences, session).asInstanceOf[Rule[LogicalPlan]]
+
+ Seq(spark3Analysis, spark3References)
+ } else {
+ Seq.empty
+ }
+ }
+
+ def extraPostHocResolutionRules(): Seq[SparkSession => Rule[LogicalPlan]] =
+ if (HoodieSparkUtils.isSpark3_2) {
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HudiSpark3SqlUtils.scala
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
+import org.apache.spark.sql.connector.expressions.{BucketTransform, FieldReference, IdentityTransform, Transform}
+
+import scala.collection.mutable
+
+object HudiSpark3SqlUtils {
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
+ assert(hoodieCatalogTable.primaryKeys.nonEmpty,
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
+ assert(hoodieCatalogTable.primaryKeys.nonEmpty,
+ s"There are no primary key in table ${hoodieCatalogTable.table.identifier}, cannot execute update operator")
+ val enableHive = isEnableHive(sparkSession)
+
+ withSparkConf(sparkSession, catalogProperties) {
+ Map(
+ "path" -> hoodieCatalogTable.tableLocation,
+ RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
+ PRECOMBINE_FIELD.key -> preCombineColumn,
+ TBL_NAME.key -> hoodieCatalogTable.tableName,
+ HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
+ URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
+ KEYGENERATOR_CLASS_NAME.key -> classOf[SqlKeyGenerator].getCanonicalName,
+ SqlKeyGenerator.ORIGIN_KEYGEN_CLASS_NAME -> tableConfig.getKeyGeneratorClassName,
+ OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+ PARTITIONPATH_FIELD.key -> tableConfig.getPartitionFieldProp,
+ META_SYNC_ENABLED.key -> enableHive.toString,
+ HIVE_SYNC_MODE.key -> HiveSyncMode.HMS.name(),
+ HIVE_USE_JDBC.key -> "false",
+ HIVE_DATABASE.key -> tableId.database.getOrElse("default"),
+ HIVE_TABLE.key -> tableId.table,
+ HIVE_PARTITION_FIELDS.key -> tableConfig.getPartitionFieldProp,
+ HIVE_PARTITION_EXTRACTOR_CLASS.key -> classOf[MultiPartKeysValueExtractor].getCanonicalName,
+ HIVE_SUPPORT_TIMESTAMP_TYPE.key -> "true",
+ HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key -> "200",
+ SqlKeyGenerator.PARTITION_SCHEMA -> hoodieCatalogTable.partitionSchema.toDDL
+ )
+ }
+ }
+
+ def cast(exp: Expression, field: StructField, sqlConf: SQLConf): Expression = {
+ castIfNeeded(exp, field.dataType, sqlConf)
+ }
Review comment:
removed
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieConfigHelper.scala
##########
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.DataSourceWriteOptions
+import org.apache.hudi.DataSourceWriteOptions._
+import org.apache.hudi.common.model.OverwriteWithLatestAvroPayload
+import org.apache.hudi.config.HoodieWriteConfig
+import org.apache.hudi.config.HoodieWriteConfig.TBL_NAME
+import org.apache.hudi.hive.MultiPartKeysValueExtractor
+import org.apache.hudi.hive.ddl.HiveSyncMode
+import org.apache.hudi.keygen.ComplexKeyGenerator
+import org.apache.hudi.sql.InsertMode
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.SparkSession
+import org.apache.spark.sql.catalyst.catalog.HoodieCatalogTable
+import org.apache.spark.sql.catalyst.expressions.Expression
+import org.apache.spark.sql.hudi.HoodieSqlCommonUtils.{castIfNeeded, isEnableHive, withSparkConf}
+import org.apache.spark.sql.hudi.command.{SqlKeyGenerator, ValidateDuplicateKeyPayload}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.StructField
+
+import scala.collection.JavaConverters.propertiesAsScalaMapConverter
+
+trait HoodieConfigHelper extends Logging {
+
+ def buildHoodieConfig(hoodieCatalogTable: HoodieCatalogTable): Map[String, String] = {
+ val sparkSession: SparkSession = hoodieCatalogTable.spark
+ val catalogProperties = hoodieCatalogTable.catalogProperties
+ val tableConfig = hoodieCatalogTable.tableConfig
+ val tableId = hoodieCatalogTable.table.identifier
+
+ val preCombineColumn = Option(tableConfig.getPreCombineField).getOrElse("")
+ assert(hoodieCatalogTable.primaryKeys.nonEmpty,
+ s"There are no primary key in table ${hoodieCatalogTable.table.identifier}, cannot execute update operator")
+ val enableHive = isEnableHive(sparkSession)
+
+ withSparkConf(sparkSession, catalogProperties) {
+ Map(
+ "path" -> hoodieCatalogTable.tableLocation,
+ RECORDKEY_FIELD.key -> hoodieCatalogTable.primaryKeys.mkString(","),
+ PRECOMBINE_FIELD.key -> preCombineColumn,
+ TBL_NAME.key -> hoodieCatalogTable.tableName,
+ HIVE_STYLE_PARTITIONING.key -> tableConfig.getHiveStylePartitioningEnable,
+ URL_ENCODE_PARTITIONING.key -> tableConfig.getUrlEncodePartitioning,
+ KEYGENERATOR_CLASS_NAME.key -> classOf[SqlKeyGenerator].getCanonicalName,
+ SqlKeyGenerator.ORIGIN_KEYGEN_CLASS_NAME -> tableConfig.getKeyGeneratorClassName,
+ OPERATION.key -> UPSERT_OPERATION_OPT_VAL,
+ PARTITIONPATH_FIELD.key -> tableConfig.getPartitionFieldProp,
+ META_SYNC_ENABLED.key -> enableHive.toString,
+ HIVE_SYNC_MODE.key -> HiveSyncMode.HMS.name(),
+ HIVE_USE_JDBC.key -> "false",
+ HIVE_DATABASE.key -> tableId.database.getOrElse("default"),
+ HIVE_TABLE.key -> tableId.table,
+ HIVE_PARTITION_FIELDS.key -> tableConfig.getPartitionFieldProp,
+ HIVE_PARTITION_EXTRACTOR_CLASS.key -> classOf[MultiPartKeysValueExtractor].getCanonicalName,
+ HIVE_SUPPORT_TIMESTAMP_TYPE.key -> "true",
+ HoodieWriteConfig.UPSERT_PARALLELISM_VALUE.key -> "200",
+ SqlKeyGenerator.PARTITION_SCHEMA -> hoodieCatalogTable.partitionSchema.toDDL
+ )
+ }
+ }
+
+ def cast(exp: Expression, field: StructField, sqlConf: SQLConf): Expression = {
+ castIfNeeded(exp, field.dataType, sqlConf)
+ }
+
+ /**
+ * Build the default config for insert.
+ * @return
+ */
+ def buildHoodieInsertConfig(hoodieCatalogTable: HoodieCatalogTable,
+ sparkSession: SparkSession,
+ isOverwrite: Boolean,
+ insertPartitions: Map[String, Option[String]] = Map.empty,
+ extraOptions: Map[String, String]): Map[String, String] = {
+
+ if (insertPartitions.nonEmpty &&
+ (insertPartitions.keys.toSet != hoodieCatalogTable.partitionFields.toSet)) {
+ throw new IllegalArgumentException(s"Insert partition fields" +
+ s"[${insertPartitions.keys.mkString(" " )}]" +
+ s" not equal to the defined partition in table[${hoodieCatalogTable.partitionFields.mkString(",")}]")
+ }
Review comment:
here if replace it with require, the condition will be very complicated, so i just left it as it is.
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieInternalV2Table.scala
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.common.table.{HoodieTableConfig, HoodieTableMetaClient}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogTable, HoodieCatalogTable}
+import org.apache.spark.sql.connector.catalog.TableCapability._
+import org.apache.spark.sql.connector.catalog.{SupportsWrite, Table, TableCapability, V2TableWithV1Fallback}
+import org.apache.spark.sql.connector.expressions.{FieldReference, IdentityTransform, Transform}
+import org.apache.spark.sql.connector.write._
+import org.apache.spark.sql.sources.{Filter, InsertableRelation}
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
+
+import java.util
+import scala.collection.JavaConverters.{mapAsJavaMapConverter, setAsJavaSetConverter}
+
+case class HoodieInternalV2Table(spark: SparkSession,
+ path: String,
+ catalogTable: Option[CatalogTable] = None,
+ tableIdentifier: Option[String] = None,
+ options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty())
+ extends Table with SupportsWrite with V2TableWithV1Fallback {
+
+ lazy val hoodieCatalogTable: HoodieCatalogTable = if (catalogTable.isDefined) {
+ HoodieCatalogTable(spark, catalogTable.get)
+ } else {
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient.builder()
+ .setBasePath(path)
+ .setConf(SparkSession.active.sessionState.newHadoopConf)
+ .build()
+
+ val tableConfig: HoodieTableConfig = metaClient.getTableConfig
+ val tableName: String = tableConfig.getTableName
+
+ HoodieCatalogTable(spark, TableIdentifier(tableName))
+ }
+
+ private lazy val tableSchema: StructType = hoodieCatalogTable.tableSchema
+
+ override def name(): String = hoodieCatalogTable.table.identifier.unquotedString
+
+ override def schema(): StructType = tableSchema
+
+ override def capabilities(): util.Set[TableCapability] = Set(
+ BATCH_READ, V1_BATCH_WRITE, OVERWRITE_BY_FILTER, TRUNCATE, ACCEPT_ANY_SCHEMA
+ ).asJava
+
+ override def properties(): util.Map[String, String] = {
+ val map = new util.HashMap[String, String]()
+ map.put("provider", "hudi")
Review comment:
here because the catalog not only handle hudi tables, but would also handle other types of tables, such as parquet.
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/HoodieInternalV2Table.scala
##########
@@ -0,0 +1,129 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.common.table.{HoodieTableConfig, HoodieTableMetaClient}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogTable, HoodieCatalogTable}
+import org.apache.spark.sql.connector.catalog.TableCapability._
+import org.apache.spark.sql.connector.catalog.{SupportsWrite, Table, TableCapability, V2TableWithV1Fallback}
+import org.apache.spark.sql.connector.expressions.{FieldReference, IdentityTransform, Transform}
+import org.apache.spark.sql.connector.write._
+import org.apache.spark.sql.sources.{Filter, InsertableRelation}
+import org.apache.spark.sql.types.StructType
+import org.apache.spark.sql.util.CaseInsensitiveStringMap
+import org.apache.spark.sql.{DataFrame, SaveMode, SparkSession}
+
+import java.util
+import scala.collection.JavaConverters.{mapAsJavaMapConverter, setAsJavaSetConverter}
+
+case class HoodieInternalV2Table(spark: SparkSession,
+ path: String,
+ catalogTable: Option[CatalogTable] = None,
+ tableIdentifier: Option[String] = None,
+ options: CaseInsensitiveStringMap = CaseInsensitiveStringMap.empty())
+ extends Table with SupportsWrite with V2TableWithV1Fallback {
+
+ lazy val hoodieCatalogTable: HoodieCatalogTable = if (catalogTable.isDefined) {
+ HoodieCatalogTable(spark, catalogTable.get)
+ } else {
+ val metaClient: HoodieTableMetaClient = HoodieTableMetaClient.builder()
+ .setBasePath(path)
+ .setConf(SparkSession.active.sessionState.newHadoopConf)
+ .build()
+
+ val tableConfig: HoodieTableConfig = metaClient.getTableConfig
+ val tableName: String = tableConfig.getTableName
+
+ HoodieCatalogTable(spark, TableIdentifier(tableName))
+ }
+
+ private lazy val tableSchema: StructType = hoodieCatalogTable.tableSchema
+
+ override def name(): String = hoodieCatalogTable.table.identifier.unquotedString
+
+ override def schema(): StructType = tableSchema
+
+ override def capabilities(): util.Set[TableCapability] = Set(
+ BATCH_READ, V1_BATCH_WRITE, OVERWRITE_BY_FILTER, TRUNCATE, ACCEPT_ANY_SCHEMA
+ ).asJava
+
+ override def properties(): util.Map[String, String] = {
+ val map = new util.HashMap[String, String]()
+ map.put("provider", "hudi")
+ map.putAll(hoodieCatalogTable.catalogProperties.asJava)
+ map
+ }
+
+ override def newWriteBuilder(info: LogicalWriteInfo): WriteBuilder = {
+ new WriteIntoHoodieBuilder(info.options, hoodieCatalogTable, spark)
+ }
+
+ override def v1Table: CatalogTable = hoodieCatalogTable.table
+
+ override def partitioning(): Array[Transform] = {
+ hoodieCatalogTable.partitionFields.map { col =>
+ new IdentityTransform(new FieldReference(Seq(col)))
+ }.toArray
+ }
+
+}
+
+private class WriteIntoHoodieBuilder(writeOptions: CaseInsensitiveStringMap,
Review comment:
done
##########
File path: hudi-spark-datasource/hudi-spark3/src/main/scala/org/apache/spark/sql/hudi/catalog/BaseStagedTable.scala
##########
@@ -0,0 +1,51 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.catalog
+
+import org.apache.hudi.exception.HoodieException
+import org.apache.spark.sql.connector.catalog._
+import org.apache.spark.sql.connector.expressions.Transform
+import org.apache.spark.sql.connector.write.{LogicalWriteInfo, WriteBuilder}
+import org.apache.spark.sql.types.StructType
+
+import java.util
+
+case class BaseStagedTable(ident: Identifier,
Review comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038535476
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960)
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038535476
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
}, {
"hash" : "2d6d4b7549fb79965f923693245d160c976df865",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "2d6d4b7549fb79965f923693245d160c976df865",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5960)
* 2d6d4b7549fb79965f923693245d160c976df865 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1038141572
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
}, {
"hash" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865",
"triggerID" : "e93af4603cc620726fc34a7d4aa482ee0acb7c41",
"triggerType" : "PUSH"
}, {
"hash" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "70a0e2d285e938f03d9c322b0c2754b42605f6d1",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* e93af4603cc620726fc34a7d4aa482ee0acb7c41 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5865)
* 70a0e2d285e938f03d9c322b0c2754b42605f6d1 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1020744332
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439)
* 0abd457fb40ee1af67da5d283cd6c09e6e07dac9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1020724538
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439)
* 0abd457fb40ee1af67da5d283cd6c09e6e07dac9 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1020744332
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439",
"triggerID" : "1019433490",
"triggerType" : "MANUAL"
}, {
"hash" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490",
"triggerID" : "0abd457fb40ee1af67da5d283cd6c09e6e07dac9",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377) Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5439)
* 0abd457fb40ee1af67da5d283cd6c09e6e07dac9 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5490)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
leesf commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1019433490
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1020724538
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013838711
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5e6315181eb95abc55bc6b1ee4f3488a60d65d0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1013848003
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f5e6315181eb95abc55bc6b1ee4f3488a60d65d0 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280)
* 27cba00d5daf148f99be78e72dade08229491d18 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017239081
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365)
* b6b74ca2553aab849511a37f0efd411fef068285 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017405909
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
}, {
"hash" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377",
"triggerID" : "addb9f9925913e522852b2efcb0a80fccc4153f2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* b6b74ca2553aab849511a37f0efd411fef068285 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370)
* addb9f9925913e522852b2efcb0a80fccc4153f2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5377)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017229766
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283)
* 1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017227414
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 7df4b2cbf0f0e658c0508e7ca0d93725bdc48823 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283)
* 1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4611: [HUDI-3254] Introduce HoodieCatalog to manage tables for Spark Datasource V2
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4611:
URL: https://github.com/apache/hudi/pull/4611#issuecomment-1017280671
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5280",
"triggerID" : "f5e6315181eb95abc55bc6b1ee4f3488a60d65d0",
"triggerType" : "PUSH"
}, {
"hash" : "27cba00d5daf148f99be78e72dade08229491d18",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5281",
"triggerID" : "27cba00d5daf148f99be78e72dade08229491d18",
"triggerType" : "PUSH"
}, {
"hash" : "a5fecdc15707afd659887b025d1e1094543468e7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5282",
"triggerID" : "a5fecdc15707afd659887b025d1e1094543468e7",
"triggerType" : "PUSH"
}, {
"hash" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5feee50ea8accfc643e1d9fd607e9e605cd97a40",
"triggerType" : "PUSH"
}, {
"hash" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5283",
"triggerID" : "7df4b2cbf0f0e658c0508e7ca0d93725bdc48823",
"triggerType" : "PUSH"
}, {
"hash" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365",
"triggerID" : "1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6",
"triggerType" : "PUSH"
}, {
"hash" : "b6b74ca2553aab849511a37f0efd411fef068285",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370",
"triggerID" : "b6b74ca2553aab849511a37f0efd411fef068285",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5feee50ea8accfc643e1d9fd607e9e605cd97a40 UNKNOWN
* 1c72a58d6f34317ac4bf9ac89e0c4d3a8cb467c6 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5365)
* b6b74ca2553aab849511a37f0efd411fef068285 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5370)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org