You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/08/01 08:08:25 UTC
[GitHub] [hudi] YannByron opened a new pull request, #6264: [HUDI-4503] support for parsing identifier with catalog
YannByron opened a new pull request, #6264:
URL: https://github.com/apache/hudi/pull/6264
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contribute/how-to-contribute before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r940830255
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
+ val catalogManager = spark.sessionState.catalogManager
+ val nameParts = relation.multipartIdentifier
+ val expandedNameParts = expandIdentifier(spark, nameParts)
+ HoodieCatalogAndIdentifier.parse(catalogManager, expandedNameParts) match {
+ case Some((catalog, ident)) =>
+ CatalogV2Util.loadTable(catalog, ident) match {
+ case Some(table) =>
+ table match {
+ case v1Table: V1Table =>
+ Some(v1Table.v1Table)
+ case withFallback: V2TableWithV1Fallback =>
+ Some(withFallback.v1Table)
+ case _ =>
+ logWarning(s"It's not a hoodie table: $table")
Review Comment:
We want to get a CatalogTable object from `UnresolvedRelation` here.
For all the sub-classes of `Table` trait, only `V1Table` and `V2TableWithV1Fallback` can convert to a `CatalogTable` object.
And `HoodieInternalV2Table` self is a sub-class of `V2TableWithV1Fallback` and have a method `v1TableWrapper` to get a `V1Table` object. So here i catch the two pattern.
According to your comment, what if I change to "can not get a CatalogTable from $table" or just remove this warning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1200994421
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 09952a197b20d3c605500708621c2a843a7f0d0e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489)
* 444908b46455be40da0757bfcdcc41cb5e5b86b7 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1203547178
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 444908b46455be40da0757bfcdcc41cb5e5b86b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491)
* 59bc4a4301f4ee5daac0a40c6c4cc184708e8710 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r940847507
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
sure. see the user case: https://github.com/apache/hudi/issues/6223.
If we use the three part `catalog.database.table` in spark3.x like , we need the ability to parse these identifiers.
And, the reason that Spark's default `ResolveRelations` can't work is, the `UnresolvedRelation` hasn't resolved when apply the cases/rules in `HoodieAnalysis`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1210100371
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
}, {
"hash" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667",
"triggerID" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"triggerType" : "PUSH"
}, {
"hash" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669",
"triggerID" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"triggerType" : "PUSH"
}, {
"hash" : "066e20303c737b6c0b441c5a92cb406ca45386ba",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "066e20303c737b6c0b441c5a92cb406ca45386ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c378a8518c1f1538a4c3c44fd0d5541a7997c003 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669)
* 066e20303c737b6c0b441c5a92cb406ca45386ba UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r940735559
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/connector/catalog/HoodieCatalogAndIdentifier.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.catalog
+
+import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
+
+object HoodieCatalogAndIdentifier {
+
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.MultipartIdentifierHelper
+
+ private val globalTempDB = SQLConf.get.getConf(StaticSQLConf.GLOBAL_TEMP_DATABASE)
+
+ def parse(catalogManager: CatalogManager, nameParts: Seq[String])
+ : Option[(CatalogPlugin, Identifier)] = {
+ assert(nameParts.nonEmpty)
+ val currentCatalog = catalogManager.currentCatalog
+ if (nameParts.length == 1) {
+ Some((currentCatalog, Identifier.of(catalogManager.currentNamespace, nameParts.head)))
+ } else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
+ // Conceptually global temp views are in a special reserved catalog. However, the v2 catalog
Review Comment:
Was this borrowed from somewhere?
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
+ val catalogManager = spark.sessionState.catalogManager
+ val nameParts = relation.multipartIdentifier
+ val expandedNameParts = expandIdentifier(spark, nameParts)
+ HoodieCatalogAndIdentifier.parse(catalogManager, expandedNameParts) match {
+ case Some((catalog, ident)) =>
+ CatalogV2Util.loadTable(catalog, ident) match {
+ case Some(table) =>
+ table match {
+ case v1Table: V1Table =>
+ Some(v1Table.v1Table)
+ case withFallback: V2TableWithV1Fallback =>
+ Some(withFallback.v1Table)
+ case _ =>
+ logWarning(s"It's not a hoodie table: $table")
+ None
+ }
+ case _ =>
+ logWarning(s"Can not load this catalog and identifier: ${catalog.name()}, $ident")
+ None
+ }
+ case _ =>
+ logWarning(s"Can not parse this name parts: ${expandedNameParts.mkString(",")}")
Review Comment:
nit: `${expandedNameParts.mkString("[", ",", "]")}`
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
Can you please elaborate why we need to have this custom resolution rule even though there's Spark's default `ResolveRelations` one?
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
+ val catalogManager = spark.sessionState.catalogManager
+ val nameParts = relation.multipartIdentifier
+ val expandedNameParts = expandIdentifier(spark, nameParts)
+ HoodieCatalogAndIdentifier.parse(catalogManager, expandedNameParts) match {
+ case Some((catalog, ident)) =>
+ CatalogV2Util.loadTable(catalog, ident) match {
+ case Some(table) =>
+ table match {
+ case v1Table: V1Table =>
+ Some(v1Table.v1Table)
+ case withFallback: V2TableWithV1Fallback =>
+ Some(withFallback.v1Table)
+ case _ =>
+ logWarning(s"It's not a hoodie table: $table")
Review Comment:
Warning is misleading: we don't check anywhere whether this is a Hudi table or not, we will hit this if there's gonna be another type of table introduced
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestSpark3Catalog.scala:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.hudi.HoodieSparkUtils
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import java.io.File
+
+class TestSpark3Catalog extends HoodieSparkSqlTestBase {
+
+ val tempDir: File = Utils.createTempDir()
+ val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+ override def sparkConf(): SparkConf = {
+ val sparkConf = super.sparkConf()
+ if (HoodieSparkUtils.gteqSpark3_1) {
Review Comment:
But how would it mess it up? We're only running our test for Spark 3.x, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r940830378
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
+ val catalogManager = spark.sessionState.catalogManager
+ val nameParts = relation.multipartIdentifier
+ val expandedNameParts = expandIdentifier(spark, nameParts)
+ HoodieCatalogAndIdentifier.parse(catalogManager, expandedNameParts) match {
+ case Some((catalog, ident)) =>
+ CatalogV2Util.loadTable(catalog, ident) match {
+ case Some(table) =>
+ table match {
+ case v1Table: V1Table =>
+ Some(v1Table.v1Table)
+ case withFallback: V2TableWithV1Fallback =>
+ Some(withFallback.v1Table)
+ case _ =>
+ logWarning(s"It's not a hoodie table: $table")
+ None
+ }
+ case _ =>
+ logWarning(s"Can not load this catalog and identifier: ${catalog.name()}, $ident")
+ None
+ }
+ case _ =>
+ logWarning(s"Can not parse this name parts: ${expandedNameParts.mkString(",")}")
Review Comment:
ok.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron closed pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron closed pull request #6264: [HUDI-4503] support for parsing identifier with catalog
URL: https://github.com/apache/hudi/pull/6264
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1200923511
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 09952a197b20d3c605500708621c2a843a7f0d0e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r940832883
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/connector/catalog/HoodieCatalogAndIdentifier.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.catalog
+
+import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
+
+object HoodieCatalogAndIdentifier {
+
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.MultipartIdentifierHelper
+
+ private val globalTempDB = SQLConf.get.getConf(StaticSQLConf.GLOBAL_TEMP_DATABASE)
+
+ def parse(catalogManager: CatalogManager, nameParts: Seq[String])
+ : Option[(CatalogPlugin, Identifier)] = {
+ assert(nameParts.nonEmpty)
+ val currentCatalog = catalogManager.currentCatalog
+ if (nameParts.length == 1) {
+ Some((currentCatalog, Identifier.of(catalogManager.currentNamespace, nameParts.head)))
+ } else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
+ // Conceptually global temp views are in a special reserved catalog. However, the v2 catalog
Review Comment:
yep, from `org.apache.spark.sql.connector.catalog.LookupCatalog.CatalogAndIdentifier`.
But `LookupCatalog` is a trait, and `CatalogAndIdentifier` is its inner object. If i make `HoodieSpark3CatalystPlanUtils` extend to `LookupCatalog`, i have to override the `catalogManager` that is not initializeed in `LookupCatalog`, further i have to pass a `SparkSession` to here, further... It will affect the definition of `HoodieCatalystPlansUtils` and `SparkAdapter` and `SparkAdapterSupport` and other places where use them.
So i extract the logical here, use it as a method call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941538499
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
+ val catalogManager = spark.sessionState.catalogManager
+ val nameParts = relation.multipartIdentifier
+ val expandedNameParts = expandIdentifier(spark, nameParts)
+ HoodieCatalogAndIdentifier.parse(catalogManager, expandedNameParts) match {
+ case Some((catalog, ident)) =>
+ CatalogV2Util.loadTable(catalog, ident) match {
+ case Some(table) =>
+ table match {
+ case v1Table: V1Table =>
+ Some(v1Table.v1Table)
+ case withFallback: V2TableWithV1Fallback =>
+ Some(withFallback.v1Table)
+ case _ =>
+ logWarning(s"It's not a hoodie table: $table")
+ None
+ }
+ case _ =>
+ logWarning(s"Can not load this catalog and identifier: ${catalog.name()}, $ident")
+ None
+ }
+ case _ =>
+ logWarning(s"Can not parse this name parts: ${expandedNameParts.mkString(",")}")
Review Comment:
Why not fixing? The problem with existing code that you will get message like following which is very confusing:
```
Can not parse this name parts: a,b,c
```
It might momentarily seem like someone is using "," instead of "." in the name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r943050236
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
> Moreover this part is not even affected by your changes. Your changes only affect Time Travel queries, and not Insert Into statement.
not just insert statement. If you run `select * from spark_catalog.default.hudi_tbl timestamp as of 111111` in the master branch, it also throws the same exception. This pr can also solve this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1212348205
@YannByron @xushiyan I also started experimenting myself on this issue to see if my hunch is right that we can avoid pulling in more resolution logic from Spark into Hudi in: https://github.com/apache/hudi/pull/6361/files
Let's hold on merging this PR until we confirm whether we'd be able to avoid pulling in resolution logic and make things simpler in the end.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1208025400
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
}, {
"hash" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667",
"triggerID" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"triggerType" : "PUSH"
}, {
"hash" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669",
"triggerID" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* e85a0338be6f3e5f511baf56e6d188aa740bbf6c Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667)
* c378a8518c1f1538a4c3c44fd0d5541a7997c003 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r940311939
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +64,51 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(relation: UnresolvedRelation): Option[CatalogTable] = {
+ val nameParts = relation.multipartIdentifier
+ nameParts match {
+ case CatalogAndIdentifier(catalog, ident) =>
+ CatalogV2Util.loadTable(catalog, ident) match {
+ case Some(table) =>
+ table match {
+ case v1Table: V1Table =>
+ Some(v1Table.v1Table)
+ case withFallback: V2TableWithV1Fallback =>
+ Some(withFallback.v1Table)
+ case _ =>
+ logWarning("It's not a hoodie table: " + table.getClass.getName)
+ None
+ }
+ }
+ case _ =>
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+ Some(spark.sessionState.catalog.getTableMetadata(nameParts.asTableIdentifier))
+ }
+ }
+
+ protected def expandIdentifier(nameParts: Seq[String]): Seq[String] = {
Review Comment:
forget to call this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1201057634
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 444908b46455be40da0757bfcdcc41cb5e5b86b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r942011022
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
@YannByron i don't think this is the case simply b/c if that would entail that `InsertIntoStatement` wouldn't be resolved by `ResolveRelations` for Spark at large, which is not the case -- it's being resolved correctly (take a look at `FindDataSourceTable` in [here](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L282))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941989799
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
understand. It's really a question that deserve to be dived.
But a same question: why we also need to resolve `UnresolvedRelation` or `UnresolvedTable` in inside of hoodie even in spark or where no such problem need to be solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r939153771
##########
hudi-spark-datasource/hudi-spark/pom.xml:
##########
@@ -180,6 +180,14 @@
</build>
<dependencies>
+ <!-- H2 database for TestSpark3Catalog -->
Review Comment:
No need to refer to the test
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestSpark3Catalog.scala:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.hudi.HoodieSparkUtils
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import java.io.File
+
+class TestSpark3Catalog extends HoodieSparkSqlTestBase {
+
+ val tempDir: File = Utils.createTempDir()
+ val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+ override def sparkConf(): SparkConf = {
+ val sparkConf = super.sparkConf()
+ if (HoodieSparkUtils.gteqSpark3_1) {
+ sparkConf
+ .set("spark.sql.catalog.h2", "org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog")
+ .set("spark.sql.catalog.h2.url", url)
+ .set("spark.sql.catalog.h2.driver", "org.h2.Driver")
+ }
+ sparkConf
+ }
+
+ private def withConnection[T](f: Connection => T): T = {
+ val conn = DriverManager.getConnection(url, new Properties())
+ try {
+ f(conn)
+ } finally {
+ conn.close()
+ }
+ }
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ Utils.classForName("org.h2.Driver")
+ withConnection { conn =>
+ conn.prepareStatement("""CREATE SCHEMA "test"""").executeUpdate()
+ conn.prepareStatement(
+ """CREATE TABLE "test"."people" (id INTEGER NOT NULL, country TEXT(32) NOT NULL)""")
+ .executeUpdate()
+ conn.prepareStatement(
+ """INSERT INTO "test"."people" VALUES (1, 'US')""")
+ .executeUpdate()
+ }
+ }
+
+ override def afterAll(): Unit = {
+ Utils.deleteRecursively(tempDir)
+ super.afterAll()
+ }
+
+ test("Test Read And Write Cross Multi Spark Catalog") {
+ if (HoodieSparkUtils.gteqSpark3_1) {
+ checkAnswer("SHOW TABLES IN h2.test")(Seq("test", "people", false))
Review Comment:
Is the expected sequence correct?
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestSpark3Catalog.scala:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.hudi.HoodieSparkUtils
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import java.io.File
+
+class TestSpark3Catalog extends HoodieSparkSqlTestBase {
+
+ val tempDir: File = Utils.createTempDir()
+ val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+ override def sparkConf(): SparkConf = {
+ val sparkConf = super.sparkConf()
+ if (HoodieSparkUtils.gteqSpark3_1) {
Review Comment:
Do we really need this conditional here?
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -18,23 +18,35 @@
package org.apache.spark.sql
import org.apache.hudi.spark3.internal.ReflectUtil
-import org.apache.spark.sql.catalyst.analysis.{TableOutputResolver, UnresolvedRelation}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, TableOutputResolver, UnresolvedRelation}
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, Like}
import org.apache.spark.sql.catalyst.plans.JoinType
import org.apache.spark.sql.catalyst.plans.logical.{InsertIntoStatement, Join, JoinHint, LogicalPlan}
import org.apache.spark.sql.catalyst.{AliasIdentifier, TableIdentifier}
-import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.connector.catalog.{CatalogManager, CatalogV2Util, LookupCatalog, V1Table, V2TableWithV1Fallback}
import org.apache.spark.sql.execution.command.ExplainCommand
+import org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog
import org.apache.spark.sql.execution.{ExtendedMode, SimpleMode}
import org.apache.spark.sql.internal.SQLConf
-abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
+abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils
+ with LookupCatalog with Logging {
+
+ override protected val catalogManager: CatalogManager = {
Review Comment:
Why do we need this? We should not be duplicating components available w/in `Analyzer`
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -473,9 +473,9 @@ case class HoodieResolveReferences(sparkSession: SparkSession) extends Rule[Logi
"version expression is not supported for time travel")
}
- val tableIdentifier = sparkAdapter.getCatalystPlanUtils.toTableIdentifier(plan)
- if (sparkAdapter.isHoodieTable(tableIdentifier, sparkSession)) {
- val hoodieCatalogTable = HoodieCatalogTable(sparkSession, tableIdentifier)
+ val catalogTable = sparkAdapter.getCatalystPlanUtils.resolve(plan)
+ if (catalogTable.isDefined && sparkAdapter.isHoodieTable(catalogTable.get)) {
Review Comment:
Will be better with pattern-matching
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r943802235
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
> I suggest to create an other pr to trace. And this pr just makes the identifier with catalog can work in hudi.
I understand that it's not ideal to make you revisit the approach, but i don't think it makes sense to introduce something in this PR that we might need to be reverting soon after.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941962972
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -473,26 +473,25 @@ case class HoodieResolveReferences(sparkSession: SparkSession) extends Rule[Logi
"version expression is not supported for time travel")
}
- val tableIdentifier = sparkAdapter.getCatalystPlanUtils.toTableIdentifier(plan)
Review Comment:
Alright, so after taking a deeper look this seems to be a piece of code that is problematic -- `toTableIdentifier` assumes that the name is 2 name-parts, when it can be 3.
What i think we need to do is simply make sure we handle it properly here. I don't believe we need to change anything around how the catalogs/tables are resolved actually.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941989799
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
understand. It's really a question that deserve to be dived.
But a same question: why we also need to resolve `UnresolvedRelation` or `UnresolvedTable` in inside of hoodie even in spark2.x env or where no such problem need to be solved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r940825126
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestSpark3Catalog.scala:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.hudi.HoodieSparkUtils
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import java.io.File
+
+class TestSpark3Catalog extends HoodieSparkSqlTestBase {
+
+ val tempDir: File = Utils.createTempDir()
+ val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+ override def sparkConf(): SparkConf = {
+ val sparkConf = super.sparkConf()
+ if (HoodieSparkUtils.gteqSpark3_1) {
Review Comment:
yes for now. But if users run UTs in their local env and some UTs fail, maybe they will be confused.
Like `TestSpark3DDL` and `TestTimeTravelTable`, these also add this conditional.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1210176323
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
}, {
"hash" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667",
"triggerID" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"triggerType" : "PUSH"
}, {
"hash" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669",
"triggerID" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"triggerType" : "PUSH"
}, {
"hash" : "066e20303c737b6c0b441c5a92cb406ca45386ba",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10713",
"triggerID" : "066e20303c737b6c0b441c5a92cb406ca45386ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 066e20303c737b6c0b441c5a92cb406ca45386ba Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10713)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1208105804
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
}, {
"hash" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667",
"triggerID" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"triggerType" : "PUSH"
}, {
"hash" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669",
"triggerID" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c378a8518c1f1538a4c3c44fd0d5541a7997c003 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r940310284
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestSpark3Catalog.scala:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.hudi.HoodieSparkUtils
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import java.io.File
+
+class TestSpark3Catalog extends HoodieSparkSqlTestBase {
+
+ val tempDir: File = Utils.createTempDir()
+ val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+ override def sparkConf(): SparkConf = {
+ val sparkConf = super.sparkConf()
+ if (HoodieSparkUtils.gteqSpark3_1) {
Review Comment:
i think we'd better have this in case it mess up in spark 2.x env.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r943051969
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
> My suggestions are basically following: 1. 2.
your suggestions can also work. Still inject the rules to `Resolution` batch but force to ask all the relation is resolved.
but for the time travel case you also mention, we have to resolve the `UnresolvedRelation` inside of Hudi by this resolution logic in this pr.
Let hudi get rid of all the resolution logics, make all the `LogicalPlan` be resolved and optimized correctly. I suggest to create an other pr to trace. And this pr just makes the identifier with catalog can work in hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1200987767
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 09952a197b20d3c605500708621c2a843a7f0d0e Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489)
* 444908b46455be40da0757bfcdcc41cb5e5b86b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r939143695
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +64,51 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(relation: UnresolvedRelation): Option[CatalogTable] = {
+ val nameParts = relation.multipartIdentifier
+ nameParts match {
+ case CatalogAndIdentifier(catalog, ident) =>
+ CatalogV2Util.loadTable(catalog, ident) match {
+ case Some(table) =>
+ table match {
+ case v1Table: V1Table =>
+ Some(v1Table.v1Table)
+ case withFallback: V2TableWithV1Fallback =>
+ Some(withFallback.v1Table)
+ case _ =>
+ logWarning("It's not a hoodie table: " + table.getClass.getName)
Review Comment:
Why are we logging `getClass.getName`?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalystPlansUtils.scala:
##########
@@ -19,13 +19,16 @@ package org.apache.spark.sql
import org.apache.spark.sql.catalyst.{AliasIdentifier, TableIdentifier}
import org.apache.spark.sql.catalyst.analysis.UnresolvedRelation
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression}
import org.apache.spark.sql.catalyst.plans.JoinType
import org.apache.spark.sql.catalyst.plans.logical.{Join, LogicalPlan}
import org.apache.spark.sql.internal.SQLConf
trait HoodieCatalystPlansUtils {
+ protected val spark: SparkSession = SparkSession.active
Review Comment:
Let's avoid adding implicit dependencies, if you want this one to have SparkSession as a state let's provide it explicitly in ctor
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -74,12 +129,17 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def createInsertInto(table: LogicalPlan, partition: Map[String, Option[String]],
- query: LogicalPlan, overwrite: Boolean, ifPartitionNotExists: Boolean): LogicalPlan = {
+ override def createInsertInto(
Review Comment:
Please keep formatting consistent
##########
hudi-spark-datasource/hudi-spark/pom.xml:
##########
@@ -180,6 +180,14 @@
</build>
<dependencies>
+ <!-- H2 database for TestSpark3Catalog -->
Review Comment:
What do we need this for?
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/spark/sql/HoodieCatalystPlansUtils.scala:
##########
@@ -54,9 +57,9 @@ trait HoodieCatalystPlansUtils {
def toTableIdentifier(aliasId: AliasIdentifier): TableIdentifier
/**
- * Convert a UnresolvedRelation to TableIdentifier.
+ * resolve UnresolvedRelation to CatalogTable.
Review Comment:
Please it keep the doc starting w/ a cap
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -18,23 +18,35 @@
package org.apache.spark.sql
import org.apache.hudi.spark3.internal.ReflectUtil
-import org.apache.spark.sql.catalyst.analysis.{TableOutputResolver, UnresolvedRelation}
+
+import org.apache.spark.internal.Logging
+import org.apache.spark.sql.catalyst.analysis.{AnalysisContext, TableOutputResolver, UnresolvedRelation}
+import org.apache.spark.sql.catalyst.catalog.CatalogTable
import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression, Like}
import org.apache.spark.sql.catalyst.plans.JoinType
import org.apache.spark.sql.catalyst.plans.logical.{InsertIntoStatement, Join, JoinHint, LogicalPlan}
import org.apache.spark.sql.catalyst.{AliasIdentifier, TableIdentifier}
-import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+import org.apache.spark.sql.connector.catalog.{CatalogManager, CatalogV2Util, LookupCatalog, V1Table, V2TableWithV1Fallback}
import org.apache.spark.sql.execution.command.ExplainCommand
+import org.apache.spark.sql.execution.datasources.v2.V2SessionCatalog
import org.apache.spark.sql.execution.{ExtendedMode, SimpleMode}
import org.apache.spark.sql.internal.SQLConf
-abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
+abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils
+ with LookupCatalog with Logging {
+
+ override protected val catalogManager: CatalogManager = {
+ val catalog = spark.sessionState.catalog
+ val v2SessionCatalog = new V2SessionCatalog(catalog)
+ new CatalogManager(v2SessionCatalog, catalog)
+ }
- def resolveOutputColumns(tableName: String,
- expected: Seq[Attribute],
- query: LogicalPlan,
- byName: Boolean,
- conf: SQLConf): LogicalPlan =
+ def resolveOutputColumns(
Review Comment:
Let's keep formatting consistent
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +64,51 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(relation: UnresolvedRelation): Option[CatalogTable] = {
+ val nameParts = relation.multipartIdentifier
+ nameParts match {
+ case CatalogAndIdentifier(catalog, ident) =>
+ CatalogV2Util.loadTable(catalog, ident) match {
+ case Some(table) =>
+ table match {
+ case v1Table: V1Table =>
+ Some(v1Table.v1Table)
+ case withFallback: V2TableWithV1Fallback =>
+ Some(withFallback.v1Table)
+ case _ =>
+ logWarning("It's not a hoodie table: " + table.getClass.getName)
+ None
+ }
+ }
+ case _ =>
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
+ Some(spark.sessionState.catalog.getTableMetadata(nameParts.asTableIdentifier))
+ }
+ }
+
+ protected def expandIdentifier(nameParts: Seq[String]): Seq[String] = {
Review Comment:
This doesn't seem to be used
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941963153
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
Please check my comment here:
https://github.com/apache/hudi/pull/6264#discussion_r941962972
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
Please check my comment here: https://github.com/apache/hudi/pull/6264#discussion_r941962972
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941959306
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
Understood this part, my question rather was: do we understand why is this not being handled properly by Spark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941536989
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
+ val catalogManager = spark.sessionState.catalogManager
+ val nameParts = relation.multipartIdentifier
+ val expandedNameParts = expandIdentifier(spark, nameParts)
+ HoodieCatalogAndIdentifier.parse(catalogManager, expandedNameParts) match {
+ case Some((catalog, ident)) =>
+ CatalogV2Util.loadTable(catalog, ident) match {
+ case Some(table) =>
+ table match {
+ case v1Table: V1Table =>
+ Some(v1Table.v1Table)
+ case withFallback: V2TableWithV1Fallback =>
+ Some(withFallback.v1Table)
+ case _ =>
+ logWarning(s"It's not a hoodie table: $table")
Review Comment:
I'd suggest "Can not find CatalogTable for $table"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r942197333
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
Let me explain this in detail.
1) Hudi injects `HoodieResolveReferences` into the `Resolution` batch in Analyzer.
2) The `Resolution` batch will have these Rule in order: `ResolveInsertInto`, `ResolveRelations`, `ResolveTables`, ... , `FindDataSourceTable`, ..., `HoodieResolveReferences`.
3) the sql `insert into target_table select * from source_table` will be analyzed by the rules above in order.
**In the first iterator** over Analyzer's `Resolution` batch:
- `ResolveInsertInto` isn't used.
- `ResolveRelations` is used to resolve the child of `InsertIntoStatement` (that is query).Here the target_table doesn't be resolved.
- `FindDataSourceTable` is not used, b/c the first argument (`table`) of `InsertIntoStatement` is still an `UnresolvedRelation`, not an `UnresolvedCatalogRelation`.
- then `HoodieResolveReferences` will be applied. In it, hudi gets the `table` object (an `UnresolvedRelation` for now) from `InsertIntoStatement`, and want to determine it is a hudi table or not according to this. It will call the origin logical that try to transform `UnresolvedRelation` to `TableIdentifier` directly. But if it's a three-part format, an exception will be thrown like this case.
But, for normal spark which do not have this `HoodieResolveReferences` Rule, the `table` object of `InsertIntoStatement` will be resolved by `ResolveTables` **in the second iterator** over Analyzer's `Resolution` batch. That's the reason why spark can resolve this correctly.
I think the root cause is that hudi injects `HoodieResolveReferences` into `Resolution`, not into `Post-Hoc Resolution`. If so, maybe hudi never need to resolve `UnresolvedXXXX` by itself. But it's a big changes, need an other issue to trace.
So this pr is the controllable solution to solve the user case that only do a few changes in current hudi architecture.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r942012496
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
Moreover this part is not even affected by your changes. Your changes only affect Time Travel queries, and not Insert Into statement.
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
@YannByron thanks for detailed elaboration! Appreciate that!
The point i’m trying to make is that Spark _should be_ resolving the relations, NOT Hudi — we should not take away functionality from Spark that we don’t customize, extend or fix. In this case we are not doing either of these things so we should leave resolving to Spark as it is.
My suggestions are basically following:
1. Avoid copying any resolution logic in your PR, and instead
2. Fix things that fail individually:
- `isHoodieTable` should NOT be invoked on `UnresolvedRelation`. Instead we should add conditional checking whether target relation in `InsertIntoStatement` has been fully resolved by Spark.
- Same goes for `TimeTravelRelation` — it’s currently actually implemented incorrectly. `TimeTravelRelation` is implemented as `Command` , while in reality it should just be either `LeafNode` or `UnaryNode` (in which case it would be resolved by Spark automatically)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1200917628
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 09952a197b20d3c605500708621c2a843a7f0d0e UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r943792259
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
You're right `TimeTravelRelation` part will actually have to stay since it's an ephemeral construct we ultimately replace w/ `LogicalRelation`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1207759424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
}, {
"hash" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59bc4a4301f4ee5daac0a40c6c4cc184708e8710 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542)
* e85a0338be6f3e5f511baf56e6d188aa740bbf6c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941958365
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/connector/catalog/HoodieCatalogAndIdentifier.scala:
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.connector.catalog
+
+import org.apache.spark.sql.internal.{SQLConf, StaticSQLConf}
+
+object HoodieCatalogAndIdentifier {
+
+ import org.apache.spark.sql.connector.catalog.CatalogV2Implicits.MultipartIdentifierHelper
+
+ private val globalTempDB = SQLConf.get.getConf(StaticSQLConf.GLOBAL_TEMP_DATABASE)
+
+ def parse(catalogManager: CatalogManager, nameParts: Seq[String])
+ : Option[(CatalogPlugin, Identifier)] = {
+ assert(nameParts.nonEmpty)
+ val currentCatalog = catalogManager.currentCatalog
+ if (nameParts.length == 1) {
+ Some((currentCatalog, Identifier.of(catalogManager.currentNamespace, nameParts.head)))
+ } else if (nameParts.head.equalsIgnoreCase(globalTempDB)) {
+ // Conceptually global temp views are in a special reserved catalog. However, the v2 catalog
Review Comment:
yes, you're right. I don't know `Analyzer` has the `CatalogAndIdentifier` object before.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941968046
##########
hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala:
##########
@@ -473,26 +473,25 @@ case class HoodieResolveReferences(sparkSession: SparkSession) extends Rule[Logi
"version expression is not supported for time travel")
}
- val tableIdentifier = sparkAdapter.getCatalystPlanUtils.toTableIdentifier(plan)
Review Comment:
This changes can't change how the catalogs/tables are resolved, the `LogicalPlan` is not changed due to this pr. Just let HUDI internal can correctly determine whether an `UnresolvedRelation` is hudi table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r940308259
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestSpark3Catalog.scala:
##########
@@ -0,0 +1,114 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi
+
+import java.sql.{Connection, DriverManager}
+import java.util.Properties
+
+import org.apache.hudi.HoodieSparkUtils
+
+import org.apache.spark.SparkConf
+import org.apache.spark.util.Utils
+
+import java.io.File
+
+class TestSpark3Catalog extends HoodieSparkSqlTestBase {
+
+ val tempDir: File = Utils.createTempDir()
+ val url = s"jdbc:h2:${tempDir.getCanonicalPath};user=testUser;password=testPass"
+
+ override def sparkConf(): SparkConf = {
+ val sparkConf = super.sparkConf()
+ if (HoodieSparkUtils.gteqSpark3_1) {
+ sparkConf
+ .set("spark.sql.catalog.h2", "org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog")
+ .set("spark.sql.catalog.h2.url", url)
+ .set("spark.sql.catalog.h2.driver", "org.h2.Driver")
+ }
+ sparkConf
+ }
+
+ private def withConnection[T](f: Connection => T): T = {
+ val conn = DriverManager.getConnection(url, new Properties())
+ try {
+ f(conn)
+ } finally {
+ conn.close()
+ }
+ }
+
+ override def beforeAll(): Unit = {
+ super.beforeAll()
+ Utils.classForName("org.h2.Driver")
+ withConnection { conn =>
+ conn.prepareStatement("""CREATE SCHEMA "test"""").executeUpdate()
+ conn.prepareStatement(
+ """CREATE TABLE "test"."people" (id INTEGER NOT NULL, country TEXT(32) NOT NULL)""")
+ .executeUpdate()
+ conn.prepareStatement(
+ """INSERT INTO "test"."people" VALUES (1, 'US')""")
+ .executeUpdate()
+ }
+ }
+
+ override def afterAll(): Unit = {
+ Utils.deleteRecursively(tempDir)
+ super.afterAll()
+ }
+
+ test("Test Read And Write Cross Multi Spark Catalog") {
+ if (HoodieSparkUtils.gteqSpark3_1) {
+ checkAnswer("SHOW TABLES IN h2.test")(Seq("test", "people", false))
Review Comment:
yes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1209837018
@YannByron can you please also update the description to make sure it has relevant info?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941959306
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
But my question is why is this not being handled properly by Spark?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r942000073
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
@alexeykudinkin both `UnresolvedRelation` and `UnresolvedTable` call `resolveOperatorsUpWithPruning` to make itself and it's children to apply the Rule.
But for the sql `insert into target_table select col1, col2 from source_table`, only `select col1, col2 from source_table` as the query of `InsertIntoStatement` is it's child, `target_table` as the table of `InsertIntoStatement` is not. So the table LogicalPlan can't be applied by `ResolvedRelation` or `ResolvedTable`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1207898843
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
}, {
"hash" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667",
"triggerID" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"triggerType" : "PUSH"
}, {
"hash" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669",
"triggerID" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59bc4a4301f4ee5daac0a40c6c4cc184708e8710 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542)
* e85a0338be6f3e5f511baf56e6d188aa740bbf6c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667)
* c378a8518c1f1538a4c3c44fd0d5541a7997c003 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1207894083
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
}, {
"hash" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667",
"triggerID" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"triggerType" : "PUSH"
}, {
"hash" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59bc4a4301f4ee5daac0a40c6c4cc184708e8710 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542)
* e85a0338be6f3e5f511baf56e6d188aa740bbf6c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667)
* c378a8518c1f1538a4c3c44fd0d5541a7997c003 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1210102971
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
}, {
"hash" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667",
"triggerID" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"triggerType" : "PUSH"
}, {
"hash" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669",
"triggerID" : "c378a8518c1f1538a4c3c44fd0d5541a7997c003",
"triggerType" : "PUSH"
}, {
"hash" : "066e20303c737b6c0b441c5a92cb406ca45386ba",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10713",
"triggerID" : "066e20303c737b6c0b441c5a92cb406ca45386ba",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c378a8518c1f1538a4c3c44fd0d5541a7997c003 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10669)
* 066e20303c737b6c0b441c5a92cb406ca45386ba Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10713)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r942011022
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
@YannByron i don't think it is the case simply b/c if that would be the case, then `InsertIntoStatement` wouldn't be resolved by `ResolveRelations` rule which it most certainly does (take a look at `FindDataSourceTable` in [here](https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceStrategy.scala#L282))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1207764684
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
}, {
"hash" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667",
"triggerID" : "e85a0338be6f3e5f511baf56e6d188aa740bbf6c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59bc4a4301f4ee5daac0a40c6c4cc184708e8710 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542)
* e85a0338be6f3e5f511baf56e6d188aa740bbf6c Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10667)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r940847876
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
+ val catalogManager = spark.sessionState.catalogManager
+ val nameParts = relation.multipartIdentifier
+ val expandedNameParts = expandIdentifier(spark, nameParts)
+ HoodieCatalogAndIdentifier.parse(catalogManager, expandedNameParts) match {
+ case Some((catalog, ident)) =>
+ CatalogV2Util.loadTable(catalog, ident) match {
+ case Some(table) =>
+ table match {
+ case v1Table: V1Table =>
+ Some(v1Table.v1Table)
+ case withFallback: V2TableWithV1Fallback =>
+ Some(withFallback.v1Table)
+ case _ =>
+ logWarning(s"It's not a hoodie table: $table")
+ None
+ }
+ case _ =>
+ logWarning(s"Can not load this catalog and identifier: ${catalog.name()}, $ident")
+ None
+ }
+ case _ =>
+ logWarning(s"Can not parse this name parts: ${expandedNameParts.mkString(",")}")
Review Comment:
@alexeykudinkin if no other places need to be changed, can we just leave it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r941956169
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
the test I have added without these changes is same with this user's case: https://github.com/apache/hudi/issues/6223#issuecomment-1198952333.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6264:
URL: https://github.com/apache/hudi/pull/6264#discussion_r943051969
##########
hudi-spark-datasource/hudi-spark3-common/src/main/scala/org/apache/spark/sql/HoodieSpark3CatalystPlanUtils.scala:
##########
@@ -52,8 +57,56 @@ abstract class HoodieSpark3CatalystPlanUtils extends HoodieCatalystPlansUtils {
}
}
- override def toTableIdentifier(relation: UnresolvedRelation): TableIdentifier = {
- relation.multipartIdentifier.asTableIdentifier
+ override def resolve(spark: SparkSession, relation: UnresolvedRelation): Option[CatalogTable] = {
Review Comment:
> My suggestions are basically following: 1. 2.
your suggestions can also work. Still inject the rules to `Resolution` batch but force to ask all the relation is resolved.
but for the time travel case you also mention, we have to resolve the `UnresolvedRelation` inside of Hudi by this resolution logic in this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1200929823
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 09952a197b20d3c605500708621c2a843a7f0d0e Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489)
* 444908b46455be40da0757bfcdcc41cb5e5b86b7 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1203550678
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 444908b46455be40da0757bfcdcc41cb5e5b86b7 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491)
* 59bc4a4301f4ee5daac0a40c6c4cc184708e8710 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6264: [HUDI-4503] support for parsing identifier with catalog
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6264:
URL: https://github.com/apache/hudi/pull/6264#issuecomment-1203601654
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10489",
"triggerID" : "09952a197b20d3c605500708621c2a843a7f0d0e",
"triggerType" : "PUSH"
}, {
"hash" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10491",
"triggerID" : "444908b46455be40da0757bfcdcc41cb5e5b86b7",
"triggerType" : "PUSH"
}, {
"hash" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542",
"triggerID" : "59bc4a4301f4ee5daac0a40c6c4cc184708e8710",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 59bc4a4301f4ee5daac0a40c6c4cc184708e8710 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=10542)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org