You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "boneanxs (via GitHub)" <gi...@apache.org> on 2023/08/10 04:09:27 UTC
[GitHub] [hudi] boneanxs opened a new pull request, #9412: [HUDI-6676] Add command for CreateHoodieTableLike
boneanxs opened a new pull request, #9412:
URL: https://github.com/apache/hudi/pull/9412
### Change Logs
_Describe context and summary for this change. Highlight if any code was copied._
1. Create table from non-hudi table
2. Create table from hudi table(The properties related to Hudi in the source Hudi table will be carried over)
### Impact
_Describe any public API or user-facing feature change or any performance impact._
None
### Risk level (write none, low medium or high below)
_If medium or high, explain what verification was done to mitigate the risks._
None
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9412:
URL: https://github.com/apache/hudi/pull/9412#issuecomment-1673014744
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24c43e61e9a304224df2ca5e2001551974348671",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237",
"triggerID" : "24c43e61e9a304224df2ca5e2001551974348671",
"triggerType" : "PUSH"
}, {
"hash" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19239",
"triggerID" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"triggerType" : "PUSH"
}, {
"hash" : "d957e2a7226f490ca59443c8b7ca5522e7ad8754",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19240",
"triggerID" : "d957e2a7226f490ca59443c8b7ca5522e7ad8754",
"triggerType" : "PUSH"
}, {
"hash" : "d608b959809c32251e89a074a130b2a92b2883d2",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d608b959809c32251e89a074a130b2a92b2883d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 16121f00ed63bf4019bed5810e22f98c42b0dfe6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19239)
* d957e2a7226f490ca59443c8b7ca5522e7ad8754 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19240)
* d608b959809c32251e89a074a130b2a92b2883d2 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9412:
URL: https://github.com/apache/hudi/pull/9412#discussion_r1289540657
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableLikeCommand.scala:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.model.HoodieTableType
+import org.apache.hudi.common.util.ConfigUtils
+import org.apache.spark.sql.{AnalysisException, Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType, HoodieCatalogTable}
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.hudi.HoodieOptionConfig
+
+import scala.util.control.NonFatal
+
+case class CreateHoodieTableLikeCommand(targetTable: TableIdentifier,
+ sourceTable: TableIdentifier,
+ fileFormat: CatalogStorageFormat,
+ properties: Map[String, String] = Map.empty,
+ ignoreIfExists: Boolean)
+ extends HoodieLeafRunnableCommand with SparkAdapterSupport {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+
+ val tableIsExists = catalog.tableExists(targetTable)
+ if (tableIsExists) {
+ if (ignoreIfExists) {
+ // scalastyle:off
+ return Seq.empty[Row]
+ // scalastyle:on
+ } else {
+ throw new IllegalArgumentException(s"Table $targetTable already exists.")
+ }
+ }
+
+ val sourceTableDesc = catalog.getTempViewOrPermanentTableMetadata(sourceTable)
+
+ val newStorage = if (fileFormat.inputFormat.isDefined) {
+ fileFormat
+ } else {
+ sourceTableDesc.storage.copy(locationUri = fileFormat.locationUri)
+ }
+
+ // If the location is specified, we create an external table internally.
+ // Otherwise create a managed table.
+ val tblType = if (newStorage.locationUri.isEmpty) {
+ CatalogTableType.MANAGED
+ } else {
+ CatalogTableType.EXTERNAL
+ }
+
+ val targetTableProperties = if (sparkAdapter.isHoodieTable(sourceTableDesc)) {
+ HoodieOptionConfig.extractHoodieOptions(sourceTableDesc.properties) ++ properties
+ } else {
+ properties
+ }
+
+ val newTableSchema = CharVarcharUtils.getRawSchema(sourceTableDesc.schema)
+ val newTableDesc = CatalogTable(
+ identifier = targetTable,
+ tableType = tblType,
+ storage = newStorage,
+ schema = newTableSchema,
+ provider = Some("hudi"),
+ partitionColumnNames = sourceTableDesc.partitionColumnNames,
+ bucketSpec = sourceTableDesc.bucketSpec,
+ properties = targetTableProperties,
+ tracksPartitionsInCatalog = sourceTableDesc.tracksPartitionsInCatalog)
+
+ val hoodieCatalogTable = HoodieCatalogTable(sparkSession, newTableDesc)
+ // check if there are conflict between table configs defined in hoodie table and properties defined in catalog.
+ CreateHoodieTableCommand.validateTblProperties(hoodieCatalogTable)
+
+ val queryAsProp = hoodieCatalogTable.catalogProperties.get(ConfigUtils.IS_QUERY_AS_RO_TABLE)
+ if (queryAsProp.isEmpty) {
Review Comment:
Does use specify this option through sql options?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9412:
URL: https://github.com/apache/hudi/pull/9412#issuecomment-1673301617
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24c43e61e9a304224df2ca5e2001551974348671",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237",
"triggerID" : "24c43e61e9a304224df2ca5e2001551974348671",
"triggerType" : "PUSH"
}, {
"hash" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19239",
"triggerID" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"triggerType" : "PUSH"
}, {
"hash" : "d957e2a7226f490ca59443c8b7ca5522e7ad8754",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19240",
"triggerID" : "d957e2a7226f490ca59443c8b7ca5522e7ad8754",
"triggerType" : "PUSH"
}, {
"hash" : "d608b959809c32251e89a074a130b2a92b2883d2",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19241",
"triggerID" : "d608b959809c32251e89a074a130b2a92b2883d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d608b959809c32251e89a074a130b2a92b2883d2 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9412:
URL: https://github.com/apache/hudi/pull/9412#issuecomment-1672559947
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24c43e61e9a304224df2ca5e2001551974348671",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237",
"triggerID" : "24c43e61e9a304224df2ca5e2001551974348671",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24c43e61e9a304224df2ca5e2001551974348671 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on a diff in pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on code in PR #9412:
URL: https://github.com/apache/hudi/pull/9412#discussion_r1290833633
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCreateTable.scala:
##########
@@ -405,6 +405,145 @@ class TestCreateTable extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table like") {
+ if (HoodieSparkUtils.gteqSpark3_1) {
+ // 1. Test create table from an existing HUDI table
+ withTempDir { tmp =>
Review Comment:
Spark2 will use Spark own `CreateTableLikeCommand`, we can't throw error here since we can't distinguish whether the user want to create hudi table or not.
```scala
* The syntax of using this command in SQL is(it doesn't support pass targetTable's provider):
* {{{
* CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
* LIKE [other_db_name.]existing_table_name [locationSpec]
* }}}
```
Spark2 is becoming depreciated, maybe only supporting spark3+ is enough?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9412:
URL: https://github.com/apache/hudi/pull/9412#issuecomment-1672553728
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24c43e61e9a304224df2ca5e2001551974348671",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "24c43e61e9a304224df2ca5e2001551974348671",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24c43e61e9a304224df2ca5e2001551974348671 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9412:
URL: https://github.com/apache/hudi/pull/9412#issuecomment-1672854354
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24c43e61e9a304224df2ca5e2001551974348671",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237",
"triggerID" : "24c43e61e9a304224df2ca5e2001551974348671",
"triggerType" : "PUSH"
}, {
"hash" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19239",
"triggerID" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"triggerType" : "PUSH"
}, {
"hash" : "d957e2a7226f490ca59443c8b7ca5522e7ad8754",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19240",
"triggerID" : "d957e2a7226f490ca59443c8b7ca5522e7ad8754",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 16121f00ed63bf4019bed5810e22f98c42b0dfe6 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19239)
* d957e2a7226f490ca59443c8b7ca5522e7ad8754 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19240)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on a diff in pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on code in PR #9412:
URL: https://github.com/apache/hudi/pull/9412#discussion_r1289685220
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/spark/sql/hudi/command/CreateHoodieTableLikeCommand.scala:
##########
@@ -0,0 +1,112 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.hudi.command
+
+import org.apache.hudi.SparkAdapterSupport
+import org.apache.hudi.common.model.HoodieTableType
+import org.apache.hudi.common.util.ConfigUtils
+import org.apache.spark.sql.{AnalysisException, Row, SparkSession}
+import org.apache.spark.sql.catalyst.TableIdentifier
+import org.apache.spark.sql.catalyst.catalog.{CatalogStorageFormat, CatalogTable, CatalogTableType, HoodieCatalogTable}
+import org.apache.spark.sql.catalyst.util.CharVarcharUtils
+import org.apache.spark.sql.hudi.HoodieOptionConfig
+
+import scala.util.control.NonFatal
+
+case class CreateHoodieTableLikeCommand(targetTable: TableIdentifier,
+ sourceTable: TableIdentifier,
+ fileFormat: CatalogStorageFormat,
+ properties: Map[String, String] = Map.empty,
+ ignoreIfExists: Boolean)
+ extends HoodieLeafRunnableCommand with SparkAdapterSupport {
+
+ override def run(sparkSession: SparkSession): Seq[Row] = {
+ val catalog = sparkSession.sessionState.catalog
+
+ val tableIsExists = catalog.tableExists(targetTable)
+ if (tableIsExists) {
+ if (ignoreIfExists) {
+ // scalastyle:off
+ return Seq.empty[Row]
+ // scalastyle:on
+ } else {
+ throw new IllegalArgumentException(s"Table $targetTable already exists.")
+ }
+ }
+
+ val sourceTableDesc = catalog.getTempViewOrPermanentTableMetadata(sourceTable)
+
+ val newStorage = if (fileFormat.inputFormat.isDefined) {
+ fileFormat
+ } else {
+ sourceTableDesc.storage.copy(locationUri = fileFormat.locationUri)
+ }
+
+ // If the location is specified, we create an external table internally.
+ // Otherwise create a managed table.
+ val tblType = if (newStorage.locationUri.isEmpty) {
+ CatalogTableType.MANAGED
+ } else {
+ CatalogTableType.EXTERNAL
+ }
+
+ val targetTableProperties = if (sparkAdapter.isHoodieTable(sourceTableDesc)) {
+ HoodieOptionConfig.extractHoodieOptions(sourceTableDesc.properties) ++ properties
+ } else {
+ properties
+ }
+
+ val newTableSchema = CharVarcharUtils.getRawSchema(sourceTableDesc.schema)
+ val newTableDesc = CatalogTable(
+ identifier = targetTable,
+ tableType = tblType,
+ storage = newStorage,
+ schema = newTableSchema,
+ provider = Some("hudi"),
+ partitionColumnNames = sourceTableDesc.partitionColumnNames,
+ bucketSpec = sourceTableDesc.bucketSpec,
+ properties = targetTableProperties,
+ tracksPartitionsInCatalog = sourceTableDesc.tracksPartitionsInCatalog)
+
+ val hoodieCatalogTable = HoodieCatalogTable(sparkSession, newTableDesc)
+ // check if there are conflict between table configs defined in hoodie table and properties defined in catalog.
+ CreateHoodieTableCommand.validateTblProperties(hoodieCatalogTable)
+
+ val queryAsProp = hoodieCatalogTable.catalogProperties.get(ConfigUtils.IS_QUERY_AS_RO_TABLE)
+ if (queryAsProp.isEmpty) {
Review Comment:
From my internal usage, we don't see users manually specify it, here I just want to align with `CreateHoodieTableCommand`, which checks this property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 merged pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 merged PR #9412:
URL: https://github.com/apache/hudi/pull/9412
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9412:
URL: https://github.com/apache/hudi/pull/9412#discussion_r1290844141
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCreateTable.scala:
##########
@@ -405,6 +405,145 @@ class TestCreateTable extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table like") {
+ if (HoodieSparkUtils.gteqSpark3_1) {
+ // 1. Test create table from an existing HUDI table
+ withTempDir { tmp =>
Review Comment:
We should avoid misusage of possible, or make it clear on the document.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] boneanxs commented on a diff in pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "boneanxs (via GitHub)" <gi...@apache.org>.
boneanxs commented on code in PR #9412:
URL: https://github.com/apache/hudi/pull/9412#discussion_r1293042161
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCreateTable.scala:
##########
@@ -405,6 +405,145 @@ class TestCreateTable extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table like") {
+ if (HoodieSparkUtils.gteqSpark3_1) {
+ // 1. Test create table from an existing HUDI table
+ withTempDir { tmp =>
Review Comment:
Updated doc here: https://github.com/apache/hudi/pull/9441
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9412:
URL: https://github.com/apache/hudi/pull/9412#issuecomment-1672765206
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24c43e61e9a304224df2ca5e2001551974348671",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237",
"triggerID" : "24c43e61e9a304224df2ca5e2001551974348671",
"triggerType" : "PUSH"
}, {
"hash" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19239",
"triggerID" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24c43e61e9a304224df2ca5e2001551974348671 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237)
* 16121f00ed63bf4019bed5810e22f98c42b0dfe6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19239)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #9412:
URL: https://github.com/apache/hudi/pull/9412#discussion_r1290821705
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCreateTable.scala:
##########
@@ -405,6 +405,145 @@ class TestCreateTable extends HoodieSparkSqlTestBase {
}
}
+ test("Test create table like") {
+ if (HoodieSparkUtils.gteqSpark3_1) {
+ // 1. Test create table from an existing HUDI table
+ withTempDir { tmp =>
Review Comment:
So spark2 will throw exception ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9412:
URL: https://github.com/apache/hudi/pull/9412#issuecomment-1673026312
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24c43e61e9a304224df2ca5e2001551974348671",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237",
"triggerID" : "24c43e61e9a304224df2ca5e2001551974348671",
"triggerType" : "PUSH"
}, {
"hash" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19239",
"triggerID" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"triggerType" : "PUSH"
}, {
"hash" : "d957e2a7226f490ca59443c8b7ca5522e7ad8754",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19240",
"triggerID" : "d957e2a7226f490ca59443c8b7ca5522e7ad8754",
"triggerType" : "PUSH"
}, {
"hash" : "d608b959809c32251e89a074a130b2a92b2883d2",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19241",
"triggerID" : "d608b959809c32251e89a074a130b2a92b2883d2",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d957e2a7226f490ca59443c8b7ca5522e7ad8754 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19240)
* d608b959809c32251e89a074a130b2a92b2883d2 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19241)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9412:
URL: https://github.com/apache/hudi/pull/9412#issuecomment-1672751548
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24c43e61e9a304224df2ca5e2001551974348671",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237",
"triggerID" : "24c43e61e9a304224df2ca5e2001551974348671",
"triggerType" : "PUSH"
}, {
"hash" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24c43e61e9a304224df2ca5e2001551974348671 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237)
* 16121f00ed63bf4019bed5810e22f98c42b0dfe6 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #9412: [HUDI-6676] Add command for CreateHoodieTableLike
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #9412:
URL: https://github.com/apache/hudi/pull/9412#issuecomment-1672840140
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "24c43e61e9a304224df2ca5e2001551974348671",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237",
"triggerID" : "24c43e61e9a304224df2ca5e2001551974348671",
"triggerType" : "PUSH"
}, {
"hash" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19239",
"triggerID" : "16121f00ed63bf4019bed5810e22f98c42b0dfe6",
"triggerType" : "PUSH"
}, {
"hash" : "d957e2a7226f490ca59443c8b7ca5522e7ad8754",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d957e2a7226f490ca59443c8b7ca5522e7ad8754",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 24c43e61e9a304224df2ca5e2001551974348671 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19237)
* 16121f00ed63bf4019bed5810e22f98c42b0dfe6 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=19239)
* d957e2a7226f490ca59443c8b7ca5522e7ad8754 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org