You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Zouxxyy (via GitHub)" <gi...@apache.org> on 2023/01/30 10:13:50 UTC
[GitHub] [hudi] Zouxxyy opened a new pull request, #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Zouxxyy opened a new pull request, #7793:
URL: https://github.com/apache/hudi/pull/7793
### Change Logs
fix https://github.com/apache/hudi/pull/7365, spark2 and spark3's behavior needs to be consistent
### Impact
low
### Risk level (write none, low medium or high below)
low
### Documentation Update
None
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7793:
URL: https://github.com/apache/hudi/pull/7793#discussion_r1092719567
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -433,122 +433,22 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
""".stripMargin)
checkAnswer(s"select id, name, price, ts, dt from $tableName " +
s"where dt >='2021-01-04' and dt <= '2021-01-06' order by id,dt")(
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06"),
Seq(3, "a1", 10.0, 1000, "2021-01-04")
)
- // test insert overwrite non-partitioned table
+ // Test insert overwrite non-partitioned table
spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10, 1000")
checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
Seq(2, "a2", 10.0, 1000)
)
- })
- }
- test("Test Insert Overwrite Table for V2 Table") {
- withSQLConf("hoodie.schema.on.read.enable" -> "true") {
Review Comment:
V2 Table is only enabled when hoodie.schema.on.read.enable is true, otherwise v2Table.v1TableWrapper will be used(see org.apache.spark.sql.hudi.catalog.HoodieCatalog#loadTable). In V2 table, we can distinguish between insert overwrite partition and insert overwrite table while we can't do this in v1 table, so I add a v2 table test here and by design the behavior is expected to be different from v1 table
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1408578042
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
}, {
"hash" : "47d371397f8380443d326237090321074994858b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773",
"triggerID" : "47d371397f8380443d326237090321074994858b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63939a3eb6fb0f4aacc340513001b34e6d12e694 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771)
* 47d371397f8380443d326237090321074994858b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1408434781
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63939a3eb6fb0f4aacc340513001b34e6d12e694 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1409837284
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
}, {
"hash" : "47d371397f8380443d326237090321074994858b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773",
"triggerID" : "47d371397f8380443d326237090321074994858b",
"triggerType" : "PUSH"
}, {
"hash" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14779",
"triggerID" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"triggerType" : "PUSH"
}, {
"hash" : "6f3efd8db2ef71ad0861f468a491b6b22e032037",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14791",
"triggerID" : "6f3efd8db2ef71ad0861f468a491b6b22e032037",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6f3efd8db2ef71ad0861f468a491b6b22e032037 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14791)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1411425231
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1408338878
@stream2000 @leesf Could you please help to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #7793:
URL: https://github.com/apache/hudi/pull/7793#discussion_r1091333514
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -443,112 +440,14 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
Seq(2, "a2", 10.0, 1000)
)
- })
- }
-
- test("Test Insert Overwrite Table for V2 Table") {
- withSQLConf("hoodie.schema.on.read.enable" -> "true") {
- withRecordType()(withTempDir { tmp =>
- if (HoodieSparkUtils.gteqSpark3_2) {
- val tableName = generateTableName
- // Create a partitioned table
- spark.sql(
- s"""
- |create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts long,
- | dt string
- |) using hudi
- | tblproperties (primaryKey = 'id', preCombineField='dt')
- | partitioned by (dt)
- | location '${tmp.getCanonicalPath}/$tableName'
- """.stripMargin)
-
- // Test insert overwrite table
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | values(1, 'a1', 10.0, 1000, '2021-01-05')
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.0, 1000, "2021-01-05")
- )
-
- // Insert overwrite table
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | values (2, 'a2', 10.0, 1000, '2021-01-06')
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by id")(
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert overwrite static partition
- spark.sql(
- s"""
- | insert overwrite table $tableName partition(dt = '2021-01-05')
- | select * from (select 2 , 'a2', 12.0, 1000) limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by dt")(
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert data from another table
- val tblNonPartition = generateTableName
- spark.sql(
- s"""
- | create table $tblNonPartition (
- | id int,
- | name string,
- | price double,
- | ts long
- | ) using hudi
- | tblproperties (primaryKey = 'id')
- | location '${tmp.getCanonicalPath}/$tblNonPartition'
- """.stripMargin)
- spark.sql(s"insert into $tblNonPartition select 1, 'a1', 10.0, 1000")
- spark.sql(
- s"""
- | insert overwrite table $tableName partition(dt ='2021-01-04')
- | select * from $tblNonPartition limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by id,dt")(
- Seq(1, "a1", 10.0, 1000, "2021-01-04"),
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert overwrite partitioned table, all partitions will be truncated
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | select id + 2, name, price, ts , '2021-01-04' from $tblNonPartition limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName " +
- s"where dt >='2021-01-04' and dt <= '2021-01-06' order by id,dt")(
- Seq(3, "a1", 10.0, 1000, "2021-01-04")
- )
-
- // Test insert overwrite non-partitioned table
- spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10.0, 1000")
- checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
- Seq(2, "a2", 10.0, 1000)
- )
- spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10.0, 2000")
- checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
- Seq(2, "a2", 10.0, 2000)
- )
- }
- })
- }
+ spark.sql(s"insert overwrite table $tblNonPartition select 3, 'a3', 10, 1000")
Review Comment:
exist, such as lines 383 and 393
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -443,112 +440,14 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
Seq(2, "a2", 10.0, 1000)
)
- })
- }
-
- test("Test Insert Overwrite Table for V2 Table") {
- withSQLConf("hoodie.schema.on.read.enable" -> "true") {
- withRecordType()(withTempDir { tmp =>
- if (HoodieSparkUtils.gteqSpark3_2) {
- val tableName = generateTableName
- // Create a partitioned table
- spark.sql(
- s"""
- |create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts long,
- | dt string
- |) using hudi
- | tblproperties (primaryKey = 'id', preCombineField='dt')
- | partitioned by (dt)
- | location '${tmp.getCanonicalPath}/$tableName'
- """.stripMargin)
-
- // Test insert overwrite table
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | values(1, 'a1', 10.0, 1000, '2021-01-05')
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.0, 1000, "2021-01-05")
- )
-
- // Insert overwrite table
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | values (2, 'a2', 10.0, 1000, '2021-01-06')
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by id")(
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert overwrite static partition
- spark.sql(
- s"""
- | insert overwrite table $tableName partition(dt = '2021-01-05')
- | select * from (select 2 , 'a2', 12.0, 1000) limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by dt")(
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert data from another table
- val tblNonPartition = generateTableName
- spark.sql(
- s"""
- | create table $tblNonPartition (
- | id int,
- | name string,
- | price double,
- | ts long
- | ) using hudi
- | tblproperties (primaryKey = 'id')
- | location '${tmp.getCanonicalPath}/$tblNonPartition'
- """.stripMargin)
- spark.sql(s"insert into $tblNonPartition select 1, 'a1', 10.0, 1000")
- spark.sql(
- s"""
- | insert overwrite table $tableName partition(dt ='2021-01-04')
- | select * from $tblNonPartition limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by id,dt")(
- Seq(1, "a1", 10.0, 1000, "2021-01-04"),
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert overwrite partitioned table, all partitions will be truncated
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | select id + 2, name, price, ts , '2021-01-04' from $tblNonPartition limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName " +
- s"where dt >='2021-01-04' and dt <= '2021-01-06' order by id,dt")(
- Seq(3, "a1", 10.0, 1000, "2021-01-04")
- )
-
- // Test insert overwrite non-partitioned table
- spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10.0, 1000")
- checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
- Seq(2, "a2", 10.0, 1000)
- )
- spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10.0, 2000")
- checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
- Seq(2, "a2", 10.0, 2000)
- )
- }
- })
- }
+ spark.sql(s"insert overwrite table $tblNonPartition select 3, 'a3', 10, 1000")
Review Comment:
@leesf exist, such as lines 383 and 393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1408841697
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
}, {
"hash" : "47d371397f8380443d326237090321074994858b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773",
"triggerID" : "47d371397f8380443d326237090321074994858b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47d371397f8380443d326237090321074994858b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf commented on a diff in pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf commented on code in PR #7793:
URL: https://github.com/apache/hudi/pull/7793#discussion_r1091326307
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -443,112 +440,14 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
Seq(2, "a2", 10.0, 1000)
)
- })
- }
-
- test("Test Insert Overwrite Table for V2 Table") {
- withSQLConf("hoodie.schema.on.read.enable" -> "true") {
- withRecordType()(withTempDir { tmp =>
- if (HoodieSparkUtils.gteqSpark3_2) {
- val tableName = generateTableName
- // Create a partitioned table
- spark.sql(
- s"""
- |create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts long,
- | dt string
- |) using hudi
- | tblproperties (primaryKey = 'id', preCombineField='dt')
- | partitioned by (dt)
- | location '${tmp.getCanonicalPath}/$tableName'
- """.stripMargin)
-
- // Test insert overwrite table
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | values(1, 'a1', 10.0, 1000, '2021-01-05')
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.0, 1000, "2021-01-05")
- )
-
- // Insert overwrite table
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | values (2, 'a2', 10.0, 1000, '2021-01-06')
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by id")(
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert overwrite static partition
- spark.sql(
- s"""
- | insert overwrite table $tableName partition(dt = '2021-01-05')
- | select * from (select 2 , 'a2', 12.0, 1000) limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by dt")(
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert data from another table
- val tblNonPartition = generateTableName
- spark.sql(
- s"""
- | create table $tblNonPartition (
- | id int,
- | name string,
- | price double,
- | ts long
- | ) using hudi
- | tblproperties (primaryKey = 'id')
- | location '${tmp.getCanonicalPath}/$tblNonPartition'
- """.stripMargin)
- spark.sql(s"insert into $tblNonPartition select 1, 'a1', 10.0, 1000")
- spark.sql(
- s"""
- | insert overwrite table $tableName partition(dt ='2021-01-04')
- | select * from $tblNonPartition limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by id,dt")(
- Seq(1, "a1", 10.0, 1000, "2021-01-04"),
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert overwrite partitioned table, all partitions will be truncated
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | select id + 2, name, price, ts , '2021-01-04' from $tblNonPartition limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName " +
- s"where dt >='2021-01-04' and dt <= '2021-01-06' order by id,dt")(
- Seq(3, "a1", 10.0, 1000, "2021-01-04")
- )
-
- // Test insert overwrite non-partitioned table
- spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10.0, 1000")
- checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
- Seq(2, "a2", 10.0, 1000)
- )
- spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10.0, 2000")
- checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
- Seq(2, "a2", 10.0, 2000)
- )
- }
- })
- }
+ spark.sql(s"insert overwrite table $tblNonPartition select 3, 'a3', 10, 1000")
Review Comment:
is there test case for insert overwrite partitioned table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] leesf merged pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "leesf (via GitHub)" <gi...@apache.org>.
leesf merged PR #7793:
URL: https://github.com/apache/hudi/pull/7793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #7793:
URL: https://github.com/apache/hudi/pull/7793#discussion_r1091338157
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -443,112 +440,14 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
Seq(2, "a2", 10.0, 1000)
)
- })
- }
-
- test("Test Insert Overwrite Table for V2 Table") {
- withSQLConf("hoodie.schema.on.read.enable" -> "true") {
- withRecordType()(withTempDir { tmp =>
- if (HoodieSparkUtils.gteqSpark3_2) {
- val tableName = generateTableName
- // Create a partitioned table
- spark.sql(
- s"""
- |create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts long,
- | dt string
- |) using hudi
- | tblproperties (primaryKey = 'id', preCombineField='dt')
- | partitioned by (dt)
- | location '${tmp.getCanonicalPath}/$tableName'
- """.stripMargin)
-
- // Test insert overwrite table
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | values(1, 'a1', 10.0, 1000, '2021-01-05')
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.0, 1000, "2021-01-05")
- )
-
- // Insert overwrite table
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | values (2, 'a2', 10.0, 1000, '2021-01-06')
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by id")(
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert overwrite static partition
- spark.sql(
- s"""
- | insert overwrite table $tableName partition(dt = '2021-01-05')
- | select * from (select 2 , 'a2', 12.0, 1000) limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by dt")(
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert data from another table
- val tblNonPartition = generateTableName
- spark.sql(
- s"""
- | create table $tblNonPartition (
- | id int,
- | name string,
- | price double,
- | ts long
- | ) using hudi
- | tblproperties (primaryKey = 'id')
- | location '${tmp.getCanonicalPath}/$tblNonPartition'
- """.stripMargin)
- spark.sql(s"insert into $tblNonPartition select 1, 'a1', 10.0, 1000")
- spark.sql(
- s"""
- | insert overwrite table $tableName partition(dt ='2021-01-04')
- | select * from $tblNonPartition limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by id,dt")(
- Seq(1, "a1", 10.0, 1000, "2021-01-04"),
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert overwrite partitioned table, all partitions will be truncated
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | select id + 2, name, price, ts , '2021-01-04' from $tblNonPartition limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName " +
- s"where dt >='2021-01-04' and dt <= '2021-01-06' order by id,dt")(
- Seq(3, "a1", 10.0, 1000, "2021-01-04")
- )
-
- // Test insert overwrite non-partitioned table
- spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10.0, 1000")
- checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
- Seq(2, "a2", 10.0, 1000)
- )
- spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10.0, 2000")
- checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
- Seq(2, "a2", 10.0, 2000)
- )
- }
- })
- }
+ spark.sql(s"insert overwrite table $tblNonPartition select 3, 'a3', 10, 1000")
Review Comment:
`Test Insert Overwrite Table for V2 Table` is unnecessary, it is completely consistent with `Test Insert Overwrite`, and their results must be the same in spark2 and spark3.
So I just removed it, and fix spark2 by repalce `if (overwrite && catalogTable.partitionFields.isEmpty)` with `if (overwrite && partitionSpec.isEmpty)`
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -443,112 +440,14 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
Seq(2, "a2", 10.0, 1000)
)
- })
- }
-
- test("Test Insert Overwrite Table for V2 Table") {
- withSQLConf("hoodie.schema.on.read.enable" -> "true") {
- withRecordType()(withTempDir { tmp =>
- if (HoodieSparkUtils.gteqSpark3_2) {
- val tableName = generateTableName
- // Create a partitioned table
- spark.sql(
- s"""
- |create table $tableName (
- | id int,
- | name string,
- | price double,
- | ts long,
- | dt string
- |) using hudi
- | tblproperties (primaryKey = 'id', preCombineField='dt')
- | partitioned by (dt)
- | location '${tmp.getCanonicalPath}/$tableName'
- """.stripMargin)
-
- // Test insert overwrite table
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | values(1, 'a1', 10.0, 1000, '2021-01-05')
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName")(
- Seq(1, "a1", 10.0, 1000, "2021-01-05")
- )
-
- // Insert overwrite table
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | values (2, 'a2', 10.0, 1000, '2021-01-06')
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by id")(
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert overwrite static partition
- spark.sql(
- s"""
- | insert overwrite table $tableName partition(dt = '2021-01-05')
- | select * from (select 2 , 'a2', 12.0, 1000) limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by dt")(
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert data from another table
- val tblNonPartition = generateTableName
- spark.sql(
- s"""
- | create table $tblNonPartition (
- | id int,
- | name string,
- | price double,
- | ts long
- | ) using hudi
- | tblproperties (primaryKey = 'id')
- | location '${tmp.getCanonicalPath}/$tblNonPartition'
- """.stripMargin)
- spark.sql(s"insert into $tblNonPartition select 1, 'a1', 10.0, 1000")
- spark.sql(
- s"""
- | insert overwrite table $tableName partition(dt ='2021-01-04')
- | select * from $tblNonPartition limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName order by id,dt")(
- Seq(1, "a1", 10.0, 1000, "2021-01-04"),
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06")
- )
-
- // Insert overwrite partitioned table, all partitions will be truncated
- spark.sql(
- s"""
- | insert overwrite table $tableName
- | select id + 2, name, price, ts , '2021-01-04' from $tblNonPartition limit 10
- """.stripMargin)
- checkAnswer(s"select id, name, price, ts, dt from $tableName " +
- s"where dt >='2021-01-04' and dt <= '2021-01-06' order by id,dt")(
- Seq(3, "a1", 10.0, 1000, "2021-01-04")
- )
-
- // Test insert overwrite non-partitioned table
- spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10.0, 1000")
- checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
- Seq(2, "a2", 10.0, 1000)
- )
- spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10.0, 2000")
- checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
- Seq(2, "a2", 10.0, 2000)
- )
- }
- })
- }
+ spark.sql(s"insert overwrite table $tblNonPartition select 3, 'a3', 10, 1000")
Review Comment:
@leesf `Test Insert Overwrite Table for V2 Table` is unnecessary, it is completely consistent with `Test Insert Overwrite`, and their results must be the same in spark2 and spark3.
So I just removed it, and fix spark2 by repalce `if (overwrite && catalogTable.partitionFields.isEmpty)` with `if (overwrite && partitionSpec.isEmpty)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] stream2000 commented on a diff in pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "stream2000 (via GitHub)" <gi...@apache.org>.
stream2000 commented on code in PR #7793:
URL: https://github.com/apache/hudi/pull/7793#discussion_r1092719567
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -433,122 +433,22 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
""".stripMargin)
checkAnswer(s"select id, name, price, ts, dt from $tableName " +
s"where dt >='2021-01-04' and dt <= '2021-01-06' order by id,dt")(
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06"),
Seq(3, "a1", 10.0, 1000, "2021-01-04")
)
- // test insert overwrite non-partitioned table
+ // Test insert overwrite non-partitioned table
spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10, 1000")
checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
Seq(2, "a2", 10.0, 1000)
)
- })
- }
- test("Test Insert Overwrite Table for V2 Table") {
- withSQLConf("hoodie.schema.on.read.enable" -> "true") {
Review Comment:
V2 Table is only enabled when hoodie.schema.on.read.enable is true, otherwise v2Table.v1TableWrapper will be used(see org.apache.spark.sql.hudi.catalog.HoodieCatalog#loadTable). In V2 table, we can distinguish between insert overwrite partition and insert overwrite table while we can't do this in v1 table, so I add a v2 table test here to test the different behaviors between v1 and v2 table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1408446424
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63939a3eb6fb0f4aacc340513001b34e6d12e694 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1408988366
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
}, {
"hash" : "47d371397f8380443d326237090321074994858b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773",
"triggerID" : "47d371397f8380443d326237090321074994858b",
"triggerType" : "PUSH"
}, {
"hash" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47d371397f8380443d326237090321074994858b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773)
* c83b9c33ffb64efe2d986462c3ffe8461580f435 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1409001842
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
}, {
"hash" : "47d371397f8380443d326237090321074994858b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773",
"triggerID" : "47d371397f8380443d326237090321074994858b",
"triggerType" : "PUSH"
}, {
"hash" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14779",
"triggerID" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 47d371397f8380443d326237090321074994858b Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773)
* c83b9c33ffb64efe2d986462c3ffe8461580f435 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14779)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1409645483
cc @jonvex and @lokeshj1703 to take a look :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1409251694
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
}, {
"hash" : "47d371397f8380443d326237090321074994858b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773",
"triggerID" : "47d371397f8380443d326237090321074994858b",
"triggerType" : "PUSH"
}, {
"hash" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14779",
"triggerID" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c83b9c33ffb64efe2d986462c3ffe8461580f435 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14779)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1408558708
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
}, {
"hash" : "47d371397f8380443d326237090321074994858b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "47d371397f8380443d326237090321074994858b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 63939a3eb6fb0f4aacc340513001b34e6d12e694 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771)
* 47d371397f8380443d326237090321074994858b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1409646637
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
}, {
"hash" : "47d371397f8380443d326237090321074994858b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773",
"triggerID" : "47d371397f8380443d326237090321074994858b",
"triggerType" : "PUSH"
}, {
"hash" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14779",
"triggerID" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"triggerType" : "PUSH"
}, {
"hash" : "6f3efd8db2ef71ad0861f468a491b6b22e032037",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14791",
"triggerID" : "6f3efd8db2ef71ad0861f468a491b6b22e032037",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c83b9c33ffb64efe2d986462c3ffe8461580f435 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14779)
* 6f3efd8db2ef71ad0861f468a491b6b22e032037 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14791)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #7793:
URL: https://github.com/apache/hudi/pull/7793#issuecomment-1409642020
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14771",
"triggerID" : "63939a3eb6fb0f4aacc340513001b34e6d12e694",
"triggerType" : "PUSH"
}, {
"hash" : "47d371397f8380443d326237090321074994858b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14773",
"triggerID" : "47d371397f8380443d326237090321074994858b",
"triggerType" : "PUSH"
}, {
"hash" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14779",
"triggerID" : "c83b9c33ffb64efe2d986462c3ffe8461580f435",
"triggerType" : "PUSH"
}, {
"hash" : "6f3efd8db2ef71ad0861f468a491b6b22e032037",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6f3efd8db2ef71ad0861f468a491b6b22e032037",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c83b9c33ffb64efe2d986462c3ffe8461580f435 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=14779)
* 6f3efd8db2ef71ad0861f468a491b6b22e032037 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #7793:
URL: https://github.com/apache/hudi/pull/7793#discussion_r1092733332
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -433,122 +433,22 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
""".stripMargin)
checkAnswer(s"select id, name, price, ts, dt from $tableName " +
s"where dt >='2021-01-04' and dt <= '2021-01-06' order by id,dt")(
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06"),
Seq(3, "a1", 10.0, 1000, "2021-01-04")
)
- // test insert overwrite non-partitioned table
+ // Test insert overwrite non-partitioned table
spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10, 1000")
checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
Seq(2, "a2", 10.0, 1000)
)
- })
- }
- test("Test Insert Overwrite Table for V2 Table") {
- withSQLConf("hoodie.schema.on.read.enable" -> "true") {
Review Comment:
Year, I noticed that you added this config to force the use of the V2 table, but I think in the future, hudi spark3 may use v2 by default instead of being controlled by this config.
Beside, v1 table can also distinguish insert overwrite partition and insert overwrite table by checking partitionSpec is empty or not, so I think the test should be uniform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #7793: [HUDI-5317] Fix insert overwrite table for partitioned table
Posted by "Zouxxyy (via GitHub)" <gi...@apache.org>.
Zouxxyy commented on code in PR #7793:
URL: https://github.com/apache/hudi/pull/7793#discussion_r1092733332
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -433,122 +433,22 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
""".stripMargin)
checkAnswer(s"select id, name, price, ts, dt from $tableName " +
s"where dt >='2021-01-04' and dt <= '2021-01-06' order by id,dt")(
- Seq(2, "a2", 12.0, 1000, "2021-01-05"),
- Seq(2, "a2", 10.0, 1000, "2021-01-06"),
Seq(3, "a1", 10.0, 1000, "2021-01-04")
)
- // test insert overwrite non-partitioned table
+ // Test insert overwrite non-partitioned table
spark.sql(s"insert overwrite table $tblNonPartition select 2, 'a2', 10, 1000")
checkAnswer(s"select id, name, price, ts from $tblNonPartition")(
Seq(2, "a2", 10.0, 1000)
)
- })
- }
- test("Test Insert Overwrite Table for V2 Table") {
- withSQLConf("hoodie.schema.on.read.enable" -> "true") {
Review Comment:
Year, I noticed that you added this config to force the use of the V2 table, but I think in the future, hudi spark3 may use v2 by default instead of being controlled by this config.
Beside, v1 table can also distinguish insert overwrite partition and insert overwrite table by checking partitionSpec is empty not, so I think the test should be uniform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org