You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/10/14 06:53:38 UTC
[GitHub] [hudi] Zouxxyy opened a new pull request, #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Zouxxyy opened a new pull request, #6949:
URL: https://github.com/apache/hudi/pull/6949
### Change Logs
Add test case for conf hoodie.merge.allow.duplicate.on.inserts
### Impact
minor
**Risk level: none | low | medium | high**
none
### Documentation Update
_Describe any necessary documentation update if there is any new feature, config, or user-facing change_
- _The config description must be updated if new configs are added or the default value of the configs are changed_
- _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the
ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make
changes to the website._
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1282290662
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "d5493649d73c87c6565044050e3c4b3781f954da",
"triggerType" : "PUSH"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12291",
"triggerID" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"triggerType" : "PUSH"
}, {
"hash" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12300",
"triggerID" : "1282290211",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* d5493649d73c87c6565044050e3c4b3781f954da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272)
* 562b126b7a14034ad5d4ad8382428653a9fab66a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12291) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12300)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1278651362
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7ab63333a74e0fe073dda02276b3812c1cb1503 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203)
* c0614b3a245b7317d186583654ebcb61e4de7bfc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1278920137
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6b927a79d55a51282915beefe9c91b7042c81a3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1282106797
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "d5493649d73c87c6565044050e3c4b3781f954da",
"triggerType" : "PUSH"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12291",
"triggerID" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5493649d73c87c6565044050e3c4b3781f954da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272)
* 562b126b7a14034ad5d4ad8382428653a9fab66a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12291)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1278908048
> @Zouxxyy please enrich this UT so that we can know the behavior of different DML operations when enable `hoodie.merge.allow.duplicate.on.inserts`.
Since we have chosen non-strict mode (insert), we may have duplicate primary key records, which can cause a lot of uncertainty when updating records, such as only updating records in one file, so I think we should avoid using the upsert operation under non-strict mode.
Therefore, I only tested the insert and select operations for correctness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on code in PR #6949:
URL: https://github.com/apache/hudi/pull/6949#discussion_r997640885
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -901,4 +901,47 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
}
}
}
+
+ test("Test enable hoodie.merge.allow.duplicate.on.inserts when write") {
+ spark.sql("set hoodie.datasource.write.operation = insert")
+ Seq("mor", "cow").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long,
+ | dt string
+ | ) using hudi
+ | partitioned by (dt)
+ | location '${tmp.getCanonicalPath}/$tableName'
+ | tblproperties (
+ | primaryKey = 'id',
+ | preCombineField = 'ts',
+ | type = '$tableType'
+ | )
+ """.stripMargin)
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a1', 10, 1000)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a1", 10, 1000, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a2', 20, 1001)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a3', 30, 1002)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25"),
+ Seq(1, "a3", 30, 1002, "2021-12-25")
+ )
+ }
+ }
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
Review Comment:
Since it defaults to false, we need to restore it after the test is complete
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1282367690
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "d5493649d73c87c6565044050e3c4b3781f954da",
"triggerType" : "PUSH"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12291",
"triggerID" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"triggerType" : "PUSH"
}, {
"hash" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12300",
"triggerID" : "1282290211",
"triggerType" : "MANUAL"
}, {
"hash" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12291",
"triggerID" : "1282290211",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 562b126b7a14034ad5d4ad8382428653a9fab66a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12300) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12291)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1278575412
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7ab63333a74e0fe073dda02276b3812c1cb1503 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1281794367
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "d5493649d73c87c6565044050e3c4b3781f954da",
"triggerType" : "PUSH"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* d5493649d73c87c6565044050e3c4b3781f954da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1280731088
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bb6868a8d75b5d4af654ff00af9bf4ede5796878 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242)
* a7b34a73658bc9912bf8c3090d93d4482e96dc6b Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
danny0405 commented on code in PR #6949:
URL: https://github.com/apache/hudi/pull/6949#discussion_r997657575
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable2.scala:
##########
@@ -609,7 +609,6 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
| preCombineField = 'ts'
| )
""".stripMargin)
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
// Insert data
Review Comment:
Why drop this line ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
YannByron commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1281729791
@Zouxxyy please rebase master branch, not merge, then will land this when CI succeeds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1281751512
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "d5493649d73c87c6565044050e3c4b3781f954da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a7b34a73658bc9912bf8c3090d93d4482e96dc6b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255)
* d5493649d73c87c6565044050e3c4b3781f954da UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1282290211
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1278914277
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7ab63333a74e0fe073dda02276b3812c1cb1503 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203)
* c0614b3a245b7317d186583654ebcb61e4de7bfc Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206)
* c6b927a79d55a51282915beefe9c91b7042c81a3 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1282095297
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "d5493649d73c87c6565044050e3c4b3781f954da",
"triggerType" : "PUSH"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* d5493649d73c87c6565044050e3c4b3781f954da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272)
* 562b126b7a14034ad5d4ad8382428653a9fab66a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1280245690
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6b927a79d55a51282915beefe9c91b7042c81a3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212)
* bb6868a8d75b5d4af654ff00af9bf4ede5796878 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1280350116
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bb6868a8d75b5d4af654ff00af9bf4ede5796878 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1281180259
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a7b34a73658bc9912bf8c3090d93d4482e96dc6b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1280725434
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* bb6868a8d75b5d4af654ff00af9bf4ede5796878 Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242)
* a7b34a73658bc9912bf8c3090d93d4482e96dc6b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1282843319
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "d5493649d73c87c6565044050e3c4b3781f954da",
"triggerType" : "PUSH"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12291",
"triggerID" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"triggerType" : "PUSH"
}, {
"hash" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12300",
"triggerID" : "1282290211",
"triggerType" : "MANUAL"
}, {
"hash" : "562b126b7a14034ad5d4ad8382428653a9fab66a",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12291",
"triggerID" : "1282290211",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* 562b126b7a14034ad5d4ad8382428653a9fab66a Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12300) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12291)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
YannByron commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1278582251
@Zouxxyy please enrich this UT so that we can know the behavior of different DML operations when enable `hoodie.merge.allow.duplicate.on.inserts`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1278569727
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7ab63333a74e0fe073dda02276b3812c1cb1503 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on a diff in pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
YannByron commented on code in PR #6949:
URL: https://github.com/apache/hudi/pull/6949#discussion_r998868474
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -901,4 +901,47 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
}
}
}
+
+ test("Test enable hoodie.merge.allow.duplicate.on.inserts when write") {
+ spark.sql("set hoodie.datasource.write.operation = insert")
+ Seq("mor", "cow").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long,
+ | dt string
+ | ) using hudi
+ | partitioned by (dt)
+ | location '${tmp.getCanonicalPath}/$tableName'
+ | tblproperties (
+ | primaryKey = 'id',
+ | preCombineField = 'ts',
+ | type = '$tableType'
+ | )
+ """.stripMargin)
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a1', 10, 1000)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a1", 10, 1000, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a2', 20, 1001)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a3', 30, 1002)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25"),
+ Seq(1, "a3", 30, 1002, "2021-12-25")
+ )
+ }
+ }
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
Review Comment:
I saw many cases use this way to set a config at the beginning and restore a config at the ending.
IMO, better to use the method like `org.apache.spark.sql.test.SQLTestUtilsBase.withSQLConf`.
@Zouxxyy are you interesting to create a ticket to follow up this optimization of spark sql UTs?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
YannByron commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1280932026
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1281754648
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "d5493649d73c87c6565044050e3c4b3781f954da",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a7b34a73658bc9912bf8c3090d93d4482e96dc6b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255)
* d5493649d73c87c6565044050e3c4b3781f954da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1281790752
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "d5493649d73c87c6565044050e3c4b3781f954da",
"triggerType" : "PUSH"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a7b34a73658bc9912bf8c3090d93d4482e96dc6b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252) Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255)
* d5493649d73c87c6565044050e3c4b3781f954da Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] YannByron merged pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
YannByron merged PR #6949:
URL: https://github.com/apache/hudi/pull/6949
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1278580986
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7ab63333a74e0fe073dda02276b3812c1cb1503 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203)
* c0614b3a245b7317d186583654ebcb61e4de7bfc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1280249213
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* c6b927a79d55a51282915beefe9c91b7042c81a3 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212)
* bb6868a8d75b5d4af654ff00af9bf4ede5796878 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1280917902
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* a7b34a73658bc9912bf8c3090d93d4482e96dc6b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on code in PR #6949:
URL: https://github.com/apache/hudi/pull/6949#discussion_r998857447
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -901,4 +901,47 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
}
}
}
+
+ test("Test enable hoodie.merge.allow.duplicate.on.inserts when write") {
+ spark.sql("set hoodie.datasource.write.operation = insert")
+ Seq("mor", "cow").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long,
+ | dt string
+ | ) using hudi
+ | partitioned by (dt)
+ | location '${tmp.getCanonicalPath}/$tableName'
+ | tblproperties (
+ | primaryKey = 'id',
+ | preCombineField = 'ts',
+ | type = '$tableType'
+ | )
+ """.stripMargin)
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a1', 10, 1000)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a1", 10, 1000, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a2', 20, 1001)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a3', 30, 1002)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25"),
+ Seq(1, "a3", 30, 1002, "2021-12-25")
+ )
+ }
+ }
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
Review Comment:
> We should reset it at the beginning of the test
This is done to ensure that other test cases are not affected by this case, just like line 945, 535 and so on
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestMergeIntoTable2.scala:
##########
@@ -609,7 +609,6 @@ class TestMergeIntoTable2 extends HoodieSparkSqlTestBase {
| preCombineField = 'ts'
| )
""".stripMargin)
- spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
// Insert data
Review Comment:
> Why drop this line ?
Because this configuration does not work here and can be misleading
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1281777512
@hudi-bot run azure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6949:
URL: https://github.com/apache/hudi/pull/6949#discussion_r997376458
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -901,4 +901,47 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
}
}
}
+
+ test("Test enable hoodie.merge.allow.duplicate.on.inserts when write") {
+ spark.sql("set hoodie.datasource.write.operation = insert")
+ Seq("mor", "cow").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long,
+ | dt string
+ | ) using hudi
+ | partitioned by (dt)
+ | location '${tmp.getCanonicalPath}/$tableName'
+ | tblproperties (
+ | primaryKey = 'id',
+ | preCombineField = 'ts',
+ | type = '$tableType'
+ | )
+ """.stripMargin)
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a1', 10, 1000)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a1", 10, 1000, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a2', 20, 1001)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a3', 30, 1002)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25"),
+ Seq(1, "a3", 30, 1002, "2021-12-25")
+ )
+ }
+ }
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
Review Comment:
I don't think it makes sense to set these after the test is done
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -901,4 +901,47 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
}
}
}
+
+ test("Test enable hoodie.merge.allow.duplicate.on.inserts when write") {
+ spark.sql("set hoodie.datasource.write.operation = insert")
+ Seq("mor", "cow").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long,
+ | dt string
+ | ) using hudi
+ | partitioned by (dt)
+ | location '${tmp.getCanonicalPath}/$tableName'
+ | tblproperties (
+ | primaryKey = 'id',
+ | preCombineField = 'ts',
+ | type = '$tableType'
+ | )
+ """.stripMargin)
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a1', 10, 1000)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a1", 10, 1000, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a2', 20, 1001)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a3', 30, 1002)")
Review Comment:
I'd suggest we extract primary-key to a common var to make sharing apparent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1281976817
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "d5493649d73c87c6565044050e3c4b3781f954da",
"triggerType" : "PUSH"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
}, {
"hash" : "d5493649d73c87c6565044050e3c4b3781f954da",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272",
"triggerID" : "1281777512",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* d5493649d73c87c6565044050e3c4b3781f954da Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12276) Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12272)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1280935118
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12206",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
}, {
"hash" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12212",
"triggerID" : "c6b927a79d55a51282915beefe9c91b7042c81a3",
"triggerType" : "PUSH"
}, {
"hash" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12242",
"triggerID" : "bb6868a8d75b5d4af654ff00af9bf4ede5796878",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252",
"triggerID" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"triggerType" : "PUSH"
}, {
"hash" : "a7b34a73658bc9912bf8c3090d93d4482e96dc6b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255",
"triggerID" : "1280932026",
"triggerType" : "MANUAL"
} ]
}-->
## CI report:
* a7b34a73658bc9912bf8c3090d93d4482e96dc6b Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12252) Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12255)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] alexeykudinkin commented on a diff in pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
alexeykudinkin commented on code in PR #6949:
URL: https://github.com/apache/hudi/pull/6949#discussion_r998641348
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -901,4 +901,47 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
}
}
}
+
+ test("Test enable hoodie.merge.allow.duplicate.on.inserts when write") {
+ spark.sql("set hoodie.datasource.write.operation = insert")
+ Seq("mor", "cow").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long,
+ | dt string
+ | ) using hudi
+ | partitioned by (dt)
+ | location '${tmp.getCanonicalPath}/$tableName'
+ | tblproperties (
+ | primaryKey = 'id',
+ | preCombineField = 'ts',
+ | type = '$tableType'
+ | )
+ """.stripMargin)
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a1', 10, 1000)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a1", 10, 1000, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a2', 20, 1001)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a3', 30, 1002)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25"),
+ Seq(1, "a3", 30, 1002, "2021-12-25")
+ )
+ }
+ }
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
Review Comment:
We should reset it at the beginning of the test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] Zouxxyy commented on a diff in pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
Zouxxyy commented on code in PR #6949:
URL: https://github.com/apache/hudi/pull/6949#discussion_r998883271
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestInsertTable.scala:
##########
@@ -901,4 +901,47 @@ class TestInsertTable extends HoodieSparkSqlTestBase {
}
}
}
+
+ test("Test enable hoodie.merge.allow.duplicate.on.inserts when write") {
+ spark.sql("set hoodie.datasource.write.operation = insert")
+ Seq("mor", "cow").foreach { tableType =>
+ withTempDir { tmp =>
+ val tableName = generateTableName
+ spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long,
+ | dt string
+ | ) using hudi
+ | partitioned by (dt)
+ | location '${tmp.getCanonicalPath}/$tableName'
+ | tblproperties (
+ | primaryKey = 'id',
+ | preCombineField = 'ts',
+ | type = '$tableType'
+ | )
+ """.stripMargin)
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a1', 10, 1000)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a1", 10, 1000, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a2', 20, 1001)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25")
+ )
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = true")
+ spark.sql(s"insert into $tableName partition(dt='2021-12-25') values (1, 'a3', 30, 1002)")
+ checkAnswer(s"select id, name, price, ts, dt from $tableName")(
+ Seq(1, "a2", 20, 1001, "2021-12-25"),
+ Seq(1, "a3", 30, 1002, "2021-12-25")
+ )
+ }
+ }
+ spark.sql("set hoodie.merge.allow.duplicate.on.inserts = false")
Review Comment:
@YannByron OK, I create a ticket https://issues.apache.org/jira/browse/HUDI-5055
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #6949: [MINOR] Test case for hoodie.merge.allow.duplicate.on.inserts
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on PR #6949:
URL: https://github.com/apache/hudi/pull/6949#issuecomment-1278643840
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203",
"triggerID" : "f7ab63333a74e0fe073dda02276b3812c1cb1503",
"triggerType" : "PUSH"
}, {
"hash" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "c0614b3a245b7317d186583654ebcb61e4de7bfc",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f7ab63333a74e0fe073dda02276b3812c1cb1503 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=12203)
* c0614b3a245b7317d186583654ebcb61e4de7bfc UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org