You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "jonvex (via GitHub)" <gi...@apache.org> on 2023/06/15 18:12:31 UTC
[GitHub] [hudi] jonvex opened a new pull request, #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
jonvex opened a new pull request, #8990:
URL: https://github.com/apache/hudi/pull/8990
### Change Logs
beforeFileSlice and afterFileSlice were reversed as arguments to the constructor for HoodieCDCFileSplit and an assert was failing that the split didn't have a beforeFileSlice
Wrote a test that fails without the fix and succeeds with it
### Impact
Fix bug
### Risk level (write none, low medium or high below)
low
### Documentation Update
N/A
### Contributor's checklist
- [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8990:
URL: https://github.com/apache/hudi/pull/8990#discussion_r1231697257
##########
hudi-common/src/main/java/org/apache/hudi/common/table/cdc/HoodieCDCExtractor.java:
##########
@@ -265,7 +265,7 @@ private HoodieCDCFileSplit parseWriteStat(
new HoodieIOException("Can not get the previous version of the base file")
);
FileSlice beforeFileSlice = new FileSlice(fileGroupId, writeStat.getPrevCommit(), beforeBaseFile, Collections.emptyList());
- cdcFileSplit = new HoodieCDCFileSplit(instantTs, BASE_FILE_DELETE, new ArrayList<>(), Option.empty(), Option.of(beforeFileSlice));
+ cdcFileSplit = new HoodieCDCFileSplit(instantTs, BASE_FILE_DELETE, new ArrayList<>(), Option.of(beforeFileSlice), Option.empty());
} else if (writeStat.getNumUpdateWrites() == 0L && writeStat.getNumDeletes() == 0
Review Comment:
Nice catch ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua merged pull request #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #8990:
URL: https://github.com/apache/hudi/pull/8990
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8990:
URL: https://github.com/apache/hudi/pull/8990#issuecomment-1593579591
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7a1da85eaf1994379dfa22a94579d09433cde477",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17898",
"triggerID" : "7a1da85eaf1994379dfa22a94579d09433cde477",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a1da85eaf1994379dfa22a94579d09433cde477 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17898)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] danny0405 commented on a diff in pull request #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #8990:
URL: https://github.com/apache/hudi/pull/8990#discussion_r1231697487
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCDCForSparkSQL.scala:
##########
@@ -44,6 +44,47 @@ class TestCDCForSparkSQL extends HoodieSparkSqlTestBase {
assertEquals(expectedDeletedCnt, cdcData.where("op = 'd'").count())
}
+ test("Test delete all records in filegroup") {
+ withTempDir { tmp =>
+ val databaseName = "hudi_database"
+ spark.sql(s"create database if not exists $databaseName")
+ spark.sql(s"use $databaseName")
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ | ) using hudi
+ | partitioned by (name)
+ | tblproperties (
+ | 'primaryKey' = 'id',
+ | 'preCombineField' = 'ts',
+ | 'hoodie.table.cdc.enabled' = 'true',
+ | 'hoodie.table.cdc.supplemental.logging.mode' = '$DATA_BEFORE_AFTER',
+ | type = 'cow'
+ | )
+ | location '$basePath'
+ """.stripMargin)
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(basePath)
+ .setConf(spark.sessionState.newHadoopConf())
+ .build()
+ spark.sql(s"insert into $tableName values (1, 11, 1000, 'a1'), (2, 12, 1000, 'a2')")
+ val commitTime1 = metaClient.reloadActiveTimeline.lastInstant().get().getTimestamp
+ val cdcDataOnly1 = cdcDataFrame(basePath, commitTime1.toLong - 1)
+ cdcDataOnly1.show(false)
+ assertCDCOpCnt(cdcDataOnly1, 2, 0, 0)
+
+ spark.sql(s"delete from $tableName where id = 1")
+ val cdcDataOnly2 = cdcDataFrame(basePath, commitTime1.toLong)
+ assertCDCOpCnt(cdcDataOnly2, 0, 0, 1)
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8990:
URL: https://github.com/apache/hudi/pull/8990#issuecomment-1593856361
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7a1da85eaf1994379dfa22a94579d09433cde477",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17898",
"triggerID" : "7a1da85eaf1994379dfa22a94579d09433cde477",
"triggerType" : "PUSH"
}, {
"hash" : "46c421b0666395d3fb84efb9777d96c8cebe83b0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17899",
"triggerID" : "46c421b0666395d3fb84efb9777d96c8cebe83b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 46c421b0666395d3fb84efb9777d96c8cebe83b0 Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17899)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8990:
URL: https://github.com/apache/hudi/pull/8990#issuecomment-1593702927
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7a1da85eaf1994379dfa22a94579d09433cde477",
"status" : "CANCELED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17898",
"triggerID" : "7a1da85eaf1994379dfa22a94579d09433cde477",
"triggerType" : "PUSH"
}, {
"hash" : "46c421b0666395d3fb84efb9777d96c8cebe83b0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17899",
"triggerID" : "46c421b0666395d3fb84efb9777d96c8cebe83b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a1da85eaf1994379dfa22a94579d09433cde477 Azure: [CANCELED](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17898)
* 46c421b0666395d3fb84efb9777d96c8cebe83b0 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17899)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on pull request #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #8990:
URL: https://github.com/apache/hudi/pull/8990#issuecomment-1593559413
@danny0405 @YannByron could you review too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8990:
URL: https://github.com/apache/hudi/pull/8990#issuecomment-1593656272
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7a1da85eaf1994379dfa22a94579d09433cde477",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17898",
"triggerID" : "7a1da85eaf1994379dfa22a94579d09433cde477",
"triggerType" : "PUSH"
}, {
"hash" : "46c421b0666395d3fb84efb9777d96c8cebe83b0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "46c421b0666395d3fb84efb9777d96c8cebe83b0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a1da85eaf1994379dfa22a94579d09433cde477 Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=17898)
* 46c421b0666395d3fb84efb9777d96c8cebe83b0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a diff in pull request #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #8990:
URL: https://github.com/apache/hudi/pull/8990#discussion_r1231414550
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/TestCDCForSparkSQL.scala:
##########
@@ -44,6 +44,47 @@ class TestCDCForSparkSQL extends HoodieSparkSqlTestBase {
assertEquals(expectedDeletedCnt, cdcData.where("op = 'd'").count())
}
+ test("Test delete all records in filegroup") {
+ withTempDir { tmp =>
+ val databaseName = "hudi_database"
+ spark.sql(s"create database if not exists $databaseName")
+ spark.sql(s"use $databaseName")
+ val tableName = generateTableName
+ val basePath = s"${tmp.getCanonicalPath}/$tableName"
+ spark.sql(
+ s"""
+ | create table $tableName (
+ | id int,
+ | name string,
+ | price double,
+ | ts long
+ | ) using hudi
+ | partitioned by (name)
+ | tblproperties (
+ | 'primaryKey' = 'id',
+ | 'preCombineField' = 'ts',
+ | 'hoodie.table.cdc.enabled' = 'true',
+ | 'hoodie.table.cdc.supplemental.logging.mode' = '$DATA_BEFORE_AFTER',
+ | type = 'cow'
+ | )
+ | location '$basePath'
+ """.stripMargin)
+ val metaClient = HoodieTableMetaClient.builder()
+ .setBasePath(basePath)
+ .setConf(spark.sessionState.newHadoopConf())
+ .build()
+ spark.sql(s"insert into $tableName values (1, 11, 1000, 'a1'), (2, 12, 1000, 'a2')")
+ val commitTime1 = metaClient.reloadActiveTimeline.lastInstant().get().getTimestamp
+ val cdcDataOnly1 = cdcDataFrame(basePath, commitTime1.toLong - 1)
+ cdcDataOnly1.show(false)
+ assertCDCOpCnt(cdcDataOnly1, 2, 0, 0)
+
+ spark.sql(s"delete from $tableName where id = 1")
+ val cdcDataOnly2 = cdcDataFrame(basePath, commitTime1.toLong)
+ assertCDCOpCnt(cdcDataOnly2, 0, 0, 1)
Review Comment:
Could you add assert statements to guarantee the precondition of removing one file group, i.e., before deletion there are two file groups, and after deletion there is one file group, and the deleted record belongs to the removed file group?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #8990: [HUDI-6387] Fix Spark CDC when delete all records from a filegroup
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #8990:
URL: https://github.com/apache/hudi/pull/8990#issuecomment-1593569903
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7a1da85eaf1994379dfa22a94579d09433cde477",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7a1da85eaf1994379dfa22a94579d09433cde477",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7a1da85eaf1994379dfa22a94579d09433cde477 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org