You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2022/02/10 20:54:35 UTC
[GitHub] [hudi] nsivabalan opened a new pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
nsivabalan opened a new pull request #4787:
URL: https://github.com/apache/hudi/pull/4787
## What is the purpose of the pull request
Adding delete partition support to deltastreamer.
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vburenin commented on a change in pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
vburenin commented on a change in pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#discussion_r804161954
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -555,6 +555,10 @@ public void refreshTimeline() throws IOException {
case INSERT_OVERWRITE_TABLE:
writeStatusRDD = writeClient.insertOverwriteTable(records, instantTime).getWriteStatuses();
break;
+ case DELETE_PARTITION:
+ List<String> partitions = records.map(record -> record.getPartitionPath()).distinct().collect();
Review comment:
My data source is Avro encoded, the schema URL can't change. It may be hard to bypass it in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#discussion_r806215809
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##########
@@ -1944,6 +1990,18 @@ public Dataset apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Ro
}
}
+ public static class TestSpecificPartitionTransformer implements Transformer {
+
+ @Override
+ public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset,
+ TypedProperties properties) {
+ //List<Row> row0 = rowDataset.collectAsList();
Review comment:
nit: same here.
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##########
@@ -258,6 +258,20 @@ static void assertRecordCount(long expected, String tablePath, SQLContext sqlCon
assertEquals(expected, recordCount);
}
+ static Map<String, Long> getPartitionRecordCount(String basePath, SQLContext sqlContext) {
+ sqlContext.clearCache();
+ //List<Row> rows0 = sqlContext.read().format("org.apache.hudi").load(basePath).collectAsList();
Review comment:
nit: remove unused code?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -555,6 +555,10 @@ public void refreshTimeline() throws IOException {
case INSERT_OVERWRITE_TABLE:
writeStatusRDD = writeClient.insertOverwriteTable(records, instantTime).getWriteStatuses();
break;
+ case DELETE_PARTITION:
+ List<String> partitions = records.map(record -> record.getPartitionPath()).distinct().collect();
Review comment:
The logic here takes the partition list from the input records and does nothing to the records. This workflow looks strange to me. Shouldn't the input be the partition list directly? It's counter-intuitive for the users to provide a list of records for deleting corresponding partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#discussion_r804136818
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -555,6 +555,10 @@ public void refreshTimeline() throws IOException {
case INSERT_OVERWRITE_TABLE:
writeStatusRDD = writeClient.insertOverwriteTable(records, instantTime).getWriteStatuses();
break;
+ case DELETE_PARTITION:
+ List<String> partitions = records.map(record -> record.getPartitionPath()).distinct().collect();
Review comment:
yes. thats what I have been trying to convey for a long time :) Deltastreamer (as the name suggests) is meant for incremental and continual ingestion of data from some source. It goes in cycles of fetch from source -> ingest into hudi ->repeat. So, I don't see in general how come one would fetch data from a source and then trigger delete partitions.
But we have a [patch](https://github.com/apache/hudi/pull/4459) for a independent tool if you are interested. Guess that would help you. but that is a spark-submit command as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#discussion_r811487825
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -555,6 +555,10 @@ public void refreshTimeline() throws IOException {
case INSERT_OVERWRITE_TABLE:
writeStatusRDD = writeClient.insertOverwriteTable(records, instantTime).getWriteStatuses();
break;
+ case DELETE_PARTITION:
+ List<String> partitions = records.map(record -> record.getPartitionPath()).distinct().collect();
Review comment:
this is against the general nomenclature of how deltastreamer works. Delta streamer consumes records from a given source and and ingests to hudi. So, I don't think we can support a cli based argument which is more of one time standalone tool/job. So, I would probably suggest you to use the standalone [tool](https://github.com/apache/hudi/pull/4459).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1047340814
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
}, {
"hash" : "693f86e35cea5c472a54c5f8b1fab5dfa25bb06a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "693f86e35cea5c472a54c5f8b1fab5dfa25bb06a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f288ae1ff5f7a01abb9a320acf448079c44956ce Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883)
* 693f86e35cea5c472a54c5f8b1fab5dfa25bb06a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1047342399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
}, {
"hash" : "693f86e35cea5c472a54c5f8b1fab5dfa25bb06a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6185",
"triggerID" : "693f86e35cea5c472a54c5f8b1fab5dfa25bb06a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f288ae1ff5f7a01abb9a320acf448079c44956ce Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883)
* 693f86e35cea5c472a54c5f8b1fab5dfa25bb06a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6185)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1035510199
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f288ae1ff5f7a01abb9a320acf448079c44956ce UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1047378621
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "DELETED",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
}, {
"hash" : "693f86e35cea5c472a54c5f8b1fab5dfa25bb06a",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6185",
"triggerID" : "693f86e35cea5c472a54c5f8b1fab5dfa25bb06a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 693f86e35cea5c472a54c5f8b1fab5dfa25bb06a Azure: [FAILURE](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6185)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1035513151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f288ae1ff5f7a01abb9a320acf448079c44956ce Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1047340814
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
}, {
"hash" : "693f86e35cea5c472a54c5f8b1fab5dfa25bb06a",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "693f86e35cea5c472a54c5f8b1fab5dfa25bb06a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f288ae1ff5f7a01abb9a320acf448079c44956ce Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883)
* 693f86e35cea5c472a54c5f8b1fab5dfa25bb06a UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1047342399
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
}, {
"hash" : "693f86e35cea5c472a54c5f8b1fab5dfa25bb06a",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6185",
"triggerID" : "693f86e35cea5c472a54c5f8b1fab5dfa25bb06a",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f288ae1ff5f7a01abb9a320acf448079c44956ce Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883)
* 693f86e35cea5c472a54c5f8b1fab5dfa25bb06a Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=6185)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on a change in pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on a change in pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#discussion_r807448915
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -555,6 +555,10 @@ public void refreshTimeline() throws IOException {
case INSERT_OVERWRITE_TABLE:
writeStatusRDD = writeClient.insertOverwriteTable(records, instantTime).getWriteStatuses();
break;
+ case DELETE_PARTITION:
+ List<String> partitions = records.map(record -> record.getPartitionPath()).distinct().collect();
Review comment:
ok. so, does hudi-cli work for you. I can try to add support there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1035581829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f288ae1ff5f7a01abb9a320acf448079c44956ce Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#discussion_r811512115
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -555,6 +555,10 @@ public void refreshTimeline() throws IOException {
case INSERT_OVERWRITE_TABLE:
writeStatusRDD = writeClient.insertOverwriteTable(records, instantTime).getWriteStatuses();
break;
+ case DELETE_PARTITION:
+ List<String> partitions = records.map(record -> record.getPartitionPath()).distinct().collect();
Review comment:
As discussed offline, `DELETE_PARTITION` should be added as a separate operation here, as to how this PR adds the functionality. To leverage this new op, user needs to pull some data from the existing partitions which are intended to be deleted, for the Deltastreamer to ingest. We'll keep this logic for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan merged pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
nsivabalan merged pull request #4787:
URL: https://github.com/apache/hudi/pull/4787
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1035513151
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "PENDING",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f288ae1ff5f7a01abb9a320acf448079c44956ce Azure: [PENDING](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vburenin commented on a change in pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
vburenin commented on a change in pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#discussion_r807478333
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -555,6 +555,10 @@ public void refreshTimeline() throws IOException {
case INSERT_OVERWRITE_TABLE:
writeStatusRDD = writeClient.insertOverwriteTable(records, instantTime).getWriteStatuses();
break;
+ case DELETE_PARTITION:
+ List<String> partitions = records.map(record -> record.getPartitionPath()).distinct().collect();
Review comment:
Would be nice if it was a delta streamer CLI parameter, something that gets executed at the end of the ingestion. hudi-cli may work if it can be done as a single CLI command. For example:
```
hudi-cli delete-partitions --schema-name someschema --table-name sometable --location s3a://bucket/data --hive-server xxxxx --metastore=xxx --do-not-delete-data
```
--do-not-delete data maybe helpful to delete data faster. I use go tool that spins up hundreds of goroutines to delete hundreds of thousands of files data within seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot removed a comment on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot removed a comment on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1035581829
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f288ae1ff5f7a01abb9a320acf448079c44956ce Azure: [SUCCESS](https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=5883)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] hudi-bot commented on pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
hudi-bot commented on pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#issuecomment-1035510199
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "f288ae1ff5f7a01abb9a320acf448079c44956ce",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* f288ae1ff5f7a01abb9a320acf448079c44956ce UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vburenin commented on a change in pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
vburenin commented on a change in pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#discussion_r804125594
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -555,6 +555,10 @@ public void refreshTimeline() throws IOException {
case INSERT_OVERWRITE_TABLE:
writeStatusRDD = writeClient.insertOverwriteTable(records, instantTime).getWriteStatuses();
break;
+ case DELETE_PARTITION:
+ List<String> partitions = records.map(record -> record.getPartitionPath()).distinct().collect();
Review comment:
If my data source is kafka, does it mean that the list of partitions to delete should be received from kafka? If answer is yes, it maybe cumbersome to combine ingestion workflow with the data and partition deletion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] yihua commented on a change in pull request #4787: [HUDI-2189] Adding delete partitions support to DeltaStreamer
Posted by GitBox <gi...@apache.org>.
yihua commented on a change in pull request #4787:
URL: https://github.com/apache/hudi/pull/4787#discussion_r806215809
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##########
@@ -1944,6 +1990,18 @@ public Dataset apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Ro
}
}
+ public static class TestSpecificPartitionTransformer implements Transformer {
+
+ @Override
+ public Dataset<Row> apply(JavaSparkContext jsc, SparkSession sparkSession, Dataset<Row> rowDataset,
+ TypedProperties properties) {
+ //List<Row> row0 = rowDataset.collectAsList();
Review comment:
nit: same here.
##########
File path: hudi-utilities/src/test/java/org/apache/hudi/utilities/functional/TestHoodieDeltaStreamer.java
##########
@@ -258,6 +258,20 @@ static void assertRecordCount(long expected, String tablePath, SQLContext sqlCon
assertEquals(expected, recordCount);
}
+ static Map<String, Long> getPartitionRecordCount(String basePath, SQLContext sqlContext) {
+ sqlContext.clearCache();
+ //List<Row> rows0 = sqlContext.read().format("org.apache.hudi").load(basePath).collectAsList();
Review comment:
nit: remove unused code?
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/DeltaSync.java
##########
@@ -555,6 +555,10 @@ public void refreshTimeline() throws IOException {
case INSERT_OVERWRITE_TABLE:
writeStatusRDD = writeClient.insertOverwriteTable(records, instantTime).getWriteStatuses();
break;
+ case DELETE_PARTITION:
+ List<String> partitions = records.map(record -> record.getPartitionPath()).distinct().collect();
Review comment:
The logic here takes the partition list from the input records and does nothing to the records. This workflow looks strange to me. Shouldn't the input be the partition list directly? It's counter-intuitive for the users to provide a list of records for deleting corresponding partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org