You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by GitBox <gi...@apache.org> on 2021/01/15 10:48:11 UTC
[GitHub] [hudi] wangxianghu opened a new pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
wangxianghu opened a new pull request #2452:
URL: https://github.com/apache/hudi/pull/2452
…tion
## *Tips*
- *Thank you very much for contributing to Apache Hudi.*
- *Please review https://hudi.apache.org/contributing.html before opening a pull request.*
## What is the purpose of the pull request
*(For example: This pull request adds quick-start document.)*
## Brief change log
*(for example:)*
- *Modify AnnotationLocation checkstyle rule in checkstyle.xml*
## Verify this pull request
*(Please pick either of the following options)*
This pull request is a trivial rework / code cleanup without any test coverage.
*(or)*
This pull request is already covered by existing tests, such as *(please describe tests)*.
(or)
This change added tests and can be verified as follows:
*(example:)*
- *Added integration tests for end-to-end.*
- *Added HoodieClientWriteTest to verify the change.*
- *Manually verified the change by running a job locally.*
## Committer checklist
- [ ] Has a corresponding JIRA in PR title & commit
- [ ] Commit message is descriptive of the change
- [ ] CI is green
- [ ] Necessary doc changes done or have another open PR
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
wangxianghu edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-814591829
> I understand its in darft. Just before you open it up for reviews, can you add more info in the description.
> is this a standalone tool? or to be used as a strategy along w/ cleaning? or hudi-cli ?
At the beginning, I wanted to make it a stand-alone tool, but now we can consider making a cleanup strategy.
WDYT ? cc @vinothchandar @lw309637554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-769518711
@n3nash can you also please review this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] vinothchandar commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
vinothchandar commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-765711222
can we just have a this implemented as a replace of the partition where all files are replaced by an empty list. cleaner would automatically clean the partition that way. Love to keep all of our tooling to be flexible at the file level, working with existing actions and timeline
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-815878916
sounds good. yeah, cleaning strategy would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-817089012
> can we just have a this implemented as a replace of the partition where all files are replaced by an empty list. cleaner would automatically clean the partition that way. Love to keep all of our tooling to be flexible at the file level, working with existing actions and timeline
Hi @vinothchandar @nsivabalan. sorry for the delay, kind of busy these days.
The current implementation leveraged the API `org.apache.hudi.client.SparkRDDWriteClient#deletePartitions` provided by @lw309637554, IIUC, it is already implemented as a replace of the partition where all files are replaced by empty data.
BTW, I come to think that the clean strategy is more suitable for TTL scenarios, not this one.
partition deletion is a relatively fixed scene(one-shot use case), and it seems to be better to make it as a tool. In this way, the user only needs to specify the partition to be deleted, a small amount of configuration such as table information, and the partition can be deleted.
In the TTL scene, hoodie can check the outdated partitions in each commit(clean part), which could be a long-running term, so it is suitable implemented with clean strategy
WDYT ?
cc @n3nash @lw309637554 @yanghua
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-765808446
> can we just have a this implemented as a replace of the partition where all files are replaced by an empty list. cleaner would automatically clean the partition that way. Love to keep all of our tooling to be flexible at the file level, working with existing actions and timeline
I will give a try
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lw309637554 commented on a change in pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
lw309637554 commented on a change in pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#discussion_r559072569
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodiePartitionCleaner.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities;
+
+import org.apache.hudi.client.HoodieWriteResult;
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Util Class help to delete hoodie data in specific partition.
+ */
+public class HoodiePartitionCleaner {
+
+ private static final Logger LOG = LogManager.getLogger(HoodiePartitionCleaner.class);
+
+ /**
+ * Config for Cleaner.
+ */
+ private final Config cfg;
+
+ /**
+ * Spark context.
+ */
+ private transient JavaSparkContext jssc;
+
+ /**
+ * Bag of properties with source, hoodie client etc.
+ */
+ private TypedProperties props;
+
+ public HoodiePartitionCleaner(Config cfg, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ /*
+ * Filesystem used.
+ */
+ FileSystem fs = FSUtils.getFs(cfg.basePath, jssc.hadoopConfiguration());
+ this.props = cfg.propsFilePath == null ? UtilHelpers.buildProperties(cfg.configs)
+ : UtilHelpers.readConfig(fs, new Path(cfg.propsFilePath), cfg.configs).getConfig();
+ LOG.info("Creating Cleaner with configs : " + props.toString());
+ }
+
+ public void run() {
+ HoodieWriteConfig hoodieCfg = getHoodieClientConfig();
+ SparkRDDWriteClient client = new SparkRDDWriteClient<>(new HoodieSparkEngineContext(jssc), hoodieCfg, true);
+ String cleanInstant = HoodieActiveTimeline.createNewInstantTime();
+ client.startCommitWithTime(cleanInstant, HoodieTimeline.REPLACE_COMMIT_ACTION);
Review comment:
1.now with replace_commit_action at incremental scenario also have some issues, please pay attention.
2. can link the issue to https://issues.apache.org/jira/browse/HUDI-1042
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=h1) Report
> Merging [#2452](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=desc) (dc8dbbc) into [master](https://codecov.io/gh/apache/hudi/commit/749f6578561cbf065c7f74ab51b1c01881a1bd97?el=desc) (749f657) will **decrease** coverage by `41.24%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2452 +/- ##
============================================
- Coverage 50.71% 9.46% -41.25%
+ Complexity 3060 48 -3012
============================================
Files 419 54 -365
Lines 18796 1975 -16821
Branches 1922 233 -1689
============================================
- Hits 9533 187 -9346
+ Misses 8488 1775 -6713
+ Partials 775 13 -762
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.46% <0.00%> (-60.02%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodiePartitionCleaner.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVBhcnRpdGlvbkNsZWFuZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [398 more](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=h1) Report
> Merging [#2452](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=desc) (6c95dfc) into [master](https://codecov.io/gh/apache/hudi/commit/749f6578561cbf065c7f74ab51b1c01881a1bd97?el=desc) (749f657) will **decrease** coverage by `41.34%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2452 +/- ##
============================================
- Coverage 50.71% 9.37% -41.35%
+ Complexity 3060 48 -3012
============================================
Files 419 54 -365
Lines 18796 1994 -16802
Branches 1922 236 -1686
============================================
- Hits 9533 187 -9346
+ Misses 8488 1794 -6694
+ Partials 775 13 -762
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.37% <0.00%> (-60.11%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodiePartitionCleaner.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVBhcnRpdGlvbkNsZWFuZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [398 more](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
nsivabalan edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-814556277
I understand its in darft. Just before you open it up for reviews, can you add more info in the description.
is this a standalone tool? or to be used as a strategy along w/ cleaning? or hudi-cli ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu closed pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
wangxianghu closed pull request #2452:
URL: https://github.com/apache/hudi/pull/2452
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] nsivabalan commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
nsivabalan commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-814556277
Can you add more info in the description.
is this a standalone tool? or to be used as a strategy along w/ cleaning? or hudi-cli ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] lw309637554 commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
lw309637554 commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-813768823
> can we just have a this implemented as a replace of the partition where all files are replaced by an empty list. cleaner would automatically clean the partition that way. Love to keep all of our tooling to be flexible at the file level, working with existing actions and timeline
@vinothchandar @wangxianghu
Hello, does this mean we can add a auto cleaner strategy to clean the old partition? Maybe it is in cleaner or just like inline clustering. Now a user want to auto partition TTL feature.
https://github.com/apache/hudi/issues/2743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=h1) Report
> Merging [#2452](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=desc) (6c95dfc) into [master](https://codecov.io/gh/apache/hudi/commit/749f6578561cbf065c7f74ab51b1c01881a1bd97?el=desc) (749f657) will **decrease** coverage by `6.75%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2452 +/- ##
============================================
- Coverage 50.71% 43.96% -6.76%
+ Complexity 3060 2690 -370
============================================
Files 419 414 -5
Lines 18796 18480 -316
Branches 1922 1912 -10
============================================
- Hits 9533 8124 -1409
- Misses 8488 9715 +1227
+ Partials 775 641 -134
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `37.26% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.97% <ø> (-0.01%)` | `0.00 <ø> (ø)` | |
| hudiflink | `10.20% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.06% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `65.90% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `48.61% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.37% <0.00%> (-60.11%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodiePartitionCleaner.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVBhcnRpdGlvbkNsZWFuZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [39 more](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-810151041
> @wangxianghu Can you please address @vinothchandar request ? I think we can land it after that
ack, will do
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761408789
@yanghua @lw309637554 please take a look when free
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on a change in pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on a change in pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#discussion_r559272503
##########
File path: hudi-utilities/src/main/java/org/apache/hudi/utilities/HoodiePartitionCleaner.java
##########
@@ -0,0 +1,142 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.utilities;
+
+import org.apache.hudi.client.HoodieWriteResult;
+import org.apache.hudi.client.SparkRDDWriteClient;
+import org.apache.hudi.client.common.HoodieSparkEngineContext;
+import org.apache.hudi.common.config.TypedProperties;
+import org.apache.hudi.common.fs.FSUtils;
+import org.apache.hudi.common.table.timeline.HoodieActiveTimeline;
+import org.apache.hudi.common.table.timeline.HoodieTimeline;
+import org.apache.hudi.common.util.Option;
+import org.apache.hudi.config.HoodieWriteConfig;
+
+import com.beust.jcommander.JCommander;
+import com.beust.jcommander.Parameter;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hudi.exception.HoodieException;
+import org.apache.log4j.LogManager;
+import org.apache.log4j.Logger;
+import org.apache.spark.api.java.JavaSparkContext;
+
+import java.io.Serializable;
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Util Class help to delete hoodie data in specific partition.
+ */
+public class HoodiePartitionCleaner {
+
+ private static final Logger LOG = LogManager.getLogger(HoodiePartitionCleaner.class);
+
+ /**
+ * Config for Cleaner.
+ */
+ private final Config cfg;
+
+ /**
+ * Spark context.
+ */
+ private transient JavaSparkContext jssc;
+
+ /**
+ * Bag of properties with source, hoodie client etc.
+ */
+ private TypedProperties props;
+
+ public HoodiePartitionCleaner(Config cfg, JavaSparkContext jssc) {
+ this.cfg = cfg;
+ this.jssc = jssc;
+ /*
+ * Filesystem used.
+ */
+ FileSystem fs = FSUtils.getFs(cfg.basePath, jssc.hadoopConfiguration());
+ this.props = cfg.propsFilePath == null ? UtilHelpers.buildProperties(cfg.configs)
+ : UtilHelpers.readConfig(fs, new Path(cfg.propsFilePath), cfg.configs).getConfig();
+ LOG.info("Creating Cleaner with configs : " + props.toString());
+ }
+
+ public void run() {
+ HoodieWriteConfig hoodieCfg = getHoodieClientConfig();
+ SparkRDDWriteClient client = new SparkRDDWriteClient<>(new HoodieSparkEngineContext(jssc), hoodieCfg, true);
+ String cleanInstant = HoodieActiveTimeline.createNewInstantTime();
+ client.startCommitWithTime(cleanInstant, HoodieTimeline.REPLACE_COMMIT_ACTION);
Review comment:
> 1.now with replace_commit_action at incremental scenario also have some issues, please pay attention.
> 2. can link the issue to https://issues.apache.org/jira/browse/HUDI-1042
ok, thanks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=h1) Report
> Merging [#2452](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=desc) (f3e08bd) into [master](https://codecov.io/gh/apache/hudi/commit/749f6578561cbf065c7f74ab51b1c01881a1bd97?el=desc) (749f657) will **decrease** coverage by `41.23%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2452 +/- ##
============================================
- Coverage 50.71% 9.48% -41.24%
+ Complexity 3060 48 -3012
============================================
Files 419 54 -365
Lines 18796 1971 -16825
Branches 1922 233 -1689
============================================
- Hits 9533 187 -9346
+ Misses 8488 1771 -6717
+ Partials 775 13 -762
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.48% <0.00%> (-60.00%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodiePartitionCleaner.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVBhcnRpdGlvbkNsZWFuZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [398 more](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=h1) Report
> Merging [#2452](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=desc) (dc8dbbc) into [master](https://codecov.io/gh/apache/hudi/commit/749f6578561cbf065c7f74ab51b1c01881a1bd97?el=desc) (749f657) will **decrease** coverage by `8.92%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2452 +/- ##
============================================
- Coverage 50.71% 41.79% -8.93%
+ Complexity 3060 2407 -653
============================================
Files 419 365 -54
Lines 18796 16007 -2789
Branches 1922 1610 -312
============================================
- Hits 9533 6690 -2843
- Misses 8488 8819 +331
+ Partials 775 498 -277
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `37.26% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.99% <ø> (+0.01%)` | `0.00 <ø> (ø)` | |
| hudiflink | `10.20% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.06% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.46% <0.00%> (-60.02%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodiePartitionCleaner.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVBhcnRpdGlvbkNsZWFuZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [89 more](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=h1) Report
> Merging [#2452](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=desc) (f3e08bd) into [master](https://codecov.io/gh/apache/hudi/commit/749f6578561cbf065c7f74ab51b1c01881a1bd97?el=desc) (749f657) will **decrease** coverage by `41.23%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2452 +/- ##
============================================
- Coverage 50.71% 9.48% -41.24%
+ Complexity 3060 48 -3012
============================================
Files 419 54 -365
Lines 18796 1971 -16825
Branches 1922 233 -1689
============================================
- Hits 9533 187 -9346
+ Misses 8488 1771 -6717
+ Partials 775 13 -762
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `?` | `?` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.48% <0.00%> (-60.00%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodiePartitionCleaner.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVBhcnRpdGlvbkNsZWFuZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [398 more](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=h1) Report
> Merging [#2452](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=desc) (8052abf) into [master](https://codecov.io/gh/apache/hudi/commit/e970e1f48302aec3af7eeca009a2c793757cd501?el=desc) (e970e1f) will **decrease** coverage by `0.16%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2452 +/- ##
============================================
- Coverage 52.32% 52.16% -0.17%
Complexity 3689 3689
============================================
Files 483 484 +1
Lines 23095 23159 +64
Branches 2460 2466 +6
============================================
- Hits 12084 12080 -4
- Misses 9942 10010 +68
Partials 1069 1069
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `37.01% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `∅ <ø> (∅)` | `0.00 <ø> (ø)` | |
| hudicommon | `50.78% <ø> (-0.05%)` | `0.00 <ø> (ø)` | |
| hudiflink | `56.71% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.44% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `71.33% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `45.47% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `64.36% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `67.57% <0.00%> (-2.12%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodiePartitionCleaner.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVBhcnRpdGlvbkNsZWFuZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...ache/hudi/common/fs/inline/InMemoryFileSystem.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2ZzL2lubGluZS9Jbk1lbW9yeUZpbGVTeXN0ZW0uamF2YQ==) | `79.31% <0.00%> (-10.35%)` | `15.00% <0.00%> (-1.00%)` | |
| [...e/hudi/common/table/log/HoodieLogFormatWriter.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL3RhYmxlL2xvZy9Ib29kaWVMb2dGb3JtYXRXcml0ZXIuamF2YQ==) | `78.12% <0.00%> (-1.57%)` | `26.00% <0.00%> (ø%)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `71.42% <0.00%> (+0.34%)` | `56.00% <0.00%> (+1.00%)` | |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] n3nash commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
n3nash commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-809994216
@wangxianghu Can you please address @vinothchandar request ? I think we can land it after that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] wangxianghu commented on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific partition
Posted by GitBox <gi...@apache.org>.
wangxianghu commented on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-814591829
> I understand its in darft. Just before you open it up for reviews, can you add more info in the description.
> is this a standalone tool? or to be used as a strategy along w/ cleaning? or hudi-cli ?
At the beginning, I wanted to make it a stand-alone tool, but now we can consider making a cleanup strategy.
WDYT ? cc @lw309637554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=h1) Report
> Merging [#2452](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=desc) (6c95dfc) into [master](https://codecov.io/gh/apache/hudi/commit/749f6578561cbf065c7f74ab51b1c01881a1bd97?el=desc) (749f657) will **decrease** coverage by `0.18%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2452 +/- ##
============================================
- Coverage 50.71% 50.53% -0.19%
+ Complexity 3060 3059 -1
============================================
Files 419 420 +1
Lines 18796 18863 +67
Branches 1922 1928 +6
============================================
Hits 9533 9533
- Misses 8488 8554 +66
- Partials 775 776 +1
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `37.26% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `51.97% <ø> (-0.01%)` | `0.00 <ø> (ø)` | |
| hudiflink | `10.20% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudihadoopmr | `33.06% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisparkdatasource | `65.90% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudisync | `48.61% <ø> (ø)` | `0.00 <ø> (ø)` | |
| huditimelineservice | `66.84% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudiutilities | `67.20% <0.00%> (-2.29%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodiePartitionCleaner.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVBhcnRpdGlvbkNsZWFuZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [.../apache/hudi/common/config/SerializableSchema.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvY29tbW9uL2NvbmZpZy9TZXJpYWxpemFibGVTY2hlbWEuamF2YQ==) | `54.54% <0.00%> (-3.35%)` | `6.00% <0.00%> (ø%)` | |
| [...apache/hudi/utilities/deltastreamer/DeltaSync.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL2RlbHRhc3RyZWFtZXIvRGVsdGFTeW5jLmphdmE=) | `70.50% <0.00%> (-0.36%)` | `50.00% <0.00%> (-1.00%)` | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [hudi] codecov-io edited a comment on pull request #2452: [HUDI-1531] Introduce HoodiePartitionCleaner to delete specific parti…
Posted by GitBox <gi...@apache.org>.
codecov-io edited a comment on pull request #2452:
URL: https://github.com/apache/hudi/pull/2452#issuecomment-761259726
# [Codecov](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=h1) Report
> Merging [#2452](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=desc) (6c95dfc) into [master](https://codecov.io/gh/apache/hudi/commit/749f6578561cbf065c7f74ab51b1c01881a1bd97?el=desc) (749f657) will **decrease** coverage by `41.34%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #2452 +/- ##
============================================
- Coverage 50.71% 9.37% -41.35%
+ Complexity 3060 48 -3012
============================================
Files 419 54 -365
Lines 18796 1994 -16802
Branches 1922 236 -1686
============================================
- Hits 9533 187 -9346
+ Misses 8488 1794 -6694
+ Partials 775 13 -762
```
| Flag | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| hudicli | `?` | `?` | |
| hudiclient | `100.00% <ø> (ø)` | `0.00 <ø> (ø)` | |
| hudicommon | `?` | `?` | |
| hudiflink | `?` | `?` | |
| hudihadoopmr | `?` | `?` | |
| hudisparkdatasource | `?` | `?` | |
| hudisync | `?` | `?` | |
| huditimelineservice | `?` | `?` | |
| hudiutilities | `9.37% <0.00%> (-60.11%)` | `0.00 <0.00> (ø)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/hudi/pull/2452?src=pr&el=tree) | Coverage Δ | Complexity Δ | |
|---|---|---|---|
| [.../apache/hudi/utilities/HoodiePartitionCleaner.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0hvb2RpZVBhcnRpdGlvbkNsZWFuZXIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | |
| [...va/org/apache/hudi/utilities/IdentitySplitter.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL0lkZW50aXR5U3BsaXR0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-2.00%)` | |
| [...va/org/apache/hudi/utilities/schema/SchemaSet.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NjaGVtYS9TY2hlbWFTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-3.00%)` | |
| [...a/org/apache/hudi/utilities/sources/RowSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUm93U291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [.../org/apache/hudi/utilities/sources/AvroSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQXZyb1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [.../org/apache/hudi/utilities/sources/JsonSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvblNvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-1.00%)` | |
| [...rg/apache/hudi/utilities/sources/CsvDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvQ3N2REZTU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-10.00%)` | |
| [...g/apache/hudi/utilities/sources/JsonDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkRGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-4.00%)` | |
| [...apache/hudi/utilities/sources/JsonKafkaSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvSnNvbkthZmthU291cmNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-6.00%)` | |
| [...pache/hudi/utilities/sources/ParquetDFSSource.java](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree#diff-aHVkaS11dGlsaXRpZXMvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2h1ZGkvdXRpbGl0aWVzL3NvdXJjZXMvUGFycXVldERGU1NvdXJjZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | `0.00% <0.00%> (-5.00%)` | |
| ... and [398 more](https://codecov.io/gh/apache/hudi/pull/2452/diff?src=pr&el=tree-more) | |
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org