You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/06/17 07:31:55 UTC
[PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
nsivabalan opened a new pull request, #19033:
URL: https://github.com/apache/hudi/pull/19033
## Summary
- Parallelizes `dropPartitionsToTable` in the HiveQL sync mode (`hoodie.datasource.hive_sync.mode=hiveql`).
- Stacked on **#18984**. The diff currently includes both the HMS commit (#18983) and the HiveQL commit (#18984); once those merge, this PR will rebase cleanly onto master.
- Reviewing the top commit `7022e7d49341` in isolation gives the DROP-only delta (~119 added / 16 removed).
## What this fixes
HiveQL DROP goes through `IMetaStoreClient.dropPartition` (Thrift), not Hive Driver — so it can't reuse the `HiveDriverPool` introduced in #18984. Today it loops sequentially against the session metastore client. This PR wires the existing `IMetaStoreClientPool` from #18983 into `HiveQueryDDLExecutor` and uses it to fan drop batches across the pool's workers.
Behavior:
- `batching.enabled=false` (default): unchanged. `dropPartitionsToTable` iterates the partition list sequentially on the session metastore client, exactly as before.
- `batching.enabled=true`: partitions split into batches of `HIVE_BATCH_SYNC_PARTITION_NUM`, batches fan out across the pool's workers (one independent `IMetaStoreClient` per worker), first-error semantics match the HMS-mode implementation (first failure thrown, subsequent suppressed at WARN).
## Configs
No new configs. Reuses everything from #18983 / #18984:
| Key | Default |
|---|---|
| `hoodie.datasource.hive_sync.batching.enabled` | `false` |
| `hoodie.datasource.hive_sync.batching.threads` | `4` |
| `hoodie.datasource.hive_sync.batch_num` | `1000` |
## Test plan
- [x] `mvn compile` on `hudi-sync/hudi-hive-sync` — clean, 0 Checkstyle violations, 0 RAT issues
- [x] `mvn test` on `hudi-sync/hudi-hive-sync` — **309 tests, 0 failures, 0 errors** (was 308 on the parent branch)
- [x] `TestHiveSyncTool#testHiveQLDropPartitionsWithBatching` (new) — creates 8 partitions, drops 4 through the parallel pool path with `threads=3` and `batch_num=2` (multiple drop batches dispatched in parallel), asserts the remaining set matches.
- [x] Existing 308 tests across all three sync modes pass unchanged.
## Files touched (top commit only)
- `HiveQueryDDLExecutor.java` — new constructor accepting `Option<IMetaStoreClientPool>`; `dropPartitionsToTable` now batches and fans out when the pool is present, falls back to sequential single-client path otherwise.
- `HoodieHiveSyncClient.java` — build `IMetaStoreClientPool` in HIVEQL branches as well (HMS branch already did so).
- `TestHiveSyncTool.java` — new end-to-end test.
## Out of scope
- JDBC DROP parallelization — JDBC's `constructDropPartitions` already batches by `HIVE_BATCH_SYNC_PARTITION_NUM` but runs sequentially; parallelizing it needs a JDBC `Connection` pool, tracked separately.
- Benchmarks vs HDrone on a production HMS — same blocker as #18984; pinning to a follow-up after #18983 lands.
Related: #18331
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727641891
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 60.31%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 16 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|7|
>|spark-scala-tests|12|7|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 60.31% -7.96%
+ Complexity 29513 26595 -2918
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
- Hits 97369 87756 -9613
- Misses 37253 49914 +12661
+ Partials 8005 7820 -185
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `30.80% <3.08%> (-17.97%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.21% <3.08%> (-12.63%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `49.87% <67.85%> (+0.94%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [657 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727744070
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.53%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.53% -0.74%
- Complexity 29513 29795 +282
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98261 +892
- Misses 37253 39034 +1781
- Partials 8005 8195 +190
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.86% <3.08%> (-0.91%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.21% <7.45%> (-0.64%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [172 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727703120
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.53%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 66.53% -1.74%
+ Complexity 29513 29330 -183
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
- Hits 97369 96796 -573
- Misses 37253 40509 +3256
- Partials 8005 8185 +180
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.58% <3.08%> (-5.19%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.03% <7.45%> (-0.81%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [315 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3737119697
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -209,21 +217,57 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
- }
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(partitionsToDrop, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ *
+ * <p>First-error semantics come from {@link ParallelDispatch}, shared with
+ * {@code HiveDriverPool}: the first failure is rethrown, batches that have not started
+ * are stopped via the task-side abort flag, and later failures are logged at WARN.
+ */
+ private void runDropBatches(String tableName, List<List<String>> batches) throws Exception {
+ if (!metaStoreClientPool.isPresent()) {
+ for (List<String> batch : batches) {
+ applyDropBatch(metaStoreClient, tableName, batch);
+ }
+ return;
+ }
+ IMetaStoreClientPool pool = metaStoreClientPool.get();
+ pool.awaitAll(
+ pool.dispatchAll(batches, (client, batch) -> applyDropBatch(client, tableName, batch)),
+ "drop partition");
+ }
+
+ private void applyDropBatch(IMetaStoreClient client, String tableName, List<String> batch) throws Exception {
+ int dropped = 0;
+ for (String dropPartition : batch) {
+ if (HivePartitionUtil.partitionExists(client, tableName, dropPartition,
Review Comment:
Filed as a nit, but I agree with your read that it is more than that — this is the data-loss-shaped one. Went with your second suggestion (extract on the calling thread before fan-out) rather than documenting the requirement, in 3a1694a.
Confirmed it was real before fixing it. A test that records the thread of every `extractPartitionValuesInPath` call, with a 4-client pool and 16 partitions, fails on the old code with:
```
expected: <[main]>
but was: <[hudi-hive-sync-pool-1-1, hudi-hive-sync-pool-1-2,
hudi-hive-sync-pool-1-3, hudi-hive-sync-pool-1-4, main]>
```
All four workers were in the shared extractor concurrently, exactly as you described.
The fix resolves each partition once on the calling thread into an immutable carrier (path + values + drop clause) and fans out only that. Three side benefits beyond removing the concurrency assumption:
- **Halves the extractor calls.** `partitionExists` and `getPartitionClauseForDrop` were each extracting the same partition separately; they now share one result.
- **Fails before the first drop.** A malformed partition previously threw partway through the batch, after some partitions had already been dropped. It now fails ahead of any drop — same `HoodieHiveSyncException`, just earlier, which matters for a destructive operation.
- **`HMSDDLExecutor` is untouched.** `HivePartitionUtil` gained overloads taking pre-extracted values and the existing signatures delegate to them, so the sequential HMS path keeps its current behaviour.
New test: `TestDropPartitionExtractorThreading#extractorIsNeverInvokedFromAPoolWorker`. It lives in `hive.util` rather than beside the executor because it needs the package-private `IMetaStoreClientPool` constructor that accepts pre-built clients. It also asserts the drops actually reached the pool, so it cannot pass by silently never fanning out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3670419426
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/ParallelDispatch.java:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.common.util.VisibleForTesting;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Handle to one fan-out batch of partition work: the submitted futures plus the shared
+ * abort flag the tasks consult before running.
+ *
+ * <p>The abort flag exists because waiting on futures in <i>submission</i> order is not
+ * enough to stop queued work. If a later task fails quickly while an earlier one is slow,
+ * the awaiting thread is still parked on the earlier {@code Future.get()}, and the
+ * executor happily keeps starting every queued task in the meantime. By the time the
+ * failure is observed, most of the "not-yet-started" work has already run.
+ *
+ * <p>Two mechanisms fix that, and both are needed:
+ * <ul>
+ * <li>a {@link CountDownLatch} tripped by the <i>first</i> abort, so the awaiting
+ * thread wakes on failure rather than on its turn in submission order;</li>
+ * <li>a task-side {@link #aborted()} check on entry, because cancelling from the
+ * awaiting thread is inherently late — a worker can pull its next task off the
+ * queue at any moment.</li>
+ * </ul>
+ *
+ * <p>Shared by {@link HiveDriverPool} (Hive {@code Driver} statements) and
+ * {@link IMetaStoreClientPool} (Thrift {@code dropPartition} batches), which fan out over
+ * different execution models but need identical abort-on-first-error semantics.
+ */
+public final class ParallelDispatch {
+
+ private final List<Future<?>> futures;
+ private final int total;
+ private final AtomicInteger settled = new AtomicInteger(0);
+ private final AtomicBoolean aborted = new AtomicBoolean(false);
+ private final CountDownLatch done = new CountDownLatch(1);
+ private volatile boolean sealed;
+
+ ParallelDispatch(int total) {
+ this.total = total;
+ this.futures = new ArrayList<>(total);
+ }
+
+ void add(Future<?> future) {
+ futures.add(future);
+ }
+
+ // Called once submission finishes. A task that settles before the last submit
+ // would otherwise see settled < total and never trip the latch, so re-check here.
+ void sealed() {
+ sealed = true;
+ signalIfComplete();
+ }
+
+ boolean aborted() {
+ return aborted.get();
+ }
+
+ void abort() {
+ aborted.set(true);
+ done.countDown();
+ }
+
+ void taskSettled() {
+ settled.incrementAndGet();
+ signalIfComplete();
+ }
+
+ private void signalIfComplete() {
+ if (sealed && settled.get() >= total) {
+ done.countDown();
+ }
+ }
+
+ void awaitSettledOrAborted() {
+ if (total == 0) {
+ return;
+ }
+ try {
+ done.await();
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ aborted.set(true);
+ }
+ }
+
+ // mayInterruptIfRunning=false: a worker may be mid-statement against a Hive Driver or
+ // a Thrift client, and we don't want to tear that down partway. Cancel only tasks that
+ // haven't started; in-flight work runs to completion.
+ int cancelPending() {
+ int cancelled = 0;
+ for (Future<?> f : futures) {
+ if (f.cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ List<Future<?>> futures() {
+ return futures;
+ }
+
+ /**
+ * Wraps {@code body} so it observes this batch's abort flag: it skips itself if a
+ * sibling has already failed, trips the flag if it fails, and always records that it
+ * settled so the awaiting thread can be released.
+ */
+ Callable<Void> guard(Callable<Void> body, String skipMessage) {
+ return () -> {
+ if (aborted()) {
+ throw new CancellationException(skipMessage);
+ }
+ try {
+ return body.call();
+ } catch (Throwable t) {
+ abort();
+ throw t;
+ } finally {
+ taskSettled();
+ }
+ };
+ }
+
+ /**
+ * Waits for the batch to settle (or abort), cancels whatever had not started, and
+ * returns the outcome. Errors are observed in <i>completion</i> order, not submission
+ * order, so a failure on a fast worker stops the other queues even while a slow worker
+ * is still mid-statement.
+ */
+ Outcome awaitOutcome() {
+ awaitSettledOrAborted();
+ int cancelled = cancelPending();
+
+ Exception firstError = null;
+ int completed = 0;
+ List<Exception> suppressed = new ArrayList<>();
+ for (Future<?> f : futures) {
Review Comment:
**[P2] Preserve the failure that actually triggered the abort**
Although the latch now wakes on the first failure, the exception itself is still selected by walking `futures` in submission order. If future 1 fails quickly and triggers `abort()`, but earlier-submitted future 0 is still running and later fails too, this loop blocks on future 0 and makes its later exception `firstError`; the failure that actually aborted the batch is only suppressed. I reproduced this deterministically with a `fast-first` future at index 1 and a `slow-later` future at index 0: `outcome.firstError()` was `slow-later`. That contradicts the completion-order contract and can surface the wrong root cause to operators. Please record the first throwable atomically in `guard`/`abort` (for example with `AtomicReference.compareAndSet`) and use that recorded cause as the primary error while collecting the rest as suppressed.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/ParallelDispatch.java:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.common.util.VisibleForTesting;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Handle to one fan-out batch of partition work: the submitted futures plus the shared
+ * abort flag the tasks consult before running.
+ *
+ * <p>The abort flag exists because waiting on futures in <i>submission</i> order is not
+ * enough to stop queued work. If a later task fails quickly while an earlier one is slow,
+ * the awaiting thread is still parked on the earlier {@code Future.get()}, and the
+ * executor happily keeps starting every queued task in the meantime. By the time the
+ * failure is observed, most of the "not-yet-started" work has already run.
+ *
+ * <p>Two mechanisms fix that, and both are needed:
+ * <ul>
+ * <li>a {@link CountDownLatch} tripped by the <i>first</i> abort, so the awaiting
+ * thread wakes on failure rather than on its turn in submission order;</li>
+ * <li>a task-side {@link #aborted()} check on entry, because cancelling from the
+ * awaiting thread is inherently late — a worker can pull its next task off the
+ * queue at any moment.</li>
+ * </ul>
+ *
+ * <p>Shared by {@link HiveDriverPool} (Hive {@code Driver} statements) and
+ * {@link IMetaStoreClientPool} (Thrift {@code dropPartition} batches), which fan out over
+ * different execution models but need identical abort-on-first-error semantics.
+ */
+public final class ParallelDispatch {
+
+ private final List<Future<?>> futures;
+ private final int total;
+ private final AtomicInteger settled = new AtomicInteger(0);
+ private final AtomicBoolean aborted = new AtomicBoolean(false);
+ private final CountDownLatch done = new CountDownLatch(1);
+ private volatile boolean sealed;
+
+ ParallelDispatch(int total) {
+ this.total = total;
+ this.futures = new ArrayList<>(total);
+ }
+
+ void add(Future<?> future) {
+ futures.add(future);
+ }
+
+ // Called once submission finishes. A task that settles before the last submit
+ // would otherwise see settled < total and never trip the latch, so re-check here.
+ void sealed() {
+ sealed = true;
+ signalIfComplete();
+ }
+
+ boolean aborted() {
+ return aborted.get();
+ }
+
+ void abort() {
+ aborted.set(true);
+ done.countDown();
+ }
+
+ void taskSettled() {
+ settled.incrementAndGet();
+ signalIfComplete();
+ }
+
+ private void signalIfComplete() {
+ if (sealed && settled.get() >= total) {
+ done.countDown();
+ }
+ }
+
+ void awaitSettledOrAborted() {
+ if (total == 0) {
+ return;
+ }
+ try {
+ done.await();
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ aborted.set(true);
+ }
+ }
+
+ // mayInterruptIfRunning=false: a worker may be mid-statement against a Hive Driver or
+ // a Thrift client, and we don't want to tear that down partway. Cancel only tasks that
+ // haven't started; in-flight work runs to completion.
+ int cancelPending() {
+ int cancelled = 0;
+ for (Future<?> f : futures) {
+ if (f.cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ List<Future<?>> futures() {
+ return futures;
+ }
+
+ /**
+ * Wraps {@code body} so it observes this batch's abort flag: it skips itself if a
+ * sibling has already failed, trips the flag if it fails, and always records that it
+ * settled so the awaiting thread can be released.
+ */
+ Callable<Void> guard(Callable<Void> body, String skipMessage) {
+ return () -> {
+ if (aborted()) {
+ throw new CancellationException(skipMessage);
+ }
+ try {
+ return body.call();
+ } catch (Throwable t) {
+ abort();
+ throw t;
+ } finally {
+ taskSettled();
+ }
+ };
+ }
+
+ /**
+ * Waits for the batch to settle (or abort), cancels whatever had not started, and
+ * returns the outcome. Errors are observed in <i>completion</i> order, not submission
+ * order, so a failure on a fast worker stops the other queues even while a slow worker
+ * is still mid-statement.
+ */
+ Outcome awaitOutcome() {
+ awaitSettledOrAborted();
+ int cancelled = cancelPending();
+
+ Exception firstError = null;
+ int completed = 0;
+ List<Exception> suppressed = new ArrayList<>();
+ for (Future<?> f : futures) {
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // Either we cancelled it before it started, or the task itself observed the
+ // abort flag and bailed. Not a new failure; just note it for the summary.
+ cancelled++;
Review Comment:
**[P3] Count each canceled future only once**
`cancelPending()` already increments `cancelled` for every future where `cancel(false)` succeeds. Calling `get()` on those same futures then throws `CancellationException` here and increments the count again. A deterministic dispatch with one failed task and two never-started tasks returns `outcome.cancelled() == 4`, although only two tasks were canceled, so both pool logs can report more canceled batches/statements than were submitted. Please count successful cancellations in only one of these two places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3762993800
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -209,21 +217,84 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
+ // Resolved here, on the calling thread, rather than inside the workers: this is the
+ // only sync path that would otherwise call a user-supplied PartitionValueExtractor
+ // from several threads at once. Extractors are pluggable and not required to be
+ // thread-safe, and a garbled clause would drop the wrong partition. It also halves
+ // the extractor calls, since partitionExists and the drop clause share the values.
+ List<PartitionToDrop> resolved = new ArrayList<>(partitionsToDrop.size());
+ for (String partition : partitionsToDrop) {
+ List<String> values = partitionValueExtractor.extractPartitionValuesInPath(partition);
+ resolved.add(new PartitionToDrop(partition, values,
+ HivePartitionUtil.getPartitionClauseForDrop(values, config)));
}
+
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<PartitionToDrop>> batches = CollectionUtils.batches(resolved, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ *
+ * <p>First-error semantics come from {@link ParallelDispatch}, shared with
+ * {@code HiveDriverPool}: the first failure is rethrown, batches that have not started
+ * are stopped via the task-side abort flag, and later failures are logged at WARN.
+ */
+ private void runDropBatches(String tableName, List<List<PartitionToDrop>> batches) throws Exception {
+ if (!metaStoreClientPool.isPresent()) {
+ for (List<PartitionToDrop> batch : batches) {
+ applyDropBatch(metaStoreClient, tableName, batch);
+ }
+ return;
+ }
+ HiveMetaStoreClientPool pool = metaStoreClientPool.get();
+ pool.awaitAll(
+ pool.dispatchAll(batches, (client, batch) -> applyDropBatch(client, tableName, batch)),
+ "drop partition");
+ }
+
+ private void applyDropBatch(IMetaStoreClient client, String tableName, List<PartitionToDrop> batch) throws Exception {
+ int dropped = 0;
+ for (PartitionToDrop dropPartition : batch) {
+ if (HivePartitionUtil.partitionExists(client, tableName, dropPartition.path,
+ dropPartition.values, config)) {
+ client.dropPartition(databaseName, tableName, dropPartition.clause, false);
+ dropped++;
+ }
+ // Per-partition detail stays at debug: a batch can hold thousands of partitions
+ // and N workers log concurrently, so INFO carries the per-batch summary instead.
+ log.debug("Dropped partition {} on {}", dropPartition.path, tableName);
+ }
+ log.info("Dropped {} of {} partitions in batch on {}", dropped, batch.size(), tableName);
+ }
+
+ /**
+ * A partition to drop with its extractor-derived values already resolved, so worker
+ * threads never touch the shared {@link PartitionValueExtractor}. Immutable: the
+ * {@code values} list is wrapped unmodifiable at construction.
+ */
+ private static final class PartitionToDrop {
+ private final String path;
+ private final List<String> values;
+ private final String clause;
+
+ private PartitionToDrop(String path, List<String> values, String clause) {
+ this.path = path;
+ this.values = Collections.unmodifiableList(values);
Review Comment:
Good catch — fixed in 749d6719, using exactly the form you suggested:
```java
this.values = Collections.unmodifiableList(new ArrayList<>(values));
```
This one is squarely my miss. The whole point of that change was to make the DROP path safe against a *stateful* extractor, and I guarded the concurrency angle while leaving the aliasing one wide open — `unmodifiableList` protects the caller from mutating the list, not the list from being mutated underneath us.
Reproduced before fixing, with an extractor that clears and refills one list per call across four partitions:
```
each partition must be checked with the values extracted for it
==> expected: <[2026-08-01, 2026-08-02, 2026-08-03, 2026-08-04]>
but was: <[2026-08-04]>
```
All four collapsed onto the final extraction, so `partitionExists` checked `2026-08-04` four times — three valid drops silently skipped, exactly as you described. Covered by `extractorReusingOneBufferStillYieldsPerPartitionValues`.
I also swept the other collections crossing a boundary in this PR while I was here: `HiveDriverPool.workers`, `HiveMetaStoreClientPool.all`, and `ParallelDispatch.futures` all already copy at construction (`new ArrayList<>(...)`). `PartitionToDrop` was the only one wrapping without copying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727604261
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `3.08483%` with `377 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.12%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 0.00% | [137 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 0.00% | [68 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 0.00% | [67 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 0.00% | [28 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 22 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|6|
>|spark-scala-tests|12|4|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
=============================================
- Coverage 68.26% 46.12% -22.15%
+ Complexity 29513 16902 -12611
=============================================
Files 2542 2087 -455
Lines 142627 120973 -21654
Branches 17788 16034 -1754
=============================================
- Hits 97369 55802 -41567
- Misses 37253 59046 +21793
+ Partials 8005 6125 -1880
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `30.80% <3.08%> (-17.97%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.55% <3.08%> (-12.29%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (-86.05%)` | :arrow_down: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-48.93%)` | :arrow_down: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-80.35%)` | :arrow_down: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-65.68%)` | :arrow_down: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
... and [1644 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727586720
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `3.08483%` with `377 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.75%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 0.00% | [137 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 0.00% | [68 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 0.00% | [67 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 0.00% | [28 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 26 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|3|
>|spark-scala-tests|12|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
=============================================
- Coverage 68.26% 46.75% -21.52%
+ Complexity 29513 16798 -12715
=============================================
Files 2542 2066 -476
Lines 142627 118408 -24219
Branches 17788 15488 -2300
=============================================
- Hits 97369 55357 -42012
- Misses 37253 56984 +19731
+ Partials 8005 6067 -1938
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.88% <3.08%> (-16.89%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.50% <3.08%> (-12.34%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (-86.05%)` | :arrow_down: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-48.93%)` | :arrow_down: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-80.35%)` | :arrow_down: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-65.68%)` | :arrow_down: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
... and [1653 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5067871529
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5099375696
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7610c51a4e727c97110e3dadba9868c4fff7aa0f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5175609702
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6c81322202e4b6cb251248da8a0d8c478fb3fe26 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787)
* 3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5110593824
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7610c51a4e727c97110e3dadba9868c4fff7aa0f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748)
* 6c81322202e4b6cb251248da8a0d8c478fb3fe26 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4767677658
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.65%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 51 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.65% -0.62%
- Complexity 29513 29865 +352
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98428 +1059
- Misses 37253 38828 +1575
- Partials 8005 8234 +229
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.15% <3.08%> (-0.62%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.37% <7.45%> (-0.48%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [156 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727614821
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.60%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 19 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|7|
>|spark-scala-tests|12|4|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 59.60% -8.67%
+ Complexity 29513 26123 -3390
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
- Hits 97369 86717 -10652
- Misses 37253 51274 +14021
+ Partials 8005 7499 -506
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `30.80% <3.08%> (-17.97%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.55% <3.08%> (-12.29%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `49.87% <67.85%> (+0.94%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [667 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727774691
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.62%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.62% -0.65%
- Complexity 29513 29853 +340
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98390 +1021
- Misses 37253 38882 +1629
- Partials 8005 8218 +213
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.94% <3.08%> (-0.84%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.37% <7.45%> (-0.48%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [161 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5219216980
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034)
* 97dc86a59f300dfdc692a725131481afba3bfc59 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5099111905
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684)
* 7610c51a4e727c97110e3dadba9868c4fff7aa0f UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5261340570
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
}, {
"hash" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187",
"triggerID" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "749d6719655c7fef7abd0051a0a6eeebfc4f676c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "749d6719655c7fef7abd0051a0a6eeebfc4f676c",
"triggerType" : "PUSH"
}, {
"hash" : "a9d94436b63766f92a78c8d8bdb0ee3d23133b46",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "a9d94436b63766f92a78c8d8bdb0ee3d23133b46",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 038e6c842e18130a64552da6ff5aef46864b3cf0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187)
* 749d6719655c7fef7abd0051a0a6eeebfc4f676c UNKNOWN
* a9d94436b63766f92a78c8d8bdb0ee3d23133b46 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727802173
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.64%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.64% -0.63%
- Complexity 29513 29858 +345
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98411 +1042
- Misses 37253 38857 +1604
- Partials 8005 8222 +217
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.99% <3.08%> (-0.79%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.37% <7.45%> (-0.48%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [160 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727751580
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.62%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.62% -0.65%
- Complexity 29513 29853 +340
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98387 +1018
- Misses 37253 38884 +1631
- Partials 8005 8219 +214
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.88% <3.08%> (-0.89%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.37% <7.45%> (-0.48%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [163 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727729374
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.48%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.48% -0.79%
- Complexity 29513 29762 +249
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98179 +810
- Misses 37253 39125 +1872
- Partials 8005 8186 +181
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.82% <3.08%> (-0.96%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.11% <7.45%> (-0.74%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [186 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3737109998
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/IMetaStoreClientPool.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Pool of {@link IMetaStoreClient} instances for parallel partition sync.
+ *
+ * <p>Each pooled client wraps an independent Thrift connection to the Hive Metastore.
+ * Callers borrow a client via {@link #run(ClientAction)}, which blocks until a client
+ * is available, executes the action, and returns the client to the pool. A worker
+ * thread pool of the same size is exposed via {@link #executor()} so callers can fan
+ * out their batches to match the number of available clients.
+ *
+ * <p><b>Usage contract:</b> pool clients must be used <i>only</i> for partition-row
+ * operations — {@code add_partitions}, {@code alter_partitions}, {@code dropPartition},
+ * {@code getPartition}. Table-row operations ({@code createTable}, {@code alter_table},
+ * {@code getTable} used as the read half of a read-modify-write of table parameters)
+ * must continue to go through the session client held by
+ * {@code HoodieHiveSyncClient.client} on the sync driver thread. Mixing the two would
+ * risk lost updates on table parameters such as the last-commit-time-synced marker.
+ *
+ * <p>The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled} and
+ * is constructed for sync mode HIVEQL, where it backs the DROP path only. DROP goes
+ * through {@code IMetaStoreClient.dropPartition} (Thrift), whereas ADD/UPDATE/TOUCH go
+ * through the thread-bound Hive {@code Driver} and use {@code HiveDriverPool} instead.
Review Comment:
🤖 nit: could you rename this to `HiveMetaStoreClientPool` (or `MetaStoreClientPool`)? In the Hive ecosystem the `I` prefix signals an interface (as in `IMetaStoreClient` itself), so a concrete class named `IMetaStoreClientPool` reads at first glance as if it's an interface — a future reader will likely do a double-take before realising it's a pool implementation.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveDriverPool.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+import org.apache.hudi.hive.HoodieHiveSyncException;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.Driver;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_DATABASE_NAME;
+
+/**
+ * Pool of Hive {@link Driver} + {@link SessionState} pairs for parallel HiveQL DDL.
+ *
+ * <p>Hive's {@code SessionState.start(state)} binds state to the calling thread's
+ * thread-local, and {@code Driver} reads from that thread-local during {@code run()}.
+ * A Driver constructed on one thread cannot be safely used from another. This pool
+ * solves that by giving each slot its own dedicated worker thread (a single-thread
+ * executor) — the Driver and SessionState are built on that thread by a bootstrap
+ * task, and all subsequent SQL for that slot runs on the same thread.
+ *
+ * <p><b>Usage contract:</b> use this pool only for partition-row DDL statements that
+ * are independent of each other and freely shuffleable across workers. Table-level
+ * statements (createTable, schema evolution, USE database) must continue to run on
+ * the session {@code Driver} held by {@code HiveQueryDDLExecutor} on the sync driver
+ * thread. The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled}
+ * and is constructed only for HiveQL sync mode.
+ */
+public class HiveDriverPool implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveDriverPool.class);
+
+ // Per-worker Driver construction has to be fast in practice (a few hundred ms
+ // for the SessionState + Driver init). A 60s ceiling per worker leaves plenty of
+ // headroom for a slow JVM warm-up but bounds the failure mode if the metastore
+ // is unreachable or Hive hangs during init.
+ private static final long BOOTSTRAP_TIMEOUT_SECONDS = 60;
+
+ private final List<Worker> workers;
+ private final int size;
+ private volatile boolean closed;
+
+ public HiveDriverPool(HiveSyncConfig config, int size) {
+ this(config, size, new DefaultDriverFactory(config));
+ }
+
+ // Package-private for tests: accepts a DriverFactory so unit tests can inject
+ // mock Driver instances without standing up a real Hive instance.
+ HiveDriverPool(HiveSyncConfig config, int size, DriverFactory factory) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ this.size = size;
+ this.workers = new ArrayList<>(size);
+ String databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
+ PoolThreadFactory threadFactory = new PoolThreadFactory();
+ List<Future<Void>> bootstrapFutures = new ArrayList<>(size);
+ try {
+ for (int i = 0; i < size; i++) {
+ Worker worker = new Worker(threadFactory);
+ workers.add(worker);
+ bootstrapFutures.add(worker.executor.submit(() -> {
+ worker.driver = factory.newDriver(databaseName);
+ return null;
+ }));
+ }
+ // Block until all bootstraps complete so we surface construction errors
+ // before any caller hands us SQL. Bounded by BOOTSTRAP_TIMEOUT_SECONDS so a
+ // hung Hive init doesn't deadlock the sync driver thread.
+ for (Future<Void> f : bootstrapFutures) {
+ f.get(BOOTSTRAP_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ }
+ } catch (Exception e) {
+ tearDown();
+ throw new HoodieException("Failed to construct HiveDriverPool of size " + size, e);
+ }
+ LOG.info("Initialized HiveDriverPool with {} workers", size);
+ }
+
+ /**
+ * Runs each given SQL on <i>every</i> worker, in order. Used for setup statements
+ * (e.g. {@code USE database}) that must establish per-thread session context
+ * before any partition statement runs. Blocks until all workers have completed
+ * the setup. Throws on first error.
+ */
+ public void runOnEachWorker(List<String> setupSqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ if (setupSqls.isEmpty()) {
+ return;
+ }
+ List<Future<?>> futures = new ArrayList<>(workers.size());
+ for (Worker worker : workers) {
+ futures.add(worker.executor.submit(() -> {
+ for (String sql : setupSqls) {
+ worker.driver.run(sql);
+ }
+ return null;
+ }));
+ }
+ awaitAll(futures);
+ }
+
+ /**
+ * Dispatches each SQL string to a worker (round-robin) and returns the list of
+ * futures. The caller is responsible for awaiting and collecting errors. SQL text
+ * is intentionally not logged per-statement here: batched TOUCH/ADD statements can
+ * be many kilobytes, and N parallel workers would multiply the log volume. See
+ * {@link #awaitAll(List)} for the per-call summary log.
+ */
+ public List<Future<?>> runAll(List<String> sqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ List<Future<?>> futures = new ArrayList<>(sqls.size());
+ for (int i = 0; i < sqls.size(); i++) {
+ String sql = sqls.get(i);
+ Worker worker = workers.get(i % workers.size());
+ futures.add(worker.executor.submit(() -> {
+ worker.driver.run(sql);
+ return null;
+ }));
+ }
+ return futures;
+ }
+
+ /**
+ * Awaits all futures and throws the first exception encountered. On first failure,
+ * cancels the remaining (not yet started) futures so workers don't keep running
+ * pointless work after a fatal error. Any errors that finished before cancellation
+ * are logged at WARN. Callers do not need per-statement results (Hive's Driver.run
+ * side-effects the metastore), so this method is void.
+ */
+ public void awaitAll(List<Future<?>> futures) {
+ long start = System.currentTimeMillis();
+ Exception firstError = null;
+ int completed = 0;
+ int cancelled = 0;
+ for (int i = 0; i < futures.size(); i++) {
+ Future<?> f = futures.get(i);
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // We cancelled this future ourselves after a prior error. Don't treat it
+ // as a new failure; just note it for the summary log.
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ cancelled += cancelRemaining(futures, i + 1);
+ }
+ } catch (ExecutionException ee) {
+ Exception cause = unwrap(ee);
+ if (firstError == null) {
+ firstError = cause;
+ cancelled += cancelRemaining(futures, i + 1);
+ } else {
+ LOG.warn("Additional SQL batch failed (suppressed in favor of first error)", cause);
+ }
+ }
+ }
+ if (firstError != null) {
+ throw new HoodieHiveSyncException("Failed in executing SQL", firstError);
+ }
+ LOG.info("Completed {} SQL statements ({} cancelled) in {} ms across {} workers",
+ completed, cancelled, System.currentTimeMillis() - start, size);
+ }
+
+ private static int cancelRemaining(List<Future<?>> futures, int fromIndex) {
+ int cancelled = 0;
+ for (int j = fromIndex; j < futures.size(); j++) {
+ // mayInterruptIfRunning=false: the worker thread is bound to a Hive Driver
+ // whose state we don't want to corrupt mid-statement. Cancel only those that
+ // haven't started yet; in-flight statements run to completion.
+ if (futures.get(j).cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ private static Exception unwrap(ExecutionException ee) {
+ Throwable cause = ee.getCause();
+ return (cause instanceof Exception) ? (Exception) cause : ee;
+ }
+
+ public int size() {
+ return size;
+ }
+
+ @Override
+ public void close() {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ tearDown();
+ }
+
+ private void tearDown() {
+ // Close each worker's Driver/SessionState on its own thread, then shut the
+ // executor down. Running close() on the bound thread keeps SessionState's
+ // thread-local cleanup correct.
+ for (Worker worker : workers) {
+ try {
+ worker.executor.submit(() -> {
+ if (worker.driver != null) {
+ try {
+ worker.driver.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled Driver", e);
+ }
+ }
+ SessionState ss = SessionState.get();
+ if (ss != null) {
+ try {
+ ss.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled SessionState", e);
+ }
+ }
+ return null;
+ }).get(30, TimeUnit.SECONDS);
+ } catch (Exception e) {
+ LOG.warn("Error during pool worker shutdown", e);
+ }
+ worker.executor.shutdown();
+ try {
+ if (!worker.executor.awaitTermination(10, TimeUnit.SECONDS)) {
+ worker.executor.shutdownNow();
+ }
+ } catch (InterruptedException ie) {
+ worker.executor.shutdownNow();
+ Thread.currentThread().interrupt();
+ }
+ }
+ workers.clear();
+ }
+
+ /**
+ * Per-slot state: a single-thread executor and the Driver bound to its thread.
+ * Driver is volatile because it is written by the bootstrap task and read by
+ * subsequent dispatch tasks on the same executor.
+ */
+ private static final class Worker {
+ final ExecutorService executor;
+ volatile Driver driver;
+
+ Worker(ThreadFactory threadFactory) {
+ this.executor = Executors.newSingleThreadExecutor(threadFactory);
+ }
+ }
+
+ @FunctionalInterface
+ interface DriverFactory {
+ Driver newDriver(String databaseName) throws Exception;
+ }
+
+ /**
+ * Builds a real Hive {@link Driver} on the calling thread. The SessionState is
+ * constructed lazily (once, on the first worker thread that builds a Driver) and
+ * shared across all worker threads — Hive uses ThreadLocal attachment, not
+ * exclusive ownership, so multiple workers calling
+ * {@code SessionState.start(sharedState)} all see the same config and scratch dir
+ * without each spending the cost of building their own SessionState (and risking
+ * resource-dir creation races during the constructor).
+ */
+ private static final class DefaultDriverFactory implements DriverFactory {
+ private final HiveConf hiveConf;
+ private volatile SessionState sharedSessionState;
+
+ DefaultDriverFactory(HiveSyncConfig config) {
+ this.hiveConf = config.getHiveConf();
+ }
+
+ @Override
+ public synchronized Driver newDriver(String databaseName) throws Exception {
+ // SessionState is shared across workers; build it once on the first call (with
+ // currentDatabase already set) and attach it to each worker's thread-local on
+ // subsequent calls. The database is a pool-wide property and never changes
+ // across workers, so setting it once at construction time is sufficient.
Review Comment:
🤖 One scoping note that might help focus this: the shared-SessionState code lives in `HiveDriverPool` and isn't actually touched by this PR — `gh pr diff` shows no SessionState delta here; it came in with #18984 for the Driver-based ADD/UPDATE/TOUCH path. This change is DROP-only, and DROP routes through `IMetaStoreClientPool` (Thrift `IMetaStoreClient.dropPartition`, via `runDropBatches` → `pool.awaitAll`), which never installs or mutates a `SessionState`. So the concurrent-SessionState question doesn't apply to the DROP path being added here — if it's a real concern for the Driver path, it'd probably be better raised against #18984 rather than blocking this diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5261482248
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
}, {
"hash" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187",
"triggerID" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "749d6719655c7fef7abd0051a0a6eeebfc4f676c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "749d6719655c7fef7abd0051a0a6eeebfc4f676c",
"triggerType" : "PUSH"
}, {
"hash" : "a9d94436b63766f92a78c8d8bdb0ee3d23133b46",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16228",
"triggerID" : "a9d94436b63766f92a78c8d8bdb0ee3d23133b46",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 038e6c842e18130a64552da6ff5aef46864b3cf0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187)
* 749d6719655c7fef7abd0051a0a6eeebfc4f676c UNKNOWN
* a9d94436b63766f92a78c8d8bdb0ee3d23133b46 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16228)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5262150231
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
}, {
"hash" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187",
"triggerID" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "749d6719655c7fef7abd0051a0a6eeebfc4f676c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "749d6719655c7fef7abd0051a0a6eeebfc4f676c",
"triggerType" : "PUSH"
}, {
"hash" : "a9d94436b63766f92a78c8d8bdb0ee3d23133b46",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16228",
"triggerID" : "a9d94436b63766f92a78c8d8bdb0ee3d23133b46",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 749d6719655c7fef7abd0051a0a6eeebfc4f676c UNKNOWN
* a9d94436b63766f92a78c8d8bdb0ee3d23133b46 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16228)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan merged PR #19033:
URL: https://github.com/apache/hudi/pull/19033
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5066935749
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e3c1fdec6c9dd95e339a2027ca5be20002dea8c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894)
* 9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727633008
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 60.10%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 17 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|7|
>|spark-scala-tests|12|6|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 60.10% -8.17%
+ Complexity 29513 26445 -3068
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
- Hits 97369 87445 -9924
- Misses 37253 50328 +13075
+ Partials 8005 7717 -288
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `30.80% <3.08%> (-17.97%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.38% <3.08%> (-12.47%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `49.87% <67.85%> (+0.94%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [659 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727296975
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7022e7d493416d600664235c3056d32ea4a1c179 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727617155
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.89%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 18 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|7|
>|spark-scala-tests|12|5|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 59.89% -8.38%
+ Complexity 29513 26300 -3213
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
- Hits 97369 87141 -10228
- Misses 37253 50738 +13485
+ Partials 8005 7611 -394
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `30.80% <3.08%> (-17.97%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.54% <3.08%> (-12.31%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `49.87% <67.85%> (+0.94%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [661 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727603118
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `3.08483%` with `377 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.12%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 0.00% | [137 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 0.00% | [68 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 0.00% | [67 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 0.00% | [28 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 22 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|6|
>|spark-scala-tests|12|4|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
=============================================
- Coverage 68.26% 46.12% -22.15%
+ Complexity 29513 16902 -12611
=============================================
Files 2542 2087 -455
Lines 142627 120973 -21654
Branches 17788 16034 -1754
=============================================
- Hits 97369 55802 -41567
- Misses 37253 59046 +21793
+ Partials 8005 6125 -1880
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `30.80% <3.08%> (-17.97%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.55% <3.08%> (-12.29%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (-86.05%)` | :arrow_down: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-48.93%)` | :arrow_down: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-80.35%)` | :arrow_down: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-65.68%)` | :arrow_down: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
... and [1644 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3662307807
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -209,21 +219,96 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
- }
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(partitionsToDrop, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ *
+ * <p>First-error semantics match {@code HiveDriverPool.awaitAll}: the first failure is
+ * rethrown, not-yet-started batches are cancelled, and later failures are logged at WARN.
+ */
+ private void runDropBatches(String tableName, List<List<String>> batches) throws Exception {
+ if (!metaStoreClientPool.isPresent()) {
+ for (List<String> batch : batches) {
+ applyDropBatch(metaStoreClient, tableName, batch);
+ }
+ return;
+ }
+ IMetaStoreClientPool pool = metaStoreClientPool.get();
+ List<Future<Void>> futures = new ArrayList<>(batches.size());
+ for (List<String> batch : batches) {
+ futures.add(pool.executor().submit(() ->
+ pool.run(poolClient -> {
+ applyDropBatch(poolClient, tableName, batch);
+ return null;
+ })
+ ));
+ }
+
+ Exception firstError = null;
+ int cancelled = 0;
+ for (Future<Void> f : futures) {
+ // Once a batch has failed the sync is going to abort anyway, so stop handing
+ // more DROPs to the metastore. mayInterruptIfRunning=false mirrors
+ // HiveDriverPool.awaitAll: a batch already mid-flight runs to completion rather
+ // than leaving the partition list half-dropped at an arbitrary point.
+ if (firstError != null && f.cancel(false)) {
+ cancelled++;
+ continue;
+ }
+ try {
+ f.get();
+ } catch (CancellationException ce) {
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ }
+ } catch (Exception e) {
+ if (firstError == null) {
+ firstError = e;
+ } else {
+ log.warn("Additional drop batch failed on {} (suppressed in favor of first error)", tableName, e);
+ }
+ }
+ }
+ if (firstError != null) {
+ log.error("Drop partition dispatch on {} aborted after first failure ({} batches cancelled)",
+ tableName, cancelled);
+ throw firstError;
+ }
+ }
+
+ private void applyDropBatch(IMetaStoreClient poolClient, String tableName, List<String> batch) throws Exception {
+ int dropped = 0;
Review Comment:
🤖 nit: the parameter is named `poolClient` but the sequential code path calls this same method with the plain session `metaStoreClient` — not a pool client. Something like `client` would read correctly in both call sites.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -227,6 +241,22 @@ private IMetaStoreClient createMetaStoreClient(HiveSyncConfig config) {
}
}
+ private Option<IMetaStoreClientPool> maybeBuildPartitionClientPool(HiveSyncConfig config) {
+ if (!config.getBooleanOrDefault(HIVE_SYNC_BATCHING_ENABLED)) {
+ return Option.empty();
+ }
+ if (config.getBooleanOrDefault(HIVE_SYNC_USE_SPARK_CATALOG)) {
+ // The Spark catalog client is constructed via reflection against a Spark-side
Review Comment:
🤖 nit: the config key names in this warning (`hive_sync.batching.enabled`, `use_spark_catalog`) don't match the actual property keys an operator would search for in logs. Could you use the real constant's `.key()` value, e.g. `HIVE_SYNC_BATCHING_ENABLED.key() + "=true is not supported with " + HIVE_SYNC_USE_SPARK_CATALOG.key() + "=true"`?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveDriverPool.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+import org.apache.hudi.hive.HoodieHiveSyncException;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.Driver;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_DATABASE_NAME;
+
+/**
+ * Pool of Hive {@link Driver} + {@link SessionState} pairs for parallel HiveQL DDL.
+ *
+ * <p>Hive's {@code SessionState.start(state)} binds state to the calling thread's
+ * thread-local, and {@code Driver} reads from that thread-local during {@code run()}.
+ * A Driver constructed on one thread cannot be safely used from another. This pool
+ * solves that by giving each slot its own dedicated worker thread (a single-thread
+ * executor) — the Driver and SessionState are built on that thread by a bootstrap
+ * task, and all subsequent SQL for that slot runs on the same thread.
+ *
+ * <p><b>Usage contract:</b> use this pool only for partition-row DDL statements that
+ * are independent of each other and freely shuffleable across workers. Table-level
+ * statements (createTable, schema evolution, USE database) must continue to run on
+ * the session {@code Driver} held by {@code HiveQueryDDLExecutor} on the sync driver
+ * thread. The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled}
+ * and is constructed only for HiveQL sync mode.
+ */
+public class HiveDriverPool implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveDriverPool.class);
+
+ // Per-worker Driver construction has to be fast in practice (a few hundred ms
+ // for the SessionState + Driver init). A 60s ceiling per worker leaves plenty of
+ // headroom for a slow JVM warm-up but bounds the failure mode if the metastore
+ // is unreachable or Hive hangs during init.
+ private static final long BOOTSTRAP_TIMEOUT_SECONDS = 60;
+
+ private final List<Worker> workers;
+ private final int size;
+ private volatile boolean closed;
+
+ public HiveDriverPool(HiveSyncConfig config, int size) {
+ this(config, size, new DefaultDriverFactory(config));
+ }
+
+ // Package-private for tests: accepts a DriverFactory so unit tests can inject
+ // mock Driver instances without standing up a real Hive instance.
+ HiveDriverPool(HiveSyncConfig config, int size, DriverFactory factory) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ this.size = size;
+ this.workers = new ArrayList<>(size);
+ String databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
+ PoolThreadFactory threadFactory = new PoolThreadFactory();
+ List<Future<Void>> bootstrapFutures = new ArrayList<>(size);
+ try {
+ for (int i = 0; i < size; i++) {
+ Worker worker = new Worker(threadFactory);
+ workers.add(worker);
+ bootstrapFutures.add(worker.executor.submit(() -> {
+ worker.driver = factory.newDriver(databaseName);
+ return null;
+ }));
+ }
+ // Block until all bootstraps complete so we surface construction errors
+ // before any caller hands us SQL. Bounded by BOOTSTRAP_TIMEOUT_SECONDS so a
+ // hung Hive init doesn't deadlock the sync driver thread.
+ for (Future<Void> f : bootstrapFutures) {
+ f.get(BOOTSTRAP_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ }
+ } catch (Exception e) {
+ tearDown();
+ throw new HoodieException("Failed to construct HiveDriverPool of size " + size, e);
+ }
+ LOG.info("Initialized HiveDriverPool with {} workers", size);
+ }
+
+ /**
+ * Runs each given SQL on <i>every</i> worker, in order. Used for setup statements
+ * (e.g. {@code USE database}) that must establish per-thread session context
+ * before any partition statement runs. Blocks until all workers have completed
+ * the setup. Throws on first error.
+ */
+ public void runOnEachWorker(List<String> setupSqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ if (setupSqls.isEmpty()) {
+ return;
+ }
+ List<Future<?>> futures = new ArrayList<>(workers.size());
+ for (Worker worker : workers) {
+ futures.add(worker.executor.submit(() -> {
+ for (String sql : setupSqls) {
+ worker.driver.run(sql);
+ }
+ return null;
+ }));
+ }
+ awaitAll(futures);
+ }
+
+ /**
+ * Dispatches each SQL string to a worker (round-robin) and returns the list of
+ * futures. The caller is responsible for awaiting and collecting errors. SQL text
+ * is intentionally not logged per-statement here: batched TOUCH/ADD statements can
+ * be many kilobytes, and N parallel workers would multiply the log volume. See
+ * {@link #awaitAll(List)} for the per-call summary log.
+ */
+ public List<Future<?>> runAll(List<String> sqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ List<Future<?>> futures = new ArrayList<>(sqls.size());
+ for (int i = 0; i < sqls.size(); i++) {
+ String sql = sqls.get(i);
+ Worker worker = workers.get(i % workers.size());
+ futures.add(worker.executor.submit(() -> {
+ worker.driver.run(sql);
+ return null;
+ }));
+ }
+ return futures;
+ }
+
+ /**
+ * Awaits all futures and throws the first exception encountered. On first failure,
+ * cancels the remaining (not yet started) futures so workers don't keep running
+ * pointless work after a fatal error. Any errors that finished before cancellation
+ * are logged at WARN. Callers do not need per-statement results (Hive's Driver.run
+ * side-effects the metastore), so this method is void.
+ */
+ public void awaitAll(List<Future<?>> futures) {
+ long start = System.currentTimeMillis();
+ Exception firstError = null;
+ int completed = 0;
+ int cancelled = 0;
+ for (int i = 0; i < futures.size(); i++) {
+ Future<?> f = futures.get(i);
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // We cancelled this future ourselves after a prior error. Don't treat it
+ // as a new failure; just note it for the summary log.
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ cancelled += cancelRemaining(futures, i + 1);
+ }
+ } catch (ExecutionException ee) {
+ Exception cause = unwrap(ee);
+ if (firstError == null) {
+ firstError = cause;
+ cancelled += cancelRemaining(futures, i + 1);
+ } else {
+ LOG.warn("Additional SQL batch failed (suppressed in favor of first error)", cause);
+ }
+ }
+ }
+ if (firstError != null) {
+ throw new HoodieHiveSyncException("Failed in executing SQL", firstError);
+ }
+ LOG.info("Completed {} SQL statements ({} cancelled) in {} ms across {} workers",
+ completed, cancelled, System.currentTimeMillis() - start, size);
+ }
+
+ private static int cancelRemaining(List<Future<?>> futures, int fromIndex) {
+ int cancelled = 0;
+ for (int j = fromIndex; j < futures.size(); j++) {
+ // mayInterruptIfRunning=false: the worker thread is bound to a Hive Driver
+ // whose state we don't want to corrupt mid-statement. Cancel only those that
+ // haven't started yet; in-flight statements run to completion.
+ if (futures.get(j).cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ private static Exception unwrap(ExecutionException ee) {
+ Throwable cause = ee.getCause();
+ return (cause instanceof Exception) ? (Exception) cause : ee;
+ }
+
+ public int size() {
+ return size;
+ }
+
+ @Override
+ public void close() {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ tearDown();
+ }
+
+ private void tearDown() {
+ // Close each worker's Driver/SessionState on its own thread, then shut the
+ // executor down. Running close() on the bound thread keeps SessionState's
+ // thread-local cleanup correct.
+ for (Worker worker : workers) {
+ try {
+ worker.executor.submit(() -> {
+ if (worker.driver != null) {
+ try {
+ worker.driver.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled Driver", e);
+ }
+ }
+ SessionState ss = SessionState.get();
+ if (ss != null) {
+ try {
+ ss.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled SessionState", e);
+ }
+ }
+ return null;
+ }).get(30, TimeUnit.SECONDS);
+ } catch (Exception e) {
+ LOG.warn("Error during pool worker shutdown", e);
+ }
+ worker.executor.shutdown();
+ try {
+ if (!worker.executor.awaitTermination(10, TimeUnit.SECONDS)) {
+ worker.executor.shutdownNow();
+ }
+ } catch (InterruptedException ie) {
+ worker.executor.shutdownNow();
+ Thread.currentThread().interrupt();
+ }
+ }
+ workers.clear();
+ }
+
+ /**
+ * Per-slot state: a single-thread executor and the Driver bound to its thread.
+ * Driver is volatile because it is written by the bootstrap task and read by
+ * subsequent dispatch tasks on the same executor.
+ */
+ private static final class Worker {
+ final ExecutorService executor;
+ volatile Driver driver;
+
+ Worker(ThreadFactory threadFactory) {
+ this.executor = Executors.newSingleThreadExecutor(threadFactory);
+ }
+ }
+
+ @FunctionalInterface
+ interface DriverFactory {
+ Driver newDriver(String databaseName) throws Exception;
+ }
+
+ /**
+ * Builds a real Hive {@link Driver} on the calling thread. The SessionState is
+ * constructed lazily (once, on the first worker thread that builds a Driver) and
+ * shared across all worker threads — Hive uses ThreadLocal attachment, not
+ * exclusive ownership, so multiple workers calling
+ * {@code SessionState.start(sharedState)} all see the same config and scratch dir
+ * without each spending the cost of building their own SessionState (and risking
+ * resource-dir creation races during the constructor).
+ */
+ private static final class DefaultDriverFactory implements DriverFactory {
+ private final HiveConf hiveConf;
+ private volatile SessionState sharedSessionState;
+
+ DefaultDriverFactory(HiveSyncConfig config) {
+ this.hiveConf = config.getHiveConf();
+ }
+
+ @Override
+ public synchronized Driver newDriver(String databaseName) throws Exception {
+ // SessionState is shared across workers; build it once on the first call (with
+ // currentDatabase already set) and attach it to each worker's thread-local on
+ // subsequent calls. The database is a pool-wide property and never changes
+ // across workers, so setting it once at construction time is sufficient.
Review Comment:
🤖 In the current HiveDriverPool design each worker actually gets its own SessionState rather than sharing one: `DefaultDriverFactory.newDriver()` does `new SessionState(new HiveConf(hiveConf), ...)` + `SessionState.start(...)` on the worker's own single-thread executor, and every statement for that slot runs pinned to that one thread (the Driver/SessionState are never touched from another thread). Each worker also gets its own HiveConf copy specifically so per-query keys like HIVEQUERYID don't collide. So there's no shared SessionState across threads here, and the concurrent USE/conf-mutation concern shouldn't apply — the per-worker isolation you suggested is what the code does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5261177155
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
}, {
"hash" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187",
"triggerID" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"triggerType" : "PUSH"
}, {
"hash" : "749d6719655c7fef7abd0051a0a6eeebfc4f676c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "749d6719655c7fef7abd0051a0a6eeebfc4f676c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 038e6c842e18130a64552da6ff5aef46864b3cf0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187)
* 749d6719655c7fef7abd0051a0a6eeebfc4f676c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4783462576
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7022e7d493416d600664235c3056d32ea4a1c179 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784)
* 5e3c1fdec6c9dd95e339a2027ca5be20002dea8c UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3662480717
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -227,6 +241,22 @@ private IMetaStoreClient createMetaStoreClient(HiveSyncConfig config) {
}
}
+ private Option<IMetaStoreClientPool> maybeBuildPartitionClientPool(HiveSyncConfig config) {
+ if (!config.getBooleanOrDefault(HIVE_SYNC_BATCHING_ENABLED)) {
+ return Option.empty();
+ }
+ if (config.getBooleanOrDefault(HIVE_SYNC_USE_SPARK_CATALOG)) {
+ // The Spark catalog client is constructed via reflection against a Spark-side
+ // class and isn't compatible with the direct RetryingMetaStoreClient pool path.
+ // Fall back to single-client sequential behavior rather than failing the sync.
+ log.warn("hive_sync.batching.enabled=true is not supported with use_spark_catalog=true; "
+ + "falling back to sequential partition sync.");
+ return Option.empty();
+ }
+ int size = config.getIntOrDefault(HIVE_SYNC_BATCHING_THREADS);
+ return Option.of(new IMetaStoreClientPool(config, size));
Review Comment:
**[P2] Update the public batching config contract for DROP**
`HIVE_SYNC_BATCHING_ENABLED` in `HiveSyncConfigHolder` still documents that "DROP remains serial," but this pool makes DROP parallel under that exact flag. That text is the user-facing configuration contract, so it currently tells operators the opposite of the new behavior. Please update it to cover the Thrift-client DROP fan-out (including the Spark-catalog fallback) as part of this change.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -209,21 +219,96 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
- }
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(partitionsToDrop, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ *
+ * <p>First-error semantics match {@code HiveDriverPool.awaitAll}: the first failure is
+ * rethrown, not-yet-started batches are cancelled, and later failures are logged at WARN.
+ */
+ private void runDropBatches(String tableName, List<List<String>> batches) throws Exception {
+ if (!metaStoreClientPool.isPresent()) {
+ for (List<String> batch : batches) {
+ applyDropBatch(metaStoreClient, tableName, batch);
+ }
+ return;
+ }
+ IMetaStoreClientPool pool = metaStoreClientPool.get();
+ List<Future<Void>> futures = new ArrayList<>(batches.size());
+ for (List<String> batch : batches) {
+ futures.add(pool.executor().submit(() ->
+ pool.run(poolClient -> {
+ applyDropBatch(poolClient, tableName, batch);
+ return null;
+ })
+ ));
+ }
+
+ Exception firstError = null;
+ int cancelled = 0;
+ for (Future<Void> f : futures) {
+ // Once a batch has failed the sync is going to abort anyway, so stop handing
+ // more DROPs to the metastore. mayInterruptIfRunning=false mirrors
+ // HiveDriverPool.awaitAll: a batch already mid-flight runs to completion rather
+ // than leaving the partition list half-dropped at an arbitrary point.
+ if (firstError != null && f.cancel(false)) {
+ cancelled++;
+ continue;
+ }
+ try {
+ f.get();
Review Comment:
**[P2] Stop queued DROP batches as soon as a worker fails**
This waits on futures in submission order. If a later batch fails quickly while an earlier batch is slow, this thread remains blocked on the earlier `f.get()`, and the shared executor can keep starting every queued batch because the tasks do not observe an abort flag. By the time `firstError` is set, most or all "not-yet-started" DROPs may already have run, so the advertised abort-on-first-error behavior is not achieved. Please mirror `HiveDriverPool.Dispatch` (shared task-side abort plus completion-order notification), or use a completion service/abort flag so pending batches are stopped immediately after any worker fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727822634
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.65%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.65% -0.62%
- Complexity 29513 29865 +352
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98428 +1059
- Misses 37253 38828 +1575
- Partials 8005 8234 +229
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.15% <3.08%> (-0.62%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.37% <7.45%> (-0.48%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [156 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727613565
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 59.60%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 19 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|7|
>|spark-scala-tests|12|4|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 59.60% -8.67%
+ Complexity 29513 26123 -3390
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
- Hits 97369 86717 -10652
- Misses 37253 51274 +14021
+ Partials 8005 7499 -506
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `30.80% <3.08%> (-17.97%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.55% <3.08%> (-12.29%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `49.87% <67.85%> (+0.94%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [667 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727569108
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `3.08483%` with `377 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 45.91%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 0.00% | [137 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 0.00% | [68 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 0.00% | [67 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 0.00% | [28 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 30 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|0|
>|spark-scala-tests|12|2|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
=============================================
- Coverage 68.26% 45.91% -22.36%
+ Complexity 29513 15507 -14006
=============================================
Files 2542 1997 -545
Lines 142627 110692 -31935
Branches 17788 13742 -4046
=============================================
- Hits 97369 50819 -46550
- Misses 37253 54364 +17111
+ Partials 8005 5509 -2496
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.64% <3.08%> (-12.21%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (-86.05%)` | :arrow_down: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-48.93%)` | :arrow_down: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-80.35%)` | :arrow_down: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-65.68%)` | :arrow_down: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
... and [1733 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5110905298
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6c81322202e4b6cb251248da8a0d8c478fb3fe26 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3710076654
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/ParallelDispatch.java:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.common.util.VisibleForTesting;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Handle to one fan-out batch of partition work: the submitted futures plus the shared
+ * abort flag the tasks consult before running.
+ *
+ * <p>The abort flag exists because waiting on futures in <i>submission</i> order is not
+ * enough to stop queued work. If a later task fails quickly while an earlier one is slow,
+ * the awaiting thread is still parked on the earlier {@code Future.get()}, and the
+ * executor happily keeps starting every queued task in the meantime. By the time the
+ * failure is observed, most of the "not-yet-started" work has already run.
+ *
+ * <p>Two mechanisms fix that, and both are needed:
+ * <ul>
+ * <li>a {@link CountDownLatch} tripped by the <i>first</i> abort, so the awaiting
+ * thread wakes on failure rather than on its turn in submission order;</li>
+ * <li>a task-side {@link #aborted()} check on entry, because cancelling from the
+ * awaiting thread is inherently late — a worker can pull its next task off the
+ * queue at any moment.</li>
+ * </ul>
+ *
+ * <p>Shared by {@link HiveDriverPool} (Hive {@code Driver} statements) and
+ * {@link IMetaStoreClientPool} (Thrift {@code dropPartition} batches), which fan out over
+ * different execution models but need identical abort-on-first-error semantics.
+ */
+public final class ParallelDispatch {
+
+ private final List<Future<?>> futures;
+ private final int total;
+ private final AtomicInteger settled = new AtomicInteger(0);
+ private final AtomicBoolean aborted = new AtomicBoolean(false);
+ private final CountDownLatch done = new CountDownLatch(1);
+ private volatile boolean sealed;
+
+ ParallelDispatch(int total) {
+ this.total = total;
+ this.futures = new ArrayList<>(total);
+ }
+
+ void add(Future<?> future) {
+ futures.add(future);
+ }
+
+ // Called once submission finishes. A task that settles before the last submit
+ // would otherwise see settled < total and never trip the latch, so re-check here.
+ void sealed() {
+ sealed = true;
+ signalIfComplete();
+ }
+
+ boolean aborted() {
+ return aborted.get();
+ }
+
+ void abort() {
+ aborted.set(true);
+ done.countDown();
+ }
+
+ void taskSettled() {
+ settled.incrementAndGet();
+ signalIfComplete();
+ }
+
+ private void signalIfComplete() {
+ if (sealed && settled.get() >= total) {
+ done.countDown();
+ }
+ }
+
+ void awaitSettledOrAborted() {
+ if (total == 0) {
+ return;
+ }
+ try {
+ done.await();
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ aborted.set(true);
+ }
+ }
+
+ // mayInterruptIfRunning=false: a worker may be mid-statement against a Hive Driver or
+ // a Thrift client, and we don't want to tear that down partway. Cancel only tasks that
+ // haven't started; in-flight work runs to completion.
+ int cancelPending() {
+ int cancelled = 0;
+ for (Future<?> f : futures) {
+ if (f.cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ List<Future<?>> futures() {
+ return futures;
+ }
+
+ /**
+ * Wraps {@code body} so it observes this batch's abort flag: it skips itself if a
+ * sibling has already failed, trips the flag if it fails, and always records that it
+ * settled so the awaiting thread can be released.
+ */
+ Callable<Void> guard(Callable<Void> body, String skipMessage) {
+ return () -> {
+ if (aborted()) {
+ throw new CancellationException(skipMessage);
+ }
+ try {
+ return body.call();
+ } catch (Throwable t) {
+ abort();
+ throw t;
+ } finally {
+ taskSettled();
+ }
+ };
+ }
+
+ /**
+ * Waits for the batch to settle (or abort), cancels whatever had not started, and
+ * returns the outcome. Errors are observed in <i>completion</i> order, not submission
+ * order, so a failure on a fast worker stops the other queues even while a slow worker
+ * is still mid-statement.
+ */
+ Outcome awaitOutcome() {
+ awaitSettledOrAborted();
+ int cancelled = cancelPending();
+
+ Exception firstError = null;
+ int completed = 0;
+ List<Exception> suppressed = new ArrayList<>();
+ for (Future<?> f : futures) {
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // Either we cancelled it before it started, or the task itself observed the
+ // abort flag and bailed. Not a new failure; just note it for the summary.
+ cancelled++;
Review Comment:
The double-count is real and reproduces exactly as you describe (`cancelled == 4` for one failure plus two never-started). I tried to fix it and am deliberately **not** shipping the fix in this PR, because every version I wrote traded a wrong log number for a swallowed exception. Details, since I think this is worth your eyes:
Counting in only one place looks safe — a future cancelled in `cancelPending()` also surfaces as `CancellationException` from `get()`, so dropping the first count should be lossless. But it removes the thing that was masking a pre-existing race in the wake protocol.
In `guard`:
```java
} catch (Throwable t) {
abort(t); // trips the latch -> awaitOutcome wakes here
throw t;
} finally {
taskSettled(); // has not run yet; the future is not yet complete
}
```
`abort()` counts down `done`, so `awaitSettledOrAborted()` returns while the failing task is still between its `catch` and its `finally`. `awaitOutcome` then calls `cancelPending()`, which cancels **the failing task's own future** before its result was ever collected. Measured on this branch, one failure plus two never-started:
```
current FAIL=EXEC:IllegalStateException cancelled=4 (wrong) failure preserved
naive fix FAIL=BARE_CANCEL cancelled=2 (right) failure destroyed
```
So today's over-count is exactly what keeps the real cause alive — `cancelPending()`'s tally papers over the future it just corrupted. With the count deduplicated, a genuine metastore error on a DROP batch is reported to the operator as a bare `CancellationException`.
I tried four variants — gating `signalIfComplete` on `sealed`, moving the wake into `taskSettled`, a separate task-side `skipped` counter, and an `isDone()` guard before `cancel(false)`. The best got the failure rate to ~1/120 in a 120-iteration stress run, but `isDone()` can only narrow the window, never close it: the task is mid-unwind, so `isDone()` is false right up until it is not.
A correct fix means the failing task must fully settle before the awaiting thread wakes — i.e. reworking the `abort()` / `taskSettled()` / `signalIfComplete()` protocol. That is the same machinery implementing the abort-on-first-error guarantee from your earlier review, so I would rather not rework it as a side effect of a log-counter fix in this PR. As it stands the cost is one wrong number in an INFO/ERROR line that nothing reads programmatically.
Happy to do it here if you would prefer, or as a focused follow-up — your call. Either way the race is worth recording, since it is latent in the current code and the obvious cleanup is what exposes it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3643376543
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveDriverPool.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+import org.apache.hudi.hive.HoodieHiveSyncException;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.Driver;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_DATABASE_NAME;
+
+/**
+ * Pool of Hive {@link Driver} + {@link SessionState} pairs for parallel HiveQL DDL.
+ *
+ * <p>Hive's {@code SessionState.start(state)} binds state to the calling thread's
+ * thread-local, and {@code Driver} reads from that thread-local during {@code run()}.
+ * A Driver constructed on one thread cannot be safely used from another. This pool
+ * solves that by giving each slot its own dedicated worker thread (a single-thread
+ * executor) — the Driver and SessionState are built on that thread by a bootstrap
+ * task, and all subsequent SQL for that slot runs on the same thread.
+ *
+ * <p><b>Usage contract:</b> use this pool only for partition-row DDL statements that
+ * are independent of each other and freely shuffleable across workers. Table-level
+ * statements (createTable, schema evolution, USE database) must continue to run on
+ * the session {@code Driver} held by {@code HiveQueryDDLExecutor} on the sync driver
+ * thread. The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled}
+ * and is constructed only for HiveQL sync mode.
+ */
+public class HiveDriverPool implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveDriverPool.class);
+
+ // Per-worker Driver construction has to be fast in practice (a few hundred ms
+ // for the SessionState + Driver init). A 60s ceiling per worker leaves plenty of
+ // headroom for a slow JVM warm-up but bounds the failure mode if the metastore
+ // is unreachable or Hive hangs during init.
+ private static final long BOOTSTRAP_TIMEOUT_SECONDS = 60;
+
+ private final List<Worker> workers;
+ private final int size;
+ private volatile boolean closed;
+
+ public HiveDriverPool(HiveSyncConfig config, int size) {
+ this(config, size, new DefaultDriverFactory(config));
+ }
+
+ // Package-private for tests: accepts a DriverFactory so unit tests can inject
+ // mock Driver instances without standing up a real Hive instance.
+ HiveDriverPool(HiveSyncConfig config, int size, DriverFactory factory) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ this.size = size;
+ this.workers = new ArrayList<>(size);
+ String databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
+ PoolThreadFactory threadFactory = new PoolThreadFactory();
+ List<Future<Void>> bootstrapFutures = new ArrayList<>(size);
+ try {
+ for (int i = 0; i < size; i++) {
+ Worker worker = new Worker(threadFactory);
+ workers.add(worker);
+ bootstrapFutures.add(worker.executor.submit(() -> {
+ worker.driver = factory.newDriver(databaseName);
+ return null;
+ }));
+ }
+ // Block until all bootstraps complete so we surface construction errors
+ // before any caller hands us SQL. Bounded by BOOTSTRAP_TIMEOUT_SECONDS so a
+ // hung Hive init doesn't deadlock the sync driver thread.
+ for (Future<Void> f : bootstrapFutures) {
+ f.get(BOOTSTRAP_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ }
+ } catch (Exception e) {
+ tearDown();
+ throw new HoodieException("Failed to construct HiveDriverPool of size " + size, e);
+ }
+ LOG.info("Initialized HiveDriverPool with {} workers", size);
+ }
+
+ /**
+ * Runs each given SQL on <i>every</i> worker, in order. Used for setup statements
+ * (e.g. {@code USE database}) that must establish per-thread session context
+ * before any partition statement runs. Blocks until all workers have completed
+ * the setup. Throws on first error.
+ */
+ public void runOnEachWorker(List<String> setupSqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ if (setupSqls.isEmpty()) {
+ return;
+ }
+ List<Future<?>> futures = new ArrayList<>(workers.size());
+ for (Worker worker : workers) {
+ futures.add(worker.executor.submit(() -> {
+ for (String sql : setupSqls) {
+ worker.driver.run(sql);
+ }
+ return null;
+ }));
+ }
+ awaitAll(futures);
+ }
+
+ /**
+ * Dispatches each SQL string to a worker (round-robin) and returns the list of
+ * futures. The caller is responsible for awaiting and collecting errors. SQL text
+ * is intentionally not logged per-statement here: batched TOUCH/ADD statements can
+ * be many kilobytes, and N parallel workers would multiply the log volume. See
+ * {@link #awaitAll(List)} for the per-call summary log.
+ */
+ public List<Future<?>> runAll(List<String> sqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ List<Future<?>> futures = new ArrayList<>(sqls.size());
+ for (int i = 0; i < sqls.size(); i++) {
+ String sql = sqls.get(i);
+ Worker worker = workers.get(i % workers.size());
+ futures.add(worker.executor.submit(() -> {
+ worker.driver.run(sql);
+ return null;
+ }));
+ }
+ return futures;
+ }
+
+ /**
+ * Awaits all futures and throws the first exception encountered. On first failure,
+ * cancels the remaining (not yet started) futures so workers don't keep running
+ * pointless work after a fatal error. Any errors that finished before cancellation
+ * are logged at WARN. Callers do not need per-statement results (Hive's Driver.run
+ * side-effects the metastore), so this method is void.
+ */
+ public void awaitAll(List<Future<?>> futures) {
+ long start = System.currentTimeMillis();
+ Exception firstError = null;
+ int completed = 0;
+ int cancelled = 0;
+ for (int i = 0; i < futures.size(); i++) {
+ Future<?> f = futures.get(i);
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // We cancelled this future ourselves after a prior error. Don't treat it
+ // as a new failure; just note it for the summary log.
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ cancelled += cancelRemaining(futures, i + 1);
+ }
+ } catch (ExecutionException ee) {
+ Exception cause = unwrap(ee);
+ if (firstError == null) {
+ firstError = cause;
+ cancelled += cancelRemaining(futures, i + 1);
+ } else {
+ LOG.warn("Additional SQL batch failed (suppressed in favor of first error)", cause);
+ }
+ }
+ }
+ if (firstError != null) {
+ throw new HoodieHiveSyncException("Failed in executing SQL", firstError);
+ }
+ LOG.info("Completed {} SQL statements ({} cancelled) in {} ms across {} workers",
+ completed, cancelled, System.currentTimeMillis() - start, size);
+ }
+
+ private static int cancelRemaining(List<Future<?>> futures, int fromIndex) {
+ int cancelled = 0;
+ for (int j = fromIndex; j < futures.size(); j++) {
+ // mayInterruptIfRunning=false: the worker thread is bound to a Hive Driver
+ // whose state we don't want to corrupt mid-statement. Cancel only those that
+ // haven't started yet; in-flight statements run to completion.
+ if (futures.get(j).cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ private static Exception unwrap(ExecutionException ee) {
+ Throwable cause = ee.getCause();
+ return (cause instanceof Exception) ? (Exception) cause : ee;
+ }
+
+ public int size() {
+ return size;
+ }
+
+ @Override
+ public void close() {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ tearDown();
+ }
+
+ private void tearDown() {
+ // Close each worker's Driver/SessionState on its own thread, then shut the
+ // executor down. Running close() on the bound thread keeps SessionState's
+ // thread-local cleanup correct.
+ for (Worker worker : workers) {
+ try {
+ worker.executor.submit(() -> {
+ if (worker.driver != null) {
+ try {
+ worker.driver.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled Driver", e);
+ }
+ }
+ SessionState ss = SessionState.get();
+ if (ss != null) {
+ try {
+ ss.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled SessionState", e);
+ }
+ }
+ return null;
+ }).get(30, TimeUnit.SECONDS);
+ } catch (Exception e) {
+ LOG.warn("Error during pool worker shutdown", e);
+ }
+ worker.executor.shutdown();
+ try {
+ if (!worker.executor.awaitTermination(10, TimeUnit.SECONDS)) {
+ worker.executor.shutdownNow();
+ }
+ } catch (InterruptedException ie) {
+ worker.executor.shutdownNow();
+ Thread.currentThread().interrupt();
+ }
+ }
+ workers.clear();
+ }
+
+ /**
+ * Per-slot state: a single-thread executor and the Driver bound to its thread.
+ * Driver is volatile because it is written by the bootstrap task and read by
+ * subsequent dispatch tasks on the same executor.
+ */
+ private static final class Worker {
+ final ExecutorService executor;
+ volatile Driver driver;
+
+ Worker(ThreadFactory threadFactory) {
+ this.executor = Executors.newSingleThreadExecutor(threadFactory);
+ }
+ }
+
+ @FunctionalInterface
+ interface DriverFactory {
+ Driver newDriver(String databaseName) throws Exception;
+ }
+
+ /**
+ * Builds a real Hive {@link Driver} on the calling thread. The SessionState is
+ * constructed lazily (once, on the first worker thread that builds a Driver) and
+ * shared across all worker threads — Hive uses ThreadLocal attachment, not
+ * exclusive ownership, so multiple workers calling
+ * {@code SessionState.start(sharedState)} all see the same config and scratch dir
+ * without each spending the cost of building their own SessionState (and risking
+ * resource-dir creation races during the constructor).
+ */
+ private static final class DefaultDriverFactory implements DriverFactory {
+ private final HiveConf hiveConf;
+ private volatile SessionState sharedSessionState;
+
+ DefaultDriverFactory(HiveSyncConfig config) {
+ this.hiveConf = config.getHiveConf();
+ }
+
+ @Override
+ public synchronized Driver newDriver(String databaseName) throws Exception {
+ // SessionState is shared across workers; build it once on the first call (with
+ // currentDatabase already set) and attach it to each worker's thread-local on
+ // subsequent calls. The database is a pool-wide property and never changes
+ // across workers, so setting it once at construction time is sufficient.
Review Comment:
This is the same shared-SessionState concern already fixed on #18984 (each HiveDriverPool worker now owns its own SessionState, built via sequential bootstrap). This branch is currently rebased on #18984's pre-fix commit, so the fix isn't visible here yet — it'll show up once #18984 merges and this branch rebases onto it. Leaving open until then rather than resolving.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfigHolder.java:
##########
@@ -122,6 +122,23 @@ public class HiveSyncConfigHolder {
.defaultValue(1000)
.markAdvanced()
.withDocumentation("The number of partitions one batch when synchronous partitions to hive.");
+ public static final ConfigProperty<Boolean> HIVE_SYNC_BATCHING_ENABLED = ConfigProperty
+ .key("hoodie.datasource.hive_sync.batching.enabled")
+ .defaultValue(false)
+ .markAdvanced()
+ .sinceVersion("1.1.0")
+ .withDocumentation("When true, partition operations (add/update/touch/drop) in the active sync mode "
+ + "are split into batches of `hoodie.datasource.hive_sync.batch_num` and dispatched in parallel "
+ + "to a pool of workers. Table-level operations (create/alter table, last commit time, writer "
+ + "version) continue to use the single session client/driver. Default off; existing behavior is "
+ + "unchanged unless explicitly opted in.");
+ public static final ConfigProperty<Integer> HIVE_SYNC_BATCHING_THREADS = ConfigProperty
+ .key("hoodie.datasource.hive_sync.batching.threads")
+ .defaultValue(4)
+ .markAdvanced()
Review Comment:
This is the same config-doc/sinceVersion staleness already fixed on #18984 (sinceVersion corrected to 1.3.0, docs reworded to scope precisely to HiveQL). This branch is rebased on #18984's pre-fix commit, so it's not visible here yet — will resolve on rebase once #18984 merges.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -86,6 +90,16 @@ public class HoodieHiveSyncClient extends HoodieSyncClient {
private final Map<String, Table> initialTableByName = new HashMap<>();
DDLExecutor ddlExecutor;
private IMetaStoreClient client;
+ // Non-null only when HIVE_SYNC_BATCHING_ENABLED and sync mode is HMS or HIVEQL.
+ // Owned by this class; closed in close() before Hive.closeCurrent(). Hands clients
+ // to the active DDL executor for partition-row work only — see IMetaStoreClientPool
+ // javadoc. HMSDDLExecutor uses it for add/alter/drop; HiveQueryDDLExecutor uses it
+ // only for DROP (Hive Thrift, not Hive Driver).
+ private IMetaStoreClientPool partitionClientPool;
+ // Present only when HIVE_SYNC_BATCHING_ENABLED and sync mode is HIVEQL (explicit
+ // or legacy default). Owned by HiveQueryDDLExecutor; this field is kept for
+ // reference only — close() is delegated through ddlExecutor.close().
+ private Option<HiveDriverPool> partitionDriverPool = Option.empty();
Review Comment:
Noted — this partitionDriverPool field-vs-local-variable point applies to code introduced in #18984, not this PR. Tracking it there; will pick up on rebase.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -132,23 +196,81 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
- }
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(partitionsToDrop, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ */
+ private void runDropBatches(String tableName, List<List<String>> batches) throws Exception {
Review Comment:
Good suggestion — once #18984 merges and this branch rebases, HiveDriverPool.awaitAll(List<Future<?>>) will be available; will look at whether IMetaStoreClientPool's runDropBatches error-collection loop here can delegate to an analogous awaitAll on IMetaStoreClientPool at that point, to de-duplicate the two nearly-identical loops.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3669746061
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -209,21 +219,96 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
- }
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(partitionsToDrop, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ *
+ * <p>First-error semantics match {@code HiveDriverPool.awaitAll}: the first failure is
+ * rethrown, not-yet-started batches are cancelled, and later failures are logged at WARN.
+ */
+ private void runDropBatches(String tableName, List<List<String>> batches) throws Exception {
+ if (!metaStoreClientPool.isPresent()) {
+ for (List<String> batch : batches) {
+ applyDropBatch(metaStoreClient, tableName, batch);
+ }
+ return;
+ }
+ IMetaStoreClientPool pool = metaStoreClientPool.get();
+ List<Future<Void>> futures = new ArrayList<>(batches.size());
+ for (List<String> batch : batches) {
+ futures.add(pool.executor().submit(() ->
+ pool.run(poolClient -> {
+ applyDropBatch(poolClient, tableName, batch);
+ return null;
+ })
+ ));
+ }
+
+ Exception firstError = null;
+ int cancelled = 0;
+ for (Future<Void> f : futures) {
+ // Once a batch has failed the sync is going to abort anyway, so stop handing
+ // more DROPs to the metastore. mayInterruptIfRunning=false mirrors
+ // HiveDriverPool.awaitAll: a batch already mid-flight runs to completion rather
+ // than leaving the partition list half-dropped at an arbitrary point.
+ if (firstError != null && f.cancel(false)) {
+ cancelled++;
+ continue;
+ }
+ try {
+ f.get();
Review Comment:
Good catch — you are right, and the abort was not merely weaker than advertised, it was doing nothing. I wrote a unit test against the previous code and every queued batch still ran after the first failure:
```
expected: <[FAIL]> but was: <[FAIL, AFTER_A, AFTER_B]>
```
The loop waited on futures in submission order, so a fast failure on a later batch went unobserved while the awaiting thread was parked on an earlier slow one, and the executor kept starting queued tasks the whole time. `Future.cancel` from the awaiting thread is always too late once a worker has dequeued its next task.
Fixed by taking your suggestion and extracting `HiveDriverPool.Dispatch` into a shared `ParallelDispatch` rather than reimplementing the pattern a second time — so both mechanisms that make it work are now shared: the latch tripped by the first abort (awaiting thread wakes on failure, not on its turn in submission order) and the task-side abort check on entry.
`IMetaStoreClientPool` now has `dispatchAll`/`awaitAll` built on it, and `runDropBatches` delegates instead of hand-rolling the loop — which also resolves the duplicate-loop nit hudi-agent raised earlier. Net effect is -139 lines.
Covered by two regression tests in the new `TestIMetaStoreClientPool`: `abortStopsQueuedBatchesAfterFirstFailure` and `abortStopsQueuedBatchesWhenEarlierBatchIsSlow` (the latter pins exactly the slow-earlier-future interleaving you described). Both fail against the previous implementation. Pushed in `8721c9a`.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -227,6 +241,22 @@ private IMetaStoreClient createMetaStoreClient(HiveSyncConfig config) {
}
}
+ private Option<IMetaStoreClientPool> maybeBuildPartitionClientPool(HiveSyncConfig config) {
+ if (!config.getBooleanOrDefault(HIVE_SYNC_BATCHING_ENABLED)) {
+ return Option.empty();
+ }
+ if (config.getBooleanOrDefault(HIVE_SYNC_USE_SPARK_CATALOG)) {
+ // The Spark catalog client is constructed via reflection against a Spark-side
+ // class and isn't compatible with the direct RetryingMetaStoreClient pool path.
+ // Fall back to single-client sequential behavior rather than failing the sync.
+ log.warn("hive_sync.batching.enabled=true is not supported with use_spark_catalog=true; "
+ + "falling back to sequential partition sync.");
+ return Option.empty();
+ }
+ int size = config.getIntOrDefault(HIVE_SYNC_BATCHING_THREADS);
+ return Option.of(new IMetaStoreClientPool(config, size));
Review Comment:
Agreed — that text was the pre-DROP contract and now states the opposite of the behavior. Fixed in `6c81322`.
`HIVE_SYNC_BATCHING_ENABLED` now documents that DROP is parallelized too, that it fans out over metastore (Thrift) clients rather than Hive Driver workers (since it is issued as `dropPartition` calls rather than SQL), that drops are split into `batch_num`-sized batches, and that it falls back to sequential execution on the single session client when `use_spark_catalog=true`.
Also updated `batching.threads`, which the same value now sizes for both pools — that was equally misleading since it only mentioned Driver workers. And switched the spark-catalog fallback warning to emit real config keys via `.key()` (hudi-agent nit on the same change), so operators can grep the property they actually set.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3737161425
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveMetaStoreClientPool.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Pool of {@link IMetaStoreClient} instances for parallel partition sync.
+ *
+ * <p>Each pooled client wraps an independent Thrift connection to the Hive Metastore.
+ * Callers borrow a client via {@link #run(ClientAction)}, which blocks until a client
+ * is available, executes the action, and returns the client to the pool. A worker
+ * thread pool of the same size is exposed via {@link #executor()} so callers can fan
+ * out their batches to match the number of available clients.
+ *
+ * <p><b>Usage contract:</b> pool clients must be used <i>only</i> for partition-row
+ * operations — {@code add_partitions}, {@code alter_partitions}, {@code dropPartition},
+ * {@code getPartition}. Table-row operations ({@code createTable}, {@code alter_table},
+ * {@code getTable} used as the read half of a read-modify-write of table parameters)
+ * must continue to go through the session client held by
+ * {@code HoodieHiveSyncClient.client} on the sync driver thread. Mixing the two would
+ * risk lost updates on table parameters such as the last-commit-time-synced marker.
+ *
+ * <p>The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled} and
+ * is constructed for sync mode HIVEQL, where it backs the DROP path only. DROP goes
+ * through {@code IMetaStoreClient.dropPartition} (Thrift), whereas ADD/UPDATE/TOUCH go
+ * through the thread-bound Hive {@code Driver} and use {@code HiveDriverPool} instead.
Review Comment:
Done in 4d0ac71 — renamed to `HiveMetaStoreClientPool`.
Agreed on the reasoning: with `IMetaStoreClient` (a genuine interface) imported in the same files, the `I` prefix on a concrete pool class is actively misleading. The class is introduced by this PR and has no callers outside it, so there is no deprecation path to worry about — free to do now, awkward after merge.
Left the log and exception strings that read "IMetaStoreClient pool" as they were: those describe a pool *of* `IMetaStoreClient` instances, which is still accurate. Renaming them would have made them less correct, not more.
Test class renamed to `TestHiveMetaStoreClientPool` alongside it. Git recorded both as renames, so the diff should stay readable.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveMetaStoreClientPool.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Pool of {@link IMetaStoreClient} instances for parallel partition sync.
+ *
+ * <p>Each pooled client wraps an independent Thrift connection to the Hive Metastore.
+ * Callers borrow a client via {@link #run(ClientAction)}, which blocks until a client
+ * is available, executes the action, and returns the client to the pool. A worker
+ * thread pool of the same size is exposed via {@link #executor()} so callers can fan
+ * out their batches to match the number of available clients.
+ *
+ * <p><b>Usage contract:</b> pool clients must be used <i>only</i> for partition-row
+ * operations — {@code add_partitions}, {@code alter_partitions}, {@code dropPartition},
+ * {@code getPartition}. Table-row operations ({@code createTable}, {@code alter_table},
+ * {@code getTable} used as the read half of a read-modify-write of table parameters)
+ * must continue to go through the session client held by
+ * {@code HoodieHiveSyncClient.client} on the sync driver thread. Mixing the two would
+ * risk lost updates on table parameters such as the last-commit-time-synced marker.
+ *
+ * <p>The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled} and
+ * is constructed for sync mode HIVEQL, where it backs the DROP path only. DROP goes
+ * through {@code IMetaStoreClient.dropPartition} (Thrift), whereas ADD/UPDATE/TOUCH go
+ * through the thread-bound Hive {@code Driver} and use {@code HiveDriverPool} instead.
Review Comment:
Done in 4d0ac71 — renamed to `HiveMetaStoreClientPool`.
Agreed on the reasoning: with `IMetaStoreClient` (a genuine interface) imported in the same files, the `I` prefix on a concrete pool class is actively misleading. The class is introduced by this PR and has no callers outside it, so there is no deprecation path to worry about — free to do now, awkward after merge.
Left the log and exception strings that read "IMetaStoreClient pool" as they were: those describe a pool *of* `IMetaStoreClient` instances, which is still accurate. Renaming them would have made them less correct, not more.
Test class renamed to `TestHiveMetaStoreClientPool` alongside it. Git recorded both as renames, so the diff should stay readable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5221058306
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
}, {
"hash" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 4d0ac7105b4dc361317b936bc2c85a8683f0e8aa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183)
* 038e6c842e18130a64552da6ff5aef46864b3cf0 UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5099143837
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684)
* 7610c51a4e727c97110e3dadba9868c4fff7aa0f Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3707878142
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -227,6 +241,22 @@ private IMetaStoreClient createMetaStoreClient(HiveSyncConfig config) {
}
}
+ private Option<IMetaStoreClientPool> maybeBuildPartitionClientPool(HiveSyncConfig config) {
+ if (!config.getBooleanOrDefault(HIVE_SYNC_BATCHING_ENABLED)) {
+ return Option.empty();
+ }
+ if (config.getBooleanOrDefault(HIVE_SYNC_USE_SPARK_CATALOG)) {
+ // The Spark catalog client is constructed via reflection against a Spark-side
Review Comment:
Fixed in `6c81322` — the warning now emits `HIVE_SYNC_BATCHING_ENABLED.key()` and `HIVE_SYNC_USE_SPARK_CATALOG.key()` rather than the abbreviated names, so the strings match the properties an operator would actually grep for. This went in alongside the batching-contract doc update on the neighbouring thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3739710143
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/ParallelDispatch.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.common.util.VisibleForTesting;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Handle to one fan-out batch of partition work: the submitted futures plus the shared
+ * abort flag the tasks consult before running.
+ *
+ * <p>The abort flag exists because waiting on futures in <i>submission</i> order is not
+ * enough to stop queued work. If a later task fails quickly while an earlier one is slow,
+ * the awaiting thread is still parked on the earlier {@code Future.get()}, and the
+ * executor happily keeps starting every queued task in the meantime. By the time the
+ * failure is observed, most of the "not-yet-started" work has already run.
+ *
+ * <p>Two mechanisms fix that, and both are needed:
+ * <ul>
+ * <li>a {@link CountDownLatch} tripped by the <i>first</i> abort, so the awaiting
+ * thread wakes on failure rather than on its turn in submission order;</li>
+ * <li>a task-side {@link #aborted()} check on entry, because cancelling from the
+ * awaiting thread is inherently late — a worker can pull its next task off the
+ * queue at any moment.</li>
+ * </ul>
+ *
+ * <p>The failing task also records its own throwable as it aborts, so the reported root
+ * cause is the failure that stopped the batch. Picking the error by scanning futures in
+ * submission order would instead surface whichever failure happens to sit earliest in the
+ * list, which is not necessarily the one that aborted the run.
+ *
+ * <p>Shared by {@link HiveDriverPool} (Hive {@code Driver} statements) and
+ * {@link HiveMetaStoreClientPool} (Thrift {@code dropPartition} batches), which fan out over
+ * different execution models but need identical abort-on-first-error semantics.
+ */
+public final class ParallelDispatch {
+
+ private final List<Future<?>> futures;
+ private final int total;
+ private final AtomicInteger settled = new AtomicInteger(0);
+ private final AtomicBoolean aborted = new AtomicBoolean(false);
+ private final AtomicReference<Throwable> abortCause = new AtomicReference<>();
+ private final CountDownLatch done = new CountDownLatch(1);
+ private volatile boolean sealed;
+
+ ParallelDispatch(int total) {
+ this.total = total;
+ this.futures = new ArrayList<>(total);
+ }
+
+ void add(Future<?> future) {
+ futures.add(future);
+ }
+
+ // Called once submission finishes. A task that settles before the last submit
+ // would otherwise see settled < total and never trip the latch, so re-check here.
+ void sealed() {
+ sealed = true;
+ signalIfComplete();
+ }
+
+ boolean aborted() {
+ return aborted.get();
+ }
+
+ void abort() {
+ aborted.set(true);
+ done.countDown();
+ }
+
+ // Records the failure that triggered the abort, first writer wins. Selecting the error
+ // by walking futures in submission order instead would report whichever failure sits
+ // earliest in the list, not the one that actually stopped the batch: a fast failure at
+ // index 1 aborts the run, and a slow index 0 that fails later would take its place.
+ void abort(Throwable cause) {
+ abortCause.compareAndSet(null, cause);
+ abort();
+ }
+
+ void taskSettled() {
+ settled.incrementAndGet();
+ signalIfComplete();
+ }
+
+ private void signalIfComplete() {
+ if (sealed && settled.get() >= total) {
+ done.countDown();
+ }
+ }
+
+ void awaitSettledOrAborted() {
+ if (total == 0) {
+ return;
+ }
+ try {
+ done.await();
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ aborted.set(true);
+ }
Review Comment:
**[P1] Propagate interruption as a failed dispatch**
If the awaiting thread is interrupted here, this restores the interrupt flag and sets `aborted`, but never records the `InterruptedException` in `abortCause` or otherwise returns it to `awaitOutcome()`. The next `cancelPending()` can mark every future cancelled, so the future drain sees only `CancellationException`; `firstError` remains null and both pool `awaitAll()` methods report success. On the DROP path, `HiveSyncTool` can then advance the last-synced commit marker even though some partition operations were cancelled or are still running. Please preserve the interrupt as the dispatch failure (and ensure the caller cannot observe success before in-flight work settles). A regression test can block a dispatched action, interrupt the thread in `awaitAll`, and assert that `awaitAll` throws rather than returning normally.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -209,21 +217,84 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
+ // Resolved here, on the calling thread, rather than inside the workers: this is the
+ // only sync path that would otherwise call a user-supplied PartitionValueExtractor
+ // from several threads at once. Extractors are pluggable and not required to be
+ // thread-safe, and a garbled clause would drop the wrong partition. It also halves
+ // the extractor calls, since partitionExists and the drop clause share the values.
+ List<PartitionToDrop> resolved = new ArrayList<>(partitionsToDrop.size());
+ for (String partition : partitionsToDrop) {
+ List<String> values = partitionValueExtractor.extractPartitionValuesInPath(partition);
+ resolved.add(new PartitionToDrop(partition, values,
+ HivePartitionUtil.getPartitionClauseForDrop(values, config)));
}
+
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<PartitionToDrop>> batches = CollectionUtils.batches(resolved, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ *
+ * <p>First-error semantics come from {@link ParallelDispatch}, shared with
+ * {@code HiveDriverPool}: the first failure is rethrown, batches that have not started
+ * are stopped via the task-side abort flag, and later failures are logged at WARN.
+ */
+ private void runDropBatches(String tableName, List<List<PartitionToDrop>> batches) throws Exception {
+ if (!metaStoreClientPool.isPresent()) {
+ for (List<PartitionToDrop> batch : batches) {
+ applyDropBatch(metaStoreClient, tableName, batch);
+ }
+ return;
+ }
+ HiveMetaStoreClientPool pool = metaStoreClientPool.get();
+ pool.awaitAll(
+ pool.dispatchAll(batches, (client, batch) -> applyDropBatch(client, tableName, batch)),
+ "drop partition");
+ }
+
+ private void applyDropBatch(IMetaStoreClient client, String tableName, List<PartitionToDrop> batch) throws Exception {
+ int dropped = 0;
+ for (PartitionToDrop dropPartition : batch) {
+ if (HivePartitionUtil.partitionExists(client, tableName, dropPartition.path,
+ dropPartition.values, config)) {
+ client.dropPartition(databaseName, tableName, dropPartition.clause, false);
+ dropped++;
+ }
+ // Per-partition detail stays at debug: a batch can hold thousands of partitions
+ // and N workers log concurrently, so INFO carries the per-batch summary instead.
+ log.debug("Dropped partition {} on {}", dropPartition.path, tableName);
+ }
+ log.info("Dropped {} of {} partitions in batch on {}", dropped, batch.size(), tableName);
+ }
+
+ /**
+ * A partition to drop with its extractor-derived values already resolved, so worker
+ * threads never touch the shared {@link PartitionValueExtractor}. Immutable: the
+ * {@code values} list is wrapped unmodifiable at construction.
+ */
+ private static final class PartitionToDrop {
+ private final String path;
+ private final List<String> values;
+ private final String clause;
+
+ private PartitionToDrop(String path, List<String> values, String clause) {
+ this.path = path;
+ this.values = Collections.unmodifiableList(values);
Review Comment:
**[P2] Defensively copy the extractor result**
`Collections.unmodifiableList(values)` only wraps the supplied list; it does not make a snapshot. `PartitionValueExtractor` is pluggable and its interface does not require returning a fresh or immutable list, so a custom extractor that reuses a mutable buffer leaves every `PartitionToDrop.values` pointing at the values from the final extraction. The clauses were materialized correctly, but `partitionExists()` then checks the wrong partition for earlier entries, which can skip valid drops or cause a spurious failure. Please copy before wrapping, for example `Collections.unmodifiableList(new ArrayList<>(values))`, and cover this with an extractor that deliberately reuses one list.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3643375058
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -597,6 +642,17 @@ public void deleteLastReplicatedTimeStamp(String tableName) {
public void close() {
try {
ddlExecutor.close();
Review Comment:
Fixed in 9e35ec8 — partitionClientPool.close() and Hive.closeCurrent() now run in a finally, so they still execute even if ddlExecutor.close() throws.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -124,7 +137,10 @@ public HoodieHiveSyncClient(HiveSyncConfig config, HoodieTableMetaClient metaCli
ddlExecutor = new HMSDDLExecutor(config, this.client);
break;
case HIVEQL:
- ddlExecutor = new HiveQueryDDLExecutor(config, this.client);
+ this.partitionDriverPool = maybeBuildHiveDriverPool(config);
Review Comment:
Fixed in 9e35ec8 — extracted buildHiveQueryDDLExecutor(config), which builds partitionDriverPool, partitionClientPool, and the HiveQueryDDLExecutor as one sequence and closes whichever pool(s) were already built if any later step throws, before rethrowing. Same pattern applied to both call sites (the explicit HIVEQL branch and the legacy default-mode branch).
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -86,6 +90,15 @@ public class HoodieHiveSyncClient extends HoodieSyncClient {
private final Map<String, Table> initialTableByName = new HashMap<>();
DDLExecutor ddlExecutor;
private IMetaStoreClient client;
+ // Non-null only when HIVE_SYNC_BATCHING_ENABLED and sync mode is HIVEQL. Owned by
+ // this class; closed in close() before Hive.closeCurrent(). HiveQueryDDLExecutor
+ // uses it only for DROP (Hive Thrift, not Hive Driver) — see IMetaStoreClientPool
+ // javadoc.
+ private IMetaStoreClientPool partitionClientPool;
Review Comment:
Fixed in 9e35ec8 — partitionClientPool is now Option<IMetaStoreClientPool>, matching partitionDriverPool.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5110635469
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7610c51a4e727c97110e3dadba9868c4fff7aa0f Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748)
* 6c81322202e4b6cb251248da8a0d8c478fb3fe26 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3426437347
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -597,6 +642,17 @@ public void deleteLastReplicatedTimeStamp(String tableName) {
public void close() {
try {
ddlExecutor.close();
Review Comment:
🤖 If `ddlExecutor.close()` throws, control jumps to the outer `catch (Exception e) { log.error(...) }` (which doesn't rethrow), so `partitionClientPool.close()` never runs and the Thrift sockets + worker threads leak silently. Could you wrap the pool cleanup in a try-finally so it runs regardless of whether `ddlExecutor.close()` succeeds? E.g. `try { ddlExecutor.close(); } finally { if (partitionClientPool != null) { ... } }`.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -86,6 +90,16 @@ public class HoodieHiveSyncClient extends HoodieSyncClient {
private final Map<String, Table> initialTableByName = new HashMap<>();
DDLExecutor ddlExecutor;
private IMetaStoreClient client;
+ // Non-null only when HIVE_SYNC_BATCHING_ENABLED and sync mode is HMS or HIVEQL.
+ // Owned by this class; closed in close() before Hive.closeCurrent(). Hands clients
+ // to the active DDL executor for partition-row work only — see IMetaStoreClientPool
+ // javadoc. HMSDDLExecutor uses it for add/alter/drop; HiveQueryDDLExecutor uses it
+ // only for DROP (Hive Thrift, not Hive Driver).
+ private IMetaStoreClientPool partitionClientPool;
+ // Present only when HIVE_SYNC_BATCHING_ENABLED and sync mode is HIVEQL (explicit
+ // or legacy default). Owned by HiveQueryDDLExecutor; this field is kept for
+ // reference only — close() is delegated through ddlExecutor.close().
+ private Option<HiveDriverPool> partitionDriverPool = Option.empty();
Review Comment:
🤖 nit: the comment says this field is 'kept for reference only — close() is delegated through ddlExecutor.close()', which means it's never read after being passed into `HiveQueryDDLExecutor`. Could you make it a local variable in the constructor instead? A field that isn't read after construction will make future readers wonder what lifecycle role it plays.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -291,21 +312,71 @@ public void close() {
private void registerAlterPartitionEvent(String tableName, List<String> alteredPartitions) {
try {
+ // Read the StorageDescriptor once on the session client; each worker deep-copies
+ // it per partition (alter_partitions semantics today).
StorageDescriptor sd = client.getTable(databaseName, tableName).getSd();
- List<Partition> partitionList = alteredPartitions.stream().map(partition -> {
- Path partitionPath = HadoopFSUtils.constructAbsolutePathInHadoopPath(syncConfig.getString(META_SYNC_BASE_PATH), partition);
- String partitionScheme = partitionPath.toUri().getScheme();
- String fullPartitionPath = StorageSchemes.HDFS.getScheme().equals(partitionScheme)
- ? HadoopFSUtils.getDFSFullPartitionPath(syncConfig.getHadoopFileSystem(), partitionPath) : partitionPath.toString();
- List<String> partitionValues = partitionValueExtractor.extractPartitionValuesInPath(partition);
- StorageDescriptor partitionSd = sd.deepCopy();
- partitionSd.setLocation(fullPartitionPath);
- return new Partition(partitionValues, databaseName, tableName, 0, 0, partitionSd, null);
- }).collect(Collectors.toList());
- client.alter_partitions(databaseName, tableName, partitionList, null);
- } catch (TException e) {
+ int batchSyncPartitionNum = syncConfig.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(alteredPartitions, batchSyncPartitionNum);
+ runBatches("alter", tableName, batches, (poolClient, batch) -> {
+ List<Partition> partitionList = batch.stream().map(partition -> {
+ Path partitionPath = HadoopFSUtils.constructAbsolutePathInHadoopPath(syncConfig.getString(META_SYNC_BASE_PATH), partition);
+ String partitionScheme = partitionPath.toUri().getScheme();
+ String fullPartitionPath = StorageSchemes.HDFS.getScheme().equals(partitionScheme)
+ ? HadoopFSUtils.getDFSFullPartitionPath(syncConfig.getHadoopFileSystem(), partitionPath) : partitionPath.toString();
+ List<String> partitionValues = partitionValueExtractor.extractPartitionValuesInPath(partition);
+ StorageDescriptor partitionSd = sd.deepCopy();
+ partitionSd.setLocation(fullPartitionPath);
+ return new Partition(partitionValues, databaseName, tableName, 0, 0, partitionSd, null);
+ }).collect(Collectors.toList());
+ poolClient.alter_partitions(databaseName, tableName, partitionList, null);
+ });
+ } catch (Exception e) {
log.error("{}.{} update partition failed", databaseName, tableName, e);
throw new HoodieHiveSyncException(databaseName + "." + tableName + " update partition failed", e);
}
}
+
+ /**
+ * Dispatches partition batches either in parallel against the client pool (if
+ * configured) or sequentially against the session client. The sequential path
+ * preserves the exact behavior we had before the pool existed, including failure
+ * semantics where batch N+1 never runs if batch N throws.
+ */
+ @FunctionalInterface
+ private interface BatchAction {
+ void apply(IMetaStoreClient client, List<String> batch) throws Exception;
+ }
+
+ private void runBatches(String opName, String tableName, List<List<String>> batches, BatchAction action) throws Exception {
+ if (partitionClientPool == null) {
+ for (List<String> batch : batches) {
+ action.apply(client, batch);
+ }
+ return;
+ }
+ List<Future<Void>> futures = new ArrayList<>(batches.size());
+ for (List<String> batch : batches) {
+ futures.add(partitionClientPool.executor().submit(() ->
+ partitionClientPool.run(poolClient -> {
+ action.apply(poolClient, batch);
Review Comment:
🤖 When a Future fails, `f.get()` throws `ExecutionException` wrapping the real cause. `firstError` then stores the `ExecutionException`, which gets rethrown and caught by the outer `catch (Exception e)` in `addPartitionsToTable`/`dropPartitionsToTable`/`registerAlterPartitionEvent`, which wraps it again in `HoodieHiveSyncException`. The result is `HoodieHiveSyncException -> ExecutionException -> RealCause` — one more wrapping layer than the sequential path produced. HiveDriverPool.awaitAll already has an `unwrap()` helper; could we do the same here for consistency?
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveDriverPool.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+import org.apache.hudi.hive.HoodieHiveSyncException;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.Driver;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_DATABASE_NAME;
+
+/**
+ * Pool of Hive {@link Driver} + {@link SessionState} pairs for parallel HiveQL DDL.
+ *
+ * <p>Hive's {@code SessionState.start(state)} binds state to the calling thread's
+ * thread-local, and {@code Driver} reads from that thread-local during {@code run()}.
+ * A Driver constructed on one thread cannot be safely used from another. This pool
+ * solves that by giving each slot its own dedicated worker thread (a single-thread
+ * executor) — the Driver and SessionState are built on that thread by a bootstrap
+ * task, and all subsequent SQL for that slot runs on the same thread.
+ *
+ * <p><b>Usage contract:</b> use this pool only for partition-row DDL statements that
+ * are independent of each other and freely shuffleable across workers. Table-level
+ * statements (createTable, schema evolution, USE database) must continue to run on
+ * the session {@code Driver} held by {@code HiveQueryDDLExecutor} on the sync driver
+ * thread. The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled}
+ * and is constructed only for HiveQL sync mode.
+ */
+public class HiveDriverPool implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveDriverPool.class);
+
+ // Per-worker Driver construction has to be fast in practice (a few hundred ms
+ // for the SessionState + Driver init). A 60s ceiling per worker leaves plenty of
+ // headroom for a slow JVM warm-up but bounds the failure mode if the metastore
+ // is unreachable or Hive hangs during init.
+ private static final long BOOTSTRAP_TIMEOUT_SECONDS = 60;
+
+ private final List<Worker> workers;
+ private final int size;
+ private volatile boolean closed;
+
+ public HiveDriverPool(HiveSyncConfig config, int size) {
+ this(config, size, new DefaultDriverFactory(config));
+ }
+
+ // Package-private for tests: accepts a DriverFactory so unit tests can inject
+ // mock Driver instances without standing up a real Hive instance.
+ HiveDriverPool(HiveSyncConfig config, int size, DriverFactory factory) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ this.size = size;
+ this.workers = new ArrayList<>(size);
+ String databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
+ PoolThreadFactory threadFactory = new PoolThreadFactory();
+ List<Future<Void>> bootstrapFutures = new ArrayList<>(size);
+ try {
+ for (int i = 0; i < size; i++) {
+ Worker worker = new Worker(threadFactory);
+ workers.add(worker);
+ bootstrapFutures.add(worker.executor.submit(() -> {
+ worker.driver = factory.newDriver(databaseName);
+ return null;
+ }));
+ }
+ // Block until all bootstraps complete so we surface construction errors
+ // before any caller hands us SQL. Bounded by BOOTSTRAP_TIMEOUT_SECONDS so a
+ // hung Hive init doesn't deadlock the sync driver thread.
+ for (Future<Void> f : bootstrapFutures) {
+ f.get(BOOTSTRAP_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ }
+ } catch (Exception e) {
+ tearDown();
+ throw new HoodieException("Failed to construct HiveDriverPool of size " + size, e);
+ }
+ LOG.info("Initialized HiveDriverPool with {} workers", size);
+ }
+
+ /**
+ * Runs each given SQL on <i>every</i> worker, in order. Used for setup statements
+ * (e.g. {@code USE database}) that must establish per-thread session context
+ * before any partition statement runs. Blocks until all workers have completed
+ * the setup. Throws on first error.
+ */
+ public void runOnEachWorker(List<String> setupSqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ if (setupSqls.isEmpty()) {
+ return;
+ }
+ List<Future<?>> futures = new ArrayList<>(workers.size());
+ for (Worker worker : workers) {
+ futures.add(worker.executor.submit(() -> {
+ for (String sql : setupSqls) {
+ worker.driver.run(sql);
+ }
+ return null;
+ }));
+ }
+ awaitAll(futures);
+ }
+
+ /**
+ * Dispatches each SQL string to a worker (round-robin) and returns the list of
+ * futures. The caller is responsible for awaiting and collecting errors. SQL text
+ * is intentionally not logged per-statement here: batched TOUCH/ADD statements can
+ * be many kilobytes, and N parallel workers would multiply the log volume. See
+ * {@link #awaitAll(List)} for the per-call summary log.
+ */
+ public List<Future<?>> runAll(List<String> sqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ List<Future<?>> futures = new ArrayList<>(sqls.size());
+ for (int i = 0; i < sqls.size(); i++) {
+ String sql = sqls.get(i);
+ Worker worker = workers.get(i % workers.size());
+ futures.add(worker.executor.submit(() -> {
+ worker.driver.run(sql);
+ return null;
+ }));
+ }
+ return futures;
+ }
+
+ /**
+ * Awaits all futures and throws the first exception encountered. On first failure,
+ * cancels the remaining (not yet started) futures so workers don't keep running
+ * pointless work after a fatal error. Any errors that finished before cancellation
+ * are logged at WARN. Callers do not need per-statement results (Hive's Driver.run
+ * side-effects the metastore), so this method is void.
+ */
+ public void awaitAll(List<Future<?>> futures) {
+ long start = System.currentTimeMillis();
+ Exception firstError = null;
+ int completed = 0;
+ int cancelled = 0;
+ for (int i = 0; i < futures.size(); i++) {
+ Future<?> f = futures.get(i);
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // We cancelled this future ourselves after a prior error. Don't treat it
+ // as a new failure; just note it for the summary log.
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ cancelled += cancelRemaining(futures, i + 1);
+ }
+ } catch (ExecutionException ee) {
+ Exception cause = unwrap(ee);
+ if (firstError == null) {
+ firstError = cause;
+ cancelled += cancelRemaining(futures, i + 1);
+ } else {
+ LOG.warn("Additional SQL batch failed (suppressed in favor of first error)", cause);
+ }
+ }
+ }
+ if (firstError != null) {
+ throw new HoodieHiveSyncException("Failed in executing SQL", firstError);
+ }
+ LOG.info("Completed {} SQL statements ({} cancelled) in {} ms across {} workers",
+ completed, cancelled, System.currentTimeMillis() - start, size);
+ }
+
+ private static int cancelRemaining(List<Future<?>> futures, int fromIndex) {
+ int cancelled = 0;
+ for (int j = fromIndex; j < futures.size(); j++) {
+ // mayInterruptIfRunning=false: the worker thread is bound to a Hive Driver
+ // whose state we don't want to corrupt mid-statement. Cancel only those that
+ // haven't started yet; in-flight statements run to completion.
+ if (futures.get(j).cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ private static Exception unwrap(ExecutionException ee) {
+ Throwable cause = ee.getCause();
+ return (cause instanceof Exception) ? (Exception) cause : ee;
+ }
+
+ public int size() {
+ return size;
+ }
+
+ @Override
+ public void close() {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ tearDown();
+ }
+
+ private void tearDown() {
+ // Close each worker's Driver/SessionState on its own thread, then shut the
+ // executor down. Running close() on the bound thread keeps SessionState's
+ // thread-local cleanup correct.
+ for (Worker worker : workers) {
+ try {
+ worker.executor.submit(() -> {
+ if (worker.driver != null) {
+ try {
+ worker.driver.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled Driver", e);
+ }
+ }
+ SessionState ss = SessionState.get();
+ if (ss != null) {
+ try {
+ ss.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled SessionState", e);
+ }
+ }
+ return null;
+ }).get(30, TimeUnit.SECONDS);
+ } catch (Exception e) {
+ LOG.warn("Error during pool worker shutdown", e);
+ }
+ worker.executor.shutdown();
+ try {
+ if (!worker.executor.awaitTermination(10, TimeUnit.SECONDS)) {
+ worker.executor.shutdownNow();
+ }
+ } catch (InterruptedException ie) {
+ worker.executor.shutdownNow();
+ Thread.currentThread().interrupt();
+ }
+ }
+ workers.clear();
+ }
+
+ /**
+ * Per-slot state: a single-thread executor and the Driver bound to its thread.
+ * Driver is volatile because it is written by the bootstrap task and read by
+ * subsequent dispatch tasks on the same executor.
+ */
+ private static final class Worker {
+ final ExecutorService executor;
+ volatile Driver driver;
+
+ Worker(ThreadFactory threadFactory) {
+ this.executor = Executors.newSingleThreadExecutor(threadFactory);
+ }
+ }
+
+ @FunctionalInterface
+ interface DriverFactory {
+ Driver newDriver(String databaseName) throws Exception;
+ }
+
+ /**
+ * Builds a real Hive {@link Driver} on the calling thread. The SessionState is
+ * constructed lazily (once, on the first worker thread that builds a Driver) and
+ * shared across all worker threads — Hive uses ThreadLocal attachment, not
+ * exclusive ownership, so multiple workers calling
+ * {@code SessionState.start(sharedState)} all see the same config and scratch dir
+ * without each spending the cost of building their own SessionState (and risking
+ * resource-dir creation races during the constructor).
+ */
+ private static final class DefaultDriverFactory implements DriverFactory {
+ private final HiveConf hiveConf;
+ private volatile SessionState sharedSessionState;
+
+ DefaultDriverFactory(HiveSyncConfig config) {
+ this.hiveConf = config.getHiveConf();
+ }
+
+ @Override
+ public synchronized Driver newDriver(String databaseName) throws Exception {
+ // SessionState is shared across workers; build it once on the first call (with
+ // currentDatabase already set) and attach it to each worker's thread-local on
+ // subsequent calls. The database is a pool-wide property and never changes
+ // across workers, so setting it once at construction time is sufficient.
Review Comment:
🤖 The shared SessionState pattern here makes me a bit nervous — every worker thread calls `SessionState.start(sharedSessionState)` and then runs concurrent `Driver.run()` against the same instance. Hive's SessionState has plenty of mutable fields the Driver writes during query execution (e.g. `currentDatabase` via USE, `hiveOperation`, `cmd`, `lineageState`, `tablesAlreadyMarkedAsRead`). Are we confident none of these get written concurrently by parallel ALTER PARTITION / TOUCH statements? @yihua could you weigh in on whether sharing SessionState across worker threads is safe in practice, or whether each worker should construct its own (paying the SessionState init cost in exchange for isolation)?
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -132,23 +196,81 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
- }
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(partitionsToDrop, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ */
+ private void runDropBatches(String tableName, List<List<String>> batches) throws Exception {
Review Comment:
🤖 nit: the parallel-path error-collection loop inside `runDropBatches` (`Exception firstError = null; for (Future<Void> f : futures) …`) is nearly identical to the one in `HMSDDLExecutor.runBatches`. Since `HiveDriverPool` already has a well-factored `awaitAll(List<Future<?>>)` for exactly this pattern, it might be worth adding a similar `awaitAll(List<Future<Void>>)` to `IMetaStoreClientPool` so both callers can delegate to it instead.
<sub><i>- AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727584991
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `3.08483%` with `377 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 47.44%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 0.00% | [137 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 0.00% | [68 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 0.00% | [67 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 0.00% | [28 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 28 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|1|
>|spark-scala-tests|12|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
=============================================
- Coverage 68.26% 47.44% -20.83%
+ Complexity 29513 16702 -12811
=============================================
Files 2542 2045 -497
Lines 142627 115857 -26770
Branches 17788 14936 -2852
=============================================
- Hits 97369 54963 -42406
- Misses 37253 54910 +17657
+ Partials 8005 5984 -2021
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.65% <3.08%> (-16.12%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.50% <3.08%> (-12.34%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (-86.05%)` | :arrow_down: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-48.93%)` | :arrow_down: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-80.35%)` | :arrow_down: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-65.68%)` | :arrow_down: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
... and [1662 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727564612
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `3.08483%` with `377 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.31%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 0.00% | [137 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 0.00% | [68 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 0.00% | [67 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 0.00% | [28 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 31 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|0|
>|spark-scala-tests|12|1|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
=============================================
- Coverage 68.26% 46.31% -21.96%
+ Complexity 29513 15293 -14220
=============================================
Files 2542 1980 -562
Lines 142627 108306 -34321
Branches 17788 13230 -4558
=============================================
- Hits 97369 50159 -47210
- Misses 37253 52838 +15585
+ Partials 8005 5309 -2696
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.75% <3.08%> (-12.10%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (-86.05%)` | :arrow_down: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-48.93%)` | :arrow_down: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-80.35%)` | :arrow_down: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-65.68%)` | :arrow_down: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
... and [1744 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727693556
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 66.44%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 66.44% -1.82%
+ Complexity 29513 29296 -217
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
- Hits 97369 96678 -691
- Misses 37253 40634 +3381
- Partials 8005 8178 +173
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.58% <3.08%> (-5.19%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.89% <7.45%> (-0.96%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [339 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727714295
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.37%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.37% -0.90%
- Complexity 29513 29701 +188
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98021 +652
- Misses 37253 39303 +2050
- Partials 8005 8166 +161
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.36% <3.08%> (-1.42%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.11% <7.45%> (-0.74%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [203 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4728394472
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7022e7d493416d600664235c3056d32ea4a1c179 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3643359984
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java:
##########
@@ -291,21 +312,71 @@ public void close() {
private void registerAlterPartitionEvent(String tableName, List<String> alteredPartitions) {
try {
+ // Read the StorageDescriptor once on the session client; each worker deep-copies
+ // it per partition (alter_partitions semantics today).
StorageDescriptor sd = client.getTable(databaseName, tableName).getSd();
- List<Partition> partitionList = alteredPartitions.stream().map(partition -> {
- Path partitionPath = HadoopFSUtils.constructAbsolutePathInHadoopPath(syncConfig.getString(META_SYNC_BASE_PATH), partition);
- String partitionScheme = partitionPath.toUri().getScheme();
- String fullPartitionPath = StorageSchemes.HDFS.getScheme().equals(partitionScheme)
- ? HadoopFSUtils.getDFSFullPartitionPath(syncConfig.getHadoopFileSystem(), partitionPath) : partitionPath.toString();
- List<String> partitionValues = partitionValueExtractor.extractPartitionValuesInPath(partition);
- StorageDescriptor partitionSd = sd.deepCopy();
- partitionSd.setLocation(fullPartitionPath);
- return new Partition(partitionValues, databaseName, tableName, 0, 0, partitionSd, null);
- }).collect(Collectors.toList());
- client.alter_partitions(databaseName, tableName, partitionList, null);
- } catch (TException e) {
+ int batchSyncPartitionNum = syncConfig.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(alteredPartitions, batchSyncPartitionNum);
+ runBatches("alter", tableName, batches, (poolClient, batch) -> {
+ List<Partition> partitionList = batch.stream().map(partition -> {
+ Path partitionPath = HadoopFSUtils.constructAbsolutePathInHadoopPath(syncConfig.getString(META_SYNC_BASE_PATH), partition);
+ String partitionScheme = partitionPath.toUri().getScheme();
+ String fullPartitionPath = StorageSchemes.HDFS.getScheme().equals(partitionScheme)
+ ? HadoopFSUtils.getDFSFullPartitionPath(syncConfig.getHadoopFileSystem(), partitionPath) : partitionPath.toString();
+ List<String> partitionValues = partitionValueExtractor.extractPartitionValuesInPath(partition);
+ StorageDescriptor partitionSd = sd.deepCopy();
+ partitionSd.setLocation(fullPartitionPath);
+ return new Partition(partitionValues, databaseName, tableName, 0, 0, partitionSd, null);
+ }).collect(Collectors.toList());
+ poolClient.alter_partitions(databaseName, tableName, partitionList, null);
+ });
+ } catch (Exception e) {
log.error("{}.{} update partition failed", databaseName, tableName, e);
throw new HoodieHiveSyncException(databaseName + "." + tableName + " update partition failed", e);
}
}
+
+ /**
+ * Dispatches partition batches either in parallel against the client pool (if
+ * configured) or sequentially against the session client. The sequential path
+ * preserves the exact behavior we had before the pool existed, including failure
+ * semantics where batch N+1 never runs if batch N throws.
+ */
+ @FunctionalInterface
+ private interface BatchAction {
+ void apply(IMetaStoreClient client, List<String> batch) throws Exception;
+ }
+
+ private void runBatches(String opName, String tableName, List<List<String>> batches, BatchAction action) throws Exception {
+ if (partitionClientPool == null) {
+ for (List<String> batch : batches) {
+ action.apply(client, batch);
+ }
+ return;
+ }
+ List<Future<Void>> futures = new ArrayList<>(batches.size());
+ for (List<String> batch : batches) {
+ futures.add(partitionClientPool.executor().submit(() ->
+ partitionClientPool.run(poolClient -> {
+ action.apply(poolClient, batch);
Review Comment:
This references HMSDDLExecutor.runBatches / registerAlterPartitionEvent(with pool), which isn't part of this PR's diff — HMSDDLExecutor is untouched here. That code belonged to an HMS-parallelization commit that was on this branch at an earlier point but has since been dropped (that work will land as its own separate PR). Resolving as not applicable to the current diff; will revisit the awaitAll-reuse suggestion when that HMS PR is opened.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3686724923
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -209,21 +217,57 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
- }
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(partitionsToDrop, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ *
+ * <p>First-error semantics come from {@link ParallelDispatch}, shared with
+ * {@code HiveDriverPool}: the first failure is rethrown, batches that have not started
+ * are stopped via the task-side abort flag, and later failures are logged at WARN.
+ */
+ private void runDropBatches(String tableName, List<List<String>> batches) throws Exception {
+ if (!metaStoreClientPool.isPresent()) {
+ for (List<String> batch : batches) {
+ applyDropBatch(metaStoreClient, tableName, batch);
+ }
+ return;
+ }
+ IMetaStoreClientPool pool = metaStoreClientPool.get();
+ pool.awaitAll(
+ pool.dispatchAll(batches, (client, batch) -> applyDropBatch(client, tableName, batch)),
+ "drop partition");
+ }
+
+ private void applyDropBatch(IMetaStoreClient client, String tableName, List<String> batch) throws Exception {
+ int dropped = 0;
+ for (String dropPartition : batch) {
+ if (HivePartitionUtil.partitionExists(client, tableName, dropPartition,
Review Comment:
🤖 This looks like the first path that calls partitionValueExtractor.extractPartitionValuesInPath(...) concurrently -- applyDropBatch invokes it (via partitionExists and getPartitionClauseForDrop) on the shared extractor from multiple pool workers at once. The ADD/TOUCH/SET_LOCATION paths build their clauses on the driver thread before dispatch, so they never hit this. The built-in extractors are stateless/immutable, but PartitionValueExtractor is user-pluggable and a custom impl holding mutable state (e.g. a SimpleDateFormat) could return a garbled clause under concurrency and drop the wrong partition. Worth confirming we're comfortable requiring extractors to be thread-safe here -- a note on the interface, or extracting the clauses on the driver thread before fan-out, would remove the assumption.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/IMetaStoreClientPool.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Pool of {@link IMetaStoreClient} instances for parallel partition sync.
+ *
+ * <p>Each pooled client wraps an independent Thrift connection to the Hive Metastore.
+ * Callers borrow a client via {@link #run(ClientAction)}, which blocks until a client
+ * is available, executes the action, and returns the client to the pool. A worker
+ * thread pool of the same size is exposed via {@link #executor()} so callers can fan
+ * out their batches to match the number of available clients.
+ *
+ * <p><b>Usage contract:</b> pool clients must be used <i>only</i> for partition-row
+ * operations — {@code add_partitions}, {@code alter_partitions}, {@code dropPartition},
+ * {@code getPartition}. Table-row operations ({@code createTable}, {@code alter_table},
+ * {@code getTable} used as the read half of a read-modify-write of table parameters)
+ * must continue to go through the session client held by
+ * {@code HoodieHiveSyncClient.client} on the sync driver thread. Mixing the two would
+ * risk lost updates on table parameters such as the last-commit-time-synced marker.
+ *
+ * <p>The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled} and
+ * is constructed for sync mode HIVEQL, where it backs the DROP path only. DROP goes
+ * through {@code IMetaStoreClient.dropPartition} (Thrift), whereas ADD/UPDATE/TOUCH go
+ * through the thread-bound Hive {@code Driver} and use {@code HiveDriverPool} instead.
Review Comment:
🤖 nit: could you rename this to `MetaStoreClientPool` (or `HoodieMetaStoreClientPool`)? The `I` prefix is a Java interface naming convention, and with `IMetaStoreClient` (an actual interface) living right next door, a reader landing here for the first time will assume this is also an interface rather than a concrete pool class.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveDriverPool.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+import org.apache.hudi.hive.HoodieHiveSyncException;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.Driver;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_DATABASE_NAME;
+
+/**
+ * Pool of Hive {@link Driver} + {@link SessionState} pairs for parallel HiveQL DDL.
+ *
+ * <p>Hive's {@code SessionState.start(state)} binds state to the calling thread's
+ * thread-local, and {@code Driver} reads from that thread-local during {@code run()}.
+ * A Driver constructed on one thread cannot be safely used from another. This pool
+ * solves that by giving each slot its own dedicated worker thread (a single-thread
+ * executor) — the Driver and SessionState are built on that thread by a bootstrap
+ * task, and all subsequent SQL for that slot runs on the same thread.
+ *
+ * <p><b>Usage contract:</b> use this pool only for partition-row DDL statements that
+ * are independent of each other and freely shuffleable across workers. Table-level
+ * statements (createTable, schema evolution, USE database) must continue to run on
+ * the session {@code Driver} held by {@code HiveQueryDDLExecutor} on the sync driver
+ * thread. The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled}
+ * and is constructed only for HiveQL sync mode.
+ */
+public class HiveDriverPool implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveDriverPool.class);
+
+ // Per-worker Driver construction has to be fast in practice (a few hundred ms
+ // for the SessionState + Driver init). A 60s ceiling per worker leaves plenty of
+ // headroom for a slow JVM warm-up but bounds the failure mode if the metastore
+ // is unreachable or Hive hangs during init.
+ private static final long BOOTSTRAP_TIMEOUT_SECONDS = 60;
+
+ private final List<Worker> workers;
+ private final int size;
+ private volatile boolean closed;
+
+ public HiveDriverPool(HiveSyncConfig config, int size) {
+ this(config, size, new DefaultDriverFactory(config));
+ }
+
+ // Package-private for tests: accepts a DriverFactory so unit tests can inject
+ // mock Driver instances without standing up a real Hive instance.
+ HiveDriverPool(HiveSyncConfig config, int size, DriverFactory factory) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ this.size = size;
+ this.workers = new ArrayList<>(size);
+ String databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
+ PoolThreadFactory threadFactory = new PoolThreadFactory();
+ List<Future<Void>> bootstrapFutures = new ArrayList<>(size);
+ try {
+ for (int i = 0; i < size; i++) {
+ Worker worker = new Worker(threadFactory);
+ workers.add(worker);
+ bootstrapFutures.add(worker.executor.submit(() -> {
+ worker.driver = factory.newDriver(databaseName);
+ return null;
+ }));
+ }
+ // Block until all bootstraps complete so we surface construction errors
+ // before any caller hands us SQL. Bounded by BOOTSTRAP_TIMEOUT_SECONDS so a
+ // hung Hive init doesn't deadlock the sync driver thread.
+ for (Future<Void> f : bootstrapFutures) {
+ f.get(BOOTSTRAP_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ }
+ } catch (Exception e) {
+ tearDown();
+ throw new HoodieException("Failed to construct HiveDriverPool of size " + size, e);
+ }
+ LOG.info("Initialized HiveDriverPool with {} workers", size);
+ }
+
+ /**
+ * Runs each given SQL on <i>every</i> worker, in order. Used for setup statements
+ * (e.g. {@code USE database}) that must establish per-thread session context
+ * before any partition statement runs. Blocks until all workers have completed
+ * the setup. Throws on first error.
+ */
+ public void runOnEachWorker(List<String> setupSqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ if (setupSqls.isEmpty()) {
+ return;
+ }
+ List<Future<?>> futures = new ArrayList<>(workers.size());
+ for (Worker worker : workers) {
+ futures.add(worker.executor.submit(() -> {
+ for (String sql : setupSqls) {
+ worker.driver.run(sql);
+ }
+ return null;
+ }));
+ }
+ awaitAll(futures);
+ }
+
+ /**
+ * Dispatches each SQL string to a worker (round-robin) and returns the list of
+ * futures. The caller is responsible for awaiting and collecting errors. SQL text
+ * is intentionally not logged per-statement here: batched TOUCH/ADD statements can
+ * be many kilobytes, and N parallel workers would multiply the log volume. See
+ * {@link #awaitAll(List)} for the per-call summary log.
+ */
+ public List<Future<?>> runAll(List<String> sqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ List<Future<?>> futures = new ArrayList<>(sqls.size());
+ for (int i = 0; i < sqls.size(); i++) {
+ String sql = sqls.get(i);
+ Worker worker = workers.get(i % workers.size());
+ futures.add(worker.executor.submit(() -> {
+ worker.driver.run(sql);
+ return null;
+ }));
+ }
+ return futures;
+ }
+
+ /**
+ * Awaits all futures and throws the first exception encountered. On first failure,
+ * cancels the remaining (not yet started) futures so workers don't keep running
+ * pointless work after a fatal error. Any errors that finished before cancellation
+ * are logged at WARN. Callers do not need per-statement results (Hive's Driver.run
+ * side-effects the metastore), so this method is void.
+ */
+ public void awaitAll(List<Future<?>> futures) {
+ long start = System.currentTimeMillis();
+ Exception firstError = null;
+ int completed = 0;
+ int cancelled = 0;
+ for (int i = 0; i < futures.size(); i++) {
+ Future<?> f = futures.get(i);
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // We cancelled this future ourselves after a prior error. Don't treat it
+ // as a new failure; just note it for the summary log.
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ cancelled += cancelRemaining(futures, i + 1);
+ }
+ } catch (ExecutionException ee) {
+ Exception cause = unwrap(ee);
+ if (firstError == null) {
+ firstError = cause;
+ cancelled += cancelRemaining(futures, i + 1);
+ } else {
+ LOG.warn("Additional SQL batch failed (suppressed in favor of first error)", cause);
+ }
+ }
+ }
+ if (firstError != null) {
+ throw new HoodieHiveSyncException("Failed in executing SQL", firstError);
+ }
+ LOG.info("Completed {} SQL statements ({} cancelled) in {} ms across {} workers",
+ completed, cancelled, System.currentTimeMillis() - start, size);
+ }
+
+ private static int cancelRemaining(List<Future<?>> futures, int fromIndex) {
+ int cancelled = 0;
+ for (int j = fromIndex; j < futures.size(); j++) {
+ // mayInterruptIfRunning=false: the worker thread is bound to a Hive Driver
+ // whose state we don't want to corrupt mid-statement. Cancel only those that
+ // haven't started yet; in-flight statements run to completion.
+ if (futures.get(j).cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ private static Exception unwrap(ExecutionException ee) {
+ Throwable cause = ee.getCause();
+ return (cause instanceof Exception) ? (Exception) cause : ee;
+ }
+
+ public int size() {
+ return size;
+ }
+
+ @Override
+ public void close() {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ tearDown();
+ }
+
+ private void tearDown() {
+ // Close each worker's Driver/SessionState on its own thread, then shut the
+ // executor down. Running close() on the bound thread keeps SessionState's
+ // thread-local cleanup correct.
+ for (Worker worker : workers) {
+ try {
+ worker.executor.submit(() -> {
+ if (worker.driver != null) {
+ try {
+ worker.driver.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled Driver", e);
+ }
+ }
+ SessionState ss = SessionState.get();
+ if (ss != null) {
+ try {
+ ss.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled SessionState", e);
+ }
+ }
+ return null;
+ }).get(30, TimeUnit.SECONDS);
+ } catch (Exception e) {
+ LOG.warn("Error during pool worker shutdown", e);
+ }
+ worker.executor.shutdown();
+ try {
+ if (!worker.executor.awaitTermination(10, TimeUnit.SECONDS)) {
+ worker.executor.shutdownNow();
+ }
+ } catch (InterruptedException ie) {
+ worker.executor.shutdownNow();
+ Thread.currentThread().interrupt();
+ }
+ }
+ workers.clear();
+ }
+
+ /**
+ * Per-slot state: a single-thread executor and the Driver bound to its thread.
+ * Driver is volatile because it is written by the bootstrap task and read by
+ * subsequent dispatch tasks on the same executor.
+ */
+ private static final class Worker {
+ final ExecutorService executor;
+ volatile Driver driver;
+
+ Worker(ThreadFactory threadFactory) {
+ this.executor = Executors.newSingleThreadExecutor(threadFactory);
+ }
+ }
+
+ @FunctionalInterface
+ interface DriverFactory {
+ Driver newDriver(String databaseName) throws Exception;
+ }
+
+ /**
+ * Builds a real Hive {@link Driver} on the calling thread. The SessionState is
+ * constructed lazily (once, on the first worker thread that builds a Driver) and
+ * shared across all worker threads — Hive uses ThreadLocal attachment, not
+ * exclusive ownership, so multiple workers calling
+ * {@code SessionState.start(sharedState)} all see the same config and scratch dir
+ * without each spending the cost of building their own SessionState (and risking
+ * resource-dir creation races during the constructor).
+ */
+ private static final class DefaultDriverFactory implements DriverFactory {
+ private final HiveConf hiveConf;
+ private volatile SessionState sharedSessionState;
+
+ DefaultDriverFactory(HiveSyncConfig config) {
+ this.hiveConf = config.getHiveConf();
+ }
+
+ @Override
+ public synchronized Driver newDriver(String databaseName) throws Exception {
+ // SessionState is shared across workers; build it once on the first call (with
+ // currentDatabase already set) and attach it to each worker's thread-local on
+ // subsequent calls. The database is a pool-wide property and never changes
+ // across workers, so setting it once at construction time is sufficient.
Review Comment:
🤖 Worth separating the two paths here: this PR's DROP fan-out goes through `IMetaStoreClient.dropPartition`, with each worker borrowing its own client from `partitionClientPool`, so it never touches `SessionState`. The shared-`SessionState` risk you're describing is specific to the ADD/UPDATE/TOUCH `Driver` path in `HiveDriverPool`, which landed in #18984 and is unchanged in this diff — so it's likely a follow-up against that merged code rather than a blocker on the DROP change. Does that match your read of how the batched statements use the session?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727349158
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7022e7d493416d600664235c3056d32ea4a1c179 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727710209
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.37%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.37% -0.90%
- Complexity 29513 29701 +188
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98021 +652
- Misses 37253 39303 +2050
- Partials 8005 8166 +161
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.36% <3.08%> (-1.42%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.11% <7.45%> (-0.74%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [203 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727591106
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `3.08483%` with `377 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.04%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 0.00% | [137 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 0.00% | [68 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 0.00% | [67 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 0.00% | [28 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 25 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|4|
>|spark-scala-tests|12|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
=============================================
- Coverage 68.26% 46.04% -22.23%
+ Complexity 29513 16882 -12631
=============================================
Files 2542 2087 -455
Lines 142627 120973 -21654
Branches 17788 16034 -1754
=============================================
- Hits 97369 55705 -41664
- Misses 37253 59114 +21861
+ Partials 8005 6154 -1851
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.47% <3.08%> (-17.30%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.50% <3.08%> (-12.34%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (-86.05%)` | :arrow_down: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-48.93%)` | :arrow_down: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-80.35%)` | :arrow_down: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-65.68%)` | :arrow_down: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
... and [1646 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5175658353
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 6c81322202e4b6cb251248da8a0d8c478fb3fe26 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787)
* 3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5219457215
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034)
* 97dc86a59f300dfdc692a725131481afba3bfc59 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178)
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 4d0ac7105b4dc361317b936bc2c85a8683f0e8aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727747813
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.58%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.58% -0.69%
- Complexity 29513 29824 +311
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98324 +955
- Misses 37253 38959 +1706
- Partials 8005 8207 +202
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.86% <3.08%> (-0.91%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.29% <7.45%> (-0.56%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [167 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5219274117
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034)
* 97dc86a59f300dfdc692a725131481afba3bfc59 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5219862849
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 97dc86a59f300dfdc692a725131481afba3bfc59 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178)
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 4d0ac7105b4dc361317b936bc2c85a8683f0e8aa Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3737015422
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/ParallelDispatch.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.common.util.VisibleForTesting;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Handle to one fan-out batch of partition work: the submitted futures plus the shared
+ * abort flag the tasks consult before running.
+ *
+ * <p>The abort flag exists because waiting on futures in <i>submission</i> order is not
+ * enough to stop queued work. If a later task fails quickly while an earlier one is slow,
+ * the awaiting thread is still parked on the earlier {@code Future.get()}, and the
+ * executor happily keeps starting every queued task in the meantime. By the time the
+ * failure is observed, most of the "not-yet-started" work has already run.
+ *
+ * <p>Two mechanisms fix that, and both are needed:
+ * <ul>
+ * <li>a {@link CountDownLatch} tripped by the <i>first</i> abort, so the awaiting
+ * thread wakes on failure rather than on its turn in submission order;</li>
+ * <li>a task-side {@link #aborted()} check on entry, because cancelling from the
+ * awaiting thread is inherently late — a worker can pull its next task off the
+ * queue at any moment.</li>
+ * </ul>
+ *
+ * <p>The failing task also records its own throwable as it aborts, so the reported root
+ * cause is the failure that stopped the batch. Picking the error by scanning futures in
+ * submission order would instead surface whichever failure happens to sit earliest in the
+ * list, which is not necessarily the one that aborted the run.
+ *
+ * <p>Shared by {@link HiveDriverPool} (Hive {@code Driver} statements) and
+ * {@link IMetaStoreClientPool} (Thrift {@code dropPartition} batches), which fan out over
+ * different execution models but need identical abort-on-first-error semantics.
+ */
+public final class ParallelDispatch {
+
+ private final List<Future<?>> futures;
+ private final int total;
+ private final AtomicInteger settled = new AtomicInteger(0);
+ private final AtomicBoolean aborted = new AtomicBoolean(false);
+ private final AtomicReference<Throwable> abortCause = new AtomicReference<>();
+ private final CountDownLatch done = new CountDownLatch(1);
+ private volatile boolean sealed;
+
+ ParallelDispatch(int total) {
+ this.total = total;
+ this.futures = new ArrayList<>(total);
+ }
+
+ void add(Future<?> future) {
+ futures.add(future);
+ }
+
+ // Called once submission finishes. A task that settles before the last submit
+ // would otherwise see settled < total and never trip the latch, so re-check here.
+ void sealed() {
+ sealed = true;
+ signalIfComplete();
+ }
+
+ boolean aborted() {
+ return aborted.get();
+ }
+
+ void abort() {
+ aborted.set(true);
+ done.countDown();
+ }
+
+ // Records the failure that triggered the abort, first writer wins. Selecting the error
+ // by walking futures in submission order instead would report whichever failure sits
+ // earliest in the list, not the one that actually stopped the batch: a fast failure at
+ // index 1 aborts the run, and a slow index 0 that fails later would take its place.
+ void abort(Throwable cause) {
+ abortCause.compareAndSet(null, cause);
+ abort();
+ }
+
+ void taskSettled() {
+ settled.incrementAndGet();
+ signalIfComplete();
+ }
+
+ private void signalIfComplete() {
+ if (sealed && settled.get() >= total) {
+ done.countDown();
+ }
+ }
+
+ void awaitSettledOrAborted() {
+ if (total == 0) {
+ return;
+ }
+ try {
+ done.await();
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ aborted.set(true);
+ }
+ }
+
+ // mayInterruptIfRunning=false: a worker may be mid-statement against a Hive Driver or
+ // a Thrift client, and we don't want to tear that down partway. Cancel only tasks that
+ // haven't started; in-flight work runs to completion.
+ int cancelPending() {
+ int cancelled = 0;
+ for (Future<?> f : futures) {
+ if (f.cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ List<Future<?>> futures() {
+ return futures;
+ }
+
+ /**
+ * Wraps {@code body} so it observes this batch's abort flag: it skips itself if a
+ * sibling has already failed, trips the flag if it fails, and always records that it
+ * settled so the awaiting thread can be released.
+ */
+ Callable<Void> guard(Callable<Void> body, String skipMessage) {
+ return () -> {
+ if (aborted()) {
+ throw new CancellationException(skipMessage);
+ }
+ try {
+ return body.call();
+ } catch (Throwable t) {
+ abort(t);
+ throw t;
+ } finally {
+ taskSettled();
+ }
+ };
+ }
+
+ /**
+ * Waits for the batch to settle (or abort), cancels whatever had not started, and
+ * returns the outcome. Errors are observed in <i>completion</i> order, not submission
+ * order, so a failure on a fast worker stops the other queues even while a slow worker
+ * is still mid-statement.
+ */
+ Outcome awaitOutcome() {
+ awaitSettledOrAborted();
+ int cancelled = cancelPending();
+
+ // Seeded from the task that tripped the abort, so the reported root cause is the
+ // failure that actually stopped the batch rather than the lowest-indexed one.
+ Exception firstError = asException(abortCause.get());
+ int completed = 0;
+ List<Exception> suppressed = new ArrayList<>();
+ for (Future<?> f : futures) {
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // Either we cancelled it before it started, or the task itself observed the
+ // abort flag and bailed. Not a new failure; just note it for the summary.
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ }
+ } catch (ExecutionException ee) {
+ Exception cause = unwrap(ee);
+ if (cause instanceof CancellationException) {
+ cancelled++;
+ } else if (firstError == null) {
+ firstError = cause;
+ } else if (cause != firstError) {
Review Comment:
Good catch — confirmed and fixed in 97dc86a.
You traced it exactly: `guard` catches `Throwable`, so an `Error` is what trips the abort and lands in `abortCause`, but it cannot be reported as-is through an `Exception`-typed `Outcome`, so `asException()` wraps it in a *new* `RuntimeException`. The identity check then compared the unwrapped cause against that wrapper, which can never match — so the one real failure was reported twice: once as the root cause, once as suppressed.
The fix keeps the raw `Throwable` that tripped the abort and compares against that instead:
```java
Throwable abortedBy = abortCause.get();
Exception firstError = asException(abortedBy);
...
} else if (ee.getCause() != abortedBy) {
suppressed.add(cause);
}
```
One deliberate deviation worth flagging: `HiveDriverPool` on master (#19391) solves the same problem by typing `firstError` as a raw `Throwable`. I kept `Outcome.firstError()` as an `Exception` here, because `IMetaStoreClientPool.awaitAll()` does `throw outcome.firstError();` under `throws Exception` — retyping would force an unrelated signature change up the DROP path for no behavioural gain. Comparing against `abortedBy` gets the same result without touching the API.
Added `anErrorThatAbortsTheBatchIsNotAlsoReportedAsSuppressed`, verified failing before the fix (`expected: <true> but was: <false>`). It asserts on the `Outcome` rather than on `awaitAll()` on purpose — `awaitAll` only *logs* the suppressed list, so a duplicate there is invisible to the thrown exception and the test would not have been able to fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3669747523
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -209,21 +219,96 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
- }
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(partitionsToDrop, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ *
+ * <p>First-error semantics match {@code HiveDriverPool.awaitAll}: the first failure is
+ * rethrown, not-yet-started batches are cancelled, and later failures are logged at WARN.
+ */
+ private void runDropBatches(String tableName, List<List<String>> batches) throws Exception {
+ if (!metaStoreClientPool.isPresent()) {
+ for (List<String> batch : batches) {
+ applyDropBatch(metaStoreClient, tableName, batch);
+ }
+ return;
+ }
+ IMetaStoreClientPool pool = metaStoreClientPool.get();
+ List<Future<Void>> futures = new ArrayList<>(batches.size());
+ for (List<String> batch : batches) {
+ futures.add(pool.executor().submit(() ->
+ pool.run(poolClient -> {
+ applyDropBatch(poolClient, tableName, batch);
+ return null;
+ })
+ ));
+ }
+
+ Exception firstError = null;
+ int cancelled = 0;
+ for (Future<Void> f : futures) {
+ // Once a batch has failed the sync is going to abort anyway, so stop handing
+ // more DROPs to the metastore. mayInterruptIfRunning=false mirrors
+ // HiveDriverPool.awaitAll: a batch already mid-flight runs to completion rather
+ // than leaving the partition list half-dropped at an arbitrary point.
+ if (firstError != null && f.cancel(false)) {
+ cancelled++;
+ continue;
+ }
+ try {
+ f.get();
+ } catch (CancellationException ce) {
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ }
+ } catch (Exception e) {
+ if (firstError == null) {
+ firstError = e;
+ } else {
+ log.warn("Additional drop batch failed on {} (suppressed in favor of first error)", tableName, e);
+ }
+ }
+ }
+ if (firstError != null) {
+ log.error("Drop partition dispatch on {} aborted after first failure ({} batches cancelled)",
+ tableName, cancelled);
+ throw firstError;
+ }
+ }
+
+ private void applyDropBatch(IMetaStoreClient poolClient, String tableName, List<String> batch) throws Exception {
+ int dropped = 0;
Review Comment:
Fixed in `6c81322` — renamed to `client`, since the sequential path does pass the session `metaStoreClient` here.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java:
##########
@@ -132,23 +196,81 @@ public void dropPartitionsToTable(String tableName, List<String> partitionsToDro
log.info("Drop partitions {} on {}", partitionsToDrop.size(), tableName);
try {
- for (String dropPartition : partitionsToDrop) {
- if (HivePartitionUtil.partitionExists(metaStoreClient, tableName, dropPartition, partitionValueExtractor,
- config)) {
- String partitionClause =
- HivePartitionUtil.getPartitionClauseForDrop(dropPartition, partitionValueExtractor, config);
- metaStoreClient.dropPartition(databaseName, tableName, partitionClause, false);
- }
- log.info("Drop partition {} on {}", dropPartition, tableName);
- }
+ int batchSyncPartitionNum = config.getIntOrDefault(HIVE_BATCH_SYNC_PARTITION_NUM);
+ List<List<String>> batches = CollectionUtils.batches(partitionsToDrop, batchSyncPartitionNum);
+ runDropBatches(tableName, batches);
} catch (Exception e) {
log.error("{} drop partition failed", tableId(databaseName, tableName), e);
throw new HoodieHiveSyncException(tableId(databaseName, tableName) + " drop partition failed", e);
}
}
+ /**
+ * Drops partitions one batch at a time. When {@link #metaStoreClientPool} is present,
+ * batches fan out across the pool's worker threads (each borrowing an independent
+ * IMetaStoreClient); otherwise batches are dispatched sequentially against the
+ * session client. Hive has no batch-drop primitive that matches dropPartition's
+ * semantics, so each worker still iterates its chunk one partition at a time — the
+ * win is fanning chunks across independent Thrift clients.
+ */
+ private void runDropBatches(String tableName, List<List<String>> batches) throws Exception {
Review Comment:
Revisited now that #18984 has merged and this branch is rebased on it: done in `8721c9a`. Rather than adding a parallel `awaitAll` to `IMetaStoreClientPool`, the shared coordination is extracted into `ParallelDispatch` and both pools build on it, so there is one implementation instead of two similar ones. This also turned out to fix a real bug in the drop path — see the thread on `runDropBatches`.
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveDriverPool.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+import org.apache.hudi.hive.HoodieHiveSyncException;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.Driver;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_DATABASE_NAME;
+
+/**
+ * Pool of Hive {@link Driver} + {@link SessionState} pairs for parallel HiveQL DDL.
+ *
+ * <p>Hive's {@code SessionState.start(state)} binds state to the calling thread's
+ * thread-local, and {@code Driver} reads from that thread-local during {@code run()}.
+ * A Driver constructed on one thread cannot be safely used from another. This pool
+ * solves that by giving each slot its own dedicated worker thread (a single-thread
+ * executor) — the Driver and SessionState are built on that thread by a bootstrap
+ * task, and all subsequent SQL for that slot runs on the same thread.
+ *
+ * <p><b>Usage contract:</b> use this pool only for partition-row DDL statements that
+ * are independent of each other and freely shuffleable across workers. Table-level
+ * statements (createTable, schema evolution, USE database) must continue to run on
+ * the session {@code Driver} held by {@code HiveQueryDDLExecutor} on the sync driver
+ * thread. The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled}
+ * and is constructed only for HiveQL sync mode.
+ */
+public class HiveDriverPool implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveDriverPool.class);
+
+ // Per-worker Driver construction has to be fast in practice (a few hundred ms
+ // for the SessionState + Driver init). A 60s ceiling per worker leaves plenty of
+ // headroom for a slow JVM warm-up but bounds the failure mode if the metastore
+ // is unreachable or Hive hangs during init.
+ private static final long BOOTSTRAP_TIMEOUT_SECONDS = 60;
+
+ private final List<Worker> workers;
+ private final int size;
+ private volatile boolean closed;
+
+ public HiveDriverPool(HiveSyncConfig config, int size) {
+ this(config, size, new DefaultDriverFactory(config));
+ }
+
+ // Package-private for tests: accepts a DriverFactory so unit tests can inject
+ // mock Driver instances without standing up a real Hive instance.
+ HiveDriverPool(HiveSyncConfig config, int size, DriverFactory factory) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ this.size = size;
+ this.workers = new ArrayList<>(size);
+ String databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
+ PoolThreadFactory threadFactory = new PoolThreadFactory();
+ List<Future<Void>> bootstrapFutures = new ArrayList<>(size);
+ try {
+ for (int i = 0; i < size; i++) {
+ Worker worker = new Worker(threadFactory);
+ workers.add(worker);
+ bootstrapFutures.add(worker.executor.submit(() -> {
+ worker.driver = factory.newDriver(databaseName);
+ return null;
+ }));
+ }
+ // Block until all bootstraps complete so we surface construction errors
+ // before any caller hands us SQL. Bounded by BOOTSTRAP_TIMEOUT_SECONDS so a
+ // hung Hive init doesn't deadlock the sync driver thread.
+ for (Future<Void> f : bootstrapFutures) {
+ f.get(BOOTSTRAP_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ }
+ } catch (Exception e) {
+ tearDown();
+ throw new HoodieException("Failed to construct HiveDriverPool of size " + size, e);
+ }
+ LOG.info("Initialized HiveDriverPool with {} workers", size);
+ }
+
+ /**
+ * Runs each given SQL on <i>every</i> worker, in order. Used for setup statements
+ * (e.g. {@code USE database}) that must establish per-thread session context
+ * before any partition statement runs. Blocks until all workers have completed
+ * the setup. Throws on first error.
+ */
+ public void runOnEachWorker(List<String> setupSqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ if (setupSqls.isEmpty()) {
+ return;
+ }
+ List<Future<?>> futures = new ArrayList<>(workers.size());
+ for (Worker worker : workers) {
+ futures.add(worker.executor.submit(() -> {
+ for (String sql : setupSqls) {
+ worker.driver.run(sql);
+ }
+ return null;
+ }));
+ }
+ awaitAll(futures);
+ }
+
+ /**
+ * Dispatches each SQL string to a worker (round-robin) and returns the list of
+ * futures. The caller is responsible for awaiting and collecting errors. SQL text
+ * is intentionally not logged per-statement here: batched TOUCH/ADD statements can
+ * be many kilobytes, and N parallel workers would multiply the log volume. See
+ * {@link #awaitAll(List)} for the per-call summary log.
+ */
+ public List<Future<?>> runAll(List<String> sqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ List<Future<?>> futures = new ArrayList<>(sqls.size());
+ for (int i = 0; i < sqls.size(); i++) {
+ String sql = sqls.get(i);
+ Worker worker = workers.get(i % workers.size());
+ futures.add(worker.executor.submit(() -> {
+ worker.driver.run(sql);
+ return null;
+ }));
+ }
+ return futures;
+ }
+
+ /**
+ * Awaits all futures and throws the first exception encountered. On first failure,
+ * cancels the remaining (not yet started) futures so workers don't keep running
+ * pointless work after a fatal error. Any errors that finished before cancellation
+ * are logged at WARN. Callers do not need per-statement results (Hive's Driver.run
+ * side-effects the metastore), so this method is void.
+ */
+ public void awaitAll(List<Future<?>> futures) {
+ long start = System.currentTimeMillis();
+ Exception firstError = null;
+ int completed = 0;
+ int cancelled = 0;
+ for (int i = 0; i < futures.size(); i++) {
+ Future<?> f = futures.get(i);
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // We cancelled this future ourselves after a prior error. Don't treat it
+ // as a new failure; just note it for the summary log.
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ cancelled += cancelRemaining(futures, i + 1);
+ }
+ } catch (ExecutionException ee) {
+ Exception cause = unwrap(ee);
+ if (firstError == null) {
+ firstError = cause;
+ cancelled += cancelRemaining(futures, i + 1);
+ } else {
+ LOG.warn("Additional SQL batch failed (suppressed in favor of first error)", cause);
+ }
+ }
+ }
+ if (firstError != null) {
+ throw new HoodieHiveSyncException("Failed in executing SQL", firstError);
+ }
+ LOG.info("Completed {} SQL statements ({} cancelled) in {} ms across {} workers",
+ completed, cancelled, System.currentTimeMillis() - start, size);
+ }
+
+ private static int cancelRemaining(List<Future<?>> futures, int fromIndex) {
+ int cancelled = 0;
+ for (int j = fromIndex; j < futures.size(); j++) {
+ // mayInterruptIfRunning=false: the worker thread is bound to a Hive Driver
+ // whose state we don't want to corrupt mid-statement. Cancel only those that
+ // haven't started yet; in-flight statements run to completion.
+ if (futures.get(j).cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ private static Exception unwrap(ExecutionException ee) {
+ Throwable cause = ee.getCause();
+ return (cause instanceof Exception) ? (Exception) cause : ee;
+ }
+
+ public int size() {
+ return size;
+ }
+
+ @Override
+ public void close() {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ tearDown();
+ }
+
+ private void tearDown() {
+ // Close each worker's Driver/SessionState on its own thread, then shut the
+ // executor down. Running close() on the bound thread keeps SessionState's
+ // thread-local cleanup correct.
+ for (Worker worker : workers) {
+ try {
+ worker.executor.submit(() -> {
+ if (worker.driver != null) {
+ try {
+ worker.driver.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled Driver", e);
+ }
+ }
+ SessionState ss = SessionState.get();
+ if (ss != null) {
+ try {
+ ss.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled SessionState", e);
+ }
+ }
+ return null;
+ }).get(30, TimeUnit.SECONDS);
+ } catch (Exception e) {
+ LOG.warn("Error during pool worker shutdown", e);
+ }
+ worker.executor.shutdown();
+ try {
+ if (!worker.executor.awaitTermination(10, TimeUnit.SECONDS)) {
+ worker.executor.shutdownNow();
+ }
+ } catch (InterruptedException ie) {
+ worker.executor.shutdownNow();
+ Thread.currentThread().interrupt();
+ }
+ }
+ workers.clear();
+ }
+
+ /**
+ * Per-slot state: a single-thread executor and the Driver bound to its thread.
+ * Driver is volatile because it is written by the bootstrap task and read by
+ * subsequent dispatch tasks on the same executor.
+ */
+ private static final class Worker {
+ final ExecutorService executor;
+ volatile Driver driver;
+
+ Worker(ThreadFactory threadFactory) {
+ this.executor = Executors.newSingleThreadExecutor(threadFactory);
+ }
+ }
+
+ @FunctionalInterface
+ interface DriverFactory {
+ Driver newDriver(String databaseName) throws Exception;
+ }
+
+ /**
+ * Builds a real Hive {@link Driver} on the calling thread. The SessionState is
+ * constructed lazily (once, on the first worker thread that builds a Driver) and
+ * shared across all worker threads — Hive uses ThreadLocal attachment, not
+ * exclusive ownership, so multiple workers calling
+ * {@code SessionState.start(sharedState)} all see the same config and scratch dir
+ * without each spending the cost of building their own SessionState (and risking
+ * resource-dir creation races during the constructor).
+ */
+ private static final class DefaultDriverFactory implements DriverFactory {
+ private final HiveConf hiveConf;
+ private volatile SessionState sharedSessionState;
+
+ DefaultDriverFactory(HiveSyncConfig config) {
+ this.hiveConf = config.getHiveConf();
+ }
+
+ @Override
+ public synchronized Driver newDriver(String databaseName) throws Exception {
+ // SessionState is shared across workers; build it once on the first call (with
+ // currentDatabase already set) and attach it to each worker's thread-local on
+ // subsequent calls. The database is a pool-wide property and never changes
+ // across workers, so setting it once at construction time is sufficient.
Review Comment:
Now resolved: this branch is rebased onto master with #18984 merged, so each `HiveDriverPool` worker owns its own `SessionState` and `HiveConf`. hudi-agent independently confirmed this on the latest push.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727365712
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 44.67%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 41 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 33 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-client-hadoop-common|1|0|
>|spark-java-tests|18|0|
>|spark-scala-tests|12|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
=============================================
- Coverage 68.26% 44.67% -23.60%
+ Complexity 29513 8503 -21010
=============================================
Files 2542 1204 -1338
Lines 142627 62882 -79745
Branches 17788 6892 -10896
=============================================
- Hits 97369 28094 -69275
+ Misses 37253 31644 -5609
+ Partials 8005 3144 -4861
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 2152 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727789013
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.64%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.64% -0.63%
- Complexity 29513 29858 +345
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98411 +1042
- Misses 37253 38857 +1604
- Partials 8005 8222 +217
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.99% <3.08%> (-0.79%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.37% <7.45%> (-0.48%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [160 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727462712
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 53.93%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 32 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|0|
>|spark-scala-tests|12|0|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
=============================================
- Coverage 68.26% 53.93% -14.34%
+ Complexity 29513 12494 -17019
=============================================
Files 2542 1436 -1106
Lines 142627 72287 -70340
Branches 17788 8334 -9454
=============================================
- Hits 97369 38986 -58383
+ Misses 37253 29752 -7501
+ Partials 8005 3549 -4456
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
[see 1909 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4784087323
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e3c1fdec6c9dd95e339a2027ca5be20002dea8c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5066898782
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 5e3c1fdec6c9dd95e339a2027ca5be20002dea8c Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894)
* 9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5219399082
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034)
* 97dc86a59f300dfdc692a725131481afba3bfc59 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178)
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5222324190
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
}, {
"hash" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187",
"triggerID" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 038e6c842e18130a64552da6ff5aef46864b3cf0 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3763124242
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveMetaStoreClientPool.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Pool of {@link IMetaStoreClient} instances for parallel partition sync.
+ *
+ * <p>Each pooled client wraps an independent Thrift connection to the Hive Metastore.
+ * Callers borrow a client via {@link #run(ClientAction)}, which blocks until a client
+ * is available, executes the action, and returns the client to the pool. A worker
+ * thread pool of the same size is exposed via {@link #executor()} so callers can fan
+ * out their batches to match the number of available clients.
+ *
+ * <p><b>Usage contract:</b> pool clients must be used <i>only</i> for partition-row
+ * operations — {@code add_partitions}, {@code alter_partitions}, {@code dropPartition},
+ * {@code getPartition}. Table-row operations ({@code createTable}, {@code alter_table},
+ * {@code getTable} used as the read half of a read-modify-write of table parameters)
+ * must continue to go through the session client held by
+ * {@code HoodieHiveSyncClient.client} on the sync driver thread. Mixing the two would
+ * risk lost updates on table parameters such as the last-commit-time-synced marker.
+ *
+ * <p>The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled} and
+ * is constructed for sync mode HIVEQL, where it backs the DROP path only. DROP goes
+ * through {@code IMetaStoreClient.dropPartition} (Thrift), whereas ADD/UPDATE/TOUCH go
+ * through the thread-bound Hive {@code Driver} and use {@code HiveDriverPool} instead.
+ */
+public class HiveMetaStoreClientPool implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveMetaStoreClientPool.class);
+
+ private final ArrayBlockingQueue<IMetaStoreClient> available;
+ private final List<IMetaStoreClient> all;
+ private final ExecutorService executor;
+ private final int size;
+ private volatile boolean closed;
+
+ public HiveMetaStoreClientPool(HiveSyncConfig config, int size) {
+ this(buildClients(config, size), size);
+ }
+
+ // Package-private for tests: accepts a pre-built list of clients so we can
+ // exercise borrow/return/close semantics without a live metastore.
+ HiveMetaStoreClientPool(List<IMetaStoreClient> clients, int size) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ if (clients.size() != size) {
+ throw new IllegalArgumentException("Expected " + size + " clients, got " + clients.size());
+ }
+ this.size = size;
+ this.available = new ArrayBlockingQueue<>(size);
+ this.all = new ArrayList<>(clients);
+ this.available.addAll(clients);
+ this.executor = Executors.newFixedThreadPool(size, new PoolThreadFactory());
+ LOG.info("Initialized IMetaStoreClient pool with {} clients", size);
+ }
+
+ private static List<IMetaStoreClient> buildClients(HiveSyncConfig config, int size) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ HiveConf hiveConf = config.getHiveConf();
+ List<IMetaStoreClient> clients = new ArrayList<>(size);
+ try {
Review Comment:
Kept the duplicate here, but you are right that it needed explaining — added a comment and a test in a9d9443.
The premise holds for `size == 0` but not for negatives. The public constructor is `this(buildClients(config, size), size)`, so `buildClients` is evaluated *before* the constructor validates anything. Two consequences:
- With the guard removed and `size = -1`, `new ArrayList<>(-1)` throws first, and the caller sees `Illegal Capacity: -1` instead of `Pool size must be >= 1`.
- The guard also fails before the loop opens any Thrift connection, rather than after.
So it is deliberate rather than drift. I documented that at the call site so the next reader does not have to re-derive it:
```java
// Duplicated with the constructor deliberately: this runs first (the public
// constructor evaluates buildClients before delegating), so it both fails before any
// Thrift connection is opened and keeps the message meaningful for a negative size,
// which would otherwise surface as ArrayList's "Illegal Capacity".
```
Also added `invalidPoolSizeIsRejectedWithAClearMessage`, covering `0` and `-1` and asserting on the message — there was no coverage for invalid size at all, so the drift you were worried about could genuinely have happened unnoticed. If either guard is dropped later, that test fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5220364185
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 4d0ac7105b4dc361317b936bc2c85a8683f0e8aa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3710168826
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/ParallelDispatch.java:
##########
@@ -0,0 +1,267 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.common.util.VisibleForTesting;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Handle to one fan-out batch of partition work: the submitted futures plus the shared
+ * abort flag the tasks consult before running.
+ *
+ * <p>The abort flag exists because waiting on futures in <i>submission</i> order is not
+ * enough to stop queued work. If a later task fails quickly while an earlier one is slow,
+ * the awaiting thread is still parked on the earlier {@code Future.get()}, and the
+ * executor happily keeps starting every queued task in the meantime. By the time the
+ * failure is observed, most of the "not-yet-started" work has already run.
+ *
+ * <p>Two mechanisms fix that, and both are needed:
+ * <ul>
+ * <li>a {@link CountDownLatch} tripped by the <i>first</i> abort, so the awaiting
+ * thread wakes on failure rather than on its turn in submission order;</li>
+ * <li>a task-side {@link #aborted()} check on entry, because cancelling from the
+ * awaiting thread is inherently late — a worker can pull its next task off the
+ * queue at any moment.</li>
+ * </ul>
+ *
+ * <p>The failing task also records its own throwable as it aborts, so the reported root
+ * cause is the failure that stopped the batch. Picking the error by scanning futures in
+ * submission order would instead surface whichever failure happens to sit earliest in the
+ * list, which is not necessarily the one that aborted the run.
+ *
+ * <p>Shared by {@link HiveDriverPool} (Hive {@code Driver} statements) and
+ * {@link IMetaStoreClientPool} (Thrift {@code dropPartition} batches), which fan out over
+ * different execution models but need identical abort-on-first-error semantics.
+ */
+public final class ParallelDispatch {
+
+ private final List<Future<?>> futures;
+ private final int total;
+ private final AtomicInteger settled = new AtomicInteger(0);
+ private final AtomicBoolean aborted = new AtomicBoolean(false);
+ private final AtomicReference<Throwable> abortCause = new AtomicReference<>();
+ private final CountDownLatch done = new CountDownLatch(1);
+ private volatile boolean sealed;
+
+ ParallelDispatch(int total) {
+ this.total = total;
+ this.futures = new ArrayList<>(total);
+ }
+
+ void add(Future<?> future) {
+ futures.add(future);
+ }
+
+ // Called once submission finishes. A task that settles before the last submit
+ // would otherwise see settled < total and never trip the latch, so re-check here.
+ void sealed() {
+ sealed = true;
+ signalIfComplete();
+ }
+
+ boolean aborted() {
+ return aborted.get();
+ }
+
+ void abort() {
+ aborted.set(true);
+ done.countDown();
+ }
+
+ // Records the failure that triggered the abort, first writer wins. Selecting the error
+ // by walking futures in submission order instead would report whichever failure sits
+ // earliest in the list, not the one that actually stopped the batch: a fast failure at
+ // index 1 aborts the run, and a slow index 0 that fails later would take its place.
+ void abort(Throwable cause) {
+ abortCause.compareAndSet(null, cause);
+ abort();
+ }
+
+ void taskSettled() {
+ settled.incrementAndGet();
+ signalIfComplete();
+ }
+
+ private void signalIfComplete() {
+ if (sealed && settled.get() >= total) {
+ done.countDown();
+ }
+ }
+
+ void awaitSettledOrAborted() {
+ if (total == 0) {
+ return;
+ }
+ try {
+ done.await();
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ aborted.set(true);
+ }
+ }
+
+ // mayInterruptIfRunning=false: a worker may be mid-statement against a Hive Driver or
+ // a Thrift client, and we don't want to tear that down partway. Cancel only tasks that
+ // haven't started; in-flight work runs to completion.
+ int cancelPending() {
+ int cancelled = 0;
+ for (Future<?> f : futures) {
+ if (f.cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ List<Future<?>> futures() {
+ return futures;
+ }
+
+ /**
+ * Wraps {@code body} so it observes this batch's abort flag: it skips itself if a
+ * sibling has already failed, trips the flag if it fails, and always records that it
+ * settled so the awaiting thread can be released.
+ */
+ Callable<Void> guard(Callable<Void> body, String skipMessage) {
+ return () -> {
+ if (aborted()) {
+ throw new CancellationException(skipMessage);
+ }
+ try {
+ return body.call();
+ } catch (Throwable t) {
+ abort(t);
+ throw t;
+ } finally {
+ taskSettled();
+ }
+ };
+ }
+
+ /**
+ * Waits for the batch to settle (or abort), cancels whatever had not started, and
+ * returns the outcome. Errors are observed in <i>completion</i> order, not submission
+ * order, so a failure on a fast worker stops the other queues even while a slow worker
+ * is still mid-statement.
+ */
+ Outcome awaitOutcome() {
+ awaitSettledOrAborted();
+ int cancelled = cancelPending();
+
+ // Seeded from the task that tripped the abort, so the reported root cause is the
+ // failure that actually stopped the batch rather than the lowest-indexed one.
+ Exception firstError = asException(abortCause.get());
+ int completed = 0;
+ List<Exception> suppressed = new ArrayList<>();
+ for (Future<?> f : futures) {
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // Either we cancelled it before it started, or the task itself observed the
+ // abort flag and bailed. Not a new failure; just note it for the summary.
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ }
+ } catch (ExecutionException ee) {
+ Exception cause = unwrap(ee);
+ if (cause instanceof CancellationException) {
+ cancelled++;
+ } else if (firstError == null) {
+ firstError = cause;
+ } else if (cause != firstError) {
Review Comment:
🤖 Nice fix on seeding the root cause. One gap: the identity guard here only holds when the aborting task threw an `Exception`. `guard` catches `Throwable` (line 158), so if a task throws an `Error`, `asException()` (line 215) wraps it in a *new* `RuntimeException` as `firstError`, while `unwrap(ee)` returns the `ExecutionException` (an `Error` isn't an `Exception`) — so `cause != firstError` is true and that same root failure also lands in `suppressed`, i.e. reported twice. Is the non-Exception case worth handling, or are `Error`s considered out of scope here?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727660502
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 62.51%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 15 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|8|
>|spark-scala-tests|12|7|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 62.51% -5.76%
+ Complexity 29513 27558 -1955
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
- Hits 97369 90946 -6423
- Misses 37253 46767 +9514
+ Partials 8005 7777 -228
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `43.58% <3.08%> (-5.19%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.21% <3.08%> (-12.63%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `49.87% <67.85%> (+0.94%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [506 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727593992
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `3.08483%` with `377 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 46.04%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 0.00% | [137 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 0.00% | [68 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 0.00% | [67 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 0.00% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 0.00% | [28 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | 0.00% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (8933224) and HEAD (7022e7d). Click for more details.
>
> <details><summary>HEAD has 25 uploads less than BASE</summary>
>
>| Flag | BASE (8933224) | HEAD (7022e7d) |
>|------|------|------|
>|spark-java-tests|18|4|
>|spark-scala-tests|12|3|
>|common-and-other-modules|1|0|
>|utilities|1|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
=============================================
- Coverage 68.26% 46.04% -22.23%
+ Complexity 29513 16882 -12631
=============================================
Files 2542 2087 -455
Lines 142627 120973 -21654
Branches 17788 16034 -1754
=============================================
- Hits 97369 55705 -41664
- Misses 37253 59114 +21861
+ Partials 8005 6154 -1851
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `31.47% <3.08%> (-17.30%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `32.50% <3.08%> (-12.34%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `0.00% <0.00%> (-86.05%)` | :arrow_down: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `0.00% <0.00%> (-48.93%)` | :arrow_down: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-80.35%)` | :arrow_down: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `0.00% <0.00%> (-65.68%)` | :arrow_down: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `0.00% <0.00%> (ø)` | |
... and [1646 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727735660
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.49%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.49% -0.78%
- Complexity 29513 29770 +257
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98194 +825
- Misses 37253 39107 +1854
- Partials 8005 8189 +184
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.85% <3.08%> (-0.93%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.11% <7.45%> (-0.74%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [186 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4727723321
## [Codecov](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `80.20566%` with `77 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 67.40%. Comparing base ([`8933224`](https://app.codecov.io/gh/apache/hudi/commit/89332241ea0b42671dae81b5ef06cb7ab9fad64b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`7022e7d`](https://app.codecov.io/gh/apache/hudi/commit/7022e7d493416d600664235c3056d32ea4a1c179?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 42 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | 82.48% | [18 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | 72.05% | [16 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | 77.61% | [9 Missing and 6 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | 84.12% | [7 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | 67.85% | [8 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #19033 +/- ##
============================================
- Coverage 68.26% 67.40% -0.87%
- Complexity 29513 29722 +209
============================================
Files 2542 2564 +22
Lines 142627 145490 +2863
Branches 17788 18371 +583
============================================
+ Hits 97369 98071 +702
- Misses 37253 39238 +1985
- Partials 8005 8181 +176
```
| [Flag](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [common-and-other-modules](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.87% <80.20%> (+0.08%)` | :arrow_up: |
| [hadoop-mr-java-client](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.67% <ø> (-0.08%)` | :arrow_down: |
| [spark-client-hadoop-common](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `48.28% <ø> (+0.24%)` | :arrow_up: |
| [spark-java-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `47.55% <3.08%> (-1.22%)` | :arrow_down: |
| [spark-scala-tests](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `44.11% <7.45%> (-0.74%)` | :arrow_down: |
| [utilities](https://app.codecov.io/gh/apache/hudi/pull/19033/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `37.09% <4.62%> (-0.16%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi/pull/19033?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/hudi/hive/HiveSyncConfigHolder.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHiveSyncConfigHolder.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSGl2ZVN5bmNDb25maWdIb2xkZXIuamF2YQ==) | `99.21% <100.00%> (+0.08%)` | :arrow_up: |
| [...rg/apache/hudi/hive/ddl/QueryBasedDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FQueryBasedDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL1F1ZXJ5QmFzZWRERExFeGVjdXRvci5qYXZh) | `87.12% <100.00%> (+1.07%)` | :arrow_up: |
| [...ava/org/apache/hudi/hive/HoodieHiveSyncClient.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2FHoodieHiveSyncClient.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvSG9vZGllSGl2ZVN5bmNDbGllbnQuamF2YQ==) | `50.12% <67.85%> (+1.20%)` | :arrow_up: |
| [.../java/org/apache/hudi/hive/ddl/HMSDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHMSDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hNU0RETEV4ZWN1dG9yLmphdmE=) | `80.86% <84.12%> (+0.51%)` | :arrow_up: |
| [...org/apache/hudi/hive/ddl/HiveQueryDDLExecutor.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Fddl%2FHiveQueryDDLExecutor.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvZGRsL0hpdmVRdWVyeURETEV4ZWN1dG9yLmphdmE=) | `70.40% <77.61%> (+4.72%)` | :arrow_up: |
| [...rg/apache/hudi/hive/util/IMetaStoreClientPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FIMetaStoreClientPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9JTWV0YVN0b3JlQ2xpZW50UG9vbC5qYXZh) | `72.05% <72.05%> (ø)` | |
| [...java/org/apache/hudi/hive/util/HiveDriverPool.java](https://app.codecov.io/gh/apache/hudi/pull/19033?src=pr&el=tree&filepath=hudi-sync%2Fhudi-hive-sync%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fhudi%2Fhive%2Futil%2FHiveDriverPool.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-aHVkaS1zeW5jL2h1ZGktaGl2ZS1zeW5jL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9odWRpL2hpdmUvdXRpbC9IaXZlRHJpdmVyUG9vbC5qYXZh) | `82.48% <82.48%> (ø)` | |
... and [197 files with indirect coverage changes](https://app.codecov.io/gh/apache/hudi/pull/19033/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-4783495752
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "SUCCESS",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 7022e7d493416d600664235c3056d32ea4a1c179 Azure: [SUCCESS](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784)
* 5e3c1fdec6c9dd95e339a2027ca5be20002dea8c Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3463342744
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -124,7 +137,10 @@ public HoodieHiveSyncClient(HiveSyncConfig config, HoodieTableMetaClient metaCli
ddlExecutor = new HMSDDLExecutor(config, this.client);
break;
case HIVEQL:
- ddlExecutor = new HiveQueryDDLExecutor(config, this.client);
+ this.partitionDriverPool = maybeBuildHiveDriverPool(config);
Review Comment:
🤖 These two pools are built eagerly here, but if `maybeBuildPartitionClientPool(...)` or the `HiveQueryDDLExecutor` constructor on the next line throws, the outer `catch` (line ~148) just rethrows — and since the constructor failed, `close()` is never called, so the already-built pool's worker threads + Thrift connections leak. Could we close `partitionDriverPool`/`partitionClientPool` in the catch before rethrowing? Same pattern in the legacy default-mode branch below.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HoodieHiveSyncClient.java:
##########
@@ -86,6 +90,15 @@ public class HoodieHiveSyncClient extends HoodieSyncClient {
private final Map<String, Table> initialTableByName = new HashMap<>();
DDLExecutor ddlExecutor;
private IMetaStoreClient client;
+ // Non-null only when HIVE_SYNC_BATCHING_ENABLED and sync mode is HIVEQL. Owned by
+ // this class; closed in close() before Hive.closeCurrent(). HiveQueryDDLExecutor
+ // uses it only for DROP (Hive Thrift, not Hive Driver) — see IMetaStoreClientPool
+ // javadoc.
+ private IMetaStoreClientPool partitionClientPool;
Review Comment:
🤖 nit: `partitionClientPool` is a raw nullable while `partitionDriverPool` just below (line 101) is wrapped in `Option<>` — both represent the same "batching enabled" condition, so it might be worth making the container type consistent (either both `Option<>`, or both nullable with the same guard pattern) to avoid a future reader wondering if there's a semantic difference.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/HiveSyncConfigHolder.java:
##########
@@ -122,6 +122,23 @@ public class HiveSyncConfigHolder {
.defaultValue(1000)
.markAdvanced()
.withDocumentation("The number of partitions one batch when synchronous partitions to hive.");
+ public static final ConfigProperty<Boolean> HIVE_SYNC_BATCHING_ENABLED = ConfigProperty
+ .key("hoodie.datasource.hive_sync.batching.enabled")
+ .defaultValue(false)
+ .markAdvanced()
+ .sinceVersion("1.1.0")
+ .withDocumentation("When true, partition operations (add/update/touch/drop) in the active sync mode "
+ + "are split into batches of `hoodie.datasource.hive_sync.batch_num` and dispatched in parallel "
+ + "to a pool of workers. Table-level operations (create/alter table, last commit time, writer "
+ + "version) continue to use the single session client/driver. Default off; existing behavior is "
+ + "unchanged unless explicitly opted in.");
+ public static final ConfigProperty<Integer> HIVE_SYNC_BATCHING_THREADS = ConfigProperty
+ .key("hoodie.datasource.hive_sync.batching.threads")
+ .defaultValue(4)
+ .markAdvanced()
Review Comment:
🤖 nit: the doc says "Pool size (number of worker resources) and worker-thread count" — could you simplify to something like "Number of worker threads (and pooled metastore clients) for parallel partition sync when `hoodie.datasource.hive_sync.batching.enabled` is true"? The current wording lists "pool size" and "worker-thread count" as if they're distinct, but they're the same number.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveDriverPool.java:
##########
@@ -0,0 +1,340 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+import org.apache.hudi.hive.HoodieHiveSyncException;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.Driver;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.Future;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.hudi.sync.common.HoodieSyncConfig.META_SYNC_DATABASE_NAME;
+
+/**
+ * Pool of Hive {@link Driver} + {@link SessionState} pairs for parallel HiveQL DDL.
+ *
+ * <p>Hive's {@code SessionState.start(state)} binds state to the calling thread's
+ * thread-local, and {@code Driver} reads from that thread-local during {@code run()}.
+ * A Driver constructed on one thread cannot be safely used from another. This pool
+ * solves that by giving each slot its own dedicated worker thread (a single-thread
+ * executor) — the Driver and SessionState are built on that thread by a bootstrap
+ * task, and all subsequent SQL for that slot runs on the same thread.
+ *
+ * <p><b>Usage contract:</b> use this pool only for partition-row DDL statements that
+ * are independent of each other and freely shuffleable across workers. Table-level
+ * statements (createTable, schema evolution, USE database) must continue to run on
+ * the session {@code Driver} held by {@code HiveQueryDDLExecutor} on the sync driver
+ * thread. The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled}
+ * and is constructed only for HiveQL sync mode.
+ */
+public class HiveDriverPool implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveDriverPool.class);
+
+ // Per-worker Driver construction has to be fast in practice (a few hundred ms
+ // for the SessionState + Driver init). A 60s ceiling per worker leaves plenty of
+ // headroom for a slow JVM warm-up but bounds the failure mode if the metastore
+ // is unreachable or Hive hangs during init.
+ private static final long BOOTSTRAP_TIMEOUT_SECONDS = 60;
+
+ private final List<Worker> workers;
+ private final int size;
+ private volatile boolean closed;
+
+ public HiveDriverPool(HiveSyncConfig config, int size) {
+ this(config, size, new DefaultDriverFactory(config));
+ }
+
+ // Package-private for tests: accepts a DriverFactory so unit tests can inject
+ // mock Driver instances without standing up a real Hive instance.
+ HiveDriverPool(HiveSyncConfig config, int size, DriverFactory factory) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ this.size = size;
+ this.workers = new ArrayList<>(size);
+ String databaseName = config.getStringOrDefault(META_SYNC_DATABASE_NAME);
+ PoolThreadFactory threadFactory = new PoolThreadFactory();
+ List<Future<Void>> bootstrapFutures = new ArrayList<>(size);
+ try {
+ for (int i = 0; i < size; i++) {
+ Worker worker = new Worker(threadFactory);
+ workers.add(worker);
+ bootstrapFutures.add(worker.executor.submit(() -> {
+ worker.driver = factory.newDriver(databaseName);
+ return null;
+ }));
+ }
+ // Block until all bootstraps complete so we surface construction errors
+ // before any caller hands us SQL. Bounded by BOOTSTRAP_TIMEOUT_SECONDS so a
+ // hung Hive init doesn't deadlock the sync driver thread.
+ for (Future<Void> f : bootstrapFutures) {
+ f.get(BOOTSTRAP_TIMEOUT_SECONDS, TimeUnit.SECONDS);
+ }
+ } catch (Exception e) {
+ tearDown();
+ throw new HoodieException("Failed to construct HiveDriverPool of size " + size, e);
+ }
+ LOG.info("Initialized HiveDriverPool with {} workers", size);
+ }
+
+ /**
+ * Runs each given SQL on <i>every</i> worker, in order. Used for setup statements
+ * (e.g. {@code USE database}) that must establish per-thread session context
+ * before any partition statement runs. Blocks until all workers have completed
+ * the setup. Throws on first error.
+ */
+ public void runOnEachWorker(List<String> setupSqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ if (setupSqls.isEmpty()) {
+ return;
+ }
+ List<Future<?>> futures = new ArrayList<>(workers.size());
+ for (Worker worker : workers) {
+ futures.add(worker.executor.submit(() -> {
+ for (String sql : setupSqls) {
+ worker.driver.run(sql);
+ }
+ return null;
+ }));
+ }
+ awaitAll(futures);
+ }
+
+ /**
+ * Dispatches each SQL string to a worker (round-robin) and returns the list of
+ * futures. The caller is responsible for awaiting and collecting errors. SQL text
+ * is intentionally not logged per-statement here: batched TOUCH/ADD statements can
+ * be many kilobytes, and N parallel workers would multiply the log volume. See
+ * {@link #awaitAll(List)} for the per-call summary log.
+ */
+ public List<Future<?>> runAll(List<String> sqls) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed HiveDriverPool");
+ }
+ List<Future<?>> futures = new ArrayList<>(sqls.size());
+ for (int i = 0; i < sqls.size(); i++) {
+ String sql = sqls.get(i);
+ Worker worker = workers.get(i % workers.size());
+ futures.add(worker.executor.submit(() -> {
+ worker.driver.run(sql);
+ return null;
+ }));
+ }
+ return futures;
+ }
+
+ /**
+ * Awaits all futures and throws the first exception encountered. On first failure,
+ * cancels the remaining (not yet started) futures so workers don't keep running
+ * pointless work after a fatal error. Any errors that finished before cancellation
+ * are logged at WARN. Callers do not need per-statement results (Hive's Driver.run
+ * side-effects the metastore), so this method is void.
+ */
+ public void awaitAll(List<Future<?>> futures) {
+ long start = System.currentTimeMillis();
+ Exception firstError = null;
+ int completed = 0;
+ int cancelled = 0;
+ for (int i = 0; i < futures.size(); i++) {
+ Future<?> f = futures.get(i);
+ try {
+ f.get();
+ completed++;
+ } catch (CancellationException ce) {
+ // We cancelled this future ourselves after a prior error. Don't treat it
+ // as a new failure; just note it for the summary log.
+ cancelled++;
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ if (firstError == null) {
+ firstError = ie;
+ cancelled += cancelRemaining(futures, i + 1);
+ }
+ } catch (ExecutionException ee) {
+ Exception cause = unwrap(ee);
+ if (firstError == null) {
+ firstError = cause;
+ cancelled += cancelRemaining(futures, i + 1);
+ } else {
+ LOG.warn("Additional SQL batch failed (suppressed in favor of first error)", cause);
+ }
+ }
+ }
+ if (firstError != null) {
+ throw new HoodieHiveSyncException("Failed in executing SQL", firstError);
+ }
+ LOG.info("Completed {} SQL statements ({} cancelled) in {} ms across {} workers",
+ completed, cancelled, System.currentTimeMillis() - start, size);
+ }
+
+ private static int cancelRemaining(List<Future<?>> futures, int fromIndex) {
+ int cancelled = 0;
+ for (int j = fromIndex; j < futures.size(); j++) {
+ // mayInterruptIfRunning=false: the worker thread is bound to a Hive Driver
+ // whose state we don't want to corrupt mid-statement. Cancel only those that
+ // haven't started yet; in-flight statements run to completion.
+ if (futures.get(j).cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ private static Exception unwrap(ExecutionException ee) {
+ Throwable cause = ee.getCause();
+ return (cause instanceof Exception) ? (Exception) cause : ee;
+ }
+
+ public int size() {
+ return size;
+ }
+
+ @Override
+ public void close() {
+ if (closed) {
+ return;
+ }
+ closed = true;
+ tearDown();
+ }
+
+ private void tearDown() {
+ // Close each worker's Driver/SessionState on its own thread, then shut the
+ // executor down. Running close() on the bound thread keeps SessionState's
+ // thread-local cleanup correct.
+ for (Worker worker : workers) {
+ try {
+ worker.executor.submit(() -> {
+ if (worker.driver != null) {
+ try {
+ worker.driver.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled Driver", e);
+ }
+ }
+ SessionState ss = SessionState.get();
+ if (ss != null) {
+ try {
+ ss.close();
+ } catch (Exception e) {
+ LOG.warn("Error closing pooled SessionState", e);
+ }
+ }
+ return null;
+ }).get(30, TimeUnit.SECONDS);
+ } catch (Exception e) {
+ LOG.warn("Error during pool worker shutdown", e);
+ }
+ worker.executor.shutdown();
+ try {
+ if (!worker.executor.awaitTermination(10, TimeUnit.SECONDS)) {
+ worker.executor.shutdownNow();
+ }
+ } catch (InterruptedException ie) {
+ worker.executor.shutdownNow();
+ Thread.currentThread().interrupt();
+ }
+ }
+ workers.clear();
+ }
+
+ /**
+ * Per-slot state: a single-thread executor and the Driver bound to its thread.
+ * Driver is volatile because it is written by the bootstrap task and read by
+ * subsequent dispatch tasks on the same executor.
+ */
+ private static final class Worker {
+ final ExecutorService executor;
+ volatile Driver driver;
+
+ Worker(ThreadFactory threadFactory) {
+ this.executor = Executors.newSingleThreadExecutor(threadFactory);
+ }
+ }
+
+ @FunctionalInterface
+ interface DriverFactory {
+ Driver newDriver(String databaseName) throws Exception;
+ }
+
+ /**
+ * Builds a real Hive {@link Driver} on the calling thread. The SessionState is
+ * constructed lazily (once, on the first worker thread that builds a Driver) and
+ * shared across all worker threads — Hive uses ThreadLocal attachment, not
+ * exclusive ownership, so multiple workers calling
+ * {@code SessionState.start(sharedState)} all see the same config and scratch dir
+ * without each spending the cost of building their own SessionState (and risking
+ * resource-dir creation races during the constructor).
+ */
+ private static final class DefaultDriverFactory implements DriverFactory {
+ private final HiveConf hiveConf;
+ private volatile SessionState sharedSessionState;
+
+ DefaultDriverFactory(HiveSyncConfig config) {
+ this.hiveConf = config.getHiveConf();
+ }
+
+ @Override
+ public synchronized Driver newDriver(String databaseName) throws Exception {
+ // SessionState is shared across workers; build it once on the first call (with
+ // currentDatabase already set) and attach it to each worker's thread-local on
+ // subsequent calls. The database is a pool-wide property and never changes
+ // across workers, so setting it once at construction time is sufficient.
Review Comment:
🤖 Sharing one SessionState across worker threads is risky here. Hive stores the active SessionState in a ThreadLocal (SessionState.start() sets tss.set(...)), so this pattern installs the *same* mutable instance into N threads' ThreadLocals. The Hive Driver isn't documented as thread-safe, and Driver execution mutates session-scoped state — currentDatabase (USE), conf overrides, and the compile/execute lineage/read-entity sets — so concurrent ALTER PARTITION/TOUCH runs can interleave on those fields. Worth confirming whether any of the batched statements can issue a USE or set conf; if so, per-worker SessionState (or serializing the session-mutating bits) is the safer default even at the init cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3710075182
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/ParallelDispatch.java:
##########
@@ -0,0 +1,240 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.common.util.VisibleForTesting;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Handle to one fan-out batch of partition work: the submitted futures plus the shared
+ * abort flag the tasks consult before running.
+ *
+ * <p>The abort flag exists because waiting on futures in <i>submission</i> order is not
+ * enough to stop queued work. If a later task fails quickly while an earlier one is slow,
+ * the awaiting thread is still parked on the earlier {@code Future.get()}, and the
+ * executor happily keeps starting every queued task in the meantime. By the time the
+ * failure is observed, most of the "not-yet-started" work has already run.
+ *
+ * <p>Two mechanisms fix that, and both are needed:
+ * <ul>
+ * <li>a {@link CountDownLatch} tripped by the <i>first</i> abort, so the awaiting
+ * thread wakes on failure rather than on its turn in submission order;</li>
+ * <li>a task-side {@link #aborted()} check on entry, because cancelling from the
+ * awaiting thread is inherently late — a worker can pull its next task off the
+ * queue at any moment.</li>
+ * </ul>
+ *
+ * <p>Shared by {@link HiveDriverPool} (Hive {@code Driver} statements) and
+ * {@link IMetaStoreClientPool} (Thrift {@code dropPartition} batches), which fan out over
+ * different execution models but need identical abort-on-first-error semantics.
+ */
+public final class ParallelDispatch {
+
+ private final List<Future<?>> futures;
+ private final int total;
+ private final AtomicInteger settled = new AtomicInteger(0);
+ private final AtomicBoolean aborted = new AtomicBoolean(false);
+ private final CountDownLatch done = new CountDownLatch(1);
+ private volatile boolean sealed;
+
+ ParallelDispatch(int total) {
+ this.total = total;
+ this.futures = new ArrayList<>(total);
+ }
+
+ void add(Future<?> future) {
+ futures.add(future);
+ }
+
+ // Called once submission finishes. A task that settles before the last submit
+ // would otherwise see settled < total and never trip the latch, so re-check here.
+ void sealed() {
+ sealed = true;
+ signalIfComplete();
+ }
+
+ boolean aborted() {
+ return aborted.get();
+ }
+
+ void abort() {
+ aborted.set(true);
+ done.countDown();
+ }
+
+ void taskSettled() {
+ settled.incrementAndGet();
+ signalIfComplete();
+ }
+
+ private void signalIfComplete() {
+ if (sealed && settled.get() >= total) {
+ done.countDown();
+ }
+ }
+
+ void awaitSettledOrAborted() {
+ if (total == 0) {
+ return;
+ }
+ try {
+ done.await();
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ aborted.set(true);
+ }
+ }
+
+ // mayInterruptIfRunning=false: a worker may be mid-statement against a Hive Driver or
+ // a Thrift client, and we don't want to tear that down partway. Cancel only tasks that
+ // haven't started; in-flight work runs to completion.
+ int cancelPending() {
+ int cancelled = 0;
+ for (Future<?> f : futures) {
+ if (f.cancel(false)) {
+ cancelled++;
+ }
+ }
+ return cancelled;
+ }
+
+ List<Future<?>> futures() {
+ return futures;
+ }
+
+ /**
+ * Wraps {@code body} so it observes this batch's abort flag: it skips itself if a
+ * sibling has already failed, trips the flag if it fails, and always records that it
+ * settled so the awaiting thread can be released.
+ */
+ Callable<Void> guard(Callable<Void> body, String skipMessage) {
+ return () -> {
+ if (aborted()) {
+ throw new CancellationException(skipMessage);
+ }
+ try {
+ return body.call();
+ } catch (Throwable t) {
+ abort();
+ throw t;
+ } finally {
+ taskSettled();
+ }
+ };
+ }
+
+ /**
+ * Waits for the batch to settle (or abort), cancels whatever had not started, and
+ * returns the outcome. Errors are observed in <i>completion</i> order, not submission
+ * order, so a failure on a fast worker stops the other queues even while a slow worker
+ * is still mid-statement.
+ */
+ Outcome awaitOutcome() {
+ awaitSettledOrAborted();
+ int cancelled = cancelPending();
+
+ Exception firstError = null;
+ int completed = 0;
+ List<Exception> suppressed = new ArrayList<>();
+ for (Future<?> f : futures) {
Review Comment:
You are right, and the reproduction holds exactly as you described. Fixed in `3b2e4bbc`.
The latch was waking on the first abort, but the exception was still being *selected* by walking `futures` in submission order — so whichever failure happened to sit earliest in the list won, regardless of which one actually stopped the batch. Two separate things, and only the first had been fixed.
Taking your suggestion: the aborting task now records its own throwable via `AtomicReference.compareAndSet` (first writer wins) as it aborts, and `awaitOutcome` seeds `firstError` from that recorded cause instead of from list position. Remaining failures are collected as suppressed.
One wrinkle worth noting: the aborting task's own failure also comes back through the `futures` drain, so the suppressed-collection skips it by *identity* (`cause != firstError`) rather than equality — otherwise the primary error ends up inside its own suppressed list.
Covered by `firstErrorIsTheFailureThatAbortedTheBatch`. It pins the interleaving on the abort flag itself rather than on a latch counted down before the throw — only the flag proves `fast-first` has actually reached `abort()`, so the ordering is guaranteed rather than merely likely. Against the previous implementation it fails with exactly what you reported:
```
expected: <fast-first> but was: <slow-later>
```
`TestIMetaStoreClientPool` 12/12 and `TestHiveDriverPool` 10/10 green (both build on `ParallelDispatch`).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5219603313
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 97dc86a59f300dfdc692a725131481afba3bfc59 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178)
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 4d0ac7105b4dc361317b936bc2c85a8683f0e8aa UNKNOWN
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5221564262
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
}, {
"hash" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16178",
"triggerID" : "97dc86a59f300dfdc692a725131481afba3bfc59",
"triggerType" : "PUSH"
}, {
"hash" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"status" : "UNKNOWN",
"url" : "TBD",
"triggerID" : "3a1694a787f4229c298d2cfc6fa98494a2110eef",
"triggerType" : "PUSH"
}, {
"hash" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183",
"triggerID" : "4d0ac7105b4dc361317b936bc2c85a8683f0e8aa",
"triggerType" : "PUSH"
}, {
"hash" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"status" : "PENDING",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187",
"triggerID" : "038e6c842e18130a64552da6ff5aef46864b3cf0",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3a1694a787f4229c298d2cfc6fa98494a2110eef UNKNOWN
* 4d0ac7105b4dc361317b936bc2c85a8683f0e8aa Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16183)
* 038e6c842e18130a64552da6ff5aef46864b3cf0 Azure: [PENDING](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16187)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5222221497
@danny0405 this is ready for another look when you have time.
**Since your last review, the branch was rebased onto master.** That mattered more than usual here: #19391 landed the same class of fix in `HiveDriverPool` that we were making in this PR, and since this PR *extracts* that inner `Dispatch` class into a shared `ParallelDispatch` (so the DROP Thrift pool can reuse it), the two collided. Resolution keeps the extraction and folds your/Vova's refinements into `ParallelDispatch`. Branch is `MERGEABLE` again and CI is green.
Worth calling out: **#19391 independently confirms the race described in the P3 thread below** — its `recordFailure` javadoc documents exactly the `cancelPending()`-wins-the-`FutureTask`-CAS hazard that was measured on this branch.
**The one thing genuinely waiting on you: P3 on `ParallelDispatch:190`** (the double-counted cancellations). It is still deferred, and the reasoning is in that thread: fixing the count naively removes what currently *masks* that race, turning a real metastore exception into a bare `CancellationException`. Measured, not theorized:
```
pre-fix FAIL=EXEC:IllegalStateException count 4 (wrong) failure PRESERVED
naive P3 FAIL=BARE_CANCEL count 2 (right) failure DESTROYED
```
Four variants were tried and rejected. A correct fix means reworking the `abort()` / `taskSettled()` / `signalIfComplete()` wake protocol — the same machinery implementing the abort-on-first-error guarantee — so it seemed to deserve its own PR rather than riding along here. Happy to do it either way; just want your call.
**Addressed since you last looked:**
| Fix | What |
|---|---|
| `97dc86a` | An `Error` that aborted a dispatch was reported twice — as root cause *and* suppressed. `guard` catches `Throwable`, so `asException()` wrapped it and the identity check could never match. Now compares the raw cause. |
| `3a1694a` | **The one I would most like your eyes on.** DROP was the only sync path invoking a user-supplied `PartitionValueExtractor` from several pool workers at once. Extractors are pluggable with no thread-safety contract, so a stateful one could return a garbled clause and drop the *wrong* partition. Values are now resolved on the calling thread before fan-out. A test recording call threads fails on the old code with all four workers inside the extractor. |
| `4d0ac71` | Renamed `IMetaStoreClientPool` → `HiveMetaStoreClientPool` (the `I` prefix read as an interface next to the real `IMetaStoreClient`). |
| `038e6c8` | Test-only: `TestHoodieHiveSyncClientClose` uses `CALLS_REAL_METHODS`, which skips the constructor, so the `partitionClientPool` field this PR adds was null and `close()` NPE'd before reaching the client — surfacing as a misleading "zero interactions with this mock". |
Current size is 12 files, +1273/−255. No new config keys — everything reuses the batching configs from #18984, and `batching.enabled=false` (the default) is byte-identical to pre-PR behaviour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on PR #19033:
URL: https://github.com/apache/hudi/pull/19033#issuecomment-5176736089
<!--
Meta data
{
"version" : 1,
"metaDataEntries" : [ {
"hash" : "7022e7d493416d600664235c3056d32ea4a1c179",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14784",
"triggerID" : "7022e7d493416d600664235c3056d32ea4a1c179",
"triggerType" : "PUSH"
}, {
"hash" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=14894",
"triggerID" : "5e3c1fdec6c9dd95e339a2027ca5be20002dea8c",
"triggerType" : "PUSH"
}, {
"hash" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15684",
"triggerID" : "9e35ec8eaa25e646b0559b0b9dc5485d2622ea9b",
"triggerType" : "PUSH"
}, {
"hash" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15748",
"triggerID" : "7610c51a4e727c97110e3dadba9868c4fff7aa0f",
"triggerType" : "PUSH"
}, {
"hash" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"status" : "DELETED",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=15787",
"triggerID" : "6c81322202e4b6cb251248da8a0d8c478fb3fe26",
"triggerType" : "PUSH"
}, {
"hash" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"status" : "FAILURE",
"url" : "https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034",
"triggerID" : "3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545",
"triggerType" : "PUSH"
} ]
}-->
## CI report:
* 3b2e4bbcdfdc15be6cd5caaa0eb689aac1dff545 Azure: [FAILURE](https://dev.azure.com/apachehudi/a1a51da7-8592-47d4-88dc-fd67bed336bb/_build/results?buildId=16034)
<details>
<summary>Bot commands</summary>
@hudi-bot supports the following commands:
- `@hudi-bot run azure` re-run the last Azure build
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3763123193
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveMetaStoreClientPool.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.exception.HoodieException;
+import org.apache.hudi.hive.HiveSyncConfig;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.IMetaStoreClient;
+import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.ArrayBlockingQueue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.ThreadFactory;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Pool of {@link IMetaStoreClient} instances for parallel partition sync.
+ *
+ * <p>Each pooled client wraps an independent Thrift connection to the Hive Metastore.
+ * Callers borrow a client via {@link #run(ClientAction)}, which blocks until a client
+ * is available, executes the action, and returns the client to the pool. A worker
+ * thread pool of the same size is exposed via {@link #executor()} so callers can fan
+ * out their batches to match the number of available clients.
+ *
+ * <p><b>Usage contract:</b> pool clients must be used <i>only</i> for partition-row
+ * operations — {@code add_partitions}, {@code alter_partitions}, {@code dropPartition},
+ * {@code getPartition}. Table-row operations ({@code createTable}, {@code alter_table},
+ * {@code getTable} used as the read half of a read-modify-write of table parameters)
+ * must continue to go through the session client held by
+ * {@code HoodieHiveSyncClient.client} on the sync driver thread. Mixing the two would
+ * risk lost updates on table parameters such as the last-commit-time-synced marker.
+ *
+ * <p>The pool is gated behind {@code hoodie.datasource.hive_sync.batching.enabled} and
+ * is constructed for sync mode HIVEQL, where it backs the DROP path only. DROP goes
+ * through {@code IMetaStoreClient.dropPartition} (Thrift), whereas ADD/UPDATE/TOUCH go
+ * through the thread-bound Hive {@code Driver} and use {@code HiveDriverPool} instead.
+ */
+public class HiveMetaStoreClientPool implements AutoCloseable {
+
+ private static final Logger LOG = LoggerFactory.getLogger(HiveMetaStoreClientPool.class);
+
+ private final ArrayBlockingQueue<IMetaStoreClient> available;
+ private final List<IMetaStoreClient> all;
+ private final ExecutorService executor;
+ private final int size;
+ private volatile boolean closed;
+
+ public HiveMetaStoreClientPool(HiveSyncConfig config, int size) {
+ this(buildClients(config, size), size);
+ }
+
+ // Package-private for tests: accepts a pre-built list of clients so we can
+ // exercise borrow/return/close semantics without a live metastore.
+ HiveMetaStoreClientPool(List<IMetaStoreClient> clients, int size) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ if (clients.size() != size) {
+ throw new IllegalArgumentException("Expected " + size + " clients, got " + clients.size());
+ }
+ this.size = size;
+ this.available = new ArrayBlockingQueue<>(size);
+ this.all = new ArrayList<>(clients);
+ this.available.addAll(clients);
+ this.executor = Executors.newFixedThreadPool(size, new PoolThreadFactory());
+ LOG.info("Initialized IMetaStoreClient pool with {} clients", size);
+ }
+
+ private static List<IMetaStoreClient> buildClients(HiveSyncConfig config, int size) {
+ if (size < 1) {
+ throw new IllegalArgumentException("Pool size must be >= 1, got " + size);
+ }
+ HiveConf hiveConf = config.getHiveConf();
+ List<IMetaStoreClient> clients = new ArrayList<>(size);
+ try {
+ for (int i = 0; i < size; i++) {
+ clients.add(newClient(hiveConf));
+ }
+ return clients;
+ } catch (Exception e) {
+ // Construction failed mid-way; close any clients we already built before
+ // surfacing the error so we don't leak Thrift sockets.
+ for (IMetaStoreClient c : clients) {
+ try {
+ c.close();
+ } catch (Exception ignore) {
+ // intentional: best-effort cleanup during failure
+ }
+ }
+ throw new HoodieException("Failed to construct IMetaStoreClient pool of size " + size, e);
+ }
+ }
+
+ private static IMetaStoreClient newClient(HiveConf hiveConf) {
+ try {
+ // RetryingMetaStoreClient.getProxy returns an independent IMetaStoreClient
+ // (one Thrift socket per call), bypassing the Hive thread-local cache that
+ // Hive.get(conf) would use. This is what gives us N truly independent clients.
+ return RetryingMetaStoreClient.getProxy(hiveConf, true);
+ } catch (Exception e) {
+ throw new HoodieException("Failed to create IMetaStoreClient for pool", e);
+ }
+ }
+
+ /**
+ * Borrows a client, runs the action, and returns the client to the pool.
+ * Blocks if all clients are in use until one becomes available.
+ */
+ public <T> T run(ClientAction<T> action) throws Exception {
+ if (closed) {
+ throw new IllegalStateException("Cannot borrow from a closed IMetaStoreClient pool");
+ }
+ IMetaStoreClient client = available.take();
+ try {
+ return action.apply(client);
+ } finally {
+ // Always return the client to the pool, even on failure. Thrift clients
+ // recover transparently from transactional errors at the HMS layer;
+ // RetryingMetaStoreClient handles transient socket failures internally.
+ if (!closed) {
+ available.offer(client);
+ }
+ }
+ }
+
+ /**
+ * Submits one task per item, each borrowing a pooled client for the duration of its
+ * call, and returns a handle to the in-flight batch — this method does not block. The
+ * caller awaits completion via {@link #awaitAll(ParallelDispatch, String)}.
+ *
+ * <p>Tasks observe a shared abort flag, so a failure on any item stops items that have
+ * not started yet even while a slower sibling is still mid-call. See
+ * {@link ParallelDispatch} for why waiting on futures alone does not achieve that.
+ */
+ public <T> ParallelDispatch dispatchAll(List<T> items, ClientConsumer<T> action) {
+ if (closed) {
+ throw new IllegalStateException("Cannot dispatch to a closed IMetaStoreClient pool");
+ }
+ ParallelDispatch dispatch = new ParallelDispatch(items.size());
+ for (T item : items) {
+ dispatch.add(executor.submit(dispatch.guard(() -> {
+ run(client -> {
+ action.accept(client, item);
+ return null;
+ });
+ return null;
+ }, "Skipped after an earlier batch failed")));
+ }
+ dispatch.sealed();
+ return dispatch;
+ }
+
+ /**
+ * Awaits a batch from {@link #dispatchAll(List, ClientConsumer)}, cancels whatever had
+ * not started, and rethrows the first real failure. Later failures are logged at WARN.
+ */
+ public void awaitAll(ParallelDispatch dispatch, String description) throws Exception {
+ long start = System.currentTimeMillis();
+ ParallelDispatch.Outcome outcome = dispatch.awaitOutcome();
+ outcome.suppressed().forEach(e ->
+ LOG.warn("Additional {} batch failed (suppressed in favor of first error)", description, e));
Review Comment:
Confirmed — no callers anywhere, removed in a9d9443.
The class javadoc was also still pointing at `executor()` as the fan-out mechanism, which stopped being true when `dispatchAll` took that over; updated it to describe the internal worker pool instead. Public surface is now borrow / dispatch / await / close.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(hive-sync): parallelize DROP partitions in HiveQL sync mode [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on code in PR #19033:
URL: https://github.com/apache/hudi/pull/19033#discussion_r3762992921
##########
hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/ParallelDispatch.java:
##########
@@ -0,0 +1,272 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.hive.util;
+
+import org.apache.hudi.common.util.VisibleForTesting;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.concurrent.Callable;
+import java.util.concurrent.CancellationException;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicReference;
+
+/**
+ * Handle to one fan-out batch of partition work: the submitted futures plus the shared
+ * abort flag the tasks consult before running.
+ *
+ * <p>The abort flag exists because waiting on futures in <i>submission</i> order is not
+ * enough to stop queued work. If a later task fails quickly while an earlier one is slow,
+ * the awaiting thread is still parked on the earlier {@code Future.get()}, and the
+ * executor happily keeps starting every queued task in the meantime. By the time the
+ * failure is observed, most of the "not-yet-started" work has already run.
+ *
+ * <p>Two mechanisms fix that, and both are needed:
+ * <ul>
+ * <li>a {@link CountDownLatch} tripped by the <i>first</i> abort, so the awaiting
+ * thread wakes on failure rather than on its turn in submission order;</li>
+ * <li>a task-side {@link #aborted()} check on entry, because cancelling from the
+ * awaiting thread is inherently late — a worker can pull its next task off the
+ * queue at any moment.</li>
+ * </ul>
+ *
+ * <p>The failing task also records its own throwable as it aborts, so the reported root
+ * cause is the failure that stopped the batch. Picking the error by scanning futures in
+ * submission order would instead surface whichever failure happens to sit earliest in the
+ * list, which is not necessarily the one that aborted the run.
+ *
+ * <p>Shared by {@link HiveDriverPool} (Hive {@code Driver} statements) and
+ * {@link HiveMetaStoreClientPool} (Thrift {@code dropPartition} batches), which fan out over
+ * different execution models but need identical abort-on-first-error semantics.
+ */
+public final class ParallelDispatch {
+
+ private final List<Future<?>> futures;
+ private final int total;
+ private final AtomicInteger settled = new AtomicInteger(0);
+ private final AtomicBoolean aborted = new AtomicBoolean(false);
+ private final AtomicReference<Throwable> abortCause = new AtomicReference<>();
+ private final CountDownLatch done = new CountDownLatch(1);
+ private volatile boolean sealed;
+
+ ParallelDispatch(int total) {
+ this.total = total;
+ this.futures = new ArrayList<>(total);
+ }
+
+ void add(Future<?> future) {
+ futures.add(future);
+ }
+
+ // Called once submission finishes. A task that settles before the last submit
+ // would otherwise see settled < total and never trip the latch, so re-check here.
+ void sealed() {
+ sealed = true;
+ signalIfComplete();
+ }
+
+ boolean aborted() {
+ return aborted.get();
+ }
+
+ void abort() {
+ aborted.set(true);
+ done.countDown();
+ }
+
+ // Records the failure that triggered the abort, first writer wins. Selecting the error
+ // by walking futures in submission order instead would report whichever failure sits
+ // earliest in the list, not the one that actually stopped the batch: a fast failure at
+ // index 1 aborts the run, and a slow index 0 that fails later would take its place.
+ void abort(Throwable cause) {
+ abortCause.compareAndSet(null, cause);
+ abort();
+ }
+
+ void taskSettled() {
+ settled.incrementAndGet();
+ signalIfComplete();
+ }
+
+ private void signalIfComplete() {
+ if (sealed && settled.get() >= total) {
+ done.countDown();
+ }
+ }
+
+ void awaitSettledOrAborted() {
+ if (total == 0) {
+ return;
+ }
+ try {
+ done.await();
+ } catch (InterruptedException ie) {
+ Thread.currentThread().interrupt();
+ aborted.set(true);
+ }
Review Comment:
Confirmed and fixed in 749d6719.
You traced the whole chain correctly: `aborted.set(true)` without recording the cause means `cancelPending()` marks the remaining futures CANCELLED, the drain sees only `CancellationException`s, `firstError` stays null, and `awaitAll` reports a clean batch. On the DROP path that lets `HiveSyncTool` advance the last-synced commit marker over work that was cancelled or is still running — silent under-sync, which is the worst shape for this.
The fix is to route it through the existing abort-cause channel:
```java
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
abort(ie);
}
```
`abort(Throwable)` already does `compareAndSet(null, cause)` then `abort()`, so `awaitOutcome()` seeds `firstError` from it and both `awaitAll()` methods throw. Since `HiveDriverPool` and `HiveMetaStoreClientPool` share `ParallelDispatch`, this closes the gap on both.
Added `interruptedAwaitIsReportedAsAFailureNotASuccess` along the lines you suggested — blocks a dispatched action, interrupts the thread parked in `awaitAll`, and asserts it does not return normally. Verified against the pre-fix code:
```
awaitAll must not report success when the wait was interrupted;
the batch's work was cancelled or is still in flight
==> expected: <false> but was: <true>
```
While I was in here I checked the other interrupt handlers in the dispatch path. The two in `close()` (`HiveDriverPool` and `HiveMetaStoreClientPool` executor shutdown) restore the flag and force `shutdownNow()`, which is right — there is no outcome there to corrupt. `HiveMetaStoreClientPool.run()` declares `throws Exception`, so an interrupt from `available.take()` propagates into `guard` and is recorded by `abort(t)` already. `awaitSettledOrAborted` was the only site with the report-success hazard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org