You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "Joy-2000 (via GitHub)" <gi...@apache.org> on 2026/08/12 06:22:43 UTC

[I] [DISCUSS] Reduce transaction lock scope for multi-writer workloads under NBCC [hudi]

Joy-2000 opened a new issue, #19587:
URL: https://github.com/apache/hudi/issues/19587

### Describe the problem you faced

## Background

In our production environment, we mainly run the following types of Flink streaming write workloads:

- COW tables with append writes
- MOR tables with Bucket Index and upsert writes

A single Hudi table may be concurrently accessed by one or more write, table-service, and backfill jobs, including:

- Flink streaming jobs writing to the latest date partition, potentially with asyn table services enabled
- Spark jobs scheduling and/or executing clustering, compaction, log compaction, or cleaning
- Multiple Spark jobs backfilling historical date partitions in parallel

Hudi currently uses its transaction mechanism—NBCC/OCC combined with locking—to support these multi-writer scenarios. Among other things, this mechanism ensures that:

- Completion times on the active timeline increase monotonically
- Metadata Table updates are serialized

## Problem

The current transaction boundary appears to cover several relatively expensive operations.

For Flink streaming writes in particular, table-service scheduling, metadata commits, and instant-time generation share the same lock. When the underlying filesystem, such as HDFS, becomes busy or unstable, these operations may hold the lock for an extended period.

This can significantly affect the stability of streaming jobs. For example, lock contention may block the sink from obtaining a new instant, resulting in prolonged backpressure, data accumulation, and delayed data availability.

We would like to discuss whether the scope between `txnManager.beginStateChange` and `txnManager.endStateChange` can be reduced under NBCC.

The following operations are possible candidates for moving partially or entirely outside the transaction boundary.

## 1. Table-service scheduling

`scheduleTableServiceInternal` currently performs the entire scheduling process for clustering, compaction, and log compaction inside a transaction.

The scheduling process may involve several expensive operations, including:

- Loading the timeline
- Building the filesystem view
- Building the table-service plan

The execution time of these operations can vary significantly, especially when the filesystem is under load.

### Proposal

Keep only the operations that require serialization inside the lock, such as `createTable` and `createNewInstantTime`, and move plan building and other expensive scheduling operations outside the lock.

### Potential risk

Without locking the entire scheduling process, multiple table-service jobs may schedule overlapping work concurrently. If this is not detected or rejected later, it could result in duplicate processing or data correctness issues (In our production environment, we use a separate scheduling lock to prevent table services from being scheduled concurrently).

### Expected benefit

Reduce the duration for which the transaction lock is held during table-service scheduling.

## 2. Request-time generation in `startCommit`

The generation of both request times and completion times currently requires acquiring the lock.

Under the current timeline model, locking completion-time generation is necessary to preserve monotonically increasing completion times. However, it may be worth discussing whether the locking requirement for request-time generation can be relaxed.

Compaction request times are an exception because they participate in FileSlice boundaries and therefore still require strict serialization.

### Proposal

Investigate whether request-time generation for regular writes and other applicable actions can be moved outside the transaction lock, while retaining locking for:

- Completion-time generation
- Compaction request-time generation
- Any action whose request time affects FileSlice boundaries or timeline ordering semantics

### Potential risk

Concurrent writers might generate the same request time. Atomic file creation by the filesystem currently ensures that only one writer can successfully create a requested-instant file with a given name. However, the losing writer would still need to detect the collision, generate a new request time, and retry.

Therefore, moving request-time generation outside the lock may require an explicit collision-detection and retry mechanism.

### Expected benefit

When a Flink sink obtains its first or next instant, it would no longer be blocked solely because another operation is holding the transaction lock. This could make data consumption and checkpoint processing smoother.

## 3. `finalizeWrite` during commit

`commitStats`, `completeClustering`, `completeCompaction`, and `completeLogCompaction` perform several operations while completing a write or table service. These operations include, but are not limited to:

- `resolveWriteConflict`
- `finalizeWrite`
- `writeTableMetadata`
- `saveAsComplete`

`finalizeWrite` removes speculative or invalid files based on the markers for the current instant. When a write touches a large number of partitions or files, `DirectWriteMarkers` may issue a large number of filesystem `list` operations.

As a result, `finalizeWrite` may hold the transaction lock for a long time when the filesystem is busy.

### Proposal

Move `finalizeWrite` outside the transaction boundary, while keeping conflict resolution, Metadata Table updates, and timeline completion inside the transaction where serialization is required.

### Potential risk

We currently do not see an obvious correctness dependency that requires `finalizeWrite` itself to run under the transaction lock. However, this needs to be verified for all marker implementations and failure-recovery paths.

In particular, we should confirm that moving it outside the transaction does not introduce races with rollback, cleaning, or another writer operating on the same files.

### Expected benefit

Reduce the lock duration caused by filesystem operations during `finalizeWrite`, especially for large writes or when HDFS is under heavy load.

## Questions for discussion

1. Which operations inside the current NBCC transaction boundary strictly require serialization?
2. Which action types can request time generation safely occur outside the lock?
3. Are there other expensive filesystem or timeline operations that can be moved outside the transaction boundary?
4. Would it make sense to introduce metrics for transaction lock wait time and lock hold time so that lock contention can be observed more directly?

Our understanding and the ideas proposed above may be incomplete or incorrect. We would greatly appreciate any corrections, feedback, or suggestions from the community.

### To Reproduce

none

### Expected behavior

The overall goal is to preserve the existing NBCC correctness guarantees while reducing lock contention and improving the stability of Flink streaming writes in multi-writer environments.

### Environment Description

Hudi：1.1.1
Flink：1.20
Spark：3.4

### Additional context

_No response_

### Stacktrace

```shell

```

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSS] Reduce transaction lock scope for multi-writer workloads under NBCC [hudi]

Posted by "Joy-2000 (via GitHub)" <gi...@apache.org>.

Joy-2000 commented on issue #19587:
URL: https://github.com/apache/hudi/issues/19587#issuecomment-5378742845

> Any lock-free allocation/retry should check/reserve the request time across the relevant comparable-action family, not just retry when the exact target path already exists. This is especially important because createNewInstantTime(false) only has the process-local monotonic guard while skipping the distributed time-generator lock.

@danny0405 Sorry for the late reply, and thanks for the detailed discussion here.

Re: 2. Request-time generation in startCommit

I agree with your point that request-time generation shouldn't simply be moved out of the lock. One additional reason from our side: duplicate request instants can cause silent MDT data loss. If two concurrent writers generate the same request time (e.g. a deltacommit and a clean landing on the same instant), they map to the same instant on the Metadata Table, and when the later commit is applied the MDT reconciliation path treats the pre-existing instant as a leftover and rolls it back — so the data the first instant wrote to the MDT gets dropped. This means "atomic file creation + loser retries" isn't sufficient on its own: even if only one writer wins on the data timeline, the collision has already corrupted MDT state, so any collision-detection/retry mechanism would have to cover the MDT side, not just the data timeline.

We encounter another problem in our test. In the Flink async-instant flow, the write task sends an RPC to the coordinator to obtain the request time for the next instant. Under lock contention this RPC can exceed its timeout (default 60s), which fails the task and triggers a full job restart. For a streaming job this is a significant stability hit — contention on the instant-request path doesn't just cost throughput, it can take the job down. So we'd still like to find a way to reduce this contention, just without weakening the guarantees.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSS] Reduce transaction lock scope for multi-writer workloads under NBCC [hudi]

Posted by "danny0405 (via GitHub)" <gi...@apache.org>.

danny0405 commented on issue #19587:
URL: https://github.com/apache/hudi/issues/19587#issuecomment-5300583751

   Thanks for writing this up. I traced the three proposed boundaries through the current code and have a few comments/questions.
   
   > **1. Table-service scheduling**
   
   Moving plan generation outside the lock looks useful, but I don't think serializing only instant creation + plan-file creation is sufficient. Both clustering and compaction currently build a plan from a timeline/filesystem-view snapshot and then publish it as one operation ([clustering](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/cluster/ClusteringPlanActionExecutor.java#L58-L102), [compaction](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/action/compact/ScheduleCompactionActionExecutor.java#L106-L134)). If schedulers A and B build concurrently before either publishes, both can select the same file groups; serializing the two publishes does not make B's already-built plan fresh.
   
   A two-phase design therefore seems to need a validation/fencing step under the transaction lock: reload the timeline/view, verify the plan's input file groups are still eligible and do not overlap any newly pending service, and reject/rebuild on failure. A concurrency test where two planners pause after selecting the same file groups, then publish in sequence, would be valuable.
   
   > **2. Request-time generation in `startCommit`**
   
   Atomic creation of only the requested-instant filename does not cover every collision. Timeline layout maps `compaction -> commit`, `logcompaction -> deltacommit`, and `clustering -> replacecommit` ([mapping](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/versioning/v2/InstantComparatorV2.java#L51-L56)); latest-state filtering groups by `(requestedTime, comparableAction)` ([grouping](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/timeline/TimelineLayout.java#L111-L121)). Thus two different action filenames can both be created successfully at the same requested time and later be treated as the same logical action family.
   
   Any lock-free allocation/retry should check/reserve the request time across the relevant comparable-action family, not just retry when the exact target path already exists. This is especially important because `createNewInstantTime(false)` only has the process-local monotonic guard while skipping the distributed time-generator lock.
   
   > **3. `finalizeWrite`**
   
   This seems separable if the order is explicitly: finalize first; then acquire the transaction lock; then reload current table/timeline state and perform conflict resolution, Metadata Table update, and timeline completion in one critical section. It should not be split as conflict-resolution-under-lock → unlock → finalize → relock, because a commit completing in that gap invalidates the conflict check.
   
   Also, `finalizeWrite` is not read-only: marker reconciliation deletes invalid data files ([implementation](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L747-L774)). Before moving it, I would add interleaving tests against rollback/failed-write cleaning for the same instant (including both marker implementations and an expired heartbeat) to prove those paths cannot delete a file selected as valid while finalize is outside the transaction.
   
   > **4. Metrics**
   
   There are already two lock timers behind `hoodie.metrics.lock.enable`: `lock.request.latency` and `lock.acquire.duration` ([metrics](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/HoodieLockMetrics.java#L38-L43), [config](https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/config/metrics/HoodieMetricsConfig.java#L99-L110)). The latter is updated on unlock and is effectively hold time. One gap is that `lock.request.latency` is started inside each retry attempt ([LockManager](https://github.com/apache/hudi/blob/master/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/client/transaction/lock/LockManager.java#L76-L90)), so it does not capture the total caller-visible wait across retry backoff. An end-to-end timer around `TransactionManager.beginStateChange`, ideally tagged by operation/action, would make the reported Flink stalls much easier to attribute.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSS] Reduce transaction lock scope for multi-writer workloads under NBCC [hudi]

Posted by "Joy-2000 (via GitHub)" <gi...@apache.org>.

Joy-2000 commented on issue #19587:
URL: https://github.com/apache/hudi/issues/19587#issuecomment-5534889586

   There are also some extreme cases — e.g. OOM or force kill — where a job fails to release the lock it holds when it stops/exits. Other jobs then have to wait for the lock to expire before they can acquire it, which is another source of instability for streaming jobs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSS] Reduce transaction lock scope for multi-writer workloads under NBCC [hudi]

Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.

HuangZhenQiu commented on issue #19587:
URL: https://github.com/apache/hudi/issues/19587#issuecomment-5429352521

   Thanks for raising the concern. We also meet the same issue. Looking forward to an agreement on the improvement.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [DISCUSS] Reduce transaction lock scope for multi-writer workloads under NBCC [hudi]

Posted by "Joy-2000 (via GitHub)" <gi...@apache.org>.

Joy-2000 commented on issue #19587:
URL: https://github.com/apache/hudi/issues/19587#issuecomment-5379007040

   Points 1 and 3 look like they have room for improvement, I've split them out into two separate issues to follow up.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org