[I] Disruptor executor can return success after a consumer write failure [hudi]

["danny0405 (via GitHub)" <gi...@apache.org>]

danny0405 opened a new issue, #20016:
URL: https://github.com/apache/hudi/issues/20016

   ### Bug Description
   
   **What happened:**
   
   The Disruptor event handler catches consumer exceptions and stores them with `markAsFailed()`. `BaseHoodieQueueBasedExecutor.execute()` checks the stored exception only inside its catch block. If the producer finishes, queue shutdown succeeds, and `consumer.finish()` returns normally, execution can return success despite a failed consume operation.
   
   This is a follow-up to [the review discussion in PR #18776](https://github.com/apache/hudi/pull/18776#discussion_r4060361891). The resource-cleanup PR deliberately does not add persistent failure state or per-record checks to write handles.
   
   **What you expected:**
   
   A fatal consumer failure must propagate to the executor caller, even when production and finalization complete normally. The original consumer exception should be preserved and executor resources released.
   
   **Steps to reproduce:**
   
   1. Create a `DisruptorExecutor` with a one-record iterator.
   2. Use a consumer whose `consume()` throws an exception and whose `finish()` returns normally.
   3. Execute it and verify that the stored consumer exception reaches the caller rather than returning the finish result.
   
   For write handles, use `hoodie.write.executor.type=DISRUPTOR` and `hoodie.write.ignore.failed=false`, then inject a record-write failure. This also matters for early-conflict exceptions that must remain fatal when individual write failures may be ignored.
   
   **Code pointers:**
   
   - [DisruptorMessageQueue](https://github.com/apache/hudi/blob/f8affff7793984a7a323988ac54598155842364f/hudi-common/src/main/java/org/apache/hudi/common/util/queue/DisruptorMessageQueue.java) catches and stores the consumer error.
   - [BaseHoodieQueueBasedExecutor](https://github.com/apache/hudi/blob/f8affff7793984a7a323988ac54598155842364f/hudi-common/src/main/java/org/apache/hudi/common/util/queue/BaseHoodieQueueBasedExecutor.java) runs finish without checking the stored error on its normal completion path.
   
   ### Environment
   
   **Hudi version:** 1.3.0-SNAPSHOT, PR #18776 at `f8affff77939`; the executor propagation gap is present in the upstream code used by this PR.
   **Query engine:** Any caller using `DisruptorExecutor`.
   **Relevant configs:** `hoodie.write.executor.type=DISRUPTOR`, `hoodie.write.ignore.failed=false`.
   
   ### Logs and Stack Trace
   
   Identified by source tracing during review. The steps above describe the required regression test; no new end-to-end reproducer log is attached.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Disruptor executor can return success after a consumer write failure [hudi]

["danny0405 (via GitHub)" <gi...@apache.org>]

danny0405 closed issue #20016: Disruptor executor can return success after a consumer write failure
URL: https://github.com/apache/hudi/issues/20016


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org