You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "matanper (via GitHub)" <gi...@apache.org> on 2026/10/11 05:36:38 UTC

[I] [Bug] BE aborts when BetaRowsetReader iterator init throws a std::exception (rethrown by DorisCallOnce, never converted to Status) [doris]

matanper opened a new issue, #68915:
URL: https://github.com/apache/doris/issues/68915

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
   
   ### Version
   
   4.1.4 (cloud mode, arm64). master is the same: `BetaRowsetReader::_init_iterator_once` (`be/src/storage/rowset/beta_rowset_reader.cpp:324-326`) returns `_init_iter_once.call(...)` with no try/catch, `DorisCallOnce::call` (`be/src/util/once.h`) stores and rethrows any exception, and `RETURN_IF_CATCH_EXCEPTION` (`be/src/common/exception.h:67`) only catches `doris::Exception`.
   
   ### What's Wrong?
   
   A BE aborted three times with `std::terminate` on a scanner thread, each time while a long full-table scan (an auto-analyze `SELECT min/max` over a large table, under heavy ingest) was being cancelled at its timeout. Stack (from a terminate handler that logs the rethrown exception):
   
   ```
   std::bad_function_call
   std::terminate <- std::rethrow_exception <- DorisCallOnce<Status>::call (util/once.h)
    <- BetaRowsetReader::_init_iterator_once <- BetaRowsetReader::_next_batch
    <- VCollectIterator::Level0Iterator::next <- BlockReader::next_block_with_aggregation
    <- OlapScanner::_get_block_impl <- ScannerScheduler::_scanner_scan
   ```
   
   `_init_iterator()` threw `std::bad_function_call` (an empty `std::function` was called) while initializing segment iterators. `DorisCallOnce` saves the exception and rethrows it on this call and every later call. Nothing on the scan path catches a `std::exception`: `RETURN_IF_CATCH_EXCEPTION` and the scanner guards only catch `doris::Exception`, so the exception escapes to `std::terminate` and the whole BE goes down.
   
   We don't know the root throw site (which `std::function` was empty); the rethrow loses the original stack.
   
   ### What You Expected?
   
   A failure while initializing a rowset reader's iterators should fail that read (or the query) with an error status, not abort the process.
   
   ### How to Reproduce?
   
   We can't reproduce the original empty `std::function` deterministically. To show the missing guard: inject a `throw std::bad_function_call()` (or any `std::exception`) in `BetaRowsetReader::_init_iterator` via a debug point, then query the table: the BE terminates instead of returning an error.
   
   ### Anything Else?
   
   What we're doing as hardening:
   
   1. `BetaRowsetReader::_init_iterator_once` converts a `doris::Exception` or `std::exception` from the call into a `Status`, on the first call and on later calls that hit the stored exception. `DorisCallOnce` itself stays unchanged.
   2. `_init_iterator` and `LazyInitSegmentIterator::init` log the step that threw, with tablet, rowset and segment ids, so the next occurrence identifies the empty `std::function`.
   
   A unit test with a debug point that calls an empty `std::function` in `_init_iterator` gets a status containing `bad_function_call` from both the first and the second `next_batch`.
   
   #68675 fixed a related cached-failure site (`BetaRowset::get_segment_num_rows`), but not this path. #61077 fixed a different `std::bad_function_call` (a data race in `AnalysisFactoryMgr`); we don't know whether that's the same root cause.
   
   ### Are you willing to submit PR?
   
   - [x] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE aborts when BetaRowsetReader iterator init throws a std::exception (rethrown by DorisCallOnce, never converted to Status) [doris]

Posted by "matanper (via GitHub)" <gi...@apache.org>.
matanper commented on issue #68915:
URL: https://github.com/apache/doris/issues/68915#issuecomment-6249075442

   Unfortunately we failed to reproduce it. All three occurrences happened during automatic statistics collection (full min/max scans) on BEs under heavy concurrent load, and the exception was captured only at the rethrow in DorisCallOnce, so the original throw site was lost.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE aborts when BetaRowsetReader iterator init throws a std::exception (rethrown by DorisCallOnce, never converted to Status) [doris]

Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on issue #68915:
URL: https://github.com/apache/doris/issues/68915#issuecomment-6210820363

   @matanper Very curious about this problem. The detail stack about the std::bad_function_call. I think we should find the stack and solve the problem.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE aborts when BetaRowsetReader iterator init throws a std::exception (rethrown by DorisCallOnce, never converted to Status) [doris]

Posted by "Doris-Breakwater (via GitHub)" <gi...@apache.org>.
Doris-Breakwater commented on issue #68915:
URL: https://github.com/apache/doris/issues/68915#issuecomment-6105923913

   Breakwater-GitHub-Analysis-Slot: slot_db219bfe8e2e
   
   This warrants high-priority BE/storage crash triage. The exception-to-Status boundary gap is supported by the source; the origin of the production `std::bad_function_call` remains unproven. These are two separate investigation items, and a focused hardening PR need not wait for a deterministic reproduction of the original empty callable.
   
   I inspected the public `4.1.4` source at `4accb51bc1f27d84980668f3a2eaf0ac96fb3cba` and master at `dd456ab7a894f54c6553415223c6d5c9c7d4e035`. The issue currently has no labels. This was a read-only source investigation; I did not run the proposed injection test or reproduce the production crashes.
   
   **Verified mechanism**
   
   - [`BetaRowsetReader::_init_iterator_once`](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/be/src/storage/rowset/beta_rowset_reader.cpp) directly returns `_init_iter_once.call(...)`. `_init_iterator` builds segment iterators, constructs the merge/union iterator, and calls its `init`. Its non-OK Status handling does not handle an exception that bypasses the return.
   - [`DorisCallOnce::call`](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/be/src/util/once.h) catches all exceptions, stores `exception_ptr`, publishes the called flag, and rethrows. Later calls rethrow the saved exception without rerunning initialization. This is deliberate behavior, with synchronized publication; the supplied stack does not establish a race in DorisCallOnce itself.
   - [`_next_batch`](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/be/src/storage/rowset/beta_rowset_reader.h) uses `RETURN_IF_ERROR` around initialization. That checks a returned Status, not a thrown exception. The scanner's [`RETURN_IF_CATCH_EXCEPTION` and `ASSIGN_STATUS_IF_CATCH_EXCEPTION`](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/be/src/common/exception.h) catch only `doris::Exception`. A plain `std::bad_function_call` therefore escapes these guards. This supports the reported terminate path, conditional on that exception being thrown.
   
   **PR scope and validation**
   
   Wrapping the entire `_init_iter_once.call(...)` invocation is a reasonable narrow boundary: it handles both the initial throw and a later cached rethrow. Catch `doris::Exception` before `std::exception`, preserve Doris error codes and the existing memory-error mapping, and retain `what()` plus reader identity for standard exceptions. Return a non-OK Status so `_next_batch` stops before using a partially initialized iterator. Keep DorisCallOnce unchanged and do not retry initialization on that same reader: initialization mutates state before completion.
   
   There is a scope limit worth documenting. [`VUnionIterator::next_batch`](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/be/src/storage/iterator/vgeneric_iterators.cpp) initializes subsequent segments after initial rowset initialization has succeeded. Those calls occur outside `_init_iterator_once`. Logging in [`LazyInitSegmentIterator::init`](https://github.com/apache/doris/blob/4accb51bc1f27d84980668f3a2eaf0ac96fb3cba/be/src/storage/segment/lazy_init_segment_iterator.cpp) followed by rethrow will provide diagnostics but will not contain such later exceptions. Include a later-segment injection case, and either cover that boundary or explicitly track it as a follow-up; do not describe the narrow patch as protecting every scan-time exception.
   
   For the proposed test, assert that the first and second `next_batch` on the **same reader** return non-OK Status values containing the exception message, and that the injected initializer ran only once. Also check ordinary returned errors, Doris exception code preservation, successful reads, and query-level error propagation without BE termination. Record the exact tested commit and commands in the PR.
   
   **Related fixes and remaining evidence**
   
   [#68675](https://github.com/apache/doris/pull/68675) changes failed segment-row-count caching on a long-lived BetaRowset. It does not change this reader's exception boundary, and its retry policy should not be copied into partially initialized reader state. [#61077](https://github.com/apache/doris/pull/61077) fixes an unsynchronized AnalysisFactoryMgr registry initialization check. In the inspected `4.1.4` and master code, `create()` already calls `initialise()` unconditionally before registry lookup. There is no evidence here that the previously fixed race is the current throw site.
   
   To establish the original cause, please attach:
   
   - The affected BE's full version/build SHA and any local patches, together with symbolized crash stacks and BE logs around the first initialization failure and timeout/cancellation, correlated by query ID.
   - A sanitized table DDL, actual auto-analyze SQL and EXPLAIN, relevant timeout/session settings, and a minimal workload description sufficient to distinguish merge/union and statistics/full-scan paths.
   - First-throw evidence identifying the callable: step diagnostics around segment loading, `Segment::new_iterator`, and inner iterator initialization, with tablet/rowset/segment IDs and exception type/message. In a controlled reproduction, breaking on the standard-library bad-function-call throw helper, or using `catch throw`, can capture the original stack before DorisCallOnce caches/rethrows it.
   - The hardening patch and reproducible debug-point/unit-test setup, including the first-call, cached-rethrow, and later-segment results.
   
   Timeout, heavy ingest, cloud mode, and arm64 are reported correlations, not verified causes. The current evidence supports containment work, but does not identify an empty callable, cancellation race, or architecture-specific defect.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org