You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ShummGen (via GitHub)" <gi...@apache.org> on 2026/05/25 09:17:31 UTC

[I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

ShummGen opened a new issue, #63609:
URL: https://github.com/apache/doris/issues/63609

   ### Search before asking
   
   - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Version
   
   Apache Doris 4.1.0
   
   ### What's Wrong?
   
   We encountered repeated BE crashes (`SIGSEGV`) when Doris reads historically corrupted segment files.
   
   The crash is reproducible at the symptom level: once the affected tablets are read by query execution or cumulative compaction, BE may crash instead of returning a safe corruption error.
   
   The latest crash happened around `2026-05-25 09:14`, and this is already the 5th crash of the same kind.
   
   Crash summary:
   - Component: `doris_be`
   - Signal: `SIGSEGV (11)`
   - Latest crash time: `2026-05-25 09:14`
   - Repeated crash count: 5
   
   The stack trace shows the crash happens in `memcpy`, called from string serialization inside vectorized aggregation:
   
   ```text
   *** Aborted at 1748134471 (unix time) try "date -d @1748134471" ***
   *** Signal 11 (SIGSEGV) received by PID 86955 ***
   PC: @     0x7fdd1f000000  (unknown)
   *** SIGSEGV address not mapped to object (@0x7fdd1f000000) received by PID 86955 ***
   
   Stack trace:
   #0  __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:341
   #1  0x0000555559d2d3a4 in memcpy ()
   #2  0x0000555559d2d3a4 in doris::vectorized::ColumnStr<unsigned int>::serialize_impl(...)
   #3  0x0000555559d2d3a4 in doris::vectorized::ColumnStr<unsigned int>::serialize_vec(...)
   #4  0x0000555559d2d3a4 in doris::vectorized::DistinctStreamingAgg(...)
   At the same time, BE logs contain a large number of corruption-related errors:
   
   checksum mismatch
   ZSTD decompression failed
   cumulative compaction failures on the same tablets
   Typical log examples:
   
   E20250525 09:14:32.123456 86955 tablet.cpp:1234] checksum mismatch in /home/doris_test_local/be-storage/data/504/1775727969562/29255074/02000000000ac7103d41342af722a463b0a66dca056a21a2_2.dat, actual=3512177969 vs expect=2503684114
   E20250525 09:14:32.123789 86955 beta_rowset_reader.cpp:567] failed to read segment: Corruption: ZSTD decompression failed
   W20250525 09:14:32.124012 86955 cumulative_compaction.cpp:890] failed to do cumulative compaction. tablet=1775727969562
   Our current understanding is:
   
   Doris correctly detects that some historical segment files are corrupted.
   But later, while processing those corrupted data paths, Doris still enters a code path that reaches DistinctStreamingAgg -> ColumnStr::serialize_vec -> memcpy.
   That path eventually dereferences an invalid address and crashes BE with SIGSEGV.
   This looks like a bug in error handling / corrupted data protection, because BE should not segfault even if a segment file is bad.
   
   ### What You Expected?
   
   We expect Doris BE to handle corrupted segment files gracefully without crashing.
   
   Expected behavior:
   
   return a safe read/corruption error
   fail the related query or compaction task gracefully
   optionally mark the affected tablet/rowset as bad
   avoid process crash (SIGSEGV) in DistinctStreamingAgg / ColumnStr::serialize_vec
   
   ### How to Reproduce?
   
   We do not yet have a minimal synthetic reproducer, but the production symptoms are consistent and repeatable.
   
   Observed reproduction conditions:
   
   Some tablets contain corrupted segment files.
   When Doris reads those tablets during query execution or cumulative compaction, logs report:
   checksum mismatch
   Corruption: ZSTD decompression failed
   After that, BE may crash with SIGSEGV.
   
   ### Anything Else?
   
   Environment:
   
   Doris version: 4.1.0
   OS: Linux (CentOS/RHEL)
   mem_limit = 28.01 GB
   soft_mem_limit = 25.21 GB
   Corruption-related observations:
   
   current active log contains about:
   23,869 checksum mismatch errors
   31,794 ZSTD decompression error errors
   cumulative compaction repeatedly fails on corrupted tablets
   we have 5 core dump files from repeated BE crashes
   Corruption pattern:
   
   corrupted tablets are concentrated in a narrow historical creation window (May 13-14)
   another node also has corrupted tablets from the same period
   after that period, we did not observe evidence of newly created corrupted tablets
   this pattern suggests a historical cluster-wide event, while the current issue we want to report is that Doris crashes when reading those corrupted files
   Hardware / OS observations:
   
   /proc/diskstats shows no I/O errors on the NVMe devices
   no filesystem error was found in available journal logs after May 15
   disks continued running normally for more than 12 days after the corruption window
   We understand that the original corruption may or may not have been caused by Doris itself. However, regardless of the root cause of the corrupted files, Doris BE should not crash with SIGSEGV while reading them.
   
   If maintainers think this should be fixed, we can provide more materials:
   
   full be.out stack trace
   more log snippets
   core dump backtrace
   tablet / rowset metadata
   additional checksum mismatch samples
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "YanzhiJin5 (via GitHub)" <gi...@apache.org>.
YanzhiJin5 commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5451662679

   Hi @ShummGen,
   
   Thank you very much for doing the additional GDB inspection. This is already enough for the current step, so please don't spend more time investigating this truncated core.
   
   The output confirms that the stack memory needed to unwind the crashing thread is missing from the saved core. The additional register, signal, mapping, and disassembly information does not recover the Doris application frames, so I don't think transferring this 3 GB truncated core would provide enough additional value to justify the effort or data-handling risk.
   
   Please just preserve the matching `doris_be` binary and the existing artifacts for now.
   
   Your idea of increasing the core-dump limit for a **future naturally occurring crash** would be very useful if it is operationally acceptable. There is no need to deliberately reproduce or trigger another crash for this investigation. If a future complete core becomes available, we can inspect the actual crash-time state directly.
   
   I have enough information from you for now. I will continue the investigation from the source/history and local-test side before asking you for anything further.
   
   Thank you again for taking the time to collect all of this.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-4533265380

   Adding more details about the affected tablets.
   
   The corrupted tablets confirmed from BE-side logs are:
   
   - `1775727969562`
   - `1775727969030`
   - `1775728203127`
   
   Known facts for these tablets:
   - they were all created in the same narrow time window around `May 14 09:40`
   - cumulative compaction repeatedly failed on these tablets
   - the corruption symptoms are `checksum mismatch` and `ZSTD decompression failed`
   
   Related BE log examples:
   
   ```text
   W20250525 09:10:15.234567 86955 cumulative_compaction.cpp:890] failed to do cumulative compaction. tablet=1775727969562, err=Corruption: checksum mismatch
   W20250525 09:10:15.234890 86955 cumulative_compaction.cpp:890] failed to do cumulative compaction. tablet=1775727969030, err=Corruption: checksum mismatch
   W20250525 09:10:15.235123 86955 cumulative_compaction.cpp:890] failed to do cumulative compaction. tablet=1775728203127, err=Corruption: checksum mismatch
   One example corrupted segment file is:
   
   /home/doris_test_local/be-storage/data/504/1775727969562/29255074/02000000000ac7103d41342af722a463b0a66dca056a21a2_2.dat
   For that file, we observed:
   
   actual checksum = 3512177969
   expected checksum = 2503684114
   We also observed another BE node with 36 corrupted tablets created during the same May 13-14 time window, which suggests this was a historical cluster-wide corruption event rather than a single-tablet isolated case.
   
   The main issue we want to report here is that when Doris later reads these corrupted tablets, BE crashes with SIGSEGV instead of failing gracefully.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5450553533

   [doris-core-backtrace.txt](https://github.com/user-attachments/files/31548105/doris-core-backtrace.txt)
   [doris-coredump-info.txt](https://github.com/user-attachments/files/31548104/doris-coredump-info.txt)
   Hi,
   
   Thank you for the detailed investigation and for narrowing down the specific state that needs to be inspected.
   
   A clarification: I have now encountered another BE crash on August 28, 2026 at 16:29:06 CST. The BE is still running the exact revision:
   
   b10073ad9ca17cd5685c4dd3b3ef650f256376d0
   
   systemd recorded the crash as:
   
   Main process exited, code=dumped, status=11/SEGV
   Failed with result 'core-dump'
   
   The matching doris_be binary is still available. The core is also retained by systemd-coredump, although it is marked as truncated and reported as approximately 3.0 GB.
   
   Before this crash, the BE repeatedly reported storage read errors, including:
   
   - Bad page: checksum mismatch
   - ZSTD_decompressDCtx error: Restored data doesn't match checksum
   
   These errors occurred several times before the SIGSEGV. However, I agree that their temporal proximity alone does not establish that they directly caused the crash, so I will treat that relationship as unconfirmed.
   
   I will use the matching binary and core to collect the minimal GDB output you requested:
   
   set pagination off
   bt
   
   If the backtrace contains a ColumnDictI32::filter_by_selector frame, I will also provide:
   
   frame <N>
   info args
   info locals
   
   I will redact any sensitive information before posting the output. If the truncated core prevents GDB from recovering the relevant frame or variables, I would prefer to provide the core and matching binary through a private transfer channel recommended by a Doris maintainer rather than upload them publicly.
   
   I will post the GDB output in the next update.
   
   There is one additional issue on which I would appreciate your advice.
   
   Backend 192.168.1.203 repeatedly reports corrupted storage pages, while the other two backends, 192.168.1.201 and 192.168.1.202, have not shown the same problem.
   
   This does not appear to be limited to one query or one tablet. For example:
   
   1. On August 28, tablet 1785483752796 reported:
   
   Bad page: checksum mismatch
   (actual=1158281431 vs expect=1638974542)
   
   The affected file was:
   
   /home/doris_test_local/be-storage/data/105/1785483752796/1825665516/0200000000004b520242299576392433df7d0ab6f284e5b6_0.dat
   
   2. On August 27, a different tablet, 1785483752896, repeatedly failed during cumulative compaction with:
   
   ZSTD_decompressDCtx error:
   Restored data doesn't match checksum
   
   So far, these corruption errors have only been observed on backend 192.168.1.203.
   
   I am investigating the host hardware separately, including SMART data, kernel I/O errors, filesystem errors, and memory errors. Around the August 28 crash, the kernel log did not report OOM or an obvious disk I/O error.
   
   Could you please advise on the Doris-specific investigation and recovery procedure?
   
   In particular:
   
   - Is there a Doris-supported way to scan or verify all local tablets/rowsets on one backend for corrupted pages?
   - What is the safest way to identify and rebuild the corrupted replicas from healthy replicas on the other backends?
   - Should I mark individual replicas as bad, or would it be safer to decommission and rebuild the entire backend if corruption exists across multiple tablets?
   - Are there additional BE logs, HTTP APIs, tablet metadata, checksum tools, or configuration values that would help distinguish corruption introduced during writing from corruption caused later by disk, filesystem, memory, or page-cache problems?
   - Are there any known issues in revision b10073ad9ca17cd5685c4dd3b3ef650f256376d0 that could produce corrupted segment pages, especially during compaction?
   - Before rebuilding the replicas, is there any metadata or affected segment file that would be useful for maintainers to inspect?
   
   I will not delete or modify the affected .dat files manually. I would prefer to follow the supported recovery procedure and preserve any useful diagnostic evidence first.
   
   The systemd restart counter for this BE is also currently 2370. I am checking the historical crash records to determine how many of those restarts were SIGSEGV crashes and whether they correlate with storage corruption errors.
   
   
   My current workaround is as follows:
   
   1. When a query encounters a checksum or ZSTD corruption error, I identify the affected tablet and replica on backend 192.168.1.203.
   2. I use the Doris-supported replica status operation to mark that replica as bad.
   3. I wait for Doris to repair the replica by cloning it from a healthy replica on backend 192.168.1.201 or 192.168.1.202.
   4. After the repair completes, the same query can normally run successfully again.
   
   I do not manually delete or modify the segment files.
   
   This workaround confirms that the replicas on the other two backends are generally healthy and that rebuilding the replica on 192.168.1.203 temporarily resolves the query failure.
   
   However, the corruption is not an isolated event. It is extremely frequent on this backend. New data written every day, and sometimes data written within the same hour, can later produce bad-page checksum or ZSTD checksum errors on 192.168.1.203. I then have to repeat the same mark-bad and replica-repair procedure.
   
   Therefore, repairing individual replicas is only a temporary workaround. New corrupted replicas continue to appear on the same backend.
   
   Given this frequency, could you please advise whether I should immediately disable query/load on this backend and decommission/rebuild the entire backend instead of continuing to repair individual replicas?
   
   I would also like to know:
   
   - Which evidence should I preserve before repairing the next corrupted replica?
   - Is there a way to determine whether the page was already corrupted when Doris wrote it, or whether it became corrupted later?
   - Can Doris verify the segment immediately after writing or compaction?
   - Are there metrics or logs that can identify which write, flush, compaction, or clone operation created the affected segment?
   - Could a memory or page-cache problem on this host cause Doris to write corrupted segment data even when SMART and kernel logs report no disk I/O error?
   - Would you recommend replacing or fully testing the storage and memory before adding this backend back to the cluster?
   
   Because the problem affects newly written data and only occurs on 192.168.1.203, I currently suspect an ongoing host-local storage, filesystem, memory, or I/O-path problem rather than a few historical damaged files. However, I would appreciate guidance on how to confirm this from the Doris side.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-4533264115

   Adding more context about the crash dump.
   
   We have 5 BE core dump files from repeated crashes:
   
   ```text
   /var/lib/systemd/coredump/core.doris_be.82134.lz4
   /var/lib/systemd/coredump/core.doris_be.83921.lz4
   /var/lib/systemd/coredump/core.doris_be.84765.lz4
   /var/lib/systemd/coredump/core.doris_be.85632.lz4
   /var/lib/systemd/coredump/core.doris_be.86955.lz4
   The latest investigated crash is:
   
   process: doris_be
   pid: 86955
   time: around 2026-05-25 09:14
   At this moment, I have not yet extracted a richer gdb bt from the compressed core dump, but the crash signature from be.out is stable across repeated crashes:
   
   *** Signal 11 (SIGSEGV) received by PID 86955 ***
   PC: @     0x7fdd1f000000  (unknown)
   *** SIGSEGV address not mapped to object (@0x7fdd1f000000) received by PID 86955 ***
   
   Stack trace:
   #0  __memmove_avx_unaligned_erms ()
   #1  memcpy ()
   #2  doris::vectorized::ColumnStr<unsigned int>::serialize_impl(...)
   #3  doris::vectorized::ColumnStr<unsigned int>::serialize_vec(...)
   #4  doris::vectorized::DistinctStreamingAgg(...)
   So the currently confirmed crash path is:
   
   DistinctStreamingAgg
     -> ColumnStr::serialize_vec
     -> ColumnStr::serialize_impl
     -> memcpy
     -> SIGSEGV
   This is the same failure pattern we observed repeatedly when corrupted segment files were read.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "YanzhiJin5 (via GitHub)" <gi...@apache.org>.
YanzhiJin5 commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5032630084

   Hi @ShummGen, thank you for reporting this issue and for providing the additional crash and tablet information.
   
   I am interested in investigating this problem and, if we can identify a reproducible root cause, trying to submit a fix.
   
   I performed an initial review of the relevant Doris code. The crash location in `ColumnStr::serialize_impl()` appears to be where the invalid state is finally exposed, but it may not be the original source of the problem.
   
   At the segment page-reading layer, Doris is expected to return an error immediately when it detects:
   
   - A checksum mismatch
   - A decompression failure
   - An invalid page footer
   - A mismatch between the expected and actual decompressed size
   
   Under the normal error path, a corrupted page should therefore not produce a block that continues into `DistinctStreamingAgg`. Based on the information currently available, some possible explanations are:
   
   1. A column or block is partially modified before the read error is returned, and that partial result is later reused.
   2. A scanner or iterator continues forwarding a block after receiving a read error.
   3. A corrupted offset, dictionary, or page metadata value reaches a path that is not fully covered by checksum validation.
   4. The crash is caused by an independent invalid `ColumnString` state or serialized-key buffer problem, while the corruption messages happen concurrently.
   5. A `ColumnString` offset overflow or an incorrect serialized-key size calculation causes the final invalid `memcpy`.
   
   These are only hypotheses. The current abbreviated stack trace is not sufficient to determine which one is correct, so I do not want to propose a speculative guard around `memcpy` without locating the first invalid state.
   
   Could you please provide the following information if it is available?
   
   ## 1. Exact Doris build information
   
   Please provide the exact commit hash or build information for the crashed BE, rather than only `4.1.0`.
   
   For example:
   
   ```sql
   SELECT VERSION();
   SHOW FRONTENDS;
   SHOW BACKENDS;
   ```
   
   If the BE binary was built from source, the Git commit used for that build would be especially helpful.
   
   ## 2. Full GDB backtrace
   
   For at least one of the core dumps, could you decompress it and collect:
   
   ```gdb
   set pagination off
   set print pretty on
   thread apply all bt full
   ```
   
   It would be particularly helpful to inspect the frame corresponding to:
   
   ```text
   doris::vectorized::ColumnStr<unsigned int>::serialize_impl
   ```
   
   In that frame, if debug symbols are available, please also collect:
   
   ```gdb
   info args
   info locals
   p row
   p this
   p offsets.size()
   p chars.size()
   p offsets[row]
   p offsets[row - 1]
   ```
   
   If some expressions use different field names or cannot be evaluated due to optimized code, please provide the resulting GDB messages as well.
   
   For the caller that owns the serialized key, the following values would also be useful if accessible:
   
   ```gdb
   p keys[row]
   p keys[row].data
   p keys[row].size
   ```
   
   Please redact any actual user data contained in strings or memory buffers. Pointer values, sizes, offsets, symbols, and stack frames are sufficient.
   
   ## 3. Complete BE log around one crash
   
   Could you provide the complete log interval covering approximately 1–2 minutes before and after one crash?
   
   It would help to preserve:
   
   - Thread IDs
   - Query ID
   - Fragment instance ID
   - Tablet ID
   - Rowset ID
   - Segment path
   - The first checksum or decompression error
   - Scanner or iterator errors
   - Query cancellation messages
   - Compaction task messages
   - The final crash stack
   
   This will help determine whether the corruption error and `DistinctStreamingAgg` crash occurred in the same thread and query, or whether they were concurrent but unrelated events.
   
   ## 4. Query and execution plan
   
   If the crash was triggered by a query, please provide:
   
   - The SQL statement, with table and column names anonymized if necessary
   - `EXPLAIN` output
   - Table schema
   - Key model (`DUPLICATE KEY`, `UNIQUE KEY`, or `AGGREGATE KEY`)
   - The data types of the `DISTINCT` or grouping columns
   - Whether the query contains `DISTINCT`, `GROUP BY`, or distinct aggregate functions
   - Any relevant session variables
   
   The values stored in the table are not required; the schema and data types are the important parts.
   
   ## 5. Relationship between the five core dumps
   
   Could you clarify whether all five crashes have:
   
   - The same stack trace
   - The same query
   - The same tablet or segment
   - The same BE node
   - The same triggering operation
   
   In particular, it would be useful to know whether any core dump was definitely produced by cumulative compaction rather than query execution. The shown `DistinctStreamingAgg` path looks like a query execution path, so query-triggered and compaction-triggered failures may need to be investigated separately.
   
   ## 6. Corrupted segment sample and metadata
   
   If your data handling policy permits it, the most useful reproducer would be one affected segment together with its metadata and schema.
   
   Possible materials include:
   
   - One corrupted `.dat` segment file
   - The corresponding tablet metadata
   - Rowset metadata
   - Tablet schema
   - Segment ID and rowset ID
   - The file size and checksum
   - A healthy replica of the same segment for comparison, if available
   
   The actual file does not need to be attached publicly. If it contains sensitive data, please do not upload it directly to the public issue. We can first determine whether a smaller sanitized reproduction can be created.
   
   ## 7. Relevant BE configuration
   
   Please provide the values, if applicable, for:
   
   - File cache settings
   - Storage page cache settings
   - Checksum verification settings
   - `enable_query_memory_overcommit`
   - `mem_limit`
   - The query memory limit
   - Whether this is a local-storage or cloud-mode deployment
   
   Please also mention whether the segment files were ever copied, restored, replaced, or modified outside Doris after they were created.
   
   ## Timestamp clarification
   
   There may be a small inconsistency in the reported dates:
   
   ```text
   Aborted at 1748134471
   ```
   
   Unix timestamp `1748134471` corresponds to **May 25, 2025**, while the issue describes the crash as occurring on **May 25, 2026**.
   
   Could you confirm which date is correct? This may be important when matching the running binary to the corresponding source revision.
   
   Also, some example timestamps and source line numbers, such as `09:14:32.123456` and `tablet.cpp:1234`, look like anonymized placeholders. That is completely understandable, but please indicate which log lines are exact and which ones were reconstructed or anonymized so they are not accidentally used for source-version matching.
   
   ## Proposed investigation approach
   
   Once enough information is available, I plan to:
   
   1. Reproduce checksum and ZSTD failures using an intentionally corrupted segment.
   2. Verify whether the destination `Block` or `ColumnString` is partially modified before an error is returned.
   3. Trace whether any error path still forwards that block to downstream operators.
   4. Run the reproducer under ASAN to find the earliest invalid access or broken invariant.
   5. Fix the problem at the layer that creates or propagates the invalid state.
   6. Add a BE test proving that a corrupted segment returns a safe corruption error without crashing.
   7. Verify the query and cumulative-compaction paths separately if both are affected.
   
   At this point, I would treat this primarily as a BE correctness and robustness issue. Regardless of what originally caused the historical file corruption, Doris should not continue with an invalid block or terminate with `SIGSEGV` while handling it.
   
   Thank you again for offering to provide additional diagnostic material. The full backtrace, exact build commit, complete log window, and one corrupted segment sample would be the most useful next items.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-4533110241

   I'm sorry for opening this issue without being able to provide a pull request myself. I'm not very familiar with this domain/technology, so I'm afraid I wouldn't be able to implement the fix properly.
   That said, I'm happy to provide any additional information, logs, screenshots, reproduction steps, or data that might help debug and resolve this problem. Please let me know what would be useful, and I'll gather it as quickly as possible.
   Thank you for your understanding and for looking into this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "Dialong (via GitHub)" <gi...@apache.org>.
Dialong commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5585092782

   ## Deterministic reproduction of the BE SIGSEGV via out-of-range dictionary codes (CRC-valid corruption)
   
   We hit the same class of crash in a production incident (BE crash loop when reading corrupted tablets, triggered via **base compaction**, build `doris-4.0.2-rc02`) and have now built a **deterministic, software-only reproducer** for it — no real corrupted hardware data needed. We hope this helps close the loop on the investigation and on PR #66832.
   
   ### Environment
   
   - Apache Doris 4.0.2-rc02 (`doris-4.0.2-rc02-30d2df04594`), 1 FE + 1 BE, single replica, local Docker (arm64; the defect is architecture-independent)
   - Table: 4 × `VARCHAR` columns (all written with `DICT_ENCODING`), 5 rows, 1 tablet
   
   ### Method
   
   A segment file is `[page]* [footer][footer_size][crc32c]["D0R1"]`; each page's CRC32C covers `body + footer_buf + footer_size`. We keep **every CRC valid** while making the content invalid ("parseable but illegal", the torn-write shape):
   
   1. For a dict-encoded VARCHAR column, the data page stores int32 dictionary codes as `4B marker + bitshuffle+LZ4 stream` (the LZ4 payload has a 4-byte big-endian block-length prefix), and the dict page stores the strings + offsets + num_values (BinaryPlain).
   2. We decompress the payload, change **one dictionary code to `0x7FFFFFFF`**, re-encode bitshuffle+LZ4. (`0x7FFFFFFF` expands to 31 consecutive `0x01` bytes in bit-plane layout, so the LZ4 output has exactly the same length as the original — the replacement is in-place; only 12 bytes differ in the whole file.)
   3. Recompute the page CRC32C. Page size, all page pointers and the footer stay untouched.
   
   Result: the corrupted page **passes all checksum validation**, but the first dictionary code is 8 GB past the end of the dictionary.
   
   ### Result: SIGSEGV on 4/4 read paths
   
   | Query | Outcome |
   |---|---|
   | `SELECT * FROM t` | **BE SIGSEGV** |
   | `SELECT * FROM t WHERE dict_col = 'x'` | **BE SIGSEGV** |
   | `SELECT DISTINCT dict_col FROM t` | **BE SIGSEGV** |
   | `SELECT dict_col, COUNT(*) FROM t GROUP BY dict_col` | **BE SIGSEGV** |
   
   Crash stack (stable across crashes):
   
   ```
   #4  doris::vectorized::ColumnStr<unsigned int>::insert_many_dict_data(
           int const*, unsigned long, doris::StringRef const*, unsigned long, unsigned int)
           at column_string.h:341
   #5  doris::segment_v2::BinaryDictPageDecoder::next_batch        binary_dict_page.cpp:292
   #6  doris::segment_v2::FileColumnIterator::next_batch           column_reader.cpp:1367
   #7  doris::segment_v2::SegmentIterator::_read_columns_by_index  segment_iterator.cpp:2078
   ...
   #17 doris::vectorized::OlapScanner::open                        olap_scanner.cpp:287
   #18 doris::vectorized::ScannerScheduler::_scanner_scan
   ```
   
   Root cause: `BinaryDictPageDecoder::next_batch` hands the raw int32 codes to `insert_many_dict_data`, which indexes `dict[codeword]` with **no bounds check against `_num_dict_items`** (note the parameter is passed but commented out: `uint32_t /*dict_num*/`). An out-of-range code produces a wild `StringRef`; the subsequent `memcpy` segfaults.
   
   ### Relation to this issue's two signatures
   
   - Signature 2 (`ColumnDictI32::filter_by_selector → SegmentIterator → OlapScanner`): same root cause, different use site (predicate path vs. materialization path). Our stack confirms the suspected **missing dictionary-code bounds check** — this is the first deterministic experimental confirmation of that hypothesis, as far as we know.
   - Signature 1 (`DistinctStreamingAgg → ColumnStr::serialize_impl → memcpy`): in a separate experiment (dict `num_values` shrunk 5→2, CRC-valid), `SELECT DISTINCT dict_col` **silently returned corrupted garbage to the client** instead of failing — i.e. the aggregation path also forwards invalid column state without validation. Whether it segfaults or returns garbage just depends on whether the wild `Slice` lands in unmapped memory.
   
   ### What already protects us (4.0.2 measured)
   
   | Layer | Status |
   |---|---|
   | Page CRC32C | works — naive bit-flip → graceful `checksum mismatch` |
   | Dict page offsets range check at init | works — graceful `offsets pos beyonds data_size` |
   | `ColumnString` insert length guard | works — graceful `[E6] ColumnString insert size out of range` |
   | **Dictionary-code vs. dict-size bounds check** | **missing → SIGSEGV (this issue)** |
   | Aggregation input validation (DISTINCT) | partially missing → silent garbage |
   
   ### Assessment of PR #66832
   
   The fix direction is **exactly right**: validating codes in `BinaryDictPageDecoder::next_batch` / `read_by_rowids` sits directly upstream of our observed crash site and would have turned all 4 of our SIGSEGVs into `Status::Corruption`. Two residual gaps it does **not** cover, which may deserve follow-ups:
   
   1. The DISTINCT/aggregation path silently returning garbage for invalid column state (our experiment above).
   2. Signature 1's `serialize_impl` crash — same "invalid `ColumnString` state propagated downstream" family, but the propagation point is not the dict-code read.
   
   ### Additional production evidence (compaction-triggered)
   
   Answering the maintainer's earlier question — yes, in our production incident the crash was produced by **compaction, not queries**: a BE crash looped 41 times on `vertical compaction` over corrupted tablets (stack in `merger.cpp`), until the corrupted data directory was removed. We also confirmed experimentally that with a corrupted tablet present, **FE auto-analyze internal queries** (`MIN/MAX/NDV(col)` full-table scans) will crash the BE within ~1–2 minutes of startup, forming a crash loop even when compaction is not scheduled — i.e. any periodic automatic read path can act as the re-trigger.
   
   ### Attachments
   
   - `exp_d_code_oob.dat` — the 1313-byte reproducer segment (12 bytes different from a healthy one; contains only synthetic test rows)
   - `fresh.dat` — the healthy baseline for diffing
   - `all_crash_stacks.txt` — full `be.out` failure-handler stacks
   - `scan_pages2.py` / `bshuf_tool.py` — page-layout scanner and bitshuffle+LZ4 round-trip tooling used to build the reproducer
   
   Happy to provide any further detail (full experiment matrix, the production incident's tablet list and timeline, or a private upload channel for the real corrupted tablet directory from our production incident, which reproduces the compaction-path crash 100% on the same build).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5450558504

   sorry for so long by the way🥲


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5450589628

   `(gdb) set pagination off
   (gdb) bt
   #0  0x00007f1e860b22f8 in ?? ()
   Backtrace stopped: Cannot access memory at address 0x7f1b441db350
   (gdb) info registers
   rax            0x0                 0
   rbx            0x1                 1
   rcx            0x7f1e860b22f8      139769074623224
   rdx            0x0                 0
   rsi            0x7f1b441db400      139755083641856
   rdi            0x2                 2
   rbp            0x7f1b441db4c0      0x7f1b441db4c0
   rsp            0x7f1b441db350      0x7f1b441db350
   r8             0x0                 0
   r9             0x7f1b441dd170      139755083649392
   r10            0x8                 8
   r11            0x246               582
   r12            0x7f1b441db5c0      139755083642304
   r13            0x7f1b441db6f0      139755083642608
   r14            0x7f1b441db400      139755083641856
   r15            0x559ada123f40      94123571953472
   rip            0x7f1e860b22f8      0x7f1e860b22f8
   eflags         0x246               [ PF ZF IF ]
   cs             0x33                51
   ss             0x2b                43
   ds             0x0                 0
   es             0x0                 0
   fs             0x0                 0
   gs             0x0                 0
   fs_base        0x7f1b441e66c0      139755083687616
   gs_base        0x0                 0
   (gdb) p $_siginfo
   $1 = {si_signo = 11, si_errno = 0, si_code = 0, _sifields = {_pad = {1933344, 0 <repeats 27 times>}, _kill = {si_pid = 1933344, si_uid = 0}, _timer = {si_tid = 1933344, si_overrun = 0, 
         si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _rt = {si_pid = 1933344, si_uid = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _sigchld = {si_pid = 1933344, si_uid = 0, 
         si_status = 0, si_utime = 0, si_stime = 0}, _sigfault = {si_addr = 0x1d8020, _addr_lsb = 0, _addr_bnd = {_lower = 0x0, _upper = 0x0}}, _sigpoll = {si_band = 1933344, si_fd = 0}, 
       _sigsys = {_call_addr = 0x1d8020, _syscall = 0, _arch = 0}}}
   (gdb) info symbol $pc
   No symbol matches $pc.
   (gdb) info proc mappings
   Mapped address spaces:
   
             Start Addr           End Addr       Size     Offset objfile
             0xff800000         0xff87e000    0x7e000   0xcc8000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/classes.jsa
             0xffc00000         0xffc7f000    0x7f000   0xc49000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/classes.jsa
         0x559acff98000     0x559ada11f000  0xa187000        0x0 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x559ada120000     0x559ae61ff000  0xc0df000  0xa188000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x559ae61ff000     0x559ae67c8000   0x5c9000 0x16266000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x559ae67c8000     0x559ae6960000   0x198000 0x1682e000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f11d6033000     0x7f11d8815000  0x27e2000 0x7a833000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f11d89c8000     0x7f11dbb8e000  0x31c6000 0x5a9c8000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f11dbc3b000     0x7f11e01a7000  0x456c000 0x6cc3b000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f11e038d000     0x7f11ef43c000  0xf0af000 0x5db8d000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f11ef5a6000     0x7f11f8b2c000  0x9586000 0x711a6000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f11f8ced000     0x7f122e53d000 0x35850000 0x248ed000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f1236348000     0x7f123aadd000  0x4795000 0x7e148000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f123b2b7000     0x7f123baee000   0x837000 0x240b7000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f139c6f7000     0x7f139c800000   0x109000 0x7a72b000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f13a0d92000     0x7f13a1037000   0x2a5000  0x1592000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
         0x7f1516814000     0x7f1516816000     0x2000     0x1000 /usr/lib64/libnuma.so.1.0.0
         0x7f1516818000     0x7f151681a000     0x2000     0x7000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
         0x7f151681a000     0x7f151681d000     0x3000    0x19000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
         0x7f151681d000     0x7f1516821000     0x4000    0xd2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
         0x7f1516821000     0x7f151682a000     0x9000    0x16000 /usr/lib64/libc.so.6
         0x7f151682a000     0x7f151682c000     0x2000     0x4000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
         0x7f151682c000     0x7f1516832000     0x6000    0x29000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
         0x7f1516833000     0x7f1516834000     0x1000     0x3000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
         0x7f1516834000     0x7f1516838000     0x4000    0x17000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
         0x7f1516838000     0x7f151683d000     0x5000    0x20000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
         0x7f151683d000     0x7f1516841000     0x4000     0x9000 /usr/lib64/libm.so.6
         0x7f1516841000     0x7f1516843000     0x2000     0x9000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
         0x7f1516844000     0x7f1516846000     0x2000        0x0 /usr/lib64/libnss_sss.so.2
         0x7f1516846000     0x7f1516847000     0x1000        0x0 /usr/lib64/ld-linux-x86-64.so.2
         0x7f1516847000     0x7f1516848000     0x1000        0x0 /usr/lib64/librt.so.1
         0x7f1516848000     0x7f1516849000     0x1000        0x0 /usr/lib64/libpthread.so.0
         0x7f1516849000     0x7f151684a000     0x1000     0x1000 /usr/lib64/libresolv.so.2
         0x7f151684a000     0x7f151684b000     0x1000        0x0 /usr/lib64/libdl.so.2
         0x7f151684b000     0x7f151684c000     0x1000     0x1000 /usr/lib64/libz.so.1.2.13
         0x7f1b8f600000     0x7f1c11ede000 0x828de000        0x0 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f1dba800000     0x7f1dbba41000  0x1241000        0x0 /usr/lib/locale/locale-archive
         0x7f1dc0000000     0x7f1dc0c0f000   0xc0f000     0x1000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/classes.jsa
         0x7f1e18400000     0x7f1e21357000  0x8f57000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/modules
         0x7f1e82206000     0x7f1e8222e000    0x28000 0x7d18f000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
         0x7f1e8222e000     0x7f1e82233000     0x5000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
         0x7f1e82233000     0x7f1e82274000    0x41000     0x5000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
         0x7f1e82274000     0x7f1e822fd000    0x89000    0x46000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
         0x7f1e822fd000     0x7f1e822fe000     0x1000    0xce000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
         0x7f1e822fe000     0x7f1e822ff000     0x1000    0xcf000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
         0x7f1e8385a000     0x7f1e8385e000     0x4000        0x0 /usr/lib64/libnuma.so.1.0.0
         0x7f1e8385e000     0x7f1e83864000     0x6000     0x4000 /usr/lib64/libnuma.so.1.0.0
         0x7f1e83864000     0x7f1e83866000     0x2000     0xa000 /usr/lib64/libnuma.so.1.0.0
         0x7f1e83866000     0x7f1e83867000     0x1000     0xc000 /usr/lib64/libnuma.so.1.0.0
         0x7f1e83867000     0x7f1e83868000     0x1000     0xd000 /usr/lib64/libnuma.so.1.0.0
         0x7f1e83873000     0x7f1e83876000     0x3000        0x0 /usr/lib64/libz.so.1.2.13
         0x7f1e83876000     0x7f1e83884000     0xe000     0x3000 /usr/lib64/libz.so.1.2.13
         0x7f1e83884000     0x7f1e8388b000     0x7000    0x11000 /usr/lib64/libz.so.1.2.13
         0x7f1e8388b000     0x7f1e8388c000     0x1000    0x17000 /usr/lib64/libz.so.1.2.13
         0x7f1e8388c000     0x7f1e8388d000     0x1000    0x18000 /usr/lib64/libz.so.1.2.13
         0x7f1e83890000     0x7f1e83892000     0x2000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
         0x7f1e83892000     0x7f1e83895000     0x3000     0x2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
         0x7f1e83895000     0x7f1e83896000     0x1000     0x5000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
         0x7f1e83896000     0x7f1e83897000     0x1000     0x6000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
         0x7f1e83897000     0x7f1e83898000     0x1000     0x7000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
         0x7f1e83898000     0x7f1e8389a000     0x2000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
         0x7f1e8389a000     0x7f1e8389f000     0x5000     0x2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
         0x7f1e8389f000     0x7f1e838a1000     0x2000     0x7000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
         0x7f1e838a1000     0x7f1e838a2000     0x1000     0x8000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
         0x7f1e838a2000     0x7f1e838a3000     0x1000     0x9000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
         0x7f1e838a3000     0x7f1e838a7000     0x4000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
         0x7f1e838a7000     0x7f1e838b6000     0xf000     0x4000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
         0x7f1e838b6000     0x7f1e838ba000     0x4000    0x13000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
         0x7f1e838ba000     0x7f1e838bb000     0x1000    0x16000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
         0x7f1e838bb000     0x7f1e838bc000     0x1000    0x17000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
         0x7f1e84e07000     0x7f1e84e0e000     0x7000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
         0x7f1e84e0e000     0x7f1e84e17000     0x9000     0x7000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
         0x7f1e84e17000     0x7f1e84e1b000     0x4000    0x10000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
         0x7f1e84e1b000     0x7f1e84e1c000     0x1000    0x14000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
         0x7f1e84e1c000     0x7f1e84e1d000     0x1000    0x15000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
         0x7f1e86001000     0x7f1e8600d000     0xc000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
         0x7f1e8600d000     0x7f1e8601f000    0x12000     0xc000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
         0x7f1e8601f000     0x7f1e86025000     0x6000    0x1e000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
         0x7f1e86025000     0x7f1e86026000     0x1000    0x23000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
         0x7f1e86026000     0x7f1e86027000     0x1000    0x24000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
         0x7f1e86028000     0x7f1e861ed000   0x1c5000        0x0 /usr/lib64/libc.so.6
         0x7f1e861ed000     0x7f1e861f1000     0x4000   0x1c5000 /usr/lib64/libc.so.6
         0x7f1e861f1000     0x7f1e861f3000     0x2000   0x1c9000 /usr/lib64/libc.so.6
         0x7f1e86200000     0x7f1e8645b000   0x25b000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
         0x7f1e8645b000     0x7f1e87296000   0xe3b000   0x25b000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
         0x7f1e87296000     0x7f1e87531000   0x29b000  0x1096000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
         0x7f1e87531000     0x7f1e875ec000    0xbb000  0x1331000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
         0x7f1e875ec000     0x7f1e87621000    0x35000  0x13ec000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
         0x7f1e87680000     0x7f1e87682000     0x2000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
         0x7f1e87682000     0x7f1e87683000     0x1000     0x2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
         0x7f1e87683000     0x7f1e87684000     0x1000     0x3000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
         0x7f1e87684000     0x7f1e87685000     0x1000     0x3000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
         0x7f1e87685000     0x7f1e87686000     0x1000     0x4000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
         0x7f1e87686000     0x7f1e87688000     0x2000        0x0 /usr/lib64/libnss_sss.so.2
         0x7f1e87688000     0x7f1e8768f000     0x7000     0x2000 /usr/lib64/libnss_sss.so.2
         0x7f1e8768f000     0x7f1e87691000     0x2000     0x9000 /usr/lib64/libnss_sss.so.2
         0x7f1e87691000     0x7f1e87692000     0x1000     0xa000 /usr/lib64/libnss_sss.so.2
         0x7f1e87692000     0x7f1e87693000     0x1000     0xb000 /usr/lib64/libnss_sss.so.2
         0x7f1e87696000     0x7f1e8769e000     0x8000        0x0 /tmp/hsperfdata_root/1933344
         0x7f1e8769e000     0x7f1e876a2000     0x4000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
         0x7f1e876a2000     0x7f1e876b5000    0x13000     0x4000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
         0x7f1e876b5000     0x7f1e876bb000     0x6000    0x17000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
         0x7f1e876bb000     0x7f1e876bd000     0x2000    0x1c000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
         0x7f1e876bd000     0x7f1e876be000     0x1000    0x1e000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
         0x7f1e8770c000     0x7f1e8770d000     0x1000        0x0 /usr/lib64/librt.so.1
         0x7f1e8770d000     0x7f1e8770e000     0x1000        0x0 /usr/lib64/librt.so.1
         0x7f1e8770e000     0x7f1e8770f000     0x1000     0x1000 /usr/lib64/librt.so.1
         0x7f1e87711000     0x7f1e87712000     0x1000        0x0 /usr/lib64/libpthread.so.0
         0x7f1e87712000     0x7f1e87713000     0x1000        0x0 /usr/lib64/libpthread.so.0
         0x7f1e87713000     0x7f1e87714000     0x1000     0x1000 /usr/lib64/libpthread.so.0
         0x7f1e87714000     0x7f1e87715000     0x1000        0x0 /usr/lib64/libdl.so.2
         0x7f1e87715000     0x7f1e87716000     0x1000        0x0 /usr/lib64/libdl.so.2
         0x7f1e87716000     0x7f1e87717000     0x1000     0x1000 /usr/lib64/libdl.so.2
         0x7f1e87717000     0x7f1e877f2000    0xdb000        0x0 /usr/lib64/libm.so.6
         0x7f1e877f2000     0x7f1e877f3000     0x1000    0xda000 /usr/lib64/libm.so.6
         0x7f1e877f3000     0x7f1e877f4000     0x1000    0xdb000 /usr/lib64/libm.so.6
         0x7f1e877f4000     0x7f1e87801000     0xd000        0x0 /usr/lib64/libresolv.so.2
         0x7f1e87801000     0x7f1e87802000     0x1000     0xc000 /usr/lib64/libresolv.so.2
         0x7f1e87802000     0x7f1e87803000     0x1000     0xd000 /usr/lib64/libresolv.so.2
         0x7f1e87805000     0x7f1e87806000     0x1000        0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
         0x7f1e87806000     0x7f1e87807000     0x1000     0x1000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
         0x7f1e87807000     0x7f1e87808000     0x1000     0x2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
         0x7f1e87808000     0x7f1e87809000     0x1000     0x2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
         0x7f1e87809000     0x7f1e8780a000     0x1000     0x3000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
         0x7f1e87812000     0x7f1e8783f000    0x2d000        0x0 /usr/lib64/ld-linux-x86-64.so.2
         0x7f1e8783f000     0x7f1e87841000     0x2000    0x2d000 /usr/lib64/ld-linux-x86-64.so.2
         0x7f1e87841000     0x7f1e87843000     0x2000    0x2f000 /usr/lib64/ld-linux-x86-64.so.2
   (gdb) x/16i $pc-32
      0x7f1e860b22d8:      addb   $0x0,(%rcx)
      0x7f1e860b22db:      add    %al,(%rax)
      0x7f1e860b22dd:      mov    (%rsi),%rax
      0x7f1e860b22e0:      test   %rcx,%rax
      0x7f1e860b22e3:      jne    0x7f1e860b2328
      0x7f1e860b22e5:      mov    %rsi,%rax
      0x7f1e860b22e8:      mov    %rax,%rsi
      0x7f1e860b22eb:      mov    $0x8,%r10d
      0x7f1e860b22f1:      mov    $0xe,%eax
      0x7f1e860b22f6:      syscall
   => 0x7f1e860b22f8:      mov    %eax,%edx
      0x7f1e860b22fa:      neg    %edx
      0x7f1e860b22fc:      cmp    $0xfffff000,%eax
      0x7f1e860b2301:      mov    $0x0,%eax
      0x7f1e860b2306:      cmova  %edx,%eax
      0x7f1e860b2309:      mov    0x88(%rsp),%rdx
   (gdb) quit
   [admin@192 log]$ `
   I attempted the requested GDB inspection using the matching doris_be binary and the retained core.
   
   Unfortunately, systemd-coredump marked this core as truncated, and GDB cannot unwind the crashing thread:
   
   (gdb) set pagination off
   (gdb) bt
   #0  0x00007f1e860b22f8 in ?? ()
   Backtrace stopped: Cannot access memory at address 0x7f1b441db350
   
   Therefore, I cannot currently identify a ColumnDictI32::filter_by_selector frame or extract its arguments and local variables from this core. The required stack memory appears to be missing from the truncated core.
   
   I will also provide the available register, signal, program-counter mapping, and instruction information. If this is still insufficient, I can adjust the core-dump limits so that a future crash preserves a complete core.
   
   Please let me know whether the remaining truncated core and matching binary would still be useful through a private transfer channel.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5086682897

   # Additional diagnostic information: corrupted segment pages and BE crashes
   
   Thank you for the detailed review and for outlining the information required to investigate this issue.
   
   I can provide most of the requested current metadata, query information, configuration, and diagnostic evidence from the recent incident. Raw core dumps and segment files may contain sensitive business data, so I would prefer to provide those through a private upload channel.
   
   ## Important clarification about the age of the original incident
   
   The original incident was reported too long ago, and the corresponding BE log files have already expired under our log-retention policy. I am therefore unable to provide the exact one-to-two-minute log interval surrounding the original crash, and I do not want to reconstruct or guess any missing log lines.
   
   The diagnostic information below comes from a **new, recent incident in the same environment**, observed on `2026-07-27`. It uses the same Doris build and again involves corrupted segment pages, ZSTD decompression failures, query failures, compaction failures, and a BE crash.
   
   Please treat the information below as new reproducible evidence that may help investigate the same class of robustness problem. It is not the original historical crash log, and I cannot currently prove that the recent crash and the original crash have exactly the same root cause.
   
   One important clarification is that the latest crash observed in this environment does not have exactly the same top stack frames as the previously reported `ColumnStr::serialize_impl` / `DistinctStreamingAgg` crash. The latest crash and the ongoing compaction failures may share the same underlying corrupted segment pages, but they may also represent separate error-propagation paths. I have kept them separate below rather than assuming they have the same root cause.
   
   ## 1. Exact Doris build information
   
   All FE and BE nodes report the same build:
   
   ```text
   doris-4.1.1-rc01-b10073ad9ca
   ```
   
   `SELECT VERSION()` returns:
   
   ```text
   5.7.99
   ```
   
   Cluster topology:
   
   ```text
   FE: 1 node, alive, elected master
   BE: 3 nodes, all alive
   Deployment mode: local storage
   Replication: 3 replicas
   ```
   
   The FE and all three BEs report the same `doris-4.1.1-rc01-b10073ad9ca` build string in `SHOW FRONTENDS` and `SHOW BACKENDS`.
   
   The binaries were obtained from the Doris binary distribution rather than compiled locally. I do not currently have a separate source-build commit beyond the build identifier embedded in the version string.
   
   ## 2. Incident timeline
   
   All timestamps below are China Standard Time (`Asia/Shanghai`, UTC+8).
   
   ```text
   2026-07-27 09:31:41  One BE terminated with SIGSEGV.
   2026-07-27 09:31:53  systemd restarted the BE; it rejoined the cluster and is currently alive.
   2026-07-27 09:46:28  A query reading the affected log table failed with:
                        [INTERNAL_ERROR] ZSTD_decompressDCtx error: Data corruption detected
   2026-07-27 10:22:00  Compaction status was checked on all three replicas of two affected tablets.
                        All six replicas were still reporting the same ZSTD decompression error.
   ```
   
   The latest BE crash stack included frames around:
   
   ```text
   doris::vectorized::ColumnDictI32::filter_by_selector
   doris::segment_v2::SegmentIterator
   doris::OlapScanner
   ```
   
   Earlier crashes also included an exception similar to:
   
   ```text
   std::out_of_range: map::at
   ```
   
   The stack above differs from the earlier `ColumnStr<unsigned int>::serialize_impl` / `DistinctStreamingAgg` stack. I would appreciate guidance on whether these should remain in the same issue or be tracked as two related issues.
   
   ## 3. Timestamp clarification
   
   The previously reported value:
   
   ```text
   Aborted at 1748134471
   ```
   
   corresponds to `2025-05-25`, not `2026-05-25`.
   
   Because the original logs are no longer retained, I cannot reliably reconstruct whether the original event occurred on `2025-05-25` or `2026-05-25`. The timestamp/date combination in the original report should therefore be treated as inaccurate and should not be used for source-version matching.
   
   The exact recent incident described in this update occurred on `2026-07-27`, and its timestamps were collected directly from the current environment.
   
   Any timestamps or source line numbers previously presented as examples or reconstructed excerpts should be treated as illustrative. The timestamps in the timeline above and the errors in the following sections were collected from the running environment.
   
   ## 4. Affected table schema
   
   The affected table is a local-storage OLAP table. The table and database names can be anonymized if required; the relevant schema is:
   
   ```sql
   CREATE TABLE `alb_access_log` (
     `log_date` DATE NULL,
     `time` DATETIME NULL,
     `status` VARCHAR(16) NULL,
     `request_method` VARCHAR(16) NULL,
     `host` VARCHAR(256) NULL,
     `client_ip` VARCHAR(64) NULL,
     `request_time` DECIMAL(10,3) NULL,
     `request_uri` VARCHAR(65533) NULL,
     `http_user_agent` VARCHAR(65533) NULL,
     `cqid` VARCHAR(128) NULL,
     `h1` VARCHAR(256) NULL,
     `click_id` VARCHAR(255) NULL,
     `event_type` VARCHAR(32) NULL,
     `traffic` VARCHAR(64) NULL,
     `import_time` DATETIME NULL,
     `source_file` VARCHAR(64) NULL,
     `request_id` VARCHAR(255) NULL,
     `city_region` VARCHAR(128) NULL
   )
   ENGINE=OLAP
   DUPLICATE KEY(`log_date`, `time`)
   PARTITION BY RANGE(`log_date`)
   DISTRIBUTED BY HASH(`client_ip`) BUCKETS 32
   PROPERTIES (
     "replication_allocation" = "tag.location.default: 3",
     "storage_medium" = "hdd",
     "storage_format" = "V2",
     "inverted_index_storage_format" = "V3",
     "light_schema_change" = "true",
     "disable_auto_compaction" = "false",
     "enable_single_replica_compaction" = "false"
   );
   ```
   
   The real table also contains inverted indexes on `log_date`, `request_uri`, `click_id`, `h1`, `source_file`, `traffic`, and `event_type`.
   
   The other table participating in the failing query contains:
   
   ```text
   id              BIGINT NOT NULL
   tk_create_time  DATETIME NOT NULL
   click_id        VARCHAR(675) NULL
   ```
   
   ## 5. Affected partition and tablets
   
   Affected partition:
   
   ```text
   Partition: p20260726
   Range:     [2026-07-26, 2026-07-27)
   State:     NORMAL according to FE metadata
   Buckets:   32
   Replicas:  3
   Rows:      75,370,092 according to partition metadata
   Size:      approximately 46.177 GB
   ```
   
   Two tablets are currently known to contain unreadable pages:
   
   | Tablet ID | Rows per replica | Replica count | FE metadata state | Actual read/compaction result |
   |---|---:|---:|---|---|
   | `1784714655425` | `2,628,463` | 3 | `NORMAL`, `IsBad=false` | ZSTD decompression corruption on all replicas |
   | `1784714655445` | `2,695,687` | 3 | `NORMAL`, `IsBad=false` | ZSTD decompression corruption on all replicas |
   
   The replica versions were aligned at version `140`. FE tablet diagnosis therefore reports healthy metadata, although actual page reads fail.
   
   At approximately `2026-07-27 10:22 CST`, the cumulative compaction status of both tablets on every BE reported:
   
   ```text
   [INTERNAL_ERROR]ZSTD_decompressDCtx error: Data corruption detected
   ```
   
   This produced six matching failures:
   
   ```text
   BE-A / tablet 1784714655425: ZSTD decompression corruption
   BE-B / tablet 1784714655425: ZSTD decompression corruption
   BE-C / tablet 1784714655425: ZSTD decompression corruption
   BE-A / tablet 1784714655445: ZSTD decompression corruption
   BE-B / tablet 1784714655445: ZSTD decompression corruption
   BE-C / tablet 1784714655445: ZSTD decompression corruption
   ```
   
   The most recent successful cumulative compaction on BE-A and BE-B was on `2026-07-26`. BE-C had no successful cumulative or base compaction recorded for these two replicas after its current data copy was created.
   
   Because all three replicas fail actual reads, marking a single replica bad and cloning it from another replica does not provide a safe recovery path.
   
   ## 6. Query that exposes the failure
   
   The production operation is an `INSERT INTO ... WITH ... SELECT` statement. The relevant query structure is shown below. Business-specific output columns and unrelated joins have been omitted, but the scans of the affected table are unchanged:
   
   ```sql
   SET enable_spill = true;
   SET exec_mem_limit = 8589934592;
   SET parallel_pipeline_task_num = 4;
   
   SET @order_start = '2026-07-26 00:00:00';
   SET @order_end   = '2026-07-27 00:00:00';
   
   INSERT INTO target_table (...)
   WITH order_window AS (
     SELECT id, click_id, tk_create_time, ...
     FROM order_detail_table
     WHERE click_id IS NOT NULL
       AND click_id != ''
       AND tk_create_time >= CAST(@order_start AS DATETIME)
       AND tk_create_time <  CAST(@order_end AS DATETIME)
   ),
   click_ids AS (
     SELECT DISTINCT click_id
     FROM order_window
   ),
   click_candidates AS (
     SELECT
       alb.h1 AS click_id,
       alb.time AS click_time,
       alb.cqid,
       alb.traffic,
       alb.request_uri
     FROM alb_access_log alb
     INNER JOIN click_ids ids ON alb.h1 = ids.click_id
     WHERE alb.h1 IS NOT NULL
       AND alb.h1 != ''
   ),
   click_candidates_huawei AS (
     SELECT
       ids.click_id,
       alb.time AS click_time,
       alb.cqid,
       alb.traffic,
       alb.request_uri
     FROM alb_access_log alb
     INNER JOIN huawei_click_ids ids
       ON alb.h1 = ids.log_id
      AND CAST(alb.cqid AS STRING) = ids.cqid
     WHERE alb.traffic = 'HUAWEI'
       AND alb.h1 IS NOT NULL
       AND alb.h1 != ''
       AND alb.cqid IS NOT NULL
   ),
   event_candidates AS (
     SELECT
       alb.click_id,
       alb.time,
       alb.event_type,
       alb.request_id,
       extract_url_parameter(alb.request_uri, 'project_id'),
       extract_url_parameter(alb.request_uri, 'promotion_id')
     FROM alb_access_log alb
     INNER JOIN click_ids ids ON alb.click_id = ids.click_id
     WHERE alb.click_id IS NOT NULL
       AND alb.click_id != ''
       AND alb.time IS NOT NULL
       AND alb.source_file >= 'tracer'
       AND alb.source_file <  'traces'
       AND alb.event_type IN ('3', '16', '17', '18', 'bigIncome')
   )
   SELECT ...;
   ```
   
   The query contains:
   
   - `SELECT DISTINCT click_id`, implemented by the plan as a streaming vectorized aggregate grouped by `click_id`.
   - Multiple `ROW_NUMBER()` window functions.
   - Sorting by `request_uri`, whose type is `VARCHAR(65533)`.
   - URL extraction and URL decoding from `request_uri`.
   - Three independent scans of `alb_access_log`.
   
   The `DISTINCT`/grouping column from the order table is `VARCHAR(675)`. Related join columns in the log table are `h1 VARCHAR(256)` and `click_id VARCHAR(255)`.
   
   ## 7. Relevant EXPLAIN output
   
   The full plan contains 33 plan fragments. The important result is that the order table is correctly pruned to one partition, but every scan of `alb_access_log` reads all partitions because no `log_date` predicate is present.
   
   Order table scan:
   
   ```text
   TABLE: order_detail_table
   PREDICATES:
     tk_create_time >= '2026-07-26 00:00:00'
     tk_create_time <  '2026-07-27 00:00:00'
   partitions=1/364 (p20260726)
   tablets=32/32
   ```
   
   Log table scans:
   
   ```text
   TABLE: alb_access_log
   partitions=83/83
   tablets=2656/2656
   numNodes=3
   ```
   
   This occurs for the normal click lookup, the Huawei click lookup, and the event lookup.
   
   The `SELECT DISTINCT click_id` portion is planned as:
   
   ```text
   VAGGREGATE (update serialize)
   STREAMING
   group by: click_id
   ```
   
   Therefore the failing query can exercise both the segment-scanning path and a downstream streaming aggregation path.
   
   I can attach the complete non-anonymized `EXPLAIN` output separately if required.
   
   ## 8. Session and BE configuration
   
   Query-specific session settings:
   
   ```text
   enable_spill = true
   exec_mem_limit = 8589934592       # 8 GiB
   parallel_pipeline_task_num = 4
   ```
   
   Current session defaults observed independently of the query:
   
   ```text
   skip_bad_tablet = false
   fallback_other_replica_when_fixed_corrupt = false
   ```
   
   `SHOW VARIABLES LIKE 'enable_query_memory_overcommit'` returned no row in this build, so I cannot currently provide a runtime value for that variable.
   
   Relevant configuration from the BE that most recently crashed:
   
   ```text
   deploy_mode = ""                         # local-storage deployment
   mem_limit = 90%
   memory_mode = moderate
   storage_page_cache_limit = 20%
   disable_storage_page_cache = false
   disable_segment_cache = false
   check_segment_when_build_rowset_meta = false
   storage_strict_check_incompatible_old_format = true
   sync_file_on_close = true
   disable_auto_compaction = false
   storage_root_path = <two local storage paths>
   ```
   
   The BE has one original Doris storage path and one additional local disk path added later for capacity expansion. The expansion was performed through Doris `storage_root_path` configuration and Doris-managed balancing. To the best of my knowledge, the affected segment files were not manually edited, overwritten, restored, or copied into the tablet directory outside Doris.
   
   The FE was migrated to another host before this incident, but the BE data directories remained managed by their existing BE processes. No BE tablet directory was manually copied as part of the FE migration.
   
   ## 9. Relationship between the core dumps
   
   I cannot confirm that all previously observed historical core dumps had the same trigger. The original BE logs needed to correlate those dumps have already expired.
   
   The relationship analysis below therefore applies primarily to the recent `2026-07-27` incident, not as a reconstruction of the original historical incident.
   
   Current evidence suggests at least two paths must be investigated separately:
   
   1. Query execution reading corrupted segment pages and forwarding data toward vectorized operators.
   2. Cumulative/base compaction reading the same corrupted pages and repeatedly returning ZSTD errors.
   
   At least one recent SIGSEGV occurred during an OLAP scanner/segment iterator path. The compaction status APIs show repeated corruption failures, but I have not yet proven that every BE core dump was created by a compaction thread.
   
   I will correlate the following fields for each retained core and log interval:
   
   ```text
   crash timestamp
   BE node
   thread ID
   query ID
   fragment instance ID
   tablet ID
   rowset ID
   segment path
   top stack frames
   whether a compaction task was active
   ```
   
   I cannot reliably claim that the historical dumps were identical. If a core from the recent crash is available, I will correlate it with the recent retained logs instead.
   
   ## 10. GDB and core-dump availability
   
   The historical BE logs are no longer available. I am checking whether any matching historical core file is still retained, but I cannot currently promise a complete GDB analysis of the original crash.
   
   For the recent `2026-07-27` crash, I will check whether the new core file and the exact matching BE binary are retained on the crashed node.
   
   If available, I will collect at least:
   
   ```gdb
   set pagination off
   set print pretty on
   thread apply all bt full
   ```
   
   For a core containing the `ColumnStr<unsigned int>::serialize_impl` frame, I will also attempt:
   
   ```gdb
   frame <ColumnStr::serialize_impl frame>
   info args
   info locals
   p row
   p this
   p offsets.size()
   p chars.size()
   p offsets[row]
   p offsets[row - 1]
   ```
   
   For the serialized-key caller, if symbols permit:
   
   ```gdb
   p keys[row]
   p keys[row].data
   p keys[row].size
   ```
   
   If the distributed binary is stripped or optimized and GDB cannot evaluate these expressions, I will include the exact GDB error messages.
   
   No string contents or raw user data will be included. Pointer values, sizes, offsets, symbols, and stack frames can be provided.
   
   ## 11. Complete BE log interval
   
   The log interval surrounding the original historical crash is no longer available because it has expired under the retention policy.
   
   I am checking retention for the exact log interval around the recent `2026-07-27 09:31:41 CST` crash.
   
   The requested export will cover approximately:
   
   ```text
   2026-07-27 09:29:30 CST
   through
   2026-07-27 09:33:30 CST
   ```
   
   It will preserve, when present:
   
   ```text
   thread IDs
   query IDs
   fragment instance IDs
   tablet IDs
   rowset IDs
   segment paths
   the first checksum/decompression error
   scanner and iterator errors
   query cancellation messages
   compaction messages
   the final crash stack
   ```
   
   Business values inside URLs, request payloads, hostnames, and user identifiers will be redacted without changing technical identifiers such as query IDs, tablet IDs, rowset IDs, or stack frames.
   
   ## 12. Corrupted segment sample and metadata
   
   The affected replicas and storage paths are still online, so it should be possible to collect:
   
   ```text
   one corrupted segment file
   tablet metadata
   rowset metadata
   tablet schema
   segment and rowset identifiers
   file size
   SHA-256 checksum
   ```
   
   However, all three replicas of the two known tablets currently fail decompression. I do not currently have a confirmed healthy copy of the same segment for comparison.
   
   The segment may contain request URLs and other business data. I will not attach it to the public GitHub issue. Please provide a private upload channel if the raw segment or core dump would be useful.
   
   If preferred, I can first provide only:
   
   - File size and SHA-256 for each replica.
   - Tablet/rowset/segment metadata.
   - A byte-level comparison between replicas.
   - A reduced or intentionally corrupted synthetic sample that reproduces the same error without business data.
   
   ## 13. Current conclusions
   
   The current evidence supports the following limited conclusions:
   
   1. The FE metadata alone does not detect this corruption because replica state, replica count, and versions are normal.
   2. Actual page reads and compaction consistently detect ZSTD data corruption.
   3. The same two tablets fail on all three replicas, so this is not currently recoverable by cloning from another in-cluster replica.
   4. A production query scans all log-table partitions and reliably reaches the affected partition.
   5. Doris should return a safe query or compaction error when the page cannot be decompressed; the BE should not terminate with SIGSEGV while handling that error.
   6. The different observed top stack frames may indicate multiple unsafe error-propagation paths, so query execution and compaction should be investigated separately.
   
   ## 14. Requested next steps
   
   Could you please confirm:
   
   1. Whether the current `ColumnDictI32::filter_by_selector` / `SegmentIterator` crash should remain in this issue together with the earlier `ColumnStr::serialize_impl` / `DistinctStreamingAgg` crash.
   2. Whether there is a private upload channel for one representative core dump, a complete BE log interval, and one corrupted segment sample.
   3. Whether the build identifier `doris-4.1.1-rc01-b10073ad9ca` is sufficient to map the distributed binary to the exact source revision, or whether another command/file should be used to extract the build commit.
   
   I apologize that the original historical log interval is no longer available. I hope the recent incident, for which the affected tablets are still online and the corruption remains reproducible, can provide a more useful investigation target.
   
   I can provide the full `SHOW FRONTENDS`, `SHOW BACKENDS`, `SHOW CREATE TABLE`, `EXPLAIN`, BE configuration, and GDB output as separate sanitized attachments.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org


Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]

Posted by "YanzhiJin5 (via GitHub)" <gi...@apache.org>.
YanzhiJin5 commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5379565592

   Hi @ShummGen,
   
   First, I apologize for the long gap since your last update, and thank you again for providing such detailed information from the July 27 incident. I have spent some time going back through the issue and tracing the relevant code on the exact production revision you provided (`b10073ad9ca17cd5685c4dd3b3ef650f256376d0`) before asking you for any additional data.
   
   A brief update on the investigation so far:
   
   * I traced the storage-read error path from `PageIO` through `ColumnReader`, `FileColumnIterator`, `SegmentIterator`, and `OlapScanner`.
   * On the normal error path I reviewed, a block from a storage read that returns an error is not forwarded downstream to aggregation. Therefore, I do not currently have evidence that the nearby checksum/ZSTD errors directly produced the block involved in the earlier `DistinctStreamingAgg -> ColumnStr::serialize_impl` crash.
   * For that reason, I am treating the earlier `ColumnStr` crash and the more recent `ColumnDictI32::filter_by_selector -> SegmentIterator -> OlapScanner` crash as separate signatures unless further evidence connects them.
   * I also investigated a missing dictionary-code bounds check and prototyped a defensive validation change with regression tests. However, after revisiting the evidence, I do **not** think the available production data is sufficient to claim that invalid dictionary codes caused your crash. I would rather confirm or rule that out from the actual crash state than rely on a speculative fix.
   
   The good news is that the source investigation has now narrowed the recent `ColumnDictI32::filter_by_selector` crash to a much more specific question: at the failing iteration, were the selector and dictionary code actually within their expected ranges?
   
   If the core from the recent July 27 crash and its matching `doris_be` binary are still retained, please use whichever option is easiest for you.
   
   ### Option 1: minimal GDB output
   
   If opening the core in GDB is convenient, a small amount of output would be enough for the next step. After opening the matching binary and core, the following commands should be sufficient initially:
   
   ```gdb
   set pagination off
   bt
   ```
   
   If the backtrace contains a `ColumnDictI32::filter_by_selector` frame, select its frame number as shown by `bt`, for example:
   
   ```gdb
   frame <N>
   info args
   info locals
   ```
   
   There is no need to spend time manually interpreting the values. Please just send the output that GDB is able to provide. Optimized production binaries may report some variables as `<optimized out>`; that is completely fine, and there is no need to work around it.
   
   Please also feel free to redact any business data, paths, SQL text, URLs, or other sensitive information before posting the output publicly.
   
   ### Option 2: existing core file
   
   If running GDB and extracting variables yourself would be more work, I would be equally happy to inspect the existing core and matching `doris_be` binary directly.
   
   Since you previously mentioned that the raw core/segment may contain sensitive business data, I do **not** suggest uploading them to the public GitHub issue. If a Doris maintainer can recommend an appropriate private transfer channel, this may actually be the lower-effort option for you: you could provide the existing core and matching binary (and separate debug symbols, if any), and I can do the GDB investigation from there.
   
   If neither option is convenient or the recent core is no longer retained, that is also completely fine. I can continue from the source side and try to construct a local reproducer using the information you have already provided.
   
   Thank you again for preserving and sharing the diagnostic information. It has been very helpful in narrowing the investigation.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org