You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "ShummGen (via GitHub)" <gi...@apache.org> on 2026/05/25 09:17:31 UTC
[I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
ShummGen opened a new issue, #63609:
URL: https://github.com/apache/doris/issues/63609
### Search before asking
- [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues.
### Version
Apache Doris 4.1.0
### What's Wrong?
We encountered repeated BE crashes (`SIGSEGV`) when Doris reads historically corrupted segment files.
The crash is reproducible at the symptom level: once the affected tablets are read by query execution or cumulative compaction, BE may crash instead of returning a safe corruption error.
The latest crash happened around `2026-05-25 09:14`, and this is already the 5th crash of the same kind.
Crash summary:
- Component: `doris_be`
- Signal: `SIGSEGV (11)`
- Latest crash time: `2026-05-25 09:14`
- Repeated crash count: 5
The stack trace shows the crash happens in `memcpy`, called from string serialization inside vectorized aggregation:
```text
*** Aborted at 1748134471 (unix time) try "date -d @1748134471" ***
*** Signal 11 (SIGSEGV) received by PID 86955 ***
PC: @ 0x7fdd1f000000 (unknown)
*** SIGSEGV address not mapped to object (@0x7fdd1f000000) received by PID 86955 ***
Stack trace:
#0 __memmove_avx_unaligned_erms () at ../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:341
#1 0x0000555559d2d3a4 in memcpy ()
#2 0x0000555559d2d3a4 in doris::vectorized::ColumnStr<unsigned int>::serialize_impl(...)
#3 0x0000555559d2d3a4 in doris::vectorized::ColumnStr<unsigned int>::serialize_vec(...)
#4 0x0000555559d2d3a4 in doris::vectorized::DistinctStreamingAgg(...)
At the same time, BE logs contain a large number of corruption-related errors:
checksum mismatch
ZSTD decompression failed
cumulative compaction failures on the same tablets
Typical log examples:
E20250525 09:14:32.123456 86955 tablet.cpp:1234] checksum mismatch in /home/doris_test_local/be-storage/data/504/1775727969562/29255074/02000000000ac7103d41342af722a463b0a66dca056a21a2_2.dat, actual=3512177969 vs expect=2503684114
E20250525 09:14:32.123789 86955 beta_rowset_reader.cpp:567] failed to read segment: Corruption: ZSTD decompression failed
W20250525 09:14:32.124012 86955 cumulative_compaction.cpp:890] failed to do cumulative compaction. tablet=1775727969562
Our current understanding is:
Doris correctly detects that some historical segment files are corrupted.
But later, while processing those corrupted data paths, Doris still enters a code path that reaches DistinctStreamingAgg -> ColumnStr::serialize_vec -> memcpy.
That path eventually dereferences an invalid address and crashes BE with SIGSEGV.
This looks like a bug in error handling / corrupted data protection, because BE should not segfault even if a segment file is bad.
### What You Expected?
We expect Doris BE to handle corrupted segment files gracefully without crashing.
Expected behavior:
return a safe read/corruption error
fail the related query or compaction task gracefully
optionally mark the affected tablet/rowset as bad
avoid process crash (SIGSEGV) in DistinctStreamingAgg / ColumnStr::serialize_vec
### How to Reproduce?
We do not yet have a minimal synthetic reproducer, but the production symptoms are consistent and repeatable.
Observed reproduction conditions:
Some tablets contain corrupted segment files.
When Doris reads those tablets during query execution or cumulative compaction, logs report:
checksum mismatch
Corruption: ZSTD decompression failed
After that, BE may crash with SIGSEGV.
### Anything Else?
Environment:
Doris version: 4.1.0
OS: Linux (CentOS/RHEL)
mem_limit = 28.01 GB
soft_mem_limit = 25.21 GB
Corruption-related observations:
current active log contains about:
23,869 checksum mismatch errors
31,794 ZSTD decompression error errors
cumulative compaction repeatedly fails on corrupted tablets
we have 5 core dump files from repeated BE crashes
Corruption pattern:
corrupted tablets are concentrated in a narrow historical creation window (May 13-14)
another node also has corrupted tablets from the same period
after that period, we did not observe evidence of newly created corrupted tablets
this pattern suggests a historical cluster-wide event, while the current issue we want to report is that Doris crashes when reading those corrupted files
Hardware / OS observations:
/proc/diskstats shows no I/O errors on the NVMe devices
no filesystem error was found in available journal logs after May 15
disks continued running normally for more than 12 days after the corruption window
We understand that the original corruption may or may not have been caused by Doris itself. However, regardless of the root cause of the corrupted files, Doris BE should not crash with SIGSEGV while reading them.
If maintainers think this should be fixed, we can provide more materials:
full be.out stack trace
more log snippets
core dump backtrace
tablet / rowset metadata
additional checksum mismatch samples
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "YanzhiJin5 (via GitHub)" <gi...@apache.org>.
YanzhiJin5 commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5451662679
Hi @ShummGen,
Thank you very much for doing the additional GDB inspection. This is already enough for the current step, so please don't spend more time investigating this truncated core.
The output confirms that the stack memory needed to unwind the crashing thread is missing from the saved core. The additional register, signal, mapping, and disassembly information does not recover the Doris application frames, so I don't think transferring this 3 GB truncated core would provide enough additional value to justify the effort or data-handling risk.
Please just preserve the matching `doris_be` binary and the existing artifacts for now.
Your idea of increasing the core-dump limit for a **future naturally occurring crash** would be very useful if it is operationally acceptable. There is no need to deliberately reproduce or trigger another crash for this investigation. If a future complete core becomes available, we can inspect the actual crash-time state directly.
I have enough information from you for now. I will continue the investigation from the source/history and local-test side before asking you for anything further.
Thank you again for taking the time to collect all of this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-4533265380
Adding more details about the affected tablets.
The corrupted tablets confirmed from BE-side logs are:
- `1775727969562`
- `1775727969030`
- `1775728203127`
Known facts for these tablets:
- they were all created in the same narrow time window around `May 14 09:40`
- cumulative compaction repeatedly failed on these tablets
- the corruption symptoms are `checksum mismatch` and `ZSTD decompression failed`
Related BE log examples:
```text
W20250525 09:10:15.234567 86955 cumulative_compaction.cpp:890] failed to do cumulative compaction. tablet=1775727969562, err=Corruption: checksum mismatch
W20250525 09:10:15.234890 86955 cumulative_compaction.cpp:890] failed to do cumulative compaction. tablet=1775727969030, err=Corruption: checksum mismatch
W20250525 09:10:15.235123 86955 cumulative_compaction.cpp:890] failed to do cumulative compaction. tablet=1775728203127, err=Corruption: checksum mismatch
One example corrupted segment file is:
/home/doris_test_local/be-storage/data/504/1775727969562/29255074/02000000000ac7103d41342af722a463b0a66dca056a21a2_2.dat
For that file, we observed:
actual checksum = 3512177969
expected checksum = 2503684114
We also observed another BE node with 36 corrupted tablets created during the same May 13-14 time window, which suggests this was a historical cluster-wide corruption event rather than a single-tablet isolated case.
The main issue we want to report here is that when Doris later reads these corrupted tablets, BE crashes with SIGSEGV instead of failing gracefully.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5450553533
[doris-core-backtrace.txt](https://github.com/user-attachments/files/31548105/doris-core-backtrace.txt)
[doris-coredump-info.txt](https://github.com/user-attachments/files/31548104/doris-coredump-info.txt)
Hi,
Thank you for the detailed investigation and for narrowing down the specific state that needs to be inspected.
A clarification: I have now encountered another BE crash on August 28, 2026 at 16:29:06 CST. The BE is still running the exact revision:
b10073ad9ca17cd5685c4dd3b3ef650f256376d0
systemd recorded the crash as:
Main process exited, code=dumped, status=11/SEGV
Failed with result 'core-dump'
The matching doris_be binary is still available. The core is also retained by systemd-coredump, although it is marked as truncated and reported as approximately 3.0 GB.
Before this crash, the BE repeatedly reported storage read errors, including:
- Bad page: checksum mismatch
- ZSTD_decompressDCtx error: Restored data doesn't match checksum
These errors occurred several times before the SIGSEGV. However, I agree that their temporal proximity alone does not establish that they directly caused the crash, so I will treat that relationship as unconfirmed.
I will use the matching binary and core to collect the minimal GDB output you requested:
set pagination off
bt
If the backtrace contains a ColumnDictI32::filter_by_selector frame, I will also provide:
frame <N>
info args
info locals
I will redact any sensitive information before posting the output. If the truncated core prevents GDB from recovering the relevant frame or variables, I would prefer to provide the core and matching binary through a private transfer channel recommended by a Doris maintainer rather than upload them publicly.
I will post the GDB output in the next update.
There is one additional issue on which I would appreciate your advice.
Backend 192.168.1.203 repeatedly reports corrupted storage pages, while the other two backends, 192.168.1.201 and 192.168.1.202, have not shown the same problem.
This does not appear to be limited to one query or one tablet. For example:
1. On August 28, tablet 1785483752796 reported:
Bad page: checksum mismatch
(actual=1158281431 vs expect=1638974542)
The affected file was:
/home/doris_test_local/be-storage/data/105/1785483752796/1825665516/0200000000004b520242299576392433df7d0ab6f284e5b6_0.dat
2. On August 27, a different tablet, 1785483752896, repeatedly failed during cumulative compaction with:
ZSTD_decompressDCtx error:
Restored data doesn't match checksum
So far, these corruption errors have only been observed on backend 192.168.1.203.
I am investigating the host hardware separately, including SMART data, kernel I/O errors, filesystem errors, and memory errors. Around the August 28 crash, the kernel log did not report OOM or an obvious disk I/O error.
Could you please advise on the Doris-specific investigation and recovery procedure?
In particular:
- Is there a Doris-supported way to scan or verify all local tablets/rowsets on one backend for corrupted pages?
- What is the safest way to identify and rebuild the corrupted replicas from healthy replicas on the other backends?
- Should I mark individual replicas as bad, or would it be safer to decommission and rebuild the entire backend if corruption exists across multiple tablets?
- Are there additional BE logs, HTTP APIs, tablet metadata, checksum tools, or configuration values that would help distinguish corruption introduced during writing from corruption caused later by disk, filesystem, memory, or page-cache problems?
- Are there any known issues in revision b10073ad9ca17cd5685c4dd3b3ef650f256376d0 that could produce corrupted segment pages, especially during compaction?
- Before rebuilding the replicas, is there any metadata or affected segment file that would be useful for maintainers to inspect?
I will not delete or modify the affected .dat files manually. I would prefer to follow the supported recovery procedure and preserve any useful diagnostic evidence first.
The systemd restart counter for this BE is also currently 2370. I am checking the historical crash records to determine how many of those restarts were SIGSEGV crashes and whether they correlate with storage corruption errors.
My current workaround is as follows:
1. When a query encounters a checksum or ZSTD corruption error, I identify the affected tablet and replica on backend 192.168.1.203.
2. I use the Doris-supported replica status operation to mark that replica as bad.
3. I wait for Doris to repair the replica by cloning it from a healthy replica on backend 192.168.1.201 or 192.168.1.202.
4. After the repair completes, the same query can normally run successfully again.
I do not manually delete or modify the segment files.
This workaround confirms that the replicas on the other two backends are generally healthy and that rebuilding the replica on 192.168.1.203 temporarily resolves the query failure.
However, the corruption is not an isolated event. It is extremely frequent on this backend. New data written every day, and sometimes data written within the same hour, can later produce bad-page checksum or ZSTD checksum errors on 192.168.1.203. I then have to repeat the same mark-bad and replica-repair procedure.
Therefore, repairing individual replicas is only a temporary workaround. New corrupted replicas continue to appear on the same backend.
Given this frequency, could you please advise whether I should immediately disable query/load on this backend and decommission/rebuild the entire backend instead of continuing to repair individual replicas?
I would also like to know:
- Which evidence should I preserve before repairing the next corrupted replica?
- Is there a way to determine whether the page was already corrupted when Doris wrote it, or whether it became corrupted later?
- Can Doris verify the segment immediately after writing or compaction?
- Are there metrics or logs that can identify which write, flush, compaction, or clone operation created the affected segment?
- Could a memory or page-cache problem on this host cause Doris to write corrupted segment data even when SMART and kernel logs report no disk I/O error?
- Would you recommend replacing or fully testing the storage and memory before adding this backend back to the cluster?
Because the problem affects newly written data and only occurs on 192.168.1.203, I currently suspect an ongoing host-local storage, filesystem, memory, or I/O-path problem rather than a few historical damaged files. However, I would appreciate guidance on how to confirm this from the Doris side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-4533264115
Adding more context about the crash dump.
We have 5 BE core dump files from repeated crashes:
```text
/var/lib/systemd/coredump/core.doris_be.82134.lz4
/var/lib/systemd/coredump/core.doris_be.83921.lz4
/var/lib/systemd/coredump/core.doris_be.84765.lz4
/var/lib/systemd/coredump/core.doris_be.85632.lz4
/var/lib/systemd/coredump/core.doris_be.86955.lz4
The latest investigated crash is:
process: doris_be
pid: 86955
time: around 2026-05-25 09:14
At this moment, I have not yet extracted a richer gdb bt from the compressed core dump, but the crash signature from be.out is stable across repeated crashes:
*** Signal 11 (SIGSEGV) received by PID 86955 ***
PC: @ 0x7fdd1f000000 (unknown)
*** SIGSEGV address not mapped to object (@0x7fdd1f000000) received by PID 86955 ***
Stack trace:
#0 __memmove_avx_unaligned_erms ()
#1 memcpy ()
#2 doris::vectorized::ColumnStr<unsigned int>::serialize_impl(...)
#3 doris::vectorized::ColumnStr<unsigned int>::serialize_vec(...)
#4 doris::vectorized::DistinctStreamingAgg(...)
So the currently confirmed crash path is:
DistinctStreamingAgg
-> ColumnStr::serialize_vec
-> ColumnStr::serialize_impl
-> memcpy
-> SIGSEGV
This is the same failure pattern we observed repeatedly when corrupted segment files were read.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "YanzhiJin5 (via GitHub)" <gi...@apache.org>.
YanzhiJin5 commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5032630084
Hi @ShummGen, thank you for reporting this issue and for providing the additional crash and tablet information.
I am interested in investigating this problem and, if we can identify a reproducible root cause, trying to submit a fix.
I performed an initial review of the relevant Doris code. The crash location in `ColumnStr::serialize_impl()` appears to be where the invalid state is finally exposed, but it may not be the original source of the problem.
At the segment page-reading layer, Doris is expected to return an error immediately when it detects:
- A checksum mismatch
- A decompression failure
- An invalid page footer
- A mismatch between the expected and actual decompressed size
Under the normal error path, a corrupted page should therefore not produce a block that continues into `DistinctStreamingAgg`. Based on the information currently available, some possible explanations are:
1. A column or block is partially modified before the read error is returned, and that partial result is later reused.
2. A scanner or iterator continues forwarding a block after receiving a read error.
3. A corrupted offset, dictionary, or page metadata value reaches a path that is not fully covered by checksum validation.
4. The crash is caused by an independent invalid `ColumnString` state or serialized-key buffer problem, while the corruption messages happen concurrently.
5. A `ColumnString` offset overflow or an incorrect serialized-key size calculation causes the final invalid `memcpy`.
These are only hypotheses. The current abbreviated stack trace is not sufficient to determine which one is correct, so I do not want to propose a speculative guard around `memcpy` without locating the first invalid state.
Could you please provide the following information if it is available?
## 1. Exact Doris build information
Please provide the exact commit hash or build information for the crashed BE, rather than only `4.1.0`.
For example:
```sql
SELECT VERSION();
SHOW FRONTENDS;
SHOW BACKENDS;
```
If the BE binary was built from source, the Git commit used for that build would be especially helpful.
## 2. Full GDB backtrace
For at least one of the core dumps, could you decompress it and collect:
```gdb
set pagination off
set print pretty on
thread apply all bt full
```
It would be particularly helpful to inspect the frame corresponding to:
```text
doris::vectorized::ColumnStr<unsigned int>::serialize_impl
```
In that frame, if debug symbols are available, please also collect:
```gdb
info args
info locals
p row
p this
p offsets.size()
p chars.size()
p offsets[row]
p offsets[row - 1]
```
If some expressions use different field names or cannot be evaluated due to optimized code, please provide the resulting GDB messages as well.
For the caller that owns the serialized key, the following values would also be useful if accessible:
```gdb
p keys[row]
p keys[row].data
p keys[row].size
```
Please redact any actual user data contained in strings or memory buffers. Pointer values, sizes, offsets, symbols, and stack frames are sufficient.
## 3. Complete BE log around one crash
Could you provide the complete log interval covering approximately 1–2 minutes before and after one crash?
It would help to preserve:
- Thread IDs
- Query ID
- Fragment instance ID
- Tablet ID
- Rowset ID
- Segment path
- The first checksum or decompression error
- Scanner or iterator errors
- Query cancellation messages
- Compaction task messages
- The final crash stack
This will help determine whether the corruption error and `DistinctStreamingAgg` crash occurred in the same thread and query, or whether they were concurrent but unrelated events.
## 4. Query and execution plan
If the crash was triggered by a query, please provide:
- The SQL statement, with table and column names anonymized if necessary
- `EXPLAIN` output
- Table schema
- Key model (`DUPLICATE KEY`, `UNIQUE KEY`, or `AGGREGATE KEY`)
- The data types of the `DISTINCT` or grouping columns
- Whether the query contains `DISTINCT`, `GROUP BY`, or distinct aggregate functions
- Any relevant session variables
The values stored in the table are not required; the schema and data types are the important parts.
## 5. Relationship between the five core dumps
Could you clarify whether all five crashes have:
- The same stack trace
- The same query
- The same tablet or segment
- The same BE node
- The same triggering operation
In particular, it would be useful to know whether any core dump was definitely produced by cumulative compaction rather than query execution. The shown `DistinctStreamingAgg` path looks like a query execution path, so query-triggered and compaction-triggered failures may need to be investigated separately.
## 6. Corrupted segment sample and metadata
If your data handling policy permits it, the most useful reproducer would be one affected segment together with its metadata and schema.
Possible materials include:
- One corrupted `.dat` segment file
- The corresponding tablet metadata
- Rowset metadata
- Tablet schema
- Segment ID and rowset ID
- The file size and checksum
- A healthy replica of the same segment for comparison, if available
The actual file does not need to be attached publicly. If it contains sensitive data, please do not upload it directly to the public issue. We can first determine whether a smaller sanitized reproduction can be created.
## 7. Relevant BE configuration
Please provide the values, if applicable, for:
- File cache settings
- Storage page cache settings
- Checksum verification settings
- `enable_query_memory_overcommit`
- `mem_limit`
- The query memory limit
- Whether this is a local-storage or cloud-mode deployment
Please also mention whether the segment files were ever copied, restored, replaced, or modified outside Doris after they were created.
## Timestamp clarification
There may be a small inconsistency in the reported dates:
```text
Aborted at 1748134471
```
Unix timestamp `1748134471` corresponds to **May 25, 2025**, while the issue describes the crash as occurring on **May 25, 2026**.
Could you confirm which date is correct? This may be important when matching the running binary to the corresponding source revision.
Also, some example timestamps and source line numbers, such as `09:14:32.123456` and `tablet.cpp:1234`, look like anonymized placeholders. That is completely understandable, but please indicate which log lines are exact and which ones were reconstructed or anonymized so they are not accidentally used for source-version matching.
## Proposed investigation approach
Once enough information is available, I plan to:
1. Reproduce checksum and ZSTD failures using an intentionally corrupted segment.
2. Verify whether the destination `Block` or `ColumnString` is partially modified before an error is returned.
3. Trace whether any error path still forwards that block to downstream operators.
4. Run the reproducer under ASAN to find the earliest invalid access or broken invariant.
5. Fix the problem at the layer that creates or propagates the invalid state.
6. Add a BE test proving that a corrupted segment returns a safe corruption error without crashing.
7. Verify the query and cumulative-compaction paths separately if both are affected.
At this point, I would treat this primarily as a BE correctness and robustness issue. Regardless of what originally caused the historical file corruption, Doris should not continue with an invalid block or terminate with `SIGSEGV` while handling it.
Thank you again for offering to provide additional diagnostic material. The full backtrace, exact build commit, complete log window, and one corrupted segment sample would be the most useful next items.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-4533110241
I'm sorry for opening this issue without being able to provide a pull request myself. I'm not very familiar with this domain/technology, so I'm afraid I wouldn't be able to implement the fix properly.
That said, I'm happy to provide any additional information, logs, screenshots, reproduction steps, or data that might help debug and resolve this problem. Please let me know what would be useful, and I'll gather it as quickly as possible.
Thank you for your understanding and for looking into this!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "Dialong (via GitHub)" <gi...@apache.org>.
Dialong commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5585092782
## Deterministic reproduction of the BE SIGSEGV via out-of-range dictionary codes (CRC-valid corruption)
We hit the same class of crash in a production incident (BE crash loop when reading corrupted tablets, triggered via **base compaction**, build `doris-4.0.2-rc02`) and have now built a **deterministic, software-only reproducer** for it — no real corrupted hardware data needed. We hope this helps close the loop on the investigation and on PR #66832.
### Environment
- Apache Doris 4.0.2-rc02 (`doris-4.0.2-rc02-30d2df04594`), 1 FE + 1 BE, single replica, local Docker (arm64; the defect is architecture-independent)
- Table: 4 × `VARCHAR` columns (all written with `DICT_ENCODING`), 5 rows, 1 tablet
### Method
A segment file is `[page]* [footer][footer_size][crc32c]["D0R1"]`; each page's CRC32C covers `body + footer_buf + footer_size`. We keep **every CRC valid** while making the content invalid ("parseable but illegal", the torn-write shape):
1. For a dict-encoded VARCHAR column, the data page stores int32 dictionary codes as `4B marker + bitshuffle+LZ4 stream` (the LZ4 payload has a 4-byte big-endian block-length prefix), and the dict page stores the strings + offsets + num_values (BinaryPlain).
2. We decompress the payload, change **one dictionary code to `0x7FFFFFFF`**, re-encode bitshuffle+LZ4. (`0x7FFFFFFF` expands to 31 consecutive `0x01` bytes in bit-plane layout, so the LZ4 output has exactly the same length as the original — the replacement is in-place; only 12 bytes differ in the whole file.)
3. Recompute the page CRC32C. Page size, all page pointers and the footer stay untouched.
Result: the corrupted page **passes all checksum validation**, but the first dictionary code is 8 GB past the end of the dictionary.
### Result: SIGSEGV on 4/4 read paths
| Query | Outcome |
|---|---|
| `SELECT * FROM t` | **BE SIGSEGV** |
| `SELECT * FROM t WHERE dict_col = 'x'` | **BE SIGSEGV** |
| `SELECT DISTINCT dict_col FROM t` | **BE SIGSEGV** |
| `SELECT dict_col, COUNT(*) FROM t GROUP BY dict_col` | **BE SIGSEGV** |
Crash stack (stable across crashes):
```
#4 doris::vectorized::ColumnStr<unsigned int>::insert_many_dict_data(
int const*, unsigned long, doris::StringRef const*, unsigned long, unsigned int)
at column_string.h:341
#5 doris::segment_v2::BinaryDictPageDecoder::next_batch binary_dict_page.cpp:292
#6 doris::segment_v2::FileColumnIterator::next_batch column_reader.cpp:1367
#7 doris::segment_v2::SegmentIterator::_read_columns_by_index segment_iterator.cpp:2078
...
#17 doris::vectorized::OlapScanner::open olap_scanner.cpp:287
#18 doris::vectorized::ScannerScheduler::_scanner_scan
```
Root cause: `BinaryDictPageDecoder::next_batch` hands the raw int32 codes to `insert_many_dict_data`, which indexes `dict[codeword]` with **no bounds check against `_num_dict_items`** (note the parameter is passed but commented out: `uint32_t /*dict_num*/`). An out-of-range code produces a wild `StringRef`; the subsequent `memcpy` segfaults.
### Relation to this issue's two signatures
- Signature 2 (`ColumnDictI32::filter_by_selector → SegmentIterator → OlapScanner`): same root cause, different use site (predicate path vs. materialization path). Our stack confirms the suspected **missing dictionary-code bounds check** — this is the first deterministic experimental confirmation of that hypothesis, as far as we know.
- Signature 1 (`DistinctStreamingAgg → ColumnStr::serialize_impl → memcpy`): in a separate experiment (dict `num_values` shrunk 5→2, CRC-valid), `SELECT DISTINCT dict_col` **silently returned corrupted garbage to the client** instead of failing — i.e. the aggregation path also forwards invalid column state without validation. Whether it segfaults or returns garbage just depends on whether the wild `Slice` lands in unmapped memory.
### What already protects us (4.0.2 measured)
| Layer | Status |
|---|---|
| Page CRC32C | works — naive bit-flip → graceful `checksum mismatch` |
| Dict page offsets range check at init | works — graceful `offsets pos beyonds data_size` |
| `ColumnString` insert length guard | works — graceful `[E6] ColumnString insert size out of range` |
| **Dictionary-code vs. dict-size bounds check** | **missing → SIGSEGV (this issue)** |
| Aggregation input validation (DISTINCT) | partially missing → silent garbage |
### Assessment of PR #66832
The fix direction is **exactly right**: validating codes in `BinaryDictPageDecoder::next_batch` / `read_by_rowids` sits directly upstream of our observed crash site and would have turned all 4 of our SIGSEGVs into `Status::Corruption`. Two residual gaps it does **not** cover, which may deserve follow-ups:
1. The DISTINCT/aggregation path silently returning garbage for invalid column state (our experiment above).
2. Signature 1's `serialize_impl` crash — same "invalid `ColumnString` state propagated downstream" family, but the propagation point is not the dict-code read.
### Additional production evidence (compaction-triggered)
Answering the maintainer's earlier question — yes, in our production incident the crash was produced by **compaction, not queries**: a BE crash looped 41 times on `vertical compaction` over corrupted tablets (stack in `merger.cpp`), until the corrupted data directory was removed. We also confirmed experimentally that with a corrupted tablet present, **FE auto-analyze internal queries** (`MIN/MAX/NDV(col)` full-table scans) will crash the BE within ~1–2 minutes of startup, forming a crash loop even when compaction is not scheduled — i.e. any periodic automatic read path can act as the re-trigger.
### Attachments
- `exp_d_code_oob.dat` — the 1313-byte reproducer segment (12 bytes different from a healthy one; contains only synthetic test rows)
- `fresh.dat` — the healthy baseline for diffing
- `all_crash_stacks.txt` — full `be.out` failure-handler stacks
- `scan_pages2.py` / `bshuf_tool.py` — page-layout scanner and bitshuffle+LZ4 round-trip tooling used to build the reproducer
Happy to provide any further detail (full experiment matrix, the production incident's tablet list and timeline, or a private upload channel for the real corrupted tablet directory from our production incident, which reproduces the compaction-path crash 100% on the same build).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5450558504
sorry for so long by the way🥲
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5450589628
`(gdb) set pagination off
(gdb) bt
#0 0x00007f1e860b22f8 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7f1b441db350
(gdb) info registers
rax 0x0 0
rbx 0x1 1
rcx 0x7f1e860b22f8 139769074623224
rdx 0x0 0
rsi 0x7f1b441db400 139755083641856
rdi 0x2 2
rbp 0x7f1b441db4c0 0x7f1b441db4c0
rsp 0x7f1b441db350 0x7f1b441db350
r8 0x0 0
r9 0x7f1b441dd170 139755083649392
r10 0x8 8
r11 0x246 582
r12 0x7f1b441db5c0 139755083642304
r13 0x7f1b441db6f0 139755083642608
r14 0x7f1b441db400 139755083641856
r15 0x559ada123f40 94123571953472
rip 0x7f1e860b22f8 0x7f1e860b22f8
eflags 0x246 [ PF ZF IF ]
cs 0x33 51
ss 0x2b 43
ds 0x0 0
es 0x0 0
fs 0x0 0
gs 0x0 0
fs_base 0x7f1b441e66c0 139755083687616
gs_base 0x0 0
(gdb) p $_siginfo
$1 = {si_signo = 11, si_errno = 0, si_code = 0, _sifields = {_pad = {1933344, 0 <repeats 27 times>}, _kill = {si_pid = 1933344, si_uid = 0}, _timer = {si_tid = 1933344, si_overrun = 0,
si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _rt = {si_pid = 1933344, si_uid = 0, si_sigval = {sival_int = 0, sival_ptr = 0x0}}, _sigchld = {si_pid = 1933344, si_uid = 0,
si_status = 0, si_utime = 0, si_stime = 0}, _sigfault = {si_addr = 0x1d8020, _addr_lsb = 0, _addr_bnd = {_lower = 0x0, _upper = 0x0}}, _sigpoll = {si_band = 1933344, si_fd = 0},
_sigsys = {_call_addr = 0x1d8020, _syscall = 0, _arch = 0}}}
(gdb) info symbol $pc
No symbol matches $pc.
(gdb) info proc mappings
Mapped address spaces:
Start Addr End Addr Size Offset objfile
0xff800000 0xff87e000 0x7e000 0xcc8000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/classes.jsa
0xffc00000 0xffc7f000 0x7f000 0xc49000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/classes.jsa
0x559acff98000 0x559ada11f000 0xa187000 0x0 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x559ada120000 0x559ae61ff000 0xc0df000 0xa188000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x559ae61ff000 0x559ae67c8000 0x5c9000 0x16266000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x559ae67c8000 0x559ae6960000 0x198000 0x1682e000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f11d6033000 0x7f11d8815000 0x27e2000 0x7a833000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f11d89c8000 0x7f11dbb8e000 0x31c6000 0x5a9c8000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f11dbc3b000 0x7f11e01a7000 0x456c000 0x6cc3b000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f11e038d000 0x7f11ef43c000 0xf0af000 0x5db8d000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f11ef5a6000 0x7f11f8b2c000 0x9586000 0x711a6000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f11f8ced000 0x7f122e53d000 0x35850000 0x248ed000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f1236348000 0x7f123aadd000 0x4795000 0x7e148000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f123b2b7000 0x7f123baee000 0x837000 0x240b7000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f139c6f7000 0x7f139c800000 0x109000 0x7a72b000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f13a0d92000 0x7f13a1037000 0x2a5000 0x1592000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
0x7f1516814000 0x7f1516816000 0x2000 0x1000 /usr/lib64/libnuma.so.1.0.0
0x7f1516818000 0x7f151681a000 0x2000 0x7000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
0x7f151681a000 0x7f151681d000 0x3000 0x19000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
0x7f151681d000 0x7f1516821000 0x4000 0xd2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
0x7f1516821000 0x7f151682a000 0x9000 0x16000 /usr/lib64/libc.so.6
0x7f151682a000 0x7f151682c000 0x2000 0x4000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
0x7f151682c000 0x7f1516832000 0x6000 0x29000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
0x7f1516833000 0x7f1516834000 0x1000 0x3000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
0x7f1516834000 0x7f1516838000 0x4000 0x17000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
0x7f1516838000 0x7f151683d000 0x5000 0x20000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
0x7f151683d000 0x7f1516841000 0x4000 0x9000 /usr/lib64/libm.so.6
0x7f1516841000 0x7f1516843000 0x2000 0x9000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
0x7f1516844000 0x7f1516846000 0x2000 0x0 /usr/lib64/libnss_sss.so.2
0x7f1516846000 0x7f1516847000 0x1000 0x0 /usr/lib64/ld-linux-x86-64.so.2
0x7f1516847000 0x7f1516848000 0x1000 0x0 /usr/lib64/librt.so.1
0x7f1516848000 0x7f1516849000 0x1000 0x0 /usr/lib64/libpthread.so.0
0x7f1516849000 0x7f151684a000 0x1000 0x1000 /usr/lib64/libresolv.so.2
0x7f151684a000 0x7f151684b000 0x1000 0x0 /usr/lib64/libdl.so.2
0x7f151684b000 0x7f151684c000 0x1000 0x1000 /usr/lib64/libz.so.1.2.13
0x7f1b8f600000 0x7f1c11ede000 0x828de000 0x0 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f1dba800000 0x7f1dbba41000 0x1241000 0x0 /usr/lib/locale/locale-archive
0x7f1dc0000000 0x7f1dc0c0f000 0xc0f000 0x1000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/classes.jsa
0x7f1e18400000 0x7f1e21357000 0x8f57000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/modules
0x7f1e82206000 0x7f1e8222e000 0x28000 0x7d18f000 /home/doris_test_local/apache-doris-4.1.1-bin-x64/be/lib/doris_be
0x7f1e8222e000 0x7f1e82233000 0x5000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
0x7f1e82233000 0x7f1e82274000 0x41000 0x5000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
0x7f1e82274000 0x7f1e822fd000 0x89000 0x46000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
0x7f1e822fd000 0x7f1e822fe000 0x1000 0xce000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
0x7f1e822fe000 0x7f1e822ff000 0x1000 0xcf000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjsvml.so
0x7f1e8385a000 0x7f1e8385e000 0x4000 0x0 /usr/lib64/libnuma.so.1.0.0
0x7f1e8385e000 0x7f1e83864000 0x6000 0x4000 /usr/lib64/libnuma.so.1.0.0
0x7f1e83864000 0x7f1e83866000 0x2000 0xa000 /usr/lib64/libnuma.so.1.0.0
0x7f1e83866000 0x7f1e83867000 0x1000 0xc000 /usr/lib64/libnuma.so.1.0.0
0x7f1e83867000 0x7f1e83868000 0x1000 0xd000 /usr/lib64/libnuma.so.1.0.0
0x7f1e83873000 0x7f1e83876000 0x3000 0x0 /usr/lib64/libz.so.1.2.13
0x7f1e83876000 0x7f1e83884000 0xe000 0x3000 /usr/lib64/libz.so.1.2.13
0x7f1e83884000 0x7f1e8388b000 0x7000 0x11000 /usr/lib64/libz.so.1.2.13
0x7f1e8388b000 0x7f1e8388c000 0x1000 0x17000 /usr/lib64/libz.so.1.2.13
0x7f1e8388c000 0x7f1e8388d000 0x1000 0x18000 /usr/lib64/libz.so.1.2.13
0x7f1e83890000 0x7f1e83892000 0x2000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
0x7f1e83892000 0x7f1e83895000 0x3000 0x2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
0x7f1e83895000 0x7f1e83896000 0x1000 0x5000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
0x7f1e83896000 0x7f1e83897000 0x1000 0x6000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
0x7f1e83897000 0x7f1e83898000 0x1000 0x7000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement_ext.so
0x7f1e83898000 0x7f1e8389a000 0x2000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
0x7f1e8389a000 0x7f1e8389f000 0x5000 0x2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
0x7f1e8389f000 0x7f1e838a1000 0x2000 0x7000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
0x7f1e838a1000 0x7f1e838a2000 0x1000 0x8000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
0x7f1e838a2000 0x7f1e838a3000 0x1000 0x9000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libzip.so
0x7f1e838a3000 0x7f1e838a7000 0x4000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
0x7f1e838a7000 0x7f1e838b6000 0xf000 0x4000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
0x7f1e838b6000 0x7f1e838ba000 0x4000 0x13000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
0x7f1e838ba000 0x7f1e838bb000 0x1000 0x16000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
0x7f1e838bb000 0x7f1e838bc000 0x1000 0x17000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnet.so
0x7f1e84e07000 0x7f1e84e0e000 0x7000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
0x7f1e84e0e000 0x7f1e84e17000 0x9000 0x7000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
0x7f1e84e17000 0x7f1e84e1b000 0x4000 0x10000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
0x7f1e84e1b000 0x7f1e84e1c000 0x1000 0x14000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
0x7f1e84e1c000 0x7f1e84e1d000 0x1000 0x15000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libnio.so
0x7f1e86001000 0x7f1e8600d000 0xc000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
0x7f1e8600d000 0x7f1e8601f000 0x12000 0xc000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
0x7f1e8601f000 0x7f1e86025000 0x6000 0x1e000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
0x7f1e86025000 0x7f1e86026000 0x1000 0x23000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
0x7f1e86026000 0x7f1e86027000 0x1000 0x24000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjava.so
0x7f1e86028000 0x7f1e861ed000 0x1c5000 0x0 /usr/lib64/libc.so.6
0x7f1e861ed000 0x7f1e861f1000 0x4000 0x1c5000 /usr/lib64/libc.so.6
0x7f1e861f1000 0x7f1e861f3000 0x2000 0x1c9000 /usr/lib64/libc.so.6
0x7f1e86200000 0x7f1e8645b000 0x25b000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
0x7f1e8645b000 0x7f1e87296000 0xe3b000 0x25b000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
0x7f1e87296000 0x7f1e87531000 0x29b000 0x1096000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
0x7f1e87531000 0x7f1e875ec000 0xbb000 0x1331000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
0x7f1e875ec000 0x7f1e87621000 0x35000 0x13ec000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/server/libjvm.so
0x7f1e87680000 0x7f1e87682000 0x2000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
0x7f1e87682000 0x7f1e87683000 0x1000 0x2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
0x7f1e87683000 0x7f1e87684000 0x1000 0x3000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
0x7f1e87684000 0x7f1e87685000 0x1000 0x3000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
0x7f1e87685000 0x7f1e87686000 0x1000 0x4000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libmanagement.so
0x7f1e87686000 0x7f1e87688000 0x2000 0x0 /usr/lib64/libnss_sss.so.2
0x7f1e87688000 0x7f1e8768f000 0x7000 0x2000 /usr/lib64/libnss_sss.so.2
0x7f1e8768f000 0x7f1e87691000 0x2000 0x9000 /usr/lib64/libnss_sss.so.2
0x7f1e87691000 0x7f1e87692000 0x1000 0xa000 /usr/lib64/libnss_sss.so.2
0x7f1e87692000 0x7f1e87693000 0x1000 0xb000 /usr/lib64/libnss_sss.so.2
0x7f1e87696000 0x7f1e8769e000 0x8000 0x0 /tmp/hsperfdata_root/1933344
0x7f1e8769e000 0x7f1e876a2000 0x4000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
0x7f1e876a2000 0x7f1e876b5000 0x13000 0x4000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
0x7f1e876b5000 0x7f1e876bb000 0x6000 0x17000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
0x7f1e876bb000 0x7f1e876bd000 0x2000 0x1c000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
0x7f1e876bd000 0x7f1e876be000 0x1000 0x1e000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjimage.so
0x7f1e8770c000 0x7f1e8770d000 0x1000 0x0 /usr/lib64/librt.so.1
0x7f1e8770d000 0x7f1e8770e000 0x1000 0x0 /usr/lib64/librt.so.1
0x7f1e8770e000 0x7f1e8770f000 0x1000 0x1000 /usr/lib64/librt.so.1
0x7f1e87711000 0x7f1e87712000 0x1000 0x0 /usr/lib64/libpthread.so.0
0x7f1e87712000 0x7f1e87713000 0x1000 0x0 /usr/lib64/libpthread.so.0
0x7f1e87713000 0x7f1e87714000 0x1000 0x1000 /usr/lib64/libpthread.so.0
0x7f1e87714000 0x7f1e87715000 0x1000 0x0 /usr/lib64/libdl.so.2
0x7f1e87715000 0x7f1e87716000 0x1000 0x0 /usr/lib64/libdl.so.2
0x7f1e87716000 0x7f1e87717000 0x1000 0x1000 /usr/lib64/libdl.so.2
0x7f1e87717000 0x7f1e877f2000 0xdb000 0x0 /usr/lib64/libm.so.6
0x7f1e877f2000 0x7f1e877f3000 0x1000 0xda000 /usr/lib64/libm.so.6
0x7f1e877f3000 0x7f1e877f4000 0x1000 0xdb000 /usr/lib64/libm.so.6
0x7f1e877f4000 0x7f1e87801000 0xd000 0x0 /usr/lib64/libresolv.so.2
0x7f1e87801000 0x7f1e87802000 0x1000 0xc000 /usr/lib64/libresolv.so.2
0x7f1e87802000 0x7f1e87803000 0x1000 0xd000 /usr/lib64/libresolv.so.2
0x7f1e87805000 0x7f1e87806000 0x1000 0x0 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
0x7f1e87806000 0x7f1e87807000 0x1000 0x1000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
0x7f1e87807000 0x7f1e87808000 0x1000 0x2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
0x7f1e87808000 0x7f1e87809000 0x1000 0x2000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
0x7f1e87809000 0x7f1e8780a000 0x1000 0x3000 /usr/lib/jvm/java-17-openjdk-17.0.18.8-8.oe2403sp3.x86_64/lib/libjaas.so
0x7f1e87812000 0x7f1e8783f000 0x2d000 0x0 /usr/lib64/ld-linux-x86-64.so.2
0x7f1e8783f000 0x7f1e87841000 0x2000 0x2d000 /usr/lib64/ld-linux-x86-64.so.2
0x7f1e87841000 0x7f1e87843000 0x2000 0x2f000 /usr/lib64/ld-linux-x86-64.so.2
(gdb) x/16i $pc-32
0x7f1e860b22d8: addb $0x0,(%rcx)
0x7f1e860b22db: add %al,(%rax)
0x7f1e860b22dd: mov (%rsi),%rax
0x7f1e860b22e0: test %rcx,%rax
0x7f1e860b22e3: jne 0x7f1e860b2328
0x7f1e860b22e5: mov %rsi,%rax
0x7f1e860b22e8: mov %rax,%rsi
0x7f1e860b22eb: mov $0x8,%r10d
0x7f1e860b22f1: mov $0xe,%eax
0x7f1e860b22f6: syscall
=> 0x7f1e860b22f8: mov %eax,%edx
0x7f1e860b22fa: neg %edx
0x7f1e860b22fc: cmp $0xfffff000,%eax
0x7f1e860b2301: mov $0x0,%eax
0x7f1e860b2306: cmova %edx,%eax
0x7f1e860b2309: mov 0x88(%rsp),%rdx
(gdb) quit
[admin@192 log]$ `
I attempted the requested GDB inspection using the matching doris_be binary and the retained core.
Unfortunately, systemd-coredump marked this core as truncated, and GDB cannot unwind the crashing thread:
(gdb) set pagination off
(gdb) bt
#0 0x00007f1e860b22f8 in ?? ()
Backtrace stopped: Cannot access memory at address 0x7f1b441db350
Therefore, I cannot currently identify a ColumnDictI32::filter_by_selector frame or extract its arguments and local variables from this core. The required stack memory appears to be missing from the truncated core.
I will also provide the available register, signal, program-counter mapping, and instruction information. If this is still insufficient, I can adjust the core-dump limits so that a future crash preserves a complete core.
Please let me know whether the remaining truncated core and matching binary would still be useful through a private transfer channel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "ShummGen (via GitHub)" <gi...@apache.org>.
ShummGen commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5086682897
# Additional diagnostic information: corrupted segment pages and BE crashes
Thank you for the detailed review and for outlining the information required to investigate this issue.
I can provide most of the requested current metadata, query information, configuration, and diagnostic evidence from the recent incident. Raw core dumps and segment files may contain sensitive business data, so I would prefer to provide those through a private upload channel.
## Important clarification about the age of the original incident
The original incident was reported too long ago, and the corresponding BE log files have already expired under our log-retention policy. I am therefore unable to provide the exact one-to-two-minute log interval surrounding the original crash, and I do not want to reconstruct or guess any missing log lines.
The diagnostic information below comes from a **new, recent incident in the same environment**, observed on `2026-07-27`. It uses the same Doris build and again involves corrupted segment pages, ZSTD decompression failures, query failures, compaction failures, and a BE crash.
Please treat the information below as new reproducible evidence that may help investigate the same class of robustness problem. It is not the original historical crash log, and I cannot currently prove that the recent crash and the original crash have exactly the same root cause.
One important clarification is that the latest crash observed in this environment does not have exactly the same top stack frames as the previously reported `ColumnStr::serialize_impl` / `DistinctStreamingAgg` crash. The latest crash and the ongoing compaction failures may share the same underlying corrupted segment pages, but they may also represent separate error-propagation paths. I have kept them separate below rather than assuming they have the same root cause.
## 1. Exact Doris build information
All FE and BE nodes report the same build:
```text
doris-4.1.1-rc01-b10073ad9ca
```
`SELECT VERSION()` returns:
```text
5.7.99
```
Cluster topology:
```text
FE: 1 node, alive, elected master
BE: 3 nodes, all alive
Deployment mode: local storage
Replication: 3 replicas
```
The FE and all three BEs report the same `doris-4.1.1-rc01-b10073ad9ca` build string in `SHOW FRONTENDS` and `SHOW BACKENDS`.
The binaries were obtained from the Doris binary distribution rather than compiled locally. I do not currently have a separate source-build commit beyond the build identifier embedded in the version string.
## 2. Incident timeline
All timestamps below are China Standard Time (`Asia/Shanghai`, UTC+8).
```text
2026-07-27 09:31:41 One BE terminated with SIGSEGV.
2026-07-27 09:31:53 systemd restarted the BE; it rejoined the cluster and is currently alive.
2026-07-27 09:46:28 A query reading the affected log table failed with:
[INTERNAL_ERROR] ZSTD_decompressDCtx error: Data corruption detected
2026-07-27 10:22:00 Compaction status was checked on all three replicas of two affected tablets.
All six replicas were still reporting the same ZSTD decompression error.
```
The latest BE crash stack included frames around:
```text
doris::vectorized::ColumnDictI32::filter_by_selector
doris::segment_v2::SegmentIterator
doris::OlapScanner
```
Earlier crashes also included an exception similar to:
```text
std::out_of_range: map::at
```
The stack above differs from the earlier `ColumnStr<unsigned int>::serialize_impl` / `DistinctStreamingAgg` stack. I would appreciate guidance on whether these should remain in the same issue or be tracked as two related issues.
## 3. Timestamp clarification
The previously reported value:
```text
Aborted at 1748134471
```
corresponds to `2025-05-25`, not `2026-05-25`.
Because the original logs are no longer retained, I cannot reliably reconstruct whether the original event occurred on `2025-05-25` or `2026-05-25`. The timestamp/date combination in the original report should therefore be treated as inaccurate and should not be used for source-version matching.
The exact recent incident described in this update occurred on `2026-07-27`, and its timestamps were collected directly from the current environment.
Any timestamps or source line numbers previously presented as examples or reconstructed excerpts should be treated as illustrative. The timestamps in the timeline above and the errors in the following sections were collected from the running environment.
## 4. Affected table schema
The affected table is a local-storage OLAP table. The table and database names can be anonymized if required; the relevant schema is:
```sql
CREATE TABLE `alb_access_log` (
`log_date` DATE NULL,
`time` DATETIME NULL,
`status` VARCHAR(16) NULL,
`request_method` VARCHAR(16) NULL,
`host` VARCHAR(256) NULL,
`client_ip` VARCHAR(64) NULL,
`request_time` DECIMAL(10,3) NULL,
`request_uri` VARCHAR(65533) NULL,
`http_user_agent` VARCHAR(65533) NULL,
`cqid` VARCHAR(128) NULL,
`h1` VARCHAR(256) NULL,
`click_id` VARCHAR(255) NULL,
`event_type` VARCHAR(32) NULL,
`traffic` VARCHAR(64) NULL,
`import_time` DATETIME NULL,
`source_file` VARCHAR(64) NULL,
`request_id` VARCHAR(255) NULL,
`city_region` VARCHAR(128) NULL
)
ENGINE=OLAP
DUPLICATE KEY(`log_date`, `time`)
PARTITION BY RANGE(`log_date`)
DISTRIBUTED BY HASH(`client_ip`) BUCKETS 32
PROPERTIES (
"replication_allocation" = "tag.location.default: 3",
"storage_medium" = "hdd",
"storage_format" = "V2",
"inverted_index_storage_format" = "V3",
"light_schema_change" = "true",
"disable_auto_compaction" = "false",
"enable_single_replica_compaction" = "false"
);
```
The real table also contains inverted indexes on `log_date`, `request_uri`, `click_id`, `h1`, `source_file`, `traffic`, and `event_type`.
The other table participating in the failing query contains:
```text
id BIGINT NOT NULL
tk_create_time DATETIME NOT NULL
click_id VARCHAR(675) NULL
```
## 5. Affected partition and tablets
Affected partition:
```text
Partition: p20260726
Range: [2026-07-26, 2026-07-27)
State: NORMAL according to FE metadata
Buckets: 32
Replicas: 3
Rows: 75,370,092 according to partition metadata
Size: approximately 46.177 GB
```
Two tablets are currently known to contain unreadable pages:
| Tablet ID | Rows per replica | Replica count | FE metadata state | Actual read/compaction result |
|---|---:|---:|---|---|
| `1784714655425` | `2,628,463` | 3 | `NORMAL`, `IsBad=false` | ZSTD decompression corruption on all replicas |
| `1784714655445` | `2,695,687` | 3 | `NORMAL`, `IsBad=false` | ZSTD decompression corruption on all replicas |
The replica versions were aligned at version `140`. FE tablet diagnosis therefore reports healthy metadata, although actual page reads fail.
At approximately `2026-07-27 10:22 CST`, the cumulative compaction status of both tablets on every BE reported:
```text
[INTERNAL_ERROR]ZSTD_decompressDCtx error: Data corruption detected
```
This produced six matching failures:
```text
BE-A / tablet 1784714655425: ZSTD decompression corruption
BE-B / tablet 1784714655425: ZSTD decompression corruption
BE-C / tablet 1784714655425: ZSTD decompression corruption
BE-A / tablet 1784714655445: ZSTD decompression corruption
BE-B / tablet 1784714655445: ZSTD decompression corruption
BE-C / tablet 1784714655445: ZSTD decompression corruption
```
The most recent successful cumulative compaction on BE-A and BE-B was on `2026-07-26`. BE-C had no successful cumulative or base compaction recorded for these two replicas after its current data copy was created.
Because all three replicas fail actual reads, marking a single replica bad and cloning it from another replica does not provide a safe recovery path.
## 6. Query that exposes the failure
The production operation is an `INSERT INTO ... WITH ... SELECT` statement. The relevant query structure is shown below. Business-specific output columns and unrelated joins have been omitted, but the scans of the affected table are unchanged:
```sql
SET enable_spill = true;
SET exec_mem_limit = 8589934592;
SET parallel_pipeline_task_num = 4;
SET @order_start = '2026-07-26 00:00:00';
SET @order_end = '2026-07-27 00:00:00';
INSERT INTO target_table (...)
WITH order_window AS (
SELECT id, click_id, tk_create_time, ...
FROM order_detail_table
WHERE click_id IS NOT NULL
AND click_id != ''
AND tk_create_time >= CAST(@order_start AS DATETIME)
AND tk_create_time < CAST(@order_end AS DATETIME)
),
click_ids AS (
SELECT DISTINCT click_id
FROM order_window
),
click_candidates AS (
SELECT
alb.h1 AS click_id,
alb.time AS click_time,
alb.cqid,
alb.traffic,
alb.request_uri
FROM alb_access_log alb
INNER JOIN click_ids ids ON alb.h1 = ids.click_id
WHERE alb.h1 IS NOT NULL
AND alb.h1 != ''
),
click_candidates_huawei AS (
SELECT
ids.click_id,
alb.time AS click_time,
alb.cqid,
alb.traffic,
alb.request_uri
FROM alb_access_log alb
INNER JOIN huawei_click_ids ids
ON alb.h1 = ids.log_id
AND CAST(alb.cqid AS STRING) = ids.cqid
WHERE alb.traffic = 'HUAWEI'
AND alb.h1 IS NOT NULL
AND alb.h1 != ''
AND alb.cqid IS NOT NULL
),
event_candidates AS (
SELECT
alb.click_id,
alb.time,
alb.event_type,
alb.request_id,
extract_url_parameter(alb.request_uri, 'project_id'),
extract_url_parameter(alb.request_uri, 'promotion_id')
FROM alb_access_log alb
INNER JOIN click_ids ids ON alb.click_id = ids.click_id
WHERE alb.click_id IS NOT NULL
AND alb.click_id != ''
AND alb.time IS NOT NULL
AND alb.source_file >= 'tracer'
AND alb.source_file < 'traces'
AND alb.event_type IN ('3', '16', '17', '18', 'bigIncome')
)
SELECT ...;
```
The query contains:
- `SELECT DISTINCT click_id`, implemented by the plan as a streaming vectorized aggregate grouped by `click_id`.
- Multiple `ROW_NUMBER()` window functions.
- Sorting by `request_uri`, whose type is `VARCHAR(65533)`.
- URL extraction and URL decoding from `request_uri`.
- Three independent scans of `alb_access_log`.
The `DISTINCT`/grouping column from the order table is `VARCHAR(675)`. Related join columns in the log table are `h1 VARCHAR(256)` and `click_id VARCHAR(255)`.
## 7. Relevant EXPLAIN output
The full plan contains 33 plan fragments. The important result is that the order table is correctly pruned to one partition, but every scan of `alb_access_log` reads all partitions because no `log_date` predicate is present.
Order table scan:
```text
TABLE: order_detail_table
PREDICATES:
tk_create_time >= '2026-07-26 00:00:00'
tk_create_time < '2026-07-27 00:00:00'
partitions=1/364 (p20260726)
tablets=32/32
```
Log table scans:
```text
TABLE: alb_access_log
partitions=83/83
tablets=2656/2656
numNodes=3
```
This occurs for the normal click lookup, the Huawei click lookup, and the event lookup.
The `SELECT DISTINCT click_id` portion is planned as:
```text
VAGGREGATE (update serialize)
STREAMING
group by: click_id
```
Therefore the failing query can exercise both the segment-scanning path and a downstream streaming aggregation path.
I can attach the complete non-anonymized `EXPLAIN` output separately if required.
## 8. Session and BE configuration
Query-specific session settings:
```text
enable_spill = true
exec_mem_limit = 8589934592 # 8 GiB
parallel_pipeline_task_num = 4
```
Current session defaults observed independently of the query:
```text
skip_bad_tablet = false
fallback_other_replica_when_fixed_corrupt = false
```
`SHOW VARIABLES LIKE 'enable_query_memory_overcommit'` returned no row in this build, so I cannot currently provide a runtime value for that variable.
Relevant configuration from the BE that most recently crashed:
```text
deploy_mode = "" # local-storage deployment
mem_limit = 90%
memory_mode = moderate
storage_page_cache_limit = 20%
disable_storage_page_cache = false
disable_segment_cache = false
check_segment_when_build_rowset_meta = false
storage_strict_check_incompatible_old_format = true
sync_file_on_close = true
disable_auto_compaction = false
storage_root_path = <two local storage paths>
```
The BE has one original Doris storage path and one additional local disk path added later for capacity expansion. The expansion was performed through Doris `storage_root_path` configuration and Doris-managed balancing. To the best of my knowledge, the affected segment files were not manually edited, overwritten, restored, or copied into the tablet directory outside Doris.
The FE was migrated to another host before this incident, but the BE data directories remained managed by their existing BE processes. No BE tablet directory was manually copied as part of the FE migration.
## 9. Relationship between the core dumps
I cannot confirm that all previously observed historical core dumps had the same trigger. The original BE logs needed to correlate those dumps have already expired.
The relationship analysis below therefore applies primarily to the recent `2026-07-27` incident, not as a reconstruction of the original historical incident.
Current evidence suggests at least two paths must be investigated separately:
1. Query execution reading corrupted segment pages and forwarding data toward vectorized operators.
2. Cumulative/base compaction reading the same corrupted pages and repeatedly returning ZSTD errors.
At least one recent SIGSEGV occurred during an OLAP scanner/segment iterator path. The compaction status APIs show repeated corruption failures, but I have not yet proven that every BE core dump was created by a compaction thread.
I will correlate the following fields for each retained core and log interval:
```text
crash timestamp
BE node
thread ID
query ID
fragment instance ID
tablet ID
rowset ID
segment path
top stack frames
whether a compaction task was active
```
I cannot reliably claim that the historical dumps were identical. If a core from the recent crash is available, I will correlate it with the recent retained logs instead.
## 10. GDB and core-dump availability
The historical BE logs are no longer available. I am checking whether any matching historical core file is still retained, but I cannot currently promise a complete GDB analysis of the original crash.
For the recent `2026-07-27` crash, I will check whether the new core file and the exact matching BE binary are retained on the crashed node.
If available, I will collect at least:
```gdb
set pagination off
set print pretty on
thread apply all bt full
```
For a core containing the `ColumnStr<unsigned int>::serialize_impl` frame, I will also attempt:
```gdb
frame <ColumnStr::serialize_impl frame>
info args
info locals
p row
p this
p offsets.size()
p chars.size()
p offsets[row]
p offsets[row - 1]
```
For the serialized-key caller, if symbols permit:
```gdb
p keys[row]
p keys[row].data
p keys[row].size
```
If the distributed binary is stripped or optimized and GDB cannot evaluate these expressions, I will include the exact GDB error messages.
No string contents or raw user data will be included. Pointer values, sizes, offsets, symbols, and stack frames can be provided.
## 11. Complete BE log interval
The log interval surrounding the original historical crash is no longer available because it has expired under the retention policy.
I am checking retention for the exact log interval around the recent `2026-07-27 09:31:41 CST` crash.
The requested export will cover approximately:
```text
2026-07-27 09:29:30 CST
through
2026-07-27 09:33:30 CST
```
It will preserve, when present:
```text
thread IDs
query IDs
fragment instance IDs
tablet IDs
rowset IDs
segment paths
the first checksum/decompression error
scanner and iterator errors
query cancellation messages
compaction messages
the final crash stack
```
Business values inside URLs, request payloads, hostnames, and user identifiers will be redacted without changing technical identifiers such as query IDs, tablet IDs, rowset IDs, or stack frames.
## 12. Corrupted segment sample and metadata
The affected replicas and storage paths are still online, so it should be possible to collect:
```text
one corrupted segment file
tablet metadata
rowset metadata
tablet schema
segment and rowset identifiers
file size
SHA-256 checksum
```
However, all three replicas of the two known tablets currently fail decompression. I do not currently have a confirmed healthy copy of the same segment for comparison.
The segment may contain request URLs and other business data. I will not attach it to the public GitHub issue. Please provide a private upload channel if the raw segment or core dump would be useful.
If preferred, I can first provide only:
- File size and SHA-256 for each replica.
- Tablet/rowset/segment metadata.
- A byte-level comparison between replicas.
- A reduced or intentionally corrupted synthetic sample that reproduces the same error without business data.
## 13. Current conclusions
The current evidence supports the following limited conclusions:
1. The FE metadata alone does not detect this corruption because replica state, replica count, and versions are normal.
2. Actual page reads and compaction consistently detect ZSTD data corruption.
3. The same two tablets fail on all three replicas, so this is not currently recoverable by cloning from another in-cluster replica.
4. A production query scans all log-table partitions and reliably reaches the affected partition.
5. Doris should return a safe query or compaction error when the page cannot be decompressed; the BE should not terminate with SIGSEGV while handling that error.
6. The different observed top stack frames may indicate multiple unsafe error-propagation paths, so query execution and compaction should be investigated separately.
## 14. Requested next steps
Could you please confirm:
1. Whether the current `ColumnDictI32::filter_by_selector` / `SegmentIterator` crash should remain in this issue together with the earlier `ColumnStr::serialize_impl` / `DistinctStreamingAgg` crash.
2. Whether there is a private upload channel for one representative core dump, a complete BE log interval, and one corrupted segment sample.
3. Whether the build identifier `doris-4.1.1-rc01-b10073ad9ca` is sufficient to map the distributed binary to the exact source revision, or whether another command/file should be used to extract the build commit.
I apologize that the original historical log interval is no longer available. I hope the recent incident, for which the affected tablets are still online and the corruption remains reproducible, can provide a more useful investigation target.
I can provide the full `SHOW FRONTENDS`, `SHOW BACKENDS`, `SHOW CREATE TABLE`, `EXPLAIN`, BE configuration, and GDB output as separate sanitized attachments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [I] [Bug] BE crashes with SIGSEGV in DistinctStreamingAgg/ColumnStr::serialize_vec when reading corrupted segment files [doris]
Posted by "YanzhiJin5 (via GitHub)" <gi...@apache.org>.
YanzhiJin5 commented on issue #63609:
URL: https://github.com/apache/doris/issues/63609#issuecomment-5379565592
Hi @ShummGen,
First, I apologize for the long gap since your last update, and thank you again for providing such detailed information from the July 27 incident. I have spent some time going back through the issue and tracing the relevant code on the exact production revision you provided (`b10073ad9ca17cd5685c4dd3b3ef650f256376d0`) before asking you for any additional data.
A brief update on the investigation so far:
* I traced the storage-read error path from `PageIO` through `ColumnReader`, `FileColumnIterator`, `SegmentIterator`, and `OlapScanner`.
* On the normal error path I reviewed, a block from a storage read that returns an error is not forwarded downstream to aggregation. Therefore, I do not currently have evidence that the nearby checksum/ZSTD errors directly produced the block involved in the earlier `DistinctStreamingAgg -> ColumnStr::serialize_impl` crash.
* For that reason, I am treating the earlier `ColumnStr` crash and the more recent `ColumnDictI32::filter_by_selector -> SegmentIterator -> OlapScanner` crash as separate signatures unless further evidence connects them.
* I also investigated a missing dictionary-code bounds check and prototyped a defensive validation change with regression tests. However, after revisiting the evidence, I do **not** think the available production data is sufficient to claim that invalid dictionary codes caused your crash. I would rather confirm or rule that out from the actual crash state than rely on a speculative fix.
The good news is that the source investigation has now narrowed the recent `ColumnDictI32::filter_by_selector` crash to a much more specific question: at the failing iteration, were the selector and dictionary code actually within their expected ranges?
If the core from the recent July 27 crash and its matching `doris_be` binary are still retained, please use whichever option is easiest for you.
### Option 1: minimal GDB output
If opening the core in GDB is convenient, a small amount of output would be enough for the next step. After opening the matching binary and core, the following commands should be sufficient initially:
```gdb
set pagination off
bt
```
If the backtrace contains a `ColumnDictI32::filter_by_selector` frame, select its frame number as shown by `bt`, for example:
```gdb
frame <N>
info args
info locals
```
There is no need to spend time manually interpreting the values. Please just send the output that GDB is able to provide. Optimized production binaries may report some variables as `<optimized out>`; that is completely fine, and there is no need to work around it.
Please also feel free to redact any business data, paths, SQL text, URLs, or other sensitive information before posting the output publicly.
### Option 2: existing core file
If running GDB and extracting variables yourself would be more work, I would be equally happy to inspect the existing core and matching `doris_be` binary directly.
Since you previously mentioned that the raw core/segment may contain sensitive business data, I do **not** suggest uploading them to the public GitHub issue. If a Doris maintainer can recommend an appropriate private transfer channel, this may actually be the lower-effort option for you: you could provide the existing core and matching binary (and separate debug symbols, if any), and I can do the GDB investigation from there.
If neither option is convenient or the recent core is no longer retained, that is also completely fine. I can continue from the source side and try to construct a local reproducer using the information you have already provided.
Thank you again for preserving and sharing the diagnostic information. It has been very helpful in narrowing the investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org