You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2026/10/18 09:57:27 UTC
[PR] [fix](regression) Scope Hive split and Paimon writer thread checks [doris]
Gabriel39 opened a new pull request, #69230:
URL: https://github.com/apache/doris/pull/69230
### What problem does this PR solve?
`test_hive_compress_type_large_data` compares exact file and split counts against streaming EXPLAIN estimates. A cold catalog can report `(approximate)inputSplitNum=1024` even though the fixture contains 16 files. Use statement-local `SET_VAR(enable_external_table_batch_mode=false)` for both exact plan checks. Keep the synchronous and explicitly forced streaming result checks.
`test_paimon_write_thread_lifecycle` treats growth in all BE JVM threads as a writer leak. Existing native blocking workers attach to the JVM on their first JNI call and remain attached until the worker exits. A stable global baseline is therefore not guaranteed, even when every writer closes its executor correctly. Check the writer-owned `doris-paimon-` threads through the existing process-thread endpoint instead, accounting for Linux's 15-byte thread-name limit. Require them to disappear within 30 seconds after each of ten write phases, and retain total process-thread counts in diagnostics. This detects even one surviving writer thread without increasing a tolerance or moving a baseline.
### Release note
None
### Check List (For Author)
- Test
- [x] Regression test
- [x] Manual test (details below)
- Behavior changed:
- [x] No production behavior changes; regression assertions now measure the intended resources.
- Does this need documentation?
- [x] No.
Local validation used the original DEBUG/ASAN build `6c27d4d8b0b4649f6517e15792fba672a54e2935`, an isolated FE/BE, Hive 3/HDFS, Spark 4/Paimon, and MinIO:
- **Before:** ran both unmodified suites. Hive failed the exact split assertion with the approximate 1024 estimate. With `parallel_pipeline_task_num=4`, Paimon failed bounded warm-up with JVM counts `451, 497, 525, 587, 606, 676`. Thread dumps mapped the growing attached threads to `p_normal_blocki`; writer compaction threads had exited. The failure reproduces the invalid global-count assumption, through the warm-up assertion rather than the later baseline assertion.
- **After:** restarted the BE, kept parallelism at 4, and ran both fixed suites through `run-regression-test.sh`; **2 suites passed, 0 failed, 0 skipped**. Paimon executed 120 inserts, verified 120,000 rows, and passed all ten writer-thread checks. Its attached blocking-worker count still grew from 54 to 368, so passing does not depend on a pre-warmed JVM. BE logs confirm the compaction executor actually ran.
- Hive used the original fixture: 16 files, 734,675,596 bytes, eight partitions. Both plan checks and all synchronous/forced-streaming result checks passed.
- Verified Paimon's actual `ExecutorThreadFactory` name truncates to `doris-paimon-co` on JDK 17/Linux. `git diff --check` passed.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](regression) Scope Hive split and Paimon writer thread checks [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69230:
URL: https://github.com/apache/doris/pull/69230#issuecomment-6248492979
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](regression) Scope Hive split and Paimon writer thread checks [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #69230:
URL: https://github.com/apache/doris/pull/69230
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](regression) Scope Hive split and Paimon writer thread checks [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69230:
URL: https://github.com/apache/doris/pull/69230#issuecomment-6248492778
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](regression) Scope Hive split and Paimon writer thread checks [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #69230:
URL: https://github.com/apache/doris/pull/69230#issuecomment-6248493223
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [fix](regression) Scope Hive split and Paimon writer thread checks [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #69230:
URL: https://github.com/apache/doris/pull/69230#issuecomment-6257664872
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org