You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "Gabriel39 (via GitHub)" <gi...@apache.org> on 2026/10/01 11:21:55 UTC
[PR] [improvement](lance) Upgrade lance-c for synchronous index prewarm (branch-4.1) [doris]
Gabriel39 opened a new pull request, #68698:
URL: https://github.com/apache/doris/pull/68698
### What problem does this PR solve?
Related issue: #68692 (M1)
Related PRs: lance-format/lance-c#94, #68687, #68689
Lance-C now exposes the synchronous index-prewarm API needed by the planned Doris SQL command, but `branch-4.1` still pins a revision without that API. Upgrade the dependency to `98468344bc9d56aa7ed4a4192e844afa62b34ba1`, which includes the merged upstream C/C++ binding, tests, and documentation. The Lance Rust revision is unchanged.
Refresh the existing Foyer patch against this revision without changing its added/removed payload. Foyer remains a separate data-file cache dependency until lance-format/lance-c#73 merges; no prewarm implementation is carried as a Doris patch.
Include the dependency installation fingerprint guard from #68687/#68689 so an old compilation image cannot silently satisfy the new pin. Missing or mismatched rebuild sources fail before existing installed dependencies are removed. Tests cover legacy installs, matching reuse, pin/patch changes, incomplete sources and artifacts, and failed publication.
This delivers the Doris dependency portion of M1. SQL parsing, BE RPCs, and shared-session integration belong to M2 and are not implemented here. The parent issue remains open.
### Validation
On the pinned upstream source with the refreshed Foyer patch (Rust 1.98.1, debug info disabled):
- `cargo test --locked`: 497 passed, 0 failed; 3 explicit C/C++ tests run separately.
- `cargo test --locked --test compile_and_run_test -- --ignored --test-threads=1`: 3 passed (C API, C++ API, and static OSS transport).
- Prewarm tests cover invalid arguments/missing indexes, multi-segment scalar indexes and historical snapshots, and multi-segment IVF cache reuse across separately opened handles sharing a session.
In Doris:
- `bash thirdparty/test/lance-install-test.sh`: passed.
- `bash thirdparty/test/lance-prefilter-patch-test.sh <archive-directory>`: passed, including fresh/cached extraction and preservation of the upstream prewarm APIs and regression.
- `bash -n` on affected shell scripts and `git diff --check`: passed.
- Verified the retained Foyer patch's added/removed payload is identical to the previous pin.
Full Doris compilation and BE/FE suites are left to PR CI; no FE/BE source files are changed.
### Release note
None. Dependency preparation for index prewarm; no new SQL interface is exposed.
### Check List (For Author)
- Test:
- [x] Unit Test
- [x] Manual test (commands above)
- Behavior changed:
- [x] Yes. Stale Lance-C installations require matching rebuild sources instead of being reused silently.
- Does this need documentation?
- [x] No. The upstream C/C++ API documentation is included in the dependency; Doris SQL documentation accompanies M2/M3.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](lance) Upgrade lance-c for synchronous index prewarm (branch-4.1) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68698:
URL: https://github.com/apache/doris/pull/68698#issuecomment-6050668654
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](lance) Upgrade lance-c for synchronous index prewarm (branch-4.1) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68698:
URL: https://github.com/apache/doris/pull/68698#issuecomment-5930286467
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](lance) Upgrade prewarm dependency and remove install guard (branch-4.1) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 merged PR #68698:
URL: https://github.com/apache/doris/pull/68698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](lance) Upgrade prewarm dependency and remove install guard (branch-4.1) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68698:
URL: https://github.com/apache/doris/pull/68698#discussion_r4213935182
##########
build.sh:
##########
@@ -456,21 +455,10 @@ if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ||
! -f "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ||
! -s "${DORIS_THIRDPARTY}/installed/lib64/libpaimon_c.a" ||
! -s "${DORIS_THIRDPARTY}/installed/include/paimon_rust/paimon.h" ||
- -e "${DORIS_THIRDPARTY}/installed/lib64/.paimon-installing" ]] ||
- ! lance_c_install_is_current "${DORIS_HOME}/thirdparty" "${DORIS_THIRDPARTY}/installed"; then
- # External trees can be partially updated or pinned to another revision. Preserve
- # the existing prefix unless their build inputs can produce the requested Lance version.
- for input in build-thirdparty.sh download-thirdparty.sh vars.sh lance-install.sh patches/lance-c-foyer.patch; do
- if [[ ! -f "${DORIS_THIRDPARTY}/${input}" || ! -r "${DORIS_THIRDPARTY}/${input}" ]]; then
- echo "Third-party dependencies require a rebuild, but ${input} is missing or unreadable." >&2
- echo "Refresh the compilation image or set DORIS_THIRDPARTY to a complete third-party source tree." >&2
- exit 1
- fi
- done
- if ! expected_lance_fingerprint="$(lance_c_install_fingerprint "${DORIS_HOME}/thirdparty")" ||
- ! rebuild_lance_fingerprint="$(lance_c_install_fingerprint "${DORIS_THIRDPARTY}")" ||
- [[ "${rebuild_lance_fingerprint}" != "${expected_lance_fingerprint}" ]]; then
- echo "Lance rebuild sources do not match this checkout; installed dependencies have been preserved." >&2
+ -e "${DORIS_THIRDPARTY}/installed/lib64/.paimon-installing" ]]; then
Review Comment:
[P2] Preserve Lance install-completeness checks when removing the fingerprint. A targeted `build-thirdparty.sh lance_c` can be interrupted just after `build_lance_c` deletes `installed/include/lance`; the previous archive and every other sentinel remain, so this gate skips repair and the next BE build fails to find `lance/lance.h`. It also accepts a zero-byte archive. Check both headers and the archive for nonempty contents, and use an installing marker spanning header and archive publication to catch mixed pairs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](lance) Upgrade lance-c for synchronous index prewarm (branch-4.1) [doris]
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #68698:
URL: https://github.com/apache/doris/pull/68698#issuecomment-5930284067
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://doris.apache.org/community/how-to-contribute/review-and-ci).
Please clearly describe your PR:
1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](lance) Upgrade lance-c for synchronous index prewarm (branch-4.1) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68698:
URL: https://github.com/apache/doris/pull/68698#issuecomment-5930286879
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](lance) Upgrade lance-c for synchronous index prewarm (branch-4.1) [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #68698:
URL: https://github.com/apache/doris/pull/68698#discussion_r4155397448
##########
thirdparty/build-thirdparty.sh:
##########
@@ -2228,9 +2230,14 @@ build_lance_c() {
env "${cargo_env[@]}" "${cargo_bin}" "${cargo_args[@]}"
mkdir -p "${TP_INSTALL_DIR}/include" "${TP_INSTALL_DIR}/lib64"
+ # Invalidate before publishing either file so interrupted installs cannot reuse
+ # a matching marker with a partial header/archive pair.
+ rm -f "${TP_INSTALL_DIR}/lib64/.lance-c-fingerprint"
Review Comment:
[P2] Serialize publication of the archive and fingerprint. When two builds share this install prefix across an in-place Lance revision update, an older process can remove the marker, pause, then let a newer process publish its header, archive, and new marker. If the older process resumes and replaces `liblance_c.a` but stops before rewriting its marker, the new marker remains and `lance_c_install_is_current` accepts the old archive. A lock or versioned staging around the whole header/archive/marker publication is needed.
##########
build.sh:
##########
@@ -471,6 +483,11 @@ if [[ ! -f "${DORIS_THIRDPARTY}/installed/lib/${LAST_THIRDPARTY_LIB}" ||
else
bash "${DORIS_THIRDPARTY}/build-thirdparty.sh" -j "${PARALLEL}" --clean
fi
+ # An external build script can itself be stale. Never link its old output silently.
+ if ! lance_c_install_is_current "${DORIS_HOME}/thirdparty" "${DORIS_THIRDPARTY}/installed"; then
Review Comment:
[P2] Preserve the installed prefix until a stale external builder is ruled out. A readable external `build-thirdparty.sh` with matching `vars.sh` and patch passes the preflight, but the script removes the whole `installed` tree before running it. The new stale-builder fixture supplies a builder that exits successfully without publishing a Lance marker, so this post-build check fails only after the previous dependencies are gone. Stage the rebuild and replace the prefix after verifying its output, or verify the builder's capability before deleting the existing install.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [improvement](lance) Upgrade lance-c for synchronous index prewarm (branch-4.1) [doris]
Posted by "Gabriel39 (via GitHub)" <gi...@apache.org>.
Gabriel39 commented on PR #68698:
URL: https://github.com/apache/doris/pull/68698#issuecomment-6050669927
/review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org