(doris) branch branch-4.2 updated: [fix](test) Keep Lance Unicode fixture paths portable (branch-4.2) (#69126)

[yi...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

yiguolei pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git


The following commit(s) were added to refs/heads/branch-4.2 by this push:
     new 5904f8c8bc4 [fix](test) Keep Lance Unicode fixture paths portable (branch-4.2) (#69126)
5904f8c8bc4 is described below

commit 5904f8c8bc4cca93ed64153ce56f7bc4d77d50e1
Author: Gabriel <li...@selectdb.com>
AuthorDate: Thu Oct 15 23:38:47 2026 +0800

    [fix](test) Keep Lance Unicode fixture paths portable (branch-4.2) (#69126)
    
    ### What problem does this PR solve?
    
    Related PR: #69124, #68707
    
    Apply the fixture portability fix from #69124 to `branch-4.2`.
    
    The Unicode Lance branch fixture tracks five files below `tree/devᲉ/`,
    causing checkout failures on filesystems that cannot represent that
    directory name. Store these files under `tree/dev_u1c89/` in Git and
    restore the original Unicode object prefix when uploading to MinIO.
    Fixture payloads and runtime Unicode branch coverage remain unchanged.
    Update the fixture checker and regeneration instructions to use the
    portable layout, and fail initialization if upload or relocation fails.
    
    ### Release note
    
    None
    
    ### Check List (For Author)
    
    - Test
        - [x] Manual test (details below)
    - Behavior changed:
        - [x] No. Runtime fixture paths and contents are unchanged.
    - Does this need documentation?
        - [x] No. Fixture regeneration instructions are updated inline.
    
    Validation on this branch:
    
    - All tracked paths are ASCII and contain no literal backslashes.
    - Fresh checkout of all 10 Unicode fixture files with
    `core.protectNTFS=true` and `core.protectHFS=true` on Linux.
    - `lance_build_unicode_branch.py --check` passes with pylance 12.0.0.
    - Python, YAML and shell syntax checks and `git diff --check` pass.
    - Verified the complete Iceberg fixture/setup subtree is identical to
    #69124. That change was validated against temporary MinIO with two
    initialization runs, matching all 515 original object keys and blob
    hashes, and successful reads of both Unicode branch versions and vector
    search. The MinIO integration test was not repeated for this identical
    port.
    
    Full Doris regression tests and a native Windows checkout were not run
    locally.
    
    ### Check List (For Reviewer who merge this PR)
    
    - [ ] Confirm the release note
    - [ ] Confirm test cases
    - [ ] Confirm document
    - [ ] Add branch pick label
---
 .../thirdparties/docker-compose/iceberg/iceberg.yaml.tpl |   5 ++++-
 .../iceberg/scripts/lance_build_unicode_branch.py        |  15 +++++++++++----
 .../1-1be00bd2-8443-4851-9ab8-4bdcee3ffa5e.txn           | Bin
 .../1-c1998c6f-6213-4b7e-ae7c-cc00cde3b7de.txn           |   0
 .../dev_u1c89/_versions/18446744073709551613.manifest    | Bin
 .../dev_u1c89/_versions/18446744073709551614.manifest    | Bin
 ...10100011000100001111040aa57401ab473a6a921298540.lance | Bin
 7 files changed, 15 insertions(+), 5 deletions(-)

diff --git a/docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl b/docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl
index f2dddc0a319..a9061003f89 100644
--- a/docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl
+++ b/docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl
@@ -229,6 +229,8 @@ services:
     # an earlier fixture would win over the one just published and point readers at tables
     # that no longer exist. This bucket is not recreated between runs, so a plain copy would
     # merge the two catalogs. --remove makes the prefix match the committed fixture exactly.
+    # The Unicode branch uses an ASCII directory in Git for portable checkout. Restore its
+    # object keys after each mirror: Lance resolves the original name recorded in the manifests.
     entrypoint: >
       /bin/sh -c "
       until (/usr/bin/mc config host add minio http://minio:9000 admin password) do echo '...waiting...' && sleep 1; done;
@@ -240,7 +242,8 @@ services:
         /usr/bin/mc policy set public minio/warehouse;
         /usr/bin/mc cp -r /mnt/data/input/minio/warehouse/* minio/warehouse/;
       fi;
-      /usr/bin/mc mirror --overwrite --remove /mnt/preinstalled_data/lance/ minio/warehouse/lance/;
+      /usr/bin/mc mirror --overwrite --remove /mnt/preinstalled_data/lance/ minio/warehouse/lance/ &&
+      /usr/bin/mc mv --recursive minio/warehouse/lance/unicode_branch.lance/tree/dev_u1c89/ minio/warehouse/lance/unicode_branch.lance/tree/devᲉ/ || exit 1;
       /usr/bin/mc cp -r /mnt/preinstalled_data/iceberg/ minio/warehouse/wh/multi_catalog/;
       "
 
diff --git a/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_unicode_branch.py b/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_unicode_branch.py
index 1179ddbe6a9..4b219bb6617 100644
--- a/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_unicode_branch.py
+++ b/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_unicode_branch.py
@@ -37,12 +37,18 @@ created against the fixture's final location (see lance_build_search_snapshot.py
       --storage-option secret_access_key=password --storage-option region=us-east-1 \\
       --storage-option allow_http=true
   # then sync tree/ and _refs/branches/ from that location back
+  # store the branch under an ASCII name before committing (Bash syntax)
+  mv preinstalled_data/lance/unicode_branch.lance/tree/$'dev\\u1c89' \\
+      preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89
   python3 lance_build_unicode_branch.py --check preinstalled_data/lance
 
 Build it with pylance 12.0.0, the Lance the FE uses, not the writer pinned in
 lance_fixture_requirements.txt: Lance 7 stores a non-ASCII branch directory percent-encoded
 (tree/dev%E1%B2%89/), which Lance 12 does not find, while Lance 12 stores it as tree/dev\u1c89/.
-The catalog build carries the fixture over as-is. --check runs with any reader.
+The repository stores that directory as tree/dev_u1c89/ so checkout does not depend on
+filesystem Unicode support. iceberg.yaml.tpl restores the original object keys after
+uploading to MinIO. The catalog build carries the portable fixture over as-is.
+--check verifies this stored layout and runs with any reader.
 """
 import argparse
 import shutil
@@ -52,7 +58,8 @@ import lance
 import pyarrow as pa
 
 UNICODE_BRANCH_DIR = "unicode_branch.lance"
-BRANCH = "devᲉ"
+BRANCH = "dev\u1c89"
+STORED_BRANCH = "dev_u1c89"
 PARENT_URI = "s3://warehouse/lance/unicode_branch.lance"
 WRITER_VERSION = "12.0.0"
 DIMENSION = 4
@@ -102,8 +109,8 @@ def check(output: Path) -> None:
     assert [v["version"] for v in dataset.versions()] == [1], dataset.versions()
     assert row_ids(dataset) == list(range(1, 9)), "main rows differ"
     assert list(dataset.branches.list()) == [BRANCH], dataset.branches.list()
-    # The branch directory keeps the name as is; Lance 12 does not find a percent-encoded one.
-    manifests = sorted((output / "tree" / BRANCH / "_versions").glob("*.manifest"))
+    # Keep Git paths portable; the upload restores the exact name Lance 12 resolves.
+    manifests = sorted((output / "tree" / STORED_BRANCH / "_versions").glob("*.manifest"))
     assert len(manifests) == 2, f"branch {BRANCH!r} must carry versions 1 and 2: {manifests}"
     for manifest in manifests:
         assert PARENT_URI.encode() in manifest.read_bytes(), f"{manifest} must reference {PARENT_URI}"
diff --git "a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_transactions/1-1be00bd2-8443-4851-9ab8-4bdcee3ffa5e.txn" b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_transactions/1-1be00bd2-8443-4851-9ab8-4bdcee3ffa5e.txn
similarity index 100%
rename from "docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_transactions/1-1be00bd2-8443-4851-9ab8-4bdcee3ffa5e.txn"
rename to docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_transactions/1-1be00bd2-8443-4851-9ab8-4bdcee3ffa5e.txn
diff --git "a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_transactions/1-c1998c6f-6213-4b7e-ae7c-cc00cde3b7de.txn" b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_transactions/1-c1998c6f-6213-4b7e-ae7c-cc00cde3b7de.txn
similarity index 100%
rename from "docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_transactions/1-c1998c6f-6213-4b7e-ae7c-cc00cde3b7de.txn"
rename to docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_transactions/1-c1998c6f-6213-4b7e-ae7c-cc00cde3b7de.txn
diff --git "a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_versions/18446744073709551613.manifest" b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_versions/18446744073709551613.manifest
similarity index 100%
rename from "docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_versions/18446744073709551613.manifest"
rename to docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_versions/18446744073709551613.manifest
diff --git "a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_versions/18446744073709551614.manifest" b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_versions/18446744073709551614.manifest
similarity index 100%
rename from "docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_versions/18446744073709551614.manifest"
rename to docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_versions/18446744073709551614.manifest
diff --git "a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/data/01110100011000100001111040aa57401ab473a6a921298540.lance" b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/data/01110100011000100001111040aa57401ab473a6a921298540.lance
similarity index 100%
rename from "docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/data/01110100011000100001111040aa57401ab473a6a921298540.lance"
rename to docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/data/01110100011000100001111040aa57401ab473a6a921298540.lance


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org