You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by ga...@apache.org on 2026/10/10 01:34:29 UTC
(doris) branch branch-4.1 updated: branch-4.1: [refactor](lance) Remove the durable Lance index job framework (#66497) (#68781)
This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 74d81085cd0 branch-4.1: [refactor](lance) Remove the durable Lance index job framework (#66497) (#68781)
74d81085cd0 is described below
commit 74d81085cd0ec7c21d43fbcedf2cad8b23f1deda
Author: kid <lu...@outlook.com>
AuthorDate: Sat Oct 10 09:34:21 2026 +0800
branch-4.1: [refactor](lance) Remove the durable Lance index job framework (#66497) (#68781)
Per the updated direction in [#66497
(comment)](https://github.com/apache/doris/issues/66497#issuecomment-6053021847),
Lance index mutations move to synchronous SQL with bounded worker
execution. A durable asynchronous job framework is no longer part of the
initial implementation, so this change reverts it explicitly rather than
leaving it disabled or maintaining two execution paths.
### Removed
- the durable Lance index job state machine, manager, same-name fences,
unresolved-job quotas, and replay infrastructure introduced by #67235
- the job-specific parts of #67630: durable job creation and JobId
results, `SHOW LANCE INDEX JOB(S)` with its grammar and plan wiring, and
the catalog DDL guards tied to unresolved jobs
- the Flight SQL JobId result-schema special case for Lance index DDL,
so Flight advertises the same generic DDL status schema the statement
now returns
- the three `lance_index_job_max_unresolved_*` quota configuration keys,
the job-specific tests, the now-unreferenced job-locator resolver pair,
and the orphaned dataset-locator helper
### Preserved
- read-only `SHOW INDEX` (#66637) and the `lance_index_entries`
inspection TVF (#66671)
- the statically validated DDL surface (#67201);
`enable_lance_index_mutation` remains the mutation gate
- the privilege, schema, parameter, snapshot, and IF-condition
validation from #67630, rewired as pure admission preflight; a mutation
that passes the whole preflight now ends in the shared not-supported
rejection until the synchronous implementation lands
- the schema-contract and name-normalization helpers, relocated out of
the removed `job` package next to their remaining callers; the
replay-path target guard keeps slimmed direct coverage
### Journal and image compatibility
`enable_lance_index_mutation` has defaulted to `false` since #67630
merged, and no released build exposes it enabled, so no deployment can
hold persisted job records unless someone manually enabled an
undocumented, gated feature on an unreleased build. The edit-log
operation `OP_LANCE_INDEX_JOB_UPSERT` (= 500) keeps its number as a
tombstone and is never reused; its old Text-delimited JSON payload is
consumed and ignored by a minimal discard branch in the journal reader,
so edit logs from such a build stay readable without restoring any
framework code. The image chunk the old manager wrote is skipped through
the existing deprecated-module list, so images from previous builds
still load.
### Evidence
Focused FE suites (`mvn test -pl fe-common,fe-core -am
-Dtest='*Lance*,DorisFlightSqlSchemaTest'`): 453 tests run, 0 failures,
0 errors, 6 skipped (known environment skips), checkstyle included,
BUILD SUCCESS — the Flight schema suite is now in the filter (33 tests
green, with the Lance DDL cases rewritten against the generic
`StatusResult` schema) and a slimmed direct test covers the retained
replay-path target guard (`hasLanceIdentityKeyChange` and its
admission-epoch advance). Net change: 62 files, +400/−8922.
Co-authored-by: u70b3 <u7...@users.noreply.github.com>
---
.../main/java/org/apache/doris/common/Config.java | 31 +-
.../doris/common/LanceIndexConfigValidator.java | 31 +-
.../common/LanceIndexConfigValidatorTest.java | 65 +-
.../antlr4/org/apache/doris/nereids/DorisLexer.g4 | 1 -
.../antlr4/org/apache/doris/nereids/DorisParser.g4 | 3 -
.../main/java/org/apache/doris/catalog/Env.java | 25 -
.../java/org/apache/doris/common/ErrorCode.java | 3 +-
.../org/apache/doris/datasource/CatalogMgr.java | 48 +-
.../doris/datasource/lance/LanceCatalogClient.java | 6 -
.../datasource/lance/LanceExternalCatalog.java | 26 -
.../datasource/lance/LanceIndexAdmission.java | 221 +-----
.../doris/datasource/lance/LanceIndexFamilies.java | 1 -
.../lance/LanceIndexMutationValidator.java | 9 +-
.../lance/{job => }/LanceIndexNameNormalizer.java | 16 +-
.../lance/{job => }/LanceIndexSchemaContract.java | 2 +-
.../lance/LanceSchemaContractBuilder.java | 1 -
.../lance/job/LanceIndexDatasetLocator.java | 143 ----
.../datasource/lance/job/LanceIndexFenceKey.java | 92 ---
.../doris/datasource/lance/job/LanceIndexJob.java | 699 ------------------
.../lance/job/LanceIndexJobCompletionReason.java | 28 -
.../datasource/lance/job/LanceIndexJobManager.java | 790 --------------------
.../lance/job/LanceIndexJobMutationState.java | 47 --
.../lance/job/LanceIndexJobMutationType.java | 28 -
.../datasource/lance/job/LanceIndexJobQuota.java | 189 -----
.../lance/job/LanceIndexJobRefreshState.java | 41 -
.../datasource/lance/job/LanceIndexJobResult.java | 112 ---
.../lance/job/LanceIndexJobResultCode.java | 158 ----
.../lance/job/LanceIndexTerminationProof.java | 33 -
.../org/apache/doris/journal/JournalEntity.java | 14 +-
.../doris/nereids/parser/LogicalPlanBuilder.java | 20 -
.../apache/doris/nereids/trees/plans/PlanType.java | 2 -
.../trees/plans/commands/AlterTableCommand.java | 35 +-
.../plans/commands/ShowLanceIndexJobCommand.java | 174 -----
.../plans/commands/ShowLanceIndexJobsCommand.java | 378 ----------
.../trees/plans/visitor/CommandVisitor.java | 10 -
.../java/org/apache/doris/persist/EditLog.java | 15 +-
.../org/apache/doris/persist/OperationType.java | 6 +-
.../doris/persist/meta/MetaPersistMethod.java | 6 -
.../doris/persist/meta/PersistMetaModules.java | 7 +-
.../arrowflight/DorisFlightSqlProducer.java | 15 +
.../service/arrowflight/FlightSqlQuerySchema.java | 15 -
.../doris/datasource/CatalogMgrLanceGuardTest.java | 505 ++-----------
.../lance/LanceCatalogLifecycleTest.java | 20 -
.../datasource/lance/LanceIndexAdmissionTest.java | 400 ++--------
.../{job => }/LanceIndexNameNormalizerTest.java | 2 +-
.../LanceSchemaContractBuilderRealDatasetTest.java | 1 -
.../lance/LanceSchemaContractBuilderTest.java | 1 -
.../lance/job/LanceIndexDatasetLocatorTest.java | 183 -----
.../lance/job/LanceIndexJobManagerPersistTest.java | 457 ------------
.../lance/job/LanceIndexJobManagerQueryTest.java | 131 ----
.../lance/job/LanceIndexJobManagerReplayTest.java | 628 ----------------
.../lance/job/LanceIndexJobQuotaTest.java | 208 ------
.../lance/job/LanceIndexJobResultClassifyTest.java | 239 ------
.../lance/job/LanceIndexJobStateMachineTest.java | 640 ----------------
.../datasource/lance/job/LanceIndexJobTest.java | 303 --------
.../lance/job/LanceIndexJobWiringTest.java | 78 --
.../parser/ShowLanceIndexJobsParserTest.java | 183 -----
.../AlterTableCommandLanceAdmissionTest.java | 159 ++--
.../commands/ShowLanceIndexJobCommandTest.java | 565 --------------
.../commands/ShowLanceIndexJobsCommandTest.java | 821 ---------------------
.../arrowflight/DorisFlightSqlSchemaTest.java | 83 ++-
.../lance/test_lance_index_admission.groovy | 147 +---
.../lance/test_lance_index_ddl.groovy | 3 +-
63 files changed, 438 insertions(+), 8865 deletions(-)
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
index 963ce97a1a8..e5b7ca1b114 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java
@@ -4237,33 +4237,14 @@ public class Config extends ConfigBase {
public static long lance_catalog_arrow_memory_limit_bytes = 256L * 1024 * 1024;
@ConfField(mutable = true, masterOnly = true, varType = VariableAnnotation.EXPERIMENTAL, description = {
- "是否启用 Lance 外表索引变更(CREATE/CREATE OR REPLACE/DROP INDEX)的 admission。默认关闭;"
- + "启用前需确认未决 job 配额均为正值。注意:在 dispatch(后续版本)与 FORCE_RELEASE(后续版本)就绪前"
- + "开启本开关会产生不可回收的 PENDING job 并冻结对应 catalog 的身份属性变更与 DROP CATALOG。",
- "Enable admission of Lance index mutations (CREATE/CREATE OR REPLACE/DROP INDEX). "
- + "Disabled by default; unresolved-job quotas must be positive before enabling. "
- + "WARNING: enabling before dispatch and FORCE_RELEASE land in a later release creates "
- + "PENDING jobs that cannot be resolved and freezes catalog identity changes and DROP CATALOG."})
+ "是否启用 Lance 外表索引变更(CREATE/CREATE OR REPLACE/DROP INDEX)的权威校验路径。默认关闭;"
+ + "在同步执行路径就绪前,开启本开关只会让通过校验的语句收到暂不支持的错误。",
+ "Enable the authoritative validation path of Lance index mutations "
+ + "(CREATE/CREATE OR REPLACE/DROP INDEX). Disabled by default; until the synchronous "
+ + "execution path lands, enabling it only makes validated statements fail with the "
+ + "not-supported error."})
public static boolean enable_lance_index_mutation = false;
- @ConfField(mutable = true, masterOnly = true,
- callback = LanceIndexConfigValidator.PositiveLongConfigHandler.class,
- description = {"单个 Lance 数据表(locator 身份)允许的最大未决索引 job 数。",
- "Max unresolved Lance index jobs per table (locator identity)."})
- public static long lance_index_job_max_unresolved_per_table = 8;
-
- @ConfField(mutable = true, masterOnly = true,
- callback = LanceIndexConfigValidator.PositiveLongConfigHandler.class,
- description = {"单个 Lance catalog 允许的最大未决索引 job 数。",
- "Max unresolved Lance index jobs per catalog."})
- public static long lance_index_job_max_unresolved_per_catalog = 64;
-
- @ConfField(mutable = true, masterOnly = true,
- callback = LanceIndexConfigValidator.PositiveLongConfigHandler.class,
- description = {"全部 catalog 合计允许的最大未决 Lance 索引 job 数。",
- "Max unresolved Lance index jobs across all catalogs (global)."})
- public static long lance_index_job_max_unresolved_global = 256;
-
@ConfField(mutable = true, masterOnly = true,
callback = LanceIndexConfigValidator.PositiveIntConfigHandler.class,
description = {"Lance IVF_PQ 索引 num_partitions 的静态上限。",
diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/LanceIndexConfigValidator.java b/fe/fe-common/src/main/java/org/apache/doris/common/LanceIndexConfigValidator.java
index 3f14c8bb885..40e061f4817 100644
--- a/fe/fe-common/src/main/java/org/apache/doris/common/LanceIndexConfigValidator.java
+++ b/fe/fe-common/src/main/java/org/apache/doris/common/LanceIndexConfigValidator.java
@@ -20,40 +20,19 @@ package org.apache.doris.common;
import java.lang.reflect.Field;
/**
- * Validators for the Lance index admission configuration items in {@link Config}. The
- * unresolved-job quotas and the static bounds must stay positive (section 9.7 enabling
- * precondition), so every handler rejects a zero, negative, or unparsable value before
- * assigning. The handlers extend {@link ConfigBase.DefaultConfHandler} and delegate to
+ * Validators for the Lance index configuration items in {@link Config}. The static bounds
+ * must stay positive, so the handler rejects a zero, negative, or unparsable value before
+ * assigning. The handler extends {@link ConfigBase.DefaultConfHandler} and delegates to
* {@code super.handle} after validation: a bare {@code ConfHandler} implementation would
* validate without ever assigning, letting ADMIN SET FRONTEND CONFIG pass with no effect.
* Note the callback only runs on the ADMIN SET path; values loaded from fe.conf bypass it
- * ({@code ConfigBase.setFields} assigns directly), so the admission path re-asserts the
- * positive invariant where the values are consumed.
+ * ({@code ConfigBase.setFields} assigns directly), so the consuming validation path
+ * re-asserts the positive invariant where the values are used.
*/
public final class LanceIndexConfigValidator {
private LanceIndexConfigValidator() {
}
- /**
- * Accepts only a positive long: validates first, then assigns via the default handler.
- */
- public static class PositiveLongConfigHandler extends ConfigBase.DefaultConfHandler {
- @Override
- public void handle(Field field, String confVal) throws Exception {
- String trimmedVal = confVal == null ? "" : confVal.trim();
- final long value;
- try {
- value = Long.parseLong(trimmedVal);
- } catch (NumberFormatException e) {
- throw new ConfigException(field.getName() + " must be a positive long, but got: " + confVal, e);
- }
- if (value <= 0) {
- throw new ConfigException(field.getName() + " must be a positive long, but got: " + confVal);
- }
- super.handle(field, trimmedVal);
- }
- }
-
/**
* Accepts only a positive int: validates first, then assigns via the default handler.
*/
diff --git a/fe/fe-common/src/test/java/org/apache/doris/common/LanceIndexConfigValidatorTest.java b/fe/fe-common/src/test/java/org/apache/doris/common/LanceIndexConfigValidatorTest.java
index c9e0de341ac..6088abe2fc0 100644
--- a/fe/fe-common/src/test/java/org/apache/doris/common/LanceIndexConfigValidatorTest.java
+++ b/fe/fe-common/src/test/java/org/apache/doris/common/LanceIndexConfigValidatorTest.java
@@ -25,8 +25,8 @@ import java.nio.file.Files;
import java.nio.file.Path;
/**
- * Coverage for the Lance index admission configuration items in {@link Config} and the
- * positive-value validators in {@link LanceIndexConfigValidator}: reviewed defaults, the
+ * Coverage for the Lance index configuration items in {@link Config} and the positive-value
+ * validator in {@link LanceIndexConfigValidator}: reviewed defaults, the
* {@link ConfigBase.ConfField} wiring (mutable/masterOnly/callback/varType), rejection of
* zero, negative and non-numeric values, and that an accepted value is really assigned to
* the field (a bare validating handler that never assigns would let ADMIN SET pass without
@@ -37,21 +37,12 @@ public class LanceIndexConfigValidatorTest {
@Test
public void testReviewedDefaults() {
Assertions.assertFalse(Config.enable_lance_index_mutation);
- Assertions.assertEquals(8L, Config.lance_index_job_max_unresolved_per_table);
- Assertions.assertEquals(64L, Config.lance_index_job_max_unresolved_per_catalog);
- Assertions.assertEquals(256L, Config.lance_index_job_max_unresolved_global);
Assertions.assertEquals(4096, Config.lance_index_max_num_partitions);
Assertions.assertEquals(256, Config.lance_index_max_num_sub_vectors);
}
@Test
public void testConfFieldWiring() throws Exception {
- assertCallbackWiring("lance_index_job_max_unresolved_per_table", true,
- LanceIndexConfigValidator.PositiveLongConfigHandler.class);
- assertCallbackWiring("lance_index_job_max_unresolved_per_catalog", true,
- LanceIndexConfigValidator.PositiveLongConfigHandler.class);
- assertCallbackWiring("lance_index_job_max_unresolved_global", true,
- LanceIndexConfigValidator.PositiveLongConfigHandler.class);
assertCallbackWiring("lance_index_max_num_partitions", true,
LanceIndexConfigValidator.PositiveIntConfigHandler.class);
assertCallbackWiring("lance_index_max_num_sub_vectors", true,
@@ -74,48 +65,6 @@ public class LanceIndexConfigValidatorTest {
Assertions.assertEquals(callback, anno.callback(), fieldName);
}
- @Test
- public void testPositiveLongHandlerAssignsAcceptedValue() throws Exception {
- assertLongAssigns("lance_index_job_max_unresolved_per_table");
- assertLongAssigns("lance_index_job_max_unresolved_per_catalog");
- assertLongAssigns("lance_index_job_max_unresolved_global");
- }
-
- private static void assertLongAssigns(String fieldName) throws Exception {
- Field field = Config.class.getField(fieldName);
- long original = field.getLong(null);
- try {
- new LanceIndexConfigValidator.PositiveLongConfigHandler().handle(field, " 12345 ");
- Assertions.assertEquals(12345L, field.getLong(null), fieldName);
- } finally {
- field.setLong(null, original);
- }
- }
-
- @Test
- public void testPositiveLongHandlerRejectsInvalidValues() throws Exception {
- assertLongRejected("lance_index_job_max_unresolved_per_table", "0");
- assertLongRejected("lance_index_job_max_unresolved_per_catalog", "0");
- assertLongRejected("lance_index_job_max_unresolved_global", "0");
- assertLongRejected("lance_index_job_max_unresolved_per_table", "-8");
- assertLongRejected("lance_index_job_max_unresolved_per_table", "not-a-number");
- assertLongRejected("lance_index_job_max_unresolved_per_table", "");
- }
-
- private static void assertLongRejected(String fieldName, String value) throws Exception {
- Field field = Config.class.getField(fieldName);
- long original = field.getLong(null);
- try {
- ConfigException e = Assertions.assertThrows(ConfigException.class,
- () -> new LanceIndexConfigValidator.PositiveLongConfigHandler().handle(field, value),
- fieldName + " <- \"" + value + "\"");
- Assertions.assertTrue(e.getMessage().contains(fieldName), e.getMessage());
- Assertions.assertEquals(original, field.getLong(null), fieldName + " must remain unchanged");
- } finally {
- field.setLong(null, original);
- }
- }
-
@Test
public void testPositiveIntHandlerAssignsAcceptedValue() throws Exception {
assertIntAssigns("lance_index_max_num_partitions");
@@ -167,18 +116,9 @@ public class LanceIndexConfigValidatorTest {
tempFile.toFile().deleteOnExit();
config.init(tempFile.toAbsolutePath().toString());
- long originalQuota = Config.lance_index_job_max_unresolved_per_catalog;
int originalBound = Config.lance_index_max_num_partitions;
boolean originalGate = Config.enable_lance_index_mutation;
try {
- ConfigBase.setMutableConfig("lance_index_job_max_unresolved_per_catalog", "96");
- Assertions.assertEquals(96L, Config.lance_index_job_max_unresolved_per_catalog);
- Assertions.assertThrows(ConfigException.class,
- () -> ConfigBase.setMutableConfig("lance_index_job_max_unresolved_per_catalog", "0"));
- Assertions.assertThrows(ConfigException.class,
- () -> ConfigBase.setMutableConfig("lance_index_job_max_unresolved_per_catalog", "xyz"));
- Assertions.assertEquals(96L, Config.lance_index_job_max_unresolved_per_catalog);
-
ConfigBase.setMutableConfig("lance_index_max_num_partitions", "8192");
Assertions.assertEquals(8192, Config.lance_index_max_num_partitions);
Assertions.assertThrows(ConfigException.class,
@@ -188,7 +128,6 @@ public class LanceIndexConfigValidatorTest {
ConfigBase.setMutableConfig("enable_lance_index_mutation", "true");
Assertions.assertTrue(Config.enable_lance_index_mutation);
} finally {
- Config.lance_index_job_max_unresolved_per_catalog = originalQuota;
Config.lance_index_max_num_partitions = originalBound;
Config.enable_lance_index_mutation = originalGate;
}
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
index 4a762c33201..f54117a4596 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisLexer.g4
@@ -333,7 +333,6 @@ KEY: 'KEY';
KEYS: 'KEYS';
KILL: 'KILL';
LABEL: 'LABEL';
-LANCE: 'LANCE';
LARGEINT: 'LARGEINT';
LAYOUT: 'LAYOUT';
LAST: 'LAST';
diff --git a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4 b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
index 9ceaeef405f..4d97fdab5b8 100644
--- a/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
+++ b/fe/fe-core/src/main/antlr4/org/apache/doris/nereids/DorisParser.g4
@@ -497,8 +497,6 @@ supportedShowStatement
| SHOW WARM UP JOB wildWhere? #showWarmUpJob
| SHOW PYTHON VERSIONS #showPythonVersions
| SHOW PYTHON PACKAGES IN STRING_LITERAL #showPythonPackages
- | SHOW LANCE INDEX JOBS ((FROM | IN) db=multipartIdentifier)? (WHERE expression)? #showLanceIndexJobs
- | SHOW LANCE INDEX JOB jobId=INTEGER_VALUE #showLanceIndexJob
;
supportedLoadStatement
@@ -2204,7 +2202,6 @@ nonReserved
| JSON
| JSONB
| LABEL
- | LANCE
| LAST
| LDAP
| LDAP_ADMIN_PASSWORD
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
index 9eccdb1374b..05b1967bc1e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java
@@ -111,7 +111,6 @@ import org.apache.doris.datasource.hive.event.MetastoreEventsProcessor;
import org.apache.doris.datasource.iceberg.IcebergExternalTable;
import org.apache.doris.datasource.iceberg.IcebergSysExternalTable;
import org.apache.doris.datasource.jdbc.JdbcExternalTable;
-import org.apache.doris.datasource.lance.job.LanceIndexJobManager;
import org.apache.doris.datasource.paimon.PaimonExternalTable;
import org.apache.doris.datasource.paimon.PaimonSysExternalTable;
import org.apache.doris.deploy.DeployManager;
@@ -572,8 +571,6 @@ public class Env {
private InsertOverwriteManager insertOverwriteManager;
- private LanceIndexJobManager lanceIndexJobManager;
-
private DNSCache dnsCache;
private final NereidsSqlCacheManager sqlCacheManager;
@@ -859,7 +856,6 @@ public class Env {
this.mtmvService = new MTMVService();
this.eventProcessor = new EventProcessor(mtmvService);
this.insertOverwriteManager = new InsertOverwriteManager();
- this.lanceIndexJobManager = new LanceIndexJobManager();
this.dnsCache = new DNSCache();
this.sqlCacheManager = new NereidsSqlCacheManager();
this.sortedPartitionsCacheManager = new NereidsSortedPartitionsCacheManager();
@@ -985,10 +981,6 @@ public class Env {
return insertOverwriteManager;
}
- public LanceIndexJobManager getLanceIndexJobManager() {
- return lanceIndexJobManager;
- }
-
public TabletScheduler getTabletScheduler() {
return tabletScheduler;
}
@@ -1830,11 +1822,6 @@ public class Env {
insertOverwriteManager.allTaskFail();
- // A durable RUNNING Lance index job at this point may have lost its result with the
- // old master: sweep it to UNKNOWN (and refresh RUNNING back to REQUIRED) before any
- // master-only dispatcher could start.
- lanceIndexJobManager.onTransferToMaster();
-
toMasterProgress = "start daemon threads";
// coz current fe was not master fe and didn't get all fes' alive session report before, which cause
@@ -2685,18 +2672,6 @@ public class Env {
return checksum;
}
- public long loadLanceIndexJobManager(DataInputStream in, long checksum) throws IOException {
- this.lanceIndexJobManager = LanceIndexJobManager.read(in);
- LOG.info("finished replay lance index job manager from image");
- return checksum;
- }
-
- public long saveLanceIndexJobManager(CountingDataOutputStream out, long checksum) throws IOException {
- this.lanceIndexJobManager.write(out);
- LOG.info("finished save lance index job manager to image");
- return checksum;
- }
-
// Only called by checkpoint thread
// return the latest image file's absolute path
public String saveImage() throws IOException {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java b/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
index 59455962bb3..b0748cef9a3 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/ErrorCode.java
@@ -1242,7 +1242,8 @@ public enum ErrorCode {
ERR_LANCE_INDEX_MUTATION_DISABLED(5102, new byte[]{'4', '2', '0', '0', '0'},
"%s is disabled for Lance catalog tables (enable_lance_index_mutation = false)"),
- ERR_LANCE_INDEX_JOB_NOT_FOUND(5103, new byte[]{'4', '2', '0', '0', '0'}, "Lance index job not found: %s"),
+ // 5103 (ERR_LANCE_INDEX_JOB_NOT_FOUND) was removed with the durable Lance index job
+ // framework; the number stays unused so old client-visible codes keep their meaning.
ERR_NOT_CLOUD_MODE(6000, new byte[]{'4', '2', '0', '0', '0'},
"Command only support in cloud mode.");
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
index 918dac65e40..bede3d4d6a0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
@@ -92,18 +92,18 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
/**
* The Lance catalog properties whose change moves the persisted target identity
- * (provider, stable locator, or namespace mapping) out from under unresolved index
- * jobs. Credentials stay unguarded and may be rotated freely: an access key, secret
- * key or session token authenticates to the same target. Storage-routing properties
- * are part of the identity instead, because a job persists only a URI and a different
- * endpoint or region can land that same URI on a different storage service. The
- * routing entries are the canonical endpoint/region spellings of every S3-compatible
- * property family the Lance storage chain consumes: an s3 dataset reads whichever
- * S3-compatible configuration the catalog carries (LanceS3StorageProvider prefers a
- * concrete family over the generic s3 keys), while an oss dataset reads the oss keys
- * only. Other properties are not target-changing and stay unguarded. Keys match
- * case-insensitively because the catalog property chain performs no key
- * normalization.
+ * (provider, stable locator, or namespace mapping) out from under an index
+ * statement that captured that identity. Credentials stay unguarded and may be
+ * rotated freely: an access key, secret key or session token authenticates to the
+ * same target. Storage-routing properties are part of the identity instead, because
+ * the target is identified by a URI and a different endpoint or region can land that
+ * same URI on a different storage service. The routing entries are the canonical
+ * endpoint/region spellings of every S3-compatible property family the Lance storage
+ * chain consumes: an s3 dataset reads whichever S3-compatible configuration the
+ * catalog carries (LanceS3StorageProvider prefers a concrete family over the generic
+ * s3 keys), while an oss dataset reads the oss keys only. Other properties are not
+ * target-changing and stay unguarded. Keys match case-insensitively because the
+ * catalog property chain performs no key normalization.
*/
private static final Set<String> LANCE_TARGET_IDENTITY_KEYS = ImmutableSet.of(
"lance.catalog.type", "warehouse",
@@ -337,12 +337,6 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
if (catalog == null) {
throw new DdlException("No catalog found with name: " + catalogName);
}
- if (catalog instanceof LanceExternalCatalog
- && Env.getCurrentEnv().getLanceIndexJobManager().hasUnresolvedJobsForCatalog(catalog.getId())) {
- throw new DdlException("catalog '" + catalogName + "' has unresolved Lance index jobs; "
- + "they must be released via FORCE_RELEASE (available in a later release) "
- + "before dropping the catalog");
- }
CatalogLog log = new CatalogLog();
log.setCatalogId(catalog.getId());
removedCatalog = removeCatalog(log.getCatalogId());
@@ -458,13 +452,6 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
.equalsIgnoreCase(newProperties.get("type"))) {
throw new DdlException("Can't modify the type of catalog property with name: " + catalogName);
}
- if (catalog instanceof LanceExternalCatalog
- && hasLanceIdentityKeyChange(oldProperties, newProperties)
- && Env.getCurrentEnv().getLanceIndexJobManager().hasUnresolvedJobsForCatalog(catalog.getId())) {
- throw new DdlException("catalog '" + catalogName + "' has unresolved Lance index jobs; "
- + "they must be released via FORCE_RELEASE (available in a later release) "
- + "before changing target identity properties");
- }
CatalogLog log = new CatalogLog();
log.setCatalogId(catalog.getId());
Map<String, String> loggedProperties = Maps.newHashMap(newProperties);
@@ -518,10 +505,9 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
}
/**
- * Revalidate the target and transfer a prepared admission to the job manager atomically
- * with DROP CATALOG and identity ALTER. The action must contain only local job creation or a no-op;
- * all metadata loading must finish before entering this short critical section.
- * Lock order is CatalogMgr then LanceIndexJobManager, as on the catalog DDL path.
+ * Revalidate the target and run the prepared admission action atomically with DROP CATALOG
+ * and identity ALTER. The action must contain only local work or a no-op; all metadata
+ * loading must finish before entering this short critical section.
*/
public <T> T withLanceIndexAdmission(LanceExternalCatalog catalog, LanceIndexTarget expectedTarget,
LanceIndexAdmissionAction<T> action) throws Exception {
@@ -561,9 +547,9 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
* Whether the supplied properties change any Lance target identity key relative to the
* currently persisted values. A same-value rewrite is an idempotent no-op and is not a
* change; a newly supplied identity key counts as a change when the persisted value
- * differs (including when it was never set).
+ * differs (including when it was never set). Package-private for the guard test.
*/
- private static boolean hasLanceIdentityKeyChange(Map<String, String> oldProperties,
+ static boolean hasLanceIdentityKeyChange(Map<String, String> oldProperties,
Map<String, String> newProperties) {
for (Map.Entry<String, String> entry : newProperties.entrySet()) {
String key = entry.getKey();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceCatalogClient.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceCatalogClient.java
index 4cd3ed64329..563b1778e42 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceCatalogClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceCatalogClient.java
@@ -24,7 +24,6 @@ import org.apache.doris.datasource.lance.index.LanceIndexInspection;
import org.apache.doris.datasource.lance.index.LanceIndexInspectionExecutor;
import org.apache.doris.datasource.lance.index.LancePhysicalIndexEntry;
import org.apache.doris.datasource.lance.index.LanceShowIndexInfo;
-import org.apache.doris.datasource.lance.job.LanceIndexDatasetLocator;
import org.apache.doris.datasource.lance.metadata.LanceMetadataLoader;
import org.apache.doris.datasource.lance.metadata.LanceReadOptions;
import org.apache.doris.datasource.lance.metadata.LanceRefSelector;
@@ -842,11 +841,6 @@ final class LanceCatalogClient implements AutoCloseable {
(dataset, uri) -> LanceIndexInspection.readPhysicalEntries(dataset));
}
- String resolveCurrentIndexJobLocator(String dbName, String tableName) {
- return LanceIndexDatasetLocator.normalize(
- namespaceClient.resolveTableAccessUncached(dbName, tableName).getDatasetUri());
- }
-
public LanceIndexAdmissionSnapshot loadTableIndexAdmissionSnapshot(String dbName, String tableName) {
return inspectTableIndexes(dbName, tableName, LanceIndexInspection::readAdmissionSnapshot);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
index dc62cb7ff63..d1080e4e942 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceExternalCatalog.java
@@ -27,7 +27,6 @@ import org.apache.doris.datasource.InitCatalogLog;
import org.apache.doris.datasource.SessionContext;
import org.apache.doris.datasource.lance.index.LancePhysicalIndexEntry;
import org.apache.doris.datasource.lance.index.LanceShowIndexInfo;
-import org.apache.doris.datasource.lance.job.LanceIndexDatasetLocator;
import org.apache.doris.datasource.lance.metadata.LanceMetadataLoader;
import org.apache.doris.datasource.lance.metadata.LanceRefSelector;
import org.apache.doris.datasource.lance.metadata.LanceTableMetadata;
@@ -80,31 +79,6 @@ public class LanceExternalCatalog extends ExternalCatalog {
indexTargetVersion++;
}
- /**
- * Resolves the dataset the given (db, table) names currently point at, through the
- * same table-access resolution the readers use, and returns its durable locator form
- * ({@link LanceIndexDatasetLocator#normalize}).
- *
- * <p>This exists for SHOW-authorization revalidation of persisted Lance index jobs:
- * once a job reaches a terminal state and releases its guard, a legitimate catalog
- * ALTER can repoint the same db.table names at a different dataset, and the job must
- * stop being readable through table-level SHOW on the new target. Any failure - the
- * catalog is not initialized, the provider is unreachable, credentials expired, or
- * the names no longer resolve - yields {@code null}; callers must treat null as
- * "not resolved" (the orphan, ADMIN-only visibility rule), never as an authorization
- * grant.
- *
- * @return the normalized locator of the dataset the names currently point at, or
- * null when it cannot be resolved
- */
- public String resolveCurrentIndexJobLocator(String dbName, String tableName) {
- try {
- return withClient(current -> current.resolveCurrentIndexJobLocator(dbName, tableName));
- } catch (Exception e) {
- return null;
- }
- }
-
public LanceExternalCatalog(long catalogId, String name, String resource, Map<String, String> props,
String comment) {
super(catalogId, name, InitCatalogLog.Type.LANCE, comment);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexAdmission.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexAdmission.java
index b408af0dbcb..6d755fb63f2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexAdmission.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexAdmission.java
@@ -20,21 +20,12 @@ package org.apache.doris.datasource.lance;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Env;
import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.Config;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.datasource.CatalogMgr;
import org.apache.doris.datasource.lance.index.LanceIndexInspection;
import org.apache.doris.datasource.lance.index.LanceShowIndexInfo;
-import org.apache.doris.datasource.lance.job.LanceIndexDatasetLocator;
-import org.apache.doris.datasource.lance.job.LanceIndexFenceKey;
-import org.apache.doris.datasource.lance.job.LanceIndexJob;
-import org.apache.doris.datasource.lance.job.LanceIndexJobMutationType;
-import org.apache.doris.datasource.lance.job.LanceIndexNameNormalizer;
-import org.apache.doris.datasource.lance.job.LanceIndexSchemaContract;
import org.apache.doris.nereids.trees.plans.commands.info.IndexDefinition;
-import org.apache.doris.persist.gson.GsonUtils;
-import org.apache.doris.qe.ConnectContext;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
@@ -46,27 +37,25 @@ import java.util.HashMap;
import java.util.List;
import java.util.Locale;
import java.util.Map;
-import java.util.TreeMap;
import javax.annotation.Nullable;
/**
* Lance index admission (design sections 2.2 and 4.1): the single place where a statically
* validated top-level CREATE [OR REPLACE]/DROP INDEX statement against a Lance catalog table is
- * turned into a durable job. The whole flow runs against one pinned admission snapshot — the
- * IF preflight never takes a second metadata read, and no catalog/db/table metadata lock is held
- * while the snapshot loader does its JNI work (design section 5.1).
+ * checked against authoritative metadata. The whole flow runs against one pinned admission
+ * snapshot — the IF preflight never takes a second metadata read, and no catalog/db/table
+ * metadata lock is held while the snapshot loader does its JNI work (design section 5.1).
*
* <p>The step order is the correctness contract: name normalization and reserved prefix come
* first, before target capture and the snapshot read (fail cheap-first — a reserved name
* rejected at admission depth costs no remote read), then case-only collision analysis, IF
* preflight (including the two-stage {@code matches}: requested-algorithm equality plus
* physical-family corroboration), column-lookup collision analysis, schema contract from
- * the stored column name, locator
- * normalization, deterministic properties JSON, positive-quota assertion, and only then exactly
- * one id allocation and the durable {@code createJob} transfer.
- * Every rejection before {@code createJob} leaves no job, no fence, no quota charge, no journal
- * record, and no id allocation; the manager's own fence/quota rejections pass through verbatim
- * (an id burned by them is accepted — ids are never required to be contiguous).
+ * the stored column name, and deterministic properties JSON. The validation collected here is
+ * the request-validation stage the synchronous execution path builds on; until that path
+ * lands, an admitted mutation terminates with the shared not-supported rejection, while the
+ * IF no-op cases complete as genuine no-ops (nothing is created or dropped). Every rejection
+ * leaves no durable state behind.
*/
public final class LanceIndexAdmission {
@@ -79,24 +68,6 @@ public final class LanceIndexAdmission {
throws Exception;
}
- /** The admission result: the durable job id, or null for an IF no-op (no job created). */
- public static final class Outcome {
- private final Long jobId;
-
- private Outcome(Long jobId) {
- this.jobId = jobId;
- }
-
- /**
- * The admitted job id, or null when the IF preflight made the statement an immediate
- * no-op (design section 2.2: "returns an immediate no-op, without creating a job").
- */
- @Nullable
- public Long getJobId() {
- return jobId;
- }
- }
-
private static final SnapshotLoader DEFAULT_LOADER = new SnapshotLoader() {
@Override
public LanceIndexAdmissionSnapshot load(LanceExternalCatalog catalog, String dbName,
@@ -113,12 +84,12 @@ public final class LanceIndexAdmission {
* ({@link LanceIndexMutationValidator#validateCreateIndex}) must already have passed for
* {@code def}.
*/
- public static Outcome admitCreate(LanceExternalCatalog catalog, LanceExternalDatabase db,
+ public static void admitCreate(LanceExternalCatalog catalog, LanceExternalDatabase db,
LanceExternalTable table, IndexDefinition def, boolean ifNotExists) throws Exception {
- return admitCreate(DEFAULT_LOADER, catalog, db, table, def, ifNotExists);
+ admitCreate(DEFAULT_LOADER, catalog, db, table, def, ifNotExists);
}
- static Outcome admitCreate(SnapshotLoader loader, LanceExternalCatalog catalog,
+ static void admitCreate(SnapshotLoader loader, LanceExternalCatalog catalog,
LanceExternalDatabase db, LanceExternalTable table, IndexDefinition def, boolean ifNotExists)
throws Exception {
// 1. Display/normalized names and the reserved system prefix, checked before any metadata
@@ -141,57 +112,30 @@ public final class LanceIndexAdmission {
}
String storedName = LanceIndexFamilies.uniqueMatch(storedNames, normalizedName);
// 4. IF preflight (design section 2.2).
- boolean orReplace = def.isOrReplace();
- if (!orReplace && storedName != null) {
+ if (!def.isOrReplace() && storedName != null) {
if (!ifNotExists) {
rejectInvalid("index '" + displayName + "' already exists");
}
if (!matchesExistingDefinition(snapshot, storedName, def)) {
rejectInvalid("index '" + displayName + "' already exists with a different definition");
}
- return catalogMgr.withLanceIndexAdmission(catalog, target, () -> new Outcome(null));
+ catalogMgr.withLanceIndexAdmission(catalog, target, () -> null);
+ return;
}
// 5. Fail closed when the table lookup relation cannot resolve the request column
// uniquely: a dataset can hold top-level fields that differ only by case (V versus v),
// and ExternalTable.getColumn returns the first hit, so admitting would journal an
// arbitrary one of them. The pinned snapshot decides, never the cached table schema.
rejectIfAmbiguousLookupColumn(snapshot, def.getCols().get(0));
- // 6. Schema contract v1 from the stored column name (never the raw user spelling).
- String storedColumnName = storedColumnName(table, def.getCols().get(0));
- LanceIndexSchemaContract contract =
- LanceSchemaContractBuilder.build(snapshot.getTopLevelFields(), storedColumnName);
- // 7. The fence locator is the normalized dataset uri of the same pinned snapshot.
- String locator = normalizeLocator(snapshot);
- // 8. Deterministic normalized properties JSON for ANN; scalar families persist null.
- boolean ann = def.getLanceIndexType() == null;
- String indexType = ann ? annIndexType(def) : def.getLanceIndexType();
- String propertiesJson = ann ? buildAnnPropertiesJson(def) : null;
- // D7 backstop: quota values from fe.conf bypass the ADMIN SET callback, so admission
- // re-asserts positivity before any id allocation or durable transfer.
- assertPositiveQuotas();
- return catalogMgr.withLanceIndexAdmission(catalog, target, () -> {
- // 9. Exactly one id allocation, after every preflight above has passed.
- long jobId = Env.getCurrentEnv().getNextId();
- String creator = ConnectContext.get().getQualifiedUser();
- // 10. REPLACE on an existing name persists the stored display name (section 4.1) so the
- // worker locates the case-sensitive target; a fresh REPLACE keeps the user's spelling.
- String persistedDisplayName = (orReplace && storedName != null) ? storedName : displayName;
- LanceIndexJob job;
- try {
- job = new LanceIndexJob(jobId, creator, catalog.getId(), db.getFullName(), table.getName(),
- LanceIndexFenceKey.PROVIDER_DIRECTORY, locator, persistedDisplayName, normalizedName,
- orReplace ? LanceIndexJobMutationType.REPLACE : LanceIndexJobMutationType.CREATE,
- ifNotExists, false, indexType, storedColumnName, propertiesJson,
- snapshot.getDatasetVersion(), contract);
- } catch (IllegalArgumentException e) {
- throw invalidAdmission(e.getMessage());
- }
- Env.getCurrentEnv().getLanceIndexJobManager().createJob(job,
- Config.lance_index_job_max_unresolved_per_table,
- Config.lance_index_job_max_unresolved_per_catalog,
- Config.lance_index_job_max_unresolved_global);
- // 11. The job and its fence are durable once createJob returns.
- return new Outcome(jobId);
+ // 6. Schema contract v1 from the stored column name (never the raw user spelling). The
+ // build itself is the admission-depth type validation against the pinned snapshot.
+ LanceSchemaContractBuilder.build(snapshot.getTopLevelFields(), storedColumnName(table, def.getCols().get(0)));
+ // Every validation above is preserved for the synchronous execution path; until that
+ // path lands, a mutation that would be admitted terminates with the shared rejection.
+ catalogMgr.withLanceIndexAdmission(catalog, target, () -> {
+ LanceIndexMutationValidator.rejectUnsupportedOperation(
+ def.isOrReplace() ? "CREATE OR REPLACE INDEX" : "CREATE INDEX", "catalog tables");
+ return null;
});
}
@@ -199,12 +143,12 @@ public final class LanceIndexAdmission {
* Admits a top-level DROP INDEX. The static name bounds
* ({@link LanceIndexMutationValidator#validateDropIndex}) must already have passed.
*/
- public static Outcome admitDrop(LanceExternalCatalog catalog, LanceExternalDatabase db,
+ public static void admitDrop(LanceExternalCatalog catalog, LanceExternalDatabase db,
LanceExternalTable table, String indexName, boolean ifExists) throws Exception {
- return admitDrop(DEFAULT_LOADER, catalog, db, table, indexName, ifExists);
+ admitDrop(DEFAULT_LOADER, catalog, db, table, indexName, ifExists);
}
- static Outcome admitDrop(SnapshotLoader loader, LanceExternalCatalog catalog,
+ static void admitDrop(SnapshotLoader loader, LanceExternalCatalog catalog,
LanceExternalDatabase db, LanceExternalTable table, String indexName, boolean ifExists)
throws Exception {
// Fail cheap-first: the reserved prefix is rejected before target capture and the
@@ -222,31 +166,16 @@ public final class LanceIndexAdmission {
String storedName = LanceIndexFamilies.uniqueMatch(storedNames, normalizedName);
if (storedName == null) {
if (ifExists) {
- return catalogMgr.withLanceIndexAdmission(catalog, target, () -> new Outcome(null));
+ catalogMgr.withLanceIndexAdmission(catalog, target, () -> null);
+ return;
}
rejectInvalid("index '" + indexName + "' not found");
}
- String locator = normalizeLocator(snapshot);
- assertPositiveQuotas();
- return catalogMgr.withLanceIndexAdmission(catalog, target, () -> {
- long jobId = Env.getCurrentEnv().getNextId();
- String creator = ConnectContext.get().getQualifiedUser();
- // DROP only runs past the preflight with a unique match, so the stored display name is
- // always persisted (section 4.1); definition fields stay null on a DROP job record.
- LanceIndexJob job;
- try {
- job = new LanceIndexJob(jobId, creator, catalog.getId(), db.getFullName(), table.getName(),
- LanceIndexFenceKey.PROVIDER_DIRECTORY, locator, storedName, normalizedName,
- LanceIndexJobMutationType.DROP, false, ifExists, null, null, null,
- snapshot.getDatasetVersion(), null);
- } catch (IllegalArgumentException e) {
- throw invalidAdmission(e.getMessage());
- }
- Env.getCurrentEnv().getLanceIndexJobManager().createJob(job,
- Config.lance_index_job_max_unresolved_per_table,
- Config.lance_index_job_max_unresolved_per_catalog,
- Config.lance_index_job_max_unresolved_global);
- return new Outcome(jobId);
+ // The preflight above is preserved for the synchronous execution path; until that path
+ // lands, a mutation that would be admitted terminates with the shared rejection.
+ catalogMgr.withLanceIndexAdmission(catalog, target, () -> {
+ LanceIndexMutationValidator.rejectUnsupportedOperation("DROP INDEX", "catalog tables");
+ return null;
});
}
@@ -344,7 +273,7 @@ public final class LanceIndexAdmission {
JsonObject compression = exposed == null || !exposed.has("compression")
? null : exposed.getAsJsonObject("compression");
// The request side of num_bits is never "unset": the validator accepts an omitted value
- // and the persisted job pins it to 8 (section 2.4), so the preflight compares an
+ // and the effective default is 8 (section 2.4), so the preflight compares an
// effective 8 against any exposed compression.num_bits instead of skipping.
String requestNumBits = request.get("num_bits") == null ? "8" : request.get("num_bits");
return numericPropertyMatches(request.get("num_sub_vectors"), compression, "num_sub_vectors")
@@ -405,47 +334,6 @@ public final class LanceIndexAdmission {
return normalizedAnnProperties(def.getProperties()).get("index_type");
}
- /** The persisted job index type: the uppercased validated ANN index_type, else the literal. */
- private static String annIndexType(IndexDefinition def) {
- String indexType = normalizedAnnProperties(def.getProperties()).get("index_type");
- return indexType == null ? null : indexType.toUpperCase(Locale.ROOT);
- }
-
- /**
- * The deterministic params JSON for an admitted ANN job: lowercased keys in TreeMap order,
- * index_type as the uppercased original value, metric lowercased, numeric values as their
- * original strings, and num_bits pinned to 8 (design section 2.4). Bounded well under
- * {@link LanceIndexJob#MAX_PROPERTIES_JSON_BYTES} by the five-key whitelist.
- */
- private static String buildAnnPropertiesJson(IndexDefinition def) throws AnalysisException {
- Map<String, String> request = normalizedAnnProperties(def.getProperties());
- TreeMap<String, String> persisted = new TreeMap<>();
- String indexType = request.get("index_type");
- if (indexType != null) {
- persisted.put("index_type", indexType.toUpperCase(Locale.ROOT));
- }
- String metric = request.get("metric");
- if (metric != null) {
- persisted.put("metric", metric.toLowerCase(Locale.ROOT));
- }
- putIfPresent(persisted, request, "num_partitions");
- putIfPresent(persisted, request, "num_sub_vectors");
- persisted.put("num_bits", "8");
- try {
- return GsonUtils.GSON.toJson(persisted);
- } catch (IllegalArgumentException e) {
- throw invalidAdmission(e.getMessage());
- }
- }
-
- private static void putIfPresent(TreeMap<String, String> target, Map<String, String> source,
- String key) {
- String value = source.get(key);
- if (value != null) {
- target.put(key, value);
- }
- }
-
/**
* The request properties with case-folded keys. Static validation has already rejected
* unknown and duplicate (case-insensitively) keys, so a plain last-wins fold is exact here.
@@ -491,15 +379,6 @@ public final class LanceIndexAdmission {
return column.getName();
}
- private static String normalizeLocator(LanceIndexAdmissionSnapshot snapshot)
- throws AnalysisException {
- try {
- return LanceIndexDatasetLocator.normalize(snapshot.getDatasetUri());
- } catch (IllegalArgumentException e) {
- throw invalidAdmission(e.getMessage());
- }
- }
-
private static List<String> logicalIndexNames(LanceIndexAdmissionSnapshot snapshot) {
List<String> names = new ArrayList<>(snapshot.getLogicalIndexes().size());
for (LanceShowIndexInfo index : snapshot.getLogicalIndexes()) {
@@ -508,39 +387,7 @@ public final class LanceIndexAdmission {
return names;
}
- /**
- * D7: the unresolved-job quotas are a section 9.7 enablement precondition. The ADMIN SET
- * callback validates them, but fe.conf loading bypasses callbacks, so admission asserts them
- * again before allocating an id. The manager independently rejects non-positive limits
- * at the durable-transfer boundary.
- */
- private static void assertPositiveQuotas() throws AnalysisException {
- if (Config.lance_index_job_max_unresolved_per_table <= 0) {
- rejectNonPositiveQuota("lance_index_job_max_unresolved_per_table",
- Config.lance_index_job_max_unresolved_per_table);
- }
- if (Config.lance_index_job_max_unresolved_per_catalog <= 0) {
- rejectNonPositiveQuota("lance_index_job_max_unresolved_per_catalog",
- Config.lance_index_job_max_unresolved_per_catalog);
- }
- if (Config.lance_index_job_max_unresolved_global <= 0) {
- rejectNonPositiveQuota("lance_index_job_max_unresolved_global",
- Config.lance_index_job_max_unresolved_global);
- }
- }
-
- private static void rejectNonPositiveQuota(String configItem, long value) throws AnalysisException {
- ErrorReport.reportAnalysisException("%s", ErrorCode.ERR_LANCE_INDEX_MUTATION_DISABLED,
- "Lance index admission requires positive unresolved-job quotas, but " + configItem
- + " = " + value + "; fix the FE configuration before enabling "
- + "enable_lance_index_mutation");
- }
-
private static void rejectInvalid(String detail) throws AnalysisException {
ErrorReport.reportAnalysisException(ErrorCode.ERR_LANCE_INDEX_INVALID, detail);
}
-
- private static AnalysisException invalidAdmission(String detail) {
- return new AnalysisException(detail, ErrorCode.ERR_LANCE_INDEX_INVALID);
- }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexFamilies.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexFamilies.java
index 50dd6119511..88a0b8ae5f0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexFamilies.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexFamilies.java
@@ -17,7 +17,6 @@
package org.apache.doris.datasource.lance;
-import org.apache.doris.datasource.lance.job.LanceIndexNameNormalizer;
import com.google.common.collect.ImmutableSet;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMutationValidator.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMutationValidator.java
index 0d2a340d710..f21e2053994 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMutationValidator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexMutationValidator.java
@@ -26,7 +26,6 @@ import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.Config;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
-import org.apache.doris.datasource.lance.job.LanceIndexNameNormalizer;
import org.apache.doris.datasource.lance.metadata.LanceTypeConverter;
import org.apache.doris.nereids.trees.plans.commands.info.IndexDefinition;
@@ -130,9 +129,9 @@ public final class LanceIndexMutationValidator {
}
/**
- * Shared Lance index-name bounds for the CREATE and DROP paths: the name becomes the durable
- * logical identity that an admitted job and its same-name fence key are built on, so
- * null/empty names are rejected here instead of being masked as unsupported operations.
+ * Shared Lance index-name bounds for the CREATE and DROP paths: the name is the durable
+ * logical identity the admission preflight matches against authoritative Lance metadata,
+ * so null/empty names are rejected here instead of being masked as unsupported operations.
*/
private static void validateIndexName(String indexName) throws AnalysisException {
if (indexName == null || indexName.isEmpty()) {
@@ -168,7 +167,7 @@ public final class LanceIndexMutationValidator {
rejectInvalidDefinition("ANN index column item type must be float type");
}
// Keys match case-insensitively; the normalized view below is validation-local only.
- // Persisting normalized keys/values into the admitted job spec is owned by admission.
+ // Comparing the normalized request against authoritative metadata is owned by admission.
Map<String, String> lowerCaseProperties = new HashMap<>();
for (Map.Entry<String, String> entry : properties.entrySet()) {
String key = entry.getKey().toLowerCase(Locale.ROOT);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexNameNormalizer.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexNameNormalizer.java
similarity index 80%
rename from fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexNameNormalizer.java
rename to fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexNameNormalizer.java
index b467aa287f7..5635792c3e4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexNameNormalizer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexNameNormalizer.java
@@ -15,28 +15,28 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.doris.datasource.lance.job;
+package org.apache.doris.datasource.lance;
import java.nio.charset.StandardCharsets;
import java.util.Locale;
/**
* Logical index name normalization v1, the only definition: the UTF-8 result
- * of Java {@code toLowerCase(Locale.ROOT)}. Both the display name and the
- * normalized bytes are persisted on the job; Doris preserves display case,
- * rejects new case-only duplicates, and fails mutation on ambiguous external
- * case-only collisions. No normalization migration or mixed-version fence
- * protocol exists.
+ * of Java {@code toLowerCase(Locale.ROOT)}. Doris preserves display case,
+ * rejects new case-only duplicates against authoritative metadata, and fails
+ * mutation on ambiguous external case-only collisions. No normalization
+ * migration exists.
*/
public final class LanceIndexNameNormalizer {
- /** Bound on the persisted logical index name, aligned with the external string bound. */
+ /** Bound on the logical index name, aligned with the external string bound. */
public static final int MAX_INDEX_NAME_BYTES = 1024;
private LanceIndexNameNormalizer() {
}
/**
- * Normalization v1. The result is the identity bytes of the same-name fence key.
+ * Normalization v1. The result is the matching key for authoritative-metadata
+ * comparison and case-only duplicate rejection during the admission preflight.
*/
public static String normalize(String displayName) {
if (displayName == null) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexSchemaContract.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexSchemaContract.java
similarity index 99%
rename from fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexSchemaContract.java
rename to fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexSchemaContract.java
index 1175de7223d..7b1c7e6a986 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexSchemaContract.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceIndexSchemaContract.java
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.doris.datasource.lance.job;
+package org.apache.doris.datasource.lance;
import com.google.gson.annotations.SerializedName;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilder.java
index 4581a6c0efa..8b59f6525a0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilder.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilder.java
@@ -20,7 +20,6 @@ package org.apache.doris.datasource.lance;
import org.apache.doris.common.AnalysisException;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
-import org.apache.doris.datasource.lance.job.LanceIndexSchemaContract;
import org.apache.doris.datasource.lance.metadata.LanceTypeConverter;
import org.apache.arrow.vector.types.DateUnit;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexDatasetLocator.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexDatasetLocator.java
deleted file mode 100644
index bed1ae696d8..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexDatasetLocator.java
+++ /dev/null
@@ -1,143 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import java.net.URI;
-import java.net.URISyntaxException;
-import java.nio.charset.StandardCharsets;
-import java.util.Locale;
-
-/**
- * Dataset locator normalization v1 for the durable fence key. The rules, in
- * order:
- *
- * <ol>
- * <li>trim surrounding whitespace;</li>
- * <li>if a {@code scheme://} prefix is present, lowercase the scheme
- * (aligned with the {@code LanceStorageProvider.schemeOf} precedent);</li>
- * <li>a URL whose authority carries userinfo is rejected: credential-bearing
- * URLs are never identity;</li>
- * <li>a locator with a scheme but neither an authority nor a path (for
- * example {@code "s3://"}) carries no identity and is rejected; an empty
- * authority with a non-empty path ({@code "file:///x"}) stays legal;</li>
- * <li>trailing {@code '/'} characters are removed, keeping the root
- * (a scheme-less {@code "/"} stays {@code "/"});</li>
- * <li>without a scheme the locator must be an absolute path (start with
- * {@code '/'}), otherwise it is rejected.</li>
- * </ol>
- *
- * <p>The authority (host/bucket) and path keep their original case: bucket and
- * path components are case-sensitive on the providers Doris supports, and
- * normalization v1 deliberately does not define cross-alias equivalence. URI
- * aliases and external writers replacing the dataset at the same URI are
- * outside Doris serialization.
- */
-public final class LanceIndexDatasetLocator {
- private static final String SCHEME_SEPARATOR = "://";
- /** Finite bound for the stable locator persisted in every durable job record. */
- public static final int MAX_LOCATOR_BYTES = 4096;
-
- private LanceIndexDatasetLocator() {
- }
-
- /**
- * Normalize a raw dataset locator into its durable identity form.
- *
- * @throws IllegalArgumentException if the locator is null/empty/oversized,
- * is not a valid hierarchical URI or absolute path, carries
- * userinfo/query/fragment data, has neither an authority nor a path,
- * or is a scheme-less relative path
- */
- public static String normalize(String rawLocator) {
- if (rawLocator == null) {
- throw new IllegalArgumentException("dataset locator must not be null");
- }
- String locator = rawLocator.trim();
- if (locator.isEmpty()) {
- throw new IllegalArgumentException("dataset locator must not be empty");
- }
- if (locator.getBytes(StandardCharsets.UTF_8).length > MAX_LOCATOR_BYTES) {
- throw new IllegalArgumentException(
- "dataset locator exceeds " + MAX_LOCATOR_BYTES + " UTF-8 bytes");
- }
-
- // Preserve the established spelling of the scheme-less filesystem root.
- // java.net.URI rejects "//" as a network-path reference without an
- // authority, while it is a valid absolute filesystem path here.
- if (containsOnlySlashes(locator)) {
- return "/";
- }
-
- URI uri;
- try {
- uri = new URI(locator);
- } catch (URISyntaxException e) {
- // Do not include the raw locator: it may contain credentials.
- throw new IllegalArgumentException("dataset locator is not a valid URI or absolute path");
- }
- if (uri.getRawUserInfo() != null) {
- throw new IllegalArgumentException(
- "credential-bearing dataset locators are never identity (userinfo is not allowed)");
- }
- if (uri.getRawQuery() != null) {
- // Presigned object-store URLs and SAS URLs carry credentials here.
- throw new IllegalArgumentException("dataset locator query parameters are not allowed");
- }
- if (uri.getRawFragment() != null) {
- throw new IllegalArgumentException("dataset locator fragments are not allowed");
- }
-
- String scheme = uri.getScheme();
- if (scheme == null) {
- if (!locator.startsWith("/")) {
- throw new IllegalArgumentException("dataset locator without a scheme must be an absolute path");
- }
- return stripTrailingSlashes(locator, 1);
- }
- if (uri.isOpaque() || !locator.regionMatches(scheme.length(), SCHEME_SEPARATOR, 0,
- SCHEME_SEPARATOR.length())) {
- throw new IllegalArgumentException("dataset locator scheme must use hierarchical '://' syntax");
- }
- String authority = uri.getRawAuthority() == null ? "" : uri.getRawAuthority();
- String rawPath = uri.getRawPath() == null ? "" : uri.getRawPath();
- String path = stripTrailingSlashes(rawPath, 0);
- if (authority.isEmpty() && path.isEmpty()) {
- // "s3://" / "file://" carry no identity at all; "file:///x" (empty
- // authority, non-empty path) is legal and does not reach this.
- throw new IllegalArgumentException("dataset locator has neither an authority nor a path");
- }
- return scheme.toLowerCase(Locale.ROOT) + SCHEME_SEPARATOR + authority + path;
- }
-
- private static boolean containsOnlySlashes(String value) {
- for (int index = 0; index < value.length(); index++) {
- if (value.charAt(index) != '/') {
- return false;
- }
- }
- return true;
- }
-
- private static String stripTrailingSlashes(String value, int minLength) {
- int end = value.length();
- while (end > minLength && value.charAt(end - 1) == '/') {
- end--;
- }
- return value.substring(0, end);
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexFenceKey.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexFenceKey.java
deleted file mode 100644
index 19cc9efe80a..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexFenceKey.java
+++ /dev/null
@@ -1,92 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import java.util.Objects;
-
-/**
- * The durable same-name target/fence key:
- * <pre>
- * ( persisted catalog identity, provider = DIRECTORY,
- * normalized stable dataset locator, persisted normalized logical-index-name bytes )
- * </pre>
- * The display name is not part of the key; it is persisted on the job itself.
- * This class is a derived in-memory index key and is not persisted directly.
- */
-public final class LanceIndexFenceKey {
- /** Provider of every job in this delivery slice, mapped from a filesystem (Directory) Lance catalog. */
- public static final String PROVIDER_DIRECTORY = "DIRECTORY";
-
- private final long catalogId;
- private final String provider;
- private final String normalizedLocator;
- private final String normalizedIndexName;
-
- public LanceIndexFenceKey(long catalogId, String provider, String normalizedLocator, String normalizedIndexName) {
- this.catalogId = catalogId;
- this.provider = Objects.requireNonNull(provider, "provider");
- this.normalizedLocator = Objects.requireNonNull(normalizedLocator, "normalizedLocator");
- this.normalizedIndexName = Objects.requireNonNull(normalizedIndexName, "normalizedIndexName");
- }
-
- public long getCatalogId() {
- return catalogId;
- }
-
- public String getProvider() {
- return provider;
- }
-
- public String getNormalizedLocator() {
- return normalizedLocator;
- }
-
- public String getNormalizedIndexName() {
- return normalizedIndexName;
- }
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (!(o instanceof LanceIndexFenceKey)) {
- return false;
- }
- LanceIndexFenceKey that = (LanceIndexFenceKey) o;
- return catalogId == that.catalogId
- && provider.equals(that.provider)
- && normalizedLocator.equals(that.normalizedLocator)
- && normalizedIndexName.equals(that.normalizedIndexName);
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(catalogId, provider, normalizedLocator, normalizedIndexName);
- }
-
- /**
- * Deliberately omits the locator: fence-conflict messages may surface to
- * users without target privileges and must not disclose it.
- */
- @Override
- public String toString() {
- return "LanceIndexFenceKey{catalogId=" + catalogId + ", provider=" + provider
- + ", normalizedIndexName=" + normalizedIndexName + '}';
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJob.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJob.java
deleted file mode 100644
index f92f734b18f..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJob.java
+++ /dev/null
@@ -1,699 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.apache.doris.common.io.Text;
-import org.apache.doris.common.io.Writable;
-import org.apache.doris.persist.gson.GsonUtils;
-
-import com.google.gson.annotations.SerializedName;
-
-import java.io.DataInput;
-import java.io.DataOutput;
-import java.io.IOException;
-import java.nio.charset.StandardCharsets;
-import java.util.Objects;
-
-/**
- * The minimal durable Lance index job record. It contains only what the
- * lifecycle invariants require: job identity/creator/revision and bounded
- * timestamps; the persisted target identity and normalized same-name fence
- * key; the mutation intent; the admitted dataset version and schema-contract
- * representation; the mutation outcome, independent refresh state, typed
- * result, and a bounded sanitized message; the dispatch identity (selected BE,
- * BE process epoch, immutable invocation ID, deadline), possible-live
- * ownership, and any matching termination proof; and the FORCE audit fields
- * (populated only by the FORCE_RELEASE slice; here they are carried for
- * durability and replay).
- *
- * <p>The record never carries secrets, raw/opaque provider data, or unbounded
- * values: name/message/note/properties fields are length-checked at
- * construction/mutation time. Credentials are resolved at execution time from
- * the current catalog and are never persisted here.
- *
- * <p>Serialization is the standard Gson stream: {@code Text.writeString} of
- * the JSON form. Every durable field carries a short {@link SerializedName};
- * fields missing in old data keep the safe in-class defaults, so replay stays
- * tolerant of additive evolution. An instance is effectively immutable while
- * published in the manager: every durable transition is applied to a private
- * copy that is logged and then swapped in, so the object written to the edit
- * log is never mutated afterwards.
- */
-public class LanceIndexJob implements Writable {
- /** Bound on the user/build properties JSON snapshot. */
- public static final int MAX_PROPERTIES_JSON_BYTES = 4096;
- /** Bound on the FORCE_RELEASE operator note and the late-commit warning text. */
- public static final int MAX_FORCE_TEXT_BYTES = 1024;
- /** Bound on persisted creator, target-name, mutation, and FORCE-actor text. */
- public static final int MAX_DURABLE_TEXT_BYTES = 1024;
- /** Dispatch identities are UUID-like tokens, not arbitrary worker output. */
- public static final int MAX_INVOCATION_ID_BYTES = 256;
-
- // ------------------------------------------------------------------
- // Identity
- // ------------------------------------------------------------------
- @SerializedName(value = "jid")
- private long jobId;
-
- @SerializedName(value = "cr")
- private String creator;
-
- /** Bumped by +1 on every durable transition; callbacks and replay are revision-checked. */
- @SerializedName(value = "rev")
- private long revision;
-
- @SerializedName(value = "ctm")
- private long createTimeMs;
-
- @SerializedName(value = "utm")
- private long updateTimeMs;
-
- // ------------------------------------------------------------------
- // Target / fence identity
- // ------------------------------------------------------------------
- @SerializedName(value = "cid")
- private long catalogId;
-
- /** Local database name, kept for privilege and FORCE resolution when the catalog still resolves. */
- @SerializedName(value = "dbn")
- private String dbName;
-
- /** Local table name, kept for privilege and FORCE resolution when the catalog still resolves. */
- @SerializedName(value = "tbn")
- private String tableName;
-
- @SerializedName(value = "prv")
- private String provider = LanceIndexFenceKey.PROVIDER_DIRECTORY;
-
- @SerializedName(value = "loc")
- private String normalizedLocator;
-
- @SerializedName(value = "din")
- private String displayIndexName;
-
- @SerializedName(value = "nin")
- private String normalizedIndexName;
-
- // ------------------------------------------------------------------
- // Mutation intent
- // ------------------------------------------------------------------
- @SerializedName(value = "mt")
- private LanceIndexJobMutationType mutationType = LanceIndexJobMutationType.CREATE;
-
- @SerializedName(value = "ine")
- private boolean ifNotExists;
-
- @SerializedName(value = "ie")
- private boolean ifExists;
-
- /** Logical Lance algorithm (IVF_PQ / BTREE / BITMAP); required for CREATE/REPLACE, nullable for DROP. */
- @SerializedName(value = "it")
- private String indexType;
-
- @SerializedName(value = "cn")
- private String columnName;
-
- @SerializedName(value = "pj")
- private String propertiesJson;
-
- // ------------------------------------------------------------------
- // Admission snapshot
- // ------------------------------------------------------------------
- @SerializedName(value = "adv")
- private long admittedDatasetVersion;
-
- /** Nullable only for DROP, which does not revalidate an indexed-field contract. */
- @SerializedName(value = "sc")
- private LanceIndexSchemaContract schemaContract;
-
- // ------------------------------------------------------------------
- // Dual state
- // ------------------------------------------------------------------
- @SerializedName(value = "ms")
- private LanceIndexJobMutationState mutationState = LanceIndexJobMutationState.UNKNOWN;
-
- /**
- * Safe default for a corrupt record missing the key: REQUIRED holds the fence on a
- * terminal record, mirroring the null fallback in {@link #isUnresolved()}. A legal
- * record always carries the state explicitly (admission sets NOT_REQUIRED).
- */
- @SerializedName(value = "rs")
- private LanceIndexJobRefreshState refreshState = LanceIndexJobRefreshState.REQUIRED;
-
- // ------------------------------------------------------------------
- // Result
- // ------------------------------------------------------------------
- @SerializedName(value = "res")
- private LanceIndexJobResult result;
-
- // ------------------------------------------------------------------
- // Dispatch
- // ------------------------------------------------------------------
- @SerializedName(value = "bid")
- private Long backendId;
-
- @SerializedName(value = "bpe")
- private Long beProcessEpoch;
-
- /**
- * Immutable revision established by PENDING -> RUNNING. Result and
- * termination-proof callbacks use this dispatch identity rather than racing
- * on the record's global revision. Null only before dispatch or in old data.
- */
- @SerializedName(value = "drv")
- private Long dispatchRevision;
-
- /** Immutable per-dispatch UUID; callbacks must present the matching identity. */
- @SerializedName(value = "iid")
- private String invocationId;
-
- /** Bounds wait/runtime only; never proves termination and never releases a possible-live slot. */
- @SerializedName(value = "dlm")
- private Long deadlineMs;
-
- @SerializedName(value = "plo")
- private boolean possibleLiveOwned;
-
- @SerializedName(value = "tp")
- private LanceIndexTerminationProof terminationProof = LanceIndexTerminationProof.NONE;
-
- // ------------------------------------------------------------------
- // FORCE_RELEASE audit (populated only by the FORCE slice; durable + replayable here)
- // ------------------------------------------------------------------
- @SerializedName(value = "fr")
- private boolean forceReleased;
-
- @SerializedName(value = "fa")
- private String forceActor;
-
- @SerializedName(value = "ftm")
- private Long forceTimeMs;
-
- @SerializedName(value = "fn")
- private String forceNote;
-
- @SerializedName(value = "fw")
- private String forceWarning;
-
- /**
- * No-arg constructor for Gson replay only; missing fields keep the safe
- * defaults declared above (UNKNOWN mutation state holds the fence, never
- * the redispatchable PENDING; REQUIRED refresh state owes a refresh rather
- * than silently releasing the fence).
- */
- public LanceIndexJob() {
- }
-
- /**
- * Admission constructor: identity, intent, and the admission snapshot. The
- * manager initializes the lifecycle fields when the job is admitted.
- */
- public LanceIndexJob(long jobId, String creator, long catalogId, String dbName, String tableName,
- String provider, String normalizedLocator, String displayIndexName, String normalizedIndexName,
- LanceIndexJobMutationType mutationType, boolean ifNotExists, boolean ifExists, String indexType,
- String columnName, String propertiesJson, long admittedDatasetVersion,
- LanceIndexSchemaContract schemaContract) {
- this.jobId = jobId;
- this.creator = checkRequiredBytes(creator, MAX_DURABLE_TEXT_BYTES, "creator");
- this.catalogId = catalogId;
- this.dbName = checkRequiredBytes(dbName, MAX_DURABLE_TEXT_BYTES, "dbName");
- this.tableName = checkRequiredBytes(tableName, MAX_DURABLE_TEXT_BYTES, "tableName");
- this.provider = Objects.requireNonNull(provider, "provider");
- this.normalizedLocator = Objects.requireNonNull(normalizedLocator, "normalizedLocator");
- setDisplayIndexName(displayIndexName);
- setNormalizedIndexName(normalizedIndexName);
- this.mutationType = Objects.requireNonNull(mutationType, "mutationType");
- this.ifNotExists = ifNotExists;
- this.ifExists = ifExists;
- setIndexType(indexType);
- setColumnName(columnName);
- setPropertiesJson(propertiesJson);
- this.admittedDatasetVersion = admittedDatasetVersion;
- this.schemaContract = schemaContract;
- validateForAdmission();
- }
-
- /**
- * Copy used by the manager to stage a durable transition: the copy is
- * mutated, written to the edit log, and then swapped in verbatim, so a
- * published instance is never mutated after being logged. Every field is
- * carried over; the {@code result} and {@code schemaContract} references
- * are shared, which is safe because both are immutable values that are
- * only ever replaced wholesale, never mutated in place.
- */
- public LanceIndexJob(LanceIndexJob other) {
- this.jobId = other.jobId;
- this.creator = other.creator;
- this.revision = other.revision;
- this.createTimeMs = other.createTimeMs;
- this.updateTimeMs = other.updateTimeMs;
- this.catalogId = other.catalogId;
- this.dbName = other.dbName;
- this.tableName = other.tableName;
- this.provider = other.provider;
- this.normalizedLocator = other.normalizedLocator;
- this.displayIndexName = other.displayIndexName;
- this.normalizedIndexName = other.normalizedIndexName;
- this.mutationType = other.mutationType;
- this.ifNotExists = other.ifNotExists;
- this.ifExists = other.ifExists;
- this.indexType = other.indexType;
- this.columnName = other.columnName;
- this.propertiesJson = other.propertiesJson;
- this.admittedDatasetVersion = other.admittedDatasetVersion;
- this.schemaContract = other.schemaContract;
- this.mutationState = other.mutationState;
- this.refreshState = other.refreshState;
- this.result = other.result;
- this.backendId = other.backendId;
- this.beProcessEpoch = other.beProcessEpoch;
- this.dispatchRevision = other.dispatchRevision;
- this.invocationId = other.invocationId;
- this.deadlineMs = other.deadlineMs;
- this.possibleLiveOwned = other.possibleLiveOwned;
- this.terminationProof = other.terminationProof;
- this.forceReleased = other.forceReleased;
- this.forceActor = other.forceActor;
- this.forceTimeMs = other.forceTimeMs;
- this.forceNote = other.forceNote;
- this.forceWarning = other.forceWarning;
- }
-
- // ------------------------------------------------------------------
- // Derived helpers
- // ------------------------------------------------------------------
-
- /**
- * The durable same-name fence key of this job.
- */
- public LanceIndexFenceKey fenceKey() {
- return new LanceIndexFenceKey(catalogId, provider, normalizedLocator, normalizedIndexName);
- }
-
- /**
- * The per persisted table/locator quota identity.
- */
- public LanceIndexJobQuota.TableQuotaKey getTableQuotaKey() {
- return new LanceIndexJobQuota.TableQuotaKey(catalogId, normalizedLocator);
- }
-
- /**
- * Whether this job is still unresolved, i.e. must keep holding its same-name
- * fence and unresolved quota. Fence and quota are released together:
- * PENDING/RUNNING always hold; a known terminal job holds until its required
- * refresh is DONE (FAILED still holds, refresh may retry); UNKNOWN holds until
- * a durable FORCE_RELEASE. A null state from a corrupt record is treated as
- * UNKNOWN, the safe direction (never redispatched). Holding the fence itself
- * requires the fence identity to survive: the manager keeps an identity-less
- * corrupt record out of the books (its fence key cannot be reconstructed) and
- * instead blocks all new admissions fail-closed until a later durable record
- * resolves it.
- */
- public boolean isUnresolved() {
- LanceIndexJobMutationState ms = mutationState == null ? LanceIndexJobMutationState.UNKNOWN : mutationState;
- switch (ms) {
- case PENDING:
- case RUNNING:
- return true;
- case UNKNOWN:
- return !forceReleased;
- case COMMITTED:
- case NOT_COMMITTED:
- default:
- LanceIndexJobRefreshState rs =
- refreshState == null ? LanceIndexJobRefreshState.REQUIRED : refreshState;
- return rs == LanceIndexJobRefreshState.REQUIRED
- || rs == LanceIndexJobRefreshState.RUNNING
- || rs == LanceIndexJobRefreshState.FAILED;
- }
- }
-
- /**
- * Whether this job still owns a possible-live worker slot: released only
- * by a matching termination proof or a durable FORCE_RELEASE, never by a
- * deadline.
- */
- public boolean holdsPossibleLiveSlot() {
- return possibleLiveOwned
- && (terminationProof == null || terminationProof == LanceIndexTerminationProof.NONE)
- && !forceReleased;
- }
-
- /**
- * Validate the invariant-bearing and bounded fields before this record is
- * admitted. This is intentionally separate from Gson replay, which must
- * remain tolerant of old or corrupt records in the safe direction.
- */
- public void validateForAdmission() {
- checkRequiredBytes(creator, MAX_DURABLE_TEXT_BYTES, "creator");
- checkRequiredBytes(dbName, MAX_DURABLE_TEXT_BYTES, "dbName");
- checkRequiredBytes(tableName, MAX_DURABLE_TEXT_BYTES, "tableName");
- if (provider == null) {
- throw new IllegalArgumentException("lance index job provider must not be null");
- }
- if (!LanceIndexFenceKey.PROVIDER_DIRECTORY.equals(provider)) {
- throw new IllegalArgumentException("lance index job provider must be DIRECTORY");
- }
- if (normalizedLocator == null) {
- throw new IllegalArgumentException("normalized dataset locator must not be null");
- }
- String canonicalLocator = LanceIndexDatasetLocator.normalize(normalizedLocator);
- if (!canonicalLocator.equals(normalizedLocator)) {
- throw new IllegalArgumentException("dataset locator is not in canonical identity form");
- }
- LanceIndexNameNormalizer.validateDisplayName(displayIndexName);
- validateNormalizedIndexName(normalizedIndexName);
- String expectedNormalizedName = LanceIndexNameNormalizer.normalize(displayIndexName);
- if (!expectedNormalizedName.equals(normalizedIndexName)) {
- throw new IllegalArgumentException("normalized index name does not match the display name");
- }
- if (mutationType == null) {
- throw new IllegalArgumentException("mutation type must not be null");
- }
- checkBytes(indexType, MAX_DURABLE_TEXT_BYTES, "indexType");
- checkBytes(columnName, MAX_DURABLE_TEXT_BYTES, "columnName");
- checkBytes(propertiesJson, MAX_PROPERTIES_JSON_BYTES, "propertiesJson");
- checkBytes(invocationId, MAX_INVOCATION_ID_BYTES, "invocationId");
- checkBytes(forceActor, MAX_DURABLE_TEXT_BYTES, "forceActor");
- checkBytes(forceNote, MAX_FORCE_TEXT_BYTES, "forceNote");
- checkBytes(forceWarning, MAX_FORCE_TEXT_BYTES, "forceWarning");
- if (result != null) {
- checkBytes(result.getSanitizedMessage(), LanceIndexJobResult.MAX_MESSAGE_BYTES, "sanitizedMessage");
- }
- if (schemaContract != null) {
- schemaContract.validateForAdmission();
- }
- }
-
- // ------------------------------------------------------------------
- // Accessors. Setters for bounded text fields re-validate the bound.
- // ------------------------------------------------------------------
-
- public long getJobId() {
- return jobId;
- }
-
- public void setJobId(long jobId) {
- this.jobId = jobId;
- }
-
- public String getCreator() {
- return creator;
- }
-
- public void setCreator(String creator) {
- this.creator = checkBytes(creator, MAX_DURABLE_TEXT_BYTES, "creator");
- }
-
- public long getRevision() {
- return revision;
- }
-
- public void setRevision(long revision) {
- this.revision = revision;
- }
-
- public long getCreateTimeMs() {
- return createTimeMs;
- }
-
- public void setCreateTimeMs(long createTimeMs) {
- this.createTimeMs = createTimeMs;
- }
-
- public long getUpdateTimeMs() {
- return updateTimeMs;
- }
-
- public void setUpdateTimeMs(long updateTimeMs) {
- this.updateTimeMs = updateTimeMs;
- }
-
- public long getCatalogId() {
- return catalogId;
- }
-
- public String getDbName() {
- return dbName;
- }
-
- public String getTableName() {
- return tableName;
- }
-
- public String getProvider() {
- return provider;
- }
-
- public String getNormalizedLocator() {
- return normalizedLocator;
- }
-
- public String getDisplayIndexName() {
- return displayIndexName;
- }
-
- public final void setDisplayIndexName(String displayIndexName) {
- LanceIndexNameNormalizer.validateDisplayName(displayIndexName);
- this.displayIndexName = displayIndexName;
- }
-
- public String getNormalizedIndexName() {
- return normalizedIndexName;
- }
-
- public final void setNormalizedIndexName(String normalizedIndexName) {
- validateNormalizedIndexName(normalizedIndexName);
- this.normalizedIndexName = normalizedIndexName;
- }
-
- private static void validateNormalizedIndexName(String normalizedIndexName) {
- if (normalizedIndexName == null || normalizedIndexName.isEmpty()) {
- throw new IllegalArgumentException("normalized index name must not be null or empty");
- }
- if (normalizedIndexName.getBytes(StandardCharsets.UTF_8).length
- > LanceIndexNameNormalizer.MAX_INDEX_NAME_BYTES) {
- throw new IllegalArgumentException(
- "normalized index name exceeds " + LanceIndexNameNormalizer.MAX_INDEX_NAME_BYTES + " UTF-8 bytes");
- }
- }
-
- public LanceIndexJobMutationType getMutationType() {
- return mutationType;
- }
-
- public boolean isIfNotExists() {
- return ifNotExists;
- }
-
- public boolean isIfExists() {
- return ifExists;
- }
-
- public String getIndexType() {
- return indexType;
- }
-
- public void setIndexType(String indexType) {
- this.indexType = checkBytes(indexType, MAX_DURABLE_TEXT_BYTES, "indexType");
- }
-
- public String getColumnName() {
- return columnName;
- }
-
- public void setColumnName(String columnName) {
- this.columnName = checkBytes(columnName, MAX_DURABLE_TEXT_BYTES, "columnName");
- }
-
- public String getPropertiesJson() {
- return propertiesJson;
- }
-
- public final void setPropertiesJson(String propertiesJson) {
- this.propertiesJson = checkBytes(propertiesJson, MAX_PROPERTIES_JSON_BYTES, "propertiesJson");
- }
-
- public long getAdmittedDatasetVersion() {
- return admittedDatasetVersion;
- }
-
- public LanceIndexSchemaContract getSchemaContract() {
- return schemaContract;
- }
-
- public LanceIndexJobMutationState getMutationState() {
- return mutationState;
- }
-
- public void setMutationState(LanceIndexJobMutationState mutationState) {
- this.mutationState = Objects.requireNonNull(mutationState, "mutationState");
- }
-
- public LanceIndexJobRefreshState getRefreshState() {
- return refreshState;
- }
-
- public void setRefreshState(LanceIndexJobRefreshState refreshState) {
- this.refreshState = Objects.requireNonNull(refreshState, "refreshState");
- }
-
- public LanceIndexJobResult getResult() {
- return result;
- }
-
- public void setResult(LanceIndexJobResult result) {
- this.result = result;
- }
-
- public Long getBackendId() {
- return backendId;
- }
-
- public void setBackendId(Long backendId) {
- this.backendId = backendId;
- }
-
- public Long getBeProcessEpoch() {
- return beProcessEpoch;
- }
-
- public void setBeProcessEpoch(Long beProcessEpoch) {
- this.beProcessEpoch = beProcessEpoch;
- }
-
- public Long getDispatchRevision() {
- return dispatchRevision;
- }
-
- public void setDispatchRevision(Long dispatchRevision) {
- this.dispatchRevision = dispatchRevision;
- }
-
- public String getInvocationId() {
- return invocationId;
- }
-
- public void setInvocationId(String invocationId) {
- this.invocationId = checkBytes(invocationId, MAX_INVOCATION_ID_BYTES, "invocationId");
- }
-
- public Long getDeadlineMs() {
- return deadlineMs;
- }
-
- public void setDeadlineMs(Long deadlineMs) {
- this.deadlineMs = deadlineMs;
- }
-
- public boolean isPossibleLiveOwned() {
- return possibleLiveOwned;
- }
-
- public void setPossibleLiveOwned(boolean possibleLiveOwned) {
- this.possibleLiveOwned = possibleLiveOwned;
- }
-
- public LanceIndexTerminationProof getTerminationProof() {
- return terminationProof;
- }
-
- public void setTerminationProof(LanceIndexTerminationProof terminationProof) {
- this.terminationProof = Objects.requireNonNull(terminationProof, "terminationProof");
- }
-
- public boolean isForceReleased() {
- return forceReleased;
- }
-
- public void setForceReleased(boolean forceReleased) {
- this.forceReleased = forceReleased;
- }
-
- public String getForceActor() {
- return forceActor;
- }
-
- public void setForceActor(String forceActor) {
- this.forceActor = checkBytes(forceActor, MAX_DURABLE_TEXT_BYTES, "forceActor");
- }
-
- public Long getForceTimeMs() {
- return forceTimeMs;
- }
-
- public void setForceTimeMs(Long forceTimeMs) {
- this.forceTimeMs = forceTimeMs;
- }
-
- public String getForceNote() {
- return forceNote;
- }
-
- public void setForceNote(String forceNote) {
- this.forceNote = checkBytes(forceNote, MAX_FORCE_TEXT_BYTES, "forceNote");
- }
-
- public String getForceWarning() {
- return forceWarning;
- }
-
- public void setForceWarning(String forceWarning) {
- this.forceWarning = checkBytes(forceWarning, MAX_FORCE_TEXT_BYTES, "forceWarning");
- }
-
- private static String checkBytes(String value, int maxBytes, String fieldName) {
- if (value != null && value.getBytes(StandardCharsets.UTF_8).length > maxBytes) {
- throw new IllegalArgumentException(fieldName + " exceeds " + maxBytes + " UTF-8 bytes");
- }
- return value;
- }
-
- private static String checkRequiredBytes(String value, int maxBytes, String fieldName) {
- if (value == null || value.isEmpty()) {
- throw new IllegalArgumentException(fieldName + " must not be null or empty");
- }
- return checkBytes(value, maxBytes, fieldName);
- }
-
- // ------------------------------------------------------------------
- // Serialization (Gson stream style, see DropIndexPolicyLog)
- // ------------------------------------------------------------------
-
- @Override
- public void write(DataOutput out) throws IOException {
- Text.writeString(out, GsonUtils.GSON.toJson(this));
- }
-
- public static LanceIndexJob read(DataInput in) throws IOException {
- return GsonUtils.GSON.fromJson(Text.readString(in), LanceIndexJob.class);
- }
-
- /**
- * Deliberately omits the locator: job lookups must not disclose target
- * details to callers without privilege, and log lines reuse this form.
- */
- @Override
- public String toString() {
- return "LanceIndexJob{jobId=" + jobId + ", revision=" + revision + ", catalogId=" + catalogId
- + ", db=" + dbName + ", table=" + tableName + ", index=" + displayIndexName
- + ", mutationType=" + mutationType + ", mutationState=" + mutationState
- + ", refreshState=" + refreshState + ", possibleLiveSlot=" + holdsPossibleLiveSlot()
- + ", forceReleased=" + forceReleased + '}';
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobCompletionReason.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobCompletionReason.java
deleted file mode 100644
index 94bfcd32053..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobCompletionReason.java
+++ /dev/null
@@ -1,28 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-/**
- * Why a job reached its terminal state. NOOP is not a state: a typed
- * post-dispatch DROP-not-found for DROP IF EXISTS is NOT_COMMITTED with
- * completion reason IF_CONDITION_NOOP.
- */
-public enum LanceIndexJobCompletionReason {
- NONE,
- IF_CONDITION_NOOP
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobManager.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobManager.java
deleted file mode 100644
index a34c9c1a087..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobManager.java
+++ /dev/null
@@ -1,790 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.apache.doris.catalog.Env;
-import org.apache.doris.common.DdlException;
-import org.apache.doris.common.io.Text;
-import org.apache.doris.common.io.Writable;
-import org.apache.doris.persist.gson.GsonPostProcessable;
-import org.apache.doris.persist.gson.GsonUtils;
-
-import com.google.common.annotations.VisibleForTesting;
-import com.google.common.collect.Maps;
-import com.google.gson.annotations.SerializedName;
-import org.apache.commons.lang3.StringUtils;
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
-
-import java.io.DataInput;
-import java.io.DataOutput;
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.Comparator;
-import java.util.List;
-import java.util.Map;
-import java.util.NavigableSet;
-import java.util.Objects;
-import java.util.TreeSet;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.locks.ReentrantReadWriteLock;
-
-/**
- * Master-owned manager of the durable Lance index job records, the same-name
- * fences, and the three-level unresolved-job quotas. It deliberately reuses
- * neither the generic JobManager scheduling framework nor the internal
- * IndexChangeJob machinery: the external one-shot CAS, no-redispatch rule,
- * same-name fence, and possible-live ownership required here are not provided
- * by either.
- *
- * <p>Every FE keeps the same in-memory image of {@link #jobs}: the master
- * writes each durable transition to the edit log and then applies the same
- * record locally; followers apply it from replay. All transitions share one
- * write-path shape:
- * <pre>
- * writeLock -> validate (state, revision CAS, callback identity)
- * -> writeEditLog(updated copy) // fails only by System.exit
- * -> applyToMemory(updated copy) // verbatim swap + fence/quota accounting
- * </pre>
- * A published {@link LanceIndexJob} is never mutated; a transition stages a
- * copy, logs it, and swaps it in. {@link #replayUpsertJob(LanceIndexJob)} is a
- * verbatim replace with a monotonic-revision guard and performs no state
- * transformation at all: a follower tailing a live master must keep a fresh
- * RUNNING record RUNNING. The only place a durable RUNNING without a complete
- * matching terminal result becomes UNKNOWN is {@link #onTransferToMaster()},
- * the master-election sweep that runs after metadata replay and before any
- * dispatcher could start.
- *
- * <p>Fence and quota live and die together ({@link LanceIndexJob#isUnresolved()}):
- * released when a known terminal job's refresh is NOT_REQUIRED or DONE, or by
- * a durable FORCE_RELEASE; an unresolved UNKNOWN holds both across failover,
- * timeout, termination proof, and metadata observations. A replayed unresolved
- * record that lacks fence identity (provider/locator/name) cannot join the
- * books — its fence key cannot be reconstructed — so it is tracked in
- * {@link #corruptUnresolvedJobIds} and blocks all new admissions fail-closed
- * until a later durable record resolves it.
- *
- * <p>The class starts no threads, so no checkpoint-thread guard is needed in
- * the constructor; the derived fence index and quota counters are rebuilt in
- * {@link #gsonPostProcess()} after image load.
- */
-public class LanceIndexJobManager implements Writable, GsonPostProcessable {
- private static final Logger LOG = LogManager.getLogger(LanceIndexJobManager.class);
-
- @SerializedName(value = "jobs")
- private ConcurrentMap<Long, LanceIndexJob> jobs = Maps.newConcurrentMap();
-
- /**
- * Derived: fence key -> all job ids holding that fence. Legal admission creates
- * exactly one owner; retaining every owner for a corrupt collision keeps the
- * fence fail-closed when one of those jobs later settles. Identity-less corrupt
- * records are kept out of the books and tracked in {@link #corruptUnresolvedJobIds}
- * instead. Rebuilt after replay/image load.
- */
- private final Map<LanceIndexFenceKey, NavigableSet<Long>> fenceIndex = Maps.newHashMap();
-
- /**
- * Derived: ids of replayed unresolved jobs that lack fence identity (a corrupt
- * journal/image record whose fence key cannot be reconstructed). While this set
- * is non-empty, {@link #createJob} rejects every admission fail-closed: such a
- * record may still track a live or already-committed mutation under an unknown
- * name, and no new job may race it. An id leaves the set when a later durable
- * record resolves the job (a known terminal state with no pending refresh, or
- * the durable force release of a follow-up PR). Rebuilt after replay/image load.
- */
- private final NavigableSet<Long> corruptUnresolvedJobIds = new TreeSet<>();
-
- /** Derived: three-level unresolved counters, rebuilt from the unresolved jobs. */
- private final LanceIndexJobQuota quota = new LanceIndexJobQuota();
-
- private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
-
- public LanceIndexJobManager() {
- }
-
- private void readLock() {
- lock.readLock().lock();
- }
-
- private void readUnlock() {
- lock.readLock().unlock();
- }
-
- private void writeLock() {
- lock.writeLock().lock();
- }
-
- private void writeUnlock() {
- lock.writeLock().unlock();
- }
-
- /**
- * Edit-log seam: the only place this manager writes the journal. Tests
- * subclass and override to capture or swallow the record.
- */
- protected void writeEditLog(LanceIndexJob job) {
- Env.getCurrentEnv().getEditLog().logLanceIndexJob(job);
- }
-
- // ------------------------------------------------------------------
- // Master-only write paths
- // ------------------------------------------------------------------
-
- /**
- * Admit a new job: same-name fence CAS and unresolved-quota admission
- * happen inside the write lock, before anything is logged. A rejection
- * leaves no job, no fence, no quota charge, and no edit-log record.
- *
- * @throws DdlException on a fence conflict, a duplicate job id, quota overload,
- * or while corrupt unresolved records keep admission fail-closed
- */
- public void createJob(LanceIndexJob job, long tableLimit, long catalogLimit, long globalLimit)
- throws DdlException {
- Objects.requireNonNull(job, "job");
- writeLock();
- try {
- try {
- job.validateForAdmission();
- } catch (IllegalArgumentException e) {
- throw new DdlException("invalid lance index job: " + e.getMessage(), e);
- }
- if (tableLimit <= 0 || catalogLimit <= 0 || globalLimit <= 0) {
- throw new DdlException("lance index job quota limits must all be positive");
- }
- if (jobs.containsKey(job.getJobId())) {
- throw new DdlException("lance index job id already exists: " + job.getJobId());
- }
- if (!corruptUnresolvedJobIds.isEmpty()) {
- // Fail closed: a corrupt unresolved record may track a live or already-committed
- // mutation whose fence key cannot be reconstructed, so no new admission may
- // race it. The message stays bounded and discloses no target identity.
- throw new DdlException("lance index admission is fail-closed: "
- + corruptUnresolvedJobIds.size() + " unresolved job(s) lack fence identity"
- + " (corrupt journal/image record), smallest job id " + corruptUnresolvedJobIds.first()
- + "; resolve them (FORCE_RELEASE) before admitting new jobs");
- }
- NavigableSet<Long> fencingJobIds = fenceIndex.get(job.fenceKey());
- if (fencingJobIds != null && !fencingJobIds.isEmpty()) {
- // Never disclose the locator in the rejection (the caller may lack target privilege).
- throw new DdlException("lance index '" + job.getDisplayIndexName()
- + "' is fenced by unresolved job " + fencingJobIds.first()
- + "; resolve that job (FORCE_RELEASE) before reusing the name");
- }
- // Pure admission check; the charge itself happens in applyToMemory together with the fence,
- // after the record is durable. An edit-log write failure exits the process, so no rollback exists.
- if (!quota.hasCapacity(job, tableLimit, catalogLimit, globalLimit)) {
- throw new DdlException("unresolved lance index job quota exceeded for index '"
- + job.getDisplayIndexName() + "'; resolve or finish existing jobs first");
- }
- long now = System.currentTimeMillis();
- // Stage a private copy and reset every lifecycle field: the caller's object is
- // never published, and admission always starts from the same PENDING record no
- // matter what the caller left in the lifecycle fields.
- LanceIndexJob admitted = new LanceIndexJob(job);
- admitted.setMutationState(LanceIndexJobMutationState.PENDING);
- admitted.setRefreshState(LanceIndexJobRefreshState.NOT_REQUIRED);
- admitted.setRevision(0);
- admitted.setCreateTimeMs(now);
- admitted.setUpdateTimeMs(now);
- admitted.setResult(null);
- admitted.setBackendId(null);
- admitted.setBeProcessEpoch(null);
- admitted.setInvocationId(null);
- admitted.setDispatchRevision(null);
- admitted.setDeadlineMs(null);
- admitted.setPossibleLiveOwned(false);
- admitted.setTerminationProof(LanceIndexTerminationProof.NONE);
- admitted.setForceReleased(false);
- admitted.setForceActor(null);
- admitted.setForceTimeMs(null);
- admitted.setForceNote(null);
- admitted.setForceWarning(null);
- writeEditLog(admitted);
- applyToMemory(admitted);
- } finally {
- writeUnlock();
- }
- }
-
- /**
- * PENDING -> RUNNING, the durable dispatch boundary. Compare-and-set on
- * (jobId, revision): only a PENDING job at the expected revision may be
- * dispatched, which is what makes redispatch after replay impossible.
- * Records the dispatch identity (backend, BE process epoch, immutable
- * invocation id, deadline) and takes the possible-live slot.
- *
- * @return false (with a warning) on any mismatch; the caller must not send
- */
- public boolean markRunning(long jobId, long expectedRevision, long backendId, long beProcessEpoch,
- String invocationId, long deadlineMs) {
- if (StringUtils.isBlank(invocationId)) {
- // A null/blank invocation identity would silently match a null field under
- // Objects.equals in completeWithResult and defeat the stale-callback guard.
- LOG.warn("reject markRunning for lance index job {}: invocation id is null or blank", jobId);
- return false;
- }
- writeLock();
- try {
- LanceIndexJob current = jobs.get(jobId);
- if (current == null || current.getRevision() != expectedRevision
- || current.getMutationState() != LanceIndexJobMutationState.PENDING) {
- LOG.warn("reject markRunning for lance index job {}: expected revision {}, current {}",
- jobId, expectedRevision, current);
- return false;
- }
- try {
- current.validateForAdmission();
- } catch (IllegalArgumentException e) {
- LOG.warn("reject markRunning for invalid lance index job {}: {}", jobId, e.getMessage());
- return false;
- }
- if (!hasFenceIdentity(current)) {
- LOG.warn("reject markRunning for lance index job {} without a valid fence identity", jobId);
- return false;
- }
- LanceIndexJob updated = new LanceIndexJob(current);
- updated.setMutationState(LanceIndexJobMutationState.RUNNING);
- updated.setBackendId(backendId);
- updated.setBeProcessEpoch(beProcessEpoch);
- updated.setInvocationId(invocationId);
- updated.setDeadlineMs(deadlineMs);
- updated.setPossibleLiveOwned(true);
- long dispatchRevision = current.getRevision() + 1;
- updated.setRevision(dispatchRevision);
- updated.setDispatchRevision(dispatchRevision);
- updated.setUpdateTimeMs(System.currentTimeMillis());
- writeEditLog(updated);
- applyToMemory(updated);
- return true;
- } finally {
- writeUnlock();
- }
- }
-
- /**
- * RUNNING -> terminal, from a worker/supervisor result. A callback must
- * match the durable dispatch identity exactly (immutable dispatch revision, invocation
- * id, and BE process epoch); a stale callback only logs a warning and
- * changes nothing. The typed result is classified into (mutation state,
- * refresh obligation, completion reason); message text is never inspected.
- * A known terminal job whose refresh is NOT_REQUIRED releases fence and
- * quota immediately; an UNKNOWN keeps both until a durable FORCE_RELEASE.
- *
- * <p>This is also the channel for marking a job UNKNOWN on an ambiguous
- * result (result code NO_TRUSTED_RESULT), including the master-transfer
- * sweep; no separate markUnknown API exists.
- *
- * @return false (with a warning) when the callback is stale or the job is not RUNNING
- */
- public boolean completeWithResult(long jobId, long expectedDispatchRevision, String invocationId,
- Long beProcessEpoch,
- LanceIndexJobResult result) {
- Objects.requireNonNull(result, "result");
- writeLock();
- try {
- LanceIndexJob current = jobs.get(jobId);
- if (current == null || dispatchRevisionOf(current) != expectedDispatchRevision) {
- LOG.warn("reject stale lance index job callback for job {}: expected dispatch revision {}, current {}",
- jobId, expectedDispatchRevision, current);
- return false;
- }
- if (current.getMutationState() != LanceIndexJobMutationState.RUNNING) {
- LOG.warn("reject lance index job callback for job {} in state {}: only RUNNING accepts a result",
- jobId, current.getMutationState());
- return false;
- }
- if (!Objects.equals(current.getInvocationId(), invocationId)
- || !Objects.equals(current.getBeProcessEpoch(), beProcessEpoch)) {
- LOG.warn("reject stale lance index job callback for job {}: invocation/epoch mismatch, current {}",
- jobId, current);
- return false;
- }
- LanceIndexJobResultCode.Classification classification = LanceIndexJobResultCode.classify(
- current.getMutationType(), result.getResultCode(), current.isIfExists(),
- result.isExternalMetadataAdvanced());
- LanceIndexJob updated = new LanceIndexJob(current);
- if (updated.getDispatchRevision() == null) {
- // Backfill old RUNNING records before the global revision advances so
- // an independent termination proof can still identify this dispatch.
- updated.setDispatchRevision(expectedDispatchRevision);
- }
- updated.setMutationState(classification.getMutationState());
- updated.setRefreshState(classification.getRefreshState());
- updated.setResult(new LanceIndexJobResult(result.getResultCode(), classification.getCompletionReason(),
- result.getSanitizedMessage(), result.isExternalMetadataAdvanced()));
- updated.setRevision(current.getRevision() + 1);
- updated.setUpdateTimeMs(System.currentTimeMillis());
- writeEditLog(updated);
- applyToMemory(updated);
- return true;
- } finally {
- writeUnlock();
- }
- }
-
- /**
- * Refresh REQUIRED -> RUNNING, or FAILED -> RUNNING for a retry
- * through the idempotent external-table refresh path.
- */
- public boolean markRefreshRunning(long jobId, long expectedRevision) {
- return transitionRefresh(jobId, expectedRevision, LanceIndexJobRefreshState.RUNNING);
- }
-
- /**
- * Refresh RUNNING -> DONE. On a known terminal job this releases the
- * same-name fence and the unresolved quota.
- */
- public boolean markRefreshDone(long jobId, long expectedRevision) {
- return transitionRefresh(jobId, expectedRevision, LanceIndexJobRefreshState.DONE);
- }
-
- /**
- * Refresh RUNNING -> FAILED. The job keeps fence and quota; a later
- * {@link #markRefreshRunning} retries through the idempotent path.
- */
- public boolean markRefreshFailed(long jobId, long expectedRevision) {
- return transitionRefresh(jobId, expectedRevision, LanceIndexJobRefreshState.FAILED);
- }
-
- private boolean transitionRefresh(long jobId, long expectedRevision, LanceIndexJobRefreshState target) {
- writeLock();
- try {
- LanceIndexJob current = jobs.get(jobId);
- if (current == null || current.getRevision() != expectedRevision) {
- LOG.warn("reject refresh transition to {} for lance index job {}: expected revision {}, current {}",
- target, jobId, expectedRevision, current);
- return false;
- }
- if (current.getMutationState() == null || !current.getMutationState().isTerminal()) {
- LOG.warn("reject refresh transition to {} for non-terminal lance index job {} in mutation state {}",
- target, jobId, current.getMutationState());
- return false;
- }
- LanceIndexJobRefreshState from = current.getRefreshState();
- boolean legal = (target == LanceIndexJobRefreshState.RUNNING
- && (from == LanceIndexJobRefreshState.REQUIRED || from == LanceIndexJobRefreshState.FAILED))
- || (target == LanceIndexJobRefreshState.DONE && from == LanceIndexJobRefreshState.RUNNING)
- || (target == LanceIndexJobRefreshState.FAILED && from == LanceIndexJobRefreshState.RUNNING);
- if (!legal) {
- LOG.warn("reject illegal refresh transition {} -> {} for lance index job {}",
- from, target, jobId);
- return false;
- }
- LanceIndexJob updated = new LanceIndexJob(current);
- updated.setRefreshState(target);
- updated.setRevision(current.getRevision() + 1);
- updated.setUpdateTimeMs(System.currentTimeMillis());
- writeEditLog(updated);
- applyToMemory(updated);
- return true;
- } finally {
- writeUnlock();
- }
- }
-
- /**
- * Record a matching termination proof for a job that still owns a
- * possible-live slot. Backend, BE process epoch, invocation id, and immutable
- * dispatch revision must all match. This releases only the slot: it never
- * changes the mutation state and never releases the fence or quota.
- */
- public boolean recordTerminationProof(long jobId, long expectedDispatchRevision, long backendId,
- long beProcessEpoch, String invocationId, LanceIndexTerminationProof proof) {
- Objects.requireNonNull(proof, "proof");
- writeLock();
- try {
- LanceIndexJob current = jobs.get(jobId);
- if (current == null || dispatchRevisionOf(current) != expectedDispatchRevision) {
- LOG.warn("reject termination proof for lance index job {}: expected dispatch revision {}, current {}",
- jobId, expectedDispatchRevision, current);
- return false;
- }
- if (StringUtils.isBlank(invocationId)
- || !Objects.equals(current.getBackendId(), backendId)
- || !Objects.equals(current.getBeProcessEpoch(), beProcessEpoch)
- || !Objects.equals(current.getInvocationId(), invocationId)) {
- LOG.warn("reject stale termination proof for lance index job {}:"
- + " dispatch identity mismatch, current {}", jobId, current);
- return false;
- }
- if (proof == LanceIndexTerminationProof.NONE || !current.isPossibleLiveOwned()
- || current.getTerminationProof() != LanceIndexTerminationProof.NONE) {
- LOG.warn("reject termination proof {} for lance index job {}: no possible-live slot owned, current {}",
- proof, jobId, current);
- return false;
- }
- LanceIndexJob updated = new LanceIndexJob(current);
- if (updated.getDispatchRevision() == null) {
- updated.setDispatchRevision(expectedDispatchRevision);
- }
- updated.setTerminationProof(proof);
- updated.setPossibleLiveOwned(false);
- updated.setRevision(current.getRevision() + 1);
- updated.setUpdateTimeMs(System.currentTimeMillis());
- writeEditLog(updated);
- applyToMemory(updated);
- return true;
- } finally {
- writeUnlock();
- }
- }
-
- /**
- * Master-election sweep, hooked from {@code Env.transferToMaster()} after
- * metadata replay and before any master-only dispatcher could start. A
- * durable RUNNING at this point means the terminal result may have been
- * lost with the old master: the job becomes UNKNOWN through the same
- * completeWithResult channel (result code NO_TRUSTED_RESULT, fence/quota/
- * possible-live ownership retained, never redispatched), and an in-flight
- * refresh is downgraded to REQUIRED so the idempotent refresh retries.
- * Both transitions are written to the edit log so followers converge.
- */
- public void onTransferToMaster() {
- List<LanceIndexJob> snapshot;
- readLock();
- try {
- snapshot = new ArrayList<>(jobs.values());
- } finally {
- readUnlock();
- }
- for (LanceIndexJob job : snapshot) {
- if (job.getMutationState() == LanceIndexJobMutationState.RUNNING) {
- boolean completed = completeWithResult(job.getJobId(), dispatchRevisionOf(job), job.getInvocationId(),
- job.getBeProcessEpoch(),
- new LanceIndexJobResult(LanceIndexJobResultCode.NO_TRUSTED_RESULT,
- LanceIndexJobCompletionReason.NONE,
- "FE master transferred while the job was RUNNING; the result is not trusted", false));
- if (completed) {
- LOG.info("lance index job {} transitioned RUNNING -> UNKNOWN on master transfer",
- job.getJobId());
- }
- }
- if (job.getRefreshState() == LanceIndexJobRefreshState.RUNNING) {
- downgradeRunningRefresh(job);
- }
- }
- }
-
- private void downgradeRunningRefresh(LanceIndexJob sweepCandidate) {
- writeLock();
- try {
- LanceIndexJob current = jobs.get(sweepCandidate.getJobId());
- if (current == null || current.getRefreshState() != LanceIndexJobRefreshState.RUNNING) {
- return;
- }
- LanceIndexJob updated = new LanceIndexJob(current);
- updated.setRefreshState(LanceIndexJobRefreshState.REQUIRED);
- updated.setRevision(current.getRevision() + 1);
- updated.setUpdateTimeMs(System.currentTimeMillis());
- writeEditLog(updated);
- applyToMemory(updated);
- } finally {
- writeUnlock();
- }
- }
-
- // ------------------------------------------------------------------
- // Replay (all FEs)
- // ------------------------------------------------------------------
-
- /**
- * Apply one durable record from the journal: a verbatim replace of the
- * in-memory record plus derived-index accounting, with a monotonic
- * revision guard so a stale record can never roll the state back. No state
- * transformation happens here on purpose: a follower tailing a live master
- * must keep a fresh RUNNING record RUNNING. Idempotent and tolerant of
- * missing fields (Gson defaults); never throws on behalf of record content.
- */
- public void replayUpsertJob(LanceIndexJob job) {
- if (job == null) {
- LOG.warn("ignore null lance index job record");
- return;
- }
- writeLock();
- try {
- LanceIndexJob replayed = new LanceIndexJob(job);
- LanceIndexJob existing = jobs.get(replayed.getJobId());
- if (existing != null && replayed.getRevision() < existing.getRevision()) {
- LOG.warn("ignore stale lance index job record for job {}: replayed revision {} < current {}",
- replayed.getJobId(), replayed.getRevision(), existing.getRevision());
- return;
- }
- applyToMemory(replayed);
- } finally {
- writeUnlock();
- }
- }
-
- /**
- * Swap a staged record into memory and settle fence/quota accounting for
- * the replaced record. Fence and quota always move together: a record
- * holds both while {@link LanceIndexJob#isUnresolved()}, provided it
- * carries fence identity (a corrupt identity-less record stays queryable
- * but out of the books on both the charge and the release side, and is
- * tracked in {@link #corruptUnresolvedJobIds} instead). A private
- * copy is always stored so neither a replay input nor an edit-log seam
- * reference can mutate the published record. Caller holds the write lock.
- */
- private void applyToMemory(LanceIndexJob job) {
- LanceIndexJob stored = new LanceIndexJob(job);
- LanceIndexJob old = jobs.put(stored.getJobId(), stored);
- // A replaced id leaves the admission blocker no matter what replaces it: the
- // new record either resolves the job, or carries identity and joins the books,
- // or is re-added below as still identity-less and unresolved.
- corruptUnresolvedJobIds.remove(stored.getJobId());
- // Release only what the identity guard below booked: an identity-less corrupt
- // record was stored without fence/quota accounting, so keying on it would throw.
- if (old != null && old.isUnresolved() && hasFenceIdentity(old)) {
- removeFenceOwner(old);
- quota.release(old);
- }
- if (stored.isUnresolved()) {
- if (hasFenceIdentity(stored)) {
- NavigableSet<Long> owners = fenceIndex.computeIfAbsent(stored.fenceKey(), ignored -> new TreeSet<>());
- if (!owners.isEmpty() && !owners.contains(stored.getJobId())) {
- LOG.warn("fence key collision between unresolved lance index jobs {} and {};"
- + " keeping fence owner {}", owners.first(), stored.getJobId(),
- Math.min(owners.first(), stored.getJobId()));
- }
- owners.add(stored.getJobId());
- quota.charge(stored);
- } else {
- // Corrupt record tolerance: keep it queryable but out of the fence/quota
- // books (keying on it would throw), and block all new admissions
- // fail-closed until a later durable record resolves it.
- corruptUnresolvedJobIds.add(stored.getJobId());
- LOG.warn("lance index job {} lacks fence identity (provider/locator/name);"
- + " stored without fence/quota accounting and blocking all new"
- + " admissions fail-closed until resolved", stored.getJobId());
- }
- }
- }
-
- private void removeFenceOwner(LanceIndexJob job) {
- LanceIndexFenceKey fenceKey = job.fenceKey();
- NavigableSet<Long> owners = fenceIndex.get(fenceKey);
- if (owners == null) {
- return;
- }
- owners.remove(job.getJobId());
- if (owners.isEmpty()) {
- fenceIndex.remove(fenceKey);
- }
- }
-
- private static long dispatchRevisionOf(LanceIndexJob job) {
- return job.getDispatchRevision() == null ? job.getRevision() : job.getDispatchRevision();
- }
-
- private static boolean hasFenceIdentity(LanceIndexJob job) {
- return job.getProvider() != null && job.getNormalizedLocator() != null
- && job.getNormalizedIndexName() != null;
- }
-
- // ------------------------------------------------------------------
- // Queries
- // ------------------------------------------------------------------
-
- public LanceIndexJob getJob(long jobId) {
- readLock();
- try {
- LanceIndexJob job = jobs.get(jobId);
- return job == null ? null : new LanceIndexJob(job);
- } finally {
- readUnlock();
- }
- }
-
- /**
- * Returns a snapshot copy of every job, ordered by jobId. Used by SHOW LANCE INDEX JOBS.
- */
- public List<LanceIndexJob> getAllJobsSnapshot() {
- readLock();
- try {
- List<LanceIndexJob> result = new ArrayList<>();
- for (LanceIndexJob job : jobs.values()) {
- if (job != null) {
- result.add(new LanceIndexJob(job));
- }
- }
- result.sort(Comparator.comparingLong(LanceIndexJob::getJobId));
- return result;
- } finally {
- readUnlock();
- }
- }
-
- /**
- * Returns true if any unresolved job targets the given catalog. Used by catalog DDL guard (Section 4.4).
- */
- public boolean hasUnresolvedJobsForCatalog(long catalogId) {
- readLock();
- try {
- for (LanceIndexJob job : jobs.values()) {
- if (job != null && job.getCatalogId() == catalogId && job.isUnresolved()) {
- return true;
- }
- }
- return false;
- } finally {
- readUnlock();
- }
- }
-
- /**
- * All jobs still holding a fence and unresolved quota: PENDING/RUNNING,
- * unforced UNKNOWN, and known terminal jobs with unfinished refresh.
- * Unresolved jobs that lack fence identity (corrupt records) hold no fence
- * and are excluded here; they are visible through
- * {@link #getCorruptUnresolvedJobIds()} instead.
- */
- public List<LanceIndexJob> getUnresolvedJobs() {
- readLock();
- try {
- List<LanceIndexJob> result = new ArrayList<>();
- for (LanceIndexJob job : jobs.values()) {
- if (job != null && job.isUnresolved() && hasFenceIdentity(job)) {
- result.add(new LanceIndexJob(job));
- }
- }
- return result;
- } finally {
- readUnlock();
- }
- }
-
- /**
- * Terminal jobs the refresh driver must pick up: refresh REQUIRED (waiting to
- * run, including jobs downgraded by the master-transfer sweep) or FAILED
- * (waiting for a retry). Both resume through the idempotent refresh path via
- * {@link #markRefreshRunning}; a FAILED job invisible here would hold its
- * fence forever with no retry channel.
- */
- public List<LanceIndexJob> getJobsNeedingRefresh() {
- readLock();
- try {
- List<LanceIndexJob> result = new ArrayList<>();
- for (LanceIndexJob job : jobs.values()) {
- if (job != null && job.getMutationState() != null && job.getMutationState().isTerminal()
- && hasFenceIdentity(job)
- && (job.getRefreshState() == LanceIndexJobRefreshState.REQUIRED
- || job.getRefreshState() == LanceIndexJobRefreshState.FAILED)) {
- result.add(new LanceIndexJob(job));
- }
- }
- return result;
- } finally {
- readUnlock();
- }
- }
-
- public boolean isFenceHeld(LanceIndexFenceKey fenceKey) {
- readLock();
- try {
- return fenceIndex.containsKey(fenceKey);
- } finally {
- readUnlock();
- }
- }
-
- /**
- * Ids of unresolved jobs that lack fence identity (corrupt journal/image
- * records whose fence key cannot be reconstructed). While non-empty, every
- * {@link #createJob} admission is rejected fail-closed; the force-release
- * transition of a follow-up PR resolves such a job by id, without needing
- * its fence key.
- */
- public List<Long> getCorruptUnresolvedJobIds() {
- readLock();
- try {
- return new ArrayList<>(corruptUnresolvedJobIds);
- } finally {
- readUnlock();
- }
- }
-
- @VisibleForTesting
- LanceIndexJobQuota getQuota() {
- return quota;
- }
-
- @VisibleForTesting
- public int getJobCount() {
- readLock();
- try {
- return jobs.size();
- } finally {
- readUnlock();
- }
- }
-
- // ------------------------------------------------------------------
- // Image serialization (whole-object Gson, IndexPolicyMgr style)
- // ------------------------------------------------------------------
-
- @Override
- public void write(DataOutput out) throws IOException {
- // Serialize against the declared base type: GsonUtils.BLOCK_INACCESSIBLE_JAVA refuses
- // reflection on non-public runtime classes, so a subclassed manager (the test seam)
- // would otherwise serialize as "null".
- Text.writeString(out, GsonUtils.GSON.toJson(this, LanceIndexJobManager.class));
- }
-
- public static LanceIndexJobManager read(DataInput in) throws IOException {
- return GsonUtils.GSON.fromJson(Text.readString(in), LanceIndexJobManager.class);
- }
-
- /**
- * Rebuild the derived fence index and quota counters from the durable
- * jobs after Gson image load. A fence-key collision between unresolved
- * jobs (only possible on a corrupt image) retains every owner so the fence
- * remains held until all colliding jobs settle; conflict reporting still
- * uses the smaller job id. Unresolved jobs without fence identity are kept
- * out of the books and recorded in {@link #corruptUnresolvedJobIds}, which
- * keeps admission fail-closed on their behalf.
- */
- @Override
- public void gsonPostProcess() throws IOException {
- fenceIndex.clear();
- corruptUnresolvedJobIds.clear();
- List<LanceIndexJob> unresolvedJobs = new ArrayList<>();
- if (jobs == null) {
- jobs = Maps.newConcurrentMap();
- }
- for (LanceIndexJob job : jobs.values()) {
- if (job == null || !job.isUnresolved()) {
- continue;
- }
- if (!hasFenceIdentity(job)) {
- corruptUnresolvedJobIds.add(job.getJobId());
- LOG.warn("lance index job {} lacks fence identity (provider/locator/name);"
- + " excluded from fence/quota rebuild and blocking all new"
- + " admissions fail-closed until resolved", job.getJobId());
- continue;
- }
- unresolvedJobs.add(job);
- NavigableSet<Long> owners = fenceIndex.computeIfAbsent(job.fenceKey(), ignored -> new TreeSet<>());
- if (!owners.isEmpty() && !owners.contains(job.getJobId())) {
- LOG.warn("fence key collision between unresolved lance index jobs {} and {}; keeping fence owner {}",
- owners.first(), job.getJobId(), Math.min(owners.first(), job.getJobId()));
- }
- owners.add(job.getJobId());
- }
- quota.rebuild(unresolvedJobs);
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobMutationState.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobMutationState.java
deleted file mode 100644
index 9add2d7c20f..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobMutationState.java
+++ /dev/null
@@ -1,47 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-/**
- * Compact durable mutation lifecycle:
- * <pre>
- * PENDING -> RUNNING -> COMMITTED
- * -> NOT_COMMITTED
- * -> UNKNOWN
- * </pre>
- * All three outcomes are terminal. UNKNOWN has no outgoing transition:
- * metadata never changes it to a known outcome, and FORCE_RELEASE (a later
- * delivery slice) only releases the fence/quota/slot without rewriting the
- * outcome.
- */
-public enum LanceIndexJobMutationState {
- /** The request and fence are durable; no execute send may have occurred. */
- PENDING,
- /** The durable dispatch boundary has been crossed; the one-shot call may execute or may already have executed. */
- RUNNING,
- /** A complete identity-matched typed success proves this job committed. */
- COMMITTED,
- /** A complete trusted result proves this job did not commit. */
- NOT_COMMITTED,
- /** Doris cannot safely prove whether this job committed. */
- UNKNOWN;
-
- public boolean isTerminal() {
- return this == COMMITTED || this == NOT_COMMITTED || this == UNKNOWN;
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobMutationType.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobMutationType.java
deleted file mode 100644
index 4d5d3e893ff..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobMutationType.java
+++ /dev/null
@@ -1,28 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-/**
- * The one-shot Lance index mutation a durable job intends to perform.
- * REPLACE is a full same-name rebuild (CREATE OR REPLACE INDEX).
- */
-public enum LanceIndexJobMutationType {
- CREATE,
- REPLACE,
- DROP
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobQuota.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobQuota.java
deleted file mode 100644
index 5634dc6f935..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobQuota.java
+++ /dev/null
@@ -1,189 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
-
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Objects;
-
-/**
- * Three-level unresolved-job quota counters: per persisted table/locator
- * identity, per catalog, and globally. An unresolved job is any job that still
- * holds its same-name fence (PENDING/RUNNING, an unforced UNKNOWN, or a known
- * terminal job whose required refresh is not DONE); active-plus-UNKNOWN counts
- * stay bounded at every level.
- *
- * <p>This class only counts. Callers hold the manager write lock, so no
- * internal synchronization exists. Config-gated limits are resolved by the
- * admission layer and passed in as positive finite values. The counters are rebuilt from the durable jobs after
- * replay/image load and are never persisted themselves.
- */
-public class LanceIndexJobQuota {
- private static final Logger LOG = LogManager.getLogger(LanceIndexJobQuota.class);
-
- private long globalCount;
- private final Map<Long, Long> catalogCounts = new HashMap<>();
- private final Map<TableQuotaKey, Long> tableCounts = new HashMap<>();
-
- /**
- * Check every level whose limit is positive, then increment all three
- * levels. Returns false (and increments nothing) when any enforced level
- * is full: "current < limit" must hold at every level.
- *
- * <p>The manager admission path uses {@link #hasCapacity} plus
- * {@link #charge} instead: the check and the charge deliberately straddle
- * the edit-log write. This self-contained variant is reserved for direct
- * use by the later admission slice and by tests.
- */
- public boolean tryAcquire(LanceIndexJob job, long tableLimit, long catalogLimit, long globalLimit) {
- if (!hasCapacity(job, tableLimit, catalogLimit, globalLimit)) {
- return false;
- }
- charge(job);
- return true;
- }
-
- /**
- * Pure check variant of {@link #tryAcquire}: true when incrementing would
- * not exceed any positive finite limit. A non-positive limit is rejected.
- */
- public boolean hasCapacity(LanceIndexJob job, long tableLimit, long catalogLimit, long globalLimit) {
- Objects.requireNonNull(job, "job");
- if (tableLimit <= 0 || catalogLimit <= 0 || globalLimit <= 0) {
- return false;
- }
- if (globalCount >= globalLimit) {
- return false;
- }
- if (getCatalogCount(job.getCatalogId()) >= catalogLimit) {
- return false;
- }
- return getTableCount(job.getTableQuotaKey()) < tableLimit;
- }
-
- /**
- * Increment all three levels unconditionally. Used when a durable record
- * (already admitted, or replayed) is applied to memory.
- */
- public void charge(LanceIndexJob job) {
- globalCount++;
- catalogCounts.merge(job.getCatalogId(), 1L, Long::sum);
- tableCounts.merge(job.getTableQuotaKey(), 1L, Long::sum);
- }
-
- /**
- * Decrement all three levels. Underflow is clamped at zero and warned
- * about: it indicates a bookkeeping bug, not a reason to fail replay.
- */
- public void release(LanceIndexJob job) {
- if (globalCount <= 0) {
- LOG.warn("lance index job quota global underflow on release of job {}", job.getJobId());
- } else {
- globalCount--;
- }
- decrement(catalogCounts, job.getCatalogId(), job.getJobId());
- decrement(tableCounts, job.getTableQuotaKey(), job.getJobId());
- }
-
- private static <K> void decrement(Map<K, Long> counts, K key, long jobId) {
- Long current = counts.get(key);
- if (current == null || current <= 0) {
- LOG.warn("lance index job quota underflow on release of job {} at key {}", jobId, key);
- counts.remove(key);
- return;
- }
- if (current == 1L) {
- counts.remove(key);
- } else {
- counts.put(key, current - 1);
- }
- }
-
- /**
- * Reset and recount from the unresolved durable jobs (replay / image load).
- */
- public void rebuild(Iterable<LanceIndexJob> unresolvedJobs) {
- globalCount = 0;
- catalogCounts.clear();
- tableCounts.clear();
- for (LanceIndexJob job : unresolvedJobs) {
- charge(job);
- }
- }
-
- public long getGlobalCount() {
- return globalCount;
- }
-
- public long getCatalogCount(long catalogId) {
- return catalogCounts.getOrDefault(catalogId, 0L);
- }
-
- public long getTableCount(TableQuotaKey key) {
- return tableCounts.getOrDefault(key, 0L);
- }
-
- /**
- * Per persisted table/locator identity: (catalogId, normalizedLocator).
- */
- public static final class TableQuotaKey {
- private final long catalogId;
- private final String normalizedLocator;
-
- public TableQuotaKey(long catalogId, String normalizedLocator) {
- this.catalogId = catalogId;
- this.normalizedLocator = Objects.requireNonNull(normalizedLocator, "normalizedLocator");
- }
-
- public long getCatalogId() {
- return catalogId;
- }
-
- public String getNormalizedLocator() {
- return normalizedLocator;
- }
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (!(o instanceof TableQuotaKey)) {
- return false;
- }
- TableQuotaKey that = (TableQuotaKey) o;
- return catalogId == that.catalogId && normalizedLocator.equals(that.normalizedLocator);
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(catalogId, normalizedLocator);
- }
-
- /**
- * Deliberately omits the locator, like {@link LanceIndexFenceKey#toString()}.
- */
- @Override
- public String toString() {
- return "TableQuotaKey{catalogId=" + catalogId + '}';
- }
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobRefreshState.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobRefreshState.java
deleted file mode 100644
index f6ece16ac93..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobRefreshState.java
+++ /dev/null
@@ -1,41 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-/**
- * Independent metadata-refresh state of a durable Lance index job. Stored and
- * transitioned independently from the mutation state: refresh success/failure
- * only sets DONE/FAILED and never changes the mutation outcome.
- * <pre>
- * NOT_REQUIRED | REQUIRED -> RUNNING -> DONE | FAILED, FAILED -> RUNNING (retry)
- * </pre>
- * Refresh replays through the existing idempotent external-table refresh path;
- * a failed refresh may be retried and still holds the same-name fence.
- */
-public enum LanceIndexJobRefreshState {
- /** No refresh is owed (e.g. a proven pre-invocation failure with no relevant metadata change). */
- NOT_REQUIRED,
- /** A refresh of the authoritative metadata is owed before the fence may be released. */
- REQUIRED,
- /** A refresh attempt is in flight. */
- RUNNING,
- /** The required refresh finished; the fence may be released. */
- DONE,
- /** The refresh attempt failed; retry through the idempotent path is allowed. */
- FAILED
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobResult.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobResult.java
deleted file mode 100644
index 49782f275b3..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobResult.java
+++ /dev/null
@@ -1,112 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import com.google.gson.annotations.SerializedName;
-
-import java.nio.charset.StandardCharsets;
-import java.util.Objects;
-
-/**
- * The immutable typed result of the one-shot invocation, persisted on the job.
- * Contains only what is needed to classify the invocation: the saved typed
- * code, the classified completion reason, a bounded sanitized message, and
- * whether trusted pre-invocation revalidation observed relevant external
- * metadata advancement. Never carries secrets, raw provider responses, or
- * unbounded text.
- */
-public class LanceIndexJobResult {
- public static final int MAX_MESSAGE_BYTES = 1024;
-
- @SerializedName(value = "rc")
- private LanceIndexJobResultCode resultCode = LanceIndexJobResultCode.NO_TRUSTED_RESULT;
-
- @SerializedName(value = "cr")
- private LanceIndexJobCompletionReason completionReason = LanceIndexJobCompletionReason.NONE;
-
- @SerializedName(value = "msg")
- private String sanitizedMessage;
-
- @SerializedName(value = "ema")
- private boolean externalMetadataAdvanced;
-
- /**
- * No-arg constructor for Gson replay only; missing fields keep the safe
- * defaults declared above.
- */
- public LanceIndexJobResult() {
- }
-
- public LanceIndexJobResult(LanceIndexJobResultCode resultCode, LanceIndexJobCompletionReason completionReason,
- String sanitizedMessage, boolean externalMetadataAdvanced) {
- this.resultCode = Objects.requireNonNull(resultCode, "resultCode");
- this.completionReason = completionReason == null ? LanceIndexJobCompletionReason.NONE : completionReason;
- this.sanitizedMessage = checkMessageBytes(sanitizedMessage);
- this.externalMetadataAdvanced = externalMetadataAdvanced;
- }
-
- private static String checkMessageBytes(String message) {
- if (message != null && message.getBytes(StandardCharsets.UTF_8).length > MAX_MESSAGE_BYTES) {
- throw new IllegalArgumentException(
- "sanitized message exceeds " + MAX_MESSAGE_BYTES + " UTF-8 bytes");
- }
- return message;
- }
-
- public LanceIndexJobResultCode getResultCode() {
- return resultCode;
- }
-
- public LanceIndexJobCompletionReason getCompletionReason() {
- return completionReason;
- }
-
- public String getSanitizedMessage() {
- return sanitizedMessage;
- }
-
- public boolean isExternalMetadataAdvanced() {
- return externalMetadataAdvanced;
- }
-
- @Override
- public boolean equals(Object o) {
- if (this == o) {
- return true;
- }
- if (!(o instanceof LanceIndexJobResult)) {
- return false;
- }
- LanceIndexJobResult that = (LanceIndexJobResult) o;
- return externalMetadataAdvanced == that.externalMetadataAdvanced
- && resultCode == that.resultCode
- && completionReason == that.completionReason
- && Objects.equals(sanitizedMessage, that.sanitizedMessage);
- }
-
- @Override
- public int hashCode() {
- return Objects.hash(resultCode, completionReason, sanitizedMessage, externalMetadataAdvanced);
- }
-
- @Override
- public String toString() {
- return "LanceIndexJobResult{resultCode=" + resultCode + ", completionReason=" + completionReason
- + ", externalMetadataAdvanced=" + externalMetadataAdvanced + '}';
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobResultCode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobResultCode.java
deleted file mode 100644
index fad096b5087..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexJobResultCode.java
+++ /dev/null
@@ -1,158 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-/**
- * Typed result codes of the one-shot Lance index invocation. Only saved typed
- * codes are ever classified; message text is never inspected to infer an
- * outcome. Codes fall into three groups:
- *
- * <ul>
- * <li>PRE_INVOCATION_*: a complete trusted rejection before the native call
- * (dataset version / schema contract / credential / resource revalidation).
- * These always prove NOT_COMMITTED.</li>
- * <li>NATIVE_*: the saved LanceErrorCode of the single native invocation,
- * read before the consuming error message.</li>
- * <li>NO_TRUSTED_RESULT: any ambiguity after the send may have occurred
- * (EOF, signal, OOM, panic, timeout, BE loss, malformed/partial protocol,
- * identity mismatch after acceptance, or a master-transfer sweep that found
- * a durable RUNNING without a complete matching terminal result).</li>
- * </ul>
- */
-public enum LanceIndexJobResultCode {
- /** Pre-invocation: admitted dataset version / schema contract no longer matches. */
- PRE_INVOCATION_STALE_ADMISSION,
- /** Pre-invocation: the recomputed contract is not a supported contract v1. */
- PRE_INVOCATION_UNSUPPORTED_SCHEMA_CONTRACT,
- /** Pre-invocation: credentials are known to be expired. */
- PRE_INVOCATION_CREDENTIAL_EXPIRED,
- /** Pre-invocation: trusted busy / pre-FFI resource rejection. */
- PRE_INVOCATION_RESOURCE_REJECTED,
- /** LANCE_OK from the one native invocation. */
- NATIVE_OK,
- /** LANCE_ERR_COMMIT_CONFLICT: the commit lost a race; the external dataset advanced. */
- NATIVE_COMMIT_CONFLICT,
- /** LANCE_ERR_NOT_FOUND after invocation. */
- NATIVE_NOT_FOUND,
- /** LANCE_ERR_INVALID_ARGUMENT after invocation. */
- NATIVE_INVALID_ARGUMENT,
- /** LANCE_ERR_NOT_SUPPORTED after invocation. */
- NATIVE_NOT_SUPPORTED,
- /** LANCE_ERR_INDEX after invocation (also the coarse duplicate-CREATE error). */
- NATIVE_INDEX,
- /** LANCE_ERR_IO after invocation. */
- NATIVE_IO,
- /** LANCE_ERR_INTERNAL after invocation. */
- NATIVE_INTERNAL,
- /** No complete trusted result exists; commitment cannot be proven either way. */
- NO_TRUSTED_RESULT;
-
- public boolean isPreInvocation() {
- return this == PRE_INVOCATION_STALE_ADMISSION
- || this == PRE_INVOCATION_UNSUPPORTED_SCHEMA_CONTRACT
- || this == PRE_INVOCATION_CREDENTIAL_EXPIRED
- || this == PRE_INVOCATION_RESOURCE_REJECTED;
- }
-
- /**
- * Classify a complete saved result into the durable (mutationState, refreshState,
- * completionReason) triple, following the provider-result classification table:
- *
- * <ul>
- * <li>PRE_INVOCATION_* -> NOT_COMMITTED; refresh REQUIRED only when the trusted
- * pre-invocation revalidation observed relevant external metadata advancement.</li>
- * <li>NATIVE_OK -> COMMITTED + REQUIRED.</li>
- * <li>NATIVE_COMMIT_CONFLICT -> NOT_COMMITTED + REQUIRED (the external dataset
- * advanced).</li>
- * <li>NATIVE_NOT_FOUND -> CREATE/REPLACE: UNKNOWN + NOT_REQUIRED (no attribution);
- * DROP: NOT_COMMITTED + REQUIRED, with IF_CONDITION_NOOP for DROP IF EXISTS.</li>
- * <li>NATIVE_INVALID_ARGUMENT / NOT_SUPPORTED / INDEX / IO / INTERNAL ->
- * UNKNOWN + NOT_REQUIRED (no outcome inference).</li>
- * <li>NO_TRUSTED_RESULT -> UNKNOWN + NOT_REQUIRED.</li>
- * </ul>
- */
- public static Classification classify(LanceIndexJobMutationType mutationType, LanceIndexJobResultCode resultCode,
- boolean ifExists, boolean externalMetadataAdvanced) {
- if (resultCode == null) {
- resultCode = NO_TRUSTED_RESULT;
- }
- switch (resultCode) {
- case PRE_INVOCATION_STALE_ADMISSION:
- case PRE_INVOCATION_UNSUPPORTED_SCHEMA_CONTRACT:
- case PRE_INVOCATION_CREDENTIAL_EXPIRED:
- case PRE_INVOCATION_RESOURCE_REJECTED:
- return new Classification(LanceIndexJobMutationState.NOT_COMMITTED,
- externalMetadataAdvanced
- ? LanceIndexJobRefreshState.REQUIRED : LanceIndexJobRefreshState.NOT_REQUIRED,
- LanceIndexJobCompletionReason.NONE);
- case NATIVE_OK:
- return new Classification(LanceIndexJobMutationState.COMMITTED, LanceIndexJobRefreshState.REQUIRED,
- LanceIndexJobCompletionReason.NONE);
- case NATIVE_COMMIT_CONFLICT:
- return new Classification(LanceIndexJobMutationState.NOT_COMMITTED, LanceIndexJobRefreshState.REQUIRED,
- LanceIndexJobCompletionReason.NONE);
- case NATIVE_NOT_FOUND:
- if (mutationType == LanceIndexJobMutationType.DROP) {
- return new Classification(LanceIndexJobMutationState.NOT_COMMITTED,
- LanceIndexJobRefreshState.REQUIRED,
- ifExists ? LanceIndexJobCompletionReason.IF_CONDITION_NOOP
- : LanceIndexJobCompletionReason.NONE);
- }
- return new Classification(LanceIndexJobMutationState.UNKNOWN, LanceIndexJobRefreshState.NOT_REQUIRED,
- LanceIndexJobCompletionReason.NONE);
- case NATIVE_INVALID_ARGUMENT:
- case NATIVE_NOT_SUPPORTED:
- case NATIVE_INDEX:
- case NATIVE_IO:
- case NATIVE_INTERNAL:
- case NO_TRUSTED_RESULT:
- default:
- return new Classification(LanceIndexJobMutationState.UNKNOWN, LanceIndexJobRefreshState.NOT_REQUIRED,
- LanceIndexJobCompletionReason.NONE);
- }
- }
-
- /**
- * The immutable outcome of {@link #classify}: the durable mutation state, the
- * independent refresh obligation, and the completion reason.
- */
- public static final class Classification {
- private final LanceIndexJobMutationState mutationState;
- private final LanceIndexJobRefreshState refreshState;
- private final LanceIndexJobCompletionReason completionReason;
-
- private Classification(LanceIndexJobMutationState mutationState, LanceIndexJobRefreshState refreshState,
- LanceIndexJobCompletionReason completionReason) {
- this.mutationState = mutationState;
- this.refreshState = refreshState;
- this.completionReason = completionReason;
- }
-
- public LanceIndexJobMutationState getMutationState() {
- return mutationState;
- }
-
- public LanceIndexJobRefreshState getRefreshState() {
- return refreshState;
- }
-
- public LanceIndexJobCompletionReason getCompletionReason() {
- return completionReason;
- }
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexTerminationProof.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexTerminationProof.java
deleted file mode 100644
index d5de9b5cbdb..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/lance/job/LanceIndexTerminationProof.java
+++ /dev/null
@@ -1,33 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-/**
- * Evidence that releases a possible-live worker slot. Termination proof
- * releases only that slot: it neither changes an UNKNOWN outcome nor releases
- * the same-name fence. Deadlines bound wait/runtime but never prove
- * termination.
- */
-public enum LanceIndexTerminationProof {
- /** No proof; the worker may still be running. */
- NONE,
- /** The supervisor reaped the exact matching child process. */
- CHILD_REAPED,
- /** The recorded BE process epoch no longer exists (the BE process was replaced). */
- BE_PROCESS_EPOCH_GONE
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java b/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java
index d2c1952211d..10420d1230a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java
@@ -49,7 +49,6 @@ import org.apache.doris.datasource.ExternalObjectLog;
import org.apache.doris.datasource.InitCatalogLog;
import org.apache.doris.datasource.InitDatabaseLog;
import org.apache.doris.datasource.MetaIdMappingsLog;
-import org.apache.doris.datasource.lance.job.LanceIndexJob;
import org.apache.doris.ha.MasterInfo;
import org.apache.doris.indexpolicy.DropIndexPolicyLog;
import org.apache.doris.indexpolicy.IndexPolicy;
@@ -1014,11 +1013,6 @@ public class JournalEntity implements Writable {
isRead = true;
break;
}
- case OperationType.OP_LANCE_INDEX_JOB_UPSERT: {
- data = LanceIndexJob.read(in);
- isRead = true;
- break;
- }
case OperationType.OP_BEGIN_SNAPSHOT: {
data = SnapshotState.read(in);
isRead = true;
@@ -1029,6 +1023,14 @@ public class JournalEntity implements Writable {
isRead = true;
break;
}
+ case OperationType.OP_LANCE_INDEX_JOB_UPSERT: {
+ // Payload of the removed durable Lance index job framework (Text-delimited
+ // JSON). Consumed and ignored so journals from builds that manually enabled
+ // the gate stay readable; the operation code is never reused.
+ Text.readString(in);
+ isRead = true;
+ break;
+ }
default: {
IOException e = new IOException();
LOG.error("UNKNOWN Operation Type {}", opCode, e);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
index 61e4dfffc7b..2830de1f61d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java
@@ -414,8 +414,6 @@ import org.apache.doris.nereids.DorisParser.ShowIndexCharFilterContext;
import org.apache.doris.nereids.DorisParser.ShowIndexNormalizerContext;
import org.apache.doris.nereids.DorisParser.ShowIndexTokenFilterContext;
import org.apache.doris.nereids.DorisParser.ShowIndexTokenizerContext;
-import org.apache.doris.nereids.DorisParser.ShowLanceIndexJobContext;
-import org.apache.doris.nereids.DorisParser.ShowLanceIndexJobsContext;
import org.apache.doris.nereids.DorisParser.ShowLastInsertContext;
import org.apache.doris.nereids.DorisParser.ShowLoadContext;
import org.apache.doris.nereids.DorisParser.ShowLoadProfileContext;
@@ -854,8 +852,6 @@ import org.apache.doris.nereids.trees.plans.commands.ShowIndexNormalizerCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowIndexStatsCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowIndexTokenFilterCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowIndexTokenizerCommand;
-import org.apache.doris.nereids.trees.plans.commands.ShowLanceIndexJobCommand;
-import org.apache.doris.nereids.trees.plans.commands.ShowLanceIndexJobsCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowLastInsertCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowLoadCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowLoadProfileCommand;
@@ -7060,22 +7056,6 @@ public class LogicalPlanBuilder extends DorisParserBaseVisitor<Object> {
return new ShowCatalogRecycleBinCommand(whereClause);
}
- @Override
- public LogicalPlan visitShowLanceIndexJobs(ShowLanceIndexJobsContext ctx) {
- List<String> nameParts = ctx.db == null ? null : visitMultipartIdentifier(ctx.db);
- Expression whereClause = null;
- if (ctx.WHERE() != null) {
- whereClause = getExpression(ctx.expression());
- }
- return new ShowLanceIndexJobsCommand(nameParts, whereClause);
- }
-
- @Override
- public LogicalPlan visitShowLanceIndexJob(ShowLanceIndexJobContext ctx) {
- long jobId = Long.parseLong(ctx.jobId.getText());
- return new ShowLanceIndexJobCommand(jobId);
- }
-
@Override
public LogicalPlan visitShowStorageEngines(ShowStorageEnginesContext ctx) {
return new ShowStorageEnginesCommand();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java
index 776d18d1dad..683939e70c8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/PlanType.java
@@ -320,8 +320,6 @@ public enum PlanType {
SHOW_GRANTS_COMMAND,
SHOW_INDEX_COMMAND,
SHOW_INDEX_STATS_COMMAND,
- SHOW_LANCE_INDEX_JOB_COMMAND,
- SHOW_LANCE_INDEX_JOBS_COMMAND,
SHOW_LAST_INSERT_COMMAND,
SHOW_LOAD_COMMAND,
SHOW_LOAD_PROFILE_COMMAND,
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java
index d7dba9958af..b2b97ed633c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommand.java
@@ -28,7 +28,6 @@ import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.MaterializedIndex;
import org.apache.doris.catalog.OlapTable;
-import org.apache.doris.catalog.ScalarType;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
@@ -77,15 +76,11 @@ import org.apache.doris.nereids.trees.plans.commands.info.ReplacePartitionFieldO
import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
import org.apache.doris.nereids.types.DataType;
import org.apache.doris.qe.ConnectContext;
-import org.apache.doris.qe.ShowResultSet;
-import org.apache.doris.qe.ShowResultSetMetaData;
import org.apache.doris.qe.StmtExecutor;
import com.google.common.base.Preconditions;
-import java.io.IOException;
import java.util.ArrayList;
-import java.util.Collections;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
@@ -554,43 +549,21 @@ public class AlterTableCommand extends Command implements ForwardWithSync {
Preconditions.checkState(lanceIndexOps.size() == 1,
"a top-level Lance index statement must carry exactly one operation");
AlterTableOp op = lanceIndexOps.get(0);
- LanceIndexAdmission.Outcome outcome;
if (op instanceof CreateIndexOp) {
IndexDefinition indexDef = ((CreateIndexOp) op).getIndexDef();
- outcome = LanceIndexAdmission.admitCreate((LanceExternalCatalog) catalog,
+ LanceIndexAdmission.admitCreate((LanceExternalCatalog) catalog,
(LanceExternalDatabase) dbIf, (LanceExternalTable) tableIf, indexDef,
indexDef.isIfNotExists());
} else {
DropIndexOp dropIndexOp = (DropIndexOp) op;
- outcome = LanceIndexAdmission.admitDrop((LanceExternalCatalog) catalog,
+ LanceIndexAdmission.admitDrop((LanceExternalCatalog) catalog,
(LanceExternalDatabase) dbIf, (LanceExternalTable) tableIf,
dropIndexOp.getIndexName(), dropIndexOp.isSetIfExists());
}
- sendJobIdResult(executor, outcome);
+ // An IF preflight no-op returns normally and completes with the default OK packet;
+ // any mutation that would be admitted rejects inside admission.
return;
}
ctx.getEnv().alterTable(this);
}
-
- /**
- * Answers the admission with a single-column JobId result set (WarmUpClusterCommand
- * pattern): one row carrying the durable job id, or zero rows for an IF no-op — "no job was
- * created" is self-evident from the empty set, and sendResultSet's trailing setEof()
- * suppresses the default OK packet on both the direct and the forwarded (proxy) link.
- */
- private void sendJobIdResult(StmtExecutor executor, LanceIndexAdmission.Outcome outcome)
- throws IOException {
- ShowResultSetMetaData.Builder builder = ShowResultSetMetaData.builder();
- builder.addColumn(new Column("JobId", ScalarType.createVarchar(30)));
- List<List<String>> rows = new ArrayList<>();
- if (outcome.getJobId() != null) {
- rows.add(Collections.singletonList(String.valueOf(outcome.getJobId())));
- }
- ShowResultSet resultSet = new ShowResultSet(builder.build(), rows);
- if (executor.isProxy()) {
- executor.setProxyShowResultSet(resultSet);
- return;
- }
- executor.sendResultSet(resultSet);
- }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobCommand.java
deleted file mode 100644
index fe74170c116..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobCommand.java
+++ /dev/null
@@ -1,174 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.nereids.trees.plans.commands;
-
-import org.apache.doris.analysis.RedirectStatus;
-import org.apache.doris.catalog.Column;
-import org.apache.doris.catalog.DatabaseIf;
-import org.apache.doris.catalog.Env;
-import org.apache.doris.catalog.ScalarType;
-import org.apache.doris.catalog.TableIf;
-import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.ErrorCode;
-import org.apache.doris.common.util.TimeUtils;
-import org.apache.doris.datasource.CatalogIf;
-import org.apache.doris.datasource.lance.job.LanceIndexJob;
-import org.apache.doris.datasource.lance.job.LanceIndexJobResult;
-import org.apache.doris.nereids.trees.plans.PlanType;
-import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
-import org.apache.doris.qe.ConnectContext;
-import org.apache.doris.qe.ShowResultSet;
-import org.apache.doris.qe.ShowResultSetMetaData;
-import org.apache.doris.qe.StmtExecutor;
-
-import com.google.common.collect.ImmutableList;
-
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-
-/**
- * SHOW LANCE INDEX JOB <jobId>.
- *
- * <p>Non-disclosing job detail (v5.1 section 8): the record is loaded first and authorized
- * against its persisted target before any field is returned. Orphan and half-orphan targets
- * (catalog gone, persisted db/table no longer resolvable, the resolution failing outright
- * because the provider is unreachable, or the table repointed at a different dataset
- * locator) require global ADMIN; a resolvable, still-matching target requires table-level
- * SHOW. A missing job and an unauthorized job share
- * the same fixed ERR_LANCE_INDEX_JOB_NOT_FOUND response that names only the job id, so the
- * existence and target of a job are never disclosed. The job locator, provider, normalized
- * names, propertiesJson and schema contract contents are never shown.
- */
-public class ShowLanceIndexJobCommand extends ShowCommand {
- public static final ImmutableList<String> TITLE_NAMES = new ImmutableList.Builder<String>()
- .addAll(ShowLanceIndexJobsCommand.TITLE_NAMES)
- .add("Creator")
- .add("ResultCode")
- .add("CompletionReason")
- .add("ExternalMetadataAdvanced")
- .add("BackendId")
- .add("BeProcessEpoch")
- .add("InvocationId")
- .add("DeadlineMs")
- .add("TerminationProof")
- .add("AdmittedDatasetVersion")
- .add("SchemaContractVersion")
- .add("Revision")
- .add("ForceNote")
- .add("ForceWarning")
- .add("IfNotExists")
- .add("IfExists")
- .add("IndexType")
- .add("ColumnName")
- .build();
-
- private final long jobId;
-
- public ShowLanceIndexJobCommand(long jobId) {
- super(PlanType.SHOW_LANCE_INDEX_JOB_COMMAND);
- this.jobId = jobId;
- }
-
- public long getJobId() {
- return jobId;
- }
-
- @Override
- public ShowResultSetMetaData getMetaData() {
- ShowResultSetMetaData.Builder builder = ShowResultSetMetaData.builder();
- for (String title : TITLE_NAMES) {
- builder.addColumn(new Column(title, ScalarType.createVarchar(30)));
- }
- return builder.build();
- }
-
- @Override
- public ShowResultSet doRun(ConnectContext ctx, StmtExecutor executor) throws Exception {
- LanceIndexJob job = Env.getCurrentEnv().getLanceIndexJobManager().getJob(jobId);
- if (job == null) {
- throw notFound();
- }
- CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog =
- Env.getCurrentEnv().getCatalogMgr().getCatalog(job.getCatalogId());
- // Missing and unauthorized jobs share the same fixed response; authorize before
- // returning any field.
- if (!ShowLanceIndexJobsCommand.isAuthorized(ctx, catalog, job)) {
- throw notFound();
- }
- return new ShowResultSet(getMetaData(), Collections.singletonList(renderRow(job, catalog)));
- }
-
- private AnalysisException notFound() {
- return new AnalysisException(ErrorCode.ERR_LANCE_INDEX_JOB_NOT_FOUND.formatErrorMsg(jobId),
- ErrorCode.ERR_LANCE_INDEX_JOB_NOT_FOUND);
- }
-
- private static List<String> renderRow(LanceIndexJob job,
- CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog) {
- List<String> row = new ArrayList<>(TITLE_NAMES.size());
- row.add(String.valueOf(job.getJobId()));
- // See ShowLanceIndexJobsCommand: the catalog name is not persisted on the job record,
- // so an orphan row renders it empty rather than fabricating one from the internal id.
- row.add(catalog == null ? "" : catalog.getName());
- row.add(job.getDbName());
- row.add(job.getTableName());
- row.add(job.getDisplayIndexName());
- row.add(job.getMutationType() == null ? "" : job.getMutationType().name());
- row.add(job.getMutationState() == null ? "" : job.getMutationState().name());
- row.add(job.getRefreshState() == null ? "" : job.getRefreshState().name());
- row.add(job.holdsPossibleLiveSlot() ? "YES" : "NO");
- row.add(TimeUtils.longToTimeString(job.getCreateTimeMs()));
- row.add(TimeUtils.longToTimeString(job.getUpdateTimeMs()));
- LanceIndexJobResult result = job.getResult();
- // The result is always null until a worker reports (no worker exists yet); the same
- // holds for every dispatch field. All of them render as empty strings, never NPE.
- row.add(result == null || result.getSanitizedMessage() == null ? "" : result.getSanitizedMessage());
- row.addAll(ShowLanceIndexJobsCommand.renderForceAudit(job));
- row.add(job.getCreator() == null ? "" : job.getCreator());
- row.add(result == null ? "" : result.getResultCode().name());
- row.add(result == null || result.getCompletionReason() == null ? "" : result.getCompletionReason().name());
- row.add(result == null ? "" : (result.isExternalMetadataAdvanced() ? "YES" : "NO"));
- row.add(job.getBackendId() == null ? "" : String.valueOf(job.getBackendId()));
- row.add(job.getBeProcessEpoch() == null ? "" : String.valueOf(job.getBeProcessEpoch()));
- row.add(job.getInvocationId() == null ? "" : job.getInvocationId());
- row.add(job.getDeadlineMs() == null ? "" : String.valueOf(job.getDeadlineMs()));
- row.add(job.getTerminationProof() == null ? "" : job.getTerminationProof().name());
- row.add(String.valueOf(job.getAdmittedDatasetVersion()));
- row.add(job.getSchemaContract() == null
- ? "" : String.valueOf(job.getSchemaContract().getSchemaContractVersion()));
- row.add(String.valueOf(job.getRevision()));
- row.add(job.isForceReleased() && job.getForceNote() != null ? job.getForceNote() : "");
- row.add(job.isForceReleased() && job.getForceWarning() != null ? job.getForceWarning() : "");
- row.add(job.isIfNotExists() ? "YES" : "NO");
- row.add(job.isIfExists() ? "YES" : "NO");
- row.add(job.getIndexType() == null ? "" : job.getIndexType());
- row.add(job.getColumnName() == null ? "" : job.getColumnName());
- return row;
- }
-
- @Override
- public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
- return visitor.visitShowLanceIndexJobCommand(this, context);
- }
-
- @Override
- public RedirectStatus toRedirectStatus() {
- return RedirectStatus.FORWARD_NO_SYNC;
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobsCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobsCommand.java
deleted file mode 100644
index 825c3e17f0a..00000000000
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobsCommand.java
+++ /dev/null
@@ -1,378 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.nereids.trees.plans.commands;
-
-import org.apache.doris.analysis.RedirectStatus;
-import org.apache.doris.catalog.Column;
-import org.apache.doris.catalog.DatabaseIf;
-import org.apache.doris.catalog.Env;
-import org.apache.doris.catalog.ScalarType;
-import org.apache.doris.catalog.TableIf;
-import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.util.TimeUtils;
-import org.apache.doris.datasource.CatalogIf;
-import org.apache.doris.datasource.lance.LanceExternalCatalog;
-import org.apache.doris.datasource.lance.job.LanceIndexJob;
-import org.apache.doris.datasource.lance.job.LanceIndexJobMutationState;
-import org.apache.doris.mysql.privilege.PrivPredicate;
-import org.apache.doris.nereids.analyzer.UnboundSlot;
-import org.apache.doris.nereids.trees.expressions.And;
-import org.apache.doris.nereids.trees.expressions.CompoundPredicate;
-import org.apache.doris.nereids.trees.expressions.EqualTo;
-import org.apache.doris.nereids.trees.expressions.Expression;
-import org.apache.doris.nereids.trees.expressions.literal.StringLikeLiteral;
-import org.apache.doris.nereids.trees.plans.PlanType;
-import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
-import org.apache.doris.qe.ConnectContext;
-import org.apache.doris.qe.ShowResultSet;
-import org.apache.doris.qe.ShowResultSetMetaData;
-import org.apache.doris.qe.StmtExecutor;
-
-import com.google.common.collect.ImmutableList;
-
-import java.util.ArrayList;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.List;
-import java.util.Locale;
-import java.util.Map;
-import java.util.Set;
-
-/**
- * SHOW LANCE INDEX JOBS [FROM [catalog.]db] [WHERE TableName = "tbl" [AND State = "PENDING"]].
- *
- * <p>Lists the durable Lance index job records held by the master. Rows whose persisted
- * target no longer resolves (the catalog is gone, the catalog is there but the db or
- * table no longer resolves, or the resolution itself fails because the provider is
- * unreachable), or whose persisted dataset locator no longer matches the catalog's
- * current one, are visible to global ADMIN only; every other row requires table-level
- * SHOW on the persisted (catalog, db, table). Rows that fail the check are omitted
- * entirely, so non-ADMIN users see no orphan trace, not even a count. The job locator,
- * provider, normalized names, propertiesJson and schema contract are never shown.
- *
- * <p>The WHERE clause is deliberately narrowed to EqualTo predicates combined with AND
- * over the case-insensitive keys TableName and State (no Like, unlike the SHOW COPY
- * precedent).
- */
-public class ShowLanceIndexJobsCommand extends ShowCommand {
- public static final ImmutableList<String> TITLE_NAMES = new ImmutableList.Builder<String>()
- .add("JobId")
- .add("CatalogName")
- .add("DbName")
- .add("TableName")
- .add("IndexName")
- .add("Operation")
- .add("State")
- .add("RefreshState")
- .add("PossibleLive")
- .add("CreateTime")
- .add("UpdateTime")
- .add("Message")
- .add("ForceReleased")
- .add("ForceActor")
- .add("ForceTime")
- .build();
-
- private static final String KEY_TABLE_NAME = "TableName";
- private static final String KEY_STATE = "State";
- private static final String WHERE_HINT = "Where clause should looks like: TableName = \"your_table_name\""
- + " or State = \"PENDING|RUNNING|COMMITTED|NOT_COMMITTED|UNKNOWN\","
- + " or compound predicate with operator AND";
-
- private final List<String> nameParts;
- private final Expression whereClause;
-
- private String ctlName;
- private String dbName;
- private String tableNameValue;
- private String stateValue;
-
- public ShowLanceIndexJobsCommand(List<String> nameParts, Expression whereClause) {
- super(PlanType.SHOW_LANCE_INDEX_JOBS_COMMAND);
- this.nameParts = nameParts;
- this.whereClause = whereClause;
- }
-
- public List<String> getNameParts() {
- return nameParts;
- }
-
- public Expression getWhereClause() {
- return whereClause;
- }
-
- @Override
- public ShowResultSetMetaData getMetaData() {
- ShowResultSetMetaData.Builder builder = ShowResultSetMetaData.builder();
- for (String title : TITLE_NAMES) {
- builder.addColumn(new Column(title, ScalarType.createVarchar(30)));
- }
- return builder.build();
- }
-
- private void validate(ConnectContext ctx) throws AnalysisException {
- if (nameParts != null) {
- if (nameParts.size() == 1) {
- dbName = nameParts.get(0);
- CatalogIf currentCatalog = ctx.getCurrentCatalog();
- ctlName = currentCatalog == null ? null : currentCatalog.getName();
- normalizeDbName(currentCatalog);
- } else if (nameParts.size() == 2) {
- ctlName = nameParts.get(0);
- dbName = nameParts.get(1);
- normalizeDbName(Env.getCurrentEnv().getCatalogMgr().getCatalog(ctlName));
- } else {
- throw new AnalysisException(
- "Only support SHOW LANCE INDEX JOBS FROM [catalog.]database, but get: " + nameParts);
- }
- }
- analyzeWhereClause();
- }
-
- /**
- * Resolves the FROM database name through the target catalog's own naming rules
- * before it is compared against the persisted job records: with
- * lower_case_database_names = 1/2, {@code FROM DB1} must still match jobs stored
- * under the resolved full name {@code db1}. A name the catalog cannot resolve (or a
- * resolution that fails outright) keeps the raw requested string, so the filter just
- * stays an exact comparison; the resolution itself must never fail the statement.
- */
- private void normalizeDbName(CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog) {
- if (dbName == null || catalog == null) {
- return;
- }
- try {
- DatabaseIf<? extends TableIf> db = catalog.getDbNullable(dbName);
- if (db == null) {
- return;
- }
- String fullName = db.getFullName();
- if (fullName != null && !fullName.isEmpty()) {
- dbName = fullName;
- }
- } catch (Exception e) {
- // Keep the raw requested name; the filter remains an exact comparison.
- }
- }
-
- private void analyzeWhereClause() throws AnalysisException {
- if (whereClause == null) {
- return;
- }
- List<Expression> children = new ArrayList<>();
- splitCompoundPredicate(whereClause, children);
- Set<String> names = new HashSet<>();
- for (Expression child : children) {
- analyzeSubPredicate(child);
- String name = ((UnboundSlot) child.child(0)).getName().toLowerCase(Locale.ROOT);
- if (!names.add(name)) {
- throw new AnalysisException("column names on both sides of operator AND should be different");
- }
- }
- }
-
- private void splitCompoundPredicate(Expression expr, List<Expression> children) throws AnalysisException {
- if (expr instanceof CompoundPredicate) {
- if (!(expr instanceof And)) {
- throw new AnalysisException("Only allow compound predicate with operator AND");
- }
- splitCompoundPredicate(expr.child(0), children);
- splitCompoundPredicate(expr.child(1), children);
- } else {
- children.add(expr);
- }
- }
-
- private void analyzeSubPredicate(Expression expr) throws AnalysisException {
- if (!(expr instanceof EqualTo)
- || !(expr.child(0) instanceof UnboundSlot)
- || !(expr.child(1) instanceof StringLikeLiteral)) {
- throw new AnalysisException(WHERE_HINT);
- }
- String key = ((UnboundSlot) expr.child(0)).getName();
- String value = ((StringLikeLiteral) expr.child(1)).getStringValue();
- if (key.equalsIgnoreCase(KEY_TABLE_NAME)) {
- tableNameValue = value;
- } else if (key.equalsIgnoreCase(KEY_STATE)) {
- stateValue = value.toUpperCase(Locale.ROOT);
- try {
- LanceIndexJobMutationState.valueOf(stateValue);
- } catch (IllegalArgumentException e) {
- throw new AnalysisException("Unknown Lance index job state: " + value + "; " + WHERE_HINT);
- }
- } else {
- throw new AnalysisException(WHERE_HINT);
- }
- }
-
- @Override
- public ShowResultSet doRun(ConnectContext ctx, StmtExecutor executor) throws Exception {
- validate(ctx);
- List<List<String>> rows = new ArrayList<>();
- // One locator resolution per unique (catalogId, db, table) for the whole listing:
- // each resolution is a provider round trip, and failed (null) resolutions are
- // cached too so a broken target is probed once, not once per sibling job.
- Map<String, String> locatorCache = new HashMap<>();
- for (LanceIndexJob job : Env.getCurrentEnv().getLanceIndexJobManager().getAllJobsSnapshot()) {
- CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog =
- Env.getCurrentEnv().getCatalogMgr().getCatalog(job.getCatalogId());
- if (!matchesFilters(catalog, job)) {
- continue;
- }
- if (!isAuthorized(ctx, catalog, job, locatorCache)) {
- continue;
- }
- rows.add(renderRow(job, catalog));
- }
- return new ShowResultSet(getMetaData(), rows);
- }
-
- private boolean matchesFilters(CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog, LanceIndexJob job) {
- if (ctlName != null && (catalog == null || !ctlName.equals(catalog.getName()))) {
- return false;
- }
- if (dbName != null && !dbName.equals(job.getDbName())) {
- return false;
- }
- if (tableNameValue != null && !tableNameValue.equals(job.getTableName())) {
- return false;
- }
- return stateValue == null
- || job.getMutationState() != null && stateValue.equals(job.getMutationState().name());
- }
-
- /**
- * Orphan and half-orphan rows (catalog gone, persisted db/table no longer resolvable,
- * or a resolution that fails outright because the provider is unreachable) are
- * visible to global ADMIN only. A Lance catalog target is additionally revalidated
- * against the catalog's current durable dataset locator: after a job reaches a
- * terminal state and releases its guard, a legitimate ALTER can repoint the same
- * db.table names at a different dataset, and SHOW on the new target must not
- * disclose the old job. Every other row needs table-level SHOW on the persisted
- * target. The caller omits the row when this returns false, so non-ADMIN users see
- * no trace of orphaned jobs, not even a count.
- */
- static boolean isAuthorized(ConnectContext ctx, CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog,
- LanceIndexJob job) {
- return isAuthorized(ctx, catalog, job, null);
- }
-
- /**
- * Same authorization as {@link #isAuthorized(ConnectContext, CatalogIf, LanceIndexJob)},
- * with an optional (catalogId, db, table)-keyed locator cache so a listing resolves
- * each unique target's current locator once per run instead of once per job row.
- */
- static boolean isAuthorized(ConnectContext ctx, CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog,
- LanceIndexJob job, Map<String, String> locatorCache) {
- if (!targetResolves(catalog, job) || !locatorMatches(catalog, job, locatorCache)) {
- return Env.getCurrentEnv().getAccessManager().checkGlobalPriv(ctx, PrivPredicate.ADMIN);
- }
- return Env.getCurrentEnv().getAccessManager().checkTblPriv(ctx, catalog.getName(),
- job.getDbName(), job.getTableName(), PrivPredicate.SHOW);
- }
-
- static boolean targetResolves(CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog, LanceIndexJob job) {
- if (catalog == null) {
- return false;
- }
- try {
- DatabaseIf<? extends TableIf> db = catalog.getDbNullable(job.getDbName());
- return db != null && db.getTableNullable(job.getTableName()) != null;
- } catch (Exception e) {
- // External metadata resolution can fail outright (credentials expired,
- // provider down). That must not leak the provider error or abort the listing:
- // a failed resolution is handled exactly like an unresolvable (orphan) target.
- return false;
- }
- }
-
- /**
- * Whether the catalog's current durable dataset locator still points at the dataset
- * this job was admitted against. Only Lance catalogs carry a durable locator; for
- * any other catalog instance (theoretically unreachable, since jobs are only
- * admitted for Lance catalogs) the name resolution above is the whole rule. A
- * locator that cannot be resolved right now counts as a mismatch, so authorization
- * fails closed to the orphan rule instead of granting SHOW through stale names.
- */
- private static boolean locatorMatches(CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog,
- LanceIndexJob job, Map<String, String> locatorCache) {
- if (!(catalog instanceof LanceExternalCatalog)) {
- return true;
- }
- String key = job.getCatalogId() + ":" + job.getDbName() + ":" + job.getTableName();
- String currentLocator;
- if (locatorCache != null && locatorCache.containsKey(key)) {
- currentLocator = locatorCache.get(key);
- } else {
- currentLocator = ((LanceExternalCatalog) catalog).resolveCurrentIndexJobLocator(
- job.getDbName(), job.getTableName());
- if (locatorCache != null) {
- locatorCache.put(key, currentLocator);
- }
- }
- return currentLocator != null && currentLocator.equals(job.getNormalizedLocator());
- }
-
- private static List<String> renderRow(LanceIndexJob job,
- CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog) {
- List<String> row = new ArrayList<>(TITLE_NAMES.size());
- row.add(String.valueOf(job.getJobId()));
- // The catalog name is not persisted on the job record; for orphan rows (catalog gone)
- // there is nothing safe to render, so the column stays empty rather than fabricating
- // a placeholder from the internal catalog id.
- row.add(catalog == null ? "" : catalog.getName());
- row.add(job.getDbName());
- row.add(job.getTableName());
- row.add(job.getDisplayIndexName());
- row.add(job.getMutationType() == null ? "" : job.getMutationType().name());
- row.add(job.getMutationState() == null ? "" : job.getMutationState().name());
- row.add(job.getRefreshState() == null ? "" : job.getRefreshState().name());
- row.add(job.holdsPossibleLiveSlot() ? "YES" : "NO");
- row.add(TimeUtils.longToTimeString(job.getCreateTimeMs()));
- row.add(TimeUtils.longToTimeString(job.getUpdateTimeMs()));
- // job.getResult() is always null until a worker reports (no worker exists yet).
- row.add(job.getResult() == null || job.getResult().getSanitizedMessage() == null
- ? "" : job.getResult().getSanitizedMessage());
- row.addAll(renderForceAudit(job));
- return row;
- }
-
- static List<String> renderForceAudit(LanceIndexJob job) {
- List<String> forceColumns = new ArrayList<>(3);
- if (!job.isForceReleased()) {
- forceColumns.add("");
- forceColumns.add("");
- forceColumns.add("");
- return forceColumns;
- }
- forceColumns.add("YES");
- forceColumns.add(job.getForceActor() == null ? "" : job.getForceActor());
- forceColumns.add(job.getForceTimeMs() == null ? "" : TimeUtils.longToTimeString(job.getForceTimeMs()));
- return forceColumns;
- }
-
- @Override
- public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
- return visitor.visitShowLanceIndexJobsCommand(this, context);
- }
-
- @Override
- public RedirectStatus toRedirectStatus() {
- return RedirectStatus.FORWARD_NO_SYNC;
- }
-}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/CommandVisitor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/CommandVisitor.java
index e0e184ca760..61f5ae23df0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/CommandVisitor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/visitor/CommandVisitor.java
@@ -229,8 +229,6 @@ import org.apache.doris.nereids.trees.plans.commands.ShowIndexNormalizerCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowIndexStatsCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowIndexTokenFilterCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowIndexTokenizerCommand;
-import org.apache.doris.nereids.trees.plans.commands.ShowLanceIndexJobCommand;
-import org.apache.doris.nereids.trees.plans.commands.ShowLanceIndexJobsCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowLastInsertCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowLoadCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowLoadProfileCommand;
@@ -685,14 +683,6 @@ public interface CommandVisitor<R, C> {
return visitCommand(showCreateRepositoryCommand, context);
}
- default R visitShowLanceIndexJobCommand(ShowLanceIndexJobCommand showLanceIndexJobCommand, C context) {
- return visitCommand(showLanceIndexJobCommand, context);
- }
-
- default R visitShowLanceIndexJobsCommand(ShowLanceIndexJobsCommand showLanceIndexJobsCommand, C context) {
- return visitCommand(showLanceIndexJobsCommand, context);
- }
-
default R visitShowLastInsertCommand(ShowLastInsertCommand showLastInsertCommand, C context) {
return visitCommand(showLastInsertCommand, context);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java b/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
index ce7ab3219cf..ea93b941cf1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
@@ -64,7 +64,6 @@ import org.apache.doris.datasource.InitCatalogLog;
import org.apache.doris.datasource.InitDatabaseLog;
import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.datasource.MetaIdMappingsLog;
-import org.apache.doris.datasource.lance.job.LanceIndexJob;
import org.apache.doris.dictionary.Dictionary;
import org.apache.doris.ha.MasterInfo;
import org.apache.doris.indexpolicy.DropIndexPolicyLog;
@@ -1445,11 +1444,6 @@ public class EditLog {
env.getKeyManager().replayKeyOperation(info);
break;
}
- case OperationType.OP_LANCE_INDEX_JOB_UPSERT: {
- LanceIndexJob job = (LanceIndexJob) journal.getData();
- env.getLanceIndexJobManager().replayUpsertJob(job);
- break;
- }
case OperationType.OP_BEGIN_SNAPSHOT: {
// SnapshotState info = (SnapshotState) journal.getData();
// TODO: implement
@@ -1460,6 +1454,11 @@ public class EditLog {
// This log is only used to keep FE/MS cut point in journal timeline.
break;
}
+ case OperationType.OP_LANCE_INDEX_JOB_UPSERT: {
+ // Consumed and ignored by JournalEntity: payload of the removed durable
+ // Lance index job framework; nothing to replay.
+ break;
+ }
default: {
IOException e = new IOException();
LOG.error("UNKNOWN Operation Type {}, log id: {}", opCode, logId, e);
@@ -2369,10 +2368,6 @@ public class EditLog {
logEdit(OperationType.OP_DROP_INDEX_POLICY, policy);
}
- public void logLanceIndexJob(LanceIndexJob job) {
- logEdit(OperationType.OP_LANCE_INDEX_JOB_UPSERT, job);
- }
-
public void logCatalogLog(short id, CatalogLog log) {
logEdit(id, log);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/OperationType.java b/fe/fe-core/src/main/java/org/apache/doris/persist/OperationType.java
index e2fbc2131f7..f3ed185dab1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/OperationType.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/OperationType.java
@@ -422,7 +422,11 @@ public class OperationType {
public static final short OP_CREATE_ROLE_MAPPING = 496;
public static final short OP_DROP_ROLE_MAPPING = 497;
- // lance index job 500 ~ 509
+ // Payload of the removed durable Lance index job framework: kept only as a tombstone so
+ // old journals stay parseable (JournalEntity consumes and ignores the record). The code
+ // must never be reused: an edit log or image written by a build that still had the
+ // framework carries that opcode. No production deployment could hold such records: the
+ // mutation gate (enable_lance_index_mutation) never left its disabled default.
public static final short OP_LANCE_INDEX_JOB_UPSERT = 500;
// For cloud.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/meta/MetaPersistMethod.java b/fe/fe-core/src/main/java/org/apache/doris/persist/meta/MetaPersistMethod.java
index 6ed7645d461..8f8425ec4a8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/meta/MetaPersistMethod.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/meta/MetaPersistMethod.java
@@ -286,12 +286,6 @@ public class MetaPersistMethod {
metaPersistMethod.writeMethod = Env.class.getDeclaredMethod("saveKeyManagerStore",
CountingDataOutputStream.class, long.class);
break;
- case "lanceIndexJobManager":
- metaPersistMethod.readMethod = Env.class.getDeclaredMethod("loadLanceIndexJobManager",
- DataInputStream.class, long.class);
- metaPersistMethod.writeMethod = Env.class.getDeclaredMethod("saveLanceIndexJobManager",
- CountingDataOutputStream.class, long.class);
- break;
default:
break;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/meta/PersistMetaModules.java b/fe/fe-core/src/main/java/org/apache/doris/persist/meta/PersistMetaModules.java
index 8f8f00dd64e..ee4f93cc801 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/meta/PersistMetaModules.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/meta/PersistMetaModules.java
@@ -44,7 +44,7 @@ public class PersistMetaModules {
"globalFunction", "workloadGroups",
"binlogs", "resourceGroups", "AnalysisMgrV2", "AsyncJobManager", "workloadSchedPolicy",
"insertOverwrite", "plsql", "dictionaryManager", "indexPolicy", "KeyManagerStore",
- "authenticationIntegrations", "roleMappings", "lanceIndexJobManager"
+ "authenticationIntegrations", "roleMappings"
);
// The modules in `CloudEnv`.
@@ -52,7 +52,10 @@ public class PersistMetaModules {
// Modules in this list is deprecated and will not be saved in meta file. (also should not be in MODULE_NAMES)
public static final ImmutableList<String> DEPRECATED_MODULE_NAMES = ImmutableList.of(
- "loadJob", "cooldownJob", "AnalysisMgr", "mtmvJobManager", "JobTaskManager", "syncJob");
+ "loadJob", "cooldownJob", "AnalysisMgr", "mtmvJobManager", "JobTaskManager", "syncJob",
+ // Removed with the durable Lance index job framework; images written while it
+ // existed carry this (possibly empty) chunk and must stay readable.
+ "lanceIndexJobManager");
static {
MODULES_MAP = Maps.newHashMap();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java
index 13b3f506fb9..40940266e07 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/DorisFlightSqlProducer.java
@@ -238,6 +238,21 @@ public class DorisFlightSqlProducer implements FlightSqlProducer, AutoCloseable
if (connectContext.getState().getStateType() == MysqlStateType.ERR) {
throw new RuntimeException("after executeQueryStatement handleQuery");
}
+ // A statement forwarded to the master that was rejected there keeps the local
+ // state OK: the MySQL protocol path returns the master's ERR packet verbatim in
+ // ConnectProcessor#finalizeCommand and the audit log reads the proxy status (see
+ // the multi-statement loop in ConnectProcessor#handleQuery). Arrow Flight has no
+ // channel for that packet, so the forwarded rejection must be surfaced here;
+ // otherwise the OK result synthesized below would report success for a statement
+ // the master refused to execute.
+ StmtExecutor stmtExecutor = connectContext.getExecutor();
+ if (stmtExecutor != null && stmtExecutor.hasForwardedToMaster()
+ && stmtExecutor.getProxyStatusCode() != 0) {
+ connectContext.getState().setError("forwarded statement failed on master FE, error code: "
+ + stmtExecutor.getProxyStatusCode() + ", error msg: "
+ + stmtExecutor.getProxyErrMsg());
+ throw new RuntimeException("after executeQueryStatement forwardToMaster");
+ }
if (connectContext.isReturnResultFromLocal()) {
// set/use etc. stmt returns an OK result by default.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlQuerySchema.java b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlQuerySchema.java
index 0ed2b77e92f..b8a739c2796 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlQuerySchema.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/service/arrowflight/FlightSqlQuerySchema.java
@@ -30,7 +30,6 @@ import org.apache.doris.catalog.Type;
import org.apache.doris.common.Pair;
import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.es.EsExternalCatalog;
-import org.apache.doris.datasource.lance.LanceExternalCatalog;
import org.apache.doris.mysql.MysqlCommand;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.nereids.CascadesContext;
@@ -46,7 +45,6 @@ import org.apache.doris.nereids.trees.expressions.literal.NullLiteral;
import org.apache.doris.nereids.trees.plans.PlaceholderId;
import org.apache.doris.nereids.trees.plans.Plan;
import org.apache.doris.nereids.trees.plans.PrepareCommandPlanner;
-import org.apache.doris.nereids.trees.plans.commands.AlterTableCommand;
import org.apache.doris.nereids.trees.plans.commands.Command;
import org.apache.doris.nereids.trees.plans.commands.DeleteFromCommand;
import org.apache.doris.nereids.trees.plans.commands.DescribeCommand;
@@ -64,8 +62,6 @@ import org.apache.doris.nereids.trees.plans.commands.ShowSnapshotCommand;
import org.apache.doris.nereids.trees.plans.commands.ShowTableCommand;
import org.apache.doris.nereids.trees.plans.commands.TransactionCommand;
import org.apache.doris.nereids.trees.plans.commands.UpdateCommand;
-import org.apache.doris.nereids.trees.plans.commands.info.CreateIndexOp;
-import org.apache.doris.nereids.trees.plans.commands.info.DropIndexOp;
import org.apache.doris.nereids.trees.plans.commands.insert.BatchInsertIntoTableCommand;
import org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTVFCommand;
import org.apache.doris.nereids.trees.plans.commands.insert.InsertIntoTableCommand;
@@ -361,17 +357,6 @@ final class FlightSqlQuerySchema {
return stringMetadata("Explain String(Nereids Planner)");
} else if (command instanceof ReplayCommand) {
return stringMetadata("Plan Replayer dump url");
- } else if (command instanceof AlterTableCommand) {
- AlterTableCommand alter = (AlterTableCommand) command;
- String catalog = alter.getTbl().getCtl();
- // Lance index admission returns a JobId header even for an IF no-op. Do not run
- // validation/admission here: those paths can resolve remote tables or allocate IDs.
- if (context.getCatalog(catalog == null ? context.getDefaultCatalog() : catalog)
- instanceof LanceExternalCatalog && alter.getNereidsOps().stream().anyMatch(op ->
- (op instanceof CreateIndexOp && !((CreateIndexOp) op).isAlter())
- || (op instanceof DropIndexOp && !((DropIndexOp) op).isAlter()))) {
- return stringMetadata("JobId");
- }
}
ResultSetMetaData metadata = command.getResultSetMetaData();
// Concrete DML commands return OK, but subclasses such as WARM UP SELECT supply rows.
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogMgrLanceGuardTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogMgrLanceGuardTest.java
index 1b667faa05a..054687076b0 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogMgrLanceGuardTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/CatalogMgrLanceGuardTest.java
@@ -19,20 +19,10 @@ package org.apache.doris.datasource;
import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
-import org.apache.doris.catalog.RefreshManager;
import org.apache.doris.catalog.TableIf;
import org.apache.doris.common.DdlException;
import org.apache.doris.datasource.lance.LanceExternalCatalog;
-import org.apache.doris.datasource.lance.job.LanceIndexFenceKey;
-import org.apache.doris.datasource.lance.job.LanceIndexJob;
-import org.apache.doris.datasource.lance.job.LanceIndexJobManager;
-import org.apache.doris.datasource.lance.job.LanceIndexJobMutationType;
-import org.apache.doris.datasource.lance.job.LanceIndexNameNormalizer;
-import org.apache.doris.persist.EditLog;
-import org.apache.doris.persist.OperationType;
-import org.apache.doris.statistics.query.QueryStats;
-import com.google.common.collect.ImmutableMap;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
import org.mockito.MockedStatic;
@@ -42,454 +32,95 @@ import java.lang.reflect.Field;
import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.CountDownLatch;
-import java.util.concurrent.ExecutorService;
-import java.util.concurrent.Executors;
-import java.util.concurrent.Future;
-import java.util.concurrent.TimeUnit;
-import java.util.concurrent.TimeoutException;
-import java.util.concurrent.atomic.AtomicLong;
-import java.util.concurrent.atomic.AtomicReference;
import java.util.function.Supplier;
/**
- * Coverage for the Lance catalog DDL guard in {@link CatalogMgr}: while the catalog has
- * unresolved Lance index jobs, ALTER CATALOG ... SET PROPERTIES must reject actual value
- * changes of the target identity keys - the namespace keys (lance.catalog.type, warehouse,
- * lance.namespace.parent, lance.namespace.delimiter, lance.namespace.root_database) and
- * the storage-routing keys (the s3.endpoint/s3.region family) that can land the same
- * persisted URI on a different storage service - and DROP CATALOG must be rejected;
- * credential-only/unrelated property changes, same-value rewrites, renames, and the
- * replay path stay unguarded. The guard message must stay neutral: no locator and no
- * release-statement syntax.
+ * Coverage for the surviving Lance target-identity guard in {@link CatalogMgr}:
+ * {@code hasLanceIdentityKeyChange} decides whether a property rewrite touches the
+ * namespace or storage-routing keys that can land the same table names on a different
+ * dataset target, and the ALTER-catalog replay path answers such a change by advancing
+ * the Lance admission epoch so in-flight admission snapshots fail revalidation.
+ * Same-value rewrites, credential-only changes, and unrelated keys stay no-ops.
*/
public class CatalogMgrLanceGuardTest {
- private static final long CATALOG_ID = 10L;
- private static final String CATALOG_NAME = "lance_catalog";
- private static final String LOCATOR = "s3://bucket/dataset";
- private static final String[] IDENTITY_KEYS = {
- "lance.catalog.type", "warehouse",
- "lance.namespace.parent", "lance.namespace.delimiter", "lance.namespace.root_database",
- "s3.endpoint", "s3.region"};
- private static final class Fixture {
- private final CatalogMgr catalogMgr = new CatalogMgr();
- private final LanceExternalCatalog catalog = Mockito.mock(LanceExternalCatalog.class);
- private final TestJobManager jobManager = new TestJobManager();
- private final Env env = Mockito.mock(Env.class);
- private final EditLog editLog = Mockito.mock(EditLog.class);
- private final ExternalMetaCacheMgr metaCacheMgr = Mockito.mock(ExternalMetaCacheMgr.class);
- private final Map<String, String> oldProperties = new HashMap<>();
+ @Test
+ public void testIdentityKeyChangeSemantics() {
+ Map<String, String> persisted = new HashMap<>();
+ persisted.put("s3.endpoint", "s3://a");
+ persisted.put("type", "hms");
+ persisted.put("s3.access_key", "old");
+
+ Assertions.assertTrue(CatalogMgr.hasLanceIdentityKeyChange(persisted, props("s3.endpoint", "s3://b")),
+ "a value change of an identity key is a target change");
+ Assertions.assertFalse(CatalogMgr.hasLanceIdentityKeyChange(persisted, props("s3.endpoint", "s3://a")),
+ "a same-value rewrite is an idempotent no-op");
+ Assertions.assertTrue(CatalogMgr.hasLanceIdentityKeyChange(persisted, props("s3.region", "us-west")),
+ "a newly supplied identity key the persisted properties never set is a change");
+ Assertions.assertTrue(CatalogMgr.hasLanceIdentityKeyChange(persisted, props("S3.ENDPOINT", "s3://b")),
+ "identity keys match case-insensitively");
+ Assertions.assertFalse(CatalogMgr.hasLanceIdentityKeyChange(persisted, props("s3.access_key", "new")),
+ "a credential-only change is not a target change");
+ Assertions.assertFalse(CatalogMgr.hasLanceIdentityKeyChange(persisted, props("type", "jdbc")),
+ "an unrelated property change is not a target change");
+ Map<String, String> withNullKey = new HashMap<>();
+ withNullKey.put(null, "x");
+ Assertions.assertFalse(CatalogMgr.hasLanceIdentityKeyChange(persisted, withNullKey),
+ "a null key is never an identity key");
+ }
+
+ @Test
+ public void testAlterReplayAdvancesAdmissionEpochOnlyOnTargetChange() throws Exception {
+ CatalogMgr catalogMgr = new CatalogMgr();
+ LanceExternalCatalog catalog = Mockito.mock(LanceExternalCatalog.class);
+ Mockito.when(catalog.getId()).thenReturn(10L);
+ Mockito.when(catalog.getProperties()).thenReturn(props("s3.endpoint", "s3://a"));
+ registerCatalog(catalogMgr, catalog);
+
+ Env env = Mockito.mock(Env.class);
+ ExternalMetaCacheMgr metaCacheMgr = Mockito.mock(ExternalMetaCacheMgr.class);
+ Mockito.when(env.getExtMetaCacheMgr()).thenReturn(metaCacheMgr);
+ Mockito.when(metaCacheMgr.withCatalogLifecycleLock(Mockito.anyLong(), Mockito.any()))
+ .thenAnswer(invocation -> {
+ Supplier<?> action = invocation.getArgument(1);
+ return action.get();
+ });
- Fixture(String lanceCatalogType) throws Exception {
- oldProperties.put("type", "lance");
- oldProperties.put("lance.catalog.type", lanceCatalogType);
- oldProperties.put("warehouse", "s3://bucket/warehouse");
- oldProperties.put("lance.namespace.parent", "s3://bucket/ns");
- oldProperties.put("lance.namespace.delimiter", ".");
- oldProperties.put("lance.namespace.root_database", "root_db");
- oldProperties.put("s3.access_key", "ak");
- oldProperties.put("s3.endpoint", "https://s3.example.com");
- oldProperties.put("s3.region", "us-east-1");
+ try (MockedStatic<Env> currentEnv = Mockito.mockStatic(Env.class)) {
+ currentEnv.when(Env::getCurrentEnv).thenReturn(env);
- AtomicReference<String> catalogName = new AtomicReference<>(CATALOG_NAME);
- Mockito.when(catalog.getId()).thenReturn(CATALOG_ID);
- Mockito.when(catalog.getType()).thenReturn("lance");
- Mockito.when(catalog.getName()).thenAnswer(invocation -> catalogName.get());
- Mockito.doAnswer(invocation -> {
- catalogName.set(invocation.getArgument(0));
- return null;
- }).when(catalog).modifyCatalogName(Mockito.anyString());
- AtomicLong targetVersion = new AtomicLong();
- Mockito.when(catalog.getIndexTargetVersion()).thenAnswer(invocation -> targetVersion.get());
- Mockito.doAnswer(invocation -> {
- targetVersion.incrementAndGet();
- return null;
- }).when(catalog).advanceIndexTargetVersion();
- Mockito.when(catalog.getProperties()).thenReturn(oldProperties);
- Mockito.doAnswer(invocation -> {
- oldProperties.putAll(invocation.getArgument(0));
- return null;
- }).when(catalog).modifyCatalogProps(Mockito.anyMap());
- registerCatalog(catalogMgr, catalog);
+ replayAlter(catalogMgr, 10L, props("s3.access_key", "new"));
+ Mockito.verify(catalog, Mockito.never()).advanceIndexTargetVersion();
- Mockito.when(env.getLanceIndexJobManager()).thenReturn(jobManager);
- Mockito.when(env.getEditLog()).thenReturn(editLog);
- Mockito.when(env.getExtMetaCacheMgr()).thenReturn(metaCacheMgr);
- Mockito.when(env.getRefreshManager()).thenReturn(Mockito.mock(RefreshManager.class));
- Mockito.when(env.getQueryStats()).thenReturn(Mockito.mock(QueryStats.class));
- Mockito.when(metaCacheMgr.withCatalogLifecycleLock(Mockito.anyLong(), Mockito.any()))
- .thenAnswer(invocation -> {
- Supplier<?> action = invocation.getArgument(1);
- return action.get();
- });
- }
+ replayAlter(catalogMgr, 10L, props("s3.endpoint", "s3://a"));
+ Mockito.verify(catalog, Mockito.never()).advanceIndexTargetVersion();
- void admitUnresolvedJob() throws Exception {
- jobManager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
+ replayAlter(catalogMgr, 10L, props("s3.endpoint", "s3://b"));
+ Mockito.verify(catalog, Mockito.times(1)).advanceIndexTargetVersion();
}
+ }
- MockedStatic<Env> mockCurrentEnv() {
- MockedStatic<Env> mockedEnv = Mockito.mockStatic(Env.class);
- mockedEnv.when(Env::getCurrentEnv).thenReturn(env);
- return mockedEnv;
- }
+ private static void replayAlter(CatalogMgr catalogMgr, long catalogId, Map<String, String> newProps)
+ throws DdlException {
+ CatalogLog log = new CatalogLog();
+ log.setCatalogId(catalogId);
+ log.setNewProps(newProps);
+ catalogMgr.replayAlterCatalogProps(log, null, true);
}
@SuppressWarnings("unchecked")
- private static void registerCatalog(CatalogMgr catalogMgr, CatalogIf catalog) throws Exception {
+ private static void registerCatalog(CatalogMgr catalogMgr, LanceExternalCatalog catalog) throws Exception {
Field idToCatalogField = CatalogMgr.class.getDeclaredField("idToCatalog");
idToCatalogField.setAccessible(true);
((ConcurrentMap<Long, CatalogIf<? extends DatabaseIf<? extends TableIf>>>)
idToCatalogField.get(catalogMgr)).put(catalog.getId(), catalog);
- Field nameToCatalogField = CatalogMgr.class.getDeclaredField("nameToCatalog");
- nameToCatalogField.setAccessible(true);
- ((Map<String, CatalogIf>) nameToCatalogField.get(catalogMgr)).put(catalog.getName(), catalog);
- }
-
- private static LanceIndexJob newCreateJob(long jobId, String displayName) {
- return new LanceIndexJob(jobId, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR,
- displayName, LanceIndexNameNormalizer.normalize(displayName),
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v",
- null, 7L, null);
- }
-
- private static void assertNeutralGuardMessage(DdlException e) {
- String message = e.getMessage();
- Assertions.assertNotNull(message);
- Assertions.assertTrue(message.contains("unresolved Lance index jobs"), message);
- Assertions.assertFalse(message.contains("RESOLVE"), message);
- Assertions.assertFalse(message.contains(LOCATOR), message);
- Assertions.assertFalse(message.contains("s3://"), message);
- }
-
- @Test
- public void testEveryIdentityKeyValueChangeIsBlocked() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- fixture.admitUnresolvedJob();
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- for (String key : IDENTITY_KEYS) {
- Map<String, String> newProperties = ImmutableMap.of(key, "changed-value");
- DdlException e = Assertions.assertThrows(DdlException.class,
- () -> fixture.catalogMgr.alterCatalogProps(CATALOG_NAME, newProperties), key);
- assertNeutralGuardMessage(e);
- }
- Mockito.verify(fixture.editLog, Mockito.never())
- .logCatalogLog(Mockito.anyShort(), Mockito.any());
- Mockito.verify(fixture.catalog, Mockito.never()).modifyCatalogProps(Mockito.any());
- }
- }
-
- @Test
- public void testAddingAbsentIdentityKeyIsBlocked() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- fixture.oldProperties.remove("lance.namespace.root_database");
- fixture.admitUnresolvedJob();
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- DdlException e = Assertions.assertThrows(DdlException.class,
- () -> fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of("lance.namespace.root_database", "other_db")));
- assertNeutralGuardMessage(e);
- }
- }
-
- @Test
- public void testSameValueRewriteIsAllowed() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- fixture.admitUnresolvedJob();
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- // Re-setting every property (identity keys included) to its current value is
- // an idempotent no-op and must not be blocked.
- fixture.catalogMgr.alterCatalogProps(CATALOG_NAME, fixture.oldProperties);
- Mockito.verify(fixture.editLog, Mockito.times(1))
- .logCatalogLog(Mockito.eq(OperationType.OP_ALTER_CATALOG_PROPS), Mockito.any());
- }
- }
-
- @Test
- public void testCaseVariantIdentityKeyChangeIsBlocked() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- fixture.admitUnresolvedJob();
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- // Catalog property keys are not normalized upstream; a case variant with a
- // changed value is still a target identity change.
- DdlException e = Assertions.assertThrows(DdlException.class,
- () -> fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of("WAREHOUSE", "s3://other/warehouse")));
- assertNeutralGuardMessage(e);
- Mockito.verify(fixture.catalog, Mockito.never()).modifyCatalogProps(Mockito.any());
- }
- }
-
- @Test
- public void testCredentialAndUnrelatedKeyChangesAreAllowed() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- fixture.admitUnresolvedJob();
- Map<String, String> newProperties = new HashMap<>();
- // Credentials authenticate to the same target, so rotating them stays allowed;
- // storage routing (s3.endpoint and family) is identity and is covered below.
- newProperties.put("s3.access_key", "rotated-ak");
- newProperties.put("s3.secret_key", "rotated-sk");
- newProperties.put("custom.unrelated.property", "value");
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- fixture.catalogMgr.alterCatalogProps(CATALOG_NAME, newProperties);
- Mockito.verify(fixture.editLog, Mockito.times(1))
- .logCatalogLog(Mockito.eq(OperationType.OP_ALTER_CATALOG_PROPS), Mockito.any());
- }
- }
-
- @Test
- public void testStorageRoutingKeyChangeWithUnresolvedJobIsBlocked() throws Exception {
- // s3.endpoint and its family route the persisted URI to storage rather than
- // authenticate to it: a changed endpoint can be a different storage service, so
- // it is target identity, not a credential.
- Fixture fixture = new Fixture("filesystem");
- fixture.admitUnresolvedJob();
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- DdlException e = Assertions.assertThrows(DdlException.class,
- () -> fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of("s3.endpoint", "https://s3.other-example.com")));
- assertNeutralGuardMessage(e);
- // A routing key the catalog never carried still counts as a change from its
- // implicit absence, exactly like the namespace identity keys.
- DdlException absent = Assertions.assertThrows(DdlException.class,
- () -> fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of("oss.endpoint", "https://oss.example.com")));
- assertNeutralGuardMessage(absent);
- Mockito.verify(fixture.editLog, Mockito.never())
- .logCatalogLog(Mockito.anyShort(), Mockito.any());
- Mockito.verify(fixture.catalog, Mockito.never()).modifyCatalogProps(Mockito.any());
- }
- }
-
- @Test
- public void testDropCatalogIsBlockedWithUnresolvedJob() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- fixture.admitUnresolvedJob();
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- DdlException e = Assertions.assertThrows(DdlException.class,
- () -> fixture.catalogMgr.dropCatalog(CATALOG_NAME, false));
- assertNeutralGuardMessage(e);
- Assertions.assertSame(fixture.catalog, fixture.catalogMgr.getCatalog(CATALOG_NAME));
- Mockito.verify(fixture.editLog, Mockito.never())
- .logCatalogLog(Mockito.anyShort(), Mockito.any());
- }
- }
-
- @Test
- public void testDropCatalogIsAllowedWithoutUnresolvedJobs() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- fixture.catalogMgr.dropCatalog(CATALOG_NAME, false);
- Assertions.assertNull(fixture.catalogMgr.getCatalog(CATALOG_NAME));
- Mockito.verify(fixture.editLog, Mockito.times(1))
- .logCatalogLog(Mockito.eq(OperationType.OP_DROP_CATALOG), Mockito.any());
- }
- }
-
- @Test
- public void testRenameIsAllowedWithUnresolvedJob() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- fixture.admitUnresolvedJob();
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- // The fence identity is the persistent catalog id, so a rename is not target-changing.
- fixture.catalogMgr.alterCatalogName(CATALOG_NAME, "lance_catalog_renamed");
- Assertions.assertSame(fixture.catalog, fixture.catalogMgr.getCatalog("lance_catalog_renamed"));
- Assertions.assertNull(fixture.catalogMgr.getCatalog(CATALOG_NAME));
- Mockito.verify(fixture.editLog, Mockito.times(1))
- .logCatalogLog(Mockito.eq(OperationType.OP_ALTER_CATALOG_NAME), Mockito.any());
- }
- }
-
- @Test
- public void testIdentityKeyChangeIsAllowedWithoutUnresolvedJobs() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of("warehouse", "s3://bucket/elsewhere"));
- Mockito.verify(fixture.editLog, Mockito.times(1))
- .logCatalogLog(Mockito.eq(OperationType.OP_ALTER_CATALOG_PROPS), Mockito.any());
- }
- }
-
- @Test
- public void testRestCatalogIsNotAffected() throws Exception {
- // Admission rejects REST catalogs, so a REST catalog can never hold a job and the
- // guard is a natural no-op for it.
- Fixture fixture = new Fixture("rest");
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of("warehouse", "s3://bucket/elsewhere"));
- fixture.catalogMgr.dropCatalog(CATALOG_NAME, false);
- Assertions.assertNull(fixture.catalogMgr.getCatalog(CATALOG_NAME));
- }
- }
-
- @Test
- public void testReplayAlterCatalogPropsIsNotGuarded() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- fixture.admitUnresolvedJob();
- CatalogLog log = new CatalogLog();
- log.setCatalogId(CATALOG_ID);
- log.setNewProps(ImmutableMap.of("warehouse", "s3://bucket/elsewhere"));
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- // Replay must apply verbatim: the guard lives only on the master DDL path.
- fixture.catalogMgr.replayAlterCatalogProps(log, null, true);
- Mockito.verify(fixture.catalog).advanceIndexTargetVersion();
- Mockito.verify(fixture.catalog).modifyCatalogProps(log.getNewProps());
- }
- }
-
- @Test
- public void targetChangeAfterCaptureRejectsAdmission() throws Exception {
- for (String key : IDENTITY_KEYS) {
- Fixture fixture = new Fixture("filesystem");
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- CatalogMgr.LanceIndexTarget target = fixture.catalogMgr.captureLanceIndexTarget(fixture.catalog);
- fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of(key, fixture.oldProperties.get(key) + "_changed"));
- DdlException error = Assertions.assertThrows(DdlException.class,
- () -> fixture.catalogMgr.withLanceIndexAdmission(fixture.catalog, target, () -> {
- fixture.admitUnresolvedJob();
- return null;
- }));
- Assertions.assertTrue(error.getMessage().contains("target changed"));
- Assertions.assertEquals(0, fixture.jobManager.getJobCount());
- }
- }
- }
-
- @Test
- public void targetChangeAndRestoreRejectsSnapshotOfIntermediateTarget() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- CatalogMgr.LanceIndexTarget target = fixture.catalogMgr.captureLanceIndexTarget(fixture.catalog);
- fixture.catalogMgr.alterCatalogProps(CATALOG_NAME, ImmutableMap.of("warehouse", "s3://bucket/B"));
- // A lock-free snapshot loader can read B before another ALTER restores A.
- fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of("warehouse", "s3://bucket/warehouse"));
- Assertions.assertThrows(DdlException.class,
- () -> fixture.catalogMgr.withLanceIndexAdmission(fixture.catalog, target, () -> {
- fixture.admitUnresolvedJob();
- return null;
- }));
- Assertions.assertEquals(0, fixture.jobManager.getJobCount());
- }
- }
-
- @Test
- public void failedIdentityAlterInvalidatesAdmissionEvenAfterRollback() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- Map<String, String> tentative = new HashMap<>(fixture.oldProperties);
- tentative.put("warehouse", "s3://bucket/temporary");
- Mockito.doAnswer(invocation -> {
- Mockito.when(fixture.catalog.getProperties()).thenReturn(tentative);
- return null;
- }).when(fixture.catalog).tryModifyCatalogProps(Mockito.anyMap());
- Mockito.doThrow(new DdlException("invalid target")).when(fixture.catalog).checkProperties();
- Mockito.doAnswer(invocation -> {
- Mockito.when(fixture.catalog.getProperties()).thenReturn(fixture.oldProperties);
- return null;
- }).when(fixture.metaCacheMgr).rollbackCatalogProperties(Mockito.eq(fixture.catalog), Mockito.anyMap());
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- CatalogMgr.LanceIndexTarget target = fixture.catalogMgr.captureLanceIndexTarget(fixture.catalog);
- Assertions.assertThrows(DdlException.class, () -> fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of("warehouse", "s3://bucket/temporary")));
- Assertions.assertSame(fixture.oldProperties, fixture.catalog.getProperties());
- Assertions.assertThrows(DdlException.class,
- () -> fixture.catalogMgr.withLanceIndexAdmission(fixture.catalog, target, () -> {
- fixture.admitUnresolvedJob();
- return null;
- }));
- Assertions.assertEquals(0, fixture.jobManager.getJobCount());
- }
- }
-
- @Test
- public void sameTargetAndCredentialChangesPreserveAdmission() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- CatalogMgr.LanceIndexTarget target = fixture.catalogMgr.captureLanceIndexTarget(fixture.catalog);
- fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of("warehouse", "s3://bucket/warehouse", "s3.access_key", "new_key"));
- fixture.catalogMgr.withLanceIndexAdmission(fixture.catalog, target, () -> {
- fixture.admitUnresolvedJob();
- return null;
- });
- Assertions.assertEquals(1, fixture.jobManager.getJobCount());
- }
- }
-
- @Test
- public void dropAfterCaptureRejectsAdmission() throws Exception {
- Fixture fixture = new Fixture("filesystem");
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- CatalogMgr.LanceIndexTarget target = fixture.catalogMgr.captureLanceIndexTarget(fixture.catalog);
- fixture.catalogMgr.dropCatalog(CATALOG_NAME, false);
- Assertions.assertThrows(DdlException.class,
- () -> fixture.catalogMgr.withLanceIndexAdmission(fixture.catalog, target, () -> {
- fixture.admitUnresolvedJob();
- return null;
- }));
- Assertions.assertEquals(0, fixture.jobManager.getJobCount());
- }
- }
-
- @Test
- public void concurrentDdlWaitsForAdmissionAndSeesItsUnresolvedJob() throws Exception {
- // Both DDL paths must be excluded throughout the final admission transfer.
- for (boolean drop : new boolean[] {true, false}) {
- Fixture fixture = new Fixture("filesystem");
- CatalogMgr.LanceIndexTarget target = fixture.catalogMgr.captureLanceIndexTarget(fixture.catalog);
- CountDownLatch admissionEntered = new CountDownLatch(1);
- CountDownLatch finishAdmission = new CountDownLatch(1);
- CountDownLatch ddlStarted = new CountDownLatch(1);
- ExecutorService executor = Executors.newFixedThreadPool(2);
- try {
- Future<?> admission = executor.submit(() -> {
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- return fixture.catalogMgr.withLanceIndexAdmission(fixture.catalog, target, () -> {
- admissionEntered.countDown();
- Assertions.assertTrue(finishAdmission.await(10, TimeUnit.SECONDS));
- fixture.admitUnresolvedJob();
- return null;
- });
- }
- });
- Assertions.assertTrue(admissionEntered.await(10, TimeUnit.SECONDS));
- Future<DdlException> ddl = executor.submit(() -> {
- try (MockedStatic<Env> mockedEnv = fixture.mockCurrentEnv()) {
- ddlStarted.countDown();
- return Assertions.assertThrows(DdlException.class, () -> {
- if (drop) {
- fixture.catalogMgr.dropCatalog(CATALOG_NAME, false);
- } else {
- fixture.catalogMgr.alterCatalogProps(CATALOG_NAME,
- ImmutableMap.of("warehouse", "s3://bucket/new"));
- }
- });
- }
- });
- Assertions.assertTrue(ddlStarted.await(10, TimeUnit.SECONDS));
- Assertions.assertThrows(TimeoutException.class, () -> ddl.get(200, TimeUnit.MILLISECONDS));
- finishAdmission.countDown();
- admission.get(10, TimeUnit.SECONDS);
- assertNeutralGuardMessage(ddl.get(10, TimeUnit.SECONDS));
- Assertions.assertSame(fixture.catalog, fixture.catalogMgr.getCatalog(CATALOG_ID));
- Assertions.assertEquals("s3://bucket/warehouse", fixture.oldProperties.get("warehouse"));
- } finally {
- finishAdmission.countDown();
- executor.shutdownNow();
- Assertions.assertTrue(executor.awaitTermination(10, TimeUnit.SECONDS));
- }
- }
}
- private static class TestJobManager extends LanceIndexJobManager {
- @Override
- protected void writeEditLog(LanceIndexJob job) {
- // No journal in a pure DDL-guard unit test.
+ private static Map<String, String> props(String... keyValue) {
+ Map<String, String> properties = new HashMap<>();
+ for (int i = 0; i < keyValue.length; i += 2) {
+ properties.put(keyValue[i], keyValue[i + 1]);
}
+ return properties;
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceCatalogLifecycleTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceCatalogLifecycleTest.java
index 8b4dbd73660..14d3f8de3f7 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceCatalogLifecycleTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceCatalogLifecycleTest.java
@@ -25,7 +25,6 @@ import org.apache.doris.datasource.ExternalDatabase;
import org.apache.doris.datasource.ExternalMetaCacheMgr;
import org.apache.doris.datasource.ExternalObjectLog;
import org.apache.doris.datasource.ExternalTable;
-import org.apache.doris.datasource.lance.job.LanceIndexDatasetLocator;
import org.apache.doris.persist.EditLog;
import org.apache.arrow.memory.BufferAllocator;
@@ -291,25 +290,6 @@ public class LanceCatalogLifecycleTest {
}
}
- @Test
- public void testIndexJobLocatorBypassesQueryAccessCache() throws Exception {
- LanceNamespace namespace = Mockito.mock(LanceNamespace.class);
- Mockito.when(namespace.describeTable(Mockito.any())).thenReturn(
- new DescribeTableResponse().tableUri("file:///warehouse/original.lance"),
- new DescribeTableResponse().tableUri("file:///warehouse/replacement.lance"));
- try (LanceCatalogClient client = new LanceCatalogClient(namespace, Mockito.mock(BufferAllocator.class),
- Mockito.mock(Session.class), "filesystem", "default", Collections.emptyList(),
- Collections.emptyList(), Collections.emptyMap(), Collections.emptyList())) {
- Field field = LanceCatalogClient.class.getDeclaredField("namespaceClient");
- field.setAccessible(true);
- LanceNamespaceClient namespaceClient = (LanceNamespaceClient) field.get(client);
- namespaceClient.resolveTableAccess("default", "items");
- Assertions.assertEquals(LanceIndexDatasetLocator.normalize("file:///warehouse/replacement.lance"),
- client.resolveCurrentIndexJobLocator("default", "items"));
- Mockito.verify(namespace, Mockito.times(2)).describeTable(Mockito.any());
- }
- }
-
@Test
public void testRoutineDatabaseObjectCleanupPreservesHotAccess() throws Exception {
try (AccessFixture fixture = new AccessFixture()) {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexAdmissionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexAdmissionTest.java
index 26ac7d69d7a..e85f971cc9f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexAdmissionTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexAdmissionTest.java
@@ -23,18 +23,12 @@ import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.Env;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.datasource.CatalogMgr;
import org.apache.doris.datasource.lance.LanceIndexAdmissionSnapshot.PhysicalIndexInfo;
import org.apache.doris.datasource.lance.index.LanceShowIndexInfo;
-import org.apache.doris.datasource.lance.job.LanceIndexFenceKey;
-import org.apache.doris.datasource.lance.job.LanceIndexJob;
-import org.apache.doris.datasource.lance.job.LanceIndexJobManager;
-import org.apache.doris.datasource.lance.job.LanceIndexJobMutationType;
-import org.apache.doris.datasource.lance.job.LanceIndexSchemaContract;
import org.apache.doris.nereids.trees.plans.commands.info.IndexDefinition;
import org.apache.doris.qe.ConnectContext;
@@ -51,21 +45,19 @@ import org.lance.schema.LanceField;
import org.mockito.MockedStatic;
import org.mockito.Mockito;
-import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
-import java.util.Locale;
import java.util.Map;
-import java.util.concurrent.atomic.AtomicLong;
/**
* Section 2.2/4.1 admission coverage for {@link LanceIndexAdmission}: snapshots are injected
- * through the {@code SnapshotLoader} seam (no FE, no JNI), the job manager is the real 3B
- * implementation with its edit-log seam captured, and {@code Env} is mocked only for id
- * allocation and the manager getter. Every rejection path must leave no job, no fence, no
- * quota charge, no journal record, and no id allocation behind (the 3B ReplayTest pattern).
+ * through the {@code SnapshotLoader} seam (no FE, no JNI) and {@code Env} is mocked only for
+ * the catalog manager. Every rejection path must leave no durable state and no id allocation
+ * behind. A mutation that passes the whole preflight terminates with the shared
+ * not-supported rejection (the synchronous execution path has not landed), while an IF
+ * preflight no-op returns normally.
*/
public class LanceIndexAdmissionTest {
private static final long CATALOG_ID = 10L;
@@ -77,9 +69,6 @@ public class LanceIndexAdmissionTest {
private static final long DATASET_VERSION = 7L;
private static final String MATCHING_ANN_PROPERTIES_JSON =
"{\"compression\":{\"num_bits\":8,\"num_sub_vectors\":16},\"metric_type\":\"l2\"}";
- private static final String NORMALIZED_ANN_PROPERTIES_JSON =
- "{\"index_type\":\"IVF_PQ\",\"metric\":\"l2\",\"num_bits\":\"8\","
- + "\"num_partitions\":\"256\",\"num_sub_vectors\":\"16\"}";
/**
* The KELVIN SIGN (U+212A) and the A-with-diaeresis pair: {@code String.equalsIgnoreCase}
* folds both pairs case-insensitively (the Kelvin sign through its per-character
@@ -91,26 +80,17 @@ public class LanceIndexAdmissionTest {
private MockedStatic<Env> mockedEnv;
private Env env;
- private TestManager manager;
- private AtomicLong idAllocator;
private LanceExternalCatalog catalog;
private LanceExternalDatabase database;
private LanceExternalTable table;
private ConnectContext connectContext;
private final Map<String, Column> tableColumns = new HashMap<>();
- private long originalTableQuota;
- private long originalCatalogQuota;
- private long originalGlobalQuota;
@BeforeEach
public void setUp() throws Exception {
mockedEnv = Mockito.mockStatic(Env.class);
env = Mockito.mock(Env.class);
- manager = new TestManager();
- idAllocator = new AtomicLong(100L);
mockedEnv.when(Env::getCurrentEnv).thenReturn(env);
- Mockito.when(env.getNextId()).thenAnswer(invocation -> idAllocator.incrementAndGet());
- Mockito.when(env.getLanceIndexJobManager()).thenReturn(manager);
catalog = Mockito.mock(LanceExternalCatalog.class);
Mockito.when(catalog.getId()).thenReturn(CATALOG_ID);
@@ -146,17 +126,10 @@ public class LanceIndexAdmissionTest {
connectContext = new ConnectContext();
connectContext.setThreadLocalInfo();
connectContext.setCurrentUserIdentity(UserIdentity.createAnalyzedUserIdentWithIp("tester", "%"));
-
- originalTableQuota = Config.lance_index_job_max_unresolved_per_table;
- originalCatalogQuota = Config.lance_index_job_max_unresolved_per_catalog;
- originalGlobalQuota = Config.lance_index_job_max_unresolved_global;
}
@AfterEach
public void tearDown() {
- Config.lance_index_job_max_unresolved_per_table = originalTableQuota;
- Config.lance_index_job_max_unresolved_per_catalog = originalCatalogQuota;
- Config.lance_index_job_max_unresolved_global = originalGlobalQuota;
ConnectContext.remove();
mockedEnv.close();
}
@@ -165,16 +138,6 @@ public class LanceIndexAdmissionTest {
// Fixtures
// ------------------------------------------------------------------
- /** Edit-log seam: captures durable records instead of writing the journal (3B pattern). */
- private static class TestManager extends LanceIndexJobManager {
- private final List<LanceIndexJob> editLog = new ArrayList<>();
-
- @Override
- protected void writeEditLog(LanceIndexJob job) {
- editLog.add(job);
- }
- }
-
private static Column notNullColumn(String name, Type type) {
return new Column(name, type, false, null, false, null, "");
}
@@ -268,92 +231,46 @@ public class LanceIndexAdmissionTest {
new HashMap<>(), "", orReplace);
}
- private LanceIndexAdmission.Outcome admitCreate(LanceIndexAdmissionSnapshot snapshot,
+ private void admitCreate(LanceIndexAdmissionSnapshot snapshot,
IndexDefinition def, boolean ifNotExists) throws Exception {
- return LanceIndexAdmission.admitCreate((cat, dbName, tblName) -> snapshot, catalog, database,
+ LanceIndexAdmission.admitCreate((cat, dbName, tblName) -> snapshot, catalog, database,
table, def, ifNotExists);
}
- private LanceIndexAdmission.Outcome admitDrop(LanceIndexAdmissionSnapshot snapshot,
+ private void admitDrop(LanceIndexAdmissionSnapshot snapshot,
String indexName, boolean ifExists) throws Exception {
- return LanceIndexAdmission.admitDrop((cat, dbName, tblName) -> snapshot, catalog, database,
+ LanceIndexAdmission.admitDrop((cat, dbName, tblName) -> snapshot, catalog, database,
table, indexName, ifExists);
}
private void assertNothingPersisted() {
- Assertions.assertEquals(0, manager.getJobCount());
- Assertions.assertTrue(manager.editLog.isEmpty());
- Assertions.assertTrue(manager.getUnresolvedJobs().isEmpty());
Mockito.verify(env, Mockito.never()).getNextId();
}
- /** Unresolved-job count on this dataset locator — the public view of the table quota. */
- private int unresolvedJobsOnDataset() {
- int count = 0;
- for (LanceIndexJob job : manager.getUnresolvedJobs()) {
- if (job.getCatalogId() == CATALOG_ID && DATASET_URI.equals(job.getNormalizedLocator())) {
- count++;
- }
- }
- return count;
- }
-
private static void assertInvalid(Executable call, String expectedMessage) {
AnalysisException exception = Assertions.assertThrows(AnalysisException.class, call);
Assertions.assertEquals(expectedMessage, exception.getDetailMessage());
Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_INVALID, exception.getMysqlErrorCode());
}
+ /** A mutation whose whole preflight passed terminates with the shared not-supported error. */
+ private static void assertUnsupported(Executable call, String operation) {
+ AnalysisException exception = Assertions.assertThrows(AnalysisException.class, call);
+ Assertions.assertEquals(operation + " is not supported for Lance catalog tables",
+ exception.getDetailMessage());
+ Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED,
+ exception.getMysqlErrorCode());
+ }
+
// ------------------------------------------------------------------
// CREATE admission
// ------------------------------------------------------------------
@Test
- public void plainCreateOnEmptySnapshotIsAdmitted() throws Exception {
- LanceIndexAdmission.Outcome outcome = admitCreate(emptySnapshot(), annDef("MyIdx", false, false),
- false);
-
- Assertions.assertNotNull(outcome.getJobId());
- LanceIndexJob job = manager.getJob(outcome.getJobId());
- Assertions.assertNotNull(job);
- Assertions.assertEquals("tester", job.getCreator());
- Assertions.assertEquals(CATALOG_ID, job.getCatalogId());
- // Local names are persisted for privilege/FORCE resolution, not the remote ones.
- Assertions.assertEquals(DB, job.getDbName());
- Assertions.assertEquals(TBL, job.getTableName());
- Assertions.assertEquals(LanceIndexFenceKey.PROVIDER_DIRECTORY, job.getProvider());
- Assertions.assertEquals(DATASET_URI, job.getNormalizedLocator());
- Assertions.assertEquals("MyIdx", job.getDisplayIndexName());
- Assertions.assertEquals("myidx", job.getNormalizedIndexName());
- Assertions.assertEquals(LanceIndexJobMutationType.CREATE, job.getMutationType());
- Assertions.assertFalse(job.isIfNotExists());
- Assertions.assertFalse(job.isIfExists());
- Assertions.assertEquals("IVF_PQ", job.getIndexType());
- Assertions.assertEquals("v", job.getColumnName());
- Assertions.assertEquals(NORMALIZED_ANN_PROPERTIES_JSON, job.getPropertiesJson());
- Assertions.assertEquals(DATASET_VERSION, job.getAdmittedDatasetVersion());
-
- LanceIndexSchemaContract contract = job.getSchemaContract();
- Assertions.assertNotNull(contract);
- Assertions.assertEquals(LanceIndexSchemaContract.SCHEMA_CONTRACT_VERSION_V1,
- contract.getSchemaContractVersion());
- Assertions.assertEquals(1, contract.getFields().size());
- LanceIndexSchemaContract.IndexedField field = contract.getFields().get(0);
- Assertions.assertEquals(1L, field.getFieldId());
- Assertions.assertEquals("v", field.getNormalizedName());
- Assertions.assertEquals("fixed_size_list", field.getNormalizedType());
- Assertions.assertFalse(field.isNullable());
- Assertions.assertEquals(4, field.getFixedSizeListDimension());
- Assertions.assertEquals("float32", field.getVectorElementType());
- // The synthesized element child is always nullable — the manifest has no slot for
- // element nullability — so the contract records the reconstructed-schema fact.
- Assertions.assertEquals(Boolean.TRUE, field.getVectorElementNullable());
-
- Assertions.assertEquals(1, manager.getJobCount());
- Assertions.assertEquals(1, manager.editLog.size());
- Assertions.assertTrue(manager.isFenceHeld(job.fenceKey()));
- Assertions.assertEquals(1, manager.getUnresolvedJobs().size());
- Mockito.verify(env, Mockito.times(1)).getNextId();
+ public void plainCreateOnEmptySnapshotValidatesThenRejectsAsUnsupported() {
+ assertUnsupported(() -> admitCreate(emptySnapshot(), annDef("MyIdx", false, false), false),
+ "CREATE INDEX");
+ assertNothingPersisted();
}
@Test
@@ -364,10 +281,8 @@ public class LanceIndexAdmissionTest {
Mockito.when(catalog.loadTableIndexAdmissionSnapshot(REMOTE_DB, REMOTE_TBL))
.thenReturn(prepared);
- LanceIndexAdmission.Outcome outcome = LanceIndexAdmission.admitCreate(catalog, database, table,
- scalarDef("Idx", "BTREE", "c", false, false), false);
-
- Assertions.assertNotNull(outcome.getJobId());
+ assertUnsupported(() -> LanceIndexAdmission.admitCreate(catalog, database, table,
+ scalarDef("Idx", "BTREE", "c", false, false), false), "CREATE INDEX");
Mockito.verify(catalog, Mockito.times(1))
.loadTableIndexAdmissionSnapshot(REMOTE_DB, REMOTE_TBL);
}
@@ -392,9 +307,7 @@ public class LanceIndexAdmissionTest {
Collections.singletonList(logicalIndex("IdxA", "v", "IVF_PQ", MATCHING_ANN_PROPERTIES_JSON)),
Collections.singletonList(physicalIndex("IdxA", "VECTOR")));
- LanceIndexAdmission.Outcome outcome = admitCreate(snapshot, annDef("idxa", true, false), true);
-
- Assertions.assertNull(outcome.getJobId());
+ Assertions.assertDoesNotThrow(() -> admitCreate(snapshot, annDef("idxa", true, false), true));
assertNothingPersisted();
}
@@ -408,10 +321,8 @@ public class LanceIndexAdmissionTest {
Collections.singletonList(logicalIndex("idx", "c", "BTREE", "[]")),
Collections.singletonList(physicalIndex("idx", "SCALAR")));
- LanceIndexAdmission.Outcome outcome = admitCreate(snapshot,
- scalarDef("idx", "BTREE", "c", true, false), true);
-
- Assertions.assertNull(outcome.getJobId());
+ Assertions.assertDoesNotThrow(() -> admitCreate(snapshot,
+ scalarDef("idx", "BTREE", "c", true, false), true));
assertNothingPersisted();
}
@@ -442,8 +353,8 @@ public class LanceIndexAdmissionTest {
MATCHING_ANN_PROPERTIES_JSON)),
Collections.singletonList(physicalIndex("idx", "VECTOR")),
Collections.singletonList(vectorField(fieldName, 1)));
- Assertions.assertNull(admitCreate(snapshot,
- annDef("idx", fieldName, true, false), true).getJobId(), fieldName);
+ Assertions.assertDoesNotThrow(() -> admitCreate(snapshot,
+ annDef("idx", fieldName, true, false), true), fieldName);
}
assertNothingPersisted();
}
@@ -519,7 +430,7 @@ public class LanceIndexAdmissionTest {
Collections.singletonList(logicalIndex("idx", "v", "IVF_PQ", "{}")),
Collections.singletonList(physicalIndex("idx", "VECTOR")));
- Assertions.assertNull(admitCreate(snapshot, annDef("idx", true, false), true).getJobId());
+ Assertions.assertDoesNotThrow(() -> admitCreate(snapshot, annDef("idx", true, false), true));
assertNothingPersisted();
}
@@ -545,7 +456,8 @@ public class LanceIndexAdmissionTest {
Collections.singletonList(logicalIndex("idx", "v", "IVF_PQ",
"{\"compression\":{\"num_bits\":8}}")),
Collections.singletonList(physicalIndex("idx", "VECTOR")));
- Assertions.assertNull(admitCreate(partialExposure, annDef("idx", true, false), true).getJobId());
+ Assertions.assertDoesNotThrow(() -> admitCreate(partialExposure,
+ annDef("idx", true, false), true));
// An exposed but non-numeric value is malformed data: fail closed, never skip.
LanceIndexAdmissionSnapshot nonNumeric = snapshot(
@@ -559,7 +471,7 @@ public class LanceIndexAdmissionTest {
@Test
public void omittedNumBitsComparesAsThePersistedEight() throws Exception {
- // The validator accepts an omitted num_bits and admission always persists 8, so the
+ // The validator accepts an omitted num_bits and the persisted value is always 8, so the
// preflight compares an effective 8 — never skips the property — against any exposed
// compression.num_bits.
LanceIndexAdmissionSnapshot fourBits = snapshot(
@@ -573,7 +485,7 @@ public class LanceIndexAdmissionTest {
Collections.singletonList(logicalIndex("idx", "v", "IVF_PQ",
"{\"compression\":{\"num_bits\":8}}")),
Collections.singletonList(physicalIndex("idx", "VECTOR")));
- Assertions.assertNull(admitCreate(eightBits, annDef("idx", true, false), true).getJobId());
+ Assertions.assertDoesNotThrow(() -> admitCreate(eightBits, annDef("idx", true, false), true));
assertNothingPersisted();
}
@@ -584,8 +496,8 @@ public class LanceIndexAdmissionTest {
Collections.singletonList(logicalIndex("idx", "v", "IVF_PQ",
"{\"compression\":{\"num_sub_vectors\":" + exposedValue + "}}")),
Collections.singletonList(physicalIndex("idx", "VECTOR")));
- Assertions.assertNull(admitCreate(snapshot, annDef("idx", true, false), true).getJobId(),
- exposedValue);
+ Assertions.assertDoesNotThrow(() -> admitCreate(snapshot,
+ annDef("idx", true, false), true), exposedValue);
}
assertNothingPersisted();
}
@@ -658,52 +570,38 @@ public class LanceIndexAdmissionTest {
"{\"metric_type\":\"L2\"}")),
Collections.singletonList(physicalIndex("idx", "VECTOR")));
- Assertions.assertNull(admitCreate(snapshot, annDef("idx", true, false), true).getJobId());
+ Assertions.assertDoesNotThrow(() -> admitCreate(snapshot, annDef("idx", true, false), true));
assertNothingPersisted();
}
@Test
- public void replaceOnAbsentNameCreatesWithUserSpelling() throws Exception {
- LanceIndexAdmission.Outcome outcome = admitCreate(emptySnapshot(),
- annDef("Fresh", false, true), false);
-
- LanceIndexJob job = manager.getJob(outcome.getJobId());
- Assertions.assertEquals(LanceIndexJobMutationType.REPLACE, job.getMutationType());
- Assertions.assertEquals("Fresh", job.getDisplayIndexName());
- Assertions.assertEquals("fresh", job.getNormalizedIndexName());
+ public void replaceOnAbsentNameValidatesThenRejectsAsUnsupported() {
+ assertUnsupported(() -> admitCreate(emptySnapshot(), annDef("Fresh", false, true), false),
+ "CREATE OR REPLACE INDEX");
+ assertNothingPersisted();
}
@Test
- public void replaceWithCaseVariantPersistsTheStoredName() throws Exception {
- // M2: the worker locates the target case-sensitively, so the stored display name is
- // persisted instead of the user's spelling.
+ public void replaceWithCaseVariantResolvesThenRejectsAsUnsupported() {
+ // M2: a unique case-insensitive match resolves to the stored display name before the
+ // terminal rejection.
LanceIndexAdmissionSnapshot snapshot = snapshot(
Collections.singletonList(logicalIndex("IdxA", "v", "IVF_PQ", MATCHING_ANN_PROPERTIES_JSON)),
Collections.singletonList(physicalIndex("IdxA", "VECTOR")));
- LanceIndexAdmission.Outcome outcome = admitCreate(snapshot, annDef("IDXA", false, true), false);
-
- LanceIndexJob job = manager.getJob(outcome.getJobId());
- Assertions.assertEquals(LanceIndexJobMutationType.REPLACE, job.getMutationType());
- Assertions.assertEquals("IdxA", job.getDisplayIndexName());
- Assertions.assertEquals("idxa", job.getNormalizedIndexName());
+ assertUnsupported(() -> admitCreate(snapshot, annDef("IDXA", false, true), false),
+ "CREATE OR REPLACE INDEX");
+ assertNothingPersisted();
}
@Test
- public void mixedCaseColumnResolvesToTheStoredColumnName() throws Exception {
- // N5: static validation is case-insensitive; the contract and the persisted column name
- // use the stored Lance field name.
- LanceIndexAdmission.Outcome outcome = admitCreate(emptySnapshot(),
+ public void mixedCaseColumnResolvesUniquelyThroughThePreflight() {
+ // N5: static validation is case-insensitive; the schema-contract build keys on the
+ // stored Lance field name, so a mixed-case request resolves, passes the preflight,
+ // and ends in the shared not-supported rejection.
+ assertUnsupported(() -> admitCreate(emptySnapshot(),
new IndexDefinition("idx", false, Collections.singletonList("embedding"), "ANN",
- annProperties(), "", false),
- false);
-
- LanceIndexJob job = manager.getJob(outcome.getJobId());
- Assertions.assertEquals("Embedding", job.getColumnName());
- LanceIndexSchemaContract.IndexedField field = job.getSchemaContract().getFields().get(0);
- Assertions.assertEquals(4L, field.getFieldId());
- Assertions.assertEquals("embedding", field.getNormalizedName());
- Assertions.assertEquals("fixed_size_list", field.getNormalizedType());
+ annProperties(), "", false), false), "CREATE INDEX");
}
@Test
@@ -731,7 +629,7 @@ public class LanceIndexAdmissionTest {
// relation folds 'Ä' onto 'ä' and, through its Character.toLowerCase fallback, the KELVIN
// SIGN (U+212A) onto ASCII 'k', so a dataset holding such a pair is genuinely
// unresolvable by the lookup and fails closed; a lone KELVIN SIGN field still resolves
- // a 'k' request uniquely and admits without a false ambiguity.
+ // a 'k' request uniquely and passes the preflight without a false ambiguity.
LanceIndexAdmissionSnapshot kelvinPair = snapshotWithFields(
vectorField(KELVIN_SIGN, 1), vectorField("k", 2));
assertInvalid(() -> admitCreate(kelvinPair, annDef("idx", "k", false, false), false),
@@ -746,49 +644,19 @@ public class LanceIndexAdmissionTest {
+ "multiple Lance fields differ only by case");
assertNothingPersisted();
- // Unique resolution: exactly one lookup hit, and the stored field name is journaled.
+ // Unique resolution: exactly one lookup hit, so the request passes the preflight and
+ // ends in the shared not-supported rejection instead of the ambiguity error.
tableColumns.put(KELVIN_SIGN, notNullColumn(KELVIN_SIGN, new ArrayType(Type.FLOAT)));
- LanceIndexAdmission.Outcome outcome = admitCreate(snapshotWithFields(
- vectorField(KELVIN_SIGN, 1)), annDef("idx", "k", false, false), false);
- Assertions.assertNotNull(outcome.getJobId());
- Assertions.assertEquals(KELVIN_SIGN, manager.getJob(outcome.getJobId()).getColumnName());
+ assertUnsupported(() -> admitCreate(snapshotWithFields(
+ vectorField(KELVIN_SIGN, 1)), annDef("idx", "k", false, false), false),
+ "CREATE INDEX");
}
@Test
- public void btreeCreatePersistsNullPropertiesAndScalarContract() throws Exception {
- LanceIndexAdmission.Outcome outcome = admitCreate(emptySnapshot(),
- scalarDef("Idx", "BTREE", "c", false, false), false);
-
- LanceIndexJob job = manager.getJob(outcome.getJobId());
- Assertions.assertEquals("BTREE", job.getIndexType());
- Assertions.assertEquals("c", job.getColumnName());
- Assertions.assertNull(job.getPropertiesJson());
- LanceIndexSchemaContract.IndexedField field = job.getSchemaContract().getFields().get(0);
- Assertions.assertEquals("c", field.getNormalizedName());
- Assertions.assertEquals("int<32>", field.getNormalizedType());
- Assertions.assertNull(field.getFixedSizeListDimension());
- Assertions.assertNull(field.getVectorElementType());
- Assertions.assertNull(field.getVectorElementNullable());
- }
-
- @Test
- public void annPropertiesJsonIsNormalizedAndDeterministic() throws Exception {
- Map<String, String> mixedCase = new HashMap<>();
- mixedCase.put("Index_Type", "ivf_pq");
- mixedCase.put("METRIC", "L2");
- mixedCase.put("num_partitions", "256");
- mixedCase.put("num_sub_vectors", "16");
- mixedCase.put("num_bits", "8");
-
- LanceIndexAdmission.Outcome outcome = admitCreate(emptySnapshot(),
- annDef("idx", false, false, mixedCase), false);
-
- LanceIndexJob job = manager.getJob(outcome.getJobId());
- // Keys are lowercase, index_type is the uppercased original value, metric is lowercased,
- // numerics stay as their original strings, num_bits is pinned to 8, and the key order is
- // the TreeMap order — byte-for-byte deterministic.
- Assertions.assertEquals(NORMALIZED_ANN_PROPERTIES_JSON, job.getPropertiesJson());
- Assertions.assertEquals("IVF_PQ", job.getIndexType());
+ public void btreeCreateValidatesTheScalarContractThenRejectsAsUnsupported() {
+ assertUnsupported(() -> admitCreate(emptySnapshot(),
+ scalarDef("Idx", "BTREE", "c", false, false), false), "CREATE INDEX");
+ assertNothingPersisted();
}
@Test
@@ -853,130 +721,34 @@ public class LanceIndexAdmissionTest {
@Test
public void dropIfExistsWithAbsentNameIsNoOp() throws Exception {
- LanceIndexAdmission.Outcome outcome = admitDrop(emptySnapshot(), "nope", true);
-
- Assertions.assertNull(outcome.getJobId());
+ Assertions.assertDoesNotThrow(() -> admitDrop(emptySnapshot(), "nope", true));
assertNothingPersisted();
}
@Test
- public void dropExistingNameIsAdmittedWithDropShape() throws Exception {
+ public void dropExistingNameValidatesThenRejectsAsUnsupported() {
LanceIndexAdmissionSnapshot snapshot = snapshot(
Collections.singletonList(logicalIndex("IdxA", "v", "IVF_PQ", MATCHING_ANN_PROPERTIES_JSON)),
Collections.singletonList(physicalIndex("IdxA", "VECTOR")));
- LanceIndexAdmission.Outcome outcome = admitDrop(snapshot, "IdxA", true);
-
- LanceIndexJob job = manager.getJob(outcome.getJobId());
- Assertions.assertEquals(LanceIndexJobMutationType.DROP, job.getMutationType());
- Assertions.assertEquals("IdxA", job.getDisplayIndexName());
- Assertions.assertEquals("idxa", job.getNormalizedIndexName());
- Assertions.assertTrue(job.isIfExists());
- Assertions.assertFalse(job.isIfNotExists());
- Assertions.assertNull(job.getIndexType());
- Assertions.assertNull(job.getColumnName());
- Assertions.assertNull(job.getPropertiesJson());
- Assertions.assertNull(job.getSchemaContract());
- Assertions.assertEquals(DATASET_VERSION, job.getAdmittedDatasetVersion());
- Assertions.assertEquals(DATASET_URI, job.getNormalizedLocator());
+ assertUnsupported(() -> admitDrop(snapshot, "IdxA", true), "DROP INDEX");
+ assertNothingPersisted();
}
@Test
- public void dropWithCaseVariantPersistsTheStoredName() throws Exception {
+ public void dropWithCaseVariantResolvesThenRejectsAsUnsupported() {
LanceIndexAdmissionSnapshot snapshot = snapshot(
Collections.singletonList(logicalIndex("IdxA", "v", "IVF_PQ", MATCHING_ANN_PROPERTIES_JSON)),
Collections.singletonList(physicalIndex("IdxA", "VECTOR")));
- LanceIndexAdmission.Outcome outcome = admitDrop(snapshot, "idxa", false);
-
- LanceIndexJob job = manager.getJob(outcome.getJobId());
- Assertions.assertEquals("IdxA", job.getDisplayIndexName());
- Assertions.assertEquals("idxa", job.getNormalizedIndexName());
+ assertUnsupported(() -> admitDrop(snapshot, "idxa", false), "DROP INDEX");
+ assertNothingPersisted();
}
// ------------------------------------------------------------------
- // Fence, quota, and id discipline
+ // Target and snapshot discipline
// ------------------------------------------------------------------
- @Test
- public void fenceConflictPassesThroughTheManagerRejection() throws Exception {
- manager.createJob(newCreateJob(1L, "idxa"), 100, 100, 100);
-
- DdlException exception = Assertions.assertThrows(DdlException.class,
- () -> admitCreate(emptySnapshot(), annDef("idxA", false, false), false));
- Assertions.assertTrue(exception.getMessage().contains("fenced by unresolved job 1"),
- exception.getMessage());
- // The rejection never discloses the locator.
- Assertions.assertFalse(exception.getMessage().contains("bucket"), exception.getMessage());
- Assertions.assertFalse(exception.getMessage().contains(DATASET_URI), exception.getMessage());
-
- // One id was burned before the durable transfer rejected the job (accepted, ids are not
- // required to be contiguous); nothing else changed.
- Assertions.assertEquals(1, manager.getJobCount());
- Assertions.assertEquals(1, manager.editLog.size());
- Assertions.assertEquals(1, manager.getUnresolvedJobs().size());
- Mockito.verify(env, Mockito.times(1)).getNextId();
- }
-
- @Test
- public void quotaOverloadPassesThroughTheManagerRejection() throws Exception {
- Config.lance_index_job_max_unresolved_per_table = 1;
- manager.createJob(newCreateJob(1L, "idxz"), 100, 100, 100);
-
- DdlException exception = Assertions.assertThrows(DdlException.class,
- () -> admitCreate(emptySnapshot(), annDef("idxA", false, false), false));
- Assertions.assertTrue(exception.getMessage().contains("quota exceeded"), exception.getMessage());
-
- Assertions.assertEquals(1, manager.getJobCount());
- Assertions.assertEquals(1, manager.editLog.size());
- Assertions.assertEquals(1, manager.getUnresolvedJobs().size());
- Mockito.verify(env, Mockito.times(1)).getNextId();
- }
-
- @Test
- public void quotaBoundaryAdmitsAtTheLimitAndRejectsBeyondIt() throws Exception {
- Config.lance_index_job_max_unresolved_per_table = 2;
- manager.createJob(newCreateJob(1L, "idxz"), 100, 100, 100);
-
- // count + 1 == limit passes.
- LanceIndexAdmission.Outcome outcome = admitCreate(emptySnapshot(),
- annDef("idxA", false, false), false);
- Assertions.assertNotNull(outcome.getJobId());
- Assertions.assertEquals(2, manager.getJobCount());
-
- // count + 1 > limit is rejected.
- Assertions.assertThrows(DdlException.class,
- () -> admitCreate(emptySnapshot(), annDef("idxB", false, false), false));
- Assertions.assertEquals(2, manager.getJobCount());
- Assertions.assertEquals(2, unresolvedJobsOnDataset());
- }
-
- @Test
- public void nonPositiveQuotaConfigIsAssertedBeforeAnyIdAllocation() {
- // D7: a quota that reached fe.conf with a non-positive value (bypassing the ADMIN SET
- // callback) must fail admission closed with 5102 naming the config item.
- Config.lance_index_job_max_unresolved_per_table = 0;
- AnalysisException exception = Assertions.assertThrows(AnalysisException.class,
- () -> admitCreate(emptySnapshot(), annDef("idxA", false, false), false));
- Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_MUTATION_DISABLED, exception.getMysqlErrorCode());
- Assertions.assertTrue(exception.getDetailMessage().contains("lance_index_job_max_unresolved_per_table"),
- exception.getDetailMessage());
- assertNothingPersisted();
-
- Config.lance_index_job_max_unresolved_per_table = originalTableQuota;
- Config.lance_index_job_max_unresolved_global = -1;
- exception = Assertions.assertThrows(AnalysisException.class,
- () -> admitDrop(snapshot(
- Collections.singletonList(
- logicalIndex("idxA", "v", "IVF_PQ", MATCHING_ANN_PROPERTIES_JSON)),
- Collections.singletonList(physicalIndex("idxA", "VECTOR"))),
- "idxA", false));
- Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_MUTATION_DISABLED, exception.getMysqlErrorCode());
- Assertions.assertTrue(exception.getDetailMessage().contains("lance_index_job_max_unresolved_global"),
- exception.getDetailMessage());
- assertNothingPersisted();
- }
-
@Test
public void identityChangeDuringSnapshotRejectsCreateAndDropBeforeAllocatingId() throws Exception {
LanceIndexAdmission.SnapshotLoader changingLoader = (cat, dbName, tblName) -> {
@@ -1010,34 +782,4 @@ public class LanceIndexAdmissionTest {
() -> LanceIndexAdmission.admitDrop(failingLoader, catalog, database, table, "idx", false));
assertNothingPersisted();
}
-
- @Test
- public void mixedPopulationAdmissionsChargeFenceAndQuotaCorrectly() throws Exception {
- LanceIndexAdmissionSnapshot withExisting = snapshot(
- Collections.singletonList(logicalIndex("IdxC", "c", "BTREE", "{}")),
- Collections.singletonList(physicalIndex("IdxC", "SCALAR")));
-
- LanceIndexAdmission.Outcome createA = admitCreate(emptySnapshot(),
- annDef("idxA", false, false), false);
- LanceIndexAdmission.Outcome createB = admitCreate(emptySnapshot(),
- scalarDef("idxB", "BITMAP", "s", false, false), false);
- LanceIndexAdmission.Outcome dropC = admitDrop(withExisting, "idxc", false);
-
- Assertions.assertEquals(3, manager.getJobCount());
- Assertions.assertEquals(3, manager.editLog.size());
- Assertions.assertEquals(3, manager.getUnresolvedJobs().size());
- Assertions.assertEquals(3, unresolvedJobsOnDataset());
- Assertions.assertTrue(manager.isFenceHeld(manager.getJob(createA.getJobId()).fenceKey()));
- Assertions.assertTrue(manager.isFenceHeld(manager.getJob(createB.getJobId()).fenceKey()));
- Assertions.assertTrue(manager.isFenceHeld(manager.getJob(dropC.getJobId()).fenceKey()));
- Assertions.assertEquals("IdxC", manager.getJob(dropC.getJobId()).getDisplayIndexName());
- }
-
- private static LanceIndexJob newCreateJob(long jobId, String displayName) {
- return new LanceIndexJob(jobId, "seeder", CATALOG_ID, DB, TBL,
- LanceIndexFenceKey.PROVIDER_DIRECTORY, DATASET_URI,
- displayName, displayName.toLowerCase(Locale.ROOT),
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v",
- null, DATASET_VERSION, null);
- }
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexNameNormalizerTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexNameNormalizerTest.java
similarity index 99%
rename from fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexNameNormalizerTest.java
rename to fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexNameNormalizerTest.java
index d2ad765697c..c6b666e055d 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexNameNormalizerTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceIndexNameNormalizerTest.java
@@ -15,7 +15,7 @@
// specific language governing permissions and limitations
// under the License.
-package org.apache.doris.datasource.lance.job;
+package org.apache.doris.datasource.lance;
import org.apache.commons.lang3.StringUtils;
import org.junit.jupiter.api.Assertions;
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilderRealDatasetTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilderRealDatasetTest.java
index 816875a0d11..713657d92fb 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilderRealDatasetTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilderRealDatasetTest.java
@@ -17,7 +17,6 @@
package org.apache.doris.datasource.lance;
-import org.apache.doris.datasource.lance.job.LanceIndexSchemaContract;
import org.apache.arrow.memory.BufferAllocator;
import org.apache.arrow.memory.RootAllocator;
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilderTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilderTest.java
index 1e14b43771f..b537c400a18 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilderTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSchemaContractBuilderTest.java
@@ -18,7 +18,6 @@
package org.apache.doris.datasource.lance;
import org.apache.doris.common.AnalysisException;
-import org.apache.doris.datasource.lance.job.LanceIndexSchemaContract;
import org.apache.arrow.vector.types.DateUnit;
import org.apache.arrow.vector.types.FloatingPointPrecision;
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexDatasetLocatorTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexDatasetLocatorTest.java
deleted file mode 100644
index 2e46203b815..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexDatasetLocatorTest.java
+++ /dev/null
@@ -1,183 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.apache.commons.lang3.StringUtils;
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-/**
- * Unit coverage for dataset-locator normalization v1, the fence-key identity for the
- * target dataset: trim, lowercase scheme, userinfo rejection (credential-bearing URLs
- * are never identity), rejection of locators with neither authority nor path,
- * trailing-slash stripping with a kept root, and the absolute-path requirement for
- * scheme-less locators.
- */
-public class LanceIndexDatasetLocatorTest {
-
- @Test
- public void lowercasesSchemeOnly() {
- Assertions.assertEquals("s3://bucket/path", LanceIndexDatasetLocator.normalize("S3://bucket/path"));
- Assertions.assertEquals("file:///data/x", LanceIndexDatasetLocator.normalize("FILE:///data/x"));
- Assertions.assertEquals("hdfs://nn:8020/data", LanceIndexDatasetLocator.normalize("HDFS://nn:8020/data"));
- }
-
- @Test
- public void preservesAuthorityAndPathCase() {
- // Bucket and path components are case-sensitive on the supported providers.
- Assertions.assertEquals("s3://MyBucket/MyPath", LanceIndexDatasetLocator.normalize("s3://MyBucket/MyPath"));
- }
-
- @Test
- public void trimsSurroundingWhitespace() {
- Assertions.assertEquals("/data/x", LanceIndexDatasetLocator.normalize(" /data/x \t\n"));
- Assertions.assertEquals("s3://bucket/p", LanceIndexDatasetLocator.normalize(" s3://bucket/p "));
- }
-
- @Test
- public void stripsTrailingSlashesFromPath() {
- Assertions.assertEquals("s3://b/p", LanceIndexDatasetLocator.normalize("s3://b/p///"));
- Assertions.assertEquals("/data/lance", LanceIndexDatasetLocator.normalize("/data/lance/"));
- }
-
- @Test
- public void keepsSchemeLessRoot() {
- Assertions.assertEquals("/", LanceIndexDatasetLocator.normalize("/"));
- Assertions.assertEquals("/", LanceIndexDatasetLocator.normalize("//"));
- Assertions.assertEquals("/", LanceIndexDatasetLocator.normalize("///"));
- }
-
- @Test
- public void stripsTrailingSlashBehindAuthority() {
- Assertions.assertEquals("s3://bucket", LanceIndexDatasetLocator.normalize("s3://bucket/"));
- Assertions.assertEquals("s3://bucket", LanceIndexDatasetLocator.normalize("s3://bucket//"));
- Assertions.assertEquals("s3://bucket", LanceIndexDatasetLocator.normalize("s3://bucket"));
- }
-
- @Test
- public void acceptsFileUrlWithEmptyAuthority() {
- Assertions.assertEquals("file:///data/x", LanceIndexDatasetLocator.normalize("file:///data/x"));
- }
-
- @Test
- public void keepsAuthorityPortAndQueryFreePathVerbatim() {
- Assertions.assertEquals("hdfs://nn:8020/base/table.lance",
- LanceIndexDatasetLocator.normalize("hdfs://nn:8020/base/table.lance"));
- }
-
- @Test
- public void allowsAtSignInPathButNotInAuthority() {
- Assertions.assertEquals("s3://bucket/p@th", LanceIndexDatasetLocator.normalize("s3://bucket/p@th"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("s3://user@bucket/path"));
- }
-
- @Test
- public void rejectsCredentialBearingUserinfo() {
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("s3://user:secret@bucket/path"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("https://user:secret@example.com/ds"));
- }
-
- @Test
- public void rejectsQueriesAndFragmentsThatCouldCarryCredentials() {
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize(
- "https://bucket.example/ds?X-Amz-Credential=AKIA_TEST&X-Amz-Signature=secret"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("https://account.blob.core.windows.net/ds?sig=secret"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("s3://bucket/ds#credential-fragment"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("/data/ds?token=secret"));
- }
-
- @Test
- public void malformedLocatorErrorsDoNotEchoPotentialSecrets() {
- String malformed = "s 3://user:top-secret@bucket/path";
- IllegalArgumentException malformedError = Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize(malformed));
- Assertions.assertFalse(malformedError.getMessage().contains(malformed));
- Assertions.assertFalse(malformedError.getMessage().contains("top-secret"));
-
- String relative = "access-key:top-secret@bucket/path";
- IllegalArgumentException relativeError = Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize(relative));
- Assertions.assertFalse(relativeError.getMessage().contains(relative));
- Assertions.assertFalse(relativeError.getMessage().contains("top-secret"));
- }
-
- @Test
- public void rejectsMalformedOrNonHierarchicalSchemes() {
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("1s3://bucket/path"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("s3:/bucket/path"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("s3:bucket/path"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("https://[malformed/path"));
- }
-
- @Test
- public void locatorBoundCountsUtf8Bytes() {
- String prefix = "s3://bucket/";
- String atLimit = prefix + StringUtils.repeat(
- "a", LanceIndexDatasetLocator.MAX_LOCATOR_BYTES - prefix.length());
- Assertions.assertEquals(atLimit, LanceIndexDatasetLocator.normalize(atLimit));
-
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize(atLimit + "a"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize(prefix + StringUtils.repeat("é",
- (LanceIndexDatasetLocator.MAX_LOCATOR_BYTES - prefix.length()) / 2 + 1)));
- }
-
- @Test
- public void rejectsSchemeLessRelativePath() {
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("data/x"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("./x"));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> LanceIndexDatasetLocator.normalize("bucket/path"));
- }
-
- @Test
- public void rejectsNullEmptyAndBlank() {
- Assertions.assertThrows(IllegalArgumentException.class, () -> LanceIndexDatasetLocator.normalize(null));
- Assertions.assertThrows(IllegalArgumentException.class, () -> LanceIndexDatasetLocator.normalize(""));
- Assertions.assertThrows(IllegalArgumentException.class, () -> LanceIndexDatasetLocator.normalize(" "));
- }
-
- @Test
- public void rejectsEmptyScheme() {
- Assertions.assertThrows(IllegalArgumentException.class, () -> LanceIndexDatasetLocator.normalize("://path"));
- }
-
- @Test
- public void rejectsLocatorWithNeitherAuthorityNorPath() {
- Assertions.assertThrows(IllegalArgumentException.class, () -> LanceIndexDatasetLocator.normalize("s3://"));
- Assertions.assertThrows(IllegalArgumentException.class, () -> LanceIndexDatasetLocator.normalize("file://"));
- // Trailing-slash stripping still leaves nothing behind the scheme.
- Assertions.assertThrows(IllegalArgumentException.class, () -> LanceIndexDatasetLocator.normalize("s3:///"));
- // An empty authority with a real path stays legal.
- Assertions.assertEquals("file:///x", LanceIndexDatasetLocator.normalize("file:///x"));
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobManagerPersistTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobManagerPersistTest.java
deleted file mode 100644
index d29cb4c3a8b..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobManagerPersistTest.java
+++ /dev/null
@@ -1,457 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.apache.doris.common.DdlException;
-import org.apache.doris.journal.JournalEntity;
-import org.apache.doris.persist.OperationType;
-import org.apache.doris.persist.gson.GsonUtils;
-
-import org.apache.commons.lang3.StringUtils;
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.io.ByteArrayInputStream;
-import java.io.ByteArrayOutputStream;
-import java.io.DataInputStream;
-import java.io.DataOutputStream;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-
-/**
- * Persistence coverage for the Lance index job infrastructure: the manager image
- * write/read round trip (including the fence-index and quota rebuild in
- * gsonPostProcess), the journal mounting point for op code 500
- * ({@link OperationType#OP_LANCE_INDEX_JOB_UPSERT}) through
- * {@link JournalEntity#write}/{@link JournalEntity#readFields}, the single-record Gson
- * round trip field by field, and the bounded-text rejection at construction time.
- */
-public class LanceIndexJobManagerPersistTest {
- private static final long CATALOG_ID = 10L;
- private static final String LOCATOR = "s3://bucket/dataset";
- private static final long BACKEND_ID = 1001L;
- private static final long BE_EPOCH = 55L;
- private static final String INVOCATION_ID = "invocation-1";
-
- @Test
- public void managerImageRoundtripRebuildsDerivedFenceAndQuota() throws Exception {
- TestManager source = new TestManager();
- source.createJob(newCreateJob(1L, "IdxPending"), 100, 100, 100);
- source.createJob(newCreateJob(2L, "IdxRunning"), 100, 100, 100);
- source.markRunning(2L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, 9999L);
- source.createJob(newCreateJob(3L, "IdxCommitted"), 100, 100, 100);
- source.markRunning(3L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, 9999L);
- source.completeWithResult(3L, 1L, INVOCATION_ID, BE_EPOCH,
- new LanceIndexJobResult(LanceIndexJobResultCode.NATIVE_OK,
- LanceIndexJobCompletionReason.NONE, "ok", false));
- source.createJob(newCreateJob(4L, "IdxUnknown"), 100, 100, 100);
- source.markRunning(4L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, 9999L);
- source.completeWithResult(4L, 1L, INVOCATION_ID, BE_EPOCH,
- new LanceIndexJobResult(LanceIndexJobResultCode.NO_TRUSTED_RESULT,
- LanceIndexJobCompletionReason.NONE, "lost", false));
- source.replayUpsertJob(forceReleasedUnknownJob(5L, "idxforced"));
-
- ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
- source.write(new DataOutputStream(byteStream));
- LanceIndexJobManager loaded =
- LanceIndexJobManager.read(new DataInputStream(new ByteArrayInputStream(byteStream.toByteArray())));
-
- Assertions.assertEquals(source.getJobCount(), loaded.getJobCount());
- for (long jobId = 1L; jobId <= 5L; jobId++) {
- Assertions.assertEquals(GsonUtils.GSON.toJson(source.getJob(jobId)),
- GsonUtils.GSON.toJson(loaded.getJob(jobId)), "job " + jobId);
- }
-
- // The derived fence index survived: unresolved jobs still fence their names.
- for (long jobId = 1L; jobId <= 4L; jobId++) {
- Assertions.assertTrue(loaded.isFenceHeld(loaded.getJob(jobId).fenceKey()), "fence of job " + jobId);
- }
- Assertions.assertFalse(loaded.isFenceHeld(loaded.getJob(5L).fenceKey()));
- Assertions.assertEquals(4L, loaded.getQuota().getGlobalCount());
- Assertions.assertEquals(4L, loaded.getQuota().getCatalogCount(CATALOG_ID));
- Assertions.assertEquals(4, loaded.getUnresolvedJobs().size());
- Assertions.assertEquals(1, loaded.getJobsNeedingRefresh().size());
-
- // The fence is enforceable after the image load: a conflict is rejected by the fence
- // CAS, which precedes any journal write.
- Assertions.assertThrows(DdlException.class,
- () -> loaded.createJob(newCreateJob(9L, "idxunknown"), 100, 100, 100));
-
- // And a FORCE-released name is free again once the loaded image is rebuilt through replay.
- TestManager rebuilt = new TestManager();
- for (long jobId = 1L; jobId <= 5L; jobId++) {
- rebuilt.replayUpsertJob(loaded.getJob(jobId));
- }
- rebuilt.createJob(newCreateJob(9L, "IdxForced"), 100, 100, 100);
- Assertions.assertEquals(5L, rebuilt.getQuota().getGlobalCount());
- }
-
- @Test
- public void emptyManagerRoundtripStaysEmpty() throws Exception {
- TestManager source = new TestManager();
- ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
- source.write(new DataOutputStream(byteStream));
- LanceIndexJobManager loaded =
- LanceIndexJobManager.read(new DataInputStream(new ByteArrayInputStream(byteStream.toByteArray())));
-
- Assertions.assertEquals(0, loaded.getJobCount());
- Assertions.assertEquals(0L, loaded.getQuota().getGlobalCount());
- Assertions.assertTrue(loaded.getUnresolvedJobs().isEmpty());
- }
-
- @Test
- public void imageLoadKeepsTheSmallerJobIdOnAFenceCollision() {
- // Only a corrupt image can hold two unresolved jobs on the same fence key;
- // gsonPostProcess keeps the smaller job id (Gson auto-runs it after fromJson).
- String jobJson = "\"jid\":%d,\"rev\":0,\"cid\":" + CATALOG_ID
- + ",\"prv\":\"" + LanceIndexFenceKey.PROVIDER_DIRECTORY + "\",\"loc\":\"" + LOCATOR
- + "\",\"din\":\"IdxA\",\"nin\":\"idxa\",\"ms\":\"PENDING\",\"rs\":\"NOT_REQUIRED\"";
- String json = "{\"jobs\":{\"2\":{" + String.format(jobJson, 2L) + "},"
- + "\"1\":{" + String.format(jobJson, 1L) + "}}}";
- LanceIndexJobManager loaded = GsonUtils.GSON.fromJson(json, LanceIndexJobManager.class);
-
- Assertions.assertNotNull(loaded.getJob(1L));
- Assertions.assertTrue(loaded.isFenceHeld(loaded.getJob(1L).fenceKey()));
- // Both unresolved jobs still charge the quota.
- Assertions.assertEquals(2L, loaded.getQuota().getGlobalCount());
- // The fence conflict message names the surviving smaller job id.
- DdlException exception = Assertions.assertThrows(DdlException.class,
- () -> loaded.createJob(newCreateJob(9L, "idxa"), 100, 100, 100));
- Assertions.assertTrue(exception.getMessage().contains("unresolved job 1"));
- }
-
- @Test
- public void imageLoadKeepsAnIdentityLessRecordBlockingAdmission() {
- // A corrupt image record without fence identity cannot join the books; it must
- // still block new admissions fail-closed after gsonPostProcess rebuilds state
- // (Gson auto-runs it after fromJson).
- String json = "{\"jobs\":{\"5\":{\"jid\":5,\"rev\":0,\"ms\":\"PENDING\"}}}";
- LanceIndexJobManager loaded = GsonUtils.GSON.fromJson(json, LanceIndexJobManager.class);
-
- Assertions.assertNotNull(loaded.getJob(5L));
- Assertions.assertEquals(Collections.singletonList(5L), loaded.getCorruptUnresolvedJobIds());
- Assertions.assertEquals(0L, loaded.getQuota().getGlobalCount());
- Assertions.assertTrue(loaded.getUnresolvedJobs().isEmpty());
-
- // The rejection precedes any journal write, so the base-class edit-log seam
- // (which needs a live Env) never runs.
- DdlException exception = Assertions.assertThrows(DdlException.class,
- () -> loaded.createJob(newCreateJob(9L, "IdxB"), 100, 100, 100));
- Assertions.assertTrue(exception.getMessage().contains("smallest job id 5"));
- }
-
- @Test
- public void schemaContractEqualityIsOrderSensitive() {
- LanceIndexSchemaContract ordered = contract();
- LanceIndexSchemaContract sameOrder = contract();
- List<LanceIndexSchemaContract.IndexedField> reversed = new ArrayList<>();
- reversed.add(ordered.getFields().get(1));
- reversed.add(ordered.getFields().get(0));
- LanceIndexSchemaContract reordered = new LanceIndexSchemaContract(reversed);
-
- Assertions.assertEquals(ordered, sameOrder);
- Assertions.assertEquals(ordered.hashCode(), sameOrder.hashCode());
- // The same fields in a different order are a different contract.
- Assertions.assertNotEquals(ordered, reordered);
- }
-
- @Test
- public void schemaContractIsBoundedNullFreeAndImmutable() {
- LanceIndexSchemaContract.IndexedField field = new LanceIndexSchemaContract.IndexedField(
- 1L, "v", "fixed_size_list[float;192]", false, 192, "float", false);
- List<LanceIndexSchemaContract.IndexedField> atLimit = new ArrayList<>();
- for (int index = 0; index < LanceIndexSchemaContract.MAX_INDEXED_FIELDS; index++) {
- atLimit.add(field);
- }
- LanceIndexSchemaContract contractAtLimit = new LanceIndexSchemaContract(atLimit);
- Assertions.assertEquals(LanceIndexSchemaContract.MAX_INDEXED_FIELDS, contractAtLimit.getFields().size());
-
- atLimit.add(field);
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexSchemaContract(atLimit));
- List<LanceIndexSchemaContract.IndexedField> withNull = new ArrayList<>();
- withNull.add(null);
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexSchemaContract(withNull));
-
- List<LanceIndexSchemaContract.IndexedField> source = new ArrayList<>();
- source.add(field);
- LanceIndexSchemaContract copied = new LanceIndexSchemaContract(source);
- source.clear();
- Assertions.assertEquals(1, copied.getFields().size());
- Assertions.assertThrows(UnsupportedOperationException.class,
- () -> copied.getFields().add(field));
- }
-
- @Test
- public void schemaContractStringBoundsCountUtf8Bytes() {
- String atLimit = StringUtils.repeat("é", LanceIndexSchemaContract.MAX_FIELD_STRING_BYTES / 2);
- new LanceIndexSchemaContract.IndexedField(1L, atLimit, atLimit, false, null, atLimit, null);
-
- String overLimit = atLimit + "é";
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexSchemaContract.IndexedField(
- 1L, overLimit, "type", false, null, null, null));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexSchemaContract.IndexedField(
- 1L, "name", overLimit, false, null, null, null));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexSchemaContract.IndexedField(
- 1L, "name", "type", false, null, overLimit, null));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexSchemaContract.IndexedField(
- 1L, null, "type", false, null, null, null));
- }
-
- @Test
- public void journalEntityRoundtripUsesOpCode500() throws Exception {
- LanceIndexJob job = fullyPopulatedJob();
-
- ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
- DataOutputStream output = new DataOutputStream(byteStream);
- JournalEntity journalEntity = new JournalEntity();
- journalEntity.setData(job);
- journalEntity.setOpCode(OperationType.OP_LANCE_INDEX_JOB_UPSERT);
- journalEntity.write(output);
- output.flush();
-
- JournalEntity replayed = new JournalEntity();
- replayed.readFields(new DataInputStream(new ByteArrayInputStream(byteStream.toByteArray())));
-
- Assertions.assertEquals(OperationType.OP_LANCE_INDEX_JOB_UPSERT, replayed.getOpCode());
- Assertions.assertEquals(500, replayed.getOpCode());
- Assertions.assertTrue(replayed.getData() instanceof LanceIndexJob);
- Assertions.assertEquals(GsonUtils.GSON.toJson(job), GsonUtils.GSON.toJson(replayed.getData()));
- }
-
- @Test
- public void journalEntityStreamPreservesRecordOrder() throws Exception {
- List<LanceIndexJob> jobs = new ArrayList<>();
- jobs.add(fullyPopulatedJob());
- LanceIndexJob second = newCreateJob(77L, "IdxSecond");
- second.setMutationState(LanceIndexJobMutationState.PENDING);
- jobs.add(second);
-
- ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
- DataOutputStream output = new DataOutputStream(byteStream);
- for (LanceIndexJob job : jobs) {
- JournalEntity journalEntity = new JournalEntity();
- journalEntity.setData(job);
- journalEntity.setOpCode(OperationType.OP_LANCE_INDEX_JOB_UPSERT);
- journalEntity.write(output);
- }
- output.flush();
-
- DataInputStream input = new DataInputStream(new ByteArrayInputStream(byteStream.toByteArray()));
- for (LanceIndexJob expected : jobs) {
- JournalEntity replayed = new JournalEntity();
- replayed.readFields(input);
- Assertions.assertEquals(OperationType.OP_LANCE_INDEX_JOB_UPSERT, replayed.getOpCode());
- Assertions.assertEquals(GsonUtils.GSON.toJson(expected), GsonUtils.GSON.toJson(replayed.getData()));
- }
- Assertions.assertEquals(0, input.available());
- }
-
- @Test
- public void jobStreamRoundtripPreservesAllFields() throws Exception {
- LanceIndexJob job = fullyPopulatedJob();
-
- ByteArrayOutputStream byteStream = new ByteArrayOutputStream();
- job.write(new DataOutputStream(byteStream));
- LanceIndexJob restored =
- LanceIndexJob.read(new DataInputStream(new ByteArrayInputStream(byteStream.toByteArray())));
-
- assertSameJobFields(job, restored);
- }
-
- @Test
- public void jobGsonRoundtripPreservesAllFields() {
- LanceIndexJob job = fullyPopulatedJob();
- LanceIndexJob restored = GsonUtils.GSON.fromJson(GsonUtils.GSON.toJson(job), LanceIndexJob.class);
- assertSameJobFields(job, restored);
- }
-
- @Test
- public void boundedTextFieldsRejectOverflow() {
- String overMessage = StringUtils.repeat("m", LanceIndexJobResult.MAX_MESSAGE_BYTES + 1);
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexJobResult(LanceIndexJobResultCode.NATIVE_OK,
- LanceIndexJobCompletionReason.NONE, overMessage, false));
- // Multibyte characters count as UTF-8 bytes, not chars.
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexJobResult(LanceIndexJobResultCode.NATIVE_OK,
- LanceIndexJobCompletionReason.NONE, StringUtils.repeat("é", 513), false));
- new LanceIndexJobResult(LanceIndexJobResultCode.NATIVE_OK, LanceIndexJobCompletionReason.NONE,
- StringUtils.repeat("m", LanceIndexJobResult.MAX_MESSAGE_BYTES), false);
-
- LanceIndexJob job = newCreateJob(1L, "IdxA");
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> job.setCreator(StringUtils.repeat("c", LanceIndexJob.MAX_DURABLE_TEXT_BYTES + 1)));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> job.setIndexType(StringUtils.repeat("t", LanceIndexJob.MAX_DURABLE_TEXT_BYTES + 1)));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> job.setColumnName(StringUtils.repeat("c", LanceIndexJob.MAX_DURABLE_TEXT_BYTES + 1)));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> job.setInvocationId(StringUtils.repeat("i", LanceIndexJob.MAX_INVOCATION_ID_BYTES + 1)));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> job.setForceActor(StringUtils.repeat("a", LanceIndexJob.MAX_DURABLE_TEXT_BYTES + 1)));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> job.setPropertiesJson(StringUtils.repeat("p", LanceIndexJob.MAX_PROPERTIES_JSON_BYTES + 1)));
- job.setPropertiesJson(StringUtils.repeat("p", LanceIndexJob.MAX_PROPERTIES_JSON_BYTES));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> job.setForceNote(StringUtils.repeat("n", LanceIndexJob.MAX_FORCE_TEXT_BYTES + 1)));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> job.setForceWarning(StringUtils.repeat("w", LanceIndexJob.MAX_FORCE_TEXT_BYTES + 1)));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> job.setNormalizedIndexName(
- StringUtils.repeat("n", LanceIndexNameNormalizer.MAX_INDEX_NAME_BYTES + 1)));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> job.setDisplayIndexName(
- StringUtils.repeat("d", LanceIndexNameNormalizer.MAX_INDEX_NAME_BYTES + 1)));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> newCreateJob(2L, StringUtils.repeat("d", LanceIndexNameNormalizer.MAX_INDEX_NAME_BYTES + 1)));
-
- // Optional durable text remains nullable; required creator identity does not.
- job.setCreator(null);
- Assertions.assertThrows(IllegalArgumentException.class, job::validateForAdmission);
- job.setCreator("tester");
- job.setIndexType(null);
- job.setColumnName(null);
- job.setInvocationId(null);
- job.setForceActor(null);
- job.setForceNote(null);
- job.setForceWarning(null);
- job.setPropertiesJson(null);
- job.validateForAdmission();
- }
-
- @Test
- public void resultRejectsNullResultCode() {
- Assertions.assertThrows(NullPointerException.class,
- () -> new LanceIndexJobResult(null, LanceIndexJobCompletionReason.NONE, "msg", false));
- }
-
- private static LanceIndexJob newCreateJob(long jobId, String displayName) {
- return new LanceIndexJob(jobId, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR,
- displayName, LanceIndexNameNormalizer.normalize(displayName),
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v",
- null, 7L, null);
- }
-
- private static LanceIndexSchemaContract contract() {
- List<LanceIndexSchemaContract.IndexedField> fields = new ArrayList<>();
- fields.add(new LanceIndexSchemaContract.IndexedField(1L, "v", "fixed_size_list[float;192]",
- false, 192, "float", false));
- fields.add(new LanceIndexSchemaContract.IndexedField(2L, "s", "decimal(10,2)", true, null, null, null));
- return new LanceIndexSchemaContract(fields);
- }
-
- /**
- * A job with every durable field populated, including result, schema contract,
- * dispatch identity, and the FORCE audit fields.
- */
- private static LanceIndexJob fullyPopulatedJob() {
- LanceIndexJob job = new LanceIndexJob(42L, "creator", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR, "IdxΩ", "idxω",
- LanceIndexJobMutationType.REPLACE, true, false, "IVF_PQ", "v",
- "{\"num_partitions\":\"256\"}", 99L, contract());
- job.setRevision(7L);
- job.setCreateTimeMs(111L);
- job.setUpdateTimeMs(222L);
- job.setMutationState(LanceIndexJobMutationState.RUNNING);
- job.setRefreshState(LanceIndexJobRefreshState.RUNNING);
- job.setResult(new LanceIndexJobResult(LanceIndexJobResultCode.NATIVE_IO,
- LanceIndexJobCompletionReason.NONE, "io error", true));
- job.setBackendId(BACKEND_ID);
- job.setBeProcessEpoch(BE_EPOCH);
- job.setDispatchRevision(7L);
- job.setInvocationId(INVOCATION_ID);
- job.setDeadlineMs(123456L);
- job.setPossibleLiveOwned(true);
- job.setTerminationProof(LanceIndexTerminationProof.NONE);
- job.setForceActor("admin");
- job.setForceTimeMs(777L);
- job.setForceNote("note");
- job.setForceWarning("warning");
- return job;
- }
-
- private static LanceIndexJob forceReleasedUnknownJob(long jobId, String normalizedName) {
- String json = "{\"jid\":" + jobId + ",\"cr\":\"tester\",\"rev\":2,\"cid\":" + CATALOG_ID
- + ",\"dbn\":\"db1\",\"tbn\":\"tbl1\",\"prv\":\"" + LanceIndexFenceKey.PROVIDER_DIRECTORY
- + "\",\"loc\":\"" + LOCATOR + "\",\"din\":\"" + normalizedName + "\",\"nin\":\"" + normalizedName
- + "\",\"mt\":\"CREATE\",\"ms\":\"UNKNOWN\",\"rs\":\"NOT_REQUIRED\",\"fr\":true}";
- return GsonUtils.GSON.fromJson(json, LanceIndexJob.class);
- }
-
- private static void assertSameJobFields(LanceIndexJob expected, LanceIndexJob actual) {
- Assertions.assertEquals(expected.getJobId(), actual.getJobId());
- Assertions.assertEquals(expected.getCreator(), actual.getCreator());
- Assertions.assertEquals(expected.getRevision(), actual.getRevision());
- Assertions.assertEquals(expected.getCreateTimeMs(), actual.getCreateTimeMs());
- Assertions.assertEquals(expected.getUpdateTimeMs(), actual.getUpdateTimeMs());
- Assertions.assertEquals(expected.getCatalogId(), actual.getCatalogId());
- Assertions.assertEquals(expected.getDbName(), actual.getDbName());
- Assertions.assertEquals(expected.getTableName(), actual.getTableName());
- Assertions.assertEquals(expected.getProvider(), actual.getProvider());
- Assertions.assertEquals(expected.getNormalizedLocator(), actual.getNormalizedLocator());
- Assertions.assertEquals(expected.getDisplayIndexName(), actual.getDisplayIndexName());
- Assertions.assertEquals(expected.getNormalizedIndexName(), actual.getNormalizedIndexName());
- Assertions.assertEquals(expected.getMutationType(), actual.getMutationType());
- Assertions.assertEquals(expected.isIfNotExists(), actual.isIfNotExists());
- Assertions.assertEquals(expected.isIfExists(), actual.isIfExists());
- Assertions.assertEquals(expected.getIndexType(), actual.getIndexType());
- Assertions.assertEquals(expected.getColumnName(), actual.getColumnName());
- Assertions.assertEquals(expected.getPropertiesJson(), actual.getPropertiesJson());
- Assertions.assertEquals(expected.getAdmittedDatasetVersion(), actual.getAdmittedDatasetVersion());
- Assertions.assertEquals(expected.getSchemaContract(), actual.getSchemaContract());
- Assertions.assertEquals(expected.getMutationState(), actual.getMutationState());
- Assertions.assertEquals(expected.getRefreshState(), actual.getRefreshState());
- Assertions.assertEquals(expected.getResult().getResultCode(), actual.getResult().getResultCode());
- Assertions.assertEquals(expected.getResult().getCompletionReason(), actual.getResult().getCompletionReason());
- Assertions.assertEquals(expected.getResult().getSanitizedMessage(), actual.getResult().getSanitizedMessage());
- Assertions.assertEquals(expected.getResult().isExternalMetadataAdvanced(),
- actual.getResult().isExternalMetadataAdvanced());
- Assertions.assertEquals(expected.getBackendId(), actual.getBackendId());
- Assertions.assertEquals(expected.getBeProcessEpoch(), actual.getBeProcessEpoch());
- Assertions.assertEquals(expected.getDispatchRevision(), actual.getDispatchRevision());
- Assertions.assertEquals(expected.getInvocationId(), actual.getInvocationId());
- Assertions.assertEquals(expected.getDeadlineMs(), actual.getDeadlineMs());
- Assertions.assertEquals(expected.isPossibleLiveOwned(), actual.isPossibleLiveOwned());
- Assertions.assertEquals(expected.getTerminationProof(), actual.getTerminationProof());
- Assertions.assertEquals(expected.isForceReleased(), actual.isForceReleased());
- Assertions.assertEquals(expected.getForceActor(), actual.getForceActor());
- Assertions.assertEquals(expected.getForceTimeMs(), actual.getForceTimeMs());
- Assertions.assertEquals(expected.getForceNote(), actual.getForceNote());
- Assertions.assertEquals(expected.getForceWarning(), actual.getForceWarning());
- Assertions.assertEquals(expected.fenceKey(), actual.fenceKey());
- Assertions.assertEquals(expected.getTableQuotaKey(), actual.getTableQuotaKey());
- }
-
- /**
- * Edit-log seam: captures every durable record instead of writing the journal.
- */
- private static class TestManager extends LanceIndexJobManager {
- @Override
- protected void writeEditLog(LanceIndexJob job) {
- // No journal in a pure persistence unit test.
- }
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobManagerQueryTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobManagerQueryTest.java
deleted file mode 100644
index 7675d05ab18..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobManagerQueryTest.java
+++ /dev/null
@@ -1,131 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.util.List;
-
-/**
- * Coverage for the two read-side queries added to {@link LanceIndexJobManager} for the
- * admission slice: {@link LanceIndexJobManager#getAllJobsSnapshot()} (the data source of
- * SHOW LANCE INDEX JOBS: every job, copies only, ordered by job id) and
- * {@link LanceIndexJobManager#hasUnresolvedJobsForCatalog(long)} (the catalog DDL guard
- * probe: any job still holding its fence and unresolved quota for the catalog).
- */
-public class LanceIndexJobManagerQueryTest {
- private static final long CATALOG_ID = 10L;
- private static final long OTHER_CATALOG_ID = 20L;
- private static final String LOCATOR = "s3://bucket/dataset";
- private static final long BACKEND_ID = 1001L;
- private static final long BE_EPOCH = 55L;
- private static final String INVOCATION_ID = "invocation-1";
- private static final long DEADLINE_MS = 9999L;
-
- @Test
- public void snapshotReturnsEveryJobOrderedByJobId() throws Exception {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(3L, "IdxC", CATALOG_ID), 100, 100, 100);
- manager.createJob(newCreateJob(1L, "IdxA", CATALOG_ID), 100, 100, 100);
- manager.createJob(newCreateJob(2L, "IdxB", OTHER_CATALOG_ID), 100, 100, 100);
-
- List<LanceIndexJob> snapshot = manager.getAllJobsSnapshot();
- Assertions.assertEquals(3, snapshot.size());
- Assertions.assertEquals(1L, snapshot.get(0).getJobId());
- Assertions.assertEquals(2L, snapshot.get(1).getJobId());
- Assertions.assertEquals(3L, snapshot.get(2).getJobId());
- }
-
- @Test
- public void snapshotCopiesAreIsolatedFromStoredRecords() throws Exception {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA", CATALOG_ID), 100, 100, 100);
-
- LanceIndexJob copy = manager.getAllJobsSnapshot().get(0);
- copy.setMutationState(LanceIndexJobMutationState.UNKNOWN);
- copy.setDisplayIndexName("mutated");
- copy.setRevision(99L);
-
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, stored.getMutationState());
- Assertions.assertEquals("IdxA", stored.getDisplayIndexName());
- Assertions.assertEquals(0L, stored.getRevision());
-
- // A later snapshot is unaffected by mutations of an earlier one.
- LanceIndexJob fresh = manager.getAllJobsSnapshot().get(0);
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, fresh.getMutationState());
- Assertions.assertEquals("IdxA", fresh.getDisplayIndexName());
- Assertions.assertEquals(0L, fresh.getRevision());
- }
-
- @Test
- public void hasUnresolvedJobsForCatalogScopesByCatalogId() throws Exception {
- TestManager manager = new TestManager();
- Assertions.assertFalse(manager.hasUnresolvedJobsForCatalog(CATALOG_ID));
-
- manager.createJob(newCreateJob(1L, "IdxA", CATALOG_ID), 100, 100, 100);
- Assertions.assertTrue(manager.hasUnresolvedJobsForCatalog(CATALOG_ID));
- Assertions.assertFalse(manager.hasUnresolvedJobsForCatalog(OTHER_CATALOG_ID));
- Assertions.assertFalse(manager.hasUnresolvedJobsForCatalog(999L));
- }
-
- @Test
- public void releasedTerminalJobDoesNotCountAsUnresolved() throws Exception {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA", CATALOG_ID), 100, 100, 100);
- Assertions.assertTrue(manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- new LanceIndexJobResult(LanceIndexJobResultCode.NATIVE_OK,
- LanceIndexJobCompletionReason.NONE, "ok", false)));
- // COMMITTED with refresh REQUIRED still holds the fence and quota.
- Assertions.assertTrue(manager.hasUnresolvedJobsForCatalog(CATALOG_ID));
-
- Assertions.assertTrue(manager.markRefreshRunning(1L, 2L));
- Assertions.assertTrue(manager.markRefreshDone(1L, 3L));
- Assertions.assertFalse(manager.hasUnresolvedJobsForCatalog(CATALOG_ID));
-
- // A released terminal job remains visible in the full snapshot.
- Assertions.assertEquals(1, manager.getAllJobsSnapshot().size());
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED,
- manager.getAllJobsSnapshot().get(0).getMutationState());
- }
-
- @Test
- public void unresolvedJobInAnotherCatalogDoesNotLeak() throws Exception {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA", OTHER_CATALOG_ID), 100, 100, 100);
- Assertions.assertFalse(manager.hasUnresolvedJobsForCatalog(CATALOG_ID));
- Assertions.assertTrue(manager.hasUnresolvedJobsForCatalog(OTHER_CATALOG_ID));
- }
-
- private static LanceIndexJob newCreateJob(long jobId, String displayName, long catalogId) {
- return new LanceIndexJob(jobId, "tester", catalogId, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR,
- displayName, LanceIndexNameNormalizer.normalize(displayName),
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v",
- null, 7L, null);
- }
-
- private static class TestManager extends LanceIndexJobManager {
- @Override
- protected void writeEditLog(LanceIndexJob job) {
- // No journal in a pure query unit test.
- }
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobManagerReplayTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobManagerReplayTest.java
deleted file mode 100644
index 0a395bee6a0..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobManagerReplayTest.java
+++ /dev/null
@@ -1,628 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.apache.doris.common.DdlException;
-import org.apache.doris.persist.gson.GsonUtils;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.List;
-
-/**
- * Replay and master-transfer coverage for {@link LanceIndexJobManager}. The journal
- * records of a scenario are captured from a source manager through the edit-log seam
- * and replayed verbatim into a fresh target manager, exactly like a follower tailing
- * the edit log. The pinned invariants: replay is a verbatim replace with a monotonic
- * revision guard (no state transformation on followers); a replayed PENDING permits
- * exactly one dispatch; only the master-election sweep turns a durable RUNNING into
- * UNKNOWN, after which redispatch is permanently refused; an unresolved UNKNOWN keeps
- * its fence and quota across replay while a FORCE-released UNKNOWN frees both; stale
- * callbacks (revision / invocation id / BE epoch) never change state; and a rejected
- * admission leaves no job, fence, quota charge, or journal record behind. A replayed
- * unresolved record that lacks fence identity stays out of the books but blocks all
- * new admissions fail-closed until a later record resolves it.
- */
-public class LanceIndexJobManagerReplayTest {
- private static final long CATALOG_ID = 10L;
- private static final String LOCATOR = "s3://bucket/dataset";
- private static final long BACKEND_ID = 1001L;
- private static final long BE_EPOCH = 55L;
- private static final String INVOCATION_ID = "invocation-1";
- private static final long DEADLINE_MS = 9999L;
-
- @Test
- public void replayedPendingPermitsExactlyOneDispatch() throws DdlException {
- List<LanceIndexJob> records = runningRecords(1L, "IdxA");
- TestManager target = new TestManager();
- target.replayUpsertJob(records.get(0));
-
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, target.getJob(1L).getMutationState());
- Assertions.assertTrue(target.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertFalse(target.markRunning(1L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertFalse(target.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertEquals(LanceIndexJobMutationState.RUNNING, target.getJob(1L).getMutationState());
- }
-
- @Test
- public void replayedRunningStaysRunningWithoutSweep() throws DdlException {
- List<LanceIndexJob> records = runningRecords(1L, "IdxA");
- TestManager target = new TestManager();
- target.replayUpsertJob(records.get(0));
- target.replayUpsertJob(records.get(1));
-
- LanceIndexJob stored = target.getJob(1L);
- // A follower tailing a live master must not transform a fresh RUNNING record.
- Assertions.assertEquals(LanceIndexJobMutationState.RUNNING, stored.getMutationState());
- Assertions.assertEquals(1L, stored.getRevision());
- Assertions.assertTrue(stored.holdsPossibleLiveSlot());
- Assertions.assertTrue(target.isFenceHeld(stored.fenceKey()));
- Assertions.assertEquals(1L, target.getQuota().getGlobalCount());
- // Replay itself never writes the journal.
- Assertions.assertTrue(target.editLog.isEmpty());
- }
-
- @Test
- public void transferToMasterConvertsRunningToUnknownAndNeverRedispatches() throws DdlException {
- List<LanceIndexJob> records = runningRecords(1L, "IdxA");
- TestManager target = new TestManager();
- target.replayUpsertJob(records.get(0));
- target.replayUpsertJob(records.get(1));
- LanceIndexFenceKey fenceKey = target.getJob(1L).fenceKey();
-
- target.onTransferToMaster();
-
- LanceIndexJob swept = target.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, swept.getMutationState());
- Assertions.assertEquals(2L, swept.getRevision());
- Assertions.assertEquals(LanceIndexJobResultCode.NO_TRUSTED_RESULT, swept.getResult().getResultCode());
- // Fence, quota, and the possible-live slot survive the sweep: only FORCE releases them.
- Assertions.assertTrue(target.isFenceHeld(fenceKey));
- Assertions.assertEquals(1L, target.getQuota().getGlobalCount());
- Assertions.assertTrue(swept.holdsPossibleLiveSlot());
- Assertions.assertTrue(containsJob(target.getUnresolvedJobs(), swept.getJobId()));
- Assertions.assertEquals(1, target.editLog.size());
-
- Assertions.assertFalse(target.markRunning(1L, 2L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertFalse(target.markRunning(1L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertFalse(target.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, target.getJob(1L).getMutationState());
- }
-
- @Test
- public void transferToMasterDowngradesRunningRefreshToRequired() throws DdlException {
- TestManager source = new TestManager();
- source.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
- source.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS);
- source.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH, okResult());
- source.markRefreshRunning(1L, 2L);
-
- TestManager target = new TestManager();
- for (LanceIndexJob record : source.editLog) {
- target.replayUpsertJob(record);
- }
- Assertions.assertEquals(LanceIndexJobRefreshState.RUNNING, target.getJob(1L).getRefreshState());
-
- target.onTransferToMaster();
-
- LanceIndexJob swept = target.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED, swept.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, swept.getRefreshState());
- Assertions.assertEquals(4L, swept.getRevision());
- Assertions.assertTrue(containsJob(target.getJobsNeedingRefresh(), swept.getJobId()));
- Assertions.assertTrue(target.isFenceHeld(swept.fenceKey()));
- }
-
- @Test
- public void replayedTerminalWithRefreshRequiredOnlyAllowsRefreshPath() throws DdlException {
- TestManager source = new TestManager();
- source.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
- source.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS);
- source.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH, okResult());
-
- TestManager target = new TestManager();
- for (LanceIndexJob record : source.editLog) {
- target.replayUpsertJob(record);
- }
- LanceIndexJob stored = target.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED, stored.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, stored.getRefreshState());
- LanceIndexFenceKey fenceKey = stored.fenceKey();
-
- // The mutation lifecycle is closed; only the refresh transitions remain.
- Assertions.assertFalse(target.markRunning(1L, 2L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertFalse(target.completeWithResult(1L, 2L, INVOCATION_ID, BE_EPOCH, okResult()));
- Assertions.assertTrue(containsJob(target.getJobsNeedingRefresh(), stored.getJobId()));
- Assertions.assertTrue(target.isFenceHeld(fenceKey));
- Assertions.assertEquals(1L, target.getQuota().getGlobalCount());
-
- Assertions.assertTrue(target.markRefreshRunning(1L, 2L));
- Assertions.assertTrue(target.markRefreshDone(1L, 3L));
- Assertions.assertFalse(target.isFenceHeld(fenceKey));
- Assertions.assertEquals(0L, target.getQuota().getGlobalCount());
- Assertions.assertTrue(target.getUnresolvedJobs().isEmpty());
- }
-
- @Test
- public void replayedUnresolvedUnknownFencesTheSameName() throws DdlException {
- TestManager source = new TestManager();
- source.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
- source.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS);
- source.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- new LanceIndexJobResult(LanceIndexJobResultCode.NO_TRUSTED_RESULT,
- LanceIndexJobCompletionReason.NONE, "ambiguous", false));
-
- TestManager target = new TestManager();
- for (LanceIndexJob record : source.editLog) {
- target.replayUpsertJob(record);
- }
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, target.getJob(1L).getMutationState());
-
- // The unforced UNKNOWN still holds the same-name fence and the quota.
- Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(9L, "IdxA"), 100, 100, 100));
- Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(9L, "idxa"), 100, 100, 100));
- target.createJob(newCreateJob(9L, "IdxB"), 100, 100, 100);
- Assertions.assertEquals(2L, target.getQuota().getGlobalCount());
- }
-
- @Test
- public void replayedForceReleasedUnknownFreesNameAndQuota() {
- TestManager target = new TestManager();
- target.replayUpsertJob(forceReleasedUnknownJob(7L, "idxforce"));
-
- Assertions.assertEquals(0L, target.getQuota().getGlobalCount());
- Assertions.assertFalse(target.isFenceHeld(target.getJob(7L).fenceKey()));
- Assertions.assertTrue(target.getUnresolvedJobs().isEmpty());
-
- Assertions.assertDoesNotThrow(() -> target.createJob(newCreateJob(8L, "IdxForce"), 100, 100, 100));
- Assertions.assertEquals(1L, target.getQuota().getGlobalCount());
- Assertions.assertEquals(1, target.editLog.size());
- }
-
- @Test
- public void staleCallbacksAreRejectedWithoutStateChange() throws DdlException {
- List<LanceIndexJob> records = runningRecords(1L, "IdxA");
- TestManager target = new TestManager();
- target.replayUpsertJob(records.get(0));
- target.replayUpsertJob(records.get(1));
-
- Assertions.assertFalse(target.completeWithResult(1L, 0L, INVOCATION_ID, BE_EPOCH, okResult()));
- Assertions.assertFalse(target.completeWithResult(1L, 1L, "invocation-x", BE_EPOCH, okResult()));
- Assertions.assertFalse(target.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH + 1, okResult()));
- Assertions.assertFalse(target.completeWithResult(1L, 1L, INVOCATION_ID, null, okResult()));
- Assertions.assertFalse(target.completeWithResult(404L, 1L, INVOCATION_ID, BE_EPOCH, okResult()));
-
- LanceIndexJob stored = target.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.RUNNING, stored.getMutationState());
- Assertions.assertEquals(1L, stored.getRevision());
- Assertions.assertNull(stored.getResult());
- Assertions.assertTrue(target.editLog.isEmpty());
-
- Assertions.assertTrue(target.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH, okResult()));
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED, target.getJob(1L).getMutationState());
- }
-
- @Test
- public void replayIsIdempotentForTheSameRecord() throws DdlException {
- List<LanceIndexJob> records = runningRecords(1L, "IdxA");
- TestManager target = new TestManager();
- target.replayUpsertJob(records.get(0));
- target.replayUpsertJob(records.get(0));
- target.replayUpsertJob(records.get(1));
- target.replayUpsertJob(records.get(1));
-
- Assertions.assertEquals(1, target.getJobCount());
- Assertions.assertEquals(1L, target.getQuota().getGlobalCount());
- Assertions.assertEquals(LanceIndexJobMutationState.RUNNING, target.getJob(1L).getMutationState());
- Assertions.assertEquals(1L, target.getJob(1L).getRevision());
- }
-
- @Test
- public void lowerRevisionRecordNeverOverwritesHigherRevision() throws DdlException {
- TestManager source = new TestManager();
- source.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
- source.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS);
- source.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH, okResult());
- // Records: PENDING rev0, RUNNING rev1, COMMITTED+REQUIRED rev2.
-
- TestManager target = new TestManager();
- target.replayUpsertJob(source.editLog.get(2));
- target.replayUpsertJob(source.editLog.get(0));
- target.replayUpsertJob(source.editLog.get(1));
-
- LanceIndexJob stored = target.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED, stored.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, stored.getRefreshState());
- Assertions.assertEquals(2L, stored.getRevision());
- Assertions.assertTrue(target.isFenceHeld(stored.fenceKey()));
- Assertions.assertEquals(1L, target.getQuota().getGlobalCount());
- Assertions.assertTrue(target.editLog.isEmpty());
-
- // An equal revision replaces verbatim (idempotent re-delivery).
- target.replayUpsertJob(source.editLog.get(2));
- Assertions.assertEquals(2L, target.getJob(1L).getRevision());
- Assertions.assertEquals(1L, target.getQuota().getGlobalCount());
- }
-
- @Test
- public void replayToleratesNullAndIdentityLessRecords() {
- TestManager target = new TestManager();
- target.replayUpsertJob(null);
- Assertions.assertEquals(0, target.getJobCount());
-
- // A corrupt record without fence identity stays queryable but out of the books.
- LanceIndexJob sparse = GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":0,\"ms\":\"PENDING\"}", LanceIndexJob.class);
- target.replayUpsertJob(sparse);
- Assertions.assertEquals(1, target.getJobCount());
- Assertions.assertEquals(0L, target.getQuota().getGlobalCount());
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, target.getJob(5L).getMutationState());
- Assertions.assertTrue(target.getUnresolvedJobs().isEmpty());
- Assertions.assertFalse(target.markRunning(5L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
-
- // Out of the books does not mean invisible: it blocks all new admissions
- // fail-closed, and the rejection leaves nothing behind.
- Assertions.assertEquals(Collections.singletonList(5L), target.getCorruptUnresolvedJobIds());
- DdlException blocked = Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(9L, "IdxB"), 100, 100, 100));
- Assertions.assertTrue(blocked.getMessage().contains("smallest job id 5"));
- Assertions.assertEquals(1, target.getJobCount());
- Assertions.assertTrue(target.editLog.isEmpty());
- Assertions.assertEquals(0L, target.getQuota().getGlobalCount());
- }
-
- @Test
- public void resolvingTheSmallestCollidingJobLeavesTheOtherFenceOwner() throws DdlException {
- TestManager target = new TestManager();
- target.replayUpsertJob(pendingRecord(1L, "IdxA"));
- target.replayUpsertJob(pendingRecord(2L, "IdxA"));
- LanceIndexFenceKey fenceKey = target.getJob(1L).fenceKey();
-
- Assertions.assertEquals(2L, target.getQuota().getGlobalCount());
- DdlException initialConflict = Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(9L, "IdxA"), 100, 100, 100));
- Assertions.assertTrue(initialConflict.getMessage().contains("unresolved job 1"));
-
- Assertions.assertTrue(target.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertTrue(target.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- new LanceIndexJobResult(LanceIndexJobResultCode.PRE_INVOCATION_RESOURCE_REJECTED,
- LanceIndexJobCompletionReason.NONE, "rejected before invocation", false)));
-
- Assertions.assertTrue(target.isFenceHeld(fenceKey));
- Assertions.assertEquals(1L, target.getQuota().getGlobalCount());
- DdlException remainingConflict = Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(9L, "IdxA"), 100, 100, 100));
- Assertions.assertTrue(remainingConflict.getMessage().contains("unresolved job 2"));
- }
-
- @Test
- public void replayAndQueryBoundariesReturnDefensiveCopies() throws DdlException {
- TestManager target = new TestManager();
- LanceIndexJob replayed = pendingRecord(1L, "IdxA");
- target.replayUpsertJob(replayed);
-
- replayed.setMutationState(LanceIndexJobMutationState.COMMITTED);
- replayed.setRefreshState(LanceIndexJobRefreshState.DONE);
- replayed.setRevision(99L);
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, target.getJob(1L).getMutationState());
- Assertions.assertEquals(0L, target.getJob(1L).getRevision());
-
- LanceIndexJob queried = target.getJob(1L);
- queried.setForceReleased(true);
- queried.setRevision(88L);
- Assertions.assertFalse(target.getJob(1L).isForceReleased());
- Assertions.assertEquals(0L, target.getJob(1L).getRevision());
-
- LanceIndexJob listed = target.getUnresolvedJobs().get(0);
- listed.setForceReleased(true);
- Assertions.assertFalse(target.getJob(1L).isForceReleased());
-
- target.createJob(newCreateJob(2L, "IdxB"), 100, 100, 100);
- Assertions.assertTrue(target.markRunning(2L, 0L, BACKEND_ID, BE_EPOCH, "invocation-2", DEADLINE_MS));
- Assertions.assertTrue(target.completeWithResult(2L, 1L, "invocation-2", BE_EPOCH, okResult()));
- LanceIndexJob refreshCandidate = target.getJobsNeedingRefresh().get(0);
- refreshCandidate.setRefreshState(LanceIndexJobRefreshState.DONE);
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, target.getJob(2L).getRefreshState());
-
- target.replayUpsertJob(GsonUtils.GSON.fromJson(
- "{\"jid\":7,\"rev\":0,\"ms\":\"PENDING\"}", LanceIndexJob.class));
- List<Long> blockingIds = target.getCorruptUnresolvedJobIds();
- blockingIds.clear();
- Assertions.assertEquals(Collections.singletonList(7L), target.getCorruptUnresolvedJobIds());
- }
-
- @Test
- public void replayingOverAnIdentityLessRecordNeverThrows() {
- TestManager target = new TestManager();
- LanceIndexJob sparse = GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":0,\"ms\":\"PENDING\"}", LanceIndexJob.class);
- target.replayUpsertJob(sparse);
- Assertions.assertEquals(0L, target.getQuota().getGlobalCount());
-
- // Re-delivering the same identity-less record must not key the release side on it.
- Assertions.assertDoesNotThrow(() -> target.replayUpsertJob(sparse));
- Assertions.assertEquals(1, target.getJobCount());
- Assertions.assertEquals(0L, target.getQuota().getGlobalCount());
-
- // Neither must a higher-revision identity-less upsert of the same job id.
- LanceIndexJob unknown = GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":1,\"ms\":\"UNKNOWN\"}", LanceIndexJob.class);
- Assertions.assertDoesNotThrow(() -> target.replayUpsertJob(unknown));
- LanceIndexJob stored = target.getJob(5L);
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, stored.getMutationState());
- Assertions.assertEquals(1L, stored.getRevision());
- Assertions.assertEquals(1, target.getJobCount());
- Assertions.assertEquals(0L, target.getQuota().getGlobalCount());
-
- // Both deliveries kept the admission blocker pinned...
- Assertions.assertEquals(Collections.singletonList(5L), target.getCorruptUnresolvedJobIds());
- // ...and a resolving upsert lifts it without ever keying the books on the record.
- Assertions.assertDoesNotThrow(() -> target.replayUpsertJob(GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":2,\"ms\":\"UNKNOWN\",\"fr\":true}", LanceIndexJob.class)));
- Assertions.assertTrue(target.getCorruptUnresolvedJobIds().isEmpty());
- Assertions.assertEquals(0L, target.getQuota().getGlobalCount());
- }
-
- @Test
- public void identityLessPendingBlocksAdmissionUntilResolved() throws DdlException {
- TestManager target = new TestManager();
- target.replayUpsertJob(GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":0,\"ms\":\"PENDING\"}", LanceIndexJob.class));
- Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(9L, "IdxB"), 100, 100, 100));
-
- // A later durable record resolving the job lifts the blockade.
- target.replayUpsertJob(GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":1,\"ms\":\"COMMITTED\",\"rs\":\"DONE\"}", LanceIndexJob.class));
- Assertions.assertTrue(target.getCorruptUnresolvedJobIds().isEmpty());
- target.createJob(newCreateJob(9L, "IdxB"), 100, 100, 100);
- Assertions.assertEquals(2, target.getJobCount());
- }
-
- @Test
- public void identityLessUnknownBlocksAdmissionUntilForceReleased() throws DdlException {
- TestManager target = new TestManager();
- target.replayUpsertJob(GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":0,\"ms\":\"UNKNOWN\"}", LanceIndexJob.class));
- Assertions.assertEquals(Collections.singletonList(5L), target.getCorruptUnresolvedJobIds());
- Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(9L, "IdxB"), 100, 100, 100));
-
- // The force-release record of a follow-up PR resolves the job by id,
- // without needing its fence key.
- target.replayUpsertJob(GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":1,\"ms\":\"UNKNOWN\",\"fr\":true}", LanceIndexJob.class));
- Assertions.assertTrue(target.getCorruptUnresolvedJobIds().isEmpty());
- target.createJob(newCreateJob(9L, "IdxB"), 100, 100, 100);
- Assertions.assertEquals(2, target.getJobCount());
- }
-
- @Test
- public void sweptIdentityLessRunningKeepsBlockingAdmission() {
- TestManager target = new TestManager();
- target.replayUpsertJob(GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":1,\"ms\":\"RUNNING\"}", LanceIndexJob.class));
- Assertions.assertEquals(Collections.singletonList(5L), target.getCorruptUnresolvedJobIds());
-
- // The master-election sweep turns the ambiguous RUNNING into UNKNOWN; the
- // blockade survives because the record still lacks fence identity.
- target.onTransferToMaster();
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, target.getJob(5L).getMutationState());
- Assertions.assertEquals(Collections.singletonList(5L), target.getCorruptUnresolvedJobIds());
- Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(9L, "IdxB"), 100, 100, 100));
- }
-
- @Test
- public void healthyJobLifecycleProceedsWhileAdmissionIsBlocked() throws DdlException {
- TestManager target = new TestManager();
- target.createJob(newCreateJob(9L, "IdxB"), 100, 100, 100);
- // Corruption can surface after a legal admission (a follower replays records in
- // journal order); only new admissions are blocked, never in-flight lifecycle steps.
- target.replayUpsertJob(GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":0,\"ms\":\"PENDING\"}", LanceIndexJob.class));
- Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(10L, "IdxC"), 100, 100, 100));
-
- Assertions.assertTrue(target.markRunning(9L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertTrue(target.completeWithResult(9L, 1L, INVOCATION_ID, BE_EPOCH, okResult()));
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED, target.getJob(9L).getMutationState());
- }
-
- @Test
- public void admissionBlockMessageStaysBoundedWithMultipleCorruptRecords() {
- TestManager target = new TestManager();
- for (long jobId = 5L; jobId <= 7L; jobId++) {
- target.replayUpsertJob(GsonUtils.GSON.fromJson(
- "{\"jid\":" + jobId + ",\"rev\":0,\"ms\":\"PENDING\"}", LanceIndexJob.class));
- }
- Assertions.assertEquals(Arrays.asList(5L, 6L, 7L), target.getCorruptUnresolvedJobIds());
-
- DdlException exception = Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(9L, "IdxB"), 100, 100, 100));
- // Bounded and non-disclosing: the count plus the smallest job id, never a list.
- Assertions.assertTrue(exception.getMessage().contains("3 unresolved"));
- Assertions.assertTrue(exception.getMessage().contains("smallest job id 5"));
- Assertions.assertFalse(exception.getMessage().contains("6"));
- Assertions.assertFalse(exception.getMessage().contains("7"));
- }
-
- @Test
- public void identityFullUpsertHealsAnIdentityLessRecord() throws DdlException {
- TestManager target = new TestManager();
- target.replayUpsertJob(GsonUtils.GSON.fromJson(
- "{\"jid\":5,\"rev\":0,\"ms\":\"PENDING\"}", LanceIndexJob.class));
- Assertions.assertEquals(Collections.singletonList(5L), target.getCorruptUnresolvedJobIds());
-
- // A record carrying fence identity heals the books at the same revision: the job
- // joins the fence/quota accounting and the admission blockade lifts.
- target.replayUpsertJob(pendingRecord(5L, "IdxA"));
- Assertions.assertTrue(target.getCorruptUnresolvedJobIds().isEmpty());
- Assertions.assertEquals(1L, target.getQuota().getGlobalCount());
- Assertions.assertTrue(target.isFenceHeld(target.getJob(5L).fenceKey()));
- DdlException fenced = Assertions.assertThrows(DdlException.class,
- () -> target.createJob(newCreateJob(9L, "idxa"), 100, 100, 100));
- Assertions.assertTrue(fenced.getMessage().contains("unresolved job 5"));
- }
-
- @Test
- public void transferToMasterSweepsOnlyTheRunningJobInAMixedPopulation() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxPending"), 100, 100, 100);
- manager.createJob(newCreateJob(2L, "IdxCommitted"), 100, 100, 100);
- manager.markRunning(2L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS);
- manager.completeWithResult(2L, 1L, INVOCATION_ID, BE_EPOCH, okResult());
- manager.markRefreshRunning(2L, 2L);
- manager.markRefreshDone(2L, 3L);
- manager.createJob(newCreateJob(3L, "IdxRunning"), 100, 100, 100);
- manager.markRunning(3L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS);
- // Journal so far: create(1), create+run+complete+refreshRun+refreshDone(2), create+run(3).
- Assertions.assertEquals(8, manager.editLog.size());
-
- manager.onTransferToMaster();
-
- // PENDING and the settled terminal job are replay-faithful: untouched.
- LanceIndexJob pending = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, pending.getMutationState());
- Assertions.assertEquals(0L, pending.getRevision());
- LanceIndexJob committed = manager.getJob(2L);
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED, committed.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.DONE, committed.getRefreshState());
- Assertions.assertEquals(4L, committed.getRevision());
-
- // Only the durable RUNNING becomes UNKNOWN, through exactly one journal record.
- LanceIndexJob swept = manager.getJob(3L);
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, swept.getMutationState());
- Assertions.assertEquals(2L, swept.getRevision());
- Assertions.assertEquals(9, manager.editLog.size());
- }
-
- @Test
- public void fenceRejectionLeavesNoJobNoQuotaAndNoJournalRecord() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
-
- DdlException exception = Assertions.assertThrows(DdlException.class,
- () -> manager.createJob(newCreateJob(2L, "idxa"), 100, 100, 100));
- // The rejection must not disclose the dataset locator to an unauthorized caller.
- Assertions.assertFalse(exception.getMessage().contains("bucket"));
- Assertions.assertFalse(exception.getMessage().contains(LOCATOR));
-
- Assertions.assertEquals(1, manager.getJobCount());
- Assertions.assertEquals(1, manager.editLog.size());
- Assertions.assertEquals(1L, manager.getQuota().getGlobalCount());
- // Only the original job's fence exists; nothing new was registered for the rejection.
- Assertions.assertFalse(manager.isFenceHeld(newCreateJob(3L, "IdxB").fenceKey()));
- }
-
- @Test
- public void quotaRejectionLeavesNoJobNoFenceAndNoJournalRecord() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 1, 1, 1);
-
- Assertions.assertThrows(DdlException.class,
- () -> manager.createJob(newCreateJob(2L, "IdxB"), 1, 1, 1));
-
- Assertions.assertEquals(1, manager.getJobCount());
- Assertions.assertEquals(1, manager.editLog.size());
- Assertions.assertEquals(1L, manager.getQuota().getGlobalCount());
- Assertions.assertFalse(manager.isFenceHeld(newCreateJob(2L, "IdxB").fenceKey()));
- }
-
- @Test
- public void duplicateJobIdIsRejectedBeforeAnythingElse() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
-
- Assertions.assertThrows(DdlException.class,
- () -> manager.createJob(newCreateJob(1L, "IdxB"), 100, 100, 100));
- Assertions.assertEquals(1, manager.getJobCount());
- Assertions.assertEquals(1, manager.editLog.size());
- Assertions.assertFalse(manager.isFenceHeld(newCreateJob(1L, "IdxB").fenceKey()));
- }
-
- private static LanceIndexJob newCreateJob(long jobId, String displayName) {
- return new LanceIndexJob(jobId, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR,
- displayName, LanceIndexNameNormalizer.normalize(displayName),
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v",
- null, 7L, null);
- }
-
- private static LanceIndexJob pendingRecord(long jobId, String displayName) {
- LanceIndexJob job = newCreateJob(jobId, displayName);
- job.setMutationState(LanceIndexJobMutationState.PENDING);
- job.setRefreshState(LanceIndexJobRefreshState.NOT_REQUIRED);
- return job;
- }
-
- private static boolean containsJob(List<LanceIndexJob> jobs, long jobId) {
- return jobs.stream().anyMatch(job -> job.getJobId() == jobId);
- }
-
- private static LanceIndexJobResult okResult() {
- return new LanceIndexJobResult(LanceIndexJobResultCode.NATIVE_OK,
- LanceIndexJobCompletionReason.NONE, "ok", false);
- }
-
- /**
- * Runs create + dispatch on a throwaway source manager and returns its journal
- * records: [PENDING rev0, RUNNING rev1].
- */
- private static List<LanceIndexJob> runningRecords(long jobId, String displayName) throws DdlException {
- TestManager source = new TestManager();
- source.createJob(newCreateJob(jobId, displayName), 100, 100, 100);
- source.markRunning(jobId, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS);
- return source.editLog;
- }
-
- /**
- * Builds the durable form of a FORCE-released UNKNOWN job. The FORCE slice owns the
- * transition itself; here the record only exists as replay input, so it is built
- * from its JSON journal form.
- */
- private static LanceIndexJob forceReleasedUnknownJob(long jobId, String normalizedName) {
- String json = "{\"jid\":" + jobId + ",\"cr\":\"tester\",\"rev\":2,\"cid\":" + CATALOG_ID
- + ",\"dbn\":\"db1\",\"tbn\":\"tbl1\",\"prv\":\"" + LanceIndexFenceKey.PROVIDER_DIRECTORY
- + "\",\"loc\":\"" + LOCATOR + "\",\"din\":\"" + normalizedName + "\",\"nin\":\"" + normalizedName
- + "\",\"mt\":\"CREATE\",\"ms\":\"UNKNOWN\",\"rs\":\"NOT_REQUIRED\",\"fr\":true,"
- + "\"fa\":\"admin\",\"ftm\":12345,\"fn\":\"operator note\"}";
- LanceIndexJob job = GsonUtils.GSON.fromJson(json, LanceIndexJob.class);
- Assertions.assertTrue(job.isForceReleased());
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, job.getMutationState());
- return job;
- }
-
- /**
- * Edit-log seam: captures every durable record instead of writing the journal.
- */
- private static class TestManager extends LanceIndexJobManager {
- private final List<LanceIndexJob> editLog = new ArrayList<>();
-
- @Override
- protected void writeEditLog(LanceIndexJob job) {
- editLog.add(job);
- }
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobQuotaTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobQuotaTest.java
deleted file mode 100644
index 9511f7f1bc0..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobQuotaTest.java
+++ /dev/null
@@ -1,208 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-
-/**
- * Unit coverage for the three-level unresolved-job quota counters (table/locator,
- * catalog, global): the "current < limit" boundary at each level, rejection of
- * non-positive limits, release recovery with underflow clamping, and rebuild
- * equivalence with live counting. The "which jobs count" semantics are
- * {@link LanceIndexJob#isUnresolved()}; the rebuild-side composition is pinned here too.
- */
-public class LanceIndexJobQuotaTest {
- private static final long CATALOG_ID = 10L;
- private static final String LOCATOR_A = "s3://bucket/table-a";
- private static final String LOCATOR_B = "s3://bucket/table-b";
-
- @Test
- public void tryAcquireChargesAllThreeLevels() {
- LanceIndexJobQuota quota = new LanceIndexJobQuota();
- LanceIndexJob job = newJob(1L, CATALOG_ID, LOCATOR_A);
-
- Assertions.assertTrue(quota.tryAcquire(job, 5, 5, 5));
- Assertions.assertEquals(1L, quota.getGlobalCount());
- Assertions.assertEquals(1L, quota.getCatalogCount(CATALOG_ID));
- Assertions.assertEquals(1L, quota.getTableCount(job.getTableQuotaKey()));
- }
-
- @Test
- public void tableLimitRejectsTheNextJobExactlyAtLimit() {
- LanceIndexJobQuota quota = new LanceIndexJobQuota();
- Assertions.assertTrue(quota.tryAcquire(newJob(1L, CATALOG_ID, LOCATOR_A), 2, 100, 100));
- Assertions.assertTrue(quota.tryAcquire(newJob(2L, CATALOG_ID, LOCATOR_A), 2, 100, 100));
-
- LanceIndexJob third = newJob(3L, CATALOG_ID, LOCATOR_A);
- Assertions.assertFalse(quota.tryAcquire(third, 2, 100, 100));
- // A rejected acquire charges nothing at any level.
- Assertions.assertEquals(2L, quota.getGlobalCount());
- Assertions.assertEquals(2L, quota.getTableCount(third.getTableQuotaKey()));
-
- // The limit is per table/locator identity: another table still has room.
- Assertions.assertTrue(quota.tryAcquire(newJob(4L, CATALOG_ID, LOCATOR_B), 2, 100, 100));
- }
-
- @Test
- public void catalogLimitRejectsAcrossTables() {
- LanceIndexJobQuota quota = new LanceIndexJobQuota();
- Assertions.assertTrue(quota.tryAcquire(newJob(1L, CATALOG_ID, LOCATOR_A), 100, 2, 100));
- Assertions.assertTrue(quota.tryAcquire(newJob(2L, CATALOG_ID, LOCATOR_B), 100, 2, 100));
- Assertions.assertFalse(quota.tryAcquire(newJob(3L, CATALOG_ID, LOCATOR_A), 100, 2, 100));
-
- // Another catalog is a separate level.
- Assertions.assertTrue(quota.tryAcquire(newJob(4L, 20L, LOCATOR_A), 100, 2, 100));
- }
-
- @Test
- public void globalLimitRejectsAcrossCatalogs() {
- LanceIndexJobQuota quota = new LanceIndexJobQuota();
- Assertions.assertTrue(quota.tryAcquire(newJob(1L, CATALOG_ID, LOCATOR_A), 100, 100, 2));
- Assertions.assertTrue(quota.tryAcquire(newJob(2L, 20L, LOCATOR_A), 100, 100, 2));
- Assertions.assertFalse(quota.tryAcquire(newJob(3L, 30L, LOCATOR_B), 100, 100, 2));
- Assertions.assertEquals(2L, quota.getGlobalCount());
- }
-
- @Test
- public void nonPositiveLimitIsRejectedWithoutACharge() {
- LanceIndexJobQuota quota = new LanceIndexJobQuota();
- LanceIndexJob job = newJob(1L, CATALOG_ID, LOCATOR_A);
- Assertions.assertFalse(quota.tryAcquire(job, 0, 1, 1));
- Assertions.assertFalse(quota.tryAcquire(job, 1, 0, 1));
- Assertions.assertFalse(quota.tryAcquire(job, 1, 1, 0));
- Assertions.assertFalse(quota.tryAcquire(job, -1, 1, 1));
- Assertions.assertEquals(0L, quota.getGlobalCount());
- }
-
- @Test
- public void releaseRecoversCapacityAtAllLevels() {
- LanceIndexJobQuota quota = new LanceIndexJobQuota();
- LanceIndexJob first = newJob(1L, CATALOG_ID, LOCATOR_A);
- Assertions.assertTrue(quota.tryAcquire(first, 1, 1, 1));
- Assertions.assertFalse(quota.tryAcquire(newJob(2L, CATALOG_ID, LOCATOR_A), 1, 1, 1));
-
- quota.release(first);
- Assertions.assertEquals(0L, quota.getGlobalCount());
- Assertions.assertEquals(0L, quota.getCatalogCount(CATALOG_ID));
- Assertions.assertEquals(0L, quota.getTableCount(first.getTableQuotaKey()));
-
- Assertions.assertTrue(quota.tryAcquire(newJob(3L, CATALOG_ID, LOCATOR_A), 1, 1, 1));
- }
-
- @Test
- public void releaseUnderflowClampsAtZero() {
- LanceIndexJobQuota quota = new LanceIndexJobQuota();
- // Releasing a job that was never charged must not fail and must not go negative.
- quota.release(newJob(1L, CATALOG_ID, LOCATOR_A));
- Assertions.assertEquals(0L, quota.getGlobalCount());
- Assertions.assertEquals(0L, quota.getCatalogCount(CATALOG_ID));
- Assertions.assertTrue(quota.tryAcquire(newJob(2L, CATALOG_ID, LOCATOR_A), 1, 1, 1));
- }
-
- @Test
- public void rebuildMatchesIncrementalCounting() {
- List<LanceIndexJob> unresolved = new ArrayList<>();
- unresolved.add(newJob(1L, CATALOG_ID, LOCATOR_A));
- unresolved.add(newJob(2L, CATALOG_ID, LOCATOR_A));
- unresolved.add(newJob(3L, CATALOG_ID, LOCATOR_B));
- unresolved.add(newJob(4L, 20L, LOCATOR_A));
-
- LanceIndexJobQuota incremental = new LanceIndexJobQuota();
- for (LanceIndexJob job : unresolved) {
- Assertions.assertTrue(incremental.tryAcquire(job, 100, 100, 100));
- }
- LanceIndexJobQuota rebuilt = new LanceIndexJobQuota();
- rebuilt.rebuild(unresolved);
-
- Assertions.assertEquals(incremental.getGlobalCount(), rebuilt.getGlobalCount());
- Assertions.assertEquals(incremental.getCatalogCount(CATALOG_ID), rebuilt.getCatalogCount(CATALOG_ID));
- Assertions.assertEquals(incremental.getCatalogCount(20L), rebuilt.getCatalogCount(20L));
- Assertions.assertEquals(incremental.getTableCount(unresolved.get(0).getTableQuotaKey()),
- rebuilt.getTableCount(unresolved.get(0).getTableQuotaKey()));
- Assertions.assertEquals(incremental.getTableCount(unresolved.get(2).getTableQuotaKey()),
- rebuilt.getTableCount(unresolved.get(2).getTableQuotaKey()));
-
- rebuilt.rebuild(Collections.<LanceIndexJob>emptyList());
- Assertions.assertEquals(0L, rebuilt.getGlobalCount());
- Assertions.assertEquals(0L, rebuilt.getCatalogCount(CATALOG_ID));
- }
-
- @Test
- public void rebuildCountsUnresolvedJobsOnly() {
- List<LanceIndexJob> jobs = new ArrayList<>();
- // Active and unforced-unknown jobs count.
- jobs.add(jobInState(1L, LanceIndexJobMutationState.PENDING, LanceIndexJobRefreshState.NOT_REQUIRED, false));
- jobs.add(jobInState(2L, LanceIndexJobMutationState.RUNNING, LanceIndexJobRefreshState.NOT_REQUIRED, false));
- jobs.add(jobInState(3L, LanceIndexJobMutationState.UNKNOWN, LanceIndexJobRefreshState.NOT_REQUIRED, false));
- jobs.add(jobInState(4L, LanceIndexJobMutationState.COMMITTED, LanceIndexJobRefreshState.REQUIRED, false));
- jobs.add(jobInState(5L, LanceIndexJobMutationState.COMMITTED, LanceIndexJobRefreshState.RUNNING, false));
- jobs.add(jobInState(6L, LanceIndexJobMutationState.COMMITTED, LanceIndexJobRefreshState.FAILED, false));
- jobs.add(jobInState(7L, LanceIndexJobMutationState.NOT_COMMITTED, LanceIndexJobRefreshState.FAILED, false));
- // Resolved jobs do not count: forced UNKNOWN and terminal jobs with a settled refresh.
- jobs.add(jobInState(8L, LanceIndexJobMutationState.UNKNOWN, LanceIndexJobRefreshState.NOT_REQUIRED, true));
- jobs.add(jobInState(9L, LanceIndexJobMutationState.COMMITTED, LanceIndexJobRefreshState.DONE, false));
- jobs.add(jobInState(10L, LanceIndexJobMutationState.COMMITTED, LanceIndexJobRefreshState.NOT_REQUIRED, false));
- jobs.add(jobInState(11L, LanceIndexJobMutationState.NOT_COMMITTED, LanceIndexJobRefreshState.NOT_REQUIRED,
- false));
-
- List<LanceIndexJob> unresolved = new ArrayList<>();
- for (LanceIndexJob job : jobs) {
- if (job.isUnresolved()) {
- unresolved.add(job);
- }
- }
- Assertions.assertEquals(7, unresolved.size());
-
- LanceIndexJobQuota quota = new LanceIndexJobQuota();
- quota.rebuild(unresolved);
- Assertions.assertEquals(7L, quota.getGlobalCount());
- Assertions.assertEquals(7L, quota.getCatalogCount(CATALOG_ID));
- Assertions.assertEquals(7L, quota.getTableCount(unresolved.get(0).getTableQuotaKey()));
- }
-
- @Test
- public void tableQuotaKeyIdentityAndLocatorHiding() {
- LanceIndexJobQuota.TableQuotaKey key = new LanceIndexJobQuota.TableQuotaKey(CATALOG_ID, LOCATOR_A);
- Assertions.assertEquals(new LanceIndexJobQuota.TableQuotaKey(CATALOG_ID, LOCATOR_A), key);
- Assertions.assertEquals(new LanceIndexJobQuota.TableQuotaKey(CATALOG_ID, LOCATOR_A).hashCode(), key.hashCode());
- Assertions.assertNotEquals(new LanceIndexJobQuota.TableQuotaKey(20L, LOCATOR_A), key);
- Assertions.assertNotEquals(new LanceIndexJobQuota.TableQuotaKey(CATALOG_ID, LOCATOR_B), key);
- Assertions.assertFalse(key.toString().contains(LOCATOR_A));
- }
-
- private static LanceIndexJob newJob(long jobId, long catalogId, String locator) {
- return new LanceIndexJob(jobId, "tester", catalogId, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, locator,
- "idx" + jobId, "idx" + jobId,
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v", null, 1L, null);
- }
-
- private static LanceIndexJob jobInState(long jobId, LanceIndexJobMutationState mutationState,
- LanceIndexJobRefreshState refreshState, boolean forceReleased) {
- LanceIndexJob job = newJob(jobId, CATALOG_ID, LOCATOR_A);
- job.setMutationState(mutationState);
- job.setRefreshState(refreshState);
- job.setForceReleased(forceReleased);
- return job;
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobResultClassifyTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobResultClassifyTest.java
deleted file mode 100644
index cbceb49caf1..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobResultClassifyTest.java
+++ /dev/null
@@ -1,239 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-/**
- * Exhaustive coverage of the provider-result classification table: every saved typed
- * result code is classified into the durable (mutationState, refreshState,
- * completionReason) triple for the full cross product of 13 codes x 3 mutation types
- * x ifExists x externalMetadataAdvanced (156 combinations). The expected values are an
- * independent restatement of the design table, so a row flipped on either side fails.
- *
- * <p>Key invariants pinned here: PRE_INVOCATION_* always prove NOT_COMMITTED and owe a
- * refresh only when trusted revalidation observed external metadata advancement;
- * NATIVE_NOT_FOUND has no attribution for CREATE/REPLACE (UNKNOWN) but is a clean
- * NOT_COMMITTED for DROP; IF_CONDITION_NOOP exists only for DROP IF EXISTS + NOT_FOUND;
- * every other ambiguous post-invocation outcome is UNKNOWN + NOT_REQUIRED.
- */
-public class LanceIndexJobResultClassifyTest {
-
- @Test
- public void fullClassificationProductMatchesDesignTable() {
- int combos = 0;
- for (LanceIndexJobResultCode code : LanceIndexJobResultCode.values()) {
- for (LanceIndexJobMutationType type : LanceIndexJobMutationType.values()) {
- for (boolean ifExists : new boolean[]{false, true}) {
- for (boolean advanced : new boolean[]{false, true}) {
- LanceIndexJobResultCode.Classification classification =
- LanceIndexJobResultCode.classify(type, code, ifExists, advanced);
- String context = "code=" + code + ", type=" + type
- + ", ifExists=" + ifExists + ", advanced=" + advanced;
- Assertions.assertEquals(expectedMutationState(type, code),
- classification.getMutationState(), context);
- Assertions.assertEquals(expectedRefreshState(type, code, advanced),
- classification.getRefreshState(), context);
- Assertions.assertEquals(expectedCompletionReason(type, code, ifExists),
- classification.getCompletionReason(), context);
- combos++;
- }
- }
- }
- }
- Assertions.assertEquals(13 * 3 * 2 * 2, combos);
- }
-
- @Test
- public void ifConditionNoopExistsOnlyForDropIfExistsNotFound() {
- int noopCombos = 0;
- for (LanceIndexJobResultCode code : LanceIndexJobResultCode.values()) {
- for (LanceIndexJobMutationType type : LanceIndexJobMutationType.values()) {
- for (boolean ifExists : new boolean[]{false, true}) {
- for (boolean advanced : new boolean[]{false, true}) {
- LanceIndexJobResultCode.Classification classification =
- LanceIndexJobResultCode.classify(type, code, ifExists, advanced);
- if (classification.getCompletionReason() == LanceIndexJobCompletionReason.IF_CONDITION_NOOP) {
- noopCombos++;
- Assertions.assertEquals(LanceIndexJobMutationType.DROP, type);
- Assertions.assertEquals(LanceIndexJobResultCode.NATIVE_NOT_FOUND, code);
- Assertions.assertTrue(ifExists);
- }
- }
- }
- }
- }
- // Exactly the two advanced-flag variants of (DROP, NATIVE_NOT_FOUND, ifExists).
- Assertions.assertEquals(2, noopCombos);
- }
-
- @Test
- public void preInvocationRefreshOwesOnlyWhenExternalMetadataAdvanced() {
- for (LanceIndexJobResultCode code : LanceIndexJobResultCode.values()) {
- if (!code.isPreInvocation()) {
- continue;
- }
- for (LanceIndexJobMutationType type : LanceIndexJobMutationType.values()) {
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED,
- LanceIndexJobResultCode.classify(type, code, false, false).getRefreshState());
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED,
- LanceIndexJobResultCode.classify(type, code, false, true).getRefreshState());
- Assertions.assertEquals(LanceIndexJobMutationState.NOT_COMMITTED,
- LanceIndexJobResultCode.classify(type, code, true, true).getMutationState());
- }
- }
- }
-
- @Test
- public void nativeOkCommitsAndOwesRefreshForEveryMutationType() {
- for (LanceIndexJobMutationType type : LanceIndexJobMutationType.values()) {
- LanceIndexJobResultCode.Classification classification =
- LanceIndexJobResultCode.classify(type, LanceIndexJobResultCode.NATIVE_OK, false, false);
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED, classification.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, classification.getRefreshState());
- Assertions.assertEquals(LanceIndexJobCompletionReason.NONE, classification.getCompletionReason());
- }
- }
-
- @Test
- public void commitConflictIsNotCommittedWithRefreshOwed() {
- for (LanceIndexJobMutationType type : LanceIndexJobMutationType.values()) {
- LanceIndexJobResultCode.Classification classification = LanceIndexJobResultCode.classify(
- type, LanceIndexJobResultCode.NATIVE_COMMIT_CONFLICT, false, false);
- Assertions.assertEquals(LanceIndexJobMutationState.NOT_COMMITTED, classification.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, classification.getRefreshState());
- Assertions.assertEquals(LanceIndexJobCompletionReason.NONE, classification.getCompletionReason());
- }
- }
-
- @Test
- public void nativeNotFoundAttributionDependsOnMutationType() {
- for (LanceIndexJobMutationType type : new LanceIndexJobMutationType[]{
- LanceIndexJobMutationType.CREATE, LanceIndexJobMutationType.REPLACE}) {
- LanceIndexJobResultCode.Classification classification = LanceIndexJobResultCode.classify(
- type, LanceIndexJobResultCode.NATIVE_NOT_FOUND, false, false);
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, classification.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED, classification.getRefreshState());
- Assertions.assertEquals(LanceIndexJobCompletionReason.NONE, classification.getCompletionReason());
- }
- LanceIndexJobResultCode.Classification dropClassification = LanceIndexJobResultCode.classify(
- LanceIndexJobMutationType.DROP, LanceIndexJobResultCode.NATIVE_NOT_FOUND, false, false);
- Assertions.assertEquals(LanceIndexJobMutationState.NOT_COMMITTED, dropClassification.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, dropClassification.getRefreshState());
- Assertions.assertEquals(LanceIndexJobCompletionReason.NONE, dropClassification.getCompletionReason());
- }
-
- @Test
- public void ambiguousPostInvocationOutcomesAreUnknownWithoutRefresh() {
- for (LanceIndexJobResultCode code : new LanceIndexJobResultCode[]{
- LanceIndexJobResultCode.NATIVE_INVALID_ARGUMENT, LanceIndexJobResultCode.NATIVE_NOT_SUPPORTED,
- LanceIndexJobResultCode.NATIVE_INDEX, LanceIndexJobResultCode.NATIVE_IO,
- LanceIndexJobResultCode.NATIVE_INTERNAL, LanceIndexJobResultCode.NO_TRUSTED_RESULT}) {
- for (LanceIndexJobMutationType type : LanceIndexJobMutationType.values()) {
- LanceIndexJobResultCode.Classification classification =
- LanceIndexJobResultCode.classify(type, code, true, true);
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, classification.getMutationState(),
- "code=" + code + ", type=" + type);
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED, classification.getRefreshState(),
- "code=" + code + ", type=" + type);
- Assertions.assertEquals(LanceIndexJobCompletionReason.NONE, classification.getCompletionReason(),
- "code=" + code + ", type=" + type);
- }
- }
- }
-
- @Test
- public void nullResultCodeFallsBackToNoTrustedResult() {
- LanceIndexJobResultCode.Classification classification =
- LanceIndexJobResultCode.classify(LanceIndexJobMutationType.CREATE, null, false, false);
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, classification.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED, classification.getRefreshState());
- Assertions.assertEquals(LanceIndexJobCompletionReason.NONE, classification.getCompletionReason());
- }
-
- @Test
- public void isPreInvocationCoversExactlyTheFourTrustedRejections() {
- int preInvocationCount = 0;
- for (LanceIndexJobResultCode code : LanceIndexJobResultCode.values()) {
- if (code.isPreInvocation()) {
- preInvocationCount++;
- }
- }
- Assertions.assertEquals(4, preInvocationCount);
- Assertions.assertEquals(13, LanceIndexJobResultCode.values().length);
- }
-
- /**
- * Independent restatement of the design table, mutation-state column.
- */
- private static LanceIndexJobMutationState expectedMutationState(
- LanceIndexJobMutationType type, LanceIndexJobResultCode code) {
- if (isPreInvocationSpec(code)) {
- return LanceIndexJobMutationState.NOT_COMMITTED;
- }
- switch (code) {
- case NATIVE_OK:
- return LanceIndexJobMutationState.COMMITTED;
- case NATIVE_COMMIT_CONFLICT:
- return LanceIndexJobMutationState.NOT_COMMITTED;
- case NATIVE_NOT_FOUND:
- return type == LanceIndexJobMutationType.DROP
- ? LanceIndexJobMutationState.NOT_COMMITTED : LanceIndexJobMutationState.UNKNOWN;
- default:
- return LanceIndexJobMutationState.UNKNOWN;
- }
- }
-
- /**
- * Independent restatement of the design table, refresh-obligation column.
- */
- private static LanceIndexJobRefreshState expectedRefreshState(
- LanceIndexJobMutationType type, LanceIndexJobResultCode code, boolean externalMetadataAdvanced) {
- if (isPreInvocationSpec(code)) {
- return externalMetadataAdvanced
- ? LanceIndexJobRefreshState.REQUIRED : LanceIndexJobRefreshState.NOT_REQUIRED;
- }
- switch (code) {
- case NATIVE_OK:
- case NATIVE_COMMIT_CONFLICT:
- return LanceIndexJobRefreshState.REQUIRED;
- case NATIVE_NOT_FOUND:
- return type == LanceIndexJobMutationType.DROP
- ? LanceIndexJobRefreshState.REQUIRED : LanceIndexJobRefreshState.NOT_REQUIRED;
- default:
- return LanceIndexJobRefreshState.NOT_REQUIRED;
- }
- }
-
- /**
- * Independent restatement of the design table, completion-reason column.
- */
- private static LanceIndexJobCompletionReason expectedCompletionReason(
- LanceIndexJobMutationType type, LanceIndexJobResultCode code, boolean ifExists) {
- return type == LanceIndexJobMutationType.DROP && code == LanceIndexJobResultCode.NATIVE_NOT_FOUND && ifExists
- ? LanceIndexJobCompletionReason.IF_CONDITION_NOOP : LanceIndexJobCompletionReason.NONE;
- }
-
- private static boolean isPreInvocationSpec(LanceIndexJobResultCode code) {
- return code == LanceIndexJobResultCode.PRE_INVOCATION_STALE_ADMISSION
- || code == LanceIndexJobResultCode.PRE_INVOCATION_UNSUPPORTED_SCHEMA_CONTRACT
- || code == LanceIndexJobResultCode.PRE_INVOCATION_CREDENTIAL_EXPIRED
- || code == LanceIndexJobResultCode.PRE_INVOCATION_RESOURCE_REJECTED;
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobStateMachineTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobStateMachineTest.java
deleted file mode 100644
index 54a5945eb79..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobStateMachineTest.java
+++ /dev/null
@@ -1,640 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.apache.doris.common.DdlException;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.util.ArrayList;
-import java.util.List;
-
-/**
- * Lifecycle state-machine coverage driven through the master write paths of
- * {@link LanceIndexJobManager} with the edit-log seam captured in memory. Covers the
- * legal mutation chain PENDING -> RUNNING -> (COMMITTED | NOT_COMMITTED |
- * UNKNOWN), the impossibility of leaving UNKNOWN, the independent refresh chain
- * REQUIRED -> RUNNING -> DONE / FAILED with FAILED -> RUNNING retry, the
- * revision compare-and-set on every transition, and the fence/quota release timing
- * (immediately on a terminal outcome with refresh NOT_REQUIRED, only at refresh DONE
- * otherwise, never for UNKNOWN without FORCE).
- */
-public class LanceIndexJobStateMachineTest {
- private static final long CATALOG_ID = 10L;
- private static final String LOCATOR = "s3://bucket/dataset";
- private static final long BACKEND_ID = 1001L;
- private static final long BE_EPOCH = 55L;
- private static final String INVOCATION_ID = "invocation-1";
- private static final long DEADLINE_MS = 9999L;
-
- @Test
- public void createInitializesPendingRecord() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
-
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, stored.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED, stored.getRefreshState());
- Assertions.assertEquals(0L, stored.getRevision());
- Assertions.assertFalse(stored.isPossibleLiveOwned());
- Assertions.assertEquals(LanceIndexTerminationProof.NONE, stored.getTerminationProof());
- Assertions.assertTrue(stored.getCreateTimeMs() > 0L);
- Assertions.assertTrue(manager.isFenceHeld(stored.fenceKey()));
- Assertions.assertEquals(1L, manager.getQuota().getGlobalCount());
- Assertions.assertEquals(1L, manager.getQuota().getCatalogCount(CATALOG_ID));
- Assertions.assertEquals(1L, manager.getQuota().getTableCount(stored.getTableQuotaKey()));
- Assertions.assertEquals(1, manager.getUnresolvedJobs().size());
- Assertions.assertEquals(1, manager.editLog.size());
- }
-
- @Test
- public void pendingToRunningToCommitted() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
- Assertions.assertTrue(manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
-
- LanceIndexJob running = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.RUNNING, running.getMutationState());
- Assertions.assertEquals(1L, running.getRevision());
- Assertions.assertEquals(BACKEND_ID, running.getBackendId().longValue());
- Assertions.assertEquals(BE_EPOCH, running.getBeProcessEpoch().longValue());
- Assertions.assertEquals(INVOCATION_ID, running.getInvocationId());
- Assertions.assertEquals(DEADLINE_MS, running.getDeadlineMs().longValue());
- Assertions.assertTrue(running.holdsPossibleLiveSlot());
-
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- LanceIndexJob committed = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED, committed.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, committed.getRefreshState());
- Assertions.assertEquals(2L, committed.getRevision());
- Assertions.assertEquals(LanceIndexJobResultCode.NATIVE_OK, committed.getResult().getResultCode());
- Assertions.assertEquals(LanceIndexJobCompletionReason.NONE, committed.getResult().getCompletionReason());
- Assertions.assertTrue(containsJob(manager.getJobsNeedingRefresh(), committed.getJobId()));
- }
-
- @Test
- public void pendingToRunningToNotCommitted() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.PRE_INVOCATION_CREDENTIAL_EXPIRED)));
-
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.NOT_COMMITTED, stored.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED, stored.getRefreshState());
- Assertions.assertEquals(2L, stored.getRevision());
- }
-
- @Test
- public void pendingToRunningToUnknown() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NO_TRUSTED_RESULT)));
-
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, stored.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED, stored.getRefreshState());
- Assertions.assertEquals(2L, stored.getRevision());
- }
-
- @Test
- public void dropIfExistsNotFoundCompletesWithIfConditionNoop() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newDropJob(1L, "IdxA", true), 100, 100, 100);
- Assertions.assertTrue(manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_NOT_FOUND)));
-
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.NOT_COMMITTED, stored.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, stored.getRefreshState());
- Assertions.assertEquals(LanceIndexJobCompletionReason.IF_CONDITION_NOOP,
- stored.getResult().getCompletionReason());
- }
-
- @Test
- public void markRunningRejectsSecondDispatch() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
-
- Assertions.assertFalse(manager.markRunning(1L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertFalse(manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertEquals(LanceIndexJobMutationState.RUNNING, manager.getJob(1L).getMutationState());
- Assertions.assertEquals(1L, manager.getJob(1L).getRevision());
- }
-
- @Test
- public void markRunningRejectsWrongRevisionAndUnknownJob() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
-
- Assertions.assertFalse(manager.markRunning(1L, 5L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertFalse(manager.markRunning(404L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, manager.getJob(1L).getMutationState());
- Assertions.assertEquals(1, manager.editLog.size());
- }
-
- @Test
- public void completeRejectsNonRunningJob() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
-
- Assertions.assertFalse(manager.completeWithResult(1L, 0L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, manager.getJob(1L).getMutationState());
-
- createAndRun(manager, 2L, "IdxB");
- Assertions.assertTrue(manager.completeWithResult(2L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- Assertions.assertFalse(manager.completeWithResult(2L, 2L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED, manager.getJob(2L).getMutationState());
- }
-
- @Test
- public void unknownHasNoOutgoingTransitions() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NO_TRUSTED_RESULT)));
- int loggedRecords = manager.editLog.size();
-
- Assertions.assertFalse(manager.markRunning(1L, 2L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertFalse(manager.completeWithResult(1L, 2L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- Assertions.assertFalse(manager.markRefreshRunning(1L, 2L));
- Assertions.assertFalse(manager.markRefreshDone(1L, 2L));
- Assertions.assertFalse(manager.markRefreshFailed(1L, 2L));
-
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, stored.getMutationState());
- Assertions.assertEquals(2L, stored.getRevision());
- Assertions.assertEquals(loggedRecords, manager.editLog.size());
- }
-
- @Test
- public void refreshLifecycleReleasesFenceAndQuotaAtDone() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
-
- LanceIndexFenceKey fenceKey = manager.getJob(1L).fenceKey();
- Assertions.assertTrue(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(1L, manager.getQuota().getGlobalCount());
-
- Assertions.assertTrue(manager.markRefreshRunning(1L, 2L));
- Assertions.assertEquals(LanceIndexJobRefreshState.RUNNING, manager.getJob(1L).getRefreshState());
- Assertions.assertTrue(manager.isFenceHeld(fenceKey));
-
- Assertions.assertTrue(manager.markRefreshDone(1L, 3L));
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobRefreshState.DONE, stored.getRefreshState());
- Assertions.assertEquals(4L, stored.getRevision());
- Assertions.assertFalse(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(0L, manager.getQuota().getGlobalCount());
- Assertions.assertTrue(manager.getUnresolvedJobs().isEmpty());
- Assertions.assertTrue(manager.getJobsNeedingRefresh().isEmpty());
- }
-
- @Test
- public void refreshFailureKeepsFenceAndRetriesThroughRunning() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- LanceIndexFenceKey fenceKey = manager.getJob(1L).fenceKey();
-
- Assertions.assertTrue(manager.markRefreshRunning(1L, 2L));
- Assertions.assertTrue(manager.markRefreshFailed(1L, 3L));
- LanceIndexJob failed = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobRefreshState.FAILED, failed.getRefreshState());
- Assertions.assertTrue(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(1L, manager.getQuota().getGlobalCount());
- Assertions.assertTrue(containsJob(manager.getUnresolvedJobs(), failed.getJobId()));
-
- // FAILED -> RUNNING is the retry entry through the idempotent refresh path.
- Assertions.assertTrue(manager.markRefreshRunning(1L, 4L));
- Assertions.assertEquals(LanceIndexJobRefreshState.RUNNING, manager.getJob(1L).getRefreshState());
- Assertions.assertTrue(manager.markRefreshDone(1L, 5L));
- Assertions.assertFalse(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(0L, manager.getQuota().getGlobalCount());
- }
-
- @Test
- public void refreshTransitionsRejectedFromNotRequired() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.PRE_INVOCATION_RESOURCE_REJECTED)));
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED, manager.getJob(1L).getRefreshState());
-
- Assertions.assertFalse(manager.markRefreshRunning(1L, 2L));
- Assertions.assertFalse(manager.markRefreshDone(1L, 2L));
- Assertions.assertFalse(manager.markRefreshFailed(1L, 2L));
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED, manager.getJob(1L).getRefreshState());
- Assertions.assertEquals(2L, manager.getJob(1L).getRevision());
- }
-
- @Test
- public void refreshTransitionsRejectedFromDone() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- Assertions.assertTrue(manager.markRefreshRunning(1L, 2L));
- Assertions.assertTrue(manager.markRefreshDone(1L, 3L));
-
- Assertions.assertFalse(manager.markRefreshRunning(1L, 4L));
- Assertions.assertFalse(manager.markRefreshDone(1L, 4L));
- Assertions.assertFalse(manager.markRefreshFailed(1L, 4L));
- Assertions.assertEquals(LanceIndexJobRefreshState.DONE, manager.getJob(1L).getRefreshState());
- }
-
- @Test
- public void refreshTransitionIsRevisionGuarded() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
-
- Assertions.assertFalse(manager.markRefreshRunning(1L, 99L));
- Assertions.assertFalse(manager.markRefreshRunning(404L, 2L));
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, manager.getJob(1L).getRefreshState());
- Assertions.assertEquals(2L, manager.getJob(1L).getRevision());
- }
-
- @Test
- public void fenceAndQuotaReleasedImmediatelyWhenRefreshNotRequired() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- LanceIndexFenceKey fenceKey = manager.getJob(1L).fenceKey();
- Assertions.assertTrue(manager.isFenceHeld(fenceKey));
-
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.PRE_INVOCATION_STALE_ADMISSION)));
-
- Assertions.assertEquals(LanceIndexJobMutationState.NOT_COMMITTED, manager.getJob(1L).getMutationState());
- Assertions.assertFalse(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(0L, manager.getQuota().getGlobalCount());
- Assertions.assertEquals(0L, manager.getQuota().getCatalogCount(CATALOG_ID));
- Assertions.assertTrue(manager.getUnresolvedJobs().isEmpty());
- }
-
- @Test
- public void fenceAndQuotaSurviveUnknown() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- LanceIndexFenceKey fenceKey = manager.getJob(1L).fenceKey();
-
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_IO)));
-
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, manager.getJob(1L).getMutationState());
- Assertions.assertTrue(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(1L, manager.getQuota().getGlobalCount());
- Assertions.assertEquals(1, manager.getUnresolvedJobs().size());
- Assertions.assertTrue(manager.getJobsNeedingRefresh().isEmpty());
- }
-
- @Test
- public void terminationProofReleasesSlotOnly() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- LanceIndexFenceKey fenceKey = manager.getJob(1L).fenceKey();
- Assertions.assertTrue(manager.getJob(1L).holdsPossibleLiveSlot());
-
- Assertions.assertFalse(manager.recordTerminationProof(1L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID,
- LanceIndexTerminationProof.NONE));
- Assertions.assertFalse(manager.recordTerminationProof(1L, 99L, BACKEND_ID, BE_EPOCH, INVOCATION_ID,
- LanceIndexTerminationProof.CHILD_REAPED));
- Assertions.assertTrue(manager.recordTerminationProof(1L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID,
- LanceIndexTerminationProof.CHILD_REAPED));
-
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.RUNNING, stored.getMutationState());
- Assertions.assertEquals(LanceIndexTerminationProof.CHILD_REAPED, stored.getTerminationProof());
- Assertions.assertFalse(stored.holdsPossibleLiveSlot());
- Assertions.assertTrue(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(1L, manager.getQuota().getGlobalCount());
-
- // A slot may be proven exactly once.
- Assertions.assertFalse(manager.recordTerminationProof(1L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID,
- LanceIndexTerminationProof.BE_PROCESS_EPOCH_GONE));
- }
-
- @Test
- public void everyAcceptedTransitionWritesExactlyOneEditLogRecord() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
- manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS);
- manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH, result(LanceIndexJobResultCode.NATIVE_OK));
- manager.markRefreshRunning(1L, 2L);
- manager.markRefreshDone(1L, 3L);
- Assertions.assertEquals(5, manager.editLog.size());
-
- // Rejected transitions never reach the journal.
- Assertions.assertFalse(manager.markRefreshDone(1L, 4L));
- Assertions.assertFalse(manager.markRunning(1L, 4L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertEquals(5, manager.editLog.size());
- }
-
- @Test
- public void markRunningRejectsBlankInvocationId() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
-
- // A null/blank invocation identity would match a null field under Objects.equals
- // in completeWithResult and silently defeat the stale-callback guard.
- Assertions.assertFalse(manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, null, DEADLINE_MS));
- Assertions.assertFalse(manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, "", DEADLINE_MS));
- Assertions.assertFalse(manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, " \t\n", DEADLINE_MS));
-
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, stored.getMutationState());
- Assertions.assertEquals(0L, stored.getRevision());
- Assertions.assertNull(stored.getInvocationId());
- Assertions.assertEquals(1, manager.editLog.size());
-
- // A well-formed dispatch is still accepted afterwards.
- Assertions.assertTrue(manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertEquals(LanceIndexJobMutationState.RUNNING, manager.getJob(1L).getMutationState());
- }
-
- @Test
- public void failedRefreshJobStaysVisibleToTheRefreshDriver() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- Assertions.assertTrue(manager.markRefreshRunning(1L, 2L));
- Assertions.assertTrue(manager.markRefreshFailed(1L, 3L));
-
- // FAILED still owes the idempotent retry: the driver must see the job.
- Assertions.assertTrue(containsJob(manager.getJobsNeedingRefresh(), 1L));
-
- Assertions.assertTrue(manager.markRefreshRunning(1L, 4L));
- Assertions.assertTrue(manager.markRefreshDone(1L, 5L));
- Assertions.assertTrue(manager.getJobsNeedingRefresh().isEmpty());
-
- // Terminal jobs with refresh DONE or NOT_REQUIRED never show up.
- createAndRun(manager, 2L, "IdxB");
- Assertions.assertTrue(manager.completeWithResult(2L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.PRE_INVOCATION_RESOURCE_REJECTED)));
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED, manager.getJob(2L).getRefreshState());
- Assertions.assertTrue(manager.getJobsNeedingRefresh().isEmpty());
- }
-
- @Test
- public void createJobResetsLifecycleFieldsAndPublishesAPrivateCopy() throws DdlException {
- LanceIndexJob dirty = newCreateJob(1L, "IdxA");
- dirty.setRevision(9L);
- dirty.setMutationState(LanceIndexJobMutationState.RUNNING);
- dirty.setRefreshState(LanceIndexJobRefreshState.RUNNING);
- dirty.setResult(result(LanceIndexJobResultCode.NATIVE_OK));
- dirty.setBackendId(BACKEND_ID);
- dirty.setBeProcessEpoch(BE_EPOCH);
- dirty.setInvocationId(INVOCATION_ID);
- dirty.setDeadlineMs(DEADLINE_MS);
- dirty.setPossibleLiveOwned(true);
- dirty.setTerminationProof(LanceIndexTerminationProof.CHILD_REAPED);
- dirty.setForceReleased(true);
- dirty.setForceActor("admin");
- dirty.setForceTimeMs(7L);
- dirty.setForceNote("note");
- dirty.setForceWarning("warning");
-
- TestManager manager = new TestManager();
- manager.createJob(dirty, 100, 100, 100);
-
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertNotSame(dirty, stored);
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, stored.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.NOT_REQUIRED, stored.getRefreshState());
- Assertions.assertEquals(0L, stored.getRevision());
- Assertions.assertNull(stored.getResult());
- Assertions.assertNull(stored.getBackendId());
- Assertions.assertNull(stored.getBeProcessEpoch());
- Assertions.assertNull(stored.getInvocationId());
- Assertions.assertNull(stored.getDeadlineMs());
- Assertions.assertFalse(stored.isPossibleLiveOwned());
- Assertions.assertEquals(LanceIndexTerminationProof.NONE, stored.getTerminationProof());
- Assertions.assertFalse(stored.isForceReleased());
- Assertions.assertNull(stored.getForceActor());
- Assertions.assertNull(stored.getForceTimeMs());
- Assertions.assertNull(stored.getForceNote());
- Assertions.assertNull(stored.getForceWarning());
-
- // Mutating the caller's object after admission touches nothing inside the manager.
- dirty.setMutationState(LanceIndexJobMutationState.UNKNOWN);
- dirty.setNormalizedIndexName("idxb");
- Assertions.assertEquals(LanceIndexJobMutationState.PENDING, manager.getJob(1L).getMutationState());
- Assertions.assertTrue(manager.isFenceHeld(newCreateJob(2L, "idxa").fenceKey()));
- Assertions.assertFalse(manager.isFenceHeld(newCreateJob(3L, "IdxB").fenceKey()));
- }
-
- @Test
- public void refreshDoneOrFailedRequiresAPrecedingRefreshRunning() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, manager.getJob(1L).getRefreshState());
-
- // No direct REQUIRED -> DONE / FAILED: a refresh must actually run first.
- Assertions.assertFalse(manager.markRefreshDone(1L, 2L));
- Assertions.assertFalse(manager.markRefreshFailed(1L, 2L));
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, manager.getJob(1L).getRefreshState());
- Assertions.assertEquals(2L, manager.getJob(1L).getRevision());
- Assertions.assertEquals(3, manager.editLog.size());
- }
-
- @Test
- public void terminationProofNeedsASlotAndStillLandsAfterTheTerminalResult() throws DdlException {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
-
- // PENDING owns no possible-live slot: a proof has nothing to release.
- Assertions.assertFalse(manager.recordTerminationProof(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID,
- LanceIndexTerminationProof.CHILD_REAPED));
- Assertions.assertEquals(0L, manager.getJob(1L).getRevision());
- Assertions.assertEquals(1, manager.editLog.size());
-
- Assertions.assertTrue(manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- // The terminal outcome does not release the slot; the proof still lands afterwards.
- LanceIndexJob committed = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED, committed.getMutationState());
- Assertions.assertTrue(committed.holdsPossibleLiveSlot());
- LanceIndexFenceKey fenceKey = committed.fenceKey();
-
- Assertions.assertTrue(manager.recordTerminationProof(1L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID,
- LanceIndexTerminationProof.CHILD_REAPED));
- LanceIndexJob proven = manager.getJob(1L);
- Assertions.assertFalse(proven.holdsPossibleLiveSlot());
- // Fence and quota still follow the refresh rule, not the proof.
- Assertions.assertTrue(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(1L, manager.getQuota().getGlobalCount());
-
- Assertions.assertTrue(manager.markRefreshRunning(1L, 3L));
- Assertions.assertTrue(manager.markRefreshDone(1L, 4L));
- Assertions.assertFalse(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(0L, manager.getQuota().getGlobalCount());
- }
-
- @Test
- public void terminationProofAndResultUseTheImmutableDispatchRevisionInEitherOrder() throws DdlException {
- TestManager proofFirst = new TestManager();
- createAndRun(proofFirst, 1L, "IdxA");
- Assertions.assertEquals(1L, proofFirst.getJob(1L).getDispatchRevision());
-
- Assertions.assertFalse(proofFirst.recordTerminationProof(1L, 1L, BACKEND_ID + 1, BE_EPOCH,
- INVOCATION_ID, LanceIndexTerminationProof.CHILD_REAPED));
- Assertions.assertFalse(proofFirst.recordTerminationProof(1L, 1L, BACKEND_ID, BE_EPOCH + 1,
- INVOCATION_ID, LanceIndexTerminationProof.CHILD_REAPED));
- Assertions.assertFalse(proofFirst.recordTerminationProof(1L, 1L, BACKEND_ID, BE_EPOCH,
- "wrong-invocation", LanceIndexTerminationProof.CHILD_REAPED));
- Assertions.assertEquals(1L, proofFirst.getJob(1L).getRevision());
-
- Assertions.assertTrue(proofFirst.recordTerminationProof(1L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID,
- LanceIndexTerminationProof.CHILD_REAPED));
- Assertions.assertEquals(2L, proofFirst.getJob(1L).getRevision());
- Assertions.assertEquals(1L, proofFirst.getJob(1L).getDispatchRevision());
- Assertions.assertTrue(proofFirst.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- Assertions.assertEquals(LanceIndexJobMutationState.COMMITTED,
- proofFirst.getJob(1L).getMutationState());
-
- TestManager resultFirst = new TestManager();
- createAndRun(resultFirst, 2L, "IdxB");
- Assertions.assertTrue(resultFirst.completeWithResult(2L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- Assertions.assertTrue(resultFirst.recordTerminationProof(2L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID,
- LanceIndexTerminationProof.CHILD_REAPED));
- Assertions.assertFalse(resultFirst.getJob(2L).holdsPossibleLiveSlot());
- Assertions.assertEquals(1L, resultFirst.getJob(2L).getDispatchRevision());
-
- TestManager legacyResultFirst = new TestManager();
- createAndRun(legacyResultFirst, 3L, "IdxC");
- LanceIndexJob legacyRunning = legacyResultFirst.getJob(3L);
- legacyRunning.setDispatchRevision(null);
- legacyResultFirst.replayUpsertJob(legacyRunning);
- Assertions.assertTrue(legacyResultFirst.completeWithResult(3L, 1L, INVOCATION_ID, BE_EPOCH,
- result(LanceIndexJobResultCode.NATIVE_OK)));
- Assertions.assertEquals(1L, legacyResultFirst.getJob(3L).getDispatchRevision());
- Assertions.assertTrue(legacyResultFirst.recordTerminationProof(3L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID,
- LanceIndexTerminationProof.CHILD_REAPED));
- Assertions.assertFalse(legacyResultFirst.getJob(3L).holdsPossibleLiveSlot());
- }
-
- @Test
- public void refreshTransitionsRejectNonTerminalMutationState() {
- TestManager manager = new TestManager();
- LanceIndexJob corrupt = newCreateJob(1L, "IdxA");
- corrupt.setMutationState(LanceIndexJobMutationState.PENDING);
- corrupt.setRefreshState(LanceIndexJobRefreshState.REQUIRED);
- manager.replayUpsertJob(corrupt);
-
- Assertions.assertFalse(manager.markRefreshRunning(1L, 0L));
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, manager.getJob(1L).getRefreshState());
- Assertions.assertTrue(manager.editLog.isEmpty());
- }
-
- @Test
- public void admissionRejectsNonPositiveQuotaLimitsBeforeDurableWrite() {
- TestManager manager = new TestManager();
- Assertions.assertThrows(DdlException.class,
- () -> manager.createJob(newCreateJob(1L, "IdxA"), 0, 1, 1));
- Assertions.assertThrows(DdlException.class,
- () -> manager.createJob(newCreateJob(1L, "IdxA"), 1, 0, 1));
- Assertions.assertThrows(DdlException.class,
- () -> manager.createJob(newCreateJob(1L, "IdxA"), 1, 1, -1));
- Assertions.assertEquals(0, manager.getJobCount());
- Assertions.assertEquals(0L, manager.getQuota().getGlobalCount());
- Assertions.assertTrue(manager.editLog.isEmpty());
- }
-
- @Test
- public void notCommittedWithRefreshRequiredReleasesFenceAtRefreshDone() throws DdlException {
- TestManager manager = new TestManager();
- createAndRun(manager, 1L, "IdxA");
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- new LanceIndexJobResult(LanceIndexJobResultCode.NATIVE_COMMIT_CONFLICT,
- LanceIndexJobCompletionReason.NONE, "commit conflict", false)));
-
- LanceIndexJob stored = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.NOT_COMMITTED, stored.getMutationState());
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, stored.getRefreshState());
- LanceIndexFenceKey fenceKey = stored.fenceKey();
- Assertions.assertTrue(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(1L, manager.getQuota().getGlobalCount());
- Assertions.assertTrue(containsJob(manager.getJobsNeedingRefresh(), stored.getJobId()));
-
- Assertions.assertTrue(manager.markRefreshRunning(1L, 2L));
- Assertions.assertTrue(manager.markRefreshDone(1L, 3L));
- Assertions.assertFalse(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(0L, manager.getQuota().getGlobalCount());
- Assertions.assertTrue(manager.getUnresolvedJobs().isEmpty());
- }
-
- private static LanceIndexJob newCreateJob(long jobId, String displayName) {
- return new LanceIndexJob(jobId, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR,
- displayName, LanceIndexNameNormalizer.normalize(displayName),
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v",
- null, 7L, null);
- }
-
- private static boolean containsJob(List<LanceIndexJob> jobs, long jobId) {
- return jobs.stream().anyMatch(job -> job.getJobId() == jobId);
- }
-
- private static LanceIndexJob newDropJob(long jobId, String displayName, boolean ifExists) {
- return new LanceIndexJob(jobId, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR,
- displayName, LanceIndexNameNormalizer.normalize(displayName),
- LanceIndexJobMutationType.DROP, false, ifExists, null, "v",
- null, 7L, null);
- }
-
- private static LanceIndexJobResult result(LanceIndexJobResultCode code) {
- return new LanceIndexJobResult(code, LanceIndexJobCompletionReason.NONE, "sanitized message", false);
- }
-
- private static LanceIndexJob createAndRun(TestManager manager, long jobId, String displayName) throws DdlException {
- manager.createJob(newCreateJob(jobId, displayName), 100, 100, 100);
- Assertions.assertTrue(manager.markRunning(jobId, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, DEADLINE_MS));
- return manager.getJob(jobId);
- }
-
- /**
- * Edit-log seam: captures every durable record instead of writing the journal.
- */
- private static class TestManager extends LanceIndexJobManager {
- private final List<LanceIndexJob> editLog = new ArrayList<>();
-
- @Override
- protected void writeEditLog(LanceIndexJob job) {
- editLog.add(job);
- }
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobTest.java
deleted file mode 100644
index 1f61b184d19..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobTest.java
+++ /dev/null
@@ -1,303 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.apache.doris.persist.gson.GsonUtils;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.util.ArrayList;
-import java.util.List;
-
-/**
- * Record-level coverage for {@link LanceIndexJob}: the derived fence and quota keys,
- * the full {@link LanceIndexJob#isUnresolved()} matrix that keeps fence and quota
- * alive together, the possible-live slot semantics (released only by a matching
- * termination proof or a durable FORCE_RELEASE, never by a deadline), corrupt-record
- * fallbacks toward the safe direction, and the deep-copy constructor.
- */
-public class LanceIndexJobTest {
- private static final long CATALOG_ID = 10L;
- private static final String LOCATOR = "s3://bucket/dataset";
- private static final long BACKEND_ID = 1001L;
- private static final long BE_EPOCH = 55L;
- private static final String INVOCATION_ID = "invocation-1";
-
- @Test
- public void fenceKeyCarriesIdentityAndHidesLocator() {
- LanceIndexJob job = newCreateJob(1L, "IdxA");
- LanceIndexFenceKey key = job.fenceKey();
-
- Assertions.assertEquals(new LanceIndexFenceKey(CATALOG_ID, LanceIndexFenceKey.PROVIDER_DIRECTORY,
- LOCATOR, "idxa"), key);
- Assertions.assertEquals(new LanceIndexFenceKey(CATALOG_ID, LanceIndexFenceKey.PROVIDER_DIRECTORY,
- LOCATOR, "idxa").hashCode(), key.hashCode());
- Assertions.assertNotEquals(new LanceIndexFenceKey(20L, LanceIndexFenceKey.PROVIDER_DIRECTORY,
- LOCATOR, "idxa"), key);
- Assertions.assertNotEquals(new LanceIndexFenceKey(CATALOG_ID, "REST", LOCATOR, "idxa"), key);
- Assertions.assertNotEquals(new LanceIndexFenceKey(CATALOG_ID, LanceIndexFenceKey.PROVIDER_DIRECTORY,
- "s3://bucket/other", "idxa"), key);
- Assertions.assertNotEquals(new LanceIndexFenceKey(CATALOG_ID, LanceIndexFenceKey.PROVIDER_DIRECTORY,
- LOCATOR, "idxb"), key);
-
- // Fence-conflict messages may surface to users without target privileges.
- Assertions.assertFalse(key.toString().contains(LOCATOR));
- Assertions.assertTrue(key.toString().contains("idxa"));
- }
-
- @Test
- public void fenceKeyIsStableAcrossJobsWithTheSameIdentity() {
- LanceIndexJob first = newCreateJob(1L, "IdxA");
- LanceIndexJob second = newCreateJob(2L, "idxa");
- // Display case differs, normalized identity does not.
- Assertions.assertEquals(first.fenceKey(), second.fenceKey());
- Assertions.assertEquals(first.fenceKey().hashCode(), second.fenceKey().hashCode());
- }
-
- @Test
- public void tableQuotaKeyDerivesFromCatalogAndLocator() {
- LanceIndexJob job = newCreateJob(1L, "IdxA");
- Assertions.assertEquals(new LanceIndexJobQuota.TableQuotaKey(CATALOG_ID, LOCATOR), job.getTableQuotaKey());
- }
-
- @Test
- public void activeStatesAreAlwaysUnresolved() {
- for (LanceIndexJobRefreshState refresh : LanceIndexJobRefreshState.values()) {
- Assertions.assertTrue(jobInState(LanceIndexJobMutationState.PENDING, refresh, false).isUnresolved());
- Assertions.assertTrue(jobInState(LanceIndexJobMutationState.RUNNING, refresh, false).isUnresolved());
- }
- }
-
- @Test
- public void knownTerminalStatesFollowTheRefreshState() {
- for (LanceIndexJobMutationState terminal : new LanceIndexJobMutationState[]{
- LanceIndexJobMutationState.COMMITTED, LanceIndexJobMutationState.NOT_COMMITTED}) {
- Assertions.assertFalse(jobInState(terminal, LanceIndexJobRefreshState.NOT_REQUIRED, false).isUnresolved());
- Assertions.assertFalse(jobInState(terminal, LanceIndexJobRefreshState.DONE, false).isUnresolved());
- Assertions.assertTrue(jobInState(terminal, LanceIndexJobRefreshState.REQUIRED, false).isUnresolved());
- Assertions.assertTrue(jobInState(terminal, LanceIndexJobRefreshState.RUNNING, false).isUnresolved());
- // A failed refresh still holds the fence: the retry goes through the idempotent path.
- Assertions.assertTrue(jobInState(terminal, LanceIndexJobRefreshState.FAILED, false).isUnresolved());
- }
- }
-
- @Test
- public void unknownIsUnresolvedUntilForceReleased() {
- for (LanceIndexJobRefreshState refresh : LanceIndexJobRefreshState.values()) {
- Assertions.assertTrue(jobInState(LanceIndexJobMutationState.UNKNOWN, refresh, false).isUnresolved());
- Assertions.assertFalse(jobInState(LanceIndexJobMutationState.UNKNOWN, refresh, true).isUnresolved());
- }
- }
-
- @Test
- public void nullStatesFallBackToTheSafeDirection() {
- // A corrupt record missing its states must keep the fence (treated as UNKNOWN)
- // and must never become redispatchable PENDING.
- LanceIndexJob job = GsonUtils.GSON.fromJson(
- "{\"jid\":1,\"ms\":null,\"rs\":null}", LanceIndexJob.class);
- Assertions.assertNull(job.getMutationState());
- Assertions.assertNull(job.getRefreshState());
- Assertions.assertTrue(job.isUnresolved());
-
- LanceIndexJob forced = GsonUtils.GSON.fromJson(
- "{\"jid\":1,\"ms\":null,\"rs\":null,\"fr\":true}", LanceIndexJob.class);
- Assertions.assertFalse(forced.isUnresolved());
-
- // A terminal record missing its refresh state owes one (treated as REQUIRED).
- LanceIndexJob committedNoRefresh = GsonUtils.GSON.fromJson(
- "{\"jid\":1,\"ms\":\"COMMITTED\",\"rs\":null}", LanceIndexJob.class);
- Assertions.assertTrue(committedNoRefresh.isUnresolved());
- }
-
- @Test
- public void missingRefreshStateKeyFallsBackToRequired() {
- // A corrupt terminal record without the "rs" key at all keeps the fence: the
- // field initial value is REQUIRED, the same safe direction as the null fallback.
- LanceIndexJob missingKey = GsonUtils.GSON.fromJson(
- "{\"jid\":1,\"ms\":\"COMMITTED\"}", LanceIndexJob.class);
- Assertions.assertEquals(LanceIndexJobRefreshState.REQUIRED, missingKey.getRefreshState());
- Assertions.assertTrue(missingKey.isUnresolved());
-
- // A legal terminal record always carries the key explicitly.
- LanceIndexJob legal = GsonUtils.GSON.fromJson(
- "{\"jid\":1,\"ms\":\"COMMITTED\",\"rs\":\"NOT_REQUIRED\"}", LanceIndexJob.class);
- Assertions.assertFalse(legal.isUnresolved());
- }
-
- @Test
- public void possibleLiveSlotMatrix() {
- LanceIndexJob job = new LanceIndexJob();
- Assertions.assertFalse(job.holdsPossibleLiveSlot());
-
- job.setPossibleLiveOwned(true);
- Assertions.assertTrue(job.holdsPossibleLiveSlot());
-
- job.setTerminationProof(LanceIndexTerminationProof.CHILD_REAPED);
- Assertions.assertFalse(job.holdsPossibleLiveSlot());
-
- job.setTerminationProof(LanceIndexTerminationProof.NONE);
- job.setForceReleased(true);
- Assertions.assertFalse(job.holdsPossibleLiveSlot());
-
- // A corrupt record missing the proof is treated as NONE (slot still owned).
- LanceIndexJob nullProof = GsonUtils.GSON.fromJson(
- "{\"jid\":1,\"plo\":true,\"tp\":null}", LanceIndexJob.class);
- Assertions.assertNull(nullProof.getTerminationProof());
- Assertions.assertTrue(nullProof.holdsPossibleLiveSlot());
- }
-
- @Test
- public void terminationProofClearsSlotButKeepsFenceAndOutcome() throws Exception {
- TestManager manager = new TestManager();
- manager.createJob(newCreateJob(1L, "IdxA"), 100, 100, 100);
- manager.markRunning(1L, 0L, BACKEND_ID, BE_EPOCH, INVOCATION_ID, 9999L);
- LanceIndexFenceKey fenceKey = manager.getJob(1L).fenceKey();
-
- Assertions.assertTrue(manager.recordTerminationProof(1L, 1L, BACKEND_ID, BE_EPOCH, INVOCATION_ID,
- LanceIndexTerminationProof.BE_PROCESS_EPOCH_GONE));
- LanceIndexJob proven = manager.getJob(1L);
- Assertions.assertFalse(proven.holdsPossibleLiveSlot());
- Assertions.assertEquals(LanceIndexJobMutationState.RUNNING, proven.getMutationState());
- Assertions.assertTrue(manager.isFenceHeld(fenceKey));
- Assertions.assertEquals(1L, manager.getQuota().getGlobalCount());
-
- // The ambiguous result still lands afterwards: UNKNOWN keeps the fence, and the
- // already-recorded proof keeps the slot released.
- Assertions.assertTrue(manager.completeWithResult(1L, 1L, INVOCATION_ID, BE_EPOCH,
- new LanceIndexJobResult(LanceIndexJobResultCode.NO_TRUSTED_RESULT,
- LanceIndexJobCompletionReason.NONE, "ambiguous", false)));
- LanceIndexJob unknown = manager.getJob(1L);
- Assertions.assertEquals(LanceIndexJobMutationState.UNKNOWN, unknown.getMutationState());
- Assertions.assertFalse(unknown.holdsPossibleLiveSlot());
- Assertions.assertTrue(unknown.isUnresolved());
- Assertions.assertTrue(manager.isFenceHeld(fenceKey));
- }
-
- @Test
- public void copyConstructorDuplicatesEveryFieldIndependently() {
- LanceIndexJob original = newCreateJob(1L, "IdxA");
- original.setRevision(3L);
- original.setMutationState(LanceIndexJobMutationState.RUNNING);
- original.setRefreshState(LanceIndexJobRefreshState.RUNNING);
- original.setResult(new LanceIndexJobResult(LanceIndexJobResultCode.NATIVE_OK,
- LanceIndexJobCompletionReason.NONE, "ok", false));
- original.setBackendId(BACKEND_ID);
- original.setBeProcessEpoch(BE_EPOCH);
- original.setDispatchRevision(1L);
- original.setInvocationId(INVOCATION_ID);
- original.setDeadlineMs(123L);
- original.setPossibleLiveOwned(true);
- original.setForceActor("admin");
- original.setForceTimeMs(9L);
- original.setForceNote("note");
- original.setForceWarning("warning");
-
- LanceIndexJob copy = new LanceIndexJob(original);
- Assertions.assertEquals(GsonUtils.GSON.toJson(original), GsonUtils.GSON.toJson(copy));
-
- copy.setRevision(99L);
- copy.setMutationState(LanceIndexJobMutationState.UNKNOWN);
- copy.setPossibleLiveOwned(false);
- Assertions.assertEquals(3L, original.getRevision());
- Assertions.assertEquals(LanceIndexJobMutationState.RUNNING, original.getMutationState());
- Assertions.assertTrue(original.isPossibleLiveOwned());
- }
-
- @Test
- public void admissionConstructorRejectsNullIdentity() {
- Assertions.assertThrows(NullPointerException.class,
- () -> new LanceIndexJob(1L, "tester", CATALOG_ID, "db1", "tbl1",
- null, LOCATOR, "IdxA", "idxa",
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v", null, 7L, null));
- Assertions.assertThrows(NullPointerException.class,
- () -> new LanceIndexJob(1L, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, null, "IdxA", "idxa",
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v", null, 7L, null));
- Assertions.assertThrows(NullPointerException.class,
- () -> new LanceIndexJob(1L, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR, "IdxA", "idxa",
- null, false, false, "IVF_PQ", "v", null, 7L, null));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexJob(1L, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR, null, "idxa",
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v", null, 7L, null));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexJob(1L, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR, "IdxA", null,
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v", null, 7L, null));
- }
-
- @Test
- public void admissionRejectsPseudoCanonicalFenceIdentity() {
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexJob(1L, "tester", CATALOG_ID, "db1", "tbl1",
- "directory", LOCATOR, "IdxA", "idxa",
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v", null, 7L, null));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexJob(1L, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, "S3://bucket/dataset/", "IdxA", "idxa",
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v", null, 7L, null));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexJob(1L, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR, "IdxA", "IdxA",
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v", null, 7L, null));
- Assertions.assertThrows(IllegalArgumentException.class,
- () -> new LanceIndexJob(1L, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY,
- "https://bucket.example/ds?X-Amz-Signature=secret", "IdxA", "idxa",
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v", null, 7L, null));
- }
-
- @Test
- public void toStringNamesTheIndexButHidesTheLocator() {
- LanceIndexJob job = newCreateJob(1L, "IdxA");
- Assertions.assertTrue(job.toString().contains("IdxA"));
- Assertions.assertFalse(job.toString().contains("bucket"));
- Assertions.assertFalse(job.toString().contains(LOCATOR));
- }
-
- private static LanceIndexJob newCreateJob(long jobId, String displayName) {
- return new LanceIndexJob(jobId, "tester", CATALOG_ID, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, LOCATOR,
- displayName, LanceIndexNameNormalizer.normalize(displayName),
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v",
- null, 7L, null);
- }
-
- private static LanceIndexJob jobInState(LanceIndexJobMutationState mutationState,
- LanceIndexJobRefreshState refreshState, boolean forceReleased) {
- LanceIndexJob job = new LanceIndexJob();
- job.setMutationState(mutationState);
- job.setRefreshState(refreshState);
- job.setForceReleased(forceReleased);
- return job;
- }
-
- /**
- * Edit-log seam: captures every durable record instead of writing the journal.
- */
- private static class TestManager extends LanceIndexJobManager {
- private final List<LanceIndexJob> editLog = new ArrayList<>();
-
- @Override
- protected void writeEditLog(LanceIndexJob job) {
- editLog.add(job);
- }
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobWiringTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobWiringTest.java
deleted file mode 100644
index ecbd2b42799..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/datasource/lance/job/LanceIndexJobWiringTest.java
+++ /dev/null
@@ -1,78 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.datasource.lance.job;
-
-import org.apache.doris.catalog.Env;
-import org.apache.doris.common.io.CountingDataOutputStream;
-import org.apache.doris.persist.OperationType;
-import org.apache.doris.persist.meta.MetaPersistMethod;
-import org.apache.doris.persist.meta.PersistMetaModules;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.io.DataInputStream;
-import java.lang.reflect.Field;
-import java.lang.reflect.Method;
-import java.lang.reflect.Modifier;
-import java.util.ArrayList;
-import java.util.Collections;
-import java.util.List;
-
-public class LanceIndexJobWiringTest {
- private static final short LANCE_INDEX_JOB_OPCODE = 500;
- private static final String LANCE_INDEX_JOB_MODULE = "lanceIndexJobManager";
-
- @Test
- public void lanceIndexJobOpcodeIsUniquelyAssigned() throws IllegalAccessException {
- List<String> fieldsUsingOpcode = new ArrayList<>();
- for (Field field : OperationType.class.getDeclaredFields()) {
- if (Modifier.isStatic(field.getModifiers())
- && field.getType() == short.class
- && field.getShort(null) == LANCE_INDEX_JOB_OPCODE) {
- fieldsUsingOpcode.add(field.getName());
- }
- }
-
- Assertions.assertEquals(
- Collections.singletonList("OP_LANCE_INDEX_JOB_UPSERT"),
- fieldsUsingOpcode,
- "operation code 500 must remain uniquely assigned to Lance index job upserts");
- Assertions.assertEquals(LANCE_INDEX_JOB_OPCODE, OperationType.OP_LANCE_INDEX_JOB_UPSERT);
- }
-
- @Test
- public void lanceIndexJobManagerIsTheLastBaseImageModuleWithEnvBindings() throws Exception {
- Assertions.assertEquals(
- LANCE_INDEX_JOB_MODULE,
- PersistMetaModules.MODULE_NAMES.get(PersistMetaModules.MODULE_NAMES.size() - 1),
- "new image modules must be appended without reordering existing base modules");
-
- MetaPersistMethod persistMethod = PersistMetaModules.MODULES_MAP.get(LANCE_INDEX_JOB_MODULE);
- Assertions.assertNotNull(persistMethod);
-
- Method expectedReadMethod = Env.class.getDeclaredMethod(
- "loadLanceIndexJobManager", DataInputStream.class, long.class);
- Method expectedWriteMethod = Env.class.getDeclaredMethod(
- "saveLanceIndexJobManager", CountingDataOutputStream.class, long.class);
- Assertions.assertEquals(expectedReadMethod, persistMethod.readMethod);
- Assertions.assertEquals(expectedWriteMethod, persistMethod.writeMethod);
- Assertions.assertEquals(long.class, persistMethod.readMethod.getReturnType());
- Assertions.assertEquals(long.class, persistMethod.writeMethod.getReturnType());
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ShowLanceIndexJobsParserTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ShowLanceIndexJobsParserTest.java
deleted file mode 100644
index ef98991e534..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/parser/ShowLanceIndexJobsParserTest.java
+++ /dev/null
@@ -1,183 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.nereids.parser;
-
-import org.apache.doris.nereids.analyzer.UnboundResultSink;
-import org.apache.doris.nereids.analyzer.UnboundSlot;
-import org.apache.doris.nereids.trees.expressions.And;
-import org.apache.doris.nereids.trees.expressions.EqualTo;
-import org.apache.doris.nereids.trees.expressions.Expression;
-import org.apache.doris.nereids.trees.expressions.Like;
-import org.apache.doris.nereids.trees.expressions.literal.StringLikeLiteral;
-import org.apache.doris.nereids.trees.plans.Plan;
-import org.apache.doris.nereids.trees.plans.commands.ShowLanceIndexJobCommand;
-import org.apache.doris.nereids.trees.plans.commands.ShowLanceIndexJobsCommand;
-import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
-
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.util.Arrays;
-
-/**
- * Parser coverage for SHOW LANCE INDEX JOBS / SHOW LANCE INDEX JOB: all FROM and WHERE
- * forms, plus the guarantee that the new LANCE token stays non-reserved and can still be
- * used as an identifier. Illegal WHERE shapes that the grammar accepts as generic
- * expressions are rejected by the command, covered in ShowLanceIndexJobsCommandTest.
- */
-public class ShowLanceIndexJobsParserTest extends ParserTestBase {
- private final NereidsParser parser = new NereidsParser();
-
- private ShowLanceIndexJobsCommand parseJobs(String sql) {
- Plan plan = parser.parseSingle(sql);
- return Assertions.assertInstanceOf(ShowLanceIndexJobsCommand.class, plan);
- }
-
- private static void assertEqualToPredicate(Expression where, String slotName, String literalValue) {
- EqualTo equalTo = Assertions.assertInstanceOf(EqualTo.class, where);
- UnboundSlot slot = Assertions.assertInstanceOf(UnboundSlot.class, equalTo.child(0));
- Assertions.assertEquals(Arrays.asList(slotName), slot.getNameParts());
- // The parser hands out VarcharLiteral for ordinary-length strings; assert the common
- // string-literal base so the test states the grammar contract, not the length heuristic.
- StringLikeLiteral literal = Assertions.assertInstanceOf(StringLikeLiteral.class, equalTo.child(1));
- Assertions.assertEquals(literalValue, literal.getStringValue());
- }
-
- @Test
- public void testShowLanceIndexJobsBare() {
- ShowLanceIndexJobsCommand command = parseJobs("SHOW LANCE INDEX JOBS");
- Assertions.assertNull(command.getNameParts());
- Assertions.assertNull(command.getWhereClause());
- }
-
- @Test
- public void testShowLanceIndexJobsFromDb() {
- ShowLanceIndexJobsCommand command = parseJobs("SHOW LANCE INDEX JOBS FROM db1");
- Assertions.assertEquals(Arrays.asList("db1"), command.getNameParts());
-
- command = parseJobs("SHOW LANCE INDEX JOBS IN db1");
- Assertions.assertEquals(Arrays.asList("db1"), command.getNameParts());
- }
-
- @Test
- public void testShowLanceIndexJobsFromCatalogDb() {
- ShowLanceIndexJobsCommand command = parseJobs("SHOW LANCE INDEX JOBS FROM ctl1.db1");
- Assertions.assertEquals(Arrays.asList("ctl1", "db1"), command.getNameParts());
- }
-
- @Test
- public void testShowLanceIndexJobsWhere() {
- ShowLanceIndexJobsCommand command = parseJobs("SHOW LANCE INDEX JOBS WHERE TableName = \"tbl1\"");
- assertEqualToPredicate(command.getWhereClause(), "TableName", "tbl1");
-
- command = parseJobs("SHOW LANCE INDEX JOBS WHERE State = \"PENDING\"");
- assertEqualToPredicate(command.getWhereClause(), "State", "PENDING");
- }
-
- @Test
- public void testShowLanceIndexJobsWhereLikeParses() {
- // The LIKE syntax parses into a Like expression; rejecting it is a command-level
- // narrowing (ShowLanceIndexJobsCommandTest covers the rejection with hand-built
- // expressions), so this pins the grammar side of that split.
- ShowLanceIndexJobsCommand command = parseJobs("SHOW LANCE INDEX JOBS WHERE TableName LIKE \"t%\"");
- Like like = Assertions.assertInstanceOf(Like.class, command.getWhereClause());
- UnboundSlot slot = Assertions.assertInstanceOf(UnboundSlot.class, like.child(0));
- Assertions.assertEquals(Arrays.asList("TableName"), slot.getNameParts());
- StringLikeLiteral pattern = Assertions.assertInstanceOf(StringLikeLiteral.class, like.child(1));
- Assertions.assertEquals("t%", pattern.getStringValue());
- }
-
- @Test
- public void testShowLanceIndexJobsWhereReversedLiteralParses() {
- // A generic boolean expression: the grammar does not order slot vs literal, the command
- // does (the reversed shape is one of the WHERE forms the command rejects).
- ShowLanceIndexJobsCommand command = parseJobs("SHOW LANCE INDEX JOBS WHERE \"t\" = TableName");
- EqualTo equalTo = Assertions.assertInstanceOf(EqualTo.class, command.getWhereClause());
- StringLikeLiteral literal = Assertions.assertInstanceOf(StringLikeLiteral.class, equalTo.child(0));
- Assertions.assertEquals("t", literal.getStringValue());
- UnboundSlot slot = Assertions.assertInstanceOf(UnboundSlot.class, equalTo.child(1));
- Assertions.assertEquals(Arrays.asList("TableName"), slot.getNameParts());
- }
-
- @Test
- public void testShowLanceIndexJobsInCatalogDbWithCombinedWhere() {
- ShowLanceIndexJobsCommand command = parseJobs("SHOW LANCE INDEX JOBS IN ctl1.db1 "
- + "WHERE TableName = \"t\" AND State = \"PENDING\"");
- Assertions.assertEquals(Arrays.asList("ctl1", "db1"), command.getNameParts());
- And where = Assertions.assertInstanceOf(And.class, command.getWhereClause());
- assertEqualToPredicate(where.child(0), "TableName", "t");
- assertEqualToPredicate(where.child(1), "State", "PENDING");
- }
-
- @Test
- public void testShowLanceIndexJobsLowercaseParses() {
- // The lexer stream is case-insensitive; the new keywords must behave like the old ones.
- ShowLanceIndexJobsCommand command = parseJobs(
- "show lance index jobs from ctl1.db1 where TableName = \"t\" and State = \"pending\"");
- Assertions.assertEquals(Arrays.asList("ctl1", "db1"), command.getNameParts());
- Assertions.assertNotNull(command.getWhereClause());
- }
-
- @Test
- public void testShowLanceIndexJobParses() {
- Plan plan = parser.parseSingle("SHOW LANCE INDEX JOB 123");
- ShowLanceIndexJobCommand command = Assertions.assertInstanceOf(ShowLanceIndexJobCommand.class, plan);
- Assertions.assertEquals(123L, command.getJobId());
- }
-
- @Test
- public void testFromWithTooManyPartsStillParses() {
- // The grammar accepts any multipart identifier; the command rejects more than two parts.
- ShowLanceIndexJobsCommand command = parseJobs("SHOW LANCE INDEX JOBS FROM ctl1.db1.tbl1");
- Assertions.assertEquals(3, command.getNameParts().size());
- }
-
- @Test
- public void testIllegalWhereShapesStillParse() {
- // The grammar parses any boolean expression; shape validation lives in the command.
- Assertions.assertNotNull(parseJobs(
- "SHOW LANCE INDEX JOBS WHERE TableName = \"t\" OR State = \"PENDING\"").getWhereClause());
- Assertions.assertNotNull(parseJobs(
- "SHOW LANCE INDEX JOBS WHERE Foo = \"t\"").getWhereClause());
- Assertions.assertNotNull(parseJobs(
- "SHOW LANCE INDEX JOBS WHERE TableName = \"t\" AND tablename = \"t\"").getWhereClause());
- Assertions.assertNotNull(parseJobs(
- "SHOW LANCE INDEX JOBS WHERE State = \"BOGUS\"").getWhereClause());
- }
-
- @Test
- public void testLanceStaysUsableAsIdentifier() {
- // LANCE is added as a non-reserved keyword: it must still work as a column name.
- Plan plan = parser.parseSingle("SELECT lance FROM t");
- UnboundResultSink<?> sink = Assertions.assertInstanceOf(UnboundResultSink.class, plan);
- Assertions.assertInstanceOf(LogicalProject.class, sink.child());
-
- Assertions.assertNotNull(parser.parseSingle("SELECT lance FROM t WHERE lance > 0"));
- }
-
- @Test
- public void testLanceStaysUsableAsDatabaseAndCatalogName() {
- // Non-reserved in qualified names too: db and catalog parts named lance parse as such,
- // not as the LANCE token of the SHOW LANCE INDEX JOBS prefix.
- ShowLanceIndexJobsCommand command = parseJobs("SHOW LANCE INDEX JOBS FROM lance");
- Assertions.assertEquals(Arrays.asList("lance"), command.getNameParts());
-
- command = parseJobs("SHOW LANCE INDEX JOBS FROM lance.lance");
- Assertions.assertEquals(Arrays.asList("lance", "lance"), command.getNameParts());
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandLanceAdmissionTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandLanceAdmissionTest.java
index 8ac5fc15cc7..a1736feccc1 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandLanceAdmissionTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/AlterTableCommandLanceAdmissionTest.java
@@ -38,15 +38,10 @@ import org.apache.doris.datasource.lance.LanceExternalTable;
import org.apache.doris.datasource.lance.LanceIndexAdmissionSnapshot;
import org.apache.doris.datasource.lance.LanceIndexAdmissionSnapshot.PhysicalIndexInfo;
import org.apache.doris.datasource.lance.index.LanceShowIndexInfo;
-import org.apache.doris.datasource.lance.job.LanceIndexJob;
-import org.apache.doris.datasource.lance.job.LanceIndexJobManager;
-import org.apache.doris.datasource.lance.job.LanceIndexJobMutationType;
import org.apache.doris.mysql.privilege.AccessControllerManager;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.nereids.parser.NereidsParser;
import org.apache.doris.qe.ConnectContext;
-import org.apache.doris.qe.ResultSet;
-import org.apache.doris.qe.ShowResultSet;
import org.apache.doris.qe.StmtExecutor;
import org.apache.arrow.vector.types.FloatingPointPrecision;
@@ -58,7 +53,6 @@ import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.lance.schema.LanceField;
-import org.mockito.ArgumentCaptor;
import org.mockito.MockedStatic;
import org.mockito.Mockito;
@@ -66,14 +60,14 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Map;
-import java.util.concurrent.atomic.AtomicLong;
/**
* Command rewiring coverage for Lance index admission: with the gate off the typed rejection is
* 5102 from {@code validate()} without any metadata read or id allocation; with the gate on the
* top-level CREATE/DROP INDEX op bypasses the generic op loop and {@code alterTable}, runs
- * admission in {@code run()}, and answers a single-column {@code JobId} result set (one row when
- * admitted, zero rows on an IF no-op). The 3A static-validation and REST/alter=true rejections
+ * admission in {@code run()}, completes an IF preflight no-op as a plain success, and rejects a
+ * mutation that would be admitted with the shared not-supported error (the synchronous
+ * execution path has not landed). The 3A static-validation and REST/alter=true rejections
* are unchanged in both modes.
*/
public class AlterTableCommandLanceAdmissionTest {
@@ -109,19 +103,9 @@ public class AlterTableCommandLanceAdmissionTest {
ConnectContext.remove();
}
- /** Edit-log seam: captures durable records instead of writing the journal (3B pattern). */
- private static class TestManager extends LanceIndexJobManager {
- private final List<LanceIndexJob> editLog = new ArrayList<>();
-
- @Override
- protected void writeEditLog(LanceIndexJob job) {
- editLog.add(job);
- }
- }
-
/**
* The 3A LanceFixture, extended for admission: remote/local names, the snapshot loader
- * entry point, real catalog/job managers behind the Env mock, and a mocked id allocator.
+ * entry point, and a real catalog manager behind the Env mock.
*/
private static class LanceFixture implements AutoCloseable {
private final MockedStatic<Env> mockedEnv;
@@ -130,8 +114,6 @@ public class AlterTableCommandLanceAdmissionTest {
private final LanceExternalCatalog catalog;
private final LanceExternalDatabase database;
private final LanceExternalTable table;
- private final TestManager manager;
- private final AtomicLong idAllocator;
private final StmtExecutor executor;
LanceFixture(boolean restCatalog) throws Exception {
@@ -142,8 +124,6 @@ public class AlterTableCommandLanceAdmissionTest {
catalog = Mockito.mock(LanceExternalCatalog.class);
database = Mockito.mock(LanceExternalDatabase.class);
table = Mockito.mock(LanceExternalTable.class);
- manager = new TestManager();
- idAllocator = new AtomicLong(100L);
executor = Mockito.mock(StmtExecutor.class);
mockedEnv.when(Env::getCurrentEnv).thenReturn(env);
@@ -162,8 +142,6 @@ public class AlterTableCommandLanceAdmissionTest {
Mockito.when(database.getFullName()).thenReturn(DB);
Mockito.when(table.getRemoteName()).thenReturn(REMOTE_TBL);
Mockito.when(table.getName()).thenReturn(TBL);
- Mockito.when(env.getNextId()).thenAnswer(invocation -> idAllocator.incrementAndGet());
- Mockito.when(env.getLanceIndexJobManager()).thenReturn(manager);
Mockito.when(table.getColumn(Mockito.anyString())).thenAnswer(invocation -> {
String name = invocation.getArgument(0);
if ("v".equalsIgnoreCase(name)) {
@@ -294,11 +272,6 @@ public class AlterTableCommandLanceAdmissionTest {
return command;
}
- private static void assertSingleJobIdColumn(ResultSet resultSet) {
- Assertions.assertEquals(1, resultSet.getMetaData().getColumnCount());
- Assertions.assertEquals("JobId", resultSet.getMetaData().getColumns().get(0).getName());
- }
-
// ------------------------------------------------------------------
// Gate off: 5102 from validate(), no metadata read, no id allocation
// ------------------------------------------------------------------
@@ -322,8 +295,6 @@ public class AlterTableCommandLanceAdmissionTest {
.loadTableIndexAdmissionSnapshot(Mockito.anyString(), Mockito.anyString());
Mockito.verify(fixture.env, Mockito.never()).getNextId();
Mockito.verify(fixture.executor, Mockito.never()).sendResultSet(Mockito.any());
- Assertions.assertEquals(0, fixture.manager.getJobCount());
- Assertions.assertTrue(fixture.manager.editLog.isEmpty());
}
}
@@ -337,8 +308,6 @@ public class AlterTableCommandLanceAdmissionTest {
"CREATE OR REPLACE INDEX idx ON " + CTL + "." + DB + "." + TBL
+ " (c) USING BTREE"));
Mockito.verify(fixture.env, Mockito.never()).getNextId();
- Assertions.assertEquals(0, fixture.manager.getJobCount());
- Assertions.assertTrue(fixture.manager.editLog.isEmpty());
}
}
@@ -353,47 +322,63 @@ public class AlterTableCommandLanceAdmissionTest {
runAndGetMessage(fixture.executor,
"DROP INDEX IF EXISTS idx ON " + CTL + "." + DB + "." + TBL));
Mockito.verify(fixture.env, Mockito.never()).getNextId();
- Assertions.assertEquals(0, fixture.manager.getJobCount());
- Assertions.assertTrue(fixture.manager.editLog.isEmpty());
}
}
// ------------------------------------------------------------------
- // Gate on: admission in run(), JobId result set, alterTable bypassed
+ // Gate on: admission in run(), IF no-ops complete, mutations reject as
+ // unsupported, alterTable bypassed
// ------------------------------------------------------------------
@Test
- public void gateOnCreateIndexAdmitsAndReturnsJobId() throws Exception {
+ public void gateOnCreateIndexValidatesThenRejectsAsUnsupported() throws Exception {
Config.enable_lance_index_mutation = true;
try (LanceFixture fixture = new LanceFixture(false)) {
fixture.respondWithSnapshot(Collections.emptyList(), Collections.emptyList());
- run(fixture.executor, "CREATE INDEX MyIdx ON " + CTL + "." + DB + "." + TBL
- + " (v) USING ANN " + VALID_ANN_PROPERTIES);
+ AnalysisException exception = runAndGetCommonAnalysisException(fixture.executor,
+ "CREATE INDEX MyIdx ON " + CTL + "." + DB + "." + TBL
+ + " (v) USING ANN " + VALID_ANN_PROPERTIES);
+ Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED,
+ exception.getMysqlErrorCode());
+ Assertions.assertEquals("CREATE INDEX is not supported for Lance catalog tables",
+ exception.getDetailMessage());
- ArgumentCaptor<ResultSet> captor = ArgumentCaptor.forClass(ResultSet.class);
- Mockito.verify(fixture.executor, Mockito.times(1)).sendResultSet(captor.capture());
- assertSingleJobIdColumn(captor.getValue());
- Assertions.assertEquals(1, captor.getValue().getResultRows().size());
- long jobId = fixture.idAllocator.get();
- Assertions.assertEquals(String.valueOf(jobId),
- captor.getValue().getResultRows().get(0).get(0));
+ // The full preflight ran against the pinned snapshot, then rejected without any
+ // result set, id allocation, or generic alter path.
+ Mockito.verify(fixture.catalog, Mockito.times(1))
+ .loadTableIndexAdmissionSnapshot(REMOTE_DB, REMOTE_TBL);
+ Mockito.verify(fixture.executor, Mockito.never()).sendResultSet(Mockito.any());
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
+ Mockito.verify(fixture.env, Mockito.never()).alterTable(Mockito.any());
+ }
+ }
+
+ @Test
+ public void gateOnDropIndexValidatesThenRejectsAsUnsupported() throws Exception {
+ Config.enable_lance_index_mutation = true;
+ try (LanceFixture fixture = new LanceFixture(false)) {
+ fixture.respondWithSnapshot(
+ Collections.singletonList(new LanceShowIndexInfo("idx",
+ Collections.singletonList("c"), "BTREE", "{}")),
+ Collections.singletonList(
+ new PhysicalIndexInfo("idx", "uuid-1", DATASET_VERSION, "SCALAR")));
- // Admission short-circuits the generic alter path entirely.
+ AnalysisException exception = runAndGetCommonAnalysisException(fixture.executor,
+ "DROP INDEX idx ON " + CTL + "." + DB + "." + TBL);
+ Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED,
+ exception.getMysqlErrorCode());
+ Assertions.assertEquals("DROP INDEX is not supported for Lance catalog tables",
+ exception.getDetailMessage());
+ Mockito.verify(fixture.catalog, Mockito.times(1))
+ .loadTableIndexAdmissionSnapshot(REMOTE_DB, REMOTE_TBL);
+ Mockito.verify(fixture.env, Mockito.never()).getNextId();
Mockito.verify(fixture.env, Mockito.never()).alterTable(Mockito.any());
- Mockito.verify(fixture.env, Mockito.times(1)).getNextId();
- Assertions.assertEquals(1, fixture.manager.getJobCount());
- Assertions.assertEquals(1, fixture.manager.editLog.size());
- LanceIndexJob job = fixture.manager.getJob(jobId);
- Assertions.assertEquals(LanceIndexJobMutationType.CREATE, job.getMutationType());
- Assertions.assertEquals("MyIdx", job.getDisplayIndexName());
- Assertions.assertEquals(DB, job.getDbName());
- Assertions.assertEquals(TBL, job.getTableName());
}
}
@Test
- public void gateOnCreateIfNotExistsMatchingReturnsZeroRows() throws Exception {
+ public void gateOnCreateIfNotExistsMatchingIsANoop() throws Exception {
Config.enable_lance_index_mutation = true;
try (LanceFixture fixture = new LanceFixture(false)) {
fixture.respondWithSnapshot(
@@ -405,32 +390,24 @@ public class AlterTableCommandLanceAdmissionTest {
run(fixture.executor, "CREATE INDEX IF NOT EXISTS idx ON " + CTL + "." + DB + "." + TBL
+ " (v) USING ANN " + VALID_ANN_PROPERTIES);
- ArgumentCaptor<ResultSet> captor = ArgumentCaptor.forClass(ResultSet.class);
- Mockito.verify(fixture.executor, Mockito.times(1)).sendResultSet(captor.capture());
- assertSingleJobIdColumn(captor.getValue());
- // The IF no-op creates no job: same schema, zero rows.
- Assertions.assertTrue(captor.getValue().getResultRows().isEmpty());
+ // The IF no-op completes as a plain success: no result set, no id, no alter path.
+ Mockito.verify(fixture.executor, Mockito.never()).sendResultSet(Mockito.any());
Mockito.verify(fixture.env, Mockito.never()).alterTable(Mockito.any());
Mockito.verify(fixture.env, Mockito.never()).getNextId();
- Assertions.assertEquals(0, fixture.manager.getJobCount());
- Assertions.assertTrue(fixture.manager.editLog.isEmpty());
}
}
@Test
- public void gateOnDropIfExistsAbsentReturnsZeroRows() throws Exception {
+ public void gateOnDropIfExistsAbsentIsANoop() throws Exception {
Config.enable_lance_index_mutation = true;
try (LanceFixture fixture = new LanceFixture(false)) {
fixture.respondWithSnapshot(Collections.emptyList(), Collections.emptyList());
run(fixture.executor, "DROP INDEX IF EXISTS idx ON " + CTL + "." + DB + "." + TBL);
- ArgumentCaptor<ResultSet> captor = ArgumentCaptor.forClass(ResultSet.class);
- Mockito.verify(fixture.executor, Mockito.times(1)).sendResultSet(captor.capture());
- assertSingleJobIdColumn(captor.getValue());
- Assertions.assertTrue(captor.getValue().getResultRows().isEmpty());
+ Mockito.verify(fixture.executor, Mockito.never()).sendResultSet(Mockito.any());
Mockito.verify(fixture.env, Mockito.never()).getNextId();
- Assertions.assertEquals(0, fixture.manager.getJobCount());
+ Mockito.verify(fixture.env, Mockito.never()).alterTable(Mockito.any());
}
}
@@ -451,31 +428,6 @@ public class AlterTableCommandLanceAdmissionTest {
Mockito.verify(fixture.executor, Mockito.never()).sendResultSet(Mockito.any());
Mockito.verify(fixture.env, Mockito.never()).getNextId();
Mockito.verify(fixture.env, Mockito.never()).alterTable(Mockito.any());
- Assertions.assertEquals(0, fixture.manager.getJobCount());
- Assertions.assertTrue(fixture.manager.editLog.isEmpty());
- }
- }
-
- @Test
- public void gateOnProxyExecutionUsesTheProxyResultSet() throws Exception {
- Config.enable_lance_index_mutation = true;
- try (LanceFixture fixture = new LanceFixture(false)) {
- fixture.respondWithSnapshot(
- Collections.singletonList(new LanceShowIndexInfo("idx",
- Collections.singletonList("c"), "BTREE", "{}")),
- Collections.singletonList(
- new PhysicalIndexInfo("idx", "uuid-1", DATASET_VERSION, "SCALAR")));
- Mockito.when(fixture.executor.isProxy()).thenReturn(true);
-
- run(fixture.executor, "DROP INDEX idx ON " + CTL + "." + DB + "." + TBL);
-
- ArgumentCaptor<ShowResultSet> captor = ArgumentCaptor.forClass(ShowResultSet.class);
- Mockito.verify(fixture.executor, Mockito.times(1)).setProxyShowResultSet(captor.capture());
- Mockito.verify(fixture.executor, Mockito.never()).sendResultSet(Mockito.any());
- assertSingleJobIdColumn(captor.getValue());
- Assertions.assertEquals(1, captor.getValue().getResultRows().size());
- Assertions.assertEquals(String.valueOf(fixture.idAllocator.get()),
- captor.getValue().getResultRows().get(0).get(0));
}
}
@@ -581,11 +533,20 @@ public class AlterTableCommandLanceAdmissionTest {
Config.enable_lance_index_mutation = true;
try (LanceFixture fixture = new LanceFixture(false)) {
fixture.respondWithSnapshot(Collections.emptyList(), Collections.emptyList());
- AlterTableCommand command = run(fixture.executor,
+ // The first run of a fresh command reaches admission (exactly one snapshot read)
+ // and ends in the shared not-supported rejection; a second run of the same
+ // command object re-validates, collects a second op, and trips the
+ // exactly-one-op precondition before any further metadata read.
+ AlterTableCommand command = (AlterTableCommand) parser.parseSingle(
"CREATE INDEX idx ON " + CTL + "." + DB + "." + TBL + " (c) USING BTREE");
+ AnalysisException first = Assertions.assertThrows(AnalysisException.class,
+ () -> command.run(connectContext, fixture.executor));
+ Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_OPERATION_NOT_SUPPORTED,
+ first.getMysqlErrorCode());
Assertions.assertThrows(IllegalStateException.class,
() -> command.run(connectContext, fixture.executor));
- Mockito.verify(fixture.env, Mockito.times(1)).getNextId();
+ Mockito.verify(fixture.catalog, Mockito.times(1))
+ .loadTableIndexAdmissionSnapshot(REMOTE_DB, REMOTE_TBL);
}
}
@@ -620,8 +581,6 @@ public class AlterTableCommandLanceAdmissionTest {
Mockito.verify(fixture.catalog, Mockito.never())
.loadTableIndexAdmissionSnapshot(Mockito.anyString(), Mockito.anyString());
Mockito.verify(fixture.executor, Mockito.never()).sendResultSet(Mockito.any());
- Assertions.assertEquals(0, fixture.manager.getJobCount());
- Assertions.assertTrue(fixture.manager.editLog.isEmpty());
}
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobCommandTest.java
deleted file mode 100644
index c49ad8ee69e..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobCommandTest.java
+++ /dev/null
@@ -1,565 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.nereids.trees.plans.commands;
-
-import org.apache.doris.analysis.RedirectStatus;
-import org.apache.doris.catalog.DatabaseIf;
-import org.apache.doris.catalog.Env;
-import org.apache.doris.catalog.TableIf;
-import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.ErrorCode;
-import org.apache.doris.common.util.TimeUtils;
-import org.apache.doris.datasource.CatalogIf;
-import org.apache.doris.datasource.CatalogMgr;
-import org.apache.doris.datasource.lance.LanceExternalCatalog;
-import org.apache.doris.datasource.lance.LanceExternalDatabase;
-import org.apache.doris.datasource.lance.LanceExternalTable;
-import org.apache.doris.datasource.lance.job.LanceIndexFenceKey;
-import org.apache.doris.datasource.lance.job.LanceIndexJob;
-import org.apache.doris.datasource.lance.job.LanceIndexJobCompletionReason;
-import org.apache.doris.datasource.lance.job.LanceIndexJobManager;
-import org.apache.doris.datasource.lance.job.LanceIndexJobMutationState;
-import org.apache.doris.datasource.lance.job.LanceIndexJobMutationType;
-import org.apache.doris.datasource.lance.job.LanceIndexJobRefreshState;
-import org.apache.doris.datasource.lance.job.LanceIndexJobResult;
-import org.apache.doris.datasource.lance.job.LanceIndexJobResultCode;
-import org.apache.doris.datasource.lance.job.LanceIndexNameNormalizer;
-import org.apache.doris.datasource.lance.job.LanceIndexSchemaContract;
-import org.apache.doris.datasource.lance.job.LanceIndexTerminationProof;
-import org.apache.doris.mysql.privilege.AccessControllerManager;
-import org.apache.doris.mysql.privilege.PrivPredicate;
-import org.apache.doris.qe.ConnectContext;
-import org.apache.doris.qe.SessionVariable;
-import org.apache.doris.qe.ShowResultSet;
-
-import mockit.Expectations;
-import mockit.Mocked;
-import mockit.Verifications;
-import mockit.VerificationsInOrder;
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.util.Collections;
-import java.util.List;
-
-/**
- * Covers SHOW LANCE INDEX JOB jobId: the record is loaded first and authorized against its
- * persisted target before any field is returned; a missing job, a job whose target the
- * user cannot see (orphan or half-orphan without ADMIN, a resolution that fails outright,
- * a repointed or unresolvable dataset locator, or a resolvable target without table
- * SHOW), all share the same fixed ERR_LANCE_INDEX_JOB_NOT_FOUND response naming only the
- * job id. Also covers full detail-column rendering with and without result/dispatch/force
- * fields.
- */
-public class ShowLanceIndexJobCommandTest {
- @Mocked
- private Env env;
- @Mocked
- private ConnectContext connectContext;
- @Mocked
- private AccessControllerManager accessControllerManager;
- @Mocked
- private LanceIndexJobManager lanceIndexJobManager;
- @Mocked
- private CatalogMgr catalogMgr;
- @Mocked
- private SessionVariable sessionVariable;
- @Mocked
- private CatalogIf catalog;
- @Mocked
- private DatabaseIf database;
- @Mocked
- private TableIf table;
- @Mocked
- private LanceExternalCatalog lanceCatalog;
- @Mocked
- private LanceExternalDatabase lanceDatabase;
- @Mocked
- private LanceExternalTable lanceTable;
-
- private static LanceIndexJob newJob(long jobId) {
- LanceIndexJob job = new LanceIndexJob(jobId, "creator", 10L, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, "s3://bucket/dataset",
- "idx1", LanceIndexNameNormalizer.normalize("idx1"),
- LanceIndexJobMutationType.CREATE, true, false, "IVF_PQ", "v", null, 7L, null);
- job.setMutationState(LanceIndexJobMutationState.PENDING);
- job.setRefreshState(LanceIndexJobRefreshState.NOT_REQUIRED);
- job.setCreateTimeMs(1000L);
- job.setUpdateTimeMs(2000L);
- return job;
- }
-
- private void expectEnv(LanceIndexJob job) {
- new Expectations() {
- {
- Env.getCurrentEnv();
- minTimes = 0;
- result = env;
-
- env.getLanceIndexJobManager();
- minTimes = 0;
- result = lanceIndexJobManager;
-
- env.getCatalogMgr();
- minTimes = 0;
- result = catalogMgr;
-
- env.getAccessManager();
- minTimes = 0;
- result = accessControllerManager;
-
- lanceIndexJobManager.getJob(anyLong);
- minTimes = 0;
- result = job;
-
- ConnectContext.get();
- minTimes = 0;
- result = connectContext;
-
- connectContext.getSessionVariable();
- minTimes = 0;
- result = sessionVariable;
-
- sessionVariable.getTimeZone();
- minTimes = 0;
- result = "Asia/Shanghai";
- }
- };
- }
-
- private void expectResolvableCatalog(boolean tableVisible) {
- new Expectations() {
- {
- catalogMgr.getCatalog(10L);
- minTimes = 0;
- result = catalog;
-
- catalog.getName();
- minTimes = 0;
- result = "lance_ctl";
-
- catalog.getDbNullable("db1");
- minTimes = 0;
- result = database;
-
- database.getTableNullable("tbl1");
- minTimes = 0;
- result = table;
-
- accessControllerManager.checkTblPriv(connectContext, "lance_ctl", "db1", "tbl1",
- PrivPredicate.SHOW);
- minTimes = 0;
- result = tableVisible;
- }
- };
- }
-
- private static int colIndex(ShowResultSet resultSet, String name) {
- for (int i = 0; i < resultSet.getMetaData().getColumns().size(); i++) {
- if (resultSet.getMetaData().getColumns().get(i).getName().equals(name)) {
- return i;
- }
- }
- throw new IllegalStateException("column not found: " + name);
- }
-
- /**
- * Catalog id 10 resolves as a Lance catalog whose db1.tbl1 names resolve and whose
- * current dataset locator is the given value: "s3://bucket/dataset" is what
- * {@link #newJob} persists, anything else simulates a table repointed at a new
- * dataset, and null a locator resolution that fails outright.
- */
- private void expectLanceCatalog(String currentLocator, boolean tableVisible) {
- new Expectations() {
- {
- catalogMgr.getCatalog(10L);
- minTimes = 0;
- result = lanceCatalog;
-
- lanceCatalog.getName();
- minTimes = 0;
- result = "lance_ctl";
-
- lanceCatalog.getDbNullable("db1");
- minTimes = 0;
- result = lanceDatabase;
-
- lanceDatabase.getTableNullable("tbl1");
- minTimes = 0;
- result = lanceTable;
-
- lanceCatalog.resolveCurrentIndexJobLocator("db1", "tbl1");
- minTimes = 0;
- result = currentLocator;
-
- accessControllerManager.checkTblPriv(connectContext, "lance_ctl", "db1", "tbl1",
- PrivPredicate.SHOW);
- minTimes = 0;
- result = tableVisible;
- }
- };
- }
-
- @Test
- public void testMissingJobAndUnauthorizedShareSameResponse() {
- // Missing job: 5103 naming only the job id.
- expectEnv(null);
- AnalysisException missing = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_JOB_NOT_FOUND, missing.getMysqlErrorCode());
- Assertions.assertTrue(missing.getMessage().contains("Lance index job not found: 42"));
-
- // Resolvable target but no table SHOW privilege: byte-identical response.
- expectEnv(newJob(42L));
- expectResolvableCatalog(false);
- AnalysisException denied = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertEquals(missing.getMessage(), denied.getMessage());
- Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_JOB_NOT_FOUND, denied.getMysqlErrorCode());
- }
-
- @Test
- public void testOrphanRequiresAdmin() {
- expectEnv(newJob(42L));
- new Expectations() {
- {
- catalogMgr.getCatalog(10L);
- minTimes = 0;
- result = null;
-
- accessControllerManager.checkGlobalPriv(connectContext, PrivPredicate.ADMIN);
- minTimes = 0;
- result = false;
- }
- };
- AnalysisException e = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_JOB_NOT_FOUND, e.getMysqlErrorCode());
- }
-
- @Test
- public void testOrphanNonAdminMatchesMissingResponse() {
- expectEnv(null);
- AnalysisException missing = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertTrue(missing.getMessage().contains("Lance index job not found: 42"));
-
- // Catalog deleted: a non-ADMIN caller gets the byte-identical response as for a
- // missing job, so the persisted catalog id behind a job is never disclosed.
- expectEnv(newJob(42L));
- new Expectations() {
- {
- catalogMgr.getCatalog(10L);
- minTimes = 0;
- result = null;
-
- accessControllerManager.checkGlobalPriv(connectContext, PrivPredicate.ADMIN);
- minTimes = 0;
- result = false;
- }
- };
- AnalysisException orphan = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertEquals(missing.getMessage(), orphan.getMessage());
- Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_JOB_NOT_FOUND, orphan.getMysqlErrorCode());
- // The job record was loaded and the ADMIN check actually ran, so the orphan branch
- // (not the missing branch) produced the response above.
- new Verifications() {
- {
- accessControllerManager.checkGlobalPriv(connectContext, PrivPredicate.ADMIN);
- times = 1;
- }
- };
- }
-
- @Test
- public void testHalfOrphanRequiresAdmin() {
- // Catalog resolves but the persisted db no longer does: same ADMIN-only rule.
- expectEnv(newJob(42L));
- new Expectations() {
- {
- catalogMgr.getCatalog(10L);
- minTimes = 0;
- result = catalog;
-
- catalog.getDbNullable("db1");
- minTimes = 0;
- result = null;
-
- accessControllerManager.checkGlobalPriv(connectContext, PrivPredicate.ADMIN);
- minTimes = 0;
- result = false;
- }
- };
- AnalysisException e = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_JOB_NOT_FOUND, e.getMysqlErrorCode());
- }
-
- @Test
- public void testProviderFailureResolvingTargetMatchesMissingResponse() {
- expectEnv(null);
- AnalysisException missing = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertTrue(missing.getMessage().contains("Lance index job not found: 42"));
-
- // A valid job whose target resolution fails outright (provider error, expired
- // credentials) gets the same fixed response for a non-ADMIN caller; the provider
- // error never surfaces.
- expectEnv(newJob(42L));
- new Expectations() {
- {
- catalogMgr.getCatalog(10L);
- minTimes = 0;
- result = catalog;
-
- catalog.getDbNullable("db1");
- minTimes = 0;
- result = new RuntimeException("provider exploded");
-
- accessControllerManager.checkGlobalPriv(connectContext, PrivPredicate.ADMIN);
- minTimes = 0;
- result = false;
- }
- };
- AnalysisException failed = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertEquals(missing.getMessage(), failed.getMessage());
- Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_JOB_NOT_FOUND, failed.getMysqlErrorCode());
- }
-
- @Test
- public void testLocatorMatchAuthorizedThroughTableShow() throws Exception {
- // The Lance catalog still points db1.tbl1 at the dataset the job was admitted
- // against, so ordinary table-level SHOW is enough to read the detail row.
- expectEnv(newJob(42L));
- expectLanceCatalog("s3://bucket/dataset", true);
-
- ShowResultSet resultSet = new ShowLanceIndexJobCommand(42L).doRun(connectContext, null);
- Assertions.assertEquals(1, resultSet.getResultRows().size());
- Assertions.assertEquals("42", resultSet.getResultRows().get(0).get(colIndex(resultSet, "JobId")));
- }
-
- @Test
- public void testLocatorMismatchMatchesMissingResponse() {
- expectEnv(null);
- AnalysisException missing = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
-
- // db1.tbl1 was repointed at a different dataset after the job terminated: even a
- // caller who would hold SHOW on the new target gets the fixed not-found response.
- expectEnv(newJob(42L));
- expectLanceCatalog("s3://bucket/repointed", true);
- new Expectations() {
- {
- accessControllerManager.checkGlobalPriv(connectContext, PrivPredicate.ADMIN);
- minTimes = 0;
- result = false;
- }
- };
- AnalysisException denied = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertEquals(missing.getMessage(), denied.getMessage());
- Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_JOB_NOT_FOUND, denied.getMysqlErrorCode());
- }
-
- @Test
- public void testLocatorResolutionFailureIsOrphan() {
- // The current locator cannot be resolved right now (provider failure): the target
- // is treated as not resolved - orphan semantics - instead of granting SHOW
- // through the stale names.
- expectEnv(newJob(42L));
- expectLanceCatalog(null, true);
- new Expectations() {
- {
- accessControllerManager.checkGlobalPriv(connectContext, PrivPredicate.ADMIN);
- minTimes = 0;
- result = false;
- }
- };
- AnalysisException e = Assertions.assertThrows(AnalysisException.class,
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertEquals(ErrorCode.ERR_LANCE_INDEX_JOB_NOT_FOUND, e.getMysqlErrorCode());
- }
-
- @Test
- public void testOrphanVisibleToAdminWithEmptyCatalogName() throws Exception {
- LanceIndexJob job = newJob(42L);
- expectEnv(job);
- new Expectations() {
- {
- catalogMgr.getCatalog(10L);
- minTimes = 0;
- result = null;
-
- accessControllerManager.checkGlobalPriv(connectContext, PrivPredicate.ADMIN);
- minTimes = 0;
- result = true;
- }
- };
- ShowResultSet resultSet = new ShowLanceIndexJobCommand(42L).doRun(connectContext, null);
- Assertions.assertEquals(1, resultSet.getResultRows().size());
- List<String> row = resultSet.getResultRows().get(0);
- Assertions.assertEquals("42", row.get(colIndex(resultSet, "JobId")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "CatalogName")));
- Assertions.assertEquals("db1", row.get(colIndex(resultSet, "DbName")));
- }
-
- @Test
- public void testNullResultAndNullDispatchFieldsRenderEmpty() throws Exception {
- LanceIndexJob job = newJob(42L);
- Assertions.assertNull(job.getResult());
- expectEnv(job);
- expectResolvableCatalog(true);
-
- ShowResultSet resultSet = Assertions.assertDoesNotThrow(
- () -> new ShowLanceIndexJobCommand(42L).doRun(connectContext, null));
- Assertions.assertEquals(1, resultSet.getResultRows().size());
- List<String> row = resultSet.getResultRows().get(0);
- Assertions.assertEquals("creator", row.get(colIndex(resultSet, "Creator")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "Message")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "ResultCode")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "CompletionReason")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "ExternalMetadataAdvanced")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "BackendId")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "BeProcessEpoch")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "InvocationId")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "DeadlineMs")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "ForceNote")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "ForceWarning")));
- Assertions.assertEquals("7", row.get(colIndex(resultSet, "AdmittedDatasetVersion")));
- Assertions.assertEquals("", row.get(colIndex(resultSet, "SchemaContractVersion")));
- Assertions.assertEquals("0", row.get(colIndex(resultSet, "Revision")));
- Assertions.assertEquals("YES", row.get(colIndex(resultSet, "IfNotExists")));
- Assertions.assertEquals("NO", row.get(colIndex(resultSet, "IfExists")));
- Assertions.assertEquals("IVF_PQ", row.get(colIndex(resultSet, "IndexType")));
- Assertions.assertEquals("v", row.get(colIndex(resultSet, "ColumnName")));
- Assertions.assertEquals("NONE", row.get(colIndex(resultSet, "TerminationProof")));
- Assertions.assertEquals(TimeUtils.longToTimeString(1000L), row.get(colIndex(resultSet, "CreateTime")));
- }
-
- @Test
- public void testResultDispatchAndForceFieldsRender() throws Exception {
- LanceIndexSchemaContract contract = new LanceIndexSchemaContract(Collections.singletonList(
- new LanceIndexSchemaContract.IndexedField(1, "v", "fixed_size_list", false, 128,
- "float32", false)));
- LanceIndexJob job = new LanceIndexJob(42L, "creator", 10L, "db1", "tbl1",
- LanceIndexFenceKey.PROVIDER_DIRECTORY, "s3://bucket/dataset",
- "idx1", LanceIndexNameNormalizer.normalize("idx1"),
- LanceIndexJobMutationType.CREATE, true, false, "IVF_PQ", "v", null, 7L, contract);
- job.setMutationState(LanceIndexJobMutationState.PENDING);
- job.setRefreshState(LanceIndexJobRefreshState.NOT_REQUIRED);
- job.setCreateTimeMs(1000L);
- job.setUpdateTimeMs(2000L);
- job.setResult(new LanceIndexJobResult(LanceIndexJobResultCode.NATIVE_OK,
- LanceIndexJobCompletionReason.NONE, "ok", true));
- job.setBackendId(1001L);
- job.setBeProcessEpoch(55L);
- job.setInvocationId("inv-1");
- job.setDeadlineMs(9999L);
- job.setTerminationProof(LanceIndexTerminationProof.CHILD_REAPED);
- job.setRevision(3L);
- job.setForceReleased(true);
- job.setForceActor("admin");
- job.setForceTimeMs(3000L);
- job.setForceNote("note");
- job.setForceWarning("warn");
- expectEnv(job);
- expectResolvableCatalog(true);
-
- ShowResultSet resultSet = new ShowLanceIndexJobCommand(42L).doRun(connectContext, null);
- List<String> row = resultSet.getResultRows().get(0);
- Assertions.assertEquals("NATIVE_OK", row.get(colIndex(resultSet, "ResultCode")));
- Assertions.assertEquals("NONE", row.get(colIndex(resultSet, "CompletionReason")));
- Assertions.assertEquals("YES", row.get(colIndex(resultSet, "ExternalMetadataAdvanced")));
- Assertions.assertEquals("ok", row.get(colIndex(resultSet, "Message")));
- Assertions.assertEquals("1001", row.get(colIndex(resultSet, "BackendId")));
- Assertions.assertEquals("55", row.get(colIndex(resultSet, "BeProcessEpoch")));
- Assertions.assertEquals("inv-1", row.get(colIndex(resultSet, "InvocationId")));
- Assertions.assertEquals("9999", row.get(colIndex(resultSet, "DeadlineMs")));
- Assertions.assertEquals("CHILD_REAPED", row.get(colIndex(resultSet, "TerminationProof")));
- Assertions.assertEquals("3", row.get(colIndex(resultSet, "Revision")));
- Assertions.assertEquals("YES", row.get(colIndex(resultSet, "ForceReleased")));
- Assertions.assertEquals("admin", row.get(colIndex(resultSet, "ForceActor")));
- Assertions.assertEquals(TimeUtils.longToTimeString(3000L), row.get(colIndex(resultSet, "ForceTime")));
- Assertions.assertEquals("note", row.get(colIndex(resultSet, "ForceNote")));
- Assertions.assertEquals("warn", row.get(colIndex(resultSet, "ForceWarning")));
- Assertions.assertEquals("1", row.get(colIndex(resultSet, "SchemaContractVersion")));
- }
-
- @Test
- public void testRedirectStatus() {
- Assertions.assertEquals(RedirectStatus.FORWARD_NO_SYNC,
- new ShowLanceIndexJobCommand(1L).toRedirectStatus());
- }
-
- @Test
- public void testProxyContextCarriesIdentity() throws Exception {
- // V3-F16.8: the privilege check runs against the context handed to doRun.
- ConnectContext proxyCtx = new ConnectContext();
- expectEnv(newJob(42L));
- new Expectations() {
- {
- catalogMgr.getCatalog(10L);
- minTimes = 0;
- result = catalog;
-
- catalog.getName();
- minTimes = 0;
- result = "lance_ctl";
-
- catalog.getDbNullable("db1");
- minTimes = 0;
- result = database;
-
- database.getTableNullable("tbl1");
- minTimes = 0;
- result = table;
-
- accessControllerManager.checkTblPriv((ConnectContext) any, "lance_ctl", "db1", "tbl1",
- PrivPredicate.SHOW);
- minTimes = 0;
- result = true;
- }
- };
- new ShowLanceIndexJobCommand(42L).doRun(proxyCtx, null);
- new Verifications() {
- {
- accessControllerManager.checkTblPriv(proxyCtx, "lance_ctl", "db1", "tbl1", PrivPredicate.SHOW);
- times = 1;
- }
- };
- }
-
- @Test
- public void testJobLoadsBeforeTargetAuthorization() throws Exception {
- expectEnv(newJob(42L));
- expectResolvableCatalog(true);
-
- new ShowLanceIndexJobCommand(42L).doRun(connectContext, null);
-
- // The record is loaded first; authorization then runs against its persisted target.
- new VerificationsInOrder() {
- {
- lanceIndexJobManager.getJob(42L);
- accessControllerManager.checkTblPriv(connectContext, "lance_ctl", "db1", "tbl1",
- PrivPredicate.SHOW);
- }
- };
- }
-}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobsCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobsCommandTest.java
deleted file mode 100644
index fb68c702893..00000000000
--- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/ShowLanceIndexJobsCommandTest.java
+++ /dev/null
@@ -1,821 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-package org.apache.doris.nereids.trees.plans.commands;
-
-import org.apache.doris.analysis.RedirectStatus;
-import org.apache.doris.catalog.DatabaseIf;
-import org.apache.doris.catalog.Env;
-import org.apache.doris.catalog.TableIf;
-import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.util.TimeUtils;
-import org.apache.doris.datasource.CatalogIf;
-import org.apache.doris.datasource.CatalogMgr;
-import org.apache.doris.datasource.lance.LanceExternalCatalog;
-import org.apache.doris.datasource.lance.LanceExternalDatabase;
-import org.apache.doris.datasource.lance.LanceExternalTable;
-import org.apache.doris.datasource.lance.job.LanceIndexFenceKey;
-import org.apache.doris.datasource.lance.job.LanceIndexJob;
-import org.apache.doris.datasource.lance.job.LanceIndexJobManager;
-import org.apache.doris.datasource.lance.job.LanceIndexJobMutationState;
-import org.apache.doris.datasource.lance.job.LanceIndexJobMutationType;
-import org.apache.doris.datasource.lance.job.LanceIndexJobRefreshState;
-import org.apache.doris.datasource.lance.job.LanceIndexNameNormalizer;
-import org.apache.doris.mysql.privilege.AccessControllerManager;
-import org.apache.doris.mysql.privilege.PrivPredicate;
-import org.apache.doris.nereids.analyzer.UnboundSlot;
-import org.apache.doris.nereids.trees.expressions.And;
-import org.apache.doris.nereids.trees.expressions.EqualTo;
-import org.apache.doris.nereids.trees.expressions.Expression;
-import org.apache.doris.nereids.trees.expressions.GreaterThan;
-import org.apache.doris.nereids.trees.expressions.Like;
-import org.apache.doris.nereids.trees.expressions.Or;
-import org.apache.doris.nereids.trees.expressions.literal.IntegerLiteral;
-import org.apache.doris.nereids.trees.expressions.literal.StringLiteral;
-import org.apache.doris.nereids.trees.expressions.literal.VarcharLiteral;
-import org.apache.doris.qe.ConnectContext;
-import org.apache.doris.qe.SessionVariable;
-import org.apache.doris.qe.ShowResultSet;
-
-import com.google.common.collect.Lists;
-import mockit.Expectations;
-import mockit.Mocked;
-import mockit.Verifications;
-import org.junit.jupiter.api.Assertions;
-import org.junit.jupiter.api.Test;
-
-import java.util.Arrays;
-import java.util.Collections;
-import java.util.List;
-
-/**
- * Covers SHOW LANCE INDEX JOBS: row-level SHOW privilege filtering, orphan and
- * half-orphan rows restricted to global ADMIN with no existence or count leakage,
- * provider failures during target resolution treated as orphans, dataset-locator
- * revalidation (matching, repointed and unresolvable locators, resolved once per
- * target within one listing), FROM db-name normalization through the target catalog's
- * own naming rules, FROM/WHERE in-memory filtering, null-result rendering, FORCE audit
- * columns, the deliberately narrowed WHERE grammar (EqualTo + AND only),
- * FORWARD_NO_SYNC redirect, and that authorization runs against the connection context
- * handed to the command (the forwarded original user identity in proxy mode).
- */
-public class ShowLanceIndexJobsCommandTest {
- @Mocked
- private Env env;
- @Mocked
- private ConnectContext connectContext;
- @Mocked
- private AccessControllerManager accessControllerManager;
- @Mocked
- private LanceIndexJobManager lanceIndexJobManager;
- @Mocked
- private CatalogMgr catalogMgr;
- @Mocked
- private SessionVariable sessionVariable;
- @Mocked
- private CatalogIf catalog;
- @Mocked
- private DatabaseIf database;
- @Mocked
- private TableIf table;
- @Mocked
- private LanceExternalCatalog lanceCatalog;
- @Mocked
- private LanceExternalDatabase lanceDatabase;
- @Mocked
- private LanceExternalTable lanceTable;
-
- private static LanceIndexJob newJob(long jobId, long catalogId, String dbName, String tableName,
- String indexName) {
- LanceIndexJob job = new LanceIndexJob(jobId, "creator", catalogId, dbName, tableName,
- LanceIndexFenceKey.PROVIDER_DIRECTORY, "s3://bucket/dataset",
- indexName, LanceIndexNameNormalizer.normalize(indexName),
- LanceIndexJobMutationType.CREATE, false, false, "IVF_PQ", "v", null, 7L, null);
- job.setMutationState(LanceIndexJobMutationState.PENDING);
- job.setRefreshState(LanceIndexJobRefreshState.NOT_REQUIRED);
- job.setCreateTimeMs(1000L);
- job.setUpdateTimeMs(2000L);
- return job;
- }
-
- private void expectEnv(List<LanceIndexJob> jobs) {
- new Expectations() {
- {
- Env.getCurrentEnv();
- minTimes = 0;
- result = env;
-
- env.getLanceIndexJobManager();
- minTimes = 0;
- result = lanceIndexJobManager;
-
- env.getCatalogMgr();
- minTimes = 0;
- result = catalogMgr;
-
- env.getAccessManager();
- minTimes = 0;
- result = accessControllerManager;
-
- lanceIndexJobManager.getAllJobsSnapshot();
- minTimes = 0;
- result = jobs;
-
- ConnectContext.get();
- minTimes = 0;
- result = connectContext;
-
- connectContext.getSessionVariable();
- minTimes = 0;
- result = sessionVariable;
-
- sessionVariable.getTimeZone();
- minTimes = 0;
- result = "Asia/Shanghai";
- }
- };
- }
-
- private void expectResolvableCatalog(boolean tableVisible) {
- new Expectations() {
- {
- catalogMgr.getCatalog(10L);
- minTimes = 0;
- result = catalog;
-
- catalog.getName();
- minTimes = 0;
- result = "lance_ctl";
-
- catalog.getDbNullable("db1");
- minTimes = 0;
- result = database;
-
- database.getFullName();
- minTimes = 0;
- result = "db1";
-
- database.getTableNullable("tbl1");
- minTimes = 0;
- result = table;
-
- accessControllerManager.checkTblPriv(connectContext, "lance_ctl", "db1", "tbl1",
- PrivPredicate.SHOW);
- minTimes = 0;
- result = tableVisible;
- }
- };
- }
-
- /**
- * Catalog id 10 resolves as a Lance catalog whose db1.tbl1 names resolve and whose
- * current dataset locator is the given value: "s3://bucket/dataset" is what
- * {@link #newJob} persists, anything else simulates a table repointed at a new
- * dataset, and null a locator resolution that fails outright.
- */
- private void expectLanceCatalog(String currentLocator, boolean tableVisible) {
- new Expectations() {
- {
- catalogMgr.getCatalog(10L);
- minTimes = 0;
- result = lanceCatalog;
-
- lanceCatalog.getName();
- minTimes = 0;
- result = "lance_ctl";
-
- lanceCatalog.getDbNullable("db1");
- minTimes = 0;
- result = lanceDatabase;
-
- lanceDatabase.getTableNullable("tbl1");
- minTimes = 0;
- result = lanceTable;
-
- lanceCatalog.resolveCurrentIndexJobLocator("db1", "tbl1");
- minTimes = 0;
- result = currentLocator;
-
- accessControllerManager.checkTblPriv(connectContext, "lance_ctl", "db1", "tbl1",
- PrivPredicate.SHOW);
- minTimes = 0;
- result = tableVisible;
- }
- };
- }
-
- private static int colIndex(ShowResultSet resultSet, String name) {
- for (int i = 0; i < resultSet.getMetaData().getColumns().size(); i++) {
- if (resultSet.getMetaData().getColumns().get(i).getName().equals(name)) {
- return i;
- }
- }
- throw new IllegalStateException("column not found: " + name);
- }
-
- @Test
- public void testMetadataColumns() throws Exception {
- expectEnv(Collections.emptyList());
- ShowLanceIndexJobsCommand command = new ShowLanceIndexJobsCommand(null, null);
- ShowResultSet resultSet = command.doRun(connectContext, null);
- List<String> expected = Arrays.asList("JobId", "CatalogName", "DbName", "TableName", "IndexName",
- "Operation", "State", "RefreshState", "PossibleLive", "CreateTime", "UpdateTime", "Message",
- "ForceReleased", "ForceActor", "ForceTime");
- Assertions.assertEquals(expected.size(), resultSet.getMetaData().getColumns().size());
- for (int i = 0; i < expected.size(); i++) {
- Assertions.assertEquals(expected.get(i), resultSet.getMetaData().getColumns().get(i).getName());
- }
- Assertions.assertTrue(resultSet.getResultRows().isEmpty());
- }
-
- @Test
- public void testRowLevelAuthFiltering() throws Exception {
- LanceIndexJob visible = newJob(1L, 10L, "db1", "tbl1", "idx1");
- LanceIndexJob hidden = newJob(2L, 10L, "db1", "tbl2", "idx2");
- expectEnv(Arrays.asList(visible, hidden));
- expectResolvableCatalog(true);
- new Expectations() {
- {
- database.getTableNullable("tbl2");
- minTimes = 0;
- result = table;
-
- accessControllerManager.checkTblPriv(connectContext, "lance_ctl", "db1", "tbl2",
- PrivPredicate.SHOW);
- minTimes = 0;
- result = false;
- }
- };
-
- ShowLanceIndexJobsCommand command = new ShowLanceIndexJobsCommand(null, null);
- ShowResultSet resultSet = command.doRun(connectContext, null);
- Assertions.assertEquals(1, resultSet.getResultRows().size());
- List<String> row = resultSet.getResultRows().get(0);
- Assertions.assertEquals("1", row.get(colIndex(resultSet, "JobId")));
- Assertions.assertEquals("lance_ctl", row.get(colIndex(resultSet, "CatalogName")));
- Assertions.assertEquals("db1", row.get(colIndex(resultSet, "DbName")));
- Assertions.assertEquals("tbl1", row.get(colIndex(resultSet, "TableName")));
- Assertions.assertEquals("idx1", row.get(colIndex(resultSet, "IndexName")));
- Assertions.assertEquals("CREATE", row.get(colIndex(resultSet, "Operation")));
- Assertions.assertEquals("PENDING", row.get(colIndex(resultSet, "State")));
- Assertions.assertEquals("NOT_REQUIRED", row.get(colIndex(resultSet, "RefreshState")));
- Assertions.assertEquals("NO", row.get(colIndex(resultSet, "PossibleLive")));
- Assertions.assertEquals(TimeUtils.longToTimeString(1000L), row.get(colIndex(resultSet, "CreateTime")));
- Assertions.assertEquals(TimeUtils.longToTimeString(2000L), row.get(colIndex(resultSet, "UpdateTime")));
- }
-
- @Test
... 902 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org