You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2022/05/06 09:20:39 UTC
[incubator-doris] branch dev-1.0.1 updated (19924dfc80 -> 973153fe9b)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
from 19924dfc80 [hotfix] track all memory in VOlapScanNode and fix track leaf from bitmap and hll (#9348)
new 706e6cda12 [fix](backup) Remove colocate_with property when backing up a table (#9142)
new 80c74b029d [improvement](hive) Support hive with HA HDFS. Pass ha configuration through hive create table properties. (#9151)
new 05cf433aa6 [fix](catalog) fix bug that replica missing version cause query -214 error (#9266)
new a927681e3f [fix](profile) Short-circuit and del predicate filter rows are not counted on vectorized exec (#9268)
new ba0b0634dd [fix](vectorized) Fix bug of outer join with delete column (#9315)
new f4bdeeebac [Bug] [Build] fix clang build fail (#9323)
new 6c3bb756d5 [fix](alter-job) Missing alter job when doing checkpoint image (#9329)
new b925413d9c [improvement] set name for scanner threads and fix compile error in clang (#9336)
new 20c3744d68 [fix](function) handle merge in window_funnel_init and add test (#9338)
new 048b3406cd [fix](truncate) fix bug that truncate partition throw NPE (#9339)
new 973153fe9b [fixbug][metadata] catalog could not load from image (#9364)
The 11 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/exec/base_scanner.h | 2 +-
be/src/exec/broker_scan_node.cpp | 2 +
be/src/exec/broker_scan_node.h | 20 ++++---
be/src/exec/broker_scanner.h | 5 +-
be/src/exprs/aggregate_functions.cpp | 7 +--
be/src/olap/olap_common.h | 2 +-
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 4 ++
be/src/util/thread.cpp | 4 ++
be/src/util/thread.h | 2 +
.../commit-format-specification.md | 4 +-
.../commit-format-specification.md | 1 +
.../java/org/apache/doris/alter/AlterJobV2.java | 2 +
.../apache/doris/alter/SchemaChangeHandler.java | 11 ++--
.../java/org/apache/doris/analysis/Analyzer.java | 4 ++
.../java/org/apache/doris/backup/BackupJob.java | 7 +++
.../java/org/apache/doris/catalog/Catalog.java | 45 ++++++++--------
.../doris/catalog/HiveMetaStoreClientHelper.java | 19 +++++--
.../java/org/apache/doris/catalog/HiveTable.java | 13 +++++
.../java/org/apache/doris/catalog/OlapTable.java | 9 ++--
.../java/org/apache/doris/catalog/Replica.java | 2 +-
.../org/apache/doris/catalog/TableProperty.java | 1 +
.../org/apache/doris/common/CaseSensibility.java | 2 +-
.../java/org/apache/doris/common/FeConstants.java | 6 +++
.../org/apache/doris/common/proc/JobsProcDir.java | 4 +-
.../java/org/apache/doris/master/MasterImpl.java | 6 +--
.../org/apache/doris/master/ReportHandler.java | 18 ++++---
.../apache/doris/persist/BackendReplicasInfo.java | 18 +++++--
.../java/org/apache/doris/persist/EditLog.java | 10 ++--
.../org/apache/doris/planner/BrokerScanNode.java | 18 +++++--
.../org/apache/doris/planner/OlapScanNode.java | 1 +
.../doris/transaction/DatabaseTransactionMgr.java | 60 ++++++++-------------
.../doris/transaction/GlobalTransactionMgr.java | 4 +-
.../doris/transaction/PublishVersionDaemon.java | 6 +--
.../org/apache/doris/alter/AlterJobV2Test.java | 10 ++--
.../java/org/apache/doris/alter/AlterTest.java | 2 +-
.../org/apache/doris/alter/BatchRollupJobTest.java | 6 +--
.../org/apache/doris/alter/RollupJobV2Test.java | 11 ++--
.../apache/doris/catalog/CatalogOperationTest.java | 2 +-
.../org/apache/doris/catalog/OlapTableTest.java | 11 ++--
.../apache/doris/catalog/TempPartitionTest.java | 6 +--
.../apache/doris/catalog/TruncateTableTest.java | 41 ++++++++++++++
.../org/apache/doris/http/DorisHttpTestCase.java | 2 +-
.../doris/persist/BackendReplicaInfosTest.java | 3 +-
.../transaction/GlobalTransactionMgrTest.java | 9 ++--
.../java/org/apache/doris/utframe/DorisAssert.java | 2 +-
fe/fe-core/src/test/resources/help-resource.zip | Bin 0 -> 1056 bytes
46 files changed, 274 insertions(+), 150 deletions(-)
create mode 100644 fe/fe-core/src/test/resources/help-resource.zip
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 03/11: [fix](catalog) fix bug that replica missing version cause query -214 error (#9266)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 05cf433aa6e528206ae68112ca7c9ccd5c575788
Author: Mingyu Chen <mo...@gmail.com>
AuthorDate: Tue May 3 17:54:19 2022 +0800
[fix](catalog) fix bug that replica missing version cause query -214 error (#9266)
1. Fix bug described in #9267
When report missing version replica, set last failed version to (replica version + 1)
2. Skip non-exist partition when handling transactions.
---
.../java/org/apache/doris/catalog/Catalog.java | 5 +-
.../java/org/apache/doris/catalog/Replica.java | 2 +-
.../org/apache/doris/master/ReportHandler.java | 18 ++++---
.../apache/doris/persist/BackendReplicasInfo.java | 18 +++++--
.../doris/transaction/DatabaseTransactionMgr.java | 60 +++++++++-------------
.../doris/transaction/GlobalTransactionMgr.java | 4 +-
.../doris/transaction/PublishVersionDaemon.java | 6 +--
.../doris/persist/BackendReplicaInfosTest.java | 3 +-
.../transaction/GlobalTransactionMgrTest.java | 9 ++--
9 files changed, 60 insertions(+), 65 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
index f3e187065a..ccae1c3b5f 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
@@ -6965,10 +6965,7 @@ public class Catalog {
replica.setBad(true);
break;
case MISSING_VERSION:
- // The absolute value is meaningless, as long as it is greater than 0.
- // This way, in other checking logic, if lastFailedVersion is found to be greater than 0,
- // it will be considered a version missing replica and will be handled accordingly.
- replica.setLastFailedVersion(1L);
+ replica.updateLastFailedVersion(info.lastFailedVersion);
break;
default:
break;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Replica.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Replica.java
index a42cebdc8a..930305db98 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Replica.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Replica.java
@@ -309,7 +309,7 @@ public class Replica implements Writable {
if (newVersion < this.version) {
// This case means that replica meta version has been updated by ReportHandler before
- // For example, the publish version daemon has already sent some publish verison tasks to one be to publish version 2, 3, 4, 5, 6,
+ // For example, the publish version daemon has already sent some publish version tasks to one be to publish version 2, 3, 4, 5, 6,
// and the be finish all publish version tasks, the be's replica version is 6 now, but publish version daemon need to wait
// for other be to finish most of publish version tasks to update replica version in fe.
// At the moment, the replica version in fe is 4, when ReportHandler sync tablet, it find reported replica version in be is 6 and then
diff --git a/fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java b/fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java
index d0e7b9ae5b..05643aa7e2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/master/ReportHandler.java
@@ -729,8 +729,8 @@ public class ReportHandler extends Daemon {
AgentTaskExecutor.submit(batchTask);
}
- LOG.info("delete {} tablet(s) from backend[{}]", deleteFromBackendCounter, backendId);
- LOG.info("add {} replica(s) to meta. backend[{}]", addToMetaCounter, backendId);
+ LOG.info("delete {} tablet(s) and add {} replica(s) to meta from backend[{}]",
+ deleteFromBackendCounter, addToMetaCounter, backendId);
}
// replica is used and no version missing
@@ -854,11 +854,15 @@ public class ReportHandler extends Daemon {
}
if (tTabletInfo.isSetVersionMiss() && tTabletInfo.isVersionMiss()) {
- // The absolute value is meaningless, as long as it is greater than 0.
- // This way, in other checking logic, if lastFailedVersion is found to be greater than 0,
- // it will be considered a version missing replica and will be handled accordingly.
- replica.setLastFailedVersion(1L);
- backendReplicasInfo.addMissingVersionReplica(tabletId);
+ // If the origin last failed version is larger than 0, not change it.
+ // Otherwise, we set last failed version to replica'version + 1.
+ // Because last failed version should always larger than replica's version.
+ long newLastFailedVersion = replica.getLastFailedVersion();
+ if (newLastFailedVersion < 0) {
+ newLastFailedVersion = replica.getVersion() + 1;
+ }
+ replica.updateLastFailedVersion(newLastFailedVersion);
+ backendReplicasInfo.addMissingVersionReplica(tabletId, newLastFailedVersion);
break;
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/BackendReplicasInfo.java b/fe/fe-core/src/main/java/org/apache/doris/persist/BackendReplicasInfo.java
index c382e1fdb5..80222f4618 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/BackendReplicasInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/BackendReplicasInfo.java
@@ -44,11 +44,11 @@ public class BackendReplicasInfo implements Writable {
}
public void addBadReplica(long tabletId) {
- replicaReportInfos.add(new ReplicaReportInfo(tabletId, ReportInfoType.BAD));
+ replicaReportInfos.add(new ReplicaReportInfo(tabletId, -1, ReportInfoType.BAD));
}
- public void addMissingVersionReplica(long tabletId) {
- replicaReportInfos.add(new ReplicaReportInfo(tabletId, ReportInfoType.MISSING_VERSION));
+ public void addMissingVersionReplica(long tabletId, long lastFailedVersion) {
+ replicaReportInfos.add(new ReplicaReportInfo(tabletId, lastFailedVersion, ReportInfoType.MISSING_VERSION));
}
public long getBackendId() {
@@ -84,9 +84,12 @@ public class BackendReplicasInfo implements Writable {
public long tabletId;
@SerializedName(value = "type")
public ReportInfoType type;
+ @SerializedName(value = "lastFailedVersion")
+ public long lastFailedVersion;
- public ReplicaReportInfo(long tabletId, ReportInfoType type) {
+ public ReplicaReportInfo(long tabletId, long lastFailedVersion, ReportInfoType type) {
this.tabletId = tabletId;
+ this.lastFailedVersion = lastFailedVersion;
this.type = type;
}
@@ -98,7 +101,12 @@ public class BackendReplicasInfo implements Writable {
public static ReplicaReportInfo read(DataInput in) throws IOException {
String json = Text.readString(in);
- return GsonUtils.GSON.fromJson(json, ReplicaReportInfo.class);
+ ReplicaReportInfo info = GsonUtils.GSON.fromJson(json, ReplicaReportInfo.class);
+ if (info.type == ReportInfoType.MISSING_VERSION && info.lastFailedVersion <= 0) {
+ // FIXME(cmy): Just for compatibility, should be remove in v1.2
+ info.lastFailedVersion = 1;
+ }
+ return info;
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
index 83585ffe32..9ce906fb79 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/transaction/DatabaseTransactionMgr.java
@@ -569,7 +569,7 @@ public class DatabaseTransactionMgr {
TxnCommitAttachment txnCommitAttachment, Boolean is2PC)
throws UserException {
// check status
- // the caller method already own db lock, we do not obtain db lock here
+ // the caller method already own tables' write lock
Database db = catalog.getDbOrMetaException(dbId);
TransactionState transactionState;
readLock();
@@ -644,7 +644,7 @@ public class DatabaseTransactionMgr {
LOG.info("transaction:[{}] successfully committed", transactionState);
}
- public boolean publishTransaction(Database db, long transactionId, long timeoutMillis) throws TransactionCommitFailedException {
+ public boolean waitForTransactionFinished(Database db, long transactionId, long timeoutMillis) throws TransactionCommitFailedException {
TransactionState transactionState = null;
readLock();
try {
@@ -857,48 +857,24 @@ public class DatabaseTransactionMgr {
}
}
+ // check success replica number for each tablet.
+ // a success replica means:
+ // 1. Not in errorReplicaIds: succeed in both commit and publish phase
+ // 2. last failed version < 0: is a health replica before
+ // 3. version catch up: not with a stale version
+ // Here we only check number, the replica version will be updated in updateCatalogAfterVisible()
for (MaterializedIndex index : allIndices) {
for (Tablet tablet : index.getTablets()) {
int healthReplicaNum = 0;
for (Replica replica : tablet.getReplicas()) {
- if (!errorReplicaIds.contains(replica.getId())
- && replica.getLastFailedVersion() < 0) {
- // this means the replica is a healthy replica,
- // it is healthy in the past and does not have error in current load
+ if (!errorReplicaIds.contains(replica.getId()) && replica.getLastFailedVersion() < 0) {
if (replica.checkVersionCatchUp(partition.getVisibleVersion(), true)) {
- // during rollup, the rollup replica's last failed version < 0,
- // it may be treated as a normal replica.
- // the replica is not failed during commit or publish
- // during upgrade, one replica's last version maybe invalid,
- // has to compare version hash.
-
- // Here we still update the replica's info even if we failed to publish
- // this txn, for the following case:
- // replica A,B,C is successfully committed, but only A is successfully
- // published,
- // B and C is crashed, now we need a Clone task to repair this tablet.
- // So, here we update A's version info, so that clone task will clone
- // the latest version of data.
-
- replica.updateVersionInfo(partitionCommitInfo.getVersion(),
- replica.getDataSize(), replica.getRowCount());
++healthReplicaNum;
- } else {
- // this means the replica has error in the past, but we did not observe it
- // during upgrade, one job maybe in quorum finished state, for example, A,B,C 3 replica
- // A,B 's version is 10, C's version is 10 but C' 10 is abnormal should be rollback
- // then we will detect this and set C's last failed version to 10 and last success version to 11
- // this logic has to be replayed in checkpoint thread
- replica.updateVersionInfo(replica.getVersion(),
- partition.getVisibleVersion(),
- partitionCommitInfo.getVersion());
- LOG.warn("transaction state {} has error, the replica [{}] not appeared in error replica list "
- + " and its version not equal to partition commit version or commit version - 1"
- + " if its not a upgrade stage, its a fatal error. ", transactionState, replica);
}
} else if (replica.getVersion() >= partitionCommitInfo.getVersion()) {
// the replica's version is larger than or equal to current transaction partition's version
// the replica is normal, then remove it from error replica ids
+ // TODO(cmy): actually I have no idea why we need this check
errorReplicaIds.remove(replica.getId());
++healthReplicaNum;
}
@@ -1490,14 +1466,19 @@ public class DatabaseTransactionMgr {
for (PartitionCommitInfo partitionCommitInfo : tableCommitInfo.getIdToPartitionCommitInfo().values()) {
long partitionId = partitionCommitInfo.getPartitionId();
Partition partition = table.getPartition(partitionId);
+ if (partition == null) {
+ LOG.warn("partition {} of table {} does not exist when update catalog after committed. transaction: {}, db: {}",
+ partitionId, tableId, transactionState.getTransactionId(), db.getId());
+ continue;
+ }
List<MaterializedIndex> allIndices = partition.getMaterializedIndices(MaterializedIndex.IndexExtState.ALL);
for (MaterializedIndex index : allIndices) {
List<Tablet> tablets = index.getTablets();
for (Tablet tablet : tablets) {
for (Replica replica : tablet.getReplicas()) {
if (errorReplicaIds.contains(replica.getId())) {
- // should not use partition.getNextVersion and partition.getNextVersionHash because partition's next version hash is generated locally
- // should get from transaction state
+ // TODO(cmy): do we need to update last failed version here?
+ // because in updateCatalogAfterVisible, it will be updated again.
replica.updateLastFailedVersion(partitionCommitInfo.getVersion());
}
}
@@ -1522,6 +1503,11 @@ public class DatabaseTransactionMgr {
long partitionId = partitionCommitInfo.getPartitionId();
long newCommitVersion = partitionCommitInfo.getVersion();
Partition partition = table.getPartition(partitionId);
+ if (partition == null) {
+ LOG.warn("partition {} in table {} does not exist when update catalog after visible. transaction: {}, db: {}",
+ partitionId, tableId, transactionState.getTransactionId(), db.getId());
+ continue;
+ }
List<MaterializedIndex> allIndices = partition.getMaterializedIndices(MaterializedIndex.IndexExtState.ALL);
for (MaterializedIndex index : allIndices) {
for (Tablet tablet : index.getTablets()) {
@@ -1531,7 +1517,7 @@ public class DatabaseTransactionMgr {
long lastSuccessVersion = replica.getLastSuccessVersion();
if (!errorReplicaIds.contains(replica.getId())) {
if (replica.getLastFailedVersion() > 0) {
- // if the replica is a failed replica, then not changing version and version hash
+ // if the replica is a failed replica, then not changing version
newVersion = replica.getVersion();
} else if (!replica.checkVersionCatchUp(partition.getVisibleVersion(), true)) {
// this means the replica has error in the past, but we did not observe it
diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
index b51cc9c0d9..9161eb525e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/transaction/GlobalTransactionMgr.java
@@ -211,7 +211,7 @@ public class GlobalTransactionMgr implements Writable {
* @throws UserException
* @throws TransactionCommitFailedException
* @note it is necessary to optimize the `lock` mechanism and `lock` scope resulting from wait lock long time
- * @note callers should get db.write lock before call this api
+ * @note callers should get all tables' write locks before call this api
*/
public void commitTransaction(long dbId, List<Table> tableList, long transactionId, List<TabletCommitInfo> tabletCommitInfos,
TxnCommitAttachment txnCommitAttachment)
@@ -263,7 +263,7 @@ public class GlobalTransactionMgr implements Writable {
// so we just return false to indicate publish timeout
return false;
}
- return dbTransactionMgr.publishTransaction(db, transactionId, publishTimeoutMillis);
+ return dbTransactionMgr.waitForTransactionFinished(db, transactionId, publishTimeoutMillis);
}
public void commitTransaction2PC(Database db, List<Table> tableList, long transactionId, long timeoutMillis)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java b/fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java
index 99da777962..7b592742f1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/transaction/PublishVersionDaemon.java
@@ -79,10 +79,7 @@ public class PublishVersionDaemon extends MasterDaemon {
return;
}
- // TODO yiguolei: could publish transaction state according to multi-tenant cluster info
- // but should do more work. for example, if a table is migrate from one cluster to another cluster
- // should publish to two clusters.
- // attention here, we publish transaction state to all backends including dead backend, if not publish to dead backend
+ // ATTN, we publish transaction state to all backends including dead backend, if not publish to dead backend
// then transaction manager will treat it as success
List<Long> allBackends = Catalog.getCurrentSystemInfo().getBackendIds(false);
if (allBackends.isEmpty()) {
@@ -198,7 +195,6 @@ public class PublishVersionDaemon extends MasterDaemon {
continue;
}
-
for (long tableId : transactionState.getTableIdList()) {
Table table = db.getTableNullable(tableId);
if (table == null || table.getType() != Table.TableType.OLAP) {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/persist/BackendReplicaInfosTest.java b/fe/fe-core/src/test/java/org/apache/doris/persist/BackendReplicaInfosTest.java
index 54ceaf44b1..e5a29a9860 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/persist/BackendReplicaInfosTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/persist/BackendReplicaInfosTest.java
@@ -49,7 +49,7 @@ public class BackendReplicaInfosTest {
BackendReplicasInfo info = new BackendReplicasInfo(beId);
info.addBadReplica(tabletId1);
- info.addMissingVersionReplica(tabletId2);
+ info.addMissingVersionReplica(tabletId2, 11);
checkInfo(info);
info.write(dos);
dos.flush();
@@ -73,6 +73,7 @@ public class BackendReplicaInfosTest {
Assert.assertEquals(BackendReplicasInfo.ReportInfoType.BAD, reportInfo.type);
} else if (reportInfo.tabletId == tabletId2) {
Assert.assertEquals(BackendReplicasInfo.ReportInfoType.MISSING_VERSION, reportInfo.type);
+ Assert.assertEquals(11, reportInfo.lastFailedVersion);
} else {
Assert.fail("unknown tablet id: " + reportInfo.tabletId);
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/transaction/GlobalTransactionMgrTest.java b/fe/fe-core/src/test/java/org/apache/doris/transaction/GlobalTransactionMgrTest.java
index 3816f73d72..abdb01f940 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/transaction/GlobalTransactionMgrTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/transaction/GlobalTransactionMgrTest.java
@@ -67,12 +67,13 @@ import java.util.Map;
import java.util.Set;
import java.util.UUID;
-import mockit.Injectable;
-import mockit.Mocked;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertNotNull;
import static org.junit.Assert.assertTrue;
+import mockit.Injectable;
+import mockit.Mocked;
+
public class GlobalTransactionMgrTest {
private static FakeEditLog fakeEditLog;
@@ -526,7 +527,9 @@ public class GlobalTransactionMgrTest {
Replica replica1 = tablet.getReplicaById(CatalogTestUtil.testReplicaId1);
Replica replica2 = tablet.getReplicaById(CatalogTestUtil.testReplicaId2);
Replica replica3 = tablet.getReplicaById(CatalogTestUtil.testReplicaId3);
- assertEquals(CatalogTestUtil.testStartVersion + 1, replica1.getVersion());
+ // because after calling `finishTransaction`, the txn state is COMMITTED, not VISIBLE,
+ // so all replicas' version are not changed.
+ assertEquals(CatalogTestUtil.testStartVersion, replica1.getVersion());
assertEquals(CatalogTestUtil.testStartVersion, replica2.getVersion());
assertEquals(CatalogTestUtil.testStartVersion, replica3.getVersion());
assertEquals(-1, replica1.getLastFailedVersion());
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 05/11: [fix](vectorized) Fix bug of outer join with delete column (#9315)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit ba0b0634dd09b81cc421227a1006a5a780f23d5e
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Thu May 5 09:42:48 2022 +0800
[fix](vectorized) Fix bug of outer join with delete column (#9315)
---
fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java | 4 ++++
fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java | 1 +
2 files changed, 5 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
index 4be8fa5308..0cf48b6afb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/Analyzer.java
@@ -1378,6 +1378,10 @@ public class Analyzer {
return globalState.fullOuterJoinedTupleIds.containsKey(tid);
}
+ public boolean isOuterMaterializedJoined(TupleId tid) {
+ return globalState.outerJoinedMaterializedTupleIds.contains(tid);
+ }
+
public boolean isFullOuterJoined(SlotId sid) {
return isFullOuterJoined(getTupleId(sid));
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
index 0ba2e7a693..7cc79ab534 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapScanNode.java
@@ -906,6 +906,7 @@ public class OlapScanNode extends ScanNode {
SlotRef deleteSignSlot = new SlotRef(desc.getAliasAsName(), Column.DELETE_SIGN);
deleteSignSlot.analyze(analyzer);
deleteSignSlot.getDesc().setIsMaterialized(true);
+ deleteSignSlot.getDesc().setIsNullable(analyzer.isOuterMaterializedJoined(desc.getId()));
Expr conjunct = new BinaryPredicate(BinaryPredicate.Operator.EQ, deleteSignSlot, new IntLiteral(0));
conjunct.analyze(analyzer);
conjuncts.add(conjunct);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 10/11: [fix](truncate) fix bug that truncate partition throw NPE (#9339)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 048b3406cd9de65259ff046845ba19575d488591
Author: Mingyu Chen <mo...@gmail.com>
AuthorDate: Sun May 1 18:26:56 2022 +0800
[fix](truncate) fix bug that truncate partition throw NPE (#9339)
1. partition name is case insensitive.
2. add a simple help-resource.zip to help pass the FE ut.
---
.../java/org/apache/doris/catalog/Catalog.java | 2 +-
.../java/org/apache/doris/catalog/OlapTable.java | 6 ++-
.../org/apache/doris/common/CaseSensibility.java | 2 +-
.../apache/doris/catalog/TruncateTableTest.java | 41 +++++++++++++++++++++
fe/fe-core/src/test/resources/help-resource.zip | Bin 0 -> 1056 bytes
5 files changed, 47 insertions(+), 4 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
index 74b0ad54ca..a6e658e535 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
@@ -6807,7 +6807,7 @@ public class Catalog {
// check partitions
for (Map.Entry<String, Long> entry : origPartitions.entrySet()) {
Partition partition = copiedTbl.getPartition(entry.getValue());
- if (partition == null || !partition.getName().equals(entry.getKey())) {
+ if (partition == null || !partition.getName().equalsIgnoreCase(entry.getKey())) {
throw new DdlException("Partition [" + entry.getKey() + "] is changed");
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index 29f7faea4a..4b015c685d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -1287,11 +1287,13 @@ public class OlapTable extends Table {
return copied;
}
- Set<String> partNames = Sets.newTreeSet(String.CASE_INSENSITIVE_ORDER);
+ Set<String> partNames = Sets.newHashSet();
partNames.addAll(copied.getPartitionNames());
+ // partition name is case insensitive:
+ Set<String> lowerReservedPartitionNames = reservedPartitions.stream().map(String::toLowerCase).collect(Collectors.toSet());
for (String partName : partNames) {
- if (!reservedPartitions.contains(partName)) {
+ if (!lowerReservedPartitionNames.contains(partName.toLowerCase())) {
copied.dropPartitionAndReserveTablet(partName);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/CaseSensibility.java b/fe/fe-core/src/main/java/org/apache/doris/common/CaseSensibility.java
index ebd7b63b5b..d0545a14dd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/CaseSensibility.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/CaseSensibility.java
@@ -22,7 +22,7 @@ public enum CaseSensibility {
DATABASE(true),
TABLE(true),
ROLLUP(true),
- PARTITION(true),
+ PARTITION(false),
COLUMN(false),
USER(true),
ROLE(false),
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/TruncateTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/TruncateTableTest.java
index db8951da4e..80aec2011f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/TruncateTableTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/TruncateTableTest.java
@@ -56,6 +56,24 @@ public class TruncateTableTest {
"properties('replication_num' = '1');";
createDb(createDbStmtStr);
createTable(createTableStr);
+
+ String createTable2 = "CREATE TABLE test.case_sensitive_table (\n" +
+ " `date_id` date NULL COMMENT \"\",\n" +
+ " `column2` tinyint(4) NULL COMMENT \"\"\n" +
+ ") ENGINE=OLAP\n" +
+ "DUPLICATE KEY(`date_id`, `column2`)\n" +
+ "COMMENT \"OLAP\"\n" +
+ "PARTITION BY RANGE(`date_id`)\n" +
+ "(\n" +
+ "PARTITION p20211006 VALUES [('2021-10-06'), ('2021-10-07')),\n" +
+ "PARTITION P20211007 VALUES [('2021-10-07'), ('2021-10-08')),\n" +
+ "PARTITION P20211008 VALUES [('2021-10-08'), ('2021-10-09')))\n" +
+ "DISTRIBUTED BY HASH(`column2`) BUCKETS 1\n" +
+ "PROPERTIES (\n" +
+ "\"replication_allocation\" = \"tag.location.default: 1\"\n" +
+ ");";
+
+ createTable(createTable2);
}
@AfterClass
@@ -64,6 +82,29 @@ public class TruncateTableTest {
file.delete();
}
+ @Test
+ public void testTruncateWithCaseInsensitivePartitionName() throws Exception {
+ Database db = Catalog.getCurrentCatalog().getDbNullable("default_cluster:test");
+ OlapTable tbl = db.getOlapTableOrDdlException("case_sensitive_table");
+ long p20211006Id = tbl.getPartition("P20211006").getId();
+ long p20211007Id = tbl.getPartition("P20211007").getId();
+ long p20211008Id = tbl.getPartition("p20211008").getId();
+ // truncate p20211008(real name is P20211008)
+ String truncateStr = "TRUNCATE TABLE test.case_sensitive_table PARTITION p20211008; \n";
+ TruncateTableStmt truncateTableStmt = (TruncateTableStmt) UtFrameUtils.parseAndAnalyzeStmt(truncateStr, connectContext);
+ Catalog.getCurrentCatalog().truncateTable(truncateTableStmt);
+ Assert.assertNotEquals(p20211008Id, tbl.getPartition("p20211008").getId());
+ // 2. truncate P20211007
+ truncateStr = "TRUNCATE TABLE test.case_sensitive_table PARTITION P20211007; \n";
+ truncateTableStmt = (TruncateTableStmt) UtFrameUtils.parseAndAnalyzeStmt(truncateStr, connectContext);
+ Catalog.getCurrentCatalog().truncateTable(truncateTableStmt);
+ Assert.assertEquals(3, tbl.getPartitionInfo().idToDataProperty.size());
+ Assert.assertNotEquals(p20211007Id, tbl.getPartition("p20211007").getId());
+ Assert.assertEquals(p20211006Id, tbl.getPartition("p20211006").getId());
+ Assert.assertNotNull(tbl.getPartition("p20211006"));
+ Assert.assertNotNull(tbl.getPartition("P20211006"));
+ }
+
@Test
public void testTruncateTable() throws Exception {
String stmtStr = "ALTER TABLE test.tbl ADD PARTITION p20210902 VALUES [('2021-09-02'), ('2021-09-03')) DISTRIBUTED BY HASH(`k1`) BUCKETS 3;";
diff --git a/fe/fe-core/src/test/resources/help-resource.zip b/fe/fe-core/src/test/resources/help-resource.zip
new file mode 100644
index 0000000000..7ab08e7bf2
Binary files /dev/null and b/fe/fe-core/src/test/resources/help-resource.zip differ
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 04/11: [fix](profile) Short-circuit and del predicate filter rows are not counted on vectorized exec (#9268)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit a927681e3f51ef30d77a42881c0042400444d73b
Author: ZenoYang <co...@qq.com>
AuthorDate: Fri Apr 29 10:45:48 2022 +0800
[fix](profile) Short-circuit and del predicate filter rows are not counted on vectorized exec (#9268)
---
be/src/olap/olap_common.h | 2 +-
be/src/olap/rowset/segment_v2/segment_iterator.cpp | 4 ++++
docs/en/community/how-to-contribute/commit-format-specification.md | 4 +++-
docs/zh-CN/community/how-to-contribute/commit-format-specification.md | 1 +
4 files changed, 9 insertions(+), 2 deletions(-)
diff --git a/be/src/olap/olap_common.h b/be/src/olap/olap_common.h
index 506308139d..6c7c0e598d 100644
--- a/be/src/olap/olap_common.h
+++ b/be/src/olap/olap_common.h
@@ -238,7 +238,7 @@ class Field;
class WrapperField;
using KeyRange = std::pair<WrapperField*, WrapperField*>;
-static const int GENERAL_DEBUG_COUNT = 4;
+static const int GENERAL_DEBUG_COUNT = 0;
// ReaderStatistics used to collect statistics when scan data from storage
struct OlapReaderStatistics {
diff --git a/be/src/olap/rowset/segment_v2/segment_iterator.cpp b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
index 9484601f12..a5767dcb08 100644
--- a/be/src/olap/rowset/segment_v2/segment_iterator.cpp
+++ b/be/src/olap/rowset/segment_v2/segment_iterator.cpp
@@ -858,6 +858,7 @@ void SegmentIterator::_evaluate_short_circuit_predicate(uint16_t* vec_sel_rowid_
return;
}
+ uint16_t original_size = *selected_size_ptr;
for (auto predicate : _short_cir_eval_predicate) {
auto column_id = predicate->column_id();
auto& short_cir_column = _current_return_columns[column_id];
@@ -869,10 +870,13 @@ void SegmentIterator::_evaluate_short_circuit_predicate(uint16_t* vec_sel_rowid_
}
predicate->evaluate(*short_cir_column, vec_sel_rowid_idx, selected_size_ptr);
}
+ _opts.stats->rows_vec_cond_filtered += original_size - *selected_size_ptr;
// evaluate delete condition
+ original_size = *selected_size_ptr;
_opts.delete_condition_predicates->evaluate(_current_return_columns, vec_sel_rowid_idx,
selected_size_ptr);
+ _opts.stats->rows_vec_del_cond_filtered += original_size - *selected_size_ptr;
}
void SegmentIterator::_read_columns_by_rowids(std::vector<ColumnId>& read_column_ids,
diff --git a/docs/en/community/how-to-contribute/commit-format-specification.md b/docs/en/community/how-to-contribute/commit-format-specification.md
index da4fb59203..b42b9a17f8 100644
--- a/docs/en/community/how-to-contribute/commit-format-specification.md
+++ b/docs/en/community/how-to-contribute/commit-format-specification.md
@@ -46,7 +46,7 @@ Commit is divided into ‘ title ’ and ‘ content ’ , the title should be l
* style: Code style adjustment
* typo: Code or Document correction
* refactor: Code refactoring (no function changes involved)
- * performance: Performance optimization
+ * performance/optimize: Performance optimization
* test: Addition or repair of unit test
* chore: Modification of build tool
* revert: Revert a previous commit
@@ -78,7 +78,9 @@ Commit is divided into ‘ title ’ and ‘ content ’ , the title should be l
* log
* cache
* config
+ * vectorization
* docs
+ * profile
Some tips：
diff --git a/docs/zh-CN/community/how-to-contribute/commit-format-specification.md b/docs/zh-CN/community/how-to-contribute/commit-format-specification.md
index 3272d0fb53..fde9828402 100644
--- a/docs/zh-CN/community/how-to-contribute/commit-format-specification.md
+++ b/docs/zh-CN/community/how-to-contribute/commit-format-specification.md
@@ -78,6 +78,7 @@ Commit 分为“标题”和“内容”。原则上标题全部小写。内容
* config
* vectorization
* docs
+ * profile
几点说明：
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 06/11: [Bug] [Build] fix clang build fail (#9323)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit f4bdeeebac3d73fa806d8842b0c9995f64e6f5b0
Author: Pxl <95...@qq.com>
AuthorDate: Mon May 2 18:04:57 2022 +0800
[Bug] [Build] fix clang build fail (#9323)
* fix clang compile fail
---
be/src/exec/broker_scan_node.h | 20 ++++++++++++--------
be/src/exec/broker_scanner.h | 2 +-
2 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/be/src/exec/broker_scan_node.h b/be/src/exec/broker_scan_node.h
index c4494731d8..ffabda7d0b 100644
--- a/be/src/exec/broker_scan_node.h
+++ b/be/src/exec/broker_scan_node.h
@@ -41,29 +41,33 @@ struct ScannerCounter;
class BrokerScanNode : public ScanNode {
public:
BrokerScanNode(ObjectPool* pool, const TPlanNode& tnode, const DescriptorTbl& descs);
- virtual ~BrokerScanNode();
+ ~BrokerScanNode() override;
// Called after create this scan node
- virtual Status init(const TPlanNode& tnode, RuntimeState* state = nullptr) override;
+ Status init(const TPlanNode& tnode, RuntimeState* state = nullptr) override;
// Prepare partition infos & set up timer
- virtual Status prepare(RuntimeState* state) override;
+ Status prepare(RuntimeState* state) override;
// Start broker scan using ParquetScanner or BrokerScanner.
- virtual Status open(RuntimeState* state) override;
+ Status open(RuntimeState* state) override;
// Fill the next row batch by calling next() on the scanner,
- virtual Status get_next(RuntimeState* state, RowBatch* row_batch, bool* eos) override;
+ Status get_next(RuntimeState* state, RowBatch* row_batch, bool* eos) override;
+
+ Status get_next(RuntimeState* state, vectorized::Block* block, bool* eos) override {
+ return Status::NotSupported("Not Implemented get block");
+ }
// Close the scanner, and report errors.
- virtual Status close(RuntimeState* state) override;
+ Status close(RuntimeState* state) override;
// No use
- virtual Status set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges) override;
+ Status set_scan_ranges(const std::vector<TScanRangeParams>& scan_ranges) override;
protected:
// Write debug string of this into out.
- virtual void debug_string(int indentation_level, std::stringstream* out) const override;
+ void debug_string(int indentation_level, std::stringstream* out) const override;
private:
// Update process status to one failed status,
diff --git a/be/src/exec/broker_scanner.h b/be/src/exec/broker_scanner.h
index ddd8bd9c78..7bc194f289 100644
--- a/be/src/exec/broker_scanner.h
+++ b/be/src/exec/broker_scanner.h
@@ -57,7 +57,7 @@ public:
const TBrokerScanRangeParams& params, const std::vector<TBrokerRangeDesc>& ranges,
const std::vector<TNetworkAddress>& broker_addresses,
const std::vector<TExpr>& pre_filter_texprs, ScannerCounter* counter);
- ~BrokerScanner();
+ ~BrokerScanner() override;
// Open this scanner, will initialize information need to
Status open() override;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 07/11: [fix](alter-job) Missing alter job when doing checkpoint image (#9329)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 6c3bb756d516d2a17534161b5d52575934c31439
Author: Mingyu Chen <mo...@gmail.com>
AuthorDate: Tue May 3 22:36:36 2022 +0800
[fix](alter-job) Missing alter job when doing checkpoint image (#9329)
This bug is introduced from #8030
---
.../java/org/apache/doris/alter/AlterJobV2.java | 2 +-
.../apache/doris/alter/SchemaChangeHandler.java | 11 ++++---
.../java/org/apache/doris/catalog/Catalog.java | 35 ++++++++++++----------
.../org/apache/doris/common/proc/JobsProcDir.java | 4 +--
.../java/org/apache/doris/master/MasterImpl.java | 6 ++--
.../java/org/apache/doris/persist/EditLog.java | 10 +++----
.../org/apache/doris/alter/AlterJobV2Test.java | 10 +++----
.../java/org/apache/doris/alter/AlterTest.java | 2 +-
.../org/apache/doris/alter/BatchRollupJobTest.java | 6 ++--
.../org/apache/doris/alter/RollupJobV2Test.java | 11 ++++---
.../apache/doris/catalog/CatalogOperationTest.java | 2 +-
.../apache/doris/catalog/TempPartitionTest.java | 6 ++--
.../org/apache/doris/http/DorisHttpTestCase.java | 2 +-
.../java/org/apache/doris/utframe/DorisAssert.java | 2 +-
14 files changed, 55 insertions(+), 54 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJobV2.java b/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJobV2.java
index 06fc79efc6..764a0b0ae8 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJobV2.java
@@ -58,7 +58,7 @@ public abstract class AlterJobV2 implements Writable {
}
public enum JobType {
- ROLLUP, SCHEMA_CHANGE, DECOMMISSION_BACKEND
+ ROLLUP, SCHEMA_CHANGE
}
@SerializedName(value = "type")
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
index 989135b3d7..d12382f926 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java
@@ -118,7 +118,7 @@ public class SchemaChangeHandler extends AlterHandler {
public final Map<Long, AlterJobV2> runnableSchemaChangeJobV2 = Maps.newConcurrentMap();
- public int cycle_count = 0;
+ public int cycleCount = 0;
public SchemaChangeHandler() {
super("schema change", Config.default_schema_change_scheduler_interval_millisecond);
@@ -1391,13 +1391,13 @@ public class SchemaChangeHandler extends AlterHandler {
@Override
protected void runAfterCatalogReady() {
- if (cycle_count >= CYCLE_COUNT_TO_CHECK_EXPIRE_SCHEMA_CHANGE_JOB) {
+ if (cycleCount >= CYCLE_COUNT_TO_CHECK_EXPIRE_SCHEMA_CHANGE_JOB) {
clearFinishedOrCancelledSchemaChangeJobV2();
super.runAfterCatalogReady();
- cycle_count = 0;
+ cycleCount = 0;
}
runAlterJobV2();
- cycle_count++;
+ cycleCount++;
}
private void runAlterJobV2() {
@@ -1868,12 +1868,11 @@ public class SchemaChangeHandler extends AlterHandler {
}
@Override
- protected void addAlterJobV2(AlterJobV2 alterJob) {
+ public void addAlterJobV2(AlterJobV2 alterJob) {
super.addAlterJobV2(alterJob);
runnableSchemaChangeJobV2.put(alterJob.getJobId(), alterJob);
}
-
private void clearFinishedOrCancelledSchemaChangeJobV2() {
Iterator<Map.Entry<Long, AlterJobV2>> iterator = runnableSchemaChangeJobV2.entrySet().iterator();
while (iterator.hasNext()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
index ccae1c3b5f..74b0ad54ca 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
@@ -1816,8 +1816,6 @@ public class Catalog {
}
public long loadAlterJob(DataInputStream dis, long checksum, JobType type) throws IOException {
- Map<Long, AlterJobV2> alterJobsV2 = Maps.newHashMap();
-
// alter jobs
int size = dis.readInt();
long newChecksum = checksum ^ size;
@@ -1826,13 +1824,11 @@ public class Catalog {
throw new IOException("There are [" + size + "] old alter jobs. Please downgrade FE to an older version and handle residual jobs");
}
- if (Catalog.getCurrentCatalogJournalVersion() >= 2) {
- // finished or cancelled jobs
- size = dis.readInt();
- newChecksum ^= size;
- if (size > 0) {
- throw new IOException("There are [" + size + "] old finished or cancelled alter jobs. Please downgrade FE to an older version and handle residual jobs");
- }
+ // finished or cancelled jobs
+ size = dis.readInt();
+ newChecksum ^= size;
+ if (size > 0) {
+ throw new IOException("There are [" + size + "] old finished or cancelled alter jobs. Please downgrade FE to an older version and handle residual jobs");
}
// alter job v2
@@ -1842,9 +1838,9 @@ public class Catalog {
AlterJobV2 alterJobV2 = AlterJobV2.read(dis);
if (type == JobType.ROLLUP || type == JobType.SCHEMA_CHANGE) {
if (type == JobType.ROLLUP) {
- this.getRollupHandler().addAlterJobV2(alterJobV2);
+ this.getMaterializedViewHandler().addAlterJobV2(alterJobV2);
} else {
- alterJobsV2.put(alterJobV2.getJobId(), alterJobV2);
+ this.getSchemaChangeHandler().addAlterJobV2(alterJobV2);
}
// ATTN : we just want to add tablet into TabletInvertedIndex when only PendingJob is checkpointed
// to prevent TabletInvertedIndex data loss,
@@ -1854,7 +1850,7 @@ public class Catalog {
LOG.info("replay pending alter job when load alter job {} ", alterJobV2.getJobId());
}
} else {
- alterJobsV2.put(alterJobV2.getJobId(), alterJobV2);
+ throw new IOException("Invalid alter job type: " + type.name());
}
}
@@ -2113,7 +2109,14 @@ public class Catalog {
}
public long saveAlterJob(CountingDataOutputStream dos, long checksum, JobType type) throws IOException {
- Map<Long, AlterJobV2> alterJobsV2 = Maps.newHashMap();
+ Map<Long, AlterJobV2> alterJobsV2;
+ if (type == JobType.ROLLUP) {
+ alterJobsV2 = this.getMaterializedViewHandler().getAlterJobsV2();
+ } else if (type == JobType.SCHEMA_CHANGE) {
+ alterJobsV2 = this.getSchemaChangeHandler().getAlterJobsV2();
+ } else {
+ throw new IOException("Invalid alter job type: " + type.name());
+ }
// alter jobs == 0
// If the FE version upgrade from old version, if it have alter jobs, the FE will failed during start process
@@ -3811,7 +3814,7 @@ public class Catalog {
}
Preconditions.checkNotNull(rollupIndexStorageType);
// set rollup index meta to olap table
- List<Column> rollupColumns = getRollupHandler().checkAndPrepareMaterializedView(addRollupClause,
+ List<Column> rollupColumns = getMaterializedViewHandler().checkAndPrepareMaterializedView(addRollupClause,
olapTable, baseRollupIndex, false);
short rollupShortKeyColumnCount = Catalog.calcShortKeyColumnCount(rollupColumns, alterClause.getProperties());
int rollupSchemaHash = Util.generateSchemaHash();
@@ -5022,7 +5025,7 @@ public class Catalog {
return (SchemaChangeHandler) this.alter.getSchemaChangeHandler();
}
- public MaterializedViewHandler getRollupHandler() {
+ public MaterializedViewHandler getMaterializedViewHandler() {
return (MaterializedViewHandler) this.alter.getMaterializedViewHandler();
}
@@ -5300,7 +5303,7 @@ public class Catalog {
*/
public void cancelAlter(CancelAlterTableStmt stmt) throws DdlException {
if (stmt.getAlterType() == AlterType.ROLLUP) {
- this.getRollupHandler().cancel(stmt);
+ this.getMaterializedViewHandler().cancel(stmt);
} else if (stmt.getAlterType() == AlterType.COLUMN) {
this.getSchemaChangeHandler().cancel(stmt);
} else {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/proc/JobsProcDir.java b/fe/fe-core/src/main/java/org/apache/doris/common/proc/JobsProcDir.java
index 85e3b089c2..dfe272ff10 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/proc/JobsProcDir.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/proc/JobsProcDir.java
@@ -72,7 +72,7 @@ public class JobsProcDir implements ProcDirInterface {
} else if (jobTypeName.equals(DELETE)) {
return new DeleteInfoProcDir(catalog.getDeleteHandler(), catalog.getLoadInstance(), db.getId());
} else if (jobTypeName.equals(ROLLUP)) {
- return new RollupProcDir(catalog.getRollupHandler(), db);
+ return new RollupProcDir(catalog.getMaterializedViewHandler(), db);
} else if (jobTypeName.equals(SCHEMA_CHANGE)) {
return new SchemaChangeProcDir(catalog.getSchemaChangeHandler(), db);
} else if (jobTypeName.equals(EXPORT)) {
@@ -119,7 +119,7 @@ public class JobsProcDir implements ProcDirInterface {
cancelledNum.toString(), totalNum.toString()));
// rollup
- MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getRollupHandler();
+ MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getMaterializedViewHandler();
pendingNum = materializedViewHandler.getAlterJobV2Num(org.apache.doris.alter.AlterJobV2.JobState.PENDING, dbId);
runningNum = materializedViewHandler.getAlterJobV2Num(org.apache.doris.alter.AlterJobV2.JobState.WAITING_TXN, dbId)
+ materializedViewHandler.getAlterJobV2Num(org.apache.doris.alter.AlterJobV2.JobState.RUNNING, dbId);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java b/fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java
index 5230f9f40a..c1a12a840c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/master/MasterImpl.java
@@ -60,12 +60,12 @@ import org.apache.doris.thrift.TStatusCode;
import org.apache.doris.thrift.TTabletInfo;
import org.apache.doris.thrift.TTaskType;
+import com.google.common.base.Preconditions;
+
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
import org.apache.thrift.TException;
-import com.google.common.base.Preconditions;
-
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.List;
@@ -758,7 +758,7 @@ public class MasterImpl {
AlterReplicaTask alterTask = (AlterReplicaTask) task;
try {
if (alterTask.getJobType() == JobType.ROLLUP) {
- Catalog.getCurrentCatalog().getRollupHandler().handleFinishAlterTask(alterTask);
+ Catalog.getCurrentCatalog().getMaterializedViewHandler().handleFinishAlterTask(alterTask);
} else if (alterTask.getJobType() == JobType.SCHEMA_CHANGE) {
Catalog.getCurrentCatalog().getSchemaChangeHandler().handleFinishAlterTask(alterTask);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java b/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
index 574c77e156..7b72ef9f4c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java
@@ -275,13 +275,13 @@ public class EditLog {
}
case OperationType.OP_DROP_ROLLUP: {
DropInfo info = (DropInfo) journal.getData();
- catalog.getRollupHandler().replayDropRollup(info, catalog);
+ catalog.getMaterializedViewHandler().replayDropRollup(info, catalog);
break;
}
case OperationType.OP_BATCH_DROP_ROLLUP: {
BatchDropInfo batchDropInfo = (BatchDropInfo) journal.getData();
for (long indexId : batchDropInfo.getIndexIdSet()) {
- catalog.getRollupHandler().replayDropRollup(
+ catalog.getMaterializedViewHandler().replayDropRollup(
new DropInfo(batchDropInfo.getDbId(), batchDropInfo.getTableId(), indexId, false), catalog);
}
break;
@@ -696,7 +696,7 @@ public class EditLog {
AlterJobV2 alterJob = (AlterJobV2) journal.getData();
switch (alterJob.getType()) {
case ROLLUP:
- catalog.getRollupHandler().replayAlterJobV2(alterJob);
+ catalog.getMaterializedViewHandler().replayAlterJobV2(alterJob);
break;
case SCHEMA_CHANGE:
catalog.getSchemaChangeHandler().replayAlterJobV2(alterJob);
@@ -709,7 +709,7 @@ public class EditLog {
case OperationType.OP_BATCH_ADD_ROLLUP: {
BatchAlterJobPersistInfo batchAlterJobV2 = (BatchAlterJobPersistInfo) journal.getData();
for (AlterJobV2 alterJobV2 : batchAlterJobV2.getAlterJobV2List()) {
- catalog.getRollupHandler().replayAlterJobV2(alterJobV2);
+ catalog.getMaterializedViewHandler().replayAlterJobV2(alterJobV2);
}
break;
}
@@ -754,7 +754,7 @@ public class EditLog {
RemoveAlterJobV2OperationLog log = (RemoveAlterJobV2OperationLog) journal.getData();
switch (log.getType()) {
case ROLLUP:
- catalog.getRollupHandler().replayRemoveAlterJobV2(log);
+ catalog.getMaterializedViewHandler().replayRemoveAlterJobV2(log);
break;
case SCHEMA_CHANGE:
catalog.getSchemaChangeHandler().replayRemoveAlterJobV2(log);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/alter/AlterJobV2Test.java b/fe/fe-core/src/test/java/org/apache/doris/alter/AlterJobV2Test.java
index 2bb1842b36..20f44dc60a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/alter/AlterJobV2Test.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/alter/AlterJobV2Test.java
@@ -134,7 +134,7 @@ public class AlterJobV2Test {
AlterTableStmt alterTableStmt = (AlterTableStmt) UtFrameUtils.parseAndAnalyzeStmt(alterStmtStr, connectContext);
Catalog.getCurrentCatalog().getAlterInstance().processAlterTable(alterTableStmt);
// 2. check alter job
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
waitAlterJobDone(alterJobs);
// 3. check show alter table column
String showAlterStmtStr = "show alter table rollup from test;";
@@ -157,7 +157,7 @@ public class AlterJobV2Test {
String alterStmtStr = "alter table test.segmentv2 add rollup r1(k2, v1)";
AlterTableStmt alterTableStmt = (AlterTableStmt) UtFrameUtils.parseAndAnalyzeStmt(alterStmtStr, connectContext);
Catalog.getCurrentCatalog().getAlterInstance().processAlterTable(alterTableStmt);
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
waitAlterJobDone(alterJobs);
String sql = "select k2, sum(v1) from test.segmentv2 group by k2";
@@ -168,7 +168,7 @@ public class AlterJobV2Test {
alterStmtStr = "alter table test.segmentv2 add rollup segmentv2(k2, v1) properties('storage_format' = 'v2')";
alterTableStmt = (AlterTableStmt) UtFrameUtils.parseAndAnalyzeStmt(alterStmtStr, connectContext);
Catalog.getCurrentCatalog().getAlterInstance().processAlterTable(alterTableStmt);
- alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
waitAlterJobDone(alterJobs);
explainString = UtFrameUtils.getSQLPlanOrErrorMsg(connectContext, "explain " + sql);
@@ -219,7 +219,7 @@ public class AlterJobV2Test {
alterTable("alter table test.dup_table add rollup r1(v1,v2,k2,k1);");
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
waitAlterJobDone(alterJobs);
ExceptionChecker.expectThrowsNoException(() -> alterTable("alter table test.dup_table modify column v2 varchar(2);"));
}
@@ -253,7 +253,7 @@ public class AlterJobV2Test {
"city,\n" +
"user_id,\n" +
"date;");
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
waitAlterJobDone(alterJobs);
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/alter/AlterTest.java b/fe/fe-core/src/test/java/org/apache/doris/alter/AlterTest.java
index c055ba20ba..08569cdc1a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/alter/AlterTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/alter/AlterTest.java
@@ -432,7 +432,7 @@ public class AlterTest {
private void waitSchemaChangeJobDone(boolean rollupJob) throws Exception {
Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getSchemaChangeHandler().getAlterJobsV2();
if (rollupJob) {
- alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
}
for (AlterJobV2 alterJobV2 : alterJobs.values()) {
while (!alterJobV2.getJobState().isFinalState()) {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/alter/BatchRollupJobTest.java b/fe/fe-core/src/test/java/org/apache/doris/alter/BatchRollupJobTest.java
index b12077855a..61384768ea 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/alter/BatchRollupJobTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/alter/BatchRollupJobTest.java
@@ -55,7 +55,7 @@ public class BatchRollupJobTest {
@Before
public void before() throws Exception {
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
alterJobs.clear();
// create database db1
@@ -77,7 +77,7 @@ public class BatchRollupJobTest {
AlterTableStmt alterTableStmt = (AlterTableStmt) UtFrameUtils.parseAndAnalyzeStmt(stmtStr, ctx);
Catalog.getCurrentCatalog().getAlterInstance().processAlterTable(alterTableStmt);
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
Assert.assertEquals(3, alterJobs.size());
Database db = Catalog.getCurrentCatalog().getDbNullable("default_cluster:db1");
@@ -127,7 +127,7 @@ public class BatchRollupJobTest {
AlterTableStmt alterTableStmt = (AlterTableStmt) UtFrameUtils.parseAndAnalyzeStmt(stmtStr, ctx);
Catalog.getCurrentCatalog().getAlterInstance().processAlterTable(alterTableStmt);
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
Assert.assertEquals(3, alterJobs.size());
List<Long> jobIds = Lists.newArrayList(alterJobs.keySet());
diff --git a/fe/fe-core/src/test/java/org/apache/doris/alter/RollupJobV2Test.java b/fe/fe-core/src/test/java/org/apache/doris/alter/RollupJobV2Test.java
index 2e00bdc54a..6e6b18536b 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/alter/RollupJobV2Test.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/alter/RollupJobV2Test.java
@@ -63,7 +63,6 @@ import org.apache.doris.transaction.FakeTransactionIDGenerator;
import org.apache.doris.transaction.GlobalTransactionMgr;
import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
import org.junit.After;
import org.junit.Assert;
@@ -150,7 +149,7 @@ public class RollupJobV2Test {
fakeCatalog = new FakeCatalog();
fakeEditLog = new FakeEditLog();
FakeCatalog.setCatalog(masterCatalog);
- MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getRollupHandler();
+ MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getMaterializedViewHandler();
ArrayList<AlterClause> alterClauses = new ArrayList<>();
alterClauses.add(clause);
alterClauses.add(clause2);
@@ -171,7 +170,7 @@ public class RollupJobV2Test {
fakeCatalog = new FakeCatalog();
fakeEditLog = new FakeEditLog();
FakeCatalog.setCatalog(masterCatalog);
- MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getRollupHandler();
+ MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getMaterializedViewHandler();
ArrayList<AlterClause> alterClauses = new ArrayList<>();
alterClauses.add(clause);
Database db = masterCatalog.getDbOrDdlException(CatalogTestUtil.testDbId1);
@@ -188,7 +187,7 @@ public class RollupJobV2Test {
fakeCatalog = new FakeCatalog();
fakeEditLog = new FakeEditLog();
FakeCatalog.setCatalog(masterCatalog);
- MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getRollupHandler();
+ MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getMaterializedViewHandler();
// add a rollup job
ArrayList<AlterClause> alterClauses = new ArrayList<>();
@@ -240,7 +239,7 @@ public class RollupJobV2Test {
fakeCatalog = new FakeCatalog();
fakeEditLog = new FakeEditLog();
FakeCatalog.setCatalog(masterCatalog);
- MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getRollupHandler();
+ MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getMaterializedViewHandler();
// add a rollup job
ArrayList<AlterClause> alterClauses = new ArrayList<>();
@@ -378,7 +377,7 @@ public class RollupJobV2Test {
fakeCatalog = new FakeCatalog();
fakeEditLog = new FakeEditLog();
FakeCatalog.setCatalog(masterCatalog);
- MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getRollupHandler();
+ MaterializedViewHandler materializedViewHandler = Catalog.getCurrentCatalog().getMaterializedViewHandler();
Database db = masterCatalog.getDbOrDdlException(CatalogTestUtil.testDbId1);
OlapTable olapTable = (OlapTable) db.getTableOrDdlException(CatalogTestUtil.testTableId2);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/CatalogOperationTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/CatalogOperationTest.java
index cf3135614a..ad21551377 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/CatalogOperationTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/CatalogOperationTest.java
@@ -122,7 +122,7 @@ public class CatalogOperationTest {
String alterStmtStr = "alter table test.newNewTest add rollup r1(k1)";
alterTableStmt = (AlterTableStmt) UtFrameUtils.parseAndAnalyzeStmt(alterStmtStr, connectContext);
Catalog.getCurrentCatalog().getAlterInstance().processAlterTable(alterTableStmt);
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
Assert.assertEquals(1, alterJobs.size());
for (AlterJobV2 alterJobV2 : alterJobs.values()) {
while (!alterJobV2.getJobState().isFinalState()) {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/TempPartitionTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/TempPartitionTest.java
index ffd3528a56..b4c80e4dee 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/TempPartitionTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/TempPartitionTest.java
@@ -432,7 +432,7 @@ public class TempPartitionTest {
alterTable(stmtStr, true);
// wait rollup finish
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
for (AlterJobV2 alterJobV2 : alterJobs.values()) {
while (!alterJobV2.getJobState().isFinalState()) {
System.out.println(
@@ -794,7 +794,7 @@ public class TempPartitionTest {
alterTable(stmtStr, true);
// wait rollup finish
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
for (AlterJobV2 alterJobV2 : alterJobs.values()) {
while (!alterJobV2.getJobState().isFinalState()) {
System.out.println(
@@ -1132,7 +1132,7 @@ public class TempPartitionTest {
alterTable(stmtStr, true);
// wait rollup finish
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
for (AlterJobV2 alterJobV2 : alterJobs.values()) {
while (!alterJobV2.getJobState().isFinalState()) {
System.out.println(
diff --git a/fe/fe-core/src/test/java/org/apache/doris/http/DorisHttpTestCase.java b/fe/fe-core/src/test/java/org/apache/doris/http/DorisHttpTestCase.java
index 71f4adaf24..dd2ce844f3 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/http/DorisHttpTestCase.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/http/DorisHttpTestCase.java
@@ -318,7 +318,7 @@ abstract public class DorisHttpTestCase {
return new SchemaChangeHandler();
}
@Mock
- MaterializedViewHandler getRollupHandler() {
+ MaterializedViewHandler getMaterializedViewHandler() {
return new MaterializedViewHandler();
}
@Mock
diff --git a/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java b/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
index eabaab95c8..520352d654 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
@@ -148,7 +148,7 @@ public class DorisAssert {
private void checkAlterJob() throws InterruptedException {
// check alter job
- Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getRollupHandler().getAlterJobsV2();
+ Map<Long, AlterJobV2> alterJobs = Catalog.getCurrentCatalog().getMaterializedViewHandler().getAlterJobsV2();
for (AlterJobV2 alterJobV2 : alterJobs.values()) {
while (!alterJobV2.getJobState().isFinalState()) {
System.out.println("alter job " + alterJobV2.getDbId()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 08/11: [improvement] set name for scanner threads and fix compile error in clang (#9336)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit b925413d9c429065753d7066acfb02420cc33129
Author: dataroaring <98...@users.noreply.github.com>
AuthorDate: Thu May 5 09:53:43 2022 +0800
[improvement] set name for scanner threads and fix compile error in clang (#9336)
---
be/src/exec/base_scanner.h | 2 +-
be/src/exec/broker_scan_node.cpp | 2 ++
be/src/exec/broker_scanner.h | 3 ++-
be/src/util/thread.cpp | 4 ++++
be/src/util/thread.h | 2 ++
5 files changed, 11 insertions(+), 2 deletions(-)
diff --git a/be/src/exec/base_scanner.h b/be/src/exec/base_scanner.h
index 21abf080f9..bf3380e077 100644
--- a/be/src/exec/base_scanner.h
+++ b/be/src/exec/base_scanner.h
@@ -54,7 +54,7 @@ public:
virtual Status open();
// Get next tuple
- virtual Status get_next(Tuple* tuple, MemPool* tuple_pool, bool* eof, bool *fill_tuple) = 0;
+ virtual Status get_next(Tuple* tuple, MemPool* tuple_pool, bool* eof, bool* fill_tuple) = 0;
// Close this scanner
virtual void close() = 0;
diff --git a/be/src/exec/broker_scan_node.cpp b/be/src/exec/broker_scan_node.cpp
index 11928480d8..0ecd7ba875 100644
--- a/be/src/exec/broker_scan_node.cpp
+++ b/be/src/exec/broker_scan_node.cpp
@@ -31,6 +31,7 @@
#include "runtime/row_batch.h"
#include "runtime/runtime_state.h"
#include "util/runtime_profile.h"
+#include "util/thread.h"
namespace doris {
@@ -247,6 +248,7 @@ Status BrokerScanNode::scanner_scan(const TBrokerScanRange& scan_range,
const std::vector<ExprContext*>& conjunct_ctxs,
ScannerCounter* counter) {
//create scanner object and open
+ Thread::set_self_name("broker_scanner");
std::unique_ptr<BaseScanner> scanner = create_scanner(scan_range, counter);
RETURN_IF_ERROR(scanner->open());
bool scanner_eof = false;
diff --git a/be/src/exec/broker_scanner.h b/be/src/exec/broker_scanner.h
index 7bc194f289..c5aa786b10 100644
--- a/be/src/exec/broker_scanner.h
+++ b/be/src/exec/broker_scanner.h
@@ -63,7 +63,8 @@ public:
Status open() override;
// Get next tuple
- Status get_next(Tuple* tuple, MemPool* tuple_pool, bool* eof, bool* fill_tuple) override;
+ virtual Status get_next(Tuple* tuple, MemPool* tuple_pool, bool* eof,
+ bool* fill_tuple) override;
// Close this scanner
void close() override;
diff --git a/be/src/util/thread.cpp b/be/src/util/thread.cpp
index 9e35cc134b..a8d79fc6a5 100644
--- a/be/src/util/thread.cpp
+++ b/be/src/util/thread.cpp
@@ -256,6 +256,10 @@ Thread::~Thread() {
}
}
+void Thread::set_self_name(const std::string& name) {
+ ThreadMgr::set_thread_name(name, current_thread_id());
+}
+
void Thread::join() {
ThreadJoiner(this).join();
}
diff --git a/be/src/util/thread.h b/be/src/util/thread.h
index c890f81286..f3f76d12d9 100644
--- a/be/src/util/thread.h
+++ b/be/src/util/thread.h
@@ -85,6 +85,8 @@ public:
return start_thread(category, name, std::bind(f, a1, a2, a3, a4, a5, a6), NO_FLAGS, holder);
}
+ static void set_self_name(const std::string& name);
+
~Thread();
// Blocks until this thread finishes execution. Once this method returns, the thread
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 01/11: [fix](backup) Remove colocate_with property when backing up a table (#9142)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 706e6cda1227218d676eb1dbb496d5f2353140ca
Author: Mingyu Chen <mo...@gmail.com>
AuthorDate: Thu May 5 20:44:27 2022 +0800
[fix](backup) Remove colocate_with property when backing up a table (#9142)
We currently not support backup table with colocation property.
So that we have to remove colocate_with property from a table when backing up.
---
.../src/main/java/org/apache/doris/backup/BackupJob.java | 7 +++++++
.../src/main/java/org/apache/doris/catalog/OlapTable.java | 3 ++-
.../src/main/java/org/apache/doris/catalog/TableProperty.java | 1 +
.../src/test/java/org/apache/doris/catalog/OlapTableTest.java | 11 +++++++----
4 files changed, 17 insertions(+), 5 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupJob.java b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupJob.java
index f23177950e..5f07552c79 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/backup/BackupJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/backup/BackupJob.java
@@ -507,6 +507,8 @@ public class BackupJob extends AbstractJob {
status = new Status(ErrCode.COMMON_ERROR, "failed to copy table: " + tblName);
return;
}
+
+ removeUnsupportProperties(copiedTbl);
copiedTables.add(copiedTbl);
} else if (table.getType() == TableType.VIEW) {
View view = (View) table;
@@ -543,6 +545,11 @@ public class BackupJob extends AbstractJob {
backupMeta = new BackupMeta(copiedTables, copiedResources);
}
+ private void removeUnsupportProperties(OlapTable tbl) {
+ // We cannot support the colocate attribute because the colocate information is not backed up
+ // synchronously when backing up.
+ tbl.setColocateGroup(null);
+ }
private void waitingAllSnapshotsFinished() {
if (unfinishedTaskIds.isEmpty()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index 0c2c99ad97..29f7faea4a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -424,10 +424,11 @@ public class OlapTable extends Table {
* Reset properties to correct values.
*/
public void resetPropertiesForRestore() {
- // disable dynamic partition
if (tableProperty != null) {
tableProperty.resetPropertiesForRestore();
}
+ // remove colocate property.
+ setColocateGroup(null);
}
public Status resetIdsForRestore(Catalog catalog, Database db, ReplicaAllocation restoreReplicaAlloc) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java
index 7c0d6f277e..38d85dc4c1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java
@@ -107,6 +107,7 @@ public class TableProperty implements Writable {
* @return this for chained
*/
public TableProperty resetPropertiesForRestore() {
+ // disable dynamic partition
if (properties.containsKey(DynamicPartitionProperty.ENABLE)) {
properties.put(DynamicPartitionProperty.ENABLE, "false");
executeBuildDynamicProperty();
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java
index cc9d7f47f5..2c57fe054d 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/OlapTableTest.java
@@ -17,10 +17,6 @@
package org.apache.doris.catalog;
-import com.google.common.collect.Maps;
-import mockit.Mock;
-import mockit.MockUp;
-
import org.apache.doris.analysis.IndexDef;
import org.apache.doris.catalog.Table.TableType;
import org.apache.doris.common.FeConstants;
@@ -28,6 +24,7 @@ import org.apache.doris.common.io.FastByteArrayOutputStream;
import org.apache.doris.common.util.UnitTestUtil;
import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
import org.junit.Assert;
import org.junit.Test;
@@ -38,6 +35,9 @@ import java.io.IOException;
import java.util.List;
import java.util.Map;
+import mockit.Mock;
+import mockit.MockUp;
+
public class OlapTableTest {
@Test
@@ -88,10 +88,13 @@ public class OlapTableTest {
OlapTable olapTable = new OlapTable();
olapTable.setTableProperty(tableProperty);
+ olapTable.setColocateGroup("test_group");
+ Assert.assertTrue(olapTable.isColocateTable());
olapTable.resetPropertiesForRestore();
Assert.assertEquals(tableProperty.getProperties(), olapTable.getTableProperty().getProperties());
Assert.assertFalse(tableProperty.getDynamicPartitionProperty().isExist());
+ Assert.assertFalse(olapTable.isColocateTable());
// restore with dynamic partition keys
properties = Maps.newHashMap();
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 02/11: [improvement](hive) Support hive with HA HDFS. Pass ha configuration through hive create table properties. (#9151)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 80c74b029dddf69f47c7457e995a0f5e825d2dd4
Author: Jibing-Li <64...@users.noreply.github.com>
AuthorDate: Thu May 5 23:43:11 2022 +0800
[improvement](hive) Support hive with HA HDFS. Pass ha configuration through hive create table properties. (#9151)
Doris couldn't resolve the defaultFS of HDFS with HA configuration, so it could query hive table on HA HDFS.
This is because there's no way to send the HA configs to hive external table.
Describe the overview of changes.
Pass the ha configs to hive external table through create table properties.
Usage:
Example of creating hive table with ha configuration properties:
CREATE TABLE region (
r_regionkey integer NOT NULL,
r_name char(25) NOT NULL,
r_comment varchar(152)
) engine=hive properties
("database"="default",
"table"="region",
"hive.metastore.uris"="thrift://172.21.16.11:7004",
"dfs.nameservices"="hacluster",
"dfs.ha.namenodes.hacluster"="3,4",
"dfs.namenode.rpc-address.hacluster.3"="192.168.0.93:8020",
"dfs.namenode.rpc-address.hacluster.4"="172.21.16.11:8020", "dfs.client.failover.proxy.provider.hacluster"="org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider");
---
.../doris/catalog/HiveMetaStoreClientHelper.java | 19 +++++++++++++++----
.../main/java/org/apache/doris/catalog/HiveTable.java | 13 +++++++++++++
.../java/org/apache/doris/common/FeConstants.java | 6 ++++++
.../java/org/apache/doris/planner/BrokerScanNode.java | 18 +++++++++++++++---
4 files changed, 49 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
index 5dcfd1273b..c378eecf24 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveMetaStoreClientHelper.java
@@ -66,6 +66,7 @@ import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Date;
import java.util.List;
+import java.util.Map;
import java.util.Stack;
/**
@@ -183,10 +184,10 @@ public class HiveMetaStoreClientHelper {
} finally {
client.close();
}
- remoteIterators = getRemoteIterator(hivePartitions);
+ remoteIterators = getRemoteIterator(hivePartitions, hiveTable.getHiveProperties());
} else {
// hive non-partitioned table, get file iterator from table sd info
- remoteIterators = getRemoteIterator(remoteHiveTbl);
+ remoteIterators = getRemoteIterator(remoteHiveTbl, hiveTable.getHiveProperties());
}
String hdfsUrl = "";
@@ -219,9 +220,14 @@ public class HiveMetaStoreClientHelper {
return hdfsUrl;
}
- private static List<RemoteIterator<LocatedFileStatus>> getRemoteIterator(List<Partition> partitions) throws DdlException {
+ private static List<RemoteIterator<LocatedFileStatus>> getRemoteIterator(List<Partition> partitions, Map<String, String> properties) throws DdlException {
List<RemoteIterator<LocatedFileStatus>> iterators = new ArrayList<>();
Configuration configuration = new Configuration(false);
+ for (Map.Entry<String, String> entry : properties.entrySet()) {
+ if (!entry.getKey().equals(HiveTable.HIVE_METASTORE_URIS)) {
+ configuration.set(entry.getKey(), entry.getValue());
+ }
+ }
for (Partition p : partitions) {
String location = p.getSd().getLocation();
org.apache.hadoop.fs.Path path = new org.apache.hadoop.fs.Path(location);
@@ -236,9 +242,14 @@ public class HiveMetaStoreClientHelper {
return iterators;
}
- private static List<RemoteIterator<LocatedFileStatus>> getRemoteIterator(Table table) throws DdlException {
+ private static List<RemoteIterator<LocatedFileStatus>> getRemoteIterator(Table table, Map<String, String> properties) throws DdlException {
List<RemoteIterator<LocatedFileStatus>> iterators = new ArrayList<>();
Configuration configuration = new Configuration(false);
+ for (Map.Entry<String, String> entry : properties.entrySet()) {
+ if (!entry.getKey().equals(HiveTable.HIVE_METASTORE_URIS)) {
+ configuration.set(entry.getKey(), entry.getValue());
+ }
+ }
String location = table.getSd().getLocation();
org.apache.hadoop.fs.Path path = new org.apache.hadoop.fs.Path(location);
try {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveTable.java
index ff4adc001f..d418e47307 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/HiveTable.java
@@ -29,6 +29,7 @@ import com.google.common.collect.Maps;
import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
+import java.util.Iterator;
import java.util.List;
import java.util.Map;
@@ -42,6 +43,7 @@ public class HiveTable extends Table {
private static final String HIVE_DB = "database";
private static final String HIVE_TABLE = "table";
public static final String HIVE_METASTORE_URIS = "hive.metastore.uris";
+ public static final String HIVE_HDFS_PREFIX = "dfs";
private String hiveDb;
private String hiveTable;
@@ -100,6 +102,17 @@ public class HiveTable extends Table {
copiedProps.remove(HIVE_METASTORE_URIS);
hiveProperties.put(HIVE_METASTORE_URIS, hiveMetastoreUris);
+ if (!copiedProps.isEmpty()) {
+ Iterator<Map.Entry<String, String>> iter = copiedProps.entrySet().iterator();
+ while(iter.hasNext()) {
+ Map.Entry<String, String> entry = iter.next();
+ if (entry.getKey().startsWith(HIVE_HDFS_PREFIX)) {
+ hiveProperties.put(entry.getKey(), entry.getValue());
+ iter.remove();
+ }
+ }
+ }
+
if (!copiedProps.isEmpty()) {
throw new DdlException("Unknown table properties: " + copiedProps.toString());
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java b/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java
index 54b755c1e8..715eb0e6f9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/FeConstants.java
@@ -57,4 +57,10 @@ public class FeConstants {
public static String null_string = "\\N";
public static long tablet_checker_interval_ms = 20 * 1000L;
+
+ public static String csv = "csv";
+ public static String csv_with_names = "csv_with_names";
+ public static String csv_with_names_and_types = "csv_with_names_and_types";
+
+ public static String text = "text";
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java
index bfab5124a9..499ef83728 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/BrokerScanNode.java
@@ -36,6 +36,7 @@ import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.UserException;
import org.apache.doris.common.util.BrokerUtil;
+import org.apache.doris.common.FeConstants;
import org.apache.doris.load.BrokerFileGroup;
import org.apache.doris.load.Load;
import org.apache.doris.load.loadv2.LoadTask;
@@ -409,7 +410,9 @@ public class BrokerScanNode extends LoadScanNode {
return TFileFormatType.FORMAT_ORC;
} else if (fileFormat.toLowerCase().equals("json")) {
return TFileFormatType.FORMAT_JSON;
- } else if (fileFormat.toLowerCase().equals("csv")) {
+ } else if (fileFormat.toLowerCase().equals(FeConstants.csv)
+ // TODO: Add TEXTFILE to TFileFormatType to Support hive text file format.
+ || fileFormat.toLowerCase().equals(FeConstants.text)) {
return TFileFormatType.FORMAT_CSV_PLAIN;
} else {
throw new UserException("Not supported file format: " + fileFormat);
@@ -483,7 +486,11 @@ public class BrokerScanNode extends LoadScanNode {
} else {
TBrokerRangeDesc rangeDesc = createBrokerRangeDesc(curFileOffset, fileStatus, formatType,
leftBytes, columnsFromPath, numberOfColumnsFromFile, brokerDesc);
- rangeDesc.setHdfsParams(tHdfsParams);
+ if (rangeDesc.hdfs_params != null && rangeDesc.hdfs_params.getFsName() == null) {
+ rangeDesc.hdfs_params.setFsName(fsName);
+ } else if (rangeDesc.hdfs_params == null) {
+ rangeDesc.setHdfsParams(tHdfsParams);
+ }
rangeDesc.setReadByColumnDef(true);
brokerScanRange(curLocations).addToRanges(rangeDesc);
curFileOffset = 0;
@@ -506,7 +513,12 @@ public class BrokerScanNode extends LoadScanNode {
rangeDesc.setNumAsString(context.fileGroup.isNumAsString());
rangeDesc.setReadJsonByLine(context.fileGroup.isReadJsonByLine());
}
- rangeDesc.setHdfsParams(tHdfsParams);
+ if (rangeDesc.hdfs_params != null && rangeDesc.hdfs_params.getFsName() == null) {
+ rangeDesc.hdfs_params.setFsName(fsName);
+ } else if (rangeDesc.hdfs_params == null) {
+ rangeDesc.setHdfsParams(tHdfsParams);
+ }
+
rangeDesc.setReadByColumnDef(true);
brokerScanRange(curLocations).addToRanges(rangeDesc);
curFileOffset = 0;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 09/11: [fix](function) handle merge in window_funnel_init and add test (#9338)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 20c3744d683fe000a032b317919642d426149f53
Author: dataroaring <98...@users.noreply.github.com>
AuthorDate: Tue May 3 22:37:06 2022 +0800
[fix](function) handle merge in window_funnel_init and add test (#9338)
---
be/src/exprs/aggregate_functions.cpp | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/be/src/exprs/aggregate_functions.cpp b/be/src/exprs/aggregate_functions.cpp
index 52883d04d9..506a87a821 100644
--- a/be/src/exprs/aggregate_functions.cpp
+++ b/be/src/exprs/aggregate_functions.cpp
@@ -2499,9 +2499,10 @@ void AggregateFunctions::window_funnel_init(FunctionContext* ctx, StringVal* dst
WindowFunnelState* state = new WindowFunnelState();
dst->ptr = (uint8_t*)state;
// constant args at index 0 and 1
- DCHECK(ctx->is_arg_constant(0));
- BigIntVal* window = reinterpret_cast<BigIntVal*>(ctx->get_constant_arg(0));
- state->window = window->val;
+ if (ctx->is_arg_constant(0)) {
+ BigIntVal* window = reinterpret_cast<BigIntVal*>(ctx->get_constant_arg(0));
+ state->window = window->val;
+ }
// TODO handle mode in the future
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 11/11: [fixbug][metadata] catalog could not load from image (#9364)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 973153fe9be05e28987d76520b9282bf77ce57db
Author: yiguolei <67...@qq.com>
AuthorDate: Thu May 5 09:58:01 2022 +0800
[fixbug][metadata] catalog could not load from image (#9364)
* [fixbug][catalog] catalog could not load from image
* fix ut failed
Co-authored-by: yiguolei <yi...@gmail.com>
---
fe/fe-core/src/main/java/org/apache/doris/alter/AlterJobV2.java | 4 +++-
fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java | 3 +++
2 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJobV2.java b/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJobV2.java
index 764a0b0ae8..1d0f69f704 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/AlterJobV2.java
@@ -58,7 +58,9 @@ public abstract class AlterJobV2 implements Writable {
}
public enum JobType {
- ROLLUP, SCHEMA_CHANGE
+ // Must not remove it or change the order, because catalog depend on it to traverse the image
+ // and load meta data
+ ROLLUP, SCHEMA_CHANGE, DECOMMISSION_BACKEND
}
@SerializedName(value = "type")
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
index a6e658e535..7621d0008c 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
@@ -2114,6 +2114,9 @@ public class Catalog {
alterJobsV2 = this.getMaterializedViewHandler().getAlterJobsV2();
} else if (type == JobType.SCHEMA_CHANGE) {
alterJobsV2 = this.getSchemaChangeHandler().getAlterJobsV2();
+ } else if (type == JobType.DECOMMISSION_BACKEND) {
+ // Load alter job need decommission backend type to load image
+ alterJobsV2 = Maps.newHashMap();
} else {
throw new IOException("Invalid alter job type: " + type.name());
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org