You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2022/05/18 08:34:38 UTC
[incubator-doris] 03/09: [config] Remove some old config and session variable (#9495)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 66b50f11066ac73799651e5decc2ea354387f650
Author: Mingyu Chen <mo...@gmail.com>
AuthorDate: Tue May 17 22:37:11 2022 +0800
[config] Remove some old config and session variable (#9495)
1. Remove session variable "enable_lateral_view"
2. Remove Fe config: enable_materialized_view
3. Remove Fe config: enable_create_sync_job
4. Fe config dynamic_partition_enable is only used for disable dynamic partition scheduler.
---
docs/en/administrator-guide/config/fe_config.md | 26 +---------------------
.../load-data/binlog-load-manual.md | 4 ----
docs/zh-CN/administrator-guide/config/fe_config.md | 26 +---------------------
.../load-data/binlog-load-manual.md | 6 +----
.../doris/analysis/CreateDataSyncJobStmt.java | 6 -----
.../doris/analysis/CreateMaterializedViewStmt.java | 4 ----
.../org/apache/doris/analysis/LateralViewRef.java | 4 ----
.../org/apache/doris/catalog/TableProperty.java | 11 +--------
.../main/java/org/apache/doris/common/Config.java | 12 ----------
.../java/org/apache/doris/qe/SessionVariable.java | 11 ---------
.../java/org/apache/doris/alter/AlterTest.java | 1 -
.../org/apache/doris/alter/RollupJobV2Test.java | 5 ++++-
.../doris/analysis/CreateDataSyncJobStmtTest.java | 6 ++---
.../analysis/CreateMaterializedViewStmtTest.java | 5 ++++-
.../planner/MaterializedViewFunctionTest.java | 2 +-
.../doris/planner/TableFunctionPlanTest.java | 1 -
.../java/org/apache/doris/utframe/DorisAssert.java | 7 ------
17 files changed, 16 insertions(+), 121 deletions(-)
diff --git a/docs/en/administrator-guide/config/fe_config.md b/docs/en/administrator-guide/config/fe_config.md
index 281d841ff5..e68e6a56ee 100644
--- a/docs/en/administrator-guide/config/fe_config.md
+++ b/docs/en/administrator-guide/config/fe_config.md
@@ -437,20 +437,6 @@ MasterOnly：true
As long as one BE is down, Routine Load cannot be automatically restored
-### enable_materialized_view
-
-Default：true
-
-IsMutable：true
-
-MasterOnly：true
-
-This configuration is used to turn on and off the creation of materialized views. If set to true, the function to create a materialized view is enabled. The user can create a materialized view through the `CREATE MATERIALIZED VIEW` command. If set to false, materialized views cannot be created.
-
-If you get an error `The materialized view is coming soon` or `The materialized view is disabled` when creating the materialized view, it means that the configuration is set to false and the function of creating the materialized view is turned off. You can start to create a materialized view by modifying the configuration to true.
-
-This variable is a dynamic configuration, and users can modify the configuration through commands after the FE process starts. You can also modify the FE configuration file and restart the FE to take effect
-
### check_java_version
Default：true
@@ -475,7 +461,7 @@ IsMutable：true
MasterOnly：true
-Whether to enable dynamic partition, enabled by default
+Whether to enable dynamic partition scheduler, enabled by default
### dynamic_partition_check_interval_seconds
@@ -2152,16 +2138,6 @@ Only for Master FE: false
If set to true, the compaction slower replica will be skipped when select get queryable replicas
-### enable_create_sync_job
-
-Enable Mysql data synchronization job function. The default is false, this function is turned off
-
-Default: false
-
-Is it possible to configure dynamically: true
-
-Whether it is a configuration item unique to the Master FE node: true
-
### sync_commit_interval_second
The maximum time interval for committing transactions. If there is still data in the channel that has not been submitted after this time, the consumer will notify the channel to submit the transaction.
diff --git a/docs/en/administrator-guide/load-data/binlog-load-manual.md b/docs/en/administrator-guide/load-data/binlog-load-manual.md
index a13e52000d..6142828408 100644
--- a/docs/en/administrator-guide/load-data/binlog-load-manual.md
+++ b/docs/en/administrator-guide/load-data/binlog-load-manual.md
@@ -478,10 +478,6 @@ You can use `HELP STOP SYNC JOB;`, `HELP PAUSE SYNC JOB`; And `HELP RESUME SYNC
### Fe configuration
The following configuration belongs to the system level configuration of SyncJob. The configuration value can be modified in configuration file fe.conf.
-
-* `enable_create_sync_job`
-
- Turn on the Binlog Load feature. The default value is false. This feature is turned off.
* `sync_commit_interval_second`
diff --git a/docs/zh-CN/administrator-guide/config/fe_config.md b/docs/zh-CN/administrator-guide/config/fe_config.md
index e16e58c365..e082155212 100644
--- a/docs/zh-CN/administrator-guide/config/fe_config.md
+++ b/docs/zh-CN/administrator-guide/config/fe_config.md
@@ -423,20 +423,6 @@ show data （其他用法：HELP SHOW DATA）
只要有一个BE宕机，Routine Load 就无法自动恢复
-### `enable_materialized_view`
-
-默认值：true
-
-是否可以动态配置：true
-
-是否为 Master FE 节点独有的配置项：true
-
-该配置用于开启和关闭创建物化视图功能。如果设置为 true，则创建物化视图功能开启。用户可以通过 `CREATE MATERIALIZED VIEW` 命令创建物化视图。如果设置为 false，则无法创建物化视图。
-
-如果在创建物化视图的时候报错 `The materialized view is coming soon` 或 `The materialized view is disabled` 则说明改配置被设置为了 false，创建物化视图功能关闭了。可以通过修改配置为 true 来启动创建物化视图功能。
-
-该变量为动态配置，用户可以在 FE 进程启动后，通过命令修改配置。也可以通过修改 FE 的配置文件，重启 FE 来生效
-
### `check_java_version`
默认值：true
@@ -461,7 +447,7 @@ Doris 将检查已编译和运行的 Java 版本是否兼容，如果不兼容
是否为 Master FE 节点独有的配置项：true
-是否启用动态分区，默认启用
+是否启用动态分区调度，默认启用
### `dynamic_partition_check_interval_seconds`
@@ -2175,16 +2161,6 @@ load 标签清理器将每隔 `label_clean_interval_second` 运行一次以清
如果设置为true，则在选择可查询副本时，将跳过 compaction 较慢的副本
-### enable_create_sync_job
-
-开启 MySQL 数据同步作业功能。默认是 false，关闭此功能
-
-默认值：false
-
-是否可以动态配置：true
-
-是否为 Master FE 节点独有的配置项：true
-
### sync_commit_interval_second
提交事务的最大时间间隔。若超过了这个时间 channel 中还有数据没有提交，consumer 会通知 channel 提交事务。
diff --git a/docs/zh-CN/administrator-guide/load-data/binlog-load-manual.md b/docs/zh-CN/administrator-guide/load-data/binlog-load-manual.md
index 8862a0a113..22ddf81775 100644
--- a/docs/zh-CN/administrator-guide/load-data/binlog-load-manual.md
+++ b/docs/zh-CN/administrator-guide/load-data/binlog-load-manual.md
@@ -458,10 +458,6 @@ ALTER TABLE canal_test.test1 ENABLE FEATURE "BATCH_DELETE";
下面配置属于数据同步作业的系统级别配置，主要通过修改 fe.conf 来调整配置值。
-* `enable_create_sync_job`
-
- 开启数据同步作业功能。默认为 false，关闭此功能。
-
* `sync_commit_interval_second`
提交事务的最大时间间隔。若超过了这个时间channel中还有数据没有提交，consumer会通知channel提交事务。
@@ -499,4 +495,4 @@ ALTER TABLE canal_test.test1 ENABLE FEATURE "BATCH_DELETE";
4. 为什么数据同步时浮点类型的数据精度在Mysql端和Doris端不一样？
Doris本身浮点类型的精度与Mysql不一样。可以选择用Decimal类型代替。
-
\ No newline at end of file
+
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateDataSyncJobStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateDataSyncJobStmt.java
index 60f02aa44d..7c939e6a41 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateDataSyncJobStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateDataSyncJobStmt.java
@@ -23,7 +23,6 @@ import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.OlapTable;
import org.apache.doris.cluster.ClusterNamespace;
import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.Config;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.UserException;
@@ -79,11 +78,6 @@ public class CreateDataSyncJobStmt extends DdlStmt {
}
dbName = ClusterNamespace.getFullName(analyzer.getClusterName(), dbName);
- if (!Config.enable_create_sync_job) {
- throw new AnalysisException("Mysql sync job is disabled." +
- " Set config 'enable_create_sync_job' = 'true' to enable this feature. ");
- }
-
if (binlogDesc != null) {
binlogDesc.analyze();
dataSyncJobType = binlogDesc.getDataSyncJobType();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java
index 420053358f..b8125417ee 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java
@@ -24,7 +24,6 @@ import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.catalog.Type;
import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.Config;
import org.apache.doris.common.ErrorCode;
import org.apache.doris.common.ErrorReport;
import org.apache.doris.common.FeConstants;
@@ -131,9 +130,6 @@ public class CreateMaterializedViewStmt extends DdlStmt {
@Override
public void analyze(Analyzer analyzer) throws UserException {
- if (!Config.enable_materialized_view) {
- throw new AnalysisException("The materialized view is disabled");
- }
super.analyze(analyzer);
FeNameFormat.checkTableName(mvName);
// TODO(ml): The mv name in from clause should pass the analyze without error.
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/LateralViewRef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/LateralViewRef.java
index 3af8b5481c..4728dc2c60 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/LateralViewRef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/LateralViewRef.java
@@ -62,10 +62,6 @@ public class LateralViewRef extends TableRef {
@Override
public void analyze(Analyzer analyzer) throws UserException {
- if (!analyzer.getContext().getSessionVariable().isEnableLateralView()) {
- throw new AnalysisException("The session variables `enable_lateral_view` is false");
- }
-
if (isAnalyzed) {
return;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java
index 38d85dc4c1..f007df52a2 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/TableProperty.java
@@ -19,8 +19,6 @@ package org.apache.doris.catalog;
import org.apache.doris.analysis.DataSortInfo;
import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.Config;
-import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeMetaVersion;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
@@ -115,14 +113,7 @@ public class TableProperty implements Writable {
return this;
}
- public TableProperty buildDynamicProperty() throws DdlException {
- if (properties.containsKey(DynamicPartitionProperty.ENABLE)
- && Boolean.valueOf(properties.get(DynamicPartitionProperty.ENABLE))
- && !Config.dynamic_partition_enable) {
- throw new DdlException("Could not create table with dynamic partition "
- + "when fe config dynamic_partition_enable is false. "
- + "Please ADMIN SET FRONTEND CONFIG (\"dynamic_partition_enable\" = \"true\") firstly.");
- }
+ public TableProperty buildDynamicProperty() {
executeBuildDynamicProperty();
return this;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/Config.java b/fe/fe-core/src/main/java/org/apache/doris/common/Config.java
index 82552ff2b3..14e827f3ab 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/Config.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/Config.java
@@ -1267,18 +1267,6 @@ public class Config extends ConfigBase {
@ConfField
public static boolean check_java_version = true;
- /**
- * control materialized view
- */
- @ConfField(mutable = true, masterOnly = true)
- public static boolean enable_materialized_view = true;
-
- /**
- * enable create sync job
- */
- @ConfField(mutable = true, masterOnly = true)
- public static boolean enable_create_sync_job = false;
-
/**
* it can't auto-resume routine load job as long as one of the backends is down
*/
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 8bc47eac47..e2aa801283 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -415,9 +415,6 @@ public class SessionVariable implements Serializable, Writable {
@VariableMgr.VarAttr(name = CPU_RESOURCE_LIMIT)
public int cpuResourceLimit = -1;
- @VariableMgr.VarAttr(name = ENABLE_LATERAL_VIEW, needForward = true)
- public boolean enableLateralView = false;
-
@VariableMgr.VarAttr(name = DISABLE_JOIN_REORDER)
private boolean disableJoinReorder = false;
@@ -876,14 +873,6 @@ public class SessionVariable implements Serializable, Writable {
return enableParallelOutfile;
}
- public boolean isEnableLateralView() {
- return enableLateralView;
- }
-
- public void setEnableLateralView(boolean enableLateralView) {
- this.enableLateralView = enableLateralView;
- }
-
public boolean isDisableJoinReorder() {
return disableJoinReorder;
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/alter/AlterTest.java b/fe/fe-core/src/test/java/org/apache/doris/alter/AlterTest.java
index 08569cdc1a..77b0ca7a6f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/alter/AlterTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/alter/AlterTest.java
@@ -65,7 +65,6 @@ public class AlterTest {
public static void beforeClass() throws Exception {
FeConstants.runningUnitTest = true;
FeConstants.default_scheduler_interval_millisecond = 100;
- Config.dynamic_partition_enable = true;
Config.dynamic_partition_check_interval_seconds = 1;
Config.disable_storage_medium_check = true;
UtFrameUtils.createDorisCluster(runningDir);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/alter/RollupJobV2Test.java b/fe/fe-core/src/test/java/org/apache/doris/alter/RollupJobV2Test.java
index 6e6b18536b..39d41054a7 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/alter/RollupJobV2Test.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/alter/RollupJobV2Test.java
@@ -79,6 +79,10 @@ import java.lang.reflect.InvocationTargetException;
import java.util.ArrayList;
import java.util.List;
import java.util.Map;
+import mockit.Expectations;
+import mockit.Mock;
+import mockit.MockUp;
+import mockit.Mocked;
import mockit.Expectations;
import mockit.Mock;
@@ -317,7 +321,6 @@ public class RollupJobV2Test {
@Test
public void testSerializeOfRollupJob(@Mocked CreateMaterializedViewStmt stmt) throws IOException,
AnalysisException {
- Config.enable_materialized_view = true;
// prepare file
File file = new File(fileName);
file.createNewFile();
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateDataSyncJobStmtTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateDataSyncJobStmtTest.java
index de3ef16cf6..6350942130 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateDataSyncJobStmtTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateDataSyncJobStmtTest.java
@@ -21,7 +21,6 @@ import org.apache.doris.catalog.Catalog;
import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.KeysType;
import org.apache.doris.catalog.OlapTable;
-import org.apache.doris.common.Config;
import org.apache.doris.common.UserException;
import org.apache.doris.load.sync.DataSyncJobType;
import org.apache.doris.mysql.privilege.PaloAuth;
@@ -39,6 +38,9 @@ import org.junit.Test;
import java.util.List;
import java.util.Map;
+import mockit.Expectations;
+import mockit.Injectable;
+import mockit.Mocked;
import mockit.Expectations;
import mockit.Injectable;
@@ -93,8 +95,6 @@ public class CreateDataSyncJobStmtTest {
result = catalog;
}
};
-
- Config.enable_create_sync_job = true;
}
@Test
public void testNoDb() {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateMaterializedViewStmtTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateMaterializedViewStmtTest.java
index 50932d5d7d..6a9e550f9c 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateMaterializedViewStmtTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/CreateMaterializedViewStmtTest.java
@@ -39,6 +39,9 @@ import org.junit.Test;
import java.util.ArrayList;
import java.util.List;
+import mockit.Expectations;
+import mockit.Injectable;
+import mockit.Mocked;
import mockit.Expectations;
import mockit.Injectable;
@@ -57,7 +60,7 @@ public class CreateMaterializedViewStmtTest {
@Before
public void initTest() {
- Deencapsulation.setField(Config.class, "enable_materialized_view", true);
+
}
@Test
diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/MaterializedViewFunctionTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/MaterializedViewFunctionTest.java
index 09d28a719e..a980ff9a67 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/planner/MaterializedViewFunctionTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/planner/MaterializedViewFunctionTest.java
@@ -58,7 +58,7 @@ public class MaterializedViewFunctionTest {
FeConstants.runningUnitTest = true;
UtFrameUtils.createDorisCluster(runningDir);
dorisAssert = new DorisAssert();
- dorisAssert.withEnableMV().withDatabase(HR_DB_NAME).useDatabase(HR_DB_NAME);
+ dorisAssert.withDatabase(HR_DB_NAME).useDatabase(HR_DB_NAME);
}
@Before
diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/TableFunctionPlanTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/TableFunctionPlanTest.java
index 15d84f7f39..0480791baf 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/planner/TableFunctionPlanTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/planner/TableFunctionPlanTest.java
@@ -48,7 +48,6 @@ public class TableFunctionPlanTest {
public static void setUp() throws Exception {
UtFrameUtils.createDorisCluster(runningDir);
ctx = UtFrameUtils.createDefaultCtx();
- ctx.getSessionVariable().setEnableLateralView(true);
String createDbStmtStr = "create database db1;";
CreateDbStmt createDbStmt = (CreateDbStmt) UtFrameUtils.parseAndAnalyzeStmt(createDbStmtStr, ctx);
Catalog.getCurrentCatalog().createDb(createDbStmt);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java b/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
index 520352d654..d6021c1dd6 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/utframe/DorisAssert.java
@@ -32,7 +32,6 @@ import org.apache.doris.analysis.StatementBase;
import org.apache.doris.catalog.Catalog;
import org.apache.doris.cluster.ClusterNamespace;
import org.apache.doris.common.AnalysisException;
-import org.apache.doris.common.Config;
import org.apache.doris.common.util.SqlParserUtils;
import org.apache.doris.planner.Planner;
import org.apache.doris.qe.ConnectContext;
@@ -62,12 +61,6 @@ public class DorisAssert {
this.ctx = ctx;
}
- public DorisAssert withEnableMV() {
- ctx.getSessionVariable().setTestMaterializedView(true);
- Config.enable_materialized_view = true;
- return this;
- }
-
public DorisAssert withDatabase(String dbName) throws Exception {
CreateDbStmt createDbStmt =
(CreateDbStmt) UtFrameUtils.parseAndAnalyzeStmt("create database " + dbName + ";", ctx);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org