You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2022/04/28 02:30:00 UTC
[incubator-doris] branch dev-1.0.1 updated (5b6fd1c681 -> 498f30ab8a)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
from 5b6fd1c681 FIX: getChannel -> getChannel() (#9217)
new db4932be00 remove gensrc/proto/palo_internal_service.proto, this removed in #6341 and add back in #6329 by mistake (#9233)
new eefe8ceab0 [improvement](account) support to account management sql (#8849)
new a3c67666c1 [fix](account) use LOG.info instead of LOG.debug (#8911)
new a20439a920 [feature] add `SHOW TABLET STORAGE FORMAT` stmt (#9037)
new 198234e1e4 [fix](planner) fix non-equal out join is not supported (#9156)
new 23bce90f3b [fix](routine-load) Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error (#9164)
new 8c8e1729e3 [Improvement] not print logs to fe.out when fe is running under daemon mode (#9195)
new 12d180a6ec remove some unused code (#9240)
new 498f30ab8a [fix](checkpoint) fix checkpoint failure when reloading new image (#9262)
The 9 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/src/olap/tablet_manager.cpp | 20 +++
be/src/olap/tablet_manager.h | 2 +
be/src/runtime/routine_load/data_consumer_pool.cpp | 6 +
.../routine_load/routine_load_task_executor.cpp | 2 +-
be/src/service/backend_service.cpp | 4 +
be/src/service/backend_service.h | 2 +
bin/start_fe.sh | 1 +
docs/.vuepress/sidebar/en.js | 1 +
docs/.vuepress/sidebar/zh-CN.js | 1 +
.../Account Management/CREATE ROLE.md | 14 +--
.../Account Management/CREATE USER.md | 26 ++--
.../sql-statements/Account Management/DROP ROLE.md | 16 +--
.../sql-statements/Account Management/DROP USER.md | 6 +-
fe/fe-core/src/main/cup/sql_parser.cup | 20 ++-
.../src/main/java/org/apache/doris/PaloFe.java | 3 +
.../org/apache/doris/analysis/CreateRoleStmt.java | 10 ++
.../org/apache/doris/analysis/DropRoleStmt.java | 10 ++
.../org/apache/doris/analysis/DropUserStmt.java | 11 ++
.../org/apache/doris/analysis/ShowTableStmt.java | 2 +
.../java/org/apache/doris/analysis/TableRef.java | 11 +-
.../java/org/apache/doris/catalog/Catalog.java | 2 +-
.../java/org/apache/doris/common/Log4jConfig.java | 14 ++-
.../load/routineload/KafkaRoutineLoadJob.java | 90 +++++++++----
.../java/org/apache/doris/master/Checkpoint.java | 16 ++-
.../org/apache/doris/mysql/privilege/PaloAuth.java | 67 ++++++----
.../java/org/apache/doris/qe/SessionVariable.java | 8 +-
.../java/org/apache/doris/qe/ShowExecutor.java | 57 +++++++--
.../java/org/apache/doris/task/AgentClient.java | 17 +++
.../apache/doris/analysis/ShowTableStmtTest.java | 6 +-
.../org/apache/doris/common/GenericPoolTest.java | 6 +
.../org/apache/doris/mysql/privilege/AuthTest.java | 140 ++++++++++++++++++++-
.../org/apache/doris/planner/QueryPlanTest.java | 34 +++++
.../java/org/apache/doris/qe/VariableMgrTest.java | 2 +
.../apache/doris/utframe/MockedBackendFactory.java | 6 +
gensrc/proto/palo_internal_service.proto | 46 -------
gensrc/thrift/BackendService.thrift | 8 ++
36 files changed, 528 insertions(+), 159 deletions(-)
delete mode 100644 gensrc/proto/palo_internal_service.proto
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 04/09: [feature] add `SHOW TABLET STORAGE FORMAT` stmt (#9037)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit a20439a920e9a77f0c8b9db853ca2c20638224b3
Author: morningman <ch...@baidu.com>
AuthorDate: Thu Apr 28 10:22:07 2022 +0800
[feature] add `SHOW TABLET STORAGE FORMAT` stmt (#9037)
use this stmt to show tablets storage format in be, if verbose is set,
will show detail message of tablet storage format.
e.g.
```
MySQL [(none)]> admin show tablet storage format;
+-----------+---------+---------+
| BackendId | V1Count | V2Count |
+-----------+---------+---------+
| 10002 | 0 | 2867 |
+-----------+---------+---------+
1 row in set (0.003 sec)
MySQL [test_query_qa]> admin show tablet storage format verbose;
+-----------+----------+---------------+
| BackendId | TabletId | StorageFormat |
+-----------+----------+---------------+
| 10002 | 39227 | V2 |
| 10002 | 39221 | V2 |
| 10002 | 39215 | V2 |
| 10002 | 39199 | V2 |
+-----------+----------+---------------+
4 rows in set (0.034 sec)
```
add storage format infomation to show full table statment.
```
MySQL [test_query_qa]> show full tables;
+-------------------------+------------+---------------+
| Tables_in_test_query_qa | Table_type | StorageFormat |
+-------------------------+------------+---------------+
| bigtable | BASE TABLE | V2 |
| test_dup | BASE TABLE | V2 |
| test | BASE TABLE | V2 |
| baseall | BASE TABLE | V2 |
| test_string | BASE TABLE | V2 |
+-------------------------+------------+---------------+
5 rows in set (0.002 sec)
```
---
be/src/olap/tablet_manager.cpp | 20 ++++++++
be/src/olap/tablet_manager.h | 2 +
be/src/service/backend_service.cpp | 4 ++
be/src/service/backend_service.h | 2 +
docs/.vuepress/sidebar/en.js | 1 +
docs/.vuepress/sidebar/zh-CN.js | 1 +
fe/fe-core/src/main/cup/sql_parser.cup | 8 +++
.../org/apache/doris/analysis/ShowTableStmt.java | 2 +
.../java/org/apache/doris/qe/ShowExecutor.java | 57 ++++++++++++++++++----
.../java/org/apache/doris/task/AgentClient.java | 17 +++++++
.../apache/doris/analysis/ShowTableStmtTest.java | 6 +--
.../org/apache/doris/common/GenericPoolTest.java | 6 +++
.../apache/doris/utframe/MockedBackendFactory.java | 6 +++
gensrc/thrift/BackendService.thrift | 8 +++
14 files changed, 128 insertions(+), 12 deletions(-)
diff --git a/be/src/olap/tablet_manager.cpp b/be/src/olap/tablet_manager.cpp
index 31c74c650f..cf75b8209c 100644
--- a/be/src/olap/tablet_manager.cpp
+++ b/be/src/olap/tablet_manager.cpp
@@ -1449,4 +1449,24 @@ void TabletManager::get_tablets_distribution_on_different_disks(
}
}
+void TabletManager::get_all_tablets_storage_format(TCheckStorageFormatResult* result) {
+ DCHECK(result != nullptr);
+ for (const auto& tablets_shard : _tablets_shards) {
+ std::shared_lock rdlock(tablets_shard.lock);
+ for (const auto& item : tablets_shard.tablet_map) {
+ uint64_t tablet_id = item.first;
+ for (auto& tablet : item.second.table_arr) {
+ if (tablet->all_beta()) {
+ result->v2_tablets.push_back(tablet_id);
+ } else {
+ result->v1_tablets.push_back(tablet_id);
+ }
+ break;
+ }
+ }
+ }
+ result->__isset.v1_tablets = true;
+ result->__isset.v2_tablets = true;
+}
+
} // end namespace doris
diff --git a/be/src/olap/tablet_manager.h b/be/src/olap/tablet_manager.h
index 9643aefbf5..90f03fa680 100644
--- a/be/src/olap/tablet_manager.h
+++ b/be/src/olap/tablet_manager.h
@@ -144,6 +144,8 @@ public:
std::map<int64_t, std::map<DataDir*, int64_t>>& tablets_num_on_disk,
std::map<int64_t, std::map<DataDir*, std::vector<TabletSize>>>& tablets_info_on_disk);
+ void get_all_tablets_storage_format(TCheckStorageFormatResult* result);
+
private:
// Add a tablet pointer to StorageEngine
// If force, drop the existing tablet add this new one
diff --git a/be/src/service/backend_service.cpp b/be/src/service/backend_service.cpp
index 05eac43034..97e35e1597 100644
--- a/be/src/service/backend_service.cpp
+++ b/be/src/service/backend_service.cpp
@@ -370,4 +370,8 @@ void BackendService::get_stream_load_record(TStreamLoadRecordResult& result,
void BackendService::clean_trash() {
StorageEngine::instance()->start_trash_sweep(nullptr, true);
}
+
+void BackendService::check_storage_format(TCheckStorageFormatResult& result) {
+ StorageEngine::instance()->tablet_manager()->get_all_tablets_storage_format(&result);
+}
} // namespace doris
diff --git a/be/src/service/backend_service.h b/be/src/service/backend_service.h
index 3c9b3bd13c..7991f59c9f 100644
--- a/be/src/service/backend_service.h
+++ b/be/src/service/backend_service.h
@@ -156,6 +156,8 @@ public:
virtual void clean_trash() override;
+ virtual void check_storage_format(TCheckStorageFormatResult& result) override;
+
private:
Status start_plan_fragment_execution(const TExecPlanFragmentParams& exec_params);
ExecEnv* _exec_env;
diff --git a/docs/.vuepress/sidebar/en.js b/docs/.vuepress/sidebar/en.js
index 2361963495..593b1519b6 100644
--- a/docs/.vuepress/sidebar/en.js
+++ b/docs/.vuepress/sidebar/en.js
@@ -589,6 +589,7 @@ module.exports = [
"ADMIN SHOW REPLICA DISTRIBUTION",
"ADMIN SHOW REPLICA STATUS",
"ADMIN-DIAGNOSE-TABLET",
+ "ADMIN SHOW TABLET STORAGE FORMAT",
"ALTER CLUSTER",
"ALTER SYSTEM",
"CANCEL DECOMMISSION",
diff --git a/docs/.vuepress/sidebar/zh-CN.js b/docs/.vuepress/sidebar/zh-CN.js
index dcb4a87597..8a804841cb 100644
--- a/docs/.vuepress/sidebar/zh-CN.js
+++ b/docs/.vuepress/sidebar/zh-CN.js
@@ -591,6 +591,7 @@ module.exports = [
"ADMIN SHOW REPLICA DISTRIBUTION",
"ADMIN SHOW REPLICA STATUS",
"ADMIN-DIAGNOSE-TABLET",
+ "ADMIN SHOW TABLET STORAGE FORMAT",
"ALTER CLUSTER",
"ALTER SYSTEM",
"CANCEL DECOMMISSION",
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup b/fe/fe-core/src/main/cup/sql_parser.cup
index 3078d8f068..e60e78c0b4 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -5287,6 +5287,14 @@ admin_stmt ::=
{:
RESULT = new AdminDiagnoseTabletStmt(tabletId);
:}
+ | KW_ADMIN KW_TABLET KW_STORAGE KW_FORMAT
+ {:
+ RESULT = new AdminShowTabletStorageFormatStmt(false);
+ :}
+ | KW_ADMIN KW_TABLET KW_STORAGE KW_FORMAT KW_VERBOSE
+ {:
+ RESULT = new AdminShowTabletStorageFormatStmt(true);
+ :}
;
truncate_stmt ::=
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTableStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTableStmt.java
index f81dc9d0d6..41dd44adfd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTableStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowTableStmt.java
@@ -37,6 +37,7 @@ public class ShowTableStmt extends ShowStmt {
private static final Logger LOG = LogManager.getLogger(ShowTableStmt.class);
private static final String NAME_COL_PREFIX = "Tables_in_";
private static final String TYPE_COL = "Table_type";
+ private static final String STORAGE_FORMAT_COL = "StorageFormat";
private static final TableName TABLE_NAME = new TableName(InfoSchemaDb.DATABASE_NAME, "tables");
private String db;
private boolean isVerbose;
@@ -146,6 +147,7 @@ public class ShowTableStmt extends ShowStmt {
new Column(NAME_COL_PREFIX + ClusterNamespace.getNameFromFullName(db), ScalarType.createVarchar(20)));
if (isVerbose) {
builder.addColumn(new Column(TYPE_COL, ScalarType.createVarchar(20)));
+ builder.addColumn(new Column(STORAGE_FORMAT_COL, ScalarType.createVarchar(20)));
}
return builder.build();
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
index 5fb6ade642..4a9d0b7dbb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
@@ -21,6 +21,7 @@ import org.apache.doris.analysis.AdminDiagnoseTabletStmt;
import org.apache.doris.analysis.AdminShowConfigStmt;
import org.apache.doris.analysis.AdminShowReplicaDistributionStmt;
import org.apache.doris.analysis.AdminShowReplicaStatusStmt;
+import org.apache.doris.analysis.AdminShowTabletStorageFormatStmt;
import org.apache.doris.analysis.DescribeStmt;
import org.apache.doris.analysis.HelpStmt;
import org.apache.doris.analysis.PartitionNames;
@@ -158,13 +159,14 @@ import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.system.Backend;
import org.apache.doris.system.Diagnoser;
import org.apache.doris.system.SystemInfoService;
+import org.apache.doris.task.AgentClient;
+import org.apache.doris.thrift.TCheckStorageFormatResult;
import org.apache.doris.thrift.TUnit;
import org.apache.doris.transaction.GlobalTransactionMgr;
import com.google.common.base.Preconditions;
import com.google.common.base.Strings;
import com.google.common.collect.Lists;
-import com.google.common.collect.Maps;
import com.google.common.collect.Sets;
import org.apache.commons.lang3.tuple.Triple;
@@ -183,7 +185,6 @@ import java.util.Collections;
import java.util.Comparator;
import java.util.HashSet;
import java.util.List;
-import java.util.Map;
import java.util.Set;
import java.util.stream.Collectors;
@@ -335,6 +336,8 @@ public class ShowExecutor {
handleShowTableCreation();
} else if (stmt instanceof ShowLastInsertStmt) {
handleShowLastInsert();
+ } else if (stmt instanceof AdminShowTabletStorageFormatStmt) {
+ handleAdminShowTabletStorageFormat();
} else if (stmt instanceof AdminDiagnoseTabletStmt) {
handleAdminDiagnoseTablet();
} else {
@@ -659,7 +662,6 @@ public class ShowExecutor {
ShowTableStmt showTableStmt = (ShowTableStmt) stmt;
List<List<String>> rows = Lists.newArrayList();
Database db = ctx.getCatalog().getDbOrAnalysisException(showTableStmt.getDb());
- Map<String, String> tableMap = Maps.newTreeMap();
PatternMatcher matcher = null;
if (showTableStmt.getPattern() != null) {
matcher = PatternMatcher.createMysqlPattern(showTableStmt.getPattern(),
@@ -675,14 +677,14 @@ public class ShowExecutor {
PrivPredicate.SHOW)) {
continue;
}
- tableMap.put(tbl.getName(), tbl.getMysqlType());
- }
-
- for (Map.Entry<String, String> entry : tableMap.entrySet()) {
if (showTableStmt.isVerbose()) {
- rows.add(Lists.newArrayList(entry.getKey(), entry.getValue()));
+ String storageFormat = "NONE";
+ if (tbl instanceof OlapTable) {
+ storageFormat = ((OlapTable) tbl).getStorageFormat().toString();
+ }
+ rows.add(Lists.newArrayList(tbl.getName(), tbl.getMysqlType(), storageFormat));
} else {
- rows.add(Lists.newArrayList(entry.getKey()));
+ rows.add(Lists.newArrayList(tbl.getName()));
}
}
resultSet = new ShowResultSet(showTableStmt.getMetaData(), rows);
@@ -2132,6 +2134,43 @@ public class ShowExecutor {
resultSet = new ShowResultSet(showMetaData, resultRowSet);
}
+ private void handleAdminShowTabletStorageFormat() throws AnalysisException {
+ List<List<String>> resultRowSet = Lists.newArrayList();
+ for (Backend be : Catalog.getCurrentSystemInfo().getIdToBackend().values()) {
+ if (be.isQueryAvailable() && be.isLoadAvailable()) {
+ AgentClient client = new AgentClient(be.getHost(), be.getBePort());
+ TCheckStorageFormatResult result = client.checkStorageFormat();
+ if (result == null) {
+ throw new AnalysisException("get tablet data from backend: " + be.getId() + "error.");
+ }
+ if (stmt.isVerbose()) {
+ for (long tabletId : result.getV1Tablets()) {
+ List<String> row = new ArrayList<>();
+ row.add(String.valueOf(be.getId()));
+ row.add(String.valueOf(tabletId));
+ row.add("V1");
+ resultRowSet.add(row);
+ }
+ for (long tabletId : result.getV2Tablets()) {
+ List<String> row = new ArrayList<>();
+ row.add(String.valueOf(be.getId()));
+ row.add(String.valueOf(tabletId));
+ row.add("V2");
+ resultRowSet.add(row);
+ }
+ } else {
+ List<String> row = new ArrayList<>();
+ row.add(String.valueOf(be.getId()));
+ row.add(String.valueOf(result.getV1Tablets().size()));
+ row.add(String.valueOf(result.getV2Tablets().size()));
+ resultRowSet.add(row);
+ }
+ }
+ }
+ ShowResultSetMetaData showMetaData = stmt.getMetaData();
+ resultSet = new ShowResultSet(showMetaData, resultRowSet);
+ }
+
private void handleAdminDiagnoseTablet() {
AdminDiagnoseTabletStmt showStmt = (AdminDiagnoseTabletStmt) stmt;
List<List<String>> resultRowSet = Diagnoser.diagnoseTablet(showStmt.getTabletId());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/task/AgentClient.java b/fe/fe-core/src/main/java/org/apache/doris/task/AgentClient.java
index a62cb0f2f0..22dce44c3c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/task/AgentClient.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/task/AgentClient.java
@@ -22,6 +22,7 @@ import org.apache.doris.common.Status;
import org.apache.doris.thrift.BackendService;
import org.apache.doris.thrift.TAgentResult;
import org.apache.doris.thrift.TAgentServiceVersion;
+import org.apache.doris.thrift.TCheckStorageFormatResult;
import org.apache.doris.thrift.TMiniLoadEtlStatusRequest;
import org.apache.doris.thrift.TMiniLoadEtlStatusResult;
import org.apache.doris.thrift.TMiniLoadEtlTaskRequest;
@@ -167,6 +168,21 @@ public class AgentClient {
return result;
}
+ public TCheckStorageFormatResult checkStorageFormat() {
+ TCheckStorageFormatResult result = null;
+ LOG.debug("submit make snapshot task.");
+ try {
+ borrowClient();
+ result = client.checkStorageFormat();
+ ok = true;
+ } catch (Exception e) {
+ LOG.warn("checkStorageFormat error", e);
+ } finally {
+ returnClient();
+ }
+ return result;
+ }
+
public void deleteEtlFiles(long dbId, long jobId, String dbName, String label) {
TDeleteEtlFilesRequest request = new TDeleteEtlFilesRequest(TAgentServiceVersion.V1,
new TUniqueId(dbId, jobId), dbName, label);
@@ -197,4 +213,5 @@ public class AgentClient {
ClientPool.backendPool.invalidateObject(address, client);
}
}
+
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowTableStmtTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowTableStmtTest.java
index 1e7a97312f..89c7336610 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowTableStmtTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/ShowTableStmtTest.java
@@ -56,7 +56,7 @@ public class ShowTableStmtTest {
stmt = new ShowTableStmt("abc", true, null);
stmt.analyze(analyzer);
Assert.assertEquals("SHOW FULL TABLES FROM testCluster:abc", stmt.toString());
- Assert.assertEquals(2, stmt.getMetaData().getColumnCount());
+ Assert.assertEquals(3, stmt.getMetaData().getColumnCount());
Assert.assertEquals("Tables_in_abc", stmt.getMetaData().getColumn(0).getName());
Assert.assertEquals("Table_type", stmt.getMetaData().getColumn(1).getName());
@@ -64,7 +64,7 @@ public class ShowTableStmtTest {
stmt.analyze(analyzer);
Assert.assertEquals("bcd", stmt.getPattern());
Assert.assertEquals("SHOW FULL TABLES FROM testCluster:abc LIKE 'bcd'", stmt.toString());
- Assert.assertEquals(2, stmt.getMetaData().getColumnCount());
+ Assert.assertEquals(3, stmt.getMetaData().getColumnCount());
Assert.assertEquals("Tables_in_abc", stmt.getMetaData().getColumn(0).getName());
Assert.assertEquals("Table_type", stmt.getMetaData().getColumn(1).getName());
}
@@ -75,4 +75,4 @@ public class ShowTableStmtTest {
stmt.analyze(AccessTestUtil.fetchEmptyDbAnalyzer());
Assert.fail("No exception throws");
}
-}
\ No newline at end of file
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/GenericPoolTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/GenericPoolTest.java
index 3247e5e09c..feab0c256e 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/common/GenericPoolTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/common/GenericPoolTest.java
@@ -24,6 +24,7 @@ import org.apache.doris.thrift.TAgentResult;
import org.apache.doris.thrift.TAgentTaskRequest;
import org.apache.doris.thrift.TCancelPlanFragmentParams;
import org.apache.doris.thrift.TCancelPlanFragmentResult;
+import org.apache.doris.thrift.TCheckStorageFormatResult;
import org.apache.doris.thrift.TDeleteEtlFilesRequest;
import org.apache.doris.thrift.TDiskTrashInfo;
import org.apache.doris.thrift.TExecPlanFragmentParams;
@@ -242,6 +243,11 @@ public class GenericPoolTest {
public void cleanTrash() throws TException {
// TODO Auto-generated method stub
}
+
+ @Override
+ public TCheckStorageFormatResult checkStorageFormat() throws TException {
+ return new TCheckStorageFormatResult();
+ }
}
@Test
diff --git a/fe/fe-core/src/test/java/org/apache/doris/utframe/MockedBackendFactory.java b/fe/fe-core/src/test/java/org/apache/doris/utframe/MockedBackendFactory.java
index 42dab10411..bc3e7928fd 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/utframe/MockedBackendFactory.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/utframe/MockedBackendFactory.java
@@ -32,6 +32,7 @@ import org.apache.doris.thrift.TBackend;
import org.apache.doris.thrift.TBackendInfo;
import org.apache.doris.thrift.TCancelPlanFragmentParams;
import org.apache.doris.thrift.TCancelPlanFragmentResult;
+import org.apache.doris.thrift.TCheckStorageFormatResult;
import org.apache.doris.thrift.TCloneReq;
import org.apache.doris.thrift.TDeleteEtlFilesRequest;
import org.apache.doris.thrift.TDiskTrashInfo;
@@ -319,6 +320,11 @@ public class MockedBackendFactory {
public void cleanTrash() throws TException {
return;
}
+
+ @Override
+ public TCheckStorageFormatResult checkStorageFormat() throws TException {
+ return new TCheckStorageFormatResult();
+ }
}
// The default Brpc service.
diff --git a/gensrc/thrift/BackendService.thrift b/gensrc/thrift/BackendService.thrift
index 4517076f00..8e534f4677 100644
--- a/gensrc/thrift/BackendService.thrift
+++ b/gensrc/thrift/BackendService.thrift
@@ -113,6 +113,11 @@ struct TDiskTrashInfo {
3: required i64 trash_used_capacity
}
+struct TCheckStorageFormatResult {
+ 1: optional list<i64> v1_tablets;
+ 2: optional list<i64> v2_tablets;
+}
+
service BackendService {
// Called by coord to start asynchronous execution of plan fragment in backend.
// Returns as soon as all incoming data streams have been set up.
@@ -174,4 +179,7 @@ service BackendService {
TStreamLoadRecordResult get_stream_load_record(1: i64 last_stream_record_time);
oneway void clean_trash();
+
+ // check tablet rowset type
+ TCheckStorageFormatResult check_storage_format();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 09/09: [fix](checkpoint) fix checkpoint failure when reloading new image (#9262)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 498f30ab8a9982e96b492e655b3c8f5cb2317fec
Author: Mingyu Chen <mo...@gmail.com>
AuthorDate: Thu Apr 28 09:47:16 2022 +0800
[fix](checkpoint) fix checkpoint failure when reloading new image (#9262)
Introduced from #9011
---
.../main/java/org/apache/doris/master/Checkpoint.java | 16 ++++++++++------
.../test/java/org/apache/doris/qe/VariableMgrTest.java | 2 ++
2 files changed, 12 insertions(+), 6 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/master/Checkpoint.java b/fe/fe-core/src/main/java/org/apache/doris/master/Checkpoint.java
index 2d3c3ac271..4eda3fb9ef 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/master/Checkpoint.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/master/Checkpoint.java
@@ -32,11 +32,11 @@ import org.apache.doris.persist.Storage;
import org.apache.doris.qe.VariableMgr;
import org.apache.doris.system.Frontend;
+import com.google.common.base.Strings;
+
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
-import com.google.common.base.Strings;
-
import java.io.IOException;
import java.io.OutputStream;
import java.net.HttpURLConnection;
@@ -73,12 +73,16 @@ public class Checkpoint extends MasterDaemon {
@Override
protected void runAfterCatalogReady() {
- doCheckpoint();
+ try {
+ doCheckpoint();
+ } catch (CheckpointException e) {
+ LOG.warn("failed to do checkpoint.", e);
+ }
}
// public for unit test, so that we can trigger checkpoint manually.
// DO NOT call it manually outside the unit test.
- public synchronized void doCheckpoint() {
+ public synchronized void doCheckpoint() throws CheckpointException {
long imageVersion = 0;
long checkPointVersion = 0;
Storage storage = null;
@@ -135,6 +139,7 @@ public class Checkpoint extends MasterDaemon {
// If success, do all the following jobs
// If failed, just return
catalog = Catalog.getCurrentCatalog();
+ createStaticFieldForCkpt();
catalog.loadImage(imageDir);
if (MetricRepo.isInit) {
MetricRepo.COUNTER_IMAGE_WRITE_SUCCESS.increase(1L);
@@ -142,12 +147,11 @@ public class Checkpoint extends MasterDaemon {
LOG.info("checkpoint finished save image.{}", replayedJournalId);
} catch (Throwable e) {
exceptionCaught = true;
- e.printStackTrace();
LOG.error("Exception when generate new image file", e);
if (MetricRepo.isInit) {
MetricRepo.COUNTER_IMAGE_WRITE_FAILED.increase(1L);
}
- return;
+ throw new CheckpointException(e.getMessage(), e);
} finally {
// destroy checkpoint catalog, reclaim memory
catalog = null;
diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/VariableMgrTest.java b/fe/fe-core/src/test/java/org/apache/doris/qe/VariableMgrTest.java
index 074ab86af4..38e235ce0f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/qe/VariableMgrTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/qe/VariableMgrTest.java
@@ -179,6 +179,8 @@ public class VariableMgrTest {
// the Catalog.isCheckpointThread() will return true.
Deencapsulation.setField(Catalog.class, "checkpointThreadId", Thread.currentThread().getId());
currentCatalog.getCheckpointer().doCheckpoint();
+ } catch (Throwable e) {
+ Assert.fail(e.getMessage());
} finally {
// Restore the ckptThreadId
Deencapsulation.setField(Catalog.class, "checkpointThreadId", ckptThreadId);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 02/09: [improvement](account) support to account management sql (#8849)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit eefe8ceab074fcdf509db56b4cc43e037c65ccab
Author: Jiading Guo <zi...@gmail.com>
AuthorDate: Fri Apr 8 09:08:08 2022 +0800
[improvement](account) support to account management sql (#8849)
Add [IF EXISTS] support to following statements:
- CREATE [IF NOT EXISTS] USER
- CREATE [IF NOT EXISTS] ROLE
- DROP [IF EXISTS] USER
- DROP [IF EXISTS] ROLE
---
.../Account Management/CREATE ROLE.md | 14 +--
.../Account Management/CREATE USER.md | 26 ++--
.../sql-statements/Account Management/DROP ROLE.md | 16 +--
.../sql-statements/Account Management/DROP USER.md | 6 +-
fe/fe-core/src/main/cup/sql_parser.cup | 12 +-
.../org/apache/doris/analysis/CreateRoleStmt.java | 10 ++
.../org/apache/doris/analysis/DropRoleStmt.java | 10 ++
.../org/apache/doris/analysis/DropUserStmt.java | 11 ++
.../java/org/apache/doris/catalog/Catalog.java | 2 +-
.../org/apache/doris/mysql/privilege/PaloAuth.java | 67 ++++++----
.../org/apache/doris/mysql/privilege/AuthTest.java | 140 ++++++++++++++++++++-
11 files changed, 254 insertions(+), 60 deletions(-)
diff --git a/docs/zh-CN/sql-reference/sql-statements/Account Management/CREATE ROLE.md b/docs/zh-CN/sql-reference/sql-statements/Account Management/CREATE ROLE.md
index 6b430a2037..f8f4879d1c 100644
--- a/docs/zh-CN/sql-reference/sql-statements/Account Management/CREATE ROLE.md
+++ b/docs/zh-CN/sql-reference/sql-statements/Account Management/CREATE ROLE.md
@@ -5,7 +5,7 @@
}
---
-<!--
+<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
@@ -27,19 +27,19 @@ under the License.
# CREATE ROLE
## description
该语句用户创建一个角色
-
+
语法：
- CREATE ROLE role1;
-
+ CREATE ROLE [IF NOT EXISTS] role1;
+
该语句创建一个无权限的角色，可以后续通过 GRANT 命令赋予该角色权限。
## example
1. 创建一个角色
-
+
CREATE ROLE role1;
-
+
## keyword
CREATE, ROLE
-
+
diff --git a/docs/zh-CN/sql-reference/sql-statements/Account Management/CREATE USER.md b/docs/zh-CN/sql-reference/sql-statements/Account Management/CREATE USER.md
index 2222e987ef..7acb73e05d 100644
--- a/docs/zh-CN/sql-reference/sql-statements/Account Management/CREATE USER.md
+++ b/docs/zh-CN/sql-reference/sql-statements/Account Management/CREATE USER.md
@@ -5,7 +5,7 @@
}
---
-<!--
+<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
@@ -28,40 +28,40 @@ under the License.
## description
Syntax:
-
- CREATE USER user_identity [IDENTIFIED BY 'password'] [DEFAULT ROLE 'role_name']
+
+ CREATE USER [IF NOT EXISTS] user_identity [IDENTIFIED BY 'password'] [DEFAULT ROLE 'role_name']
user_identity:
'user_name'@'host'
-
+
CREATE USER 命令用于创建一个 Doris 用户。在 Doris 中，一个 user_identity 唯一标识一个用户。user_identity 由两部分组成，user_name 和 host，其中 username 为用户名。host 标识用户端连接所在的主机地址。host 部分可以使用 % 进行模糊匹配。如果不指定 host，默认为 '%'，即表示该用户可以从任意 host 连接到 Doris。
-
+
host 部分也可指定为 domain，语法为：'user_name'@['domain']，即使用中括号包围，则 Doris 会认为这个是一个 domain，并尝试解析其 ip 地址。目前仅支持百度内部的 BNS 解析。
-
+
如果指定了角色（ROLE），则会自动将该角色所拥有的权限赋予新创建的这个用户。如果不指定，则该用户默认没有任何权限。指定的 ROLE 必须已经存在。
## example
1. 创建一个无密码用户（不指定 host，则等价于 jack@'%'）
-
+
CREATE USER 'jack';
2. 创建一个有密码用户，允许从 '172.10.1.10' 登陆
-
+
CREATE USER jack@'172.10.1.10' IDENTIFIED BY '123456';
3. 为了避免传递明文，用例2也可以使用下面的方式来创建
-
+
CREATE USER jack@'172.10.1.10' IDENTIFIED BY PASSWORD '*6BB4837EB74329105EE4568DDA7DC67ED2CA2AD9';
-
+
后面加密的内容可以通过PASSWORD()获得到,例如：
-
+
SELECT PASSWORD('123456');
4. 创建一个允许从 '192.168' 子网登陆的用户，同时指定其角色为 example_role
-
+
CREATE USER 'jack'@'192.168.%' DEFAULT ROLE 'example_role';
-
+
5. 创建一个允许从域名 'example_domain' 登陆的用户
CREATE USER 'jack'@['example_domain'] IDENTIFIED BY '12345';
diff --git a/docs/zh-CN/sql-reference/sql-statements/Account Management/DROP ROLE.md b/docs/zh-CN/sql-reference/sql-statements/Account Management/DROP ROLE.md
index ee0cc980bc..d9cf9fca17 100644
--- a/docs/zh-CN/sql-reference/sql-statements/Account Management/DROP ROLE.md
+++ b/docs/zh-CN/sql-reference/sql-statements/Account Management/DROP ROLE.md
@@ -5,7 +5,7 @@
}
---
-<!--
+<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
@@ -27,18 +27,18 @@ under the License.
# DROP ROLE
## description
该语句用户删除一个角色
-
+
语法：
- DROP ROLE role1;
-
+ DROP ROLE [IF EXISTS] role1;
+
删除一个角色，不会影响之前属于该角色的用户的权限。仅相当于将该角色与用户解耦。用户已经从该角色中获取到的权限，不会改变。
-
+
## example
1. 删除一个角色
-
+
DROP ROLE role1;
-
+
## keyword
- DROP, ROLE
+ DROP, ROLE
diff --git a/docs/zh-CN/sql-reference/sql-statements/Account Management/DROP USER.md b/docs/zh-CN/sql-reference/sql-statements/Account Management/DROP USER.md
index dbdebd164a..43e7fc8b12 100644
--- a/docs/zh-CN/sql-reference/sql-statements/Account Management/DROP USER.md
+++ b/docs/zh-CN/sql-reference/sql-statements/Account Management/DROP USER.md
@@ -5,7 +5,7 @@
}
---
-<!--
+<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
@@ -29,7 +29,7 @@ under the License.
Syntax:
- DROP USER 'user_identity'
+ DROP USER [IF EXISTS] 'user_identity'
`user_identity`:
@@ -41,7 +41,7 @@ Syntax:
## example
1. 删除用户 jack@'192.%'
-
+
DROP USER 'jack'@'192.%'
## keyword
diff --git a/fe/fe-core/src/main/cup/sql_parser.cup b/fe/fe-core/src/main/cup/sql_parser.cup
index 529022f94a..3078d8f068 100644
--- a/fe/fe-core/src/main/cup/sql_parser.cup
+++ b/fe/fe-core/src/main/cup/sql_parser.cup
@@ -1309,9 +1309,9 @@ create_stmt ::=
{:
RESULT = new CreateRepositoryStmt(isReadOnly, repoName, storage);
:}
- | KW_CREATE KW_ROLE ident:role
+ | KW_CREATE KW_ROLE opt_if_not_exists:ifNotExists ident:role
{:
- RESULT = new CreateRoleStmt(role);
+ RESULT = new CreateRoleStmt(ifNotExists, role);
:}
| KW_CREATE KW_FILE STRING_LITERAL:fileName opt_db:db KW_PROPERTIES LPAREN key_value_map:properties RPAREN
{:
@@ -2009,9 +2009,9 @@ drop_stmt ::=
RESULT = new DropTableStmt(ifExists, name, force);
:}
/* User */
- | KW_DROP KW_USER user_identity:userId
+ | KW_DROP KW_USER opt_if_exists:ifExists user_identity:userId
{:
- RESULT = new DropUserStmt(userId);
+ RESULT = new DropUserStmt(ifExists, userId);
:}
/* View */
| KW_DROP KW_VIEW opt_if_exists:ifExists table_name:name
@@ -2022,9 +2022,9 @@ drop_stmt ::=
{:
RESULT = new DropRepositoryStmt(repoName);
:}
- | KW_DROP KW_ROLE ident:role
+ | KW_DROP KW_ROLE opt_if_exists:ifExists ident:role
{:
- RESULT = new DropRoleStmt(role);
+ RESULT = new DropRoleStmt(ifExists, role);
:}
| KW_DROP KW_FILE STRING_LITERAL:fileName opt_db:dbName KW_PROPERTIES LPAREN key_value_map:properties RPAREN
{:
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java
index a35e10eb5b..d93632127e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateRoleStmt.java
@@ -28,12 +28,22 @@ import org.apache.doris.qe.ConnectContext;
public class CreateRoleStmt extends DdlStmt {
+ private boolean ifNotExists;
private String role;
public CreateRoleStmt(String role) {
this.role = role;
}
+ public CreateRoleStmt(boolean ifNotExists, String role) {
+ this.ifNotExists = ifNotExists;
+ this.role = role;
+ }
+
+ public boolean isSetIfNotExists() {
+ return ifNotExists;
+ }
+
public String getQualifiedRole() {
return role;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRoleStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRoleStmt.java
index 2852b0a2f1..522aebf4bf 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRoleStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropRoleStmt.java
@@ -28,12 +28,22 @@ import org.apache.doris.qe.ConnectContext;
public class DropRoleStmt extends DdlStmt {
+ private boolean ifExists;
private String role;
public DropRoleStmt(String role) {
this.role = role;
}
+ public DropRoleStmt(boolean ifExists, String role) {
+ this.ifExists = ifExists;
+ this.role = role;
+ }
+
+ public boolean isSetIfExists() {
+ return ifExists;
+ }
+
public String getQualifiedRole() {
return role;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropUserStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropUserStmt.java
index d7c5402eae..9c8453e23d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/DropUserStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/DropUserStmt.java
@@ -29,12 +29,23 @@ import org.apache.doris.qe.ConnectContext;
// drop user cmy <==> drop user cmy@'%'
// drop user cmy@'192.168.1.%'
public class DropUserStmt extends DdlStmt {
+
+ private boolean ifExists;
private UserIdentity userIdent;
public DropUserStmt(UserIdentity userIdent) {
this.userIdent = userIdent;
}
+ public DropUserStmt(boolean ifExists, UserIdentity userIdent) {
+ this.ifExists = ifExists;
+ this.userIdent = userIdent;
+ }
+
+ public boolean isSetIfExists() {
+ return ifExists;
+ }
+
public UserIdentity getUserIdentity() {
return userIdent;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
index f115f4e63c..f3e187065a 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Catalog.java
@@ -6104,7 +6104,7 @@ public class Catalog {
idToDb.remove(infoSchemaDb.getId());
}
- public void replayDropCluster(ClusterInfo info) {
+ public void replayDropCluster(ClusterInfo info) throws DdlException {
tryLock(true);
try {
unprotectDropCluster(info, true/* is replay */);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PaloAuth.java b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PaloAuth.java
index 27fc99289f..ec072bfdec 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PaloAuth.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PaloAuth.java
@@ -554,12 +554,12 @@ public class PaloAuth implements Writable {
// create user
public void createUser(CreateUserStmt stmt) throws DdlException {
- createUserInternal(stmt.getUserIdent(), stmt.getQualifiedRole(), stmt.getPassword(), false);
+ createUserInternal(stmt.getUserIdent(), stmt.getQualifiedRole(), stmt.getPassword(), stmt.isIfNotExist(), false);
}
public void replayCreateUser(PrivInfo privInfo) {
try {
- createUserInternal(privInfo.getUserIdent(), privInfo.getRole(), privInfo.getPasswd(), true);
+ createUserInternal(privInfo.getUserIdent(), privInfo.getRole(), privInfo.getPasswd(), false, true);
} catch (DdlException e) {
LOG.error("should not happen", e);
}
@@ -568,12 +568,12 @@ public class PaloAuth implements Writable {
/*
* Do following steps:
* 1. Check does specified role exist. If not, throw exception.
- * 2. Check does user already exist. If yes, throw exception.
+ * 2. Check does user already exist. If yes && ignoreIfExists, just return. Otherwise, throw exception.
* 3. set password for specified user.
* 4. grant privs of role to user, if role is specified.
*/
private void createUserInternal(UserIdentity userIdent, String roleName, byte[] password,
- boolean isReplay) throws DdlException {
+ boolean ignoreIfExists, boolean isReplay) throws DdlException {
writeLock();
try {
// 1. check if role exist
@@ -584,9 +584,13 @@ public class PaloAuth implements Writable {
throw new DdlException("Role: " + roleName + " does not exist");
}
}
-
+
// 2. check if user already exist
if (userPrivTable.doesUserExist(userIdent)) {
+ if (ignoreIfExists) {
+ LOG.debug("user exists, ignored to create user: {}, is replay: {}", userIdent, isReplay);
+ return;
+ }
throw new DdlException("User " + userIdent + " already exist");
}
@@ -646,22 +650,33 @@ public class PaloAuth implements Writable {
// drop user
public void dropUser(DropUserStmt stmt) throws DdlException {
- dropUserInternal(stmt.getUserIdentity(), false);
+ dropUserInternal(stmt.getUserIdentity(), stmt.isSetIfExists(), false);
}
- public void replayDropUser(UserIdentity userIdent) {
- dropUserInternal(userIdent, true);
+ public void replayDropUser(UserIdentity userIdent) throws DdlException {
+ dropUserInternal(userIdent, false, true);
}
- public void replayOldDropUser(String userName) {
+ public void replayOldDropUser(String userName) throws DdlException {
UserIdentity userIdentity = new UserIdentity(userName, "%");
userIdentity.setIsAnalyzed();
- dropUserInternal(userIdentity, true /* is replay */);
+ dropUserInternal(userIdentity, false /* ignore if non exists */, true /* is replay */);
}
- private void dropUserInternal(UserIdentity userIdent, boolean isReplay) {
+ private void dropUserInternal(UserIdentity userIdent, boolean ignoreIfNonExists, boolean isReplay) throws DdlException {
writeLock();
try {
+ // check if user exists
+ if (!doesUserExist(userIdent)) {
+ if (ignoreIfNonExists) {
+ LOG.info("user non exists, ignored to drop user: {}, is replay: {}",
+ userIdent.getQualifiedUser(), isReplay);
+ return;
+ }
+ throw new DdlException(String.format("User `%s`@`%s` does not exist.",
+ userIdent.getQualifiedUser(), userIdent.getHost()));
+ }
+
// we don't check if user exists
userPrivTable.dropUser(userIdent);
dbPrivTable.dropUser(userIdent);
@@ -1047,21 +1062,26 @@ public class PaloAuth implements Writable {
// create role
public void createRole(CreateRoleStmt stmt) throws DdlException {
- createRoleInternal(stmt.getQualifiedRole(), false);
+ createRoleInternal(stmt.getQualifiedRole(), stmt.isSetIfNotExists(), false);
}
public void replayCreateRole(PrivInfo info) {
try {
- createRoleInternal(info.getRole(), true);
+ createRoleInternal(info.getRole(), false, true);
} catch (DdlException e) {
LOG.error("should not happened", e);
}
}
- private void createRoleInternal(String role, boolean isReplay) throws DdlException {
+ private void createRoleInternal(String role, boolean ignoreIfExists, boolean isReplay) throws DdlException {
PaloRole emptyPrivsRole = new PaloRole(role);
writeLock();
try {
+ if (ignoreIfExists && roleManager.getRole(role) != null) {
+ LOG.info("role exists, ignored to create role: {}, is replay: {}", role, isReplay);
+ return;
+ }
+
roleManager.addRole(emptyPrivsRole, true /* err on exist */);
if (!isReplay) {
@@ -1076,20 +1096,25 @@ public class PaloAuth implements Writable {
// drop role
public void dropRole(DropRoleStmt stmt) throws DdlException {
- dropRoleInternal(stmt.getQualifiedRole(), false);
+ dropRoleInternal(stmt.getQualifiedRole(), stmt.isSetIfExists(), false);
}
public void replayDropRole(PrivInfo info) {
try {
- dropRoleInternal(info.getRole(), true);
+ dropRoleInternal(info.getRole(), false, true);
} catch (DdlException e) {
LOG.error("should not happened", e);
}
}
- private void dropRoleInternal(String role, boolean isReplay) throws DdlException {
+ private void dropRoleInternal(String role, boolean ignoreIfNonExists, boolean isReplay) throws DdlException {
writeLock();
try {
+ if (ignoreIfNonExists && roleManager.getRole(role) == null) {
+ LOG.info("role non exists, ignored to drop role: {}, is replay: {}", role, isReplay);
+ return;
+ }
+
roleManager.dropRole(role, true /* err on non exist */);
if (!isReplay) {
@@ -1429,13 +1454,13 @@ public class PaloAuth implements Writable {
}
}
- public void dropUserOfCluster(String clusterName, boolean isReplay) {
+ public void dropUserOfCluster(String clusterName, boolean isReplay) throws DdlException {
writeLock();
try {
Set<UserIdentity> allUserIdents = getAllUserIdents(true);
for (UserIdentity userIdent : allUserIdents) {
if (userIdent.getQualifiedUser().startsWith(clusterName)) {
- dropUserInternal(userIdent, isReplay);
+ dropUserInternal(userIdent, false, isReplay);
}
}
} finally {
@@ -1478,10 +1503,10 @@ public class PaloAuth implements Writable {
try {
UserIdentity rootUser = new UserIdentity(ROOT_USER, "%");
rootUser.setIsAnalyzed();
- createUserInternal(rootUser, PaloRole.OPERATOR_ROLE, new byte[0], true /* is replay */);
+ createUserInternal(rootUser, PaloRole.OPERATOR_ROLE, new byte[0], false /* ignore if exists */, true /* is replay */);
UserIdentity adminUser = new UserIdentity(ADMIN_USER, "%");
adminUser.setIsAnalyzed();
- createUserInternal(adminUser, PaloRole.ADMIN_ROLE, new byte[0], true /* is replay */);
+ createUserInternal(adminUser, PaloRole.ADMIN_ROLE, new byte[0], false /* ignore if exists */, true /* is replay */);
} catch (DdlException e) {
LOG.error("should not happened", e);
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/AuthTest.java b/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/AuthTest.java
index f05a341a3d..91182dc4cf 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/AuthTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/mysql/privilege/AuthTest.java
@@ -161,6 +161,43 @@ public class AuthTest {
Assert.fail();
}
+ // 1.1 create cmy@% again with IF NOT EXISTS
+ userIdentity = new UserIdentity("cmy", "%");
+ userDesc = new UserDesc(userIdentity, "54321", true);
+ createUserStmt = new CreateUserStmt(true, userDesc, null);
+ try {
+ createUserStmt.analyze(analyzer);
+ } catch (UserException e) {
+ e.printStackTrace();
+ Assert.fail();
+ }
+
+ try {
+ auth.createUser(createUserStmt);
+ } catch (DdlException e) {
+ Assert.fail();
+ }
+
+ // 1.2 create cmy@% again without IF NOT EXISTS
+ userIdentity = new UserIdentity("cmy", "%");
+ userDesc = new UserDesc(userIdentity, "54321", true);
+ createUserStmt = new CreateUserStmt(false, userDesc, null);
+ try {
+ createUserStmt.analyze(analyzer);
+ } catch (UserException e) {
+ e.printStackTrace();
+ Assert.fail();
+ }
+
+ boolean hasException = false;
+ try {
+ auth.createUser(createUserStmt);
+ } catch (DdlException e) {
+ e.printStackTrace();
+ hasException = true;
+ }
+ Assert.assertTrue(hasException);
+
// 2. check if cmy from specified ip can access to palo
List<UserIdentity> currentUser = Lists.newArrayList();
Assert.assertTrue(auth.checkPlainPassword(SystemInfoService.DEFAULT_CLUSTER + ":cmy", "192.168.0.1", "12345",
@@ -205,7 +242,7 @@ public class AuthTest {
Assert.fail();
}
- boolean hasException = false;
+ hasException = false;
try {
auth.createUser(createUserStmt);
} catch (DdlException e) {
@@ -769,6 +806,40 @@ public class AuthTest {
Assert.fail();
}
+ // 24.1 create role again with IF NOT EXISTS
+ roleStmt = new CreateRoleStmt(true, "role1");
+ try {
+ roleStmt.analyze(analyzer);
+ } catch (UserException e1) {
+ e1.printStackTrace();
+ Assert.fail();
+ }
+
+ try {
+ auth.createRole(roleStmt);
+ } catch (DdlException e1) {
+ e1.printStackTrace();
+ Assert.fail();
+ }
+
+ // 24.2 create role again without IF NOT EXISTS
+ roleStmt = new CreateRoleStmt(false, "role1");
+ try {
+ roleStmt.analyze(analyzer);
+ } catch (UserException e1) {
+ e1.printStackTrace();
+ Assert.fail();
+ }
+
+ hasException = false;
+ try {
+ auth.createRole(roleStmt);
+ } catch (DdlException e1) {
+ e1.printStackTrace();
+ hasException = true;
+ }
+ Assert.assertTrue(hasException);
+
// 25. grant auth to non exist role, will create this new role
privileges = Lists.newArrayList(AccessPrivilege.DROP_PRIV, AccessPrivilege.SELECT_PRIV);
grantStmt = new GrantStmt(null, "role2", new TablePattern("*", "*"), privileges);
@@ -917,6 +988,40 @@ public class AuthTest {
Assert.assertFalse(auth.checkDbPriv(currentUser2.get(0), SystemInfoService.DEFAULT_CLUSTER + ":db4",
PrivPredicate.DROP));
+ // 31.1 drop role again with IF EXISTS
+ dropRoleStmt = new DropRoleStmt(true, "role1");
+ try {
+ dropRoleStmt.analyze(analyzer);
+ } catch (UserException e) {
+ e.printStackTrace();
+ Assert.fail();
+ }
+
+ try {
+ auth.dropRole(dropRoleStmt);
+ } catch (DdlException e) {
+ e.printStackTrace();
+ Assert.fail();
+ }
+
+ // 31.2 drop role again without IF EXISTS
+ dropRoleStmt = new DropRoleStmt(false, "role1");
+ try {
+ dropRoleStmt.analyze(analyzer);
+ } catch (UserException e) {
+ e.printStackTrace();
+ Assert.fail();
+ }
+
+ hasException = false;
+ try {
+ auth.dropRole(dropRoleStmt);
+ } catch (DdlException e) {
+ e.printStackTrace();
+ hasException = true;
+ }
+ Assert.assertTrue(hasException);
+
// 32. drop user cmy@"%"
DropUserStmt dropUserStmt = new DropUserStmt(new UserIdentity("cmy", "%"));
try {
@@ -936,6 +1041,39 @@ public class AuthTest {
Assert.assertTrue(auth.checkPlainPassword(SystemInfoService.DEFAULT_CLUSTER + ":zhangsan", "192.168.0.1",
"12345", null));
+ // 32.1 drop user cmy@"%" again with IF EXISTS
+ dropUserStmt = new DropUserStmt(true, new UserIdentity("cmy", "%"));
+ try {
+ dropUserStmt.analyze(analyzer);
+ } catch (UserException e) {
+ e.printStackTrace();
+ Assert.fail();
+ }
+
+ try {
+ auth.dropUser(dropUserStmt);
+ } catch (DdlException e) {
+ Assert.fail();
+ }
+
+ // 32.2 drop user cmy@"%" again without IF EXISTS
+ dropUserStmt = new DropUserStmt(false, new UserIdentity("cmy", "%"));
+ try {
+ dropUserStmt.analyze(analyzer);
+ } catch (UserException e) {
+ e.printStackTrace();
+ Assert.fail();
+ }
+
+ hasException = false;
+ try {
+ auth.dropUser(dropUserStmt);
+ } catch (DdlException e) {
+ e.printStackTrace();
+ hasException = true;
+ }
+ Assert.assertTrue(hasException);
+
// 33. drop user zhangsan@"192.%"
dropUserStmt = new DropUserStmt(new UserIdentity("zhangsan", "192.%"));
try {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 03/09: [fix](account) use LOG.info instead of LOG.debug (#8911)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit a3c67666c124c64a37e421ea989e02e08cd32e73
Author: Jiading Guo <zi...@gmail.com>
AuthorDate: Sat Apr 9 19:18:13 2022 +0800
[fix](account) use LOG.info instead of LOG.debug (#8911)
This complements (#8849)
---
fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PaloAuth.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PaloAuth.java b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PaloAuth.java
index ec072bfdec..309d5b50a6 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PaloAuth.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/mysql/privilege/PaloAuth.java
@@ -588,7 +588,7 @@ public class PaloAuth implements Writable {
// 2. check if user already exist
if (userPrivTable.doesUserExist(userIdent)) {
if (ignoreIfExists) {
- LOG.debug("user exists, ignored to create user: {}, is replay: {}", userIdent, isReplay);
+ LOG.info("user exists, ignored to create user: {}, is replay: {}", userIdent, isReplay);
return;
}
throw new DdlException("User " + userIdent + " already exist");
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 05/09: [fix](planner) fix non-equal out join is not supported (#9156)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 198234e1e4d2e9f45f262b65e20710db86762e02
Author: shee <13...@users.noreply.github.com>
AuthorDate: Wed Apr 27 08:19:13 2022 -0700
[fix](planner) fix non-equal out join is not supported (#9156)
---
.../java/org/apache/doris/analysis/TableRef.java | 11 ++++++-
.../org/apache/doris/planner/QueryPlanTest.java | 34 ++++++++++++++++++++++
2 files changed, 44 insertions(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/TableRef.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/TableRef.java
index 7e9a509002..a911d27227 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/TableRef.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/TableRef.java
@@ -215,6 +215,7 @@ public class TableRef implements ParseNode, Writable {
return null;
}
+
public JoinOperator getJoinOp() {
// if it's not explicitly set, we're doing an inner join
return (joinOp == null ? JoinOperator.INNER_JOIN : joinOp);
@@ -572,7 +573,15 @@ public class TableRef implements ParseNode, Writable {
public void rewriteExprs(ExprRewriter rewriter, Analyzer analyzer)
throws AnalysisException {
Preconditions.checkState(isAnalyzed);
- if (onClause != null) onClause = rewriter.rewrite(onClause, analyzer, ExprRewriter.ClauseType.ON_CLAUSE);
+ if (onClause != null) {
+ Expr expr = onClause.clone();
+ onClause = rewriter.rewrite(onClause, analyzer, ExprRewriter.ClauseType.ON_CLAUSE);
+ if (joinOp.isOuterJoin() || joinOp.isSemiAntiJoin()) {
+ if (onClause instanceof BoolLiteral && !((BoolLiteral) onClause).getValue()) {
+ onClause = expr;
+ }
+ }
+ }
}
private String joinOpToSql() {
diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/QueryPlanTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/QueryPlanTest.java
index 20a5c7c704..df97fbc845 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/planner/QueryPlanTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/planner/QueryPlanTest.java
@@ -2119,4 +2119,38 @@ public class QueryPlanTest {
String explainString = UtFrameUtils.getSQLPlanOrErrorMsg(connectContext, sql);
Assert.assertTrue(explainString.contains("1 | 10 | 1 | 1 | 1"));
}
+
+ @Test
+ public void testOutJoinWithOnFalse() throws Exception {
+ connectContext.setDatabase("default_cluster:test");
+ createTable("create table out_join_1\n" +
+ "(\n" +
+ " k1 int,\n" +
+ " v int\n" +
+ ")\n" +
+ "DISTRIBUTED BY HASH(k1) BUCKETS 10\n" +
+ "PROPERTIES(\"replication_num\" = \"1\");");
+
+ createTable("create table out_join_2\n" +
+ "(\n" +
+ " k1 int,\n" +
+ " v int\n" +
+ ")\n" +
+ "DISTRIBUTED BY HASH(k1) BUCKETS 10\n" +
+ "PROPERTIES(\"replication_num\" = \"1\");");
+
+ String sql = "explain select * from out_join_1 left join out_join_2 on out_join_1.k1 = out_join_2.k1 and 1=2;";
+ String explainString = UtFrameUtils.getSQLPlanOrErrorMsg(connectContext, sql);
+ Assert.assertFalse(explainString.contains("non-equal LEFT OUTER JOIN is not supported"));
+
+ sql = "explain select * from out_join_1 right join out_join_2 on out_join_1.k1 = out_join_2.k1 and 1=2;";
+ explainString = UtFrameUtils.getSQLPlanOrErrorMsg(connectContext, sql);
+ Assert.assertFalse(explainString.contains("non-equal RIGHT OUTER JOIN is not supported"));
+
+ sql = "explain select * from out_join_1 full join out_join_2 on out_join_1.k1 = out_join_2.k1 and 1=2;";
+ explainString = UtFrameUtils.getSQLPlanOrErrorMsg(connectContext, sql);
+ Assert.assertFalse(explainString.contains("non-equal FULL OUTER JOIN is not supported"));
+
+ }
+
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 06/09: [fix](routine-load) Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error (#9164)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 23bce90f3b8f172f4acb20390d5868c97aab148e
Author: xy720 <22...@users.noreply.github.com>
AuthorDate: Wed Apr 27 23:21:17 2022 +0800
[fix](routine-load) Fix bug that new coming routine load tasks are rejected all the time and report TOO_MANY_TASK error (#9164)
```
CREATE ROUTINE LOAD iaas.dws_nat ON dws_nat
WITH APPEND PROPERTIES (
"desired_concurrent_number"="2",
"max_batch_interval" = "20",
"max_batch_rows" = "400000",
"max_batch_size" = "314572800",
"format" = "json",
"max_error_number" = "0"
)
FROM KAFKA (
"kafka_broker_list" = "xxxx:xxxx",
"kafka_topic" = "nat_nsq",
"property.kafka_default_offsets" = "2022-04-19 13:20:00"
);
```
In the create statement example below, you can see
The user didn't specify the custom partitions.
So that 1. Fe will get all kafka partitions from server in routine load's scheduler.
The user set the default offset by datetime.
So that 2. Fe will get kafka offset by time from server in routine load's scheduler.
When 1 is success, meanwhile 2 is failed, the progress of this routine load may not contains any partitions and offsets.
Nevertheless, since newCurrentKafkaPartition which is get by kafka server may be always equal to currentKafkaPartitions,
the wrong progress will never be updated.
---
be/src/runtime/routine_load/data_consumer_pool.cpp | 6 ++
.../routine_load/routine_load_task_executor.cpp | 2 +-
.../load/routineload/KafkaRoutineLoadJob.java | 90 +++++++++++++++-------
3 files changed, 71 insertions(+), 27 deletions(-)
diff --git a/be/src/runtime/routine_load/data_consumer_pool.cpp b/be/src/runtime/routine_load/data_consumer_pool.cpp
index 5b0d9fabbe..5f039982a9 100644
--- a/be/src/runtime/routine_load/data_consumer_pool.cpp
+++ b/be/src/runtime/routine_load/data_consumer_pool.cpp
@@ -68,11 +68,17 @@ Status DataConsumerPool::get_consumer_grp(StreamLoadContext* ctx,
}
DCHECK(ctx->kafka_info);
+ if (ctx->kafka_info->begin_offset.size() == 0) {
+ return Status::InternalError(
+ "PAUSE: The size of begin_offset of task should not be 0.");
+ }
+
std::shared_ptr<KafkaDataConsumerGroup> grp = std::make_shared<KafkaDataConsumerGroup>();
// one data consumer group contains at least one data consumers.
int max_consumer_num = config::max_consumer_num_per_group;
size_t consumer_num = std::min((size_t)max_consumer_num, ctx->kafka_info->begin_offset.size());
+
for (int i = 0; i < consumer_num; ++i) {
std::shared_ptr<DataConsumer> consumer;
RETURN_IF_ERROR(get_consumer(ctx, &consumer));
diff --git a/be/src/runtime/routine_load/routine_load_task_executor.cpp b/be/src/runtime/routine_load/routine_load_task_executor.cpp
index 1c72a52eaa..cce1646b76 100644
--- a/be/src/runtime/routine_load/routine_load_task_executor.cpp
+++ b/be/src/runtime/routine_load/routine_load_task_executor.cpp
@@ -356,7 +356,7 @@ void RoutineLoadTaskExecutor::exec_task(StreamLoadContext* ctx, DataConsumerPool
void RoutineLoadTaskExecutor::err_handler(StreamLoadContext* ctx, const Status& st,
const std::string& err_msg) {
- LOG(WARNING) << err_msg;
+ LOG(WARNING) << err_msg << ", routine load task: " << ctx->brief(true);
ctx->status = st;
if (ctx->need_rollback) {
_exec_env->stream_load_executor()->rollback_txn(ctx);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
index bd67bd1685..f85898f10b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/KafkaRoutineLoadJob.java
@@ -132,6 +132,22 @@ public class KafkaRoutineLoadJob extends RoutineLoadJob {
return TimeUtils.timeStringToLong(this.kafkaDefaultOffSet, timeZone);
}
+ private long convertedDefaultOffsetToLong() {
+ if (this.kafkaDefaultOffSet.isEmpty()) {
+ return KafkaProgress.OFFSET_END_VAL;
+ } else {
+ if (isOffsetForTimes()) {
+ return convertedDefaultOffsetToTimestamp();
+ } else if (this.kafkaDefaultOffSet.equalsIgnoreCase(KafkaProgress.OFFSET_BEGINNING)) {
+ return KafkaProgress.OFFSET_BEGINNING_VAL;
+ } else if (this.kafkaDefaultOffSet.equalsIgnoreCase(KafkaProgress.OFFSET_END)) {
+ return KafkaProgress.OFFSET_END_VAL;
+ } else {
+ return KafkaProgress.OFFSET_END_VAL;
+ }
+ }
+ }
+
@Override
public void prepare() throws UserException {
super.prepare();
@@ -328,6 +344,13 @@ public class KafkaRoutineLoadJob extends RoutineLoadJob {
}
return true;
} else {
+ // if the partitions of currentKafkaPartitions and progress are inconsistent,
+ // We should also update the progress
+ for (Integer kafkaPartition : currentKafkaPartitions) {
+ if (!((KafkaProgress) progress).containsPartition(kafkaPartition)) {
+ return true;
+ }
+ }
return false;
}
} else {
@@ -406,37 +429,52 @@ public class KafkaRoutineLoadJob extends RoutineLoadJob {
}
}
- private void updateNewPartitionProgress() throws LoadException {
+ private void updateNewPartitionProgress() throws UserException {
// update the progress of new partitions
- for (Integer kafkaPartition : currentKafkaPartitions) {
- if (!((KafkaProgress) progress).containsPartition(kafkaPartition)) {
- // if offset is not assigned, start from OFFSET_END
- long beginOffSet = KafkaProgress.OFFSET_END_VAL;
- if (!kafkaDefaultOffSet.isEmpty()) {
- if (isOffsetForTimes()) {
- // get offset by time
- List<Pair<Integer, Long>> offsets = Lists.newArrayList();
- offsets.add(Pair.create(kafkaPartition, convertedDefaultOffsetToTimestamp()));
- offsets = KafkaUtil.getOffsetsForTimes(this.brokerList, this.topic, convertedCustomProperties, offsets);
- Preconditions.checkState(offsets.size() == 1);
- beginOffSet = offsets.get(0).second;
- } else if (kafkaDefaultOffSet.equalsIgnoreCase(KafkaProgress.OFFSET_BEGINNING)) {
- beginOffSet = KafkaProgress.OFFSET_BEGINNING_VAL;
- } else if (kafkaDefaultOffSet.equalsIgnoreCase(KafkaProgress.OFFSET_END)) {
- beginOffSet = KafkaProgress.OFFSET_END_VAL;
- } else {
- beginOffSet = KafkaProgress.OFFSET_END_VAL;
+ try {
+ for (Integer kafkaPartition : currentKafkaPartitions) {
+ if (!((KafkaProgress) progress).containsPartition(kafkaPartition)) {
+ List<Integer> newPartitions = Lists.newArrayList();
+ newPartitions.add(kafkaPartition);
+ List<Pair<Integer, Long>> newPartitionsOffsets = getNewPartitionOffsetsFromDefaultOffset(newPartitions);
+ Preconditions.checkState(newPartitionsOffsets.size() == 1);
+ for (Pair<Integer, Long> partitionOffset : newPartitionsOffsets) {
+ ((KafkaProgress) progress).addPartitionOffset(partitionOffset);
+ if (LOG.isDebugEnabled()) {
+ LOG.debug(new LogBuilder(LogKey.ROUTINE_LOAD_JOB, id)
+ .add("kafka_partition_id", partitionOffset.first)
+ .add("begin_offset", partitionOffset.second)
+ .add("msg", "The new partition has been added in job"));
+ }
}
}
- ((KafkaProgress) progress).addPartitionOffset(Pair.create(kafkaPartition, beginOffSet));
- if (LOG.isDebugEnabled()) {
- LOG.debug(new LogBuilder(LogKey.ROUTINE_LOAD_JOB, id)
- .add("kafka_partition_id", kafkaPartition)
- .add("begin_offset", beginOffSet)
- .add("msg", "The new partition has been added in job"));
- }
+ }
+ } catch (UserException e) {
+ unprotectUpdateState(JobState.PAUSED,
+ new ErrorReason(InternalErrorCode.PARTITIONS_ERR, e.getMessage()), false /* not replay */);
+ throw e;
+ }
+ }
+
+ private List<Pair<Integer, Long>> getNewPartitionOffsetsFromDefaultOffset(List<Integer> newPartitions) throws UserException {
+ List<Pair<Integer, Long>> partitionOffsets = Lists.newArrayList();
+ // get default offset
+ long beginOffset = convertedDefaultOffsetToLong();
+ for (Integer kafkaPartition : newPartitions) {
+ partitionOffsets.add(Pair.create(kafkaPartition, beginOffset));
+ }
+ if (isOffsetForTimes()) {
+ try {
+ partitionOffsets = KafkaUtil.getOffsetsForTimes(this.brokerList, this.topic, convertedCustomProperties, partitionOffsets);
+ } catch (LoadException e) {
+ LOG.warn(new LogBuilder(LogKey.ROUTINE_LOAD_JOB, id)
+ .add("partition:timestamp", Joiner.on(",").join(partitionOffsets))
+ .add("error_msg", "Job failed to fetch current offsets from times with error " + e.getMessage())
+ .build(), e);
+ throw new UserException(e);
}
}
+ return partitionOffsets;
}
@Override
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 07/09: [Improvement] not print logs to fe.out when fe is running under daemon mode (#9195)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 8c8e1729e37a3ce0630a1b09ac2a6c8acaa6b71b
Author: yiguolei <67...@qq.com>
AuthorDate: Mon Apr 25 18:29:29 2022 +0800
[Improvement] not print logs to fe.out when fe is running under daemon mode (#9195)
Co-authored-by: yiguolei <yi...@gmail.com>
---
bin/start_fe.sh | 1 +
fe/fe-core/src/main/java/org/apache/doris/PaloFe.java | 3 +++
.../src/main/java/org/apache/doris/common/Log4jConfig.java | 14 +++++++++++++-
3 files changed, 17 insertions(+), 1 deletion(-)
diff --git a/bin/start_fe.sh b/bin/start_fe.sh
index b9f1df0b8e..37a474ca4b 100755
--- a/bin/start_fe.sh
+++ b/bin/start_fe.sh
@@ -166,6 +166,7 @@ fi
if [ ${RUN_DAEMON} -eq 1 ]; then
nohup $LIMIT $JAVA $final_java_opt org.apache.doris.PaloFe ${HELPER} "$@" >> $LOG_DIR/fe.out 2>&1 < /dev/null &
else
+ export DORIS_LOG_TO_STDERR=1
$LIMIT $JAVA $final_java_opt org.apache.doris.PaloFe ${HELPER} "$@" < /dev/null
fi
diff --git a/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java b/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java
index c8ee60a4e1..2b04cbc90c 100755
--- a/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/PaloFe.java
@@ -65,6 +65,9 @@ public class PaloFe {
// entrance for doris frontend
public static void start(String dorisHomeDir, String pidDir, String[] args) {
+ if (System.getenv("DORIS_LOG_TO_STDERR") != null) {
+ Log4jConfig.foreground = true;
+ }
if (Strings.isNullOrEmpty(dorisHomeDir)) {
System.err.println("env DORIS_HOME is not set.");
return;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java b/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
index 6f2dbf5f81..81ae11feb9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/common/Log4jConfig.java
@@ -97,7 +97,7 @@ public class Log4jConfig extends XmlConfiguration {
" <Root level=\"${sys_log_level}\">\n" +
" <AppenderRef ref=\"Sys\"/>\n" +
" <AppenderRef ref=\"SysWF\" level=\"WARN\"/>\n" +
- " <AppenderRef ref=\"Console\"/>\n" +
+ " <!--REPLACED BY Console Logger-->\n" +
" </Root>\n" +
" <Logger name=\"audit\" level=\"ERROR\" additivity=\"false\">\n" +
" <AppenderRef ref=\"Auditfile\"/>\n" +
@@ -116,6 +116,12 @@ public class Log4jConfig extends XmlConfiguration {
public static String confDir;
// custom conf dir
public static String customConfDir;
+ // Doris uses both system.out and log4j to print log messages.
+ // This variable is used to check whether to add console appender to loggers.
+ // If doris is running under daemon mode, then this variable == false, and console logger will not be added.
+ // If doris is not running under daemon mode, then this variable == true, and console logger will be added to
+ // loggers, all logs will be printed to console.
+ public static boolean foreground = false;
private static void reconfig() throws IOException {
String newXmlConfTemplate = xmlConfTemplate;
@@ -167,6 +173,12 @@ public class Log4jConfig extends XmlConfiguration {
newXmlConfTemplate = newXmlConfTemplate.replaceAll("<!--REPLACED BY AUDIT AND VERBOSE MODULE NAMES-->",
sb.toString());
+ if (foreground) {
+ StringBuilder consoleLogger = new StringBuilder();
+ consoleLogger.append("<AppenderRef ref=\"Console\"/>\n");
+ newXmlConfTemplate = newXmlConfTemplate.replaceAll("<!--REPLACED BY Console Logger-->",
+ consoleLogger.toString());
+ }
Map<String, String> properties = Maps.newHashMap();
properties.put("sys_log_dir", sysLogDir);
properties.put("sys_file_pattern", sysLogRollPattern);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 01/09: remove gensrc/proto/palo_internal_service.proto, this removed in #6341 and add back in #6329 by mistake (#9233)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit db4932be00ddeca1f7346d88cd815ae8c2b11d2f
Author: Zhengguo Yang <ya...@gmail.com>
AuthorDate: Wed Apr 27 08:25:01 2022 +0800
remove gensrc/proto/palo_internal_service.proto, this removed in #6341 and add back in #6329 by mistake (#9233)
---
gensrc/proto/palo_internal_service.proto | 46 --------------------------------
1 file changed, 46 deletions(-)
diff --git a/gensrc/proto/palo_internal_service.proto b/gensrc/proto/palo_internal_service.proto
deleted file mode 100644
index 07da2e37ea..0000000000
--- a/gensrc/proto/palo_internal_service.proto
+++ /dev/null
@@ -1,46 +0,0 @@
-// Licensed to the Apache Software Foundation (ASF) under one
-// or more contributor license agreements. See the NOTICE file
-// distributed with this work for additional information
-// regarding copyright ownership. The ASF licenses this file
-// to you under the Apache License, Version 2.0 (the
-// "License"); you may not use this file except in compliance
-// with the License. You may obtain a copy of the License at
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing,
-// software distributed under the License is distributed on an
-// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-// KIND, either express or implied. See the License for the
-// specific language governing permissions and limitations
-// under the License.
-
-// NOTE(XXX): DEPRECATED, just use to compatiple with old version.
-// Make system can grayscale upgrade
-syntax="proto2";
-
-import "internal_service.proto";
-
-package palo;
-option java_package = "org.apache.doris.proto";
-
-option cc_generic_services = true;
-
-service PInternalService {
- rpc transmit_data(doris.PTransmitDataParams) returns (doris.PTransmitDataResult);
- rpc exec_plan_fragment(doris.PExecPlanFragmentRequest) returns (doris.PExecPlanFragmentResult);
- rpc cancel_plan_fragment(doris.PCancelPlanFragmentRequest) returns (doris.PCancelPlanFragmentResult);
- rpc fetch_data(doris.PFetchDataRequest) returns (doris.PFetchDataResult);
- rpc tablet_writer_open(doris.PTabletWriterOpenRequest) returns (doris.PTabletWriterOpenResult);
- rpc tablet_writer_add_batch(doris.PTabletWriterAddBatchRequest) returns (doris.PTabletWriterAddBatchResult);
- rpc tablet_writer_cancel(doris.PTabletWriterCancelRequest) returns (doris.PTabletWriterCancelResult);
- rpc get_info(doris.PProxyRequest) returns (doris.PProxyResult);
- rpc update_cache(doris.PUpdateCacheRequest) returns (doris.PCacheResponse);
- rpc fetch_cache(doris.PFetchCacheRequest) returns (doris.PFetchCacheResult);
- rpc clear_cache(doris.PClearCacheRequest) returns (doris.PCacheResponse);
-
- rpc merge_filter(doris.PMergeFilterRequest) returns (doris.PMergeFilterResponse);
- rpc apply_filter(doris.PPublishFilterRequest) returns (doris.PPublishFilterResponse);
- rpc fold_constant_expr(doris.PConstantExprRequest) returns (doris.PConstantExprResult);
- rpc transmit_block(doris.PTransmitDataParams) returns (doris.PTransmitDataResult);
-};
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 08/09: remove some unused code (#9240)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch dev-1.0.1
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 12d180a6ecb12cb69219bbcffd9bcdcc03ad5f21
Author: GoGoWen <82...@users.noreply.github.com>
AuthorDate: Wed Apr 27 11:04:16 2022 +0800
remove some unused code (#9240)
---
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 20b1187e84..20fccab3fa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -275,7 +275,7 @@ public class SessionVariable implements Serializable, Writable {
// The number of seconds the server waits for activity on a noninteractive connection before closing it.
@VariableMgr.VarAttr(name = WAIT_TIMEOUT)
- public int waitTimeout = 28800;
+ public int waitTimeoutS = 28800;
// The number of seconds to wait for a block to be written to a connection before aborting the write
@VariableMgr.VarAttr(name = NET_WRITE_TIMEOUT)
@@ -466,7 +466,7 @@ public class SessionVariable implements Serializable, Writable {
}
public int getWaitTimeoutS() {
- return waitTimeout;
+ return waitTimeoutS;
}
public long getSqlMode() {
@@ -551,10 +551,6 @@ public class SessionVariable implements Serializable, Writable {
return interactiveTimeout;
}
- public int getWaitTimeout() {
- return waitTimeout;
- }
-
public int getNetWriteTimeout() {
return netWriteTimeout;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org