You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by mo...@apache.org on 2021/11/08 13:58:18 UTC
[incubator-doris] branch branch-0.15 updated (4719771 -> c9d3010)
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a change to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git.
from 4719771 [Enhance][Load] Reduce the number of segments when loading a large volume data in one batch (#6947)
new 0465b5c [BUG] fix Calc capacityCoefficient mistake #6898 (#6899)
new e1d8078 [Thirdparty] Fix flatbuffers download url error (#6968)
new 5266ba9 [BUG][Schedule] Fix getMixLoadScore error. (#6975)
new 0faff63 [Build] Generate compile_command.json (#6976)
new cc6a6be Fix the SQL execution error caused by tablet not being found due to Colocate join (#7002)
new 8bfc0a0 Fix hadoop load failed when enable batch delete in unique table (#6996)
new 02694af fix issue for JournalEntity (#7005)
new 0a551e4 [refactor] set `forward_to_master` true by default (#7017)
new 1d41162 Union stmt support 'OutFileClause' (#7026)
new 987ca1f [Bug] Fix bug that memtracker in delta writer will be visited before initializd. (#7013)
new c9d3010 [Bug] Change DateTimeValue Memmory Layout To Old (#7022)
The 11 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/CMakeLists.txt | 2 +
be/src/olap/delta_writer.cpp | 5 +
be/src/olap/task/engine_batch_load_task.cpp | 2 +-
be/src/runtime/datetime_value.h | 28 +++---
be/src/runtime/fragment_mgr.cpp | 8 +-
.../runtime/stream_load/stream_load_executor.cpp | 5 +-
be/src/service/internal_service.cpp | 2 +-
docs/en/administrator-guide/variables.md | 2 +-
docs/zh-CN/administrator-guide/variables.md | 2 +-
.../org/apache/doris/analysis/OutFileClause.java | 8 +-
.../java/org/apache/doris/analysis/SelectStmt.java | 2 +-
.../apache/doris/analysis/SetOperationStmt.java | 2 +
.../doris/analysis/ShowRoutineLoadTaskStmt.java | 5 +
.../apache/doris/clone/BackendLoadStatistic.java | 4 +-
.../org/apache/doris/journal/JournalEntity.java | 3 +-
.../src/main/java/org/apache/doris/load/Load.java | 23 +++++
.../doris/load/routineload/RoutineLoadManager.java | 4 +-
.../main/java/org/apache/doris/qe/Coordinator.java | 15 +--
.../java/org/apache/doris/qe/SessionVariable.java | 2 +-
.../apache/doris/task/HadoopLoadPendingTask.java | 4 +-
.../doris/analysis/OutFileClauseFunctionTest.java | 101 +++++++++++++++++++++
.../org/apache/doris/planner/ColocatePlanTest.java | 14 +++
thirdparty/vars.sh | 2 +-
23 files changed, 193 insertions(+), 52 deletions(-)
create mode 100644 fe/fe-core/src/test/java/org/apache/doris/analysis/OutFileClauseFunctionTest.java
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 02/11: [Thirdparty] Fix flatbuffers download url
error (#6968)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit e1d8078c4fb22a6034c0e72233bd3d5573687e18
Author: xuliuzhe <36...@qq.com>
AuthorDate: Tue Nov 2 16:34:17 2021 +0800
[Thirdparty] Fix flatbuffers download url error (#6968)
Change google flatbuffers download URL in thirdparty.var.sh
---
thirdparty/vars.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index 734a656..ea34e57 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -205,7 +205,7 @@ BROTLI_SOURCE="brotli-1.0.9"
BROTLI_MD5SUM="c2274f0c7af8470ad514637c35bcee7d"
# flatbuffers
-FLATBUFFERS_DOWNLOAD="https://github.com/google/flatbuffers/archive/2.0.0.tar.gz"
+FLATBUFFERS_DOWNLOAD="https://github.com/google/flatbuffers/archive/v2.0.0.tar.gz"
FLATBUFFERS_NAME=flatbuffers-2.0.0.tar.gz
FLATBUFFERS_SOURCE=flatbuffers-2.0.0
FLATBUFFERS_MD5SUM="a27992324c3cbf86dd888268a23d17bd"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 01/11: [BUG] fix Calc capacityCoefficient mistake
#6898 (#6899)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 0465b5cbd698a7c0c6a55cbaf55ebf8518326f64
Author: tarepanda1024 <li...@gmail.com>
AuthorDate: Tue Nov 2 16:32:44 2021 +0800
[BUG] fix Calc capacityCoefficient mistake #6898 (#6899)
fix #6898
---
.../src/main/java/org/apache/doris/clone/BackendLoadStatistic.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/clone/BackendLoadStatistic.java b/fe/fe-core/src/main/java/org/apache/doris/clone/BackendLoadStatistic.java
index 2210a3f..5be3ad9 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/BackendLoadStatistic.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/BackendLoadStatistic.java
@@ -183,7 +183,7 @@ public class BackendLoadStatistic {
if (diskInfo.getState() == DiskState.ONLINE) {
// we only collect online disk's capacity
totalCapacityMap.put(medium, totalCapacityMap.getOrDefault(medium, 0L) + diskInfo.getTotalCapacityB());
- totalUsedCapacityMap.put(medium, totalUsedCapacityMap.getOrDefault(medium, 0L) + diskInfo.getDataUsedCapacityB());
+ totalUsedCapacityMap.put(medium, totalUsedCapacityMap.getOrDefault(medium, 0L) + (diskInfo.getTotalCapacityB() - diskInfo.getAvailableCapacityB()));
}
RootPathLoadStatistic pathStatistic = new RootPathLoadStatistic(beId, diskInfo.getRootPath(),
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 10/11: [Bug] Fix bug that memtracker in delta
writer will be visited before initializd. (#7013)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 987ca1fbcf43a5bd05c08180688214dc579a3f38
Author: Mingyu Chen <mo...@gmail.com>
AuthorDate: Sat Nov 6 13:29:49 2021 +0800
[Bug] Fix bug that memtracker in delta writer will be visited before initializd. (#7013)
---
be/src/olap/delta_writer.cpp | 5 +++++
be/src/olap/task/engine_batch_load_task.cpp | 2 +-
be/src/runtime/fragment_mgr.cpp | 8 ++++++--
be/src/service/internal_service.cpp | 2 +-
4 files changed, 13 insertions(+), 4 deletions(-)
diff --git a/be/src/olap/delta_writer.cpp b/be/src/olap/delta_writer.cpp
index afa014b..3c0d2fc 100644
--- a/be/src/olap/delta_writer.cpp
+++ b/be/src/olap/delta_writer.cpp
@@ -343,6 +343,11 @@ OLAPStatus DeltaWriter::cancel() {
}
int64_t DeltaWriter::mem_consumption() const {
+ if (_mem_tracker == nullptr) {
+ // This method may be called before this writer is initialized.
+ // So _mem_tracker may be null.
+ return 0;
+ }
return _mem_tracker->consumption();
}
diff --git a/be/src/olap/task/engine_batch_load_task.cpp b/be/src/olap/task/engine_batch_load_task.cpp
index 8d1b6d2..2b6a95a 100644
--- a/be/src/olap/task/engine_batch_load_task.cpp
+++ b/be/src/olap/task/engine_batch_load_task.cpp
@@ -343,7 +343,7 @@ OLAPStatus EngineBatchLoadTask::_push(const TPushReq& request,
OLAPStatus EngineBatchLoadTask::_delete_data(const TPushReq& request,
std::vector<TTabletInfo>* tablet_info_vec) {
- LOG(INFO) << "begin to process delete data. request=" << ThriftDebugString(request);
+ VLOG_DEBUG << "begin to process delete data. request=" << ThriftDebugString(request);
DorisMetrics::instance()->delete_requests_total->increment(1);
OLAPStatus res = OLAP_SUCCESS;
diff --git a/be/src/runtime/fragment_mgr.cpp b/be/src/runtime/fragment_mgr.cpp
index a8ff209..5cd403d 100644
--- a/be/src/runtime/fragment_mgr.cpp
+++ b/be/src/runtime/fragment_mgr.cpp
@@ -373,7 +373,11 @@ void FragmentExecState::coordinator_callback(const Status& status, RuntimeProfil
try {
coord->reportExecStatus(res, params);
} catch (TTransportException& e) {
- LOG(WARNING) << "Retrying ReportExecStatus: " << e.what();
+ LOG(WARNING) << "Retrying ReportExecStatus. query id: "
+ << print_id(_query_id) << ", instance id: "
+ << print_id(_fragment_instance_id)
+ << " to " << _coord_addr
+ << ", err: " << e.what();
rpc_status = coord.reopen();
if (!rpc_status.ok()) {
@@ -822,7 +826,7 @@ Status FragmentMgr::apply_filter(const PPublishFilterRequest* request, const cha
auto iter = _fragment_map.find(tfragment_instance_id);
if (iter == _fragment_map.end()) {
VLOG_CRITICAL << "unknown.... fragment-id:" << fragment_instance_id;
- return Status::InvalidArgument("fragment-id");
+ return Status::InvalidArgument("fragment-id: " + fragment_instance_id.to_string());
}
fragment_state = iter->second;
}
diff --git a/be/src/service/internal_service.cpp b/be/src/service/internal_service.cpp
index 93dc6b1..3afab34 100644
--- a/be/src/service/internal_service.cpp
+++ b/be/src/service/internal_service.cpp
@@ -312,7 +312,7 @@ void PInternalServiceImpl<T>::apply_filter(::google::protobuf::RpcController* co
VLOG_NOTICE << "rpc apply_filter recv";
Status st = _exec_env->fragment_mgr()->apply_filter(request, attachment.to_string().data());
if (!st.ok()) {
- LOG(WARNING) << "apply filter meet error" << st.to_string();
+ LOG(WARNING) << "apply filter meet error: " << st.to_string();
}
st.to_protobuf(response->mutable_status());
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 06/11: Fix hadoop load failed when enable batch
delete in unique table (#6996)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 8bfc0a05f4f3732e6d1f1179f3f858509f825838
Author: Zhengguo Yang <ya...@gmail.com>
AuthorDate: Fri Nov 5 09:43:28 2021 +0800
Fix hadoop load failed when enable batch delete in unique table (#6996)
---
.../src/main/java/org/apache/doris/load/Load.java | 23 ++++++++++++++++++++++
.../apache/doris/task/HadoopLoadPendingTask.java | 4 +---
2 files changed, 24 insertions(+), 3 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/Load.java b/fe/fe-core/src/main/java/org/apache/doris/load/Load.java
index 9469e4c..4e0d4d4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/Load.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/Load.java
@@ -83,6 +83,7 @@ import org.apache.doris.common.util.MetaLockUtils;
import org.apache.doris.common.util.TimeUtils;
import org.apache.doris.load.FailMsg.CancelType;
import org.apache.doris.load.LoadJob.JobState;
+import org.apache.doris.load.loadv2.LoadTask;
import org.apache.doris.metric.MetricRepo;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.persist.ReplicaPersistInfo;
@@ -581,6 +582,11 @@ public class Load {
// set default timeout
job.setTimeoutSecond(Config.hadoop_load_default_timeout_second);
}
+ for (DataDescription dataDescription : dataDescriptions) {
+ if (dataDescription.getMergeType() != LoadTask.MergeType.APPEND) {
+ throw new DdlException("MERGE OR DELETE is not supported in hadoop load.");
+ }
+ }
} else if (etlJobType == EtlJobType.BROKER) {
if (job.getTimeoutSecond() == 0) {
// set default timeout
@@ -758,6 +764,23 @@ public class Load {
// do nothing
}
+ } else if (!column.isVisible()) {
+ /*
+ * For batch delete table add hidden column __DORIS_DELETE_SIGN__ to columns
+ * eg:
+ * (A, B, C)
+ * ->
+ * (A, B, C) SET (__DORIS_DELETE_SIGN__ = 0)
+ */
+ columnToHadoopFunction.put(column.getName(), Pair.create("default_value", Lists.newArrayList(column.getDefaultValue())));
+ ImportColumnDesc importColumnDesc = null;
+ try {
+ importColumnDesc = new ImportColumnDesc(column.getName(),
+ new FunctionCallExpr("default_value", Arrays.asList(column.getDefaultValueExpr())));
+ } catch (AnalysisException e) {
+ throw new DdlException(e.getMessage());
+ }
+ parsedColumnExprList.add(importColumnDesc);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/task/HadoopLoadPendingTask.java b/fe/fe-core/src/main/java/org/apache/doris/task/HadoopLoadPendingTask.java
index cffd772..e1944eb 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/task/HadoopLoadPendingTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/task/HadoopLoadPendingTask.java
@@ -169,7 +169,7 @@ public class HadoopLoadPendingTask extends LoadPendingTask {
private Map<String, EtlColumn> createEtlColumns(OlapTable table) {
Map<String, EtlColumn> etlColumns = Maps.newHashMap();
- for (Column column : table.getBaseSchema()) {
+ for (Column column : table.getBaseSchema(true)) {
etlColumns.put(column.getName(), new EtlColumn(column));
}
return etlColumns;
@@ -225,7 +225,6 @@ public class HadoopLoadPendingTask extends LoadPendingTask {
}
columnRefs.add(dppColumn);
}
-
// distribution infos
DistributionInfo distributionInfo = partition.getDistributionInfo();
List<String> distributionColumnRefs = Lists.newArrayList();
@@ -266,7 +265,6 @@ public class HadoopLoadPendingTask extends LoadPendingTask {
LOG.warn("unknown distribution type. type: {}", distributionInfo.getType().name());
throw new LoadException("unknown distribution type. type: " + distributionInfo.getType().name());
}
-
etlIndex.setPidKeyCount(keySize);
etlIndex.setColumnRefs(columnRefs);
etlIndices.put(String.valueOf(indexId), etlIndex);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 07/11: fix issue for JournalEntity (#7005)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 02694afad063a4d99b1182eb044029a43ea6914e
Author: GoGoWen <82...@users.noreply.github.com>
AuthorDate: Fri Nov 5 09:45:10 2021 +0800
fix issue for JournalEntity (#7005)
fix a log class incorrect issue in JournalEntity.java.
---
fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java b/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java
index 739a962..03036aa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java
@@ -106,7 +106,7 @@ import java.io.IOException;
// this is the value written to bdb or local edit files. key is an auto-increasing long.
public class JournalEntity implements Writable {
- public static final Logger LOG = LogManager.getLogger(Checkpoint.class);
+ public static final Logger LOG = LogManager.getLogger(JournalEntity.class);
private short opCode;
private Writable data;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 03/11: [BUG][Schedule] Fix getMixLoadScore error.
(#6975)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 5266ba9f4d68251c78cc44cd6fb30a10bb033a4d
Author: Lijia Liu <li...@yeah.net>
AuthorDate: Tue Nov 2 16:36:05 2021 +0800
[BUG][Schedule] Fix getMixLoadScore error. (#6975)
---
.../src/main/java/org/apache/doris/clone/BackendLoadStatistic.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/clone/BackendLoadStatistic.java b/fe/fe-core/src/main/java/org/apache/doris/clone/BackendLoadStatistic.java
index 5be3ad9..005ea1e 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/BackendLoadStatistic.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/BackendLoadStatistic.java
@@ -158,7 +158,7 @@ public class BackendLoadStatistic {
totalLoadScore += getLoadScore(medium);
}
}
- return totalLoadScore / mediumCount == 0 ? 1 : mediumCount;
+ return totalLoadScore / (mediumCount == 0 ? 1 : mediumCount);
}
public void setClazz(TStorageMedium medium, Classification clazz) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 08/11: [refactor] set `forward_to_master` true by
default (#7017)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 0a551e4dd039879c7089f07284900254170ce495
Author: Zhengguo Yang <ya...@gmail.com>
AuthorDate: Sat Nov 6 13:27:26 2021 +0800
[refactor] set `forward_to_master` true by default (#7017)
* ot set forward_to_master true by default
* Update docs/zh-CN/administrator-guide/variables.md
---
be/src/runtime/stream_load/stream_load_executor.cpp | 5 +++--
docs/en/administrator-guide/variables.md | 2 +-
docs/zh-CN/administrator-guide/variables.md | 2 +-
.../main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java | 5 +++++
.../java/org/apache/doris/load/routineload/RoutineLoadManager.java | 4 +++-
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java | 2 +-
6 files changed, 14 insertions(+), 6 deletions(-)
diff --git a/be/src/runtime/stream_load/stream_load_executor.cpp b/be/src/runtime/stream_load/stream_load_executor.cpp
index 87d8de4..c76f356 100644
--- a/be/src/runtime/stream_load/stream_load_executor.cpp
+++ b/be/src/runtime/stream_load/stream_load_executor.cpp
@@ -45,7 +45,8 @@ Status StreamLoadExecutor::execute_plan_fragment(StreamLoadContext* ctx) {
return execute_plan_fragment(ctx, nullptr);
}
-Status StreamLoadExecutor::execute_plan_fragment(StreamLoadContext* ctx, std::shared_ptr<StreamLoadPipe> pipe) {
+Status StreamLoadExecutor::execute_plan_fragment(StreamLoadContext* ctx,
+ std::shared_ptr<StreamLoadPipe> pipe) {
DorisMetrics::instance()->txn_exec_plan_total->increment(1);
// submit this params
#ifndef BE_TEST
@@ -217,7 +218,7 @@ Status StreamLoadExecutor::commit_txn(StreamLoadContext* ctx) {
// rollback this transaction
Status status(result.status);
if (!status.ok()) {
- LOG(WARNING) << "commit transaction failed, errmsg=" << status.get_error_msg()
+ LOG(WARNING) << "commit transaction failed, errmsg=" << status.get_error_msg() << ", "
<< ctx->brief();
if (status.code() == TStatusCode::PUBLISH_TIMEOUT) {
ctx->need_rollback = false;
diff --git a/docs/en/administrator-guide/variables.md b/docs/en/administrator-guide/variables.md
index 815ceef..b57be76 100644
--- a/docs/en/administrator-guide/variables.md
+++ b/docs/en/administrator-guide/variables.md
@@ -195,7 +195,7 @@ Note that the comment must start with /*+ and can only follow the SELECT.
* `forward_to_master`
- The user sets whether to forward some commands to the Master FE node for execution. The default is false, which means no forwarding. There are multiple FE nodes in Doris, one of which is the Master node. Usually users can connect to any FE node for full-featured operation. However, some of detail information can only be obtained from the Master FE node.
+ The user sets whether to forward some commands to the Master FE node for execution. The default is `true`, which means no forwarding. There are multiple FE nodes in Doris, one of which is the Master node. Usually users can connect to any FE node for full-featured operation. However, some of detail information can only be obtained from the Master FE node.
For example, the `SHOW BACKENDS;` command, if not forwarded to the Master FE node, can only see some basic information such as whether the node is alive, and forwarded to the Master FE to obtain more detailed information including the node startup time and the last heartbeat time.
diff --git a/docs/zh-CN/administrator-guide/variables.md b/docs/zh-CN/administrator-guide/variables.md
index 72b6fb7..5c54440 100644
--- a/docs/zh-CN/administrator-guide/variables.md
+++ b/docs/zh-CN/administrator-guide/variables.md
@@ -192,7 +192,7 @@ SELECT /*+ SET_VAR(query_timeout = 1, enable_partition_cache=true) */ sleep(3);
* `forward_to_master`
- 用户设置是否将一些命令转发到 Master FE 节点执行。默认为 false，即不转发。Doris 中存在多个 FE 节点，其中一个为 Master 节点。通常用户可以连接任意 FE 节点进行全功能操作。但部分信息查看指令，只有从 Master FE 节点才能获取详细信息。
+ 用户设置是否将一些show 类命令转发到 Master FE 节点执行。默认为 `true`，即转发。Doris 中存在多个 FE 节点，其中一个为 Master 节点。通常用户可以连接任意 FE 节点进行全功能操作。但部分信息查看指令，只有从 Master FE 节点才能获取详细信息。
如 `SHOW BACKENDS;` 命令，如果不转发到 Master FE 节点，则仅能看到节点是否存活等一些基本信息，而转发到 Master FE 则可以获取包括节点启动时间、最后一次心跳时间等更详细的信息。
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
index d280704..6a7dccc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ShowRoutineLoadTaskStmt.java
@@ -145,4 +145,9 @@ public class ShowRoutineLoadTaskStmt extends ShowStmt {
public static List<String> getTitleNames() {
return TITLE_NAMES;
}
+
+ @Override
+ public RedirectStatus getRedirectStatus() {
+ return RedirectStatus.FORWARD_NO_SYNC;
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
index 4d283e5..dc0806c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/load/routineload/RoutineLoadManager.java
@@ -426,7 +426,9 @@ public class RoutineLoadManager implements Writable {
// 1. Find if the given BE id has available slots
if (previoudBeId != -1L) {
int idleTaskNum = 0;
- if (beIdToConcurrentTasks.containsKey(previoudBeId)) {
+ if (beIdToMaxConcurrentTasks.containsKey(previoudBeId)) {
+ idleTaskNum = 0;
+ } else if (beIdToConcurrentTasks.containsKey(previoudBeId)) {
idleTaskNum = beIdToMaxConcurrentTasks.get(previoudBeId) - beIdToConcurrentTasks.get(previoudBeId);
} else {
idleTaskNum = Config.max_routine_load_task_num_per_be;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
index 859508b..d536f62 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
@@ -312,7 +312,7 @@ public class SessionVariable implements Serializable, Writable {
private boolean enableJoinReorderBasedCost = false;
@VariableMgr.VarAttr(name = FORWARD_TO_MASTER)
- public boolean forwardToMaster = false;
+ public boolean forwardToMaster = true;
@VariableMgr.VarAttr(name = LOAD_MEM_LIMIT)
public long loadMemLimit = 0L;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 04/11: [Build] Generate compile_command.json
(#6976)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 0faff631a12a3c24a2151e851a403ed7b214b47c
Author: flynn <fe...@mails.ucas.ac.cn>
AuthorDate: Tue Nov 2 16:36:35 2021 +0800
[Build] Generate compile_command.json (#6976)
Set cmake to generate compile_commands.json, which is useful for lsp like clangd, cquery, et.
---
be/CMakeLists.txt | 2 ++
1 file changed, 2 insertions(+)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index d8ac46d..9c3631c 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -28,6 +28,8 @@ endif()
project(doris CXX C)
+# Write compile_commands.json
+set(CMAKE_EXPORT_COMPILE_COMMANDS 1)
# set platforms
if (CMAKE_SYSTEM_PROCESSOR MATCHES "amd64|x86_64")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 09/11: Union stmt support 'OutFileClause' (#7026)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit 1d4116233458d29913ea148b27061379f2141ccd
Author: EmmyMiao87 <52...@qq.com>
AuthorDate: Sat Nov 6 13:28:52 2021 +0800
Union stmt support 'OutFileClause' (#7026)
The union(set operation) stmt also need to analyze 'OutFileClause'.
Whether the fragment is colocate only needs to check the plan node belonging to this fragment.
---
.../org/apache/doris/analysis/OutFileClause.java | 8 +-
.../java/org/apache/doris/analysis/SelectStmt.java | 2 +-
.../apache/doris/analysis/SetOperationStmt.java | 2 +
.../org/apache/doris/journal/JournalEntity.java | 1 -
.../main/java/org/apache/doris/qe/Coordinator.java | 15 +--
.../doris/analysis/OutFileClauseFunctionTest.java | 101 +++++++++++++++++++++
.../org/apache/doris/planner/ColocatePlanTest.java | 14 +++
7 files changed, 121 insertions(+), 22 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
index b2268ab..05f1569 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
@@ -158,7 +158,7 @@ public class OutFileClause {
return schema;
}
- private void analyze(Analyzer analyzer) throws UserException {
+ public void analyze(Analyzer analyzer, List<Expr> resultExprs) throws UserException {
if (isAnalyzed) {
// If the query stmt is rewritten, the whole stmt will be analyzed again.
// But some of fields in this OutfileClause has been changed,
@@ -187,13 +187,9 @@ public class OutFileClause {
throw new AnalysisException("Must specify BROKER properties in OUTFILE clause");
}
isAnalyzed = true;
- }
-
- public void analyze(Analyzer analyzer, SelectStmt stmt) throws UserException {
- analyze(analyzer);
if (isParquetFormat()) {
- analyzeForParquetFormat(stmt.getResultExprs());
+ analyzeForParquetFormat(resultExprs);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
index 0f2167e..6a0630b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SelectStmt.java
@@ -537,7 +537,7 @@ public class SelectStmt extends QueryStmt {
}
}
if (hasOutFileClause()) {
- outFileClause.analyze(analyzer, this);
+ outFileClause.analyze(analyzer, resultExprs);
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/SetOperationStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/SetOperationStmt.java
index f735315..0d43295 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/SetOperationStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/SetOperationStmt.java
@@ -258,6 +258,8 @@ public class SetOperationStmt extends QueryStmt {
setOpsResultExprs_ = Expr.cloneList(resultExprs);
if (evaluateOrderBy) createSortTupleInfo(analyzer);
baseTblResultExprs = resultExprs;
+
+ if (hasOutFileClause()) outFileClause.analyze(analyzer, resultExprs);
}
/**
diff --git a/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java b/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java
index 03036aa..d386ae0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java
@@ -48,7 +48,6 @@ import org.apache.doris.load.loadv2.LoadJob.LoadJobStateUpdateInfo;
import org.apache.doris.load.loadv2.LoadJobFinalOperation;
import org.apache.doris.load.routineload.RoutineLoadJob;
import org.apache.doris.load.sync.SyncJob;
-import org.apache.doris.master.Checkpoint;
import org.apache.doris.mysql.privilege.UserPropertyInfo;
import org.apache.doris.persist.AlterRoutineLoadJobOperationLog;
import org.apache.doris.persist.AlterViewInfo;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
index ea82f2a..a091d87 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -1222,23 +1222,10 @@ public class Coordinator {
}
if (planFragment.hasColocatePlanNode()) {
+ colocateFragmentIds.add(planFragment.getId().asInt());
return true;
}
- if (node instanceof HashJoinNode) {
- HashJoinNode joinNode = (HashJoinNode) node;
- if (joinNode.isColocate()) {
- colocateFragmentIds.add(joinNode.getFragmentId().asInt());
- return true;
- }
- }
-
- for (PlanNode childNode : node.getChildren()) {
- if (childNode.getFragmentId().asInt() == planFragment.getId().asInt() && isColocateFragment(planFragment, childNode)) {
- return true;
- }
- }
-
return false;
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/analysis/OutFileClauseFunctionTest.java b/fe/fe-core/src/test/java/org/apache/doris/analysis/OutFileClauseFunctionTest.java
new file mode 100644
index 0000000..d2bc45a
--- /dev/null
+++ b/fe/fe-core/src/test/java/org/apache/doris/analysis/OutFileClauseFunctionTest.java
@@ -0,0 +1,101 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.analysis;
+
+import org.apache.doris.catalog.Catalog;
+import org.apache.doris.common.Config;
+import org.apache.doris.common.FeConstants;
+import org.apache.doris.common.jmockit.Deencapsulation;
+import org.apache.doris.qe.ConnectContext;
+import org.apache.doris.qe.StmtExecutor;
+import org.apache.doris.thrift.TFileFormatType;
+import org.apache.doris.utframe.DorisAssert;
+import org.apache.doris.utframe.UtFrameUtils;
+
+import java.io.File;
+import java.util.UUID;
+
+import org.junit.AfterClass;
+import org.junit.Assert;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+public class OutFileClauseFunctionTest {
+
+ private static String baseDir = "fe";
+ private static String runningDir = baseDir + "/mocked/MaterializedViewFunctionTest/"
+ + UUID.randomUUID().toString() + "/";
+ private static ConnectContext ctx;
+
+ private static final String DB_NAME = "db1";
+
+ @BeforeClass
+ public static void beforeClass() throws Exception {
+ FeConstants.default_scheduler_interval_millisecond = 10;
+ FeConstants.runningUnitTest = true;
+ Config.enable_outfile_to_local = true;
+ ctx = UtFrameUtils.createDefaultCtx();
+ UtFrameUtils.createDorisCluster(runningDir);
+ CreateDbStmt createDbStmt = (CreateDbStmt) UtFrameUtils.parseAndAnalyzeStmt("create database db1;", ctx);
+ Catalog.getCurrentCatalog().createDb(createDbStmt);
+ String createTableSQL = "create table " + DB_NAME
+ + ".test (k1 int, k2 varchar ) "
+ + "distributed by hash(k1) buckets 3 properties('replication_num' = '1');";
+ CreateTableStmt createTableStmt = (CreateTableStmt) UtFrameUtils.parseAndAnalyzeStmt(createTableSQL, ctx);
+ Catalog.getCurrentCatalog().createTable(createTableStmt);
+ }
+
+ @AfterClass
+ public static void tearDown() {
+ File file = new File(runningDir);
+ file.delete();
+ }
+
+ @Test
+ public void testSelectStmtOutFileClause() throws Exception {
+ String query1 = "select * from db1.test into outfile \"file:///" + runningDir + "/result_\";";
+ QueryStmt analyzedQueryStmt = (QueryStmt) UtFrameUtils.parseAndAnalyzeStmt(query1, ctx);
+ Assert.assertTrue(analyzedQueryStmt.hasOutFileClause());
+ OutFileClause outFileClause = analyzedQueryStmt.getOutFileClause();
+ boolean isOutFileClauseAnalyzed = Deencapsulation.getField(outFileClause, "isAnalyzed");
+ Assert.assertTrue(isOutFileClauseAnalyzed);
+ Assert.assertEquals(outFileClause.getFileFormatType(), TFileFormatType.FORMAT_CSV_PLAIN);
+ }
+
+ @Test
+ public void testSetOperationStmtOutFileClause() throws Exception {
+ String query1 = "select * from db1.test union select * from db1.test into outfile \"file:///" + runningDir + "/result_\";";
+ QueryStmt analyzedSetOperationStmt = (QueryStmt) UtFrameUtils.parseAndAnalyzeStmt(query1, ctx);
+ Assert.assertTrue(analyzedSetOperationStmt.hasOutFileClause());
+ OutFileClause outFileClause = analyzedSetOperationStmt.getOutFileClause();
+ boolean isOutFileClauseAnalyzed = Deencapsulation.getField(outFileClause, "isAnalyzed");
+ Assert.assertTrue(isOutFileClauseAnalyzed);
+ Assert.assertEquals(outFileClause.getFileFormatType(), TFileFormatType.FORMAT_CSV_PLAIN);
+ }
+
+ @Test
+ public void testParquetFormat() throws Exception {
+ String query1 = "select * from db1.test union select * from db1.test into outfile \"file:///" + runningDir + "/result_\" FORMAT AS PARQUET;";
+ QueryStmt analyzedSetOperationStmt = (QueryStmt) UtFrameUtils.parseAndAnalyzeStmt(query1, ctx);
+ Assert.assertTrue(analyzedSetOperationStmt.hasOutFileClause());
+ OutFileClause outFileClause = analyzedSetOperationStmt.getOutFileClause();
+ boolean isOutFileClauseAnalyzed = Deencapsulation.getField(outFileClause, "isAnalyzed");
+ Assert.assertTrue(isOutFileClauseAnalyzed);
+ Assert.assertEquals(outFileClause.getFileFormatType(), TFileFormatType.FORMAT_PARQUET);
+ }
+}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/ColocatePlanTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/ColocatePlanTest.java
index ac288dd..43d062a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/planner/ColocatePlanTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/planner/ColocatePlanTest.java
@@ -172,4 +172,18 @@ public class ColocatePlanTest {
Assert.assertEquals(instanceInfo.size(), 2);
}
+ @Test
+ public void checkColocatePlanFragment() throws Exception {
+ String sql = "select a.k1 from db1.test_colocate a, db1.test_colocate b where a.k1=b.k1 and a.k2=b.k2 group by a.k1;";
+ StmtExecutor executor = UtFrameUtils.getSqlStmtExecutor(ctx, sql);
+ Planner planner = executor.planner();
+ Coordinator coordinator = Deencapsulation.getField(executor, "coord");
+ boolean isColocateFragment0 = Deencapsulation.invoke(coordinator, "isColocateFragment",
+ planner.getFragments().get(1), planner.getFragments().get(1).getPlanRoot());
+ Assert.assertFalse(isColocateFragment0);
+ boolean isColocateFragment1 = Deencapsulation.invoke(coordinator, "isColocateFragment",
+ planner.getFragments().get(2), planner.getFragments().get(2).getPlanRoot());
+ Assert.assertTrue(isColocateFragment1);
+ }
+
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 11/11: [Bug] Change DateTimeValue Memmory Layout
To Old (#7022)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit c9d30107a01aa60c019b9ed22a76675dafa967fb
Author: HappenLee <ha...@hotmail.com>
AuthorDate: Mon Nov 8 21:56:14 2021 +0800
[Bug] Change DateTimeValue Memmory Layout To Old (#7022)
Change DateTimeValue Memmory Layout To Old to fix compatibility problems
---
be/src/runtime/datetime_value.h | 28 +++++++++++++++-------------
1 file changed, 15 insertions(+), 13 deletions(-)
diff --git a/be/src/runtime/datetime_value.h b/be/src/runtime/datetime_value.h
index 12aa25e..9ddb1fa 100644
--- a/be/src/runtime/datetime_value.h
+++ b/be/src/runtime/datetime_value.h
@@ -167,12 +167,12 @@ public:
DateTimeValue()
: _neg(0),
_type(TIME_DATETIME),
- _second(0),
- _minute(0),
_hour(0),
- _day(0),
- _month(0),
+ _minute(0),
+ _second(0),
_year(0),
+ _month(0),
+ _day(0),
_microsecond(0) {}
explicit DateTimeValue(int64_t t) { from_date_int64(t); }
@@ -625,15 +625,17 @@ private:
bool from_date_format_str(const char* format, int format_len, const char* value, int value_len,
const char** sub_val_end);
- // 1 bits for neg. 3 bits for type. 12bit for second
+ // NOTICE: it's dangerous if you want to modify the memory structure of datetime
+ // which will cause problem in serialization/deserialization of RowBatch.
+ // 1 bits for neg. 3 bits for type. 12bit for hour
uint16_t _neg : 1; // Used for time value.
uint16_t _type : 3; // Which type of this value.
- uint16_t _second : 12;
+ uint16_t _hour : 12;
uint8_t _minute;
- uint8_t _hour;
- uint8_t _day;
- uint8_t _month;
+ uint8_t _second;
uint16_t _year;
+ uint8_t _month;
+ uint8_t _day;
// TODO(zc): used for nothing
uint64_t _microsecond;
@@ -641,12 +643,12 @@ private:
uint32_t microsecond, uint16_t year, uint8_t month, uint8_t day)
: _neg(neg),
_type(type),
- _second(second),
- _minute(minute),
_hour(hour),
- _day(day),
- _month(month),
+ _minute(minute),
+ _second(second),
_year(year),
+ _month(month),
+ _day(day),
_microsecond(microsecond) {}
// RE2 obj is thread safe
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[incubator-doris] 05/11: Fix the SQL execution error caused by
tablet not being found due to Colocate join (#7002)
Posted by mo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch branch-0.15
in repository https://gitbox.apache.org/repos/asf/incubator-doris.git
commit cc6a6be4786072d03608e23f38f7e6bcb011869a
Author: weajun <we...@gmail.com>
AuthorDate: Thu Nov 4 09:21:52 2021 +0800
Fix the SQL execution error caused by tablet not being found due to Colocate join (#7002)
* fixbug sql execution sometimes due to failed to get tablet
---
fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
index 98b4661..ea82f2a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java
@@ -1234,7 +1234,7 @@ public class Coordinator {
}
for (PlanNode childNode : node.getChildren()) {
- if (isColocateFragment(planFragment, childNode)) {
+ if (childNode.getFragmentId().asInt() == planFragment.getId().asInt() && isColocateFragment(planFragment, childNode)) {
return true;
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org