You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by kx...@apache.org on 2023/10/16 16:04:53 UTC
[doris] branch branch-2.0 updated (69f2beb050d -> 11c81822e3e)
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a change to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
from 69f2beb050d [fix](regression-test)Fix test insert from tvf with common user (#25494)
new d3f0a1856ec [Bug](delete) Use date as common type for date comparison (#25262)
new 01bca9d95b7 [chore](format) fix tablet_meta.cpp (#25410)
new 6be4994e80e [chore](build) Fix compilation errors reported by GCC-13 (#25439)
new 6d14c43721c [fix](build) aarch64 compilation fix # (#25443)
new feb1ebbe42b insert default when json array parse failed (#25447)
new 91fe09e3a28 [opt] use correct column label when execute query in FE (#25372)
new e00d4622606 [fix](planner)should not add TupleIsNullPredicate for inlineview plan (#25338)
new 68c50ee00ee [fix](hive-udf) delete Logger to avoid Kryo serialize problem. (#25312)
new 2fedad33e4b [refactor](stats) Use id instead name in analysis info (#25213)
new fc37baaf53b [Chore](sink) add index number check for table sink (#25461)
new f2ba2965222 [fix](planner)should call SlotRef'smaterializeSrcExpr() method if the slotRef is materialized (#25467)
new 56500f7b0f8 [Improvement](materialized-view) set job failed when toAgentTaskRequest meet error (#25358)
new 5e02dc719f6 [compile](fix) fix ubsan compile error (#25473)
new 4916bbf7325 [enhancement](cooldown) Improve cooldown logs (#25432)
new 11c81822e3e [fix](path gc) fix data dir path gc (#25420)
The 15 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/be-ut-mac.yml | 4 +-
be/CMakeLists.txt | 5 +-
be/src/common/config.cpp | 3 +-
be/src/common/stack_trace.cpp | 2 +-
be/src/common/status.h | 8 +-
be/src/olap/cold_data_compaction.cpp | 2 +-
be/src/olap/data_dir.cpp | 25 +++++--
be/src/olap/data_dir.h | 8 +-
be/src/olap/memtable.cpp | 2 +-
be/src/olap/olap_server.cpp | 8 +-
be/src/olap/tablet.cpp | 21 ++++--
be/src/olap/tablet_meta.cpp | 2 +-
be/src/util/bitmap_value.h | 1 +
be/src/util/perf_counters.cpp | 2 +-
.../aggregate_function_state_merge.h | 2 +
.../exprs/table_function/vexplode_json_array.cpp | 2 +-
be/src/vec/functions/uuid_numeric.cpp | 4 +-
be/src/vec/runtime/shared_hash_table_controller.h | 2 +-
.../java/org/apache/doris/alter/RollupJobV2.java | 2 +-
.../org/apache/doris/alter/SchemaChangeJobV2.java | 4 +-
.../org/apache/doris/analysis/AnalyzeTblStmt.java | 6 ++
.../org/apache/doris/analysis/BinaryPredicate.java | 2 +
.../doris/analysis/CreateMaterializedViewStmt.java | 45 ++++++++++-
.../apache/doris/analysis/ExpressionFunctions.java | 2 +-
.../doris/catalog/BuiltinScalarFunctions.java | 2 +-
.../java/org/apache/doris/catalog/OlapTable.java | 4 +
.../org/apache/doris/datasource/CatalogMgr.java | 9 ++-
.../org/apache/doris/nereids/NereidsPlanner.java | 3 +-
.../mv/AbstractSelectMaterializedIndexRule.java | 8 --
.../mv/SelectMaterializedIndexWithAggregate.java | 36 ++++-----
.../expressions/functions/scalar/CurrentUser.java | 4 +-
.../org/apache/doris/planner/JoinNodeBase.java | 1 +
.../org/apache/doris/planner/OlapTableSink.java | 10 +++
.../apache/doris/planner/SingleNodePlanner.java | 14 ----
.../java/org/apache/doris/qe/ShowExecutor.java | 15 +++-
.../org/apache/doris/statistics/AnalysisInfo.java | 87 ++++++----------------
.../doris/statistics/AnalysisInfoBuilder.java | 32 ++++----
.../apache/doris/statistics/AnalysisManager.java | 50 ++++++++-----
.../apache/doris/statistics/BaseAnalysisTask.java | 31 +++-----
.../apache/doris/statistics/HMSAnalysisTask.java | 6 +-
.../org/apache/doris/statistics/HistogramTask.java | 4 +-
.../apache/doris/statistics/JdbcAnalysisTask.java | 4 +-
.../apache/doris/statistics/MVAnalysisTask.java | 4 +-
.../apache/doris/statistics/OlapAnalysisTask.java | 12 +--
.../doris/statistics/StatisticsAutoCollector.java | 35 +--------
.../apache/doris/statistics/StatisticsCleaner.java | 2 +-
.../doris/statistics/StatisticsCollector.java | 2 +-
.../apache/doris/statistics/util/DBObjects.java | 8 +-
.../doris/statistics/util/StatisticsUtil.java | 47 ++++++++++--
.../java/org/apache/doris/task/AgentBatchTask.java | 7 +-
.../apache/doris/planner/OlapTableSinkTest.java | 6 ++
.../apache/doris/statistics/AnalysisJobTest.java | 34 ++++++++-
.../doris/statistics/AnalysisTaskExecutorTest.java | 57 +++++++++++---
.../apache/doris/statistics/HistogramTaskTest.java | 31 +++++++-
.../java/org/apache/doris/udf/BitmapAndUDF.java | 4 -
.../java/org/apache/doris/udf/BitmapCountUDF.java | 4 -
.../java/org/apache/doris/udf/BitmapOrUDF.java | 4 -
.../java/org/apache/doris/udf/BitmapXorUDF.java | 4 -
.../test_outer_join_with_cross_join.out | 3 +
.../multiple_ssb_between.out} | 0
.../table_function/explode_json_array.out | 4 +
.../test_outer_join_with_cross_join.groovy | 18 +++++
.../multiple_ssb_between.groovy} | 8 +-
.../table_function/explode_json_array.groovy | 5 ++
.../suites/statistics/analyze_stats.groovy | 2 +-
65 files changed, 468 insertions(+), 317 deletions(-)
copy regression-test/data/mv_p0/ssb/{multiple_ssb/multiple_ssb.out => multiple_ssb_between/multiple_ssb_between.out} (100%)
copy regression-test/suites/mv_p0/ssb/{multiple_ssb/multiple_ssb.groovy => multiple_ssb_between/multiple_ssb_between.groovy} (98%)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 14/15: [enhancement](cooldown) Improve cooldown logs (#25432)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 4916bbf7325af165bd56f61c739a623635f304b6
Author: plat1ko <pl...@gmail.com>
AuthorDate: Mon Oct 16 20:17:00 2023 +0800
[enhancement](cooldown) Improve cooldown logs (#25432)
---
be/src/common/config.cpp | 3 ++-
be/src/common/status.h | 8 ++++----
be/src/olap/cold_data_compaction.cpp | 2 +-
be/src/olap/tablet.cpp | 21 ++++++++++++++-------
4 files changed, 21 insertions(+), 13 deletions(-)
diff --git a/be/src/common/config.cpp b/be/src/common/config.cpp
index 63ba1a1288d..d3f03eed84c 100644
--- a/be/src/common/config.cpp
+++ b/be/src/common/config.cpp
@@ -713,7 +713,8 @@ DEFINE_mInt32(zone_map_row_num_threshold, "20");
// Info = 4,
// Debug = 5,
// Trace = 6
-DEFINE_Int32(aws_log_level, "3");
+// Default to turn off aws sdk log, because aws sdk errors that need to be cared will be output through Doris logs
+DEFINE_Int32(aws_log_level, "0");
// the buffer size when read data from remote storage like s3
DEFINE_mInt32(remote_storage_read_buffer_mb, "16");
diff --git a/be/src/common/status.h b/be/src/common/status.h
index a5dfccd01dd..ef7513b2e8a 100644
--- a/be/src/common/status.h
+++ b/be/src/common/status.h
@@ -391,10 +391,10 @@ public:
static Status OK() { return Status(); }
-#define ERROR_CTOR(name, code) \
- template <typename... Args> \
- static Status name(std::string_view msg, Args&&... args) { \
- return Error<ErrorCode::code, true>(msg, std::forward<Args>(args)...); \
+#define ERROR_CTOR(name, code) \
+ template <bool stacktrace = true, typename... Args> \
+ static Status name(std::string_view msg, Args&&... args) { \
+ return Error<ErrorCode::code, stacktrace>(msg, std::forward<Args>(args)...); \
}
ERROR_CTOR(PublishTimeout, PUBLISH_TIMEOUT)
diff --git a/be/src/olap/cold_data_compaction.cpp b/be/src/olap/cold_data_compaction.cpp
index 27812aeab35..fc3f7569aa7 100644
--- a/be/src/olap/cold_data_compaction.cpp
+++ b/be/src/olap/cold_data_compaction.cpp
@@ -63,7 +63,7 @@ Status ColdDataCompaction::execute_compact_impl() {
int64_t permits = get_compaction_permits();
std::shared_lock cooldown_conf_rlock(_tablet->get_cooldown_conf_lock());
if (_tablet->cooldown_conf_unlocked().first != _tablet->replica_id()) {
- return Status::Aborted("this replica is not cooldown replica");
+ return Status::Aborted<false>("this replica is not cooldown replica");
}
RETURN_IF_ERROR(do_compaction(permits));
_state = CompactionState::SUCCESS;
diff --git a/be/src/olap/tablet.cpp b/be/src/olap/tablet.cpp
index 1ce46547679..1f46cf2b99f 100644
--- a/be/src/olap/tablet.cpp
+++ b/be/src/olap/tablet.cpp
@@ -2101,7 +2101,8 @@ Status Tablet::_cooldown_data() {
RETURN_IF_ERROR(get_remote_file_system(storage_policy_id(), &dest_fs));
auto old_rowset = pick_cooldown_rowset();
if (!old_rowset) {
- return Status::InternalError("cannot pick cooldown rowset in tablet {}", tablet_id());
+ LOG(INFO) << "cannot pick cooldown rowset in tablet " << tablet_id();
+ return Status::OK();
}
if (old_rowset->num_segments() < 1) {
// Empty rowset, just reset rowset's resource_id
@@ -2229,8 +2230,9 @@ bool Tablet::update_cooldown_conf(int64_t cooldown_term, int64_t cooldown_replic
Status Tablet::write_cooldown_meta() {
std::shared_lock rlock(_cooldown_conf_lock);
if (_cooldown_replica_id != _tablet_meta->replica_id()) {
- return Status::Aborted("not cooldown replcia({} vs {}) tablet_id={}",
- _tablet_meta->replica_id(), _cooldown_replica_id, tablet_id());
+ return Status::Aborted<false>("not cooldown replica({} vs {}) tablet_id={}",
+ _tablet_meta->replica_id(), _cooldown_replica_id,
+ tablet_id());
}
std::shared_ptr<io::RemoteFileSystem> fs;
@@ -2295,7 +2297,11 @@ Status Tablet::_follow_cooldowned_data() {
}
TabletMetaPB cooldown_meta_pb;
- RETURN_IF_ERROR(_read_cooldown_meta(fs, &cooldown_meta_pb));
+ auto st = _read_cooldown_meta(fs, &cooldown_meta_pb);
+ if (!st.ok()) {
+ LOG(INFO) << "cannot read cooldown meta: " << st;
+ return Status::InternalError<false>("cannot read cooldown meta");
+ }
DCHECK(cooldown_meta_pb.rs_metas_size() > 0);
if (_tablet_meta->cooldown_meta_id() == cooldown_meta_pb.cooldown_meta_id()) {
// cooldowned rowsets are same, no need to follow
@@ -2311,7 +2317,7 @@ Status Tablet::_follow_cooldowned_data() {
std::lock_guard wlock(_meta_lock);
SCOPED_SIMPLE_TRACE_IF_TIMEOUT(TRACE_TABLET_LOCK_THRESHOLD);
if (tablet_state() != TABLET_RUNNING) {
- return Status::InternalError("tablet not running");
+ return Status::InternalError<false>("tablet not running");
}
for (auto& [v, rs] : _rs_version_map) {
@@ -2321,13 +2327,14 @@ Status Tablet::_follow_cooldowned_data() {
}
}
if (!version_aligned) {
- return Status::InternalError("cooldowned version is not aligned");
+ return Status::InternalError<false>("cooldowned version is not aligned");
}
for (auto& [v, rs] : _rs_version_map) {
if (v.second <= cooldowned_version) {
overlap_rowsets.push_back(rs);
} else if (!rs->is_local()) {
- return Status::InternalError("cooldowned version larger than that to follow");
+ return Status::InternalError<false>(
+ "cooldowned version larger than that to follow");
}
}
std::sort(overlap_rowsets.begin(), overlap_rowsets.end(), Rowset::comparator);
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 09/15: [refactor](stats) Use id instead name in analysis info (#25213)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 2fedad33e4b7d64c09032759142199015c878f35
Author: AKIRA <33...@users.noreply.github.com>
AuthorDate: Mon Oct 16 16:49:53 2023 +0800
[refactor](stats) Use id instead name in analysis info (#25213)
---
.../org/apache/doris/analysis/AnalyzeTblStmt.java | 6 ++
.../org/apache/doris/datasource/CatalogMgr.java | 9 ++-
.../java/org/apache/doris/qe/ShowExecutor.java | 15 +++-
.../org/apache/doris/statistics/AnalysisInfo.java | 87 ++++++----------------
.../doris/statistics/AnalysisInfoBuilder.java | 32 ++++----
.../apache/doris/statistics/AnalysisManager.java | 50 ++++++++-----
.../apache/doris/statistics/BaseAnalysisTask.java | 31 +++-----
.../apache/doris/statistics/HMSAnalysisTask.java | 6 +-
.../org/apache/doris/statistics/HistogramTask.java | 4 +-
.../apache/doris/statistics/JdbcAnalysisTask.java | 4 +-
.../apache/doris/statistics/MVAnalysisTask.java | 4 +-
.../apache/doris/statistics/OlapAnalysisTask.java | 12 +--
.../doris/statistics/StatisticsAutoCollector.java | 35 +--------
.../apache/doris/statistics/StatisticsCleaner.java | 2 +-
.../doris/statistics/StatisticsCollector.java | 2 +-
.../apache/doris/statistics/util/DBObjects.java | 8 +-
.../doris/statistics/util/StatisticsUtil.java | 47 ++++++++++--
.../apache/doris/statistics/AnalysisJobTest.java | 34 ++++++++-
.../doris/statistics/AnalysisTaskExecutorTest.java | 57 +++++++++++---
.../apache/doris/statistics/HistogramTaskTest.java | 31 +++++++-
.../suites/statistics/analyze_stats.groovy | 2 +-
21 files changed, 275 insertions(+), 203 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeTblStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeTblStmt.java
index 185bee1d132..8329d2c451a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeTblStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/AnalyzeTblStmt.java
@@ -89,6 +89,7 @@ public class AnalyzeTblStmt extends AnalyzeStmt {
private boolean isAllColumns;
// after analyzed
+ private long catalogId;
private long dbId;
private TableIf table;
@@ -130,6 +131,7 @@ public class AnalyzeTblStmt extends AnalyzeStmt {
String tblName = tableName.getTbl();
CatalogIf catalog = analyzer.getEnv().getCatalogMgr()
.getCatalogOrAnalysisException(catalogName);
+ this.catalogId = catalog.getId();
DatabaseIf db = catalog.getDbOrAnalysisException(dbName);
dbId = db.getId();
table = db.getTableOrAnalysisException(tblName);
@@ -329,4 +331,8 @@ public class AnalyzeTblStmt extends AnalyzeStmt {
public boolean isAllColumns() {
return isAllColumns;
}
+
+ public long getCatalogId() {
+ return catalogId;
+ }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
index 356c116ae42..5dd8a464119 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java
@@ -92,7 +92,7 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
private final ReentrantReadWriteLock lock = new ReentrantReadWriteLock(true);
@SerializedName(value = "idToCatalog")
- private final Map<Long, CatalogIf> idToCatalog = Maps.newConcurrentMap();
+ private final Map<Long, CatalogIf<? extends DatabaseIf<? extends TableIf>>> idToCatalog = Maps.newConcurrentMap();
// this map will be regenerated from idToCatalog, so not need to persist.
private final Map<String, CatalogIf> nameToCatalog = Maps.newConcurrentMap();
// record last used database of every catalog
@@ -163,7 +163,7 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
return nameToCatalog.get(name);
}
- public CatalogIf getCatalog(long id) {
+ public CatalogIf<? extends DatabaseIf<? extends TableIf>> getCatalog(long id) {
return idToCatalog.get(id);
}
@@ -173,7 +173,8 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
ErrorCode.ERR_UNKNOWN_CATALOG));
}
- public <E extends Exception> CatalogIf getCatalogOrException(long id, Function<Long, E> e) throws E {
+ public <E extends Exception> CatalogIf<? extends DatabaseIf<? extends TableIf>>
+ getCatalogOrException(long id, Function<Long, E> e) throws E {
CatalogIf catalog = idToCatalog.get(id);
if (catalog == null) {
throw e.apply(id);
@@ -1174,7 +1175,7 @@ public class CatalogMgr implements Writable, GsonPostProcessable {
internalCatalog = (InternalCatalog) idToCatalog.get(InternalCatalog.INTERNAL_CATALOG_ID);
}
- public Map<Long, CatalogIf> getIdToCatalog() {
+ public Map<Long, CatalogIf<? extends DatabaseIf<? extends TableIf>>> getIdToCatalog() {
return idToCatalog;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
index 2c3cdf8ff32..5eb2e4187dd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ShowExecutor.java
@@ -197,6 +197,7 @@ import org.apache.doris.statistics.Histogram;
import org.apache.doris.statistics.StatisticsRepository;
import org.apache.doris.statistics.TableStatsMeta;
import org.apache.doris.statistics.query.QueryStatsUtil;
+import org.apache.doris.statistics.util.StatisticsUtil;
import org.apache.doris.system.Backend;
import org.apache.doris.system.Diagnoser;
import org.apache.doris.system.SystemInfoService;
@@ -239,6 +240,7 @@ import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Objects;
+import java.util.Optional;
import java.util.Set;
import java.util.concurrent.TimeUnit;
import java.util.function.Predicate;
@@ -2586,9 +2588,16 @@ public class ShowExecutor {
for (AnalysisInfo analysisInfo : results) {
List<String> row = new ArrayList<>();
row.add(String.valueOf(analysisInfo.jobId));
- row.add(analysisInfo.catalogName);
- row.add(analysisInfo.dbName);
- row.add(analysisInfo.tblName);
+ CatalogIf<? extends DatabaseIf<? extends TableIf>> c = StatisticsUtil.findCatalog(analysisInfo.catalogId);
+ row.add(c.getName());
+ Optional<? extends DatabaseIf<? extends TableIf>> databaseIf = c.getDb(analysisInfo.dbId);
+ row.add(databaseIf.isPresent() ? databaseIf.get().getFullName() : "DB may get deleted");
+ if (databaseIf.isPresent()) {
+ Optional<? extends TableIf> table = databaseIf.get().getTable(analysisInfo.tblId);
+ row.add(table.isPresent() ? table.get().getName() : "Table may get deleted");
+ } else {
+ row.add("DB may get deleted");
+ }
row.add(analysisInfo.colName);
row.add(analysisInfo.jobType.toString());
row.add(analysisInfo.analysisType.toString());
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfo.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfo.java
index 4df6f9cf7af..2bf06d0c2c4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfo.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfo.java
@@ -17,8 +17,6 @@
package org.apache.doris.statistics;
-import org.apache.doris.catalog.Env;
-import org.apache.doris.common.FeMetaVersion;
import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.persist.gson.GsonUtils;
@@ -37,8 +35,6 @@ import java.io.DataOutput;
import java.io.IOException;
import java.lang.reflect.Type;
import java.text.ParseException;
-import java.util.HashMap;
-import java.util.HashSet;
import java.util.List;
import java.util.Map;
import java.util.Set;
@@ -91,14 +87,14 @@ public class AnalysisInfo implements Writable {
@SerializedName("taskIds")
public final List<Long> taskIds;
- @SerializedName("catalogName")
- public final String catalogName;
+ @SerializedName("catalogId")
+ public final long catalogId;
- @SerializedName("dbName")
- public final String dbName;
+ @SerializedName("dbId")
+ public final long dbId;
- @SerializedName("tblName")
- public final String tblName;
+ @SerializedName("tblId")
+ public final long tblId;
// TODO: Map here is wired, List is enough
@SerializedName("colToPartitions")
@@ -183,7 +179,7 @@ public class AnalysisInfo implements Writable {
@SerializedName("forceFull")
public final boolean forceFull;
- public AnalysisInfo(long jobId, long taskId, List<Long> taskIds, String catalogName, String dbName, String tblName,
+ public AnalysisInfo(long jobId, long taskId, List<Long> taskIds, long catalogId, long dbId, long tblId,
Map<String, Set<String>> colToPartitions, Set<String> partitionNames, String colName, Long indexId,
JobType jobType, AnalysisMode analysisMode, AnalysisMethod analysisMethod, AnalysisType analysisType,
int samplePercent, long sampleRows, int maxBucketNum, long periodTimeInMs, String message,
@@ -193,9 +189,9 @@ public class AnalysisInfo implements Writable {
this.jobId = jobId;
this.taskId = taskId;
this.taskIds = taskIds;
- this.catalogName = catalogName;
- this.dbName = dbName;
- this.tblName = tblName;
+ this.catalogId = catalogId;
+ this.dbId = dbId;
+ this.tblId = tblId;
this.colToPartitions = colToPartitions;
this.partitionNames = partitionNames;
this.colName = colName;
@@ -229,9 +225,9 @@ public class AnalysisInfo implements Writable {
public String toString() {
StringJoiner sj = new StringJoiner("\n", getClass().getName() + ":\n", "\n");
sj.add("JobId: " + jobId);
- sj.add("CatalogName: " + catalogName);
- sj.add("DBName: " + dbName);
- sj.add("TableName: " + tblName);
+ sj.add("catalogId: " + catalogId);
+ sj.add("dbId: " + dbId);
+ sj.add("TableName: " + tblId);
sj.add("ColumnName: " + colName);
sj.add("TaskType: " + analysisType);
sj.add("TaskMode: " + analysisMode);
@@ -291,7 +287,8 @@ public class AnalysisInfo implements Writable {
return null;
}
Gson gson = new Gson();
- Type type = new TypeToken<Map<String, Set<String>>>() {}.getType();
+ Type type = new TypeToken<Map<String, Set<String>>>() {
+ }.getType();
return gson.fromJson(colToPartitionStr, type);
}
@@ -302,53 +299,15 @@ public class AnalysisInfo implements Writable {
}
public static AnalysisInfo read(DataInput dataInput) throws IOException {
- if (Env.getCurrentEnvJournalVersion() < FeMetaVersion.VERSION_123) {
- AnalysisInfoBuilder analysisInfoBuilder = new AnalysisInfoBuilder();
- analysisInfoBuilder.setJobId(dataInput.readLong());
- long taskId = dataInput.readLong();
- analysisInfoBuilder.setTaskId(taskId);
- analysisInfoBuilder.setCatalogName(Text.readString(dataInput));
- analysisInfoBuilder.setDbName(Text.readString(dataInput));
- analysisInfoBuilder.setTblName(Text.readString(dataInput));
- int size = dataInput.readInt();
- Map<String, Set<String>> colToPartitions = new HashMap<>();
- for (int i = 0; i < size; i++) {
- String k = Text.readString(dataInput);
- int partSize = dataInput.readInt();
- Set<String> parts = new HashSet<>();
- for (int j = 0; j < partSize; j++) {
- parts.add(Text.readString(dataInput));
- }
- colToPartitions.put(k, parts);
+ String json = Text.readString(dataInput);
+ AnalysisInfo analysisInfo = GsonUtils.GSON.fromJson(json, AnalysisInfo.class);
+ if (analysisInfo.cronExprStr != null) {
+ try {
+ analysisInfo.cronExpression = new CronExpression(analysisInfo.cronExprStr);
+ } catch (ParseException e) {
+ LOG.warn("Cron expression of job is invalid, there is a bug", e);
}
- analysisInfoBuilder.setColToPartitions(colToPartitions);
- analysisInfoBuilder.setColName(Text.readString(dataInput));
- analysisInfoBuilder.setIndexId(dataInput.readLong());
- analysisInfoBuilder.setJobType(JobType.valueOf(Text.readString(dataInput)));
- analysisInfoBuilder.setAnalysisMode(AnalysisMode.valueOf(Text.readString(dataInput)));
- analysisInfoBuilder.setAnalysisMethod(AnalysisMethod.valueOf(Text.readString(dataInput)));
- analysisInfoBuilder.setAnalysisType(AnalysisType.valueOf(Text.readString(dataInput)));
- analysisInfoBuilder.setSamplePercent(dataInput.readInt());
- analysisInfoBuilder.setSampleRows(dataInput.readInt());
- analysisInfoBuilder.setMaxBucketNum(dataInput.readInt());
- analysisInfoBuilder.setPeriodTimeInMs(dataInput.readLong());
- analysisInfoBuilder.setLastExecTimeInMs(dataInput.readLong());
- analysisInfoBuilder.setState(AnalysisState.valueOf(Text.readString(dataInput)));
- analysisInfoBuilder.setScheduleType(ScheduleType.valueOf(Text.readString(dataInput)));
- analysisInfoBuilder.setMessage(Text.readString(dataInput));
- analysisInfoBuilder.setExternalTableLevelTask(dataInput.readBoolean());
- return analysisInfoBuilder.build();
- } else {
- String json = Text.readString(dataInput);
- AnalysisInfo analysisInfo = GsonUtils.GSON.fromJson(json, AnalysisInfo.class);
- if (analysisInfo.cronExprStr != null) {
- try {
- analysisInfo.cronExpression = new CronExpression(analysisInfo.cronExprStr);
- } catch (ParseException e) {
- LOG.warn("Cron expression of job is invalid, there is a bug", e);
- }
- }
- return analysisInfo;
}
+ return analysisInfo;
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfoBuilder.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfoBuilder.java
index b6d9aafe853..b6dc7b5dd74 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfoBuilder.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisInfoBuilder.java
@@ -33,9 +33,9 @@ public class AnalysisInfoBuilder {
private long jobId;
private long taskId;
private List<Long> taskIds;
- private String catalogName;
- private String dbName;
- private String tblName;
+ private long catalogId;
+ private long dbId;
+ private long tblId;
private Map<String, Set<String>> colToPartitions;
private Set<String> partitionNames;
private String colName;
@@ -68,9 +68,9 @@ public class AnalysisInfoBuilder {
jobId = info.jobId;
taskId = info.taskId;
taskIds = info.taskIds;
- catalogName = info.catalogName;
- dbName = info.dbName;
- tblName = info.tblName;
+ catalogId = info.catalogId;
+ dbId = info.dbId;
+ tblId = info.tblId;
colToPartitions = info.colToPartitions;
partitionNames = info.partitionNames;
colName = info.colName;
@@ -112,18 +112,18 @@ public class AnalysisInfoBuilder {
return this;
}
- public AnalysisInfoBuilder setCatalogName(String catalogName) {
- this.catalogName = catalogName;
+ public AnalysisInfoBuilder setCatalogId(long catalogId) {
+ this.catalogId = catalogId;
return this;
}
- public AnalysisInfoBuilder setDbName(String dbName) {
- this.dbName = dbName;
+ public AnalysisInfoBuilder setDBId(long dbId) {
+ this.dbId = dbId;
return this;
}
- public AnalysisInfoBuilder setTblName(String tblName) {
- this.tblName = tblName;
+ public AnalysisInfoBuilder setTblId(long tblId) {
+ this.tblId = tblId;
return this;
}
@@ -246,7 +246,7 @@ public class AnalysisInfoBuilder {
}
public AnalysisInfo build() {
- return new AnalysisInfo(jobId, taskId, taskIds, catalogName, dbName, tblName, colToPartitions, partitionNames,
+ return new AnalysisInfo(jobId, taskId, taskIds, catalogId, dbId, tblId, colToPartitions, partitionNames,
colName, indexId, jobType, analysisMode, analysisMethod, analysisType, samplePercent,
sampleRows, maxBucketNum, periodTimeInMs, message, lastExecTimeInMs, timeCostInMs, state, scheduleType,
externalTableLevelTask, partitionOnly, samplingPartition, isAllPartition, partitionCount,
@@ -258,9 +258,9 @@ public class AnalysisInfoBuilder {
.setJobId(jobId)
.setTaskId(taskId)
.setTaskIds(taskIds)
- .setCatalogName(catalogName)
- .setDbName(dbName)
- .setTblName(tblName)
+ .setCatalogId(catalogId)
+ .setDBId(dbId)
+ .setTblId(tblId)
.setColToPartitions(colToPartitions)
.setColName(colName)
.setIndexId(indexId)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
index 9fa576f85cd..0d8308c4ebd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java
@@ -44,6 +44,7 @@ import org.apache.doris.common.io.Text;
import org.apache.doris.common.io.Writable;
import org.apache.doris.common.util.Daemon;
import org.apache.doris.common.util.Util;
+import org.apache.doris.datasource.CatalogIf;
import org.apache.doris.mysql.privilege.PrivPredicate;
import org.apache.doris.persist.AnalyzeDeletionLog;
import org.apache.doris.persist.TableStatsDeletionLog;
@@ -56,6 +57,7 @@ import org.apache.doris.statistics.AnalysisInfo.AnalysisMode;
import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
import org.apache.doris.statistics.AnalysisInfo.JobType;
import org.apache.doris.statistics.AnalysisInfo.ScheduleType;
+import org.apache.doris.statistics.util.DBObjects;
import org.apache.doris.statistics.util.SimpleQueue;
import org.apache.doris.statistics.util.StatisticsUtil;
@@ -84,6 +86,7 @@ import java.util.LinkedList;
import java.util.List;
import java.util.Map;
import java.util.Map.Entry;
+import java.util.Optional;
import java.util.Set;
import java.util.StringJoiner;
import java.util.TreeMap;
@@ -365,7 +368,7 @@ public class AnalysisManager extends Daemon implements Writable {
Map<Long, BaseAnalysisTask> analysisTaskInfos = new HashMap<>();
createTaskForEachColumns(jobInfo, analysisTaskInfos, isSync);
if (!jobInfo.partitionOnly && stmt.isAllColumns()
- && StatisticsUtil.isExternalTable(jobInfo.catalogName, jobInfo.dbName, jobInfo.tblName)) {
+ && StatisticsUtil.isExternalTable(jobInfo.catalogId, jobInfo.dbId, jobInfo.tblId)) {
createTableLevelTaskForExternalTable(jobInfo, analysisTaskInfos, isSync);
}
if (isSync) {
@@ -398,9 +401,16 @@ public class AnalysisManager extends Daemon implements Writable {
}
List<String> row = new ArrayList<>();
row.add(String.valueOf(analysisInfo.jobId));
- row.add(analysisInfo.catalogName);
- row.add(analysisInfo.dbName);
- row.add(analysisInfo.tblName);
+ CatalogIf<? extends DatabaseIf<? extends TableIf>> c = StatisticsUtil.findCatalog(analysisInfo.catalogId);
+ row.add(c.getName());
+ Optional<? extends DatabaseIf<? extends TableIf>> databaseIf = c.getDb(analysisInfo.dbId);
+ row.add(databaseIf.isPresent() ? databaseIf.get().getFullName() : "DB may get deleted");
+ if (databaseIf.isPresent()) {
+ Optional<? extends TableIf> table = databaseIf.get().getTable(analysisInfo.tblId);
+ row.add(table.isPresent() ? table.get().getName() : "Table may get deleted");
+ } else {
+ row.add("DB not exists anymore");
+ }
row.add(analysisInfo.colName);
resultRows.add(row);
}
@@ -487,11 +497,6 @@ public class AnalysisManager extends Daemon implements Writable {
public AnalysisInfo buildAnalysisJobInfo(AnalyzeTblStmt stmt) throws DdlException {
AnalysisInfoBuilder infoBuilder = new AnalysisInfoBuilder();
long jobId = Env.getCurrentEnv().getNextId();
- String catalogName = stmt.getCatalogName();
- String db = stmt.getDBName();
- TableName tbl = stmt.getTblName();
- StatisticsUtil.convertTableNameToObjects(tbl);
- String tblName = tbl.getTbl();
TableIf table = stmt.getTable();
Set<String> columnNames = stmt.getColumnNames();
Set<String> partitionNames = stmt.getPartitionNames();
@@ -508,9 +513,9 @@ public class AnalysisManager extends Daemon implements Writable {
CronExpression cronExpression = stmt.getCron();
infoBuilder.setJobId(jobId);
- infoBuilder.setCatalogName(catalogName);
- infoBuilder.setDbName(db);
- infoBuilder.setTblName(tblName);
+ infoBuilder.setCatalogId(stmt.getCatalogId());
+ infoBuilder.setDBId(stmt.getDbId());
+ infoBuilder.setTblId(stmt.getTable().getId());
// TODO: Refactor later, DON'T MODIFY IT RIGHT NOW
StringJoiner stringJoiner = new StringJoiner(",", "[", "]");
for (String colName : columnNames) {
@@ -638,8 +643,8 @@ public class AnalysisManager extends Daemon implements Writable {
@VisibleForTesting
public void updateTableStats(AnalysisInfo jobInfo) {
- TableIf tbl = StatisticsUtil.findTable(jobInfo.catalogName,
- jobInfo.dbName, jobInfo.tblName);
+ TableIf tbl = StatisticsUtil.findTable(jobInfo.catalogId,
+ jobInfo.dbId, jobInfo.tblId);
// External Table update table stats after table level task finished.
if (tbl instanceof ExternalTable) {
return;
@@ -666,11 +671,15 @@ public class AnalysisManager extends Daemon implements Writable {
protected List<AnalysisInfo> findShowAnalyzeResult(Collection<AnalysisInfo> analysisInfos, ShowAnalyzeStmt stmt) {
String state = stmt.getStateValue();
TableName tblName = stmt.getDbTableName();
+ TableIf tbl = null;
+ if (tblName != null) {
+ tbl = StatisticsUtil.findTable(tblName.getCtl(), tblName.getDb(), tblName.getTbl());
+ }
+ long tblId = tbl == null ? -1 : tbl.getId();
return analysisInfos.stream()
.filter(a -> stmt.getJobId() == 0 || a.jobId == stmt.getJobId())
.filter(a -> state == null || a.state.equals(AnalysisState.valueOf(state)))
- .filter(a -> tblName == null || a.catalogName.equals(tblName.getCtl())
- && a.dbName.equals(tblName.getDb()) && a.tblName.equals(tblName.getTbl()))
+ .filter(a -> tblName == null || a.tblId == tblId)
.sorted(Comparator.comparingLong(a -> a.jobId))
.collect(Collectors.toList());
}
@@ -777,8 +786,11 @@ public class AnalysisManager extends Daemon implements Writable {
}
private void checkPriv(AnalysisInfo analysisInfo) {
+ DBObjects dbObjects = StatisticsUtil.convertIdToObjects(analysisInfo.catalogId,
+ analysisInfo.dbId, analysisInfo.tblId);
if (!Env.getCurrentEnv().getAccessManager()
- .checkTblPriv(ConnectContext.get(), analysisInfo.dbName, analysisInfo.tblName, PrivPredicate.SELECT)) {
+ .checkTblPriv(ConnectContext.get(), dbObjects.catalog.getName(), dbObjects.db.getFullName(),
+ dbObjects.table.getName(), PrivPredicate.SELECT)) {
throw new RuntimeException("You need at least SELECT PRIV to corresponding table to kill this analyze"
+ " job");
}
@@ -793,8 +805,8 @@ public class AnalysisManager extends Daemon implements Writable {
private BaseAnalysisTask createTask(AnalysisInfo analysisInfo) throws DdlException {
try {
- TableIf table = StatisticsUtil.findTable(analysisInfo.catalogName,
- analysisInfo.dbName, analysisInfo.tblName);
+ TableIf table = StatisticsUtil.findTable(analysisInfo.catalogId,
+ analysisInfo.dbId, analysisInfo.tblId);
return table.createAnalysisTask(analysisInfo);
} catch (Throwable t) {
LOG.warn("Failed to find table", t);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java
index af85b528247..04b775bcd63 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/BaseAnalysisTask.java
@@ -30,6 +30,7 @@ import org.apache.doris.qe.QueryState.MysqlStateType;
import org.apache.doris.qe.StmtExecutor;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
+import org.apache.doris.statistics.util.DBObjects;
import org.apache.doris.statistics.util.StatisticsUtil;
import com.google.common.annotations.VisibleForTesting;
@@ -94,9 +95,9 @@ public abstract class BaseAnalysisTask {
protected AnalysisInfo info;
- protected CatalogIf catalog;
+ protected CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog;
- protected DatabaseIf db;
+ protected DatabaseIf<? extends TableIf> db;
protected TableIf tbl;
@@ -119,25 +120,11 @@ public abstract class BaseAnalysisTask {
}
protected void init(AnalysisInfo info) {
- catalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(info.catalogName);
- if (catalog == null) {
- Env.getCurrentEnv().getAnalysisManager().updateTaskStatus(info, AnalysisState.FAILED,
- String.format("Catalog with name: %s not exists", info.dbName), System.currentTimeMillis());
- return;
- }
- db = (DatabaseIf) catalog.getDb(info.dbName).orElse(null);
- if (db == null) {
- Env.getCurrentEnv().getAnalysisManager().updateTaskStatus(info, AnalysisState.FAILED,
- String.format("DB with name %s not exists", info.dbName), System.currentTimeMillis());
- return;
- }
- tbl = (TableIf) db.getTable(info.tblName).orElse(null);
- if (tbl == null) {
- Env.getCurrentEnv().getAnalysisManager().updateTaskStatus(
- info, AnalysisState.FAILED,
- String.format("Table with name %s not exists", info.tblName), System.currentTimeMillis());
- }
tableSample = getTableSample();
+ DBObjects dbObjects = StatisticsUtil.convertIdToObjects(info.catalogId, info.dbId, info.tblId);
+ catalog = dbObjects.catalog;
+ db = dbObjects.db;
+ tbl = dbObjects.table;
// External Table level task doesn't contain a column. Don't need to do the column related analyze.
if (info.externalTableLevelTask) {
return;
@@ -146,7 +133,7 @@ public abstract class BaseAnalysisTask {
|| info.analysisType.equals(AnalysisType.HISTOGRAM))) {
col = tbl.getColumn(info.colName);
if (col == null) {
- throw new RuntimeException(String.format("Column with name %s not exists", info.tblName));
+ throw new RuntimeException(String.format("Column with name %s not exists", tbl.getName()));
}
Preconditions.checkArgument(!StatisticsUtil.isUnsupportedType(col.getType()),
String.format("Column with type %s is not supported", col.getType().toString()));
@@ -261,7 +248,7 @@ public abstract class BaseAnalysisTask {
QueryState queryState = stmtExecutor.getContext().getState();
if (queryState.getStateType().equals(MysqlStateType.ERR)) {
throw new RuntimeException(String.format("Failed to analyze %s.%s.%s, error: %s sql: %s",
- info.catalogName, info.dbName, info.colName, stmtExecutor.getOriginStmt().toString(),
+ catalog.getName(), db.getFullName(), info.colName, stmtExecutor.getOriginStmt().toString(),
queryState.getErrorMessage()));
}
} finally {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/HMSAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/HMSAnalysisTask.java
index 076479aae8f..7ef87ac9401 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/HMSAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/HMSAnalysisTask.java
@@ -224,7 +224,7 @@ public class HMSAnalysisTask extends BaseAnalysisTask {
QueryState queryState = r.connectContext.getState();
if (queryState.getStateType().equals(QueryState.MysqlStateType.ERR)) {
throw new RuntimeException(String.format("Failed to analyze %s.%s.%s, error: %s sql: %s",
- info.catalogName, info.dbName, info.colName, partitionCollectSQL,
+ catalog.getName(), db.getFullName(), info.colName, partitionCollectSQL,
queryState.getErrorMessage()));
}
}
@@ -254,11 +254,11 @@ public class HMSAnalysisTask extends BaseAnalysisTask {
QueryState queryState = r.connectContext.getState();
if (queryState.getStateType().equals(QueryState.MysqlStateType.ERR)) {
LOG.warn(String.format("Failed to analyze %s.%s.%s, sql: [%s], error: [%s]",
- info.catalogName, info.dbName, info.colName, sql, queryState.getErrorMessage()));
+ catalog.getName(), db.getFullName(), info.colName, sql, queryState.getErrorMessage()));
throw new RuntimeException(queryState.getErrorMessage());
}
LOG.debug(String.format("Analyze %s.%s.%s done. SQL: [%s]. Cost %d ms.",
- info.catalogName, info.dbName, info.colName, sql, (System.currentTimeMillis() - startTime)));
+ catalog.getName(), db.getFullName(), info.colName, sql, (System.currentTimeMillis() - startTime)));
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/HistogramTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/HistogramTask.java
index 11274ec79e6..3e02c47497b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/HistogramTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/HistogramTask.java
@@ -67,8 +67,8 @@ public class HistogramTask extends BaseAnalysisTask {
params.put("tblId", String.valueOf(tbl.getId()));
params.put("idxId", String.valueOf(info.indexId));
params.put("colId", String.valueOf(info.colName));
- params.put("dbName", info.dbName);
- params.put("tblName", String.valueOf(info.tblName));
+ params.put("dbName", db.getFullName());
+ params.put("tblName", tbl.getName());
params.put("colName", String.valueOf(info.colName));
params.put("sampleRate", getSampleRateFunction());
params.put("maxBucketNum", String.valueOf(info.maxBucketNum));
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/JdbcAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/JdbcAnalysisTask.java
index 58be1510b44..0c148b5ad8d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/JdbcAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/JdbcAnalysisTask.java
@@ -130,11 +130,11 @@ public class JdbcAnalysisTask extends BaseAnalysisTask {
QueryState queryState = r.connectContext.getState();
if (queryState.getStateType().equals(QueryState.MysqlStateType.ERR)) {
LOG.warn(String.format("Failed to analyze %s.%s.%s, sql: [%s], error: [%s]",
- info.catalogName, info.dbName, info.colName, sql, queryState.getErrorMessage()));
+ catalog.getName(), db.getFullName(), info.colName, sql, queryState.getErrorMessage()));
throw new RuntimeException(queryState.getErrorMessage());
}
LOG.debug(String.format("Analyze %s.%s.%s done. SQL: [%s]. Cost %d ms.",
- info.catalogName, info.dbName, info.colName, sql, (System.currentTimeMillis() - startTime)));
+ catalog.getName(), db.getFullName(), info.colName, sql, (System.currentTimeMillis() - startTime)));
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/MVAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/MVAnalysisTask.java
index 89d4313cf2f..6a43c5092fa 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/MVAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/MVAnalysisTask.java
@@ -114,9 +114,9 @@ public class MVAnalysisTask extends BaseAnalysisTask {
String.valueOf(olapTable.getPartition(partName).getId());
params.put("partId", partId);
params.put("dataSizeFunction", getDataSizeFunction(column));
- params.put("dbName", info.dbName);
+ params.put("dbName", db.getFullName());
params.put("colName", colName);
- params.put("tblName", String.valueOf(info.tblName));
+ params.put("tblName", tbl.getName());
params.put("sql", sql);
StatisticsUtil.execUpdate(ANALYZE_MV_PART, params);
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
index 868bd1a8882..d14cbc0f0a1 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/OlapAnalysisTask.java
@@ -137,9 +137,9 @@ public class OlapAnalysisTask extends BaseAnalysisTask {
params.put("idxId", String.valueOf(info.indexId));
params.put("colId", String.valueOf(info.colName));
params.put("dataSizeFunction", getDataSizeFunction(col));
- params.put("dbName", info.dbName);
- params.put("colName", String.valueOf(info.colName));
- params.put("tblName", String.valueOf(info.tblName));
+ params.put("dbName", db.getFullName());
+ params.put("colName", info.colName);
+ params.put("tblName", tbl.getName());
params.put("scaleFactor", String.valueOf(scaleFactor));
params.put("tablets", tabletStr.isEmpty() ? "" : String.format("TABLET(%s)", tabletStr));
StringSubstitutor stringSubstitutor = new StringSubstitutor(params);
@@ -207,9 +207,9 @@ public class OlapAnalysisTask extends BaseAnalysisTask {
params.put("idxId", String.valueOf(info.indexId));
params.put("colId", String.valueOf(info.colName));
params.put("dataSizeFunction", getDataSizeFunction(col));
- params.put("dbName", info.dbName);
+ params.put("dbName", db.getFullName());
params.put("colName", String.valueOf(info.colName));
- params.put("tblName", String.valueOf(info.tblName));
+ params.put("tblName", String.valueOf(tbl.getName()));
List<String> partitionAnalysisSQLs = new ArrayList<>();
try {
tbl.readLock();
@@ -249,7 +249,7 @@ public class OlapAnalysisTask extends BaseAnalysisTask {
QueryState queryState = r.connectContext.getState();
if (queryState.getStateType().equals(MysqlStateType.ERR)) {
throw new RuntimeException(String.format("Failed to analyze %s.%s.%s, error: %s sql: %s",
- info.catalogName, info.dbName, info.colName, partitionCollectSQL,
+ catalog.getName(), db.getFullName(), info.colName, partitionCollectSQL,
queryState.getErrorMessage()));
}
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
index fe535b0fb4a..20ebc5848d0 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java
@@ -17,7 +17,6 @@
package org.apache.doris.statistics;
-import org.apache.doris.analysis.TableName;
import org.apache.doris.catalog.Column;
import org.apache.doris.catalog.DatabaseIf;
import org.apache.doris.catalog.Env;
@@ -32,7 +31,6 @@ import org.apache.doris.statistics.AnalysisInfo.JobType;
import org.apache.doris.statistics.AnalysisInfo.ScheduleType;
import org.apache.doris.statistics.util.StatisticsUtil;
-import com.google.common.collect.Maps;
import org.apache.hudi.common.util.VisibleForTesting;
import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;
@@ -126,13 +124,11 @@ public class StatisticsAutoCollector extends StatisticsCollector {
List<AnalysisInfo> analysisInfos, TableIf table) {
AnalysisMethod analysisMethod = table.getDataSize(true) > Config.huge_table_lower_bound_size_in_bytes
? AnalysisMethod.SAMPLE : AnalysisMethod.FULL;
- TableName tableName = new TableName(db.getCatalog().getName(), db.getFullName(),
- table.getName());
AnalysisInfo jobInfo = new AnalysisInfoBuilder()
.setJobId(Env.getCurrentEnv().getNextId())
- .setCatalogName(db.getCatalog().getName())
- .setDbName(db.getFullName())
- .setTblName(tableName.getTbl())
+ .setCatalogId(db.getCatalog().getId())
+ .setDBId(db.getId())
+ .setTblId(table.getId())
.setColName(
table.getBaseSchema().stream().filter(c -> !StatisticsUtil.isUnsupportedType(c.getType()))
.map(
@@ -153,7 +149,7 @@ public class StatisticsAutoCollector extends StatisticsCollector {
@VisibleForTesting
protected AnalysisInfo getReAnalyzeRequiredPart(AnalysisInfo jobInfo) {
TableIf table = StatisticsUtil
- .findTable(jobInfo.catalogName, jobInfo.dbName, jobInfo.tblName);
+ .findTable(jobInfo.catalogId, jobInfo.dbId, jobInfo.tblId);
AnalysisManager analysisManager = Env.getServingEnv().getAnalysisManager();
TableStatsMeta tblStats = analysisManager.findTableStatsStatus(table.getId());
@@ -170,27 +166,4 @@ public class StatisticsAutoCollector extends StatisticsCollector {
return new AnalysisInfoBuilder(jobInfo).setColToPartitions(needRunPartitions).build();
}
- @VisibleForTesting
- protected AnalysisInfo getAnalysisJobInfo(AnalysisInfo jobInfo, TableIf table,
- Set<String> needRunPartitions) {
- Map<String, Set<String>> newColToPartitions = Maps.newHashMap();
- Map<String, Set<String>> colToPartitions = jobInfo.colToPartitions;
- if (colToPartitions == null) {
- for (Column c : table.getColumns()) {
- if (StatisticsUtil.isUnsupportedType(c.getType())) {
- continue;
- }
- newColToPartitions.put(c.getName(), needRunPartitions);
- }
- } else {
- colToPartitions.keySet().forEach(colName -> {
- Column column = table.getColumn(colName);
- if (column != null) {
- newColToPartitions.put(colName, needRunPartitions);
- }
- });
- }
- return new AnalysisInfoBuilder(jobInfo)
- .setColToPartitions(newColToPartitions).build();
- }
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCleaner.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCleaner.java
index 6521a8b4a59..88fa098e57b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCleaner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCleaner.java
@@ -55,7 +55,7 @@ public class StatisticsCleaner extends MasterDaemon {
private OlapTable colStatsTbl;
private OlapTable histStatsTbl;
- private Map<Long, CatalogIf> idToCatalog;
+ private Map<Long, CatalogIf<? extends DatabaseIf<? extends TableIf>>> idToCatalog;
private Map<Long, DatabaseIf> idToDb;
private Map<Long, TableIf> idToTbl;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java
index 2d5c4816835..c2f1db6bc4a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsCollector.java
@@ -76,7 +76,7 @@ public abstract class StatisticsCollector extends MasterDaemon {
Map<Long, BaseAnalysisTask> analysisTaskInfos = new HashMap<>();
AnalysisManager analysisManager = Env.getCurrentEnv().getAnalysisManager();
analysisManager.createTaskForEachColumns(jobInfo, analysisTaskInfos, false);
- if (StatisticsUtil.isExternalTable(jobInfo.catalogName, jobInfo.dbName, jobInfo.tblName)) {
+ if (StatisticsUtil.isExternalTable(jobInfo.catalogId, jobInfo.dbId, jobInfo.tblId)) {
analysisManager.createTableLevelTaskForExternalTable(jobInfo, analysisTaskInfos, false);
}
Env.getCurrentEnv().getAnalysisManager().registerSysJob(jobInfo, analysisTaskInfos);
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/util/DBObjects.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/util/DBObjects.java
index 3e68d0e0274..3e3a258030c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/util/DBObjects.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/util/DBObjects.java
@@ -23,14 +23,14 @@ import org.apache.doris.datasource.CatalogIf;
public class DBObjects {
- public final CatalogIf<DatabaseIf> catalog;
+ public final CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog;
- public final DatabaseIf<TableIf> db;
+ public final DatabaseIf<? extends TableIf> db;
public final TableIf table;
- public DBObjects(CatalogIf<DatabaseIf> catalog,
- DatabaseIf<TableIf> db, TableIf table) {
+ public DBObjects(CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog,
+ DatabaseIf<? extends TableIf> db, TableIf table) {
this.catalog = catalog;
this.db = db;
this.table = table;
diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java
index f482c812879..985dd20b635 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/util/StatisticsUtil.java
@@ -301,13 +301,13 @@ public class StatisticsUtil {
}
-
public static DBObjects convertTableNameToObjects(TableName tableName) {
- CatalogIf<DatabaseIf> catalogIf = Env.getCurrentEnv().getCatalogMgr().getCatalog(tableName.getCtl());
+ CatalogIf<? extends DatabaseIf<? extends TableIf>> catalogIf =
+ Env.getCurrentEnv().getCatalogMgr().getCatalog(tableName.getCtl());
if (catalogIf == null) {
throw new IllegalStateException(String.format("Catalog:%s doesn't exist", tableName.getCtl()));
}
- DatabaseIf<TableIf> databaseIf = catalogIf.getDbNullable(tableName.getDb());
+ DatabaseIf<? extends TableIf> databaseIf = catalogIf.getDbNullable(tableName.getDb());
if (databaseIf == null) {
throw new IllegalStateException(String.format("DB:%s doesn't exist", tableName.getDb()));
}
@@ -318,12 +318,17 @@ public class StatisticsUtil {
return new DBObjects(catalogIf, databaseIf, tableIf);
}
+ public static DBObjects convertIdToObjects(long catalogId, long dbId, long tblId) {
+ return new DBObjects(findCatalog(catalogId), findDatabase(catalogId, dbId), findTable(catalogId, dbId, tblId));
+ }
+
public static Column findColumn(long catalogId, long dbId, long tblId, long idxId, String columnName) {
- CatalogIf<DatabaseIf<TableIf>> catalogIf = Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
+ CatalogIf<? extends DatabaseIf<? extends TableIf>> catalogIf =
+ Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogId);
if (catalogIf == null) {
return null;
}
- DatabaseIf<TableIf> db = catalogIf.getDb(dbId).orElse(null);
+ DatabaseIf<? extends TableIf> db = catalogIf.getDb(dbId).orElse(null);
if (db == null) {
return null;
}
@@ -361,6 +366,16 @@ public class StatisticsUtil {
}
}
+ public static TableIf findTable(long catalogId, long dbId, long tblId) {
+ try {
+ DatabaseIf<? extends TableIf> db = findDatabase(catalogId, dbId);
+ return db.getTableOrException(tblId,
+ t -> new RuntimeException("Table: " + t + " not exists"));
+ } catch (Throwable t) {
+ throw new RuntimeException("Table: `" + catalogId + "." + dbId + "." + tblId + "` not exists");
+ }
+ }
+
/**
* Throw RuntimeException if database not exists.
*/
@@ -371,6 +386,12 @@ public class StatisticsUtil {
d -> new RuntimeException("DB: " + d + " not exists"));
}
+ public static DatabaseIf<? extends TableIf> findDatabase(long catalogId, long dbId) {
+ CatalogIf<? extends DatabaseIf<? extends TableIf>> catalog = findCatalog(catalogId);
+ return catalog.getDbOrException(dbId,
+ d -> new RuntimeException("DB: " + d + " not exists"));
+ }
+
/**
* Throw RuntimeException if catalog not exists.
*/
@@ -380,6 +401,11 @@ public class StatisticsUtil {
.getCatalogOrException(catalogName, c -> new RuntimeException("Catalog: " + c + " not exists"));
}
+ public static CatalogIf<? extends DatabaseIf<? extends TableIf>> findCatalog(long catalogId) {
+ return Env.getCurrentEnv().getCatalogMgr().getCatalogOrException(catalogId,
+ c -> new RuntimeException("Catalog: " + c + " not exists"));
+ }
+
public static boolean isNullOrEmpty(String str) {
return Optional.ofNullable(str)
.map(String::trim)
@@ -763,6 +789,17 @@ public class StatisticsUtil {
return table instanceof ExternalTable;
}
+ public static boolean isExternalTable(long catalogId, long dbId, long tblId) {
+ TableIf table;
+ try {
+ table = findTable(catalogId, dbId, tblId);
+ } catch (Throwable e) {
+ LOG.warn(e.getMessage());
+ return false;
+ }
+ return table instanceof ExternalTable;
+ }
+
public static boolean inAnalyzeTime(LocalTime now) {
try {
Pair<LocalTime, LocalTime> range = findRangeFromGlobalSessionVar();
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisJobTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisJobTest.java
index 9624c201498..f01485f642f 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisJobTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisJobTest.java
@@ -17,8 +17,13 @@
package org.apache.doris.statistics;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.InternalSchemaInitializer;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.common.FeConstants;
+import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.qe.AutoCloseConnectContext;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.qe.StmtExecutor;
@@ -26,6 +31,7 @@ import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMode;
import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
import org.apache.doris.statistics.AnalysisInfo.JobType;
+import org.apache.doris.statistics.util.DBObjects;
import org.apache.doris.statistics.util.StatisticsUtil;
import org.apache.doris.utframe.TestWithFeService;
@@ -41,6 +47,7 @@ import java.util.ArrayList;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.Set;
public class AnalysisJobTest extends TestWithFeService {
@@ -95,8 +102,18 @@ public class AnalysisJobTest extends TestWithFeService {
}
@Test
- public void testJobExecution(@Mocked StmtExecutor stmtExecutor)
+ public void testJobExecution(@Mocked StmtExecutor stmtExecutor, @Mocked InternalCatalog catalog, @Mocked
+ Database database,
+ @Mocked OlapTable olapTable)
throws Exception {
+ new MockUp<OlapTable>() {
+
+ @Mock
+ public Column getColumn(String name) {
+ return new Column("col1", PrimitiveType.INT);
+ }
+ };
+
new MockUp<StatisticsUtil>() {
@Mock
@@ -107,6 +124,11 @@ public class AnalysisJobTest extends TestWithFeService {
@Mock
public void execUpdate(String sql) throws Exception {
}
+
+ @Mock
+ public DBObjects convertIdToObjects(long catalogId, long dbId, long tblId) {
+ return new DBObjects(catalog, database, olapTable);
+ }
};
new MockUp<StatisticsCache>() {
@@ -126,10 +148,18 @@ public class AnalysisJobTest extends TestWithFeService {
return new ArrayList<>();
}
};
+
+ new MockUp<OlapAnalysisTask>() {
+
+ @Mock
+ public void execSQLs(List<String> partitionAnalysisSQLs, Map<String, String> params) throws Exception {}
+ };
HashMap<String, Set<String>> colToPartitions = Maps.newHashMap();
colToPartitions.put("col1", Collections.singleton("t1"));
AnalysisInfo analysisJobInfo = new AnalysisInfoBuilder().setJobId(0).setTaskId(0)
- .setCatalogName("internal").setDbName("default_cluster:analysis_job_test").setTblName("t1")
+ .setCatalogId(0)
+ .setDBId(0)
+ .setTblId(0)
.setColName("col1").setJobType(JobType.MANUAL)
.setAnalysisMode(AnalysisMode.FULL)
.setAnalysisMethod(AnalysisMethod.FULL)
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
index 196ac8ad9a0..19d7798041a 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/AnalysisTaskExecutorTest.java
@@ -17,24 +17,32 @@
package org.apache.doris.statistics;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.InternalSchemaInitializer;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.common.jmockit.Deencapsulation;
+import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.qe.StmtExecutor;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMode;
import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
import org.apache.doris.statistics.AnalysisInfo.JobType;
+import org.apache.doris.statistics.util.DBObjects;
+import org.apache.doris.statistics.util.StatisticsUtil;
import org.apache.doris.utframe.TestWithFeService;
import com.google.common.collect.Maps;
-import mockit.Expectations;
import mockit.Mock;
import mockit.MockUp;
+import mockit.Mocked;
import org.junit.jupiter.api.Test;
import java.util.Collections;
import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.Set;
import java.util.concurrent.BlockingQueue;
@@ -58,9 +66,26 @@ public class AnalysisTaskExecutorTest extends TestWithFeService {
}
@Test
- public void testExpiredJobCancellation() throws Exception {
+ public void testExpiredJobCancellation(@Mocked InternalCatalog catalog, @Mocked Database database,
+ @Mocked OlapTable olapTable) throws Exception {
+ new MockUp<StatisticsUtil>() {
+
+ @Mock
+ public DBObjects convertIdToObjects(long catalogId, long dbId, long tblId) {
+ return new DBObjects(catalog, database, olapTable);
+ }
+ };
+ new MockUp<OlapTable>() {
+
+ @Mock
+ public Column getColumn(String name) {
+ return new Column("col1", PrimitiveType.INT);
+ }
+ };
AnalysisInfo analysisJobInfo = new AnalysisInfoBuilder().setJobId(0).setTaskId(0)
- .setCatalogName("internal").setDbName("default_cluster:analysis_job_test").setTblName("t1")
+ .setCatalogId(0)
+ .setDBId(0)
+ .setTblId(0)
.setColName("col1").setJobType(JobType.MANUAL)
.setAnalysisMode(AnalysisMode.FULL)
.setAnalysisMethod(AnalysisMethod.FULL)
@@ -77,8 +102,22 @@ public class AnalysisTaskExecutorTest extends TestWithFeService {
}
@Test
- public void testTaskExecution() throws Exception {
+ public void testTaskExecution(@Mocked InternalCatalog catalog, @Mocked Database database,
+ @Mocked OlapTable olapTable) throws Exception {
+ new MockUp<StatisticsUtil>() {
+
+ @Mock
+ public DBObjects convertIdToObjects(long catalogId, long dbId, long tblId) {
+ return new DBObjects(catalog, database, olapTable);
+ }
+ };
+ new MockUp<OlapTable>() {
+ @Mock
+ public Column getColumn(String name) {
+ return new Column("col1", PrimitiveType.INT);
+ }
+ };
new MockUp<StmtExecutor>() {
@Mock
public List<ResultRow> executeInternalQuery() {
@@ -88,7 +127,7 @@ public class AnalysisTaskExecutorTest extends TestWithFeService {
new MockUp<OlapAnalysisTask>() {
@Mock
- public void execSQLs(List<String> sqls) throws Exception {
+ public void execSQLs(List<String> partitionAnalysisSQLs, Map<String, String> params) throws Exception {
}
@Mock
@@ -108,7 +147,7 @@ public class AnalysisTaskExecutorTest extends TestWithFeService {
HashMap<String, Set<String>> colToPartitions = Maps.newHashMap();
colToPartitions.put("col1", Collections.singleton("t1"));
AnalysisInfo analysisInfo = new AnalysisInfoBuilder().setJobId(0).setTaskId(0)
- .setCatalogName("internal").setDbName("default_cluster:analysis_job_test").setTblName("t1")
+ .setCatalogId(0).setDBId(0).setTblId(0)
.setColName("col1").setJobType(JobType.MANUAL)
.setAnalysisMode(AnalysisMode.FULL)
.setAnalysisMethod(AnalysisMethod.FULL)
@@ -122,11 +161,7 @@ public class AnalysisTaskExecutorTest extends TestWithFeService {
@Mock
public void updateTaskStatus(AnalysisInfo info, AnalysisState jobState, String message, long time) {}
};
- new Expectations() {
- {
- task.doExecute();
- }
- };
+
Deencapsulation.invoke(analysisTaskExecutor, "submitTask", task);
}
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/statistics/HistogramTaskTest.java b/fe/fe-core/src/test/java/org/apache/doris/statistics/HistogramTaskTest.java
index 0660c994a12..b856d374bef 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/statistics/HistogramTaskTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/statistics/HistogramTaskTest.java
@@ -17,19 +17,27 @@
package org.apache.doris.statistics;
+import org.apache.doris.catalog.Column;
+import org.apache.doris.catalog.Database;
import org.apache.doris.catalog.Env;
+import org.apache.doris.catalog.OlapTable;
+import org.apache.doris.catalog.PrimitiveType;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.jmockit.Deencapsulation;
+import org.apache.doris.datasource.InternalCatalog;
import org.apache.doris.qe.StmtExecutor;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMethod;
import org.apache.doris.statistics.AnalysisInfo.AnalysisMode;
import org.apache.doris.statistics.AnalysisInfo.AnalysisType;
import org.apache.doris.statistics.AnalysisInfo.JobType;
+import org.apache.doris.statistics.util.DBObjects;
+import org.apache.doris.statistics.util.StatisticsUtil;
import org.apache.doris.system.SystemInfoService;
import org.apache.doris.utframe.TestWithFeService;
import mockit.Mock;
import mockit.MockUp;
+import mockit.Mocked;
import org.junit.FixMethodOrder;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;
@@ -84,18 +92,33 @@ public class HistogramTaskTest extends TestWithFeService {
for (Entry<Long, BaseAnalysisTask> infoEntry : taskInfo.entrySet()) {
BaseAnalysisTask task = infoEntry.getValue();
Assertions.assertEquals(AnalysisType.HISTOGRAM, task.info.analysisType);
- Assertions.assertEquals("t1", task.info.tblName);
Assertions.assertEquals("col1", task.info.colName);
}
}
}
@Test
- public void test2TaskExecution() throws Exception {
+ public void test2TaskExecution(@Mocked InternalCatalog catalog, @Mocked Database database,
+ @Mocked OlapTable olapTable) throws Exception {
+ new MockUp<StatisticsUtil>() {
+
+ @Mock
+ public DBObjects convertIdToObjects(long catalogId, long dbId, long tblId) {
+ return new DBObjects(catalog, database, olapTable);
+ }
+ };
+ new MockUp<OlapTable>() {
+
+ @Mock
+ public Column getColumn(String name) {
+ return new Column("col1", PrimitiveType.INT);
+ }
+ };
AnalysisTaskExecutor analysisTaskExecutor = new AnalysisTaskExecutor(1);
AnalysisInfo analysisInfo = new AnalysisInfoBuilder()
- .setJobId(0).setTaskId(0).setCatalogName("internal")
- .setDbName(SystemInfoService.DEFAULT_CLUSTER + ":" + "histogram_task_test").setTblName("t1")
+ .setJobId(0).setTaskId(0).setCatalogId(0)
+ .setDBId(0)
+ .setTblId(0)
.setColName("col1").setJobType(JobType.MANUAL)
.setAnalysisMode(AnalysisMode.FULL)
.setAnalysisMethod(AnalysisMethod.FULL)
diff --git a/regression-test/suites/statistics/analyze_stats.groovy b/regression-test/suites/statistics/analyze_stats.groovy
index b168230b2d5..04bd9c730d4 100644
--- a/regression-test/suites/statistics/analyze_stats.groovy
+++ b/regression-test/suites/statistics/analyze_stats.groovy
@@ -18,7 +18,7 @@ import java.util.stream.Collectors
// under the License.
suite("test_analyze") {
- String db = "test_analyze"
+ String db = "regression_test_statistics"
String tbl = "analyzetestlimited_duplicate_all"
sql """
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 04/15: [fix](build) aarch64 compilation fix # (#25443)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6d14c43721c4bfa8c398728c1d6f48a6bd457611
Author: Nitin-Kashyap <66...@users.noreply.github.com>
AuthorDate: Mon Oct 16 07:23:50 2023 +0530
[fix](build) aarch64 compilation fix # (#25443)
Issue: #25442
Compilation to include execinfo when building on aarch64
---
be/src/common/stack_trace.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/common/stack_trace.cpp b/be/src/common/stack_trace.cpp
index a6768517943..f9dc0c09387 100644
--- a/be/src/common/stack_trace.cpp
+++ b/be/src/common/stack_trace.cpp
@@ -38,7 +38,7 @@
#include "vec/common/demangle.h"
#include "vec/common/hex.h"
-#if USE_UNWIND
+#if USE_UNWIND && defined(__x86_64__)
#include <libunwind.h>
#else
#include <execinfo.h>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 08/15: [fix](hive-udf) delete Logger to avoid Kryo serialize problem. (#25312)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 68c50ee00ee8a270b474c0ff506de8d8e9b7e285
Author: yagagagaga <zh...@gmail.com>
AuthorDate: Mon Oct 16 16:10:06 2023 +0800
[fix](hive-udf) delete Logger to avoid Kryo serialize problem. (#25312)
---
fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapAndUDF.java | 4 ----
fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapCountUDF.java | 4 ----
fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapOrUDF.java | 4 ----
fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapXorUDF.java | 4 ----
4 files changed, 16 deletions(-)
diff --git a/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapAndUDF.java b/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapAndUDF.java
index cc84c5afafc..8b5044b51f3 100644
--- a/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapAndUDF.java
+++ b/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapAndUDF.java
@@ -27,15 +27,12 @@ import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.BinaryObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
import java.io.IOException;
@Description(name = "bitmap_and", value = "a _FUNC_ b - Compute intersection of two or more input bitmaps,"
+ " return the new bitmap")
public class BitmapAndUDF extends GenericUDF {
- private static final Logger LOG = LogManager.getLogger(BitmapAndUDF.class);
private transient BinaryObjectInspector inputOI0;
private transient BinaryObjectInspector inputOI1;
@@ -68,7 +65,6 @@ public class BitmapAndUDF extends GenericUDF {
bitmapValue0.and(bitmapValue1);
return BitmapValueUtil.serializeToBytes(bitmapValue1);
} catch (IOException ioException) {
- LOG.warn("", ioException);
throw new RuntimeException(ioException);
}
}
diff --git a/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapCountUDF.java b/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapCountUDF.java
index 28475610ec8..408c6c1d470 100644
--- a/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapCountUDF.java
+++ b/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapCountUDF.java
@@ -27,15 +27,12 @@ import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.BinaryObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
import java.io.IOException;
@Description(name = "bitmap_count", value = "a _FUNC_ b - Returns the number of distinct integers"
+ " added to the bitmap (e.g., number of bits set)")
public class BitmapCountUDF extends GenericUDF {
- private static final Logger LOG = LogManager.getLogger(BitmapCountUDF.class);
private transient BinaryObjectInspector inputOI;
@Override
@@ -62,7 +59,6 @@ public class BitmapCountUDF extends GenericUDF {
BitmapValue bitmapValue = BitmapValueUtil.deserializeToBitmap(inputBytes);
return bitmapValue.cardinality();
} catch (IOException ioException) {
- LOG.warn("", ioException);
throw new HiveException(ioException);
}
}
diff --git a/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapOrUDF.java b/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapOrUDF.java
index 197e137946e..9ed1446d3fb 100644
--- a/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapOrUDF.java
+++ b/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapOrUDF.java
@@ -27,15 +27,12 @@ import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.BinaryObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
import java.io.IOException;
@Description(name = "bitmap_or", value = "a _FUNC_ b - Compute"
+ " union of two or more input bitmaps, returns the new bitmap")
public class BitmapOrUDF extends GenericUDF {
- private static final Logger LOG = LogManager.getLogger(BitmapOrUDF.class);
private transient BinaryObjectInspector inputOI0;
private transient BinaryObjectInspector inputOI1;
@@ -68,7 +65,6 @@ public class BitmapOrUDF extends GenericUDF {
bitmapValue0.or(bitmapValue1);
return BitmapValueUtil.serializeToBytes(bitmapValue1);
} catch (IOException ioException) {
- LOG.warn("", ioException);
throw new RuntimeException(ioException);
}
}
diff --git a/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapXorUDF.java b/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapXorUDF.java
index d6c23f7dbb7..fa46eeec7a2 100644
--- a/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapXorUDF.java
+++ b/fe/hive-udf/src/main/java/org/apache/doris/udf/BitmapXorUDF.java
@@ -27,15 +27,12 @@ import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.BinaryObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
-import org.apache.logging.log4j.LogManager;
-import org.apache.logging.log4j.Logger;
import java.io.IOException;
@Description(name = "bitmap_xor", value = "a _FUNC_ b - Compute the symmetric"
+ " union of two or more input bitmaps, return the new bitmap")
public class BitmapXorUDF extends GenericUDF {
- private static final Logger LOG = LogManager.getLogger(BitmapXorUDF.class);
private transient BinaryObjectInspector inputOI0;
private transient BinaryObjectInspector inputOI1;
@@ -68,7 +65,6 @@ public class BitmapXorUDF extends GenericUDF {
bitmapValue0.xor(bitmapValue1);
return BitmapValueUtil.serializeToBytes(bitmapValue1);
} catch (IOException ioException) {
- LOG.warn("", ioException);
throw new RuntimeException(ioException);
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 07/15: [fix](planner)should not add TupleIsNullPredicate for inlineview plan (#25338)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e00d46226060af2c58fe315436f933cc9a7da7a7
Author: starocean999 <40...@users.noreply.github.com>
AuthorDate: Mon Oct 16 15:24:13 2023 +0800
[fix](planner)should not add TupleIsNullPredicate for inlineview plan (#25338)
---
.../org/apache/doris/planner/SingleNodePlanner.java | 14 --------------
.../correctness_p0/test_outer_join_with_cross_join.out | 3 +++
.../test_outer_join_with_cross_join.groovy | 18 ++++++++++++++++++
3 files changed, 21 insertions(+), 14 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
index e3834070d57..0d148bce44d 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/SingleNodePlanner.java
@@ -51,7 +51,6 @@ import org.apache.doris.analysis.TableRef;
import org.apache.doris.analysis.TableValuedFunctionRef;
import org.apache.doris.analysis.TupleDescriptor;
import org.apache.doris.analysis.TupleId;
-import org.apache.doris.analysis.TupleIsNullPredicate;
import org.apache.doris.catalog.AggregateFunction;
import org.apache.doris.catalog.AggregateType;
import org.apache.doris.catalog.Column;
@@ -78,7 +77,6 @@ import org.apache.doris.planner.external.paimon.PaimonScanNode;
import org.apache.doris.qe.ConnectContext;
import org.apache.doris.rewrite.mvrewrite.MVSelectFailedException;
import org.apache.doris.statistics.StatisticalType;
-import org.apache.doris.thrift.TNullSide;
import org.apache.doris.thrift.TPushAggOp;
import com.google.common.base.Preconditions;
@@ -1658,18 +1656,6 @@ public class SingleNodePlanner {
//set outputSmap to substitute literal in outputExpr
unionNode.setWithoutTupleIsNullOutputSmap(inlineViewRef.getSmap());
unionNode.setOutputSmap(inlineViewRef.getSmap(), analyzer);
- if (analyzer.isOuterJoined(inlineViewRef.getId())) {
- List<Expr> nullableRhs;
- if (analyzer.isOuterJoinedLeftSide(inlineViewRef.getId())) {
- nullableRhs = TupleIsNullPredicate.wrapExprs(inlineViewRef.getSmap().getRhs(),
- unionNode.getTupleIds(), TNullSide.LEFT, analyzer);
- } else {
- nullableRhs = TupleIsNullPredicate.wrapExprs(inlineViewRef.getSmap().getRhs(),
- unionNode.getTupleIds(), TNullSide.RIGHT, analyzer);
- }
- unionNode.setOutputSmap(
- new ExprSubstitutionMap(inlineViewRef.getSmap().getLhs(), nullableRhs), analyzer);
- }
return unionNode;
}
}
diff --git a/regression-test/data/correctness_p0/test_outer_join_with_cross_join.out b/regression-test/data/correctness_p0/test_outer_join_with_cross_join.out
index 0a5825a7bb0..e2f2ee9a508 100644
--- a/regression-test/data/correctness_p0/test_outer_join_with_cross_join.out
+++ b/regression-test/data/correctness_p0/test_outer_join_with_cross_join.out
@@ -5,3 +5,6 @@
-- !select2 --
1
+-- !select3 --
+1 2023 2023 1
+
diff --git a/regression-test/suites/correctness_p0/test_outer_join_with_cross_join.groovy b/regression-test/suites/correctness_p0/test_outer_join_with_cross_join.groovy
index 507d2755dc2..e0857c31c58 100644
--- a/regression-test/suites/correctness_p0/test_outer_join_with_cross_join.groovy
+++ b/regression-test/suites/correctness_p0/test_outer_join_with_cross_join.groovy
@@ -112,6 +112,24 @@ suite("test_outer_join_with_cross_join") {
inner join test_outer_join_with_cross_join_outerjoin_C as ref_4 on true;
"""
+ sql """set enable_nereids_planner=false;"""
+ qt_select3 """
+ WITH a As(
+ select
+ (case
+ when 'å¹´' = 'å¹´' then DATE_FORMAT(date_sub(concat('2023', '-01-01'), interval 0 year), '%Y')
+
+ end) as startdate,
+ (case
+ when 'å¹´' = 'å¹´' then DATE_FORMAT(date_sub(concat('2023', '-01-01'), interval 0 year), '%Y')
+
+ end) as enddate
+ )
+ select * from test_outer_join_with_cross_join_outerjoin_A DMR_POTM, a
+ right join ( select distinct a from test_outer_join_with_cross_join_outerjoin_B ) DD
+ on DMR_POTM.a =DD.a;
+ """
+
sql """
drop table if exists test_outer_join_with_cross_join_outerjoin_A;
"""
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 02/15: [chore](format) fix tablet_meta.cpp (#25410)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 01bca9d95b7daefc2815755d0b10272152b1d44f
Author: Dongyang Li <he...@qq.com>
AuthorDate: Fri Oct 13 17:58:54 2023 +0800
[chore](format) fix tablet_meta.cpp (#25410)
fix format error introduced by #25124
The clang format check had a bug before, so PR 25124 can pass the check at that time.
---
be/src/olap/tablet_meta.cpp | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/src/olap/tablet_meta.cpp b/be/src/olap/tablet_meta.cpp
index e1637d3d945..58e2876d81a 100644
--- a/be/src/olap/tablet_meta.cpp
+++ b/be/src/olap/tablet_meta.cpp
@@ -453,7 +453,7 @@ Status TabletMeta::_save_meta(DataDir* data_dir) {
<< ", schema_hash=" << schema_hash();
}
auto t3 = MonotonicMicros();
- auto cost = t3 - t1;
+ auto cost = t3 - t1;
if (cost > 1 * 1000 * 1000) {
LOG(INFO) << "save tablet(" << full_name() << ") meta too slow. serialize cost " << t2 - t1
<< "(us), serialized binary size: " << meta_binary.length()
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 13/15: [compile](fix) fix ubsan compile error (#25473)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5e02dc719f62d50bdf4cac1f3067016cf240ec75
Author: shuke <37...@users.noreply.github.com>
AuthorDate: Mon Oct 16 07:11:17 2023 -0500
[compile](fix) fix ubsan compile error (#25473)
---
be/CMakeLists.txt | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index 030118b3268..e10da917b5b 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -370,7 +370,7 @@ set(CXX_FLAGS_LSAN "${CXX_GCC_FLAGS} -O0 -fsanitize=leak -DLEAK_SANITIZER")
# Set the flags to the undefined behavior sanitizer, also known as "ubsan"
# Turn on sanitizer and debug symbols to get stack traces:
-set(CXX_FLAGS_UBSAN "${CXX_GCC_FLAGS} -O0 -fno-wrapv -mcmodel=large -fsanitize=undefined -DUNDEFINED_BEHAVIOR_SANITIZER")
+set(CXX_FLAGS_UBSAN "${CXX_GCC_FLAGS} -O0 -fno-wrapv -mcmodel=medium -fsanitize=undefined -DUNDEFINED_BEHAVIOR_SANITIZER")
# Set the flags to the thread sanitizer, also known as "tsan"
# Turn on sanitizer and debug symbols to get stack traces:
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 15/15: [fix](path gc) fix data dir path gc (#25420)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 11c81822e3e0f71f942c12c310162037c8159600
Author: yujun <yu...@gmail.com>
AuthorDate: Mon Oct 16 20:25:20 2023 +0800
[fix](path gc) fix data dir path gc (#25420)
---
be/src/olap/data_dir.cpp | 25 +++++++++++++++++--------
be/src/olap/data_dir.h | 8 +++++---
be/src/olap/olap_server.cpp | 8 +++-----
3 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/be/src/olap/data_dir.cpp b/be/src/olap/data_dir.cpp
index 9e01b75ba89..393db80d643 100644
--- a/be/src/olap/data_dir.cpp
+++ b/be/src/olap/data_dir.cpp
@@ -603,14 +603,25 @@ void DataDir::remove_pending_ids(const std::string& id) {
_pending_path_ids.erase(id);
}
-// gc unused tablet schemahash dir
-void DataDir::perform_path_gc_by_tablet() {
+void DataDir::perform_path_gc() {
std::unique_lock<std::mutex> lck(_check_path_mutex);
- _check_path_cv.wait(
- lck, [this] { return _stop_bg_worker || !_all_tablet_schemahash_paths.empty(); });
+ _check_path_cv.wait(lck, [this] {
+ return _stop_bg_worker || !_all_tablet_schemahash_paths.empty() ||
+ !_all_check_paths.empty();
+ });
if (_stop_bg_worker) {
return;
}
+
+ _perform_path_gc_by_tablet();
+ _perform_path_gc_by_rowsetid();
+}
+
+// gc unused tablet schemahash dir
+void DataDir::_perform_path_gc_by_tablet() {
+ if (_all_tablet_schemahash_paths.empty()) {
+ return;
+ }
LOG(INFO) << "start to path gc by tablet schemahash.";
int counter = 0;
for (const auto& path : _all_tablet_schemahash_paths) {
@@ -652,12 +663,10 @@ void DataDir::perform_path_gc_by_tablet() {
LOG(INFO) << "finished one time path gc by tablet.";
}
-void DataDir::perform_path_gc_by_rowsetid() {
+void DataDir::_perform_path_gc_by_rowsetid() {
// init the set of valid path
// validate the path in data dir
- std::unique_lock<std::mutex> lck(_check_path_mutex);
- _check_path_cv.wait(lck, [this] { return _stop_bg_worker || !_all_check_paths.empty(); });
- if (_stop_bg_worker) {
+ if (_all_check_paths.empty()) {
return;
}
LOG(INFO) << "start to path gc by rowsetid.";
diff --git a/be/src/olap/data_dir.h b/be/src/olap/data_dir.h
index 7d6684ad9b3..51261a19b1b 100644
--- a/be/src/olap/data_dir.h
+++ b/be/src/olap/data_dir.h
@@ -114,9 +114,7 @@ public:
// this is a producer function. After scan, it will notify the perform_path_gc function to gc
Status perform_path_scan();
- void perform_path_gc_by_rowsetid();
-
- void perform_path_gc_by_tablet();
+ void perform_path_gc();
void perform_remote_rowset_gc();
@@ -171,6 +169,10 @@ private:
bool _check_pending_ids(const std::string& id);
+ void _perform_path_gc_by_tablet();
+
+ void _perform_path_gc_by_rowsetid();
+
private:
std::atomic<bool> _stop_bg_worker = false;
diff --git a/be/src/olap/olap_server.cpp b/be/src/olap/olap_server.cpp
index 369b0c48d87..401495bed2b 100644
--- a/be/src/olap/olap_server.cpp
+++ b/be/src/olap/olap_server.cpp
@@ -358,11 +358,8 @@ void StorageEngine::_path_gc_thread_callback(DataDir* data_dir) {
LOG(INFO) << "try to start path gc thread!";
int32_t interval = config::path_gc_check_interval_second;
do {
- LOG(INFO) << "try to perform path gc by tablet!";
- data_dir->perform_path_gc_by_tablet();
-
- LOG(INFO) << "try to perform path gc by rowsetid!";
- data_dir->perform_path_gc_by_rowsetid();
+ LOG(INFO) << "try to perform path gc!";
+ data_dir->perform_path_gc();
interval = config::path_gc_check_interval_second;
if (interval <= 0) {
@@ -372,6 +369,7 @@ void StorageEngine::_path_gc_thread_callback(DataDir* data_dir) {
}
} while (!_stop_background_threads_latch.wait_for(std::chrono::seconds(interval)) &&
!k_doris_exit);
+ LOG(INFO) << "stop path gc thread!";
}
void StorageEngine::_path_scan_thread_callback(DataDir* data_dir) {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 12/15: [Improvement](materialized-view) set job failed when toAgentTaskRequest meet error (#25358)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 56500f7b0f8b3462f75a70a416a5c18edbaa90e7
Author: Pxl <px...@qq.com>
AuthorDate: Mon Oct 16 20:10:52 2023 +0800
[Improvement](materialized-view) set job failed when toAgentTaskRequest meet error (#25358)
set job failed when toAgentTaskRequest meet error
---
.../java/org/apache/doris/alter/RollupJobV2.java | 2 +-
.../org/apache/doris/alter/SchemaChangeJobV2.java | 4 +-
.../doris/analysis/CreateMaterializedViewStmt.java | 45 +++-
.../apache/doris/analysis/ExpressionFunctions.java | 2 +-
.../mv/AbstractSelectMaterializedIndexRule.java | 8 -
.../mv/SelectMaterializedIndexWithAggregate.java | 36 +--
.../java/org/apache/doris/task/AgentBatchTask.java | 7 +-
.../multiple_ssb_between/multiple_ssb_between.out | 24 ++
.../multiple_ssb_between.groovy | 244 +++++++++++++++++++++
9 files changed, 337 insertions(+), 35 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java b/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java
index cf70af5b122..afcbe261bb4 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/RollupJobV2.java
@@ -526,7 +526,7 @@ public class RollupJobV2 extends AlterJobV2 implements GsonPostProcessable {
if (task.getFailedTimes() > 0) {
task.setFinished(true);
AgentTaskQueue.removeTask(task.getBackendId(), TTaskType.ALTER, task.getSignature());
- LOG.warn("rollup task failed after try three times: " + task.getErrorMsg());
+ LOG.warn("rollup task failed: " + task.getErrorMsg());
if (!failedAgentTasks.containsKey(task.getTabletId())) {
failedAgentTasks.put(task.getTabletId(), Lists.newArrayList(task));
} else {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
index 49c6610b354..574a1bf129f 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeJobV2.java
@@ -519,10 +519,10 @@ public class SchemaChangeJobV2 extends AlterJobV2 {
LOG.info("schema change tasks not finished. job: {}", jobId);
List<AgentTask> tasks = schemaChangeBatchTask.getUnfinishedTasks(2000);
for (AgentTask task : tasks) {
- if (task.getFailedTimes() >= 3) {
+ if (task.getFailedTimes() > 0) {
task.setFinished(true);
AgentTaskQueue.removeTask(task.getBackendId(), TTaskType.ALTER, task.getSignature());
- LOG.warn("schema change task failed after try three times: " + task.getErrorMsg());
+ LOG.warn("schema change task failed: " + task.getErrorMsg());
if (!failedAgentTasks.containsKey(task.getTabletId())) {
failedAgentTasks.put(task.getTabletId(), Lists.newArrayList(task));
} else {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java
index 195e67485ea..2f7a23ed674 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/CreateMaterializedViewStmt.java
@@ -32,9 +32,11 @@ import org.apache.doris.common.DdlException;
import org.apache.doris.common.FeConstants;
import org.apache.doris.common.FeNameFormat;
import org.apache.doris.common.UserException;
+import org.apache.doris.rewrite.ExprRewriter;
import org.apache.doris.rewrite.mvrewrite.CountFieldToSum;
import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableSet;
import com.google.common.collect.Lists;
import com.google.common.collect.Maps;
import org.apache.logging.log4j.LogManager;
@@ -79,6 +81,9 @@ public class CreateMaterializedViewStmt extends DdlStmt {
FN_NAME_TO_PATTERN.put(FunctionSet.HLL_UNION, new MVColumnHLLUnionPattern());
}
+ public static final ImmutableSet<String> invalidFn = ImmutableSet.of("now", "current_time", "current_date",
+ "utc_timestamp", "uuid", "random", "unix_timestamp", "curdate");
+
private String mvName;
private SelectStmt selectStmt;
private Map<String, String> properties;
@@ -158,14 +163,50 @@ public class CreateMaterializedViewStmt extends DdlStmt {
return selectStmt.getWhereClause();
}
+ private void checkExprValidInMv(Expr expr, String functionName) throws AnalysisException {
+ if (!isReplay && expr.haveFunction(functionName)) {
+ throw new AnalysisException("The materialized view contain " + functionName + " is disallowed");
+ }
+ }
+
+ private void checkExprValidInMv(Expr expr) throws AnalysisException {
+ if (isReplay) {
+ return;
+ }
+ for (String function : invalidFn) {
+ checkExprValidInMv(expr, function);
+ }
+ }
+
+ private void checkExprValidInMv() throws AnalysisException {
+ if (selectStmt.getWhereClause() != null) {
+ checkExprValidInMv(selectStmt.getWhereClause());
+ }
+ SelectList selectList = selectStmt.getSelectList();
+ for (SelectListItem selectListItem : selectList.getItems()) {
+ checkExprValidInMv(selectListItem.getExpr());
+ }
+ }
+
@Override
public void analyze(Analyzer analyzer) throws UserException {
super.analyze(analyzer);
+
+ checkExprValidInMv();
+
FeNameFormat.checkTableName(mvName);
rewriteToBitmapWithCheck();
// TODO(ml): The mv name in from clause should pass the analyze without error.
selectStmt.forbiddenMVRewrite();
selectStmt.analyze(analyzer);
+
+ ExprRewriter rewriter = analyzer.getExprRewriter();
+ rewriter.reset();
+ selectStmt.rewriteExprs(rewriter);
+ selectStmt.reset();
+ analyzer = new Analyzer(analyzer.getEnv(), analyzer.getContext());
+ selectStmt.analyze(analyzer);
+
if (selectStmt.getAggInfo() != null) {
mvKeysType = KeysType.AGG_KEYS;
}
@@ -227,10 +268,6 @@ public class CreateMaterializedViewStmt extends DdlStmt {
+ selectListItemExpr.toSql());
}
- if (!isReplay && selectListItemExpr.haveFunction("curdate")) {
- throw new AnalysisException(
- "The materialized view contain curdate is disallowed");
- }
if (selectListItemExpr instanceof FunctionCallExpr
&& ((FunctionCallExpr) selectListItemExpr).isAggregateFunction()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExpressionFunctions.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExpressionFunctions.java
index f7ace7ce4e8..a45954a9b93 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/ExpressionFunctions.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/ExpressionFunctions.java
@@ -51,7 +51,7 @@ public enum ExpressionFunctions {
private static final Logger LOG = LogManager.getLogger(ExpressionFunctions.class);
private ImmutableMultimap<String, FEFunctionInvoker> functions;
- private static final Set<String> unfixedFn = ImmutableSet.of(
+ public static final Set<String> unfixedFn = ImmutableSet.of(
"now",
"current_time",
"current_date",
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/AbstractSelectMaterializedIndexRule.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/AbstractSelectMaterializedIndexRule.java
index 45453f760df..52ba749cf60 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/AbstractSelectMaterializedIndexRule.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/AbstractSelectMaterializedIndexRule.java
@@ -570,14 +570,6 @@ public abstract class AbstractSelectMaterializedIndexRule {
return visit(aggregateFunction, context);
}
- @Override
- public Expression visitAlias(Alias alias, Void context) {
- if (mvNameToMvSlot.containsKey(alias.toSlot().toSql())) {
- return mvNameToMvSlot.get(alias.toSlot().toSql());
- }
- return visit(alias, context);
- }
-
@Override
public Expression visitScalarFunction(ScalarFunction scalarFunction, Void context) {
List<Expression> newChildrenWithoutCast = scalarFunction.children().stream()
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/SelectMaterializedIndexWithAggregate.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/SelectMaterializedIndexWithAggregate.java
index 1ccae2e0e33..b4286472dfd 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/SelectMaterializedIndexWithAggregate.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/mv/SelectMaterializedIndexWithAggregate.java
@@ -124,7 +124,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
.replace(agg.withChildren(mvPlan), mvPlan);
} else {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
new LogicalAggregate<>(
agg.getGroupByExpressions(),
@@ -168,12 +168,12 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
if (result.exprRewriteMap.isEmpty()) {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
agg.withChildren(filter.withChildren(mvPlan)), mvPlan));
} else {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
new LogicalAggregate<>(
agg.getGroupByExpressions(),
@@ -226,7 +226,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
generateNewOutputsWithMvOutputs(mvPlan, newProjectList),
scan.withMaterializedIndexSelected(result.preAggStatus, result.indexId));
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
new LogicalAggregate<>(
agg.getGroupByExpressions(),
@@ -271,7 +271,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
if (result.exprRewriteMap.isEmpty()) {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
agg.withChildren(
project.withProjectsAndChild(
@@ -285,7 +285,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
filter.withChildren(mvPlan));
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
new LogicalAggregate<>(
agg.getGroupByExpressions(),
@@ -328,7 +328,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
if (result.exprRewriteMap.isEmpty()) {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
agg.withChildren(
filter.withChildren(
@@ -342,7 +342,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
generateNewOutputsWithMvOutputs(mvPlan, newProjectList), mvPlan);
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
new LogicalAggregate<>(
agg.getGroupByExpressions(),
@@ -374,7 +374,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
if (result.exprRewriteMap.isEmpty()) {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
agg.withChildren(
repeat.withAggOutputAndChild(
@@ -382,7 +382,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
), mvPlan));
} else {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
new LogicalAggregate<>(
agg.getGroupByExpressions(),
@@ -428,7 +428,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
if (result.exprRewriteMap.isEmpty()) {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
agg.withChildren(
repeat.withAggOutputAndChild(
@@ -437,7 +437,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
)), mvPlan));
} else {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
new LogicalAggregate<>(
agg.getGroupByExpressions(),
@@ -480,7 +480,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
if (result.exprRewriteMap.isEmpty()) {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
agg.withChildren(
repeat.withAggOutputAndChild(
@@ -496,7 +496,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
generateNewOutputsWithMvOutputs(mvPlan, newProjectList),
mvPlan);
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
new LogicalAggregate<>(
agg.getGroupByExpressions(),
@@ -545,7 +545,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
if (result.exprRewriteMap.isEmpty()) {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
agg.withChildren(
repeat.withAggOutputAndChild(
@@ -564,7 +564,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
filter.withChildren(mvPlan));
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
new LogicalAggregate<>(
agg.getGroupByExpressions(),
@@ -611,7 +611,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
if (result.exprRewriteMap.isEmpty()) {
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
agg.withChildren(
repeat.withAggOutputAndChild(
@@ -630,7 +630,7 @@ public class SelectMaterializedIndexWithAggregate extends AbstractSelectMaterial
scan.withMaterializedIndexSelected(result.preAggStatus, result.indexId));
return new LogicalProject<>(
- generateProjectsAlias(agg.getOutput(), slotContext),
+ generateProjectsAlias(agg.getOutputs(), slotContext),
new ReplaceExpressions(slotContext).replace(
new LogicalAggregate<>(
agg.getGroupByExpressions(),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/task/AgentBatchTask.java b/fe/fe-core/src/main/java/org/apache/doris/task/AgentBatchTask.java
index 3c12160610c..aa654fcd21a 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/task/AgentBatchTask.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/task/AgentBatchTask.java
@@ -165,7 +165,12 @@ public class AgentBatchTask implements Runnable {
client = ClientPool.backendPool.borrowObject(address);
List<TAgentTaskRequest> agentTaskRequests = new LinkedList<TAgentTaskRequest>();
for (AgentTask task : tasks) {
- agentTaskRequests.add(toAgentTaskRequest(task));
+ try {
+ agentTaskRequests.add(toAgentTaskRequest(task));
+ } catch (Exception e) {
+ task.failed();
+ throw e;
+ }
}
client.submitTasks(agentTaskRequests);
if (LOG.isDebugEnabled()) {
diff --git a/regression-test/data/mv_p0/ssb/multiple_ssb_between/multiple_ssb_between.out b/regression-test/data/mv_p0/ssb/multiple_ssb_between/multiple_ssb_between.out
new file mode 100644
index 00000000000..00e05ab400e
--- /dev/null
+++ b/regression-test/data/mv_p0/ssb/multiple_ssb_between/multiple_ssb_between.out
@@ -0,0 +1,24 @@
+-- This file is automatically generated. You should know what you did if you want to edit this
+-- !select_star --
+1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2023-06-09 shipmode name address city nation AMERICA phone mktsegment name address city nation AMERICA phone name MFGR#1 category brand color type 4 container
+2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2023-06-09 shipmode name address city nation region phone mktsegment name address city nation region phone name mfgr category brand color type 4 container
+19920101 1 1 1 1 1 1 1 1 1 1 100 1 1 1 2023-06-09 ASIA ASIA ASIA ASIA ASIA ASIA ASIA ASIA ASIA ASIA ASIA ASIA ASIA ASIA ASIA MFGR#12 MFGR#12 brand color type 4 container
+19930101 1 1 1 1 1 1 1 1 1 1 100 1 1 1 2023-06-09 shipmode name address city nation AMERICA phone mktsegment name address city nation AMERICA phone name MFGR#1 category brand color type 4 container
+19930101 1 1 1 1 1 1 1 1 1 1 100 1 1 1 2023-06-09 shipmode name address city nation AMERICA phone mktsegment name address city nation AMERICA phone name MFGR#12 MFGR#12 brand color type 4 container
+19930101 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2023-06-09 shipmode name address city nation region phone mktsegment name address city nation region phone name mfgr category brand color type 4 container
+19930101 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2023-06-09 shipmode name address city nation region phone mktsegment name address city nation region phone name mfgr category brand color type 4 container
+19930101 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2023-06-09 shipmode name address city nation region phone mktsegment name address city nation region phone name mfgr category brand color type 4 container
+
+-- !select_q_1_1 --
+12
+
+-- !select_q_2_1 --
+1 1993 brand
+
+-- !select_q_3_1 --
+ASIA ASIA 1992 1
+
+-- !select_q_4_1 --
+0 nation 0
+1993 nation 0
+
diff --git a/regression-test/suites/mv_p0/ssb/multiple_ssb_between/multiple_ssb_between.groovy b/regression-test/suites/mv_p0/ssb/multiple_ssb_between/multiple_ssb_between.groovy
new file mode 100644
index 00000000000..3e787d9aee0
--- /dev/null
+++ b/regression-test/suites/mv_p0/ssb/multiple_ssb_between/multiple_ssb_between.groovy
@@ -0,0 +1,244 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import org.codehaus.groovy.runtime.IOGroovyMethods
+
+suite ("multiple_ssb_between") {
+ sql """ DROP TABLE IF EXISTS lineorder_flat; """
+ sql """set enable_nereids_planner=true"""
+ sql """SET enable_fallback_to_original_planner=false"""
+
+ sql """
+ CREATE TABLE IF NOT EXISTS `lineorder_flat` (
+ `LO_ORDERDATE` int(11) NOT NULL COMMENT "",
+ `LO_ORDERKEY` int(11) NOT NULL COMMENT "",
+ `LO_LINENUMBER` tinyint(4) NOT NULL COMMENT "",
+ `LO_CUSTKEY` int(11) NOT NULL COMMENT "",
+ `LO_PARTKEY` int(11) NOT NULL COMMENT "",
+ `LO_SUPPKEY` int(11) NOT NULL COMMENT "",
+ `LO_ORDERPRIORITY` varchar(100) NOT NULL COMMENT "",
+ `LO_SHIPPRIORITY` tinyint(4) NOT NULL COMMENT "",
+ `LO_QUANTITY` tinyint(4) NOT NULL COMMENT "",
+ `LO_EXTENDEDPRICE` int(11) NOT NULL COMMENT "",
+ `LO_ORDTOTALPRICE` int(11) NOT NULL COMMENT "",
+ `LO_DISCOUNT` tinyint(4) NOT NULL COMMENT "",
+ `LO_REVENUE` int(11) NOT NULL COMMENT "",
+ `LO_SUPPLYCOST` int(11) NOT NULL COMMENT "",
+ `LO_TAX` tinyint(4) NOT NULL COMMENT "",
+ `LO_COMMITDATE` date NOT NULL COMMENT "",
+ `LO_SHIPMODE` varchar(100) NOT NULL COMMENT "",
+ `C_NAME` varchar(100) NOT NULL COMMENT "",
+ `C_ADDRESS` varchar(100) NOT NULL COMMENT "",
+ `C_CITY` varchar(100) NOT NULL COMMENT "",
+ `C_NATION` varchar(100) NOT NULL COMMENT "",
+ `C_REGION` varchar(100) NOT NULL COMMENT "",
+ `C_PHONE` varchar(100) NOT NULL COMMENT "",
+ `C_MKTSEGMENT` varchar(100) NOT NULL COMMENT "",
+ `S_NAME` varchar(100) NOT NULL COMMENT "",
+ `S_ADDRESS` varchar(100) NOT NULL COMMENT "",
+ `S_CITY` varchar(100) NOT NULL COMMENT "",
+ `S_NATION` varchar(100) NOT NULL COMMENT "",
+ `S_REGION` varchar(100) NOT NULL COMMENT "",
+ `S_PHONE` varchar(100) NOT NULL COMMENT "",
+ `P_NAME` varchar(100) NOT NULL COMMENT "",
+ `P_MFGR` varchar(100) NOT NULL COMMENT "",
+ `P_CATEGORY` varchar(100) NOT NULL COMMENT "",
+ `P_BRAND` varchar(100) NOT NULL COMMENT "",
+ `P_COLOR` varchar(100) NOT NULL COMMENT "",
+ `P_TYPE` varchar(100) NOT NULL COMMENT "",
+ `P_SIZE` tinyint(4) NOT NULL COMMENT "",
+ `P_CONTAINER` varchar(100) NOT NULL COMMENT ""
+ ) ENGINE=OLAP
+ DUPLICATE KEY(`LO_ORDERDATE`, `LO_ORDERKEY`)
+ COMMENT "OLAP"
+ PARTITION BY RANGE(`LO_ORDERDATE`)
+ (PARTITION p1992 VALUES [("-2147483648"), ("19930101")),
+ PARTITION p1993 VALUES [("19930101"), ("19940101")),
+ PARTITION p1994 VALUES [("19940101"), ("19950101")),
+ PARTITION p1995 VALUES [("19950101"), ("19960101")),
+ PARTITION p1996 VALUES [("19960101"), ("19970101")),
+ PARTITION p1997 VALUES [("19970101"), ("19980101")),
+ PARTITION p1998 VALUES [("19980101"), ("19990101")))
+ DISTRIBUTED BY HASH(`LO_ORDERKEY`) BUCKETS 48
+ PROPERTIES (
+ "replication_num" = "1",
+ "colocate_with" = "groupxx1",
+ "in_memory" = "false",
+ "storage_format" = "DEFAULT"
+ );
+ """
+
+ sql """INSERT INTO lineorder_flat (LO_ORDERDATE, LO_ORDERKEY, LO_LINENUMBER, LO_CUSTKEY, LO_PARTKEY, LO_SUPPKEY, LO_ORDERPRIORITY, LO_SHIPPRIORITY, LO_QUANTITY, LO_EXTENDEDPRICE, LO_ORDTOTALPRICE, LO_DISCOUNT, LO_REVENUE, LO_SUPPLYCOST, LO_TAX, LO_COMMITDATE, LO_SHIPMODE, C_NAME, C_ADDRESS, C_CITY, C_NATION, C_REGION, C_PHONE, C_MKTSEGMENT, S_NAME, S_ADDRESS, S_CITY, S_NATION, S_REGION, S_PHONE, P_NAME, P_MFGR, P_CATEGORY, P_BRAND, P_COLOR,P_TYPE,P_SIZE,P_CONTAINER) VALUES (19930101 [...]
+
+ test {
+ sql """create materialized view lineorder_q_1_1 as
+ SELECT LO_ORDERKEY, SUM(LO_EXTENDEDPRICE * LO_DISCOUNT)
+ FROM lineorder_flat GROUP BY
+ LO_ORDERKEY, LO_ORDERDATE, LO_DISCOUNT, LO_QUANTITY;"""
+ exception "not in select list"
+ }
+
+ createMV ("""create materialized view lineorder_q_1_1 as
+ SELECT LO_ORDERKEY, SUM(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
+ FROM lineorder_flat
+ WHERE
+ LO_ORDERDATE >= 19930101
+ AND LO_ORDERDATE <= 19931231
+ AND LO_DISCOUNT BETWEEN 1 AND 3
+ AND LO_QUANTITY < 25
+ GROUP BY
+ LO_ORDERKEY;""")
+
+ createMV ("""create materialized view lineorder_q_2_1 as
+ SELECT
+ (LO_ORDERDATE DIV 10000) AS YEAR,
+ P_BRAND,
+ SUM(LO_REVENUE)
+ FROM lineorder_flat
+ WHERE P_CATEGORY = 'MFGR#12' AND S_REGION = 'AMERICA'
+ GROUP BY YEAR, P_BRAND
+ ORDER BY YEAR, P_BRAND;""")
+
+ createMV ("""create materialized view lineorder_q_3_1 as
+ SELECT
+ C_NATION,
+ S_NATION, (LO_ORDERDATE DIV 10000) AS YEAR,
+ SUM(LO_REVENUE) AS revenue
+ FROM lineorder_flat
+ WHERE
+ C_REGION = 'ASIA'
+ AND S_REGION = 'ASIA'
+ AND LO_ORDERDATE >= 19920101
+ AND LO_ORDERDATE <= 19971231
+ GROUP BY C_NATION, S_NATION, YEAR;""")
+
+ createMV ("""create materialized view lineorder_q_4_1 as
+ SELECT (LO_ORDERDATE DIV 10000) AS YEAR,
+ C_NATION,
+ SUM(LO_REVENUE - LO_SUPPLYCOST) AS profit
+ FROM lineorder_flat
+ WHERE
+ C_REGION = 'AMERICA'
+ AND S_REGION = 'AMERICA'
+ AND P_MFGR IN ('MFGR#1', 'MFGR#2')
+ GROUP BY YEAR, C_NATION
+ ORDER BY YEAR ASC, C_NATION ASC;""")
+
+ sql """INSERT INTO lineorder_flat (LO_ORDERDATE, LO_ORDERKEY, LO_LINENUMBER, LO_CUSTKEY, LO_PARTKEY, LO_SUPPKEY, LO_ORDERPRIORITY, LO_SHIPPRIORITY, LO_QUANTITY, LO_EXTENDEDPRICE, LO_ORDTOTALPRICE, LO_DISCOUNT, LO_REVENUE, LO_SUPPLYCOST, LO_TAX, LO_COMMITDATE, LO_SHIPMODE,C_NAME,C_ADDRESS,C_CITY,C_NATION,C_REGION,C_PHONE,C_MKTSEGMENT,S_NAME,S_ADDRESS,S_CITY,S_NATION,S_REGION,S_PHONE,P_NAME,P_MFGR,P_CATEGORY,P_BRAND,P_COLOR,P_TYPE,P_SIZE,P_CONTAINER) VALUES (19930101 , 2 , 2 , 2 , 2 , [...]
+
+ sql """INSERT INTO lineorder_flat (LO_ORDERDATE, LO_ORDERKEY, LO_LINENUMBER, LO_CUSTKEY, LO_PARTKEY, LO_SUPPKEY, LO_ORDERPRIORITY, LO_SHIPPRIORITY, LO_QUANTITY, LO_EXTENDEDPRICE, LO_ORDTOTALPRICE, LO_DISCOUNT, LO_REVENUE, LO_SUPPLYCOST, LO_TAX, LO_COMMITDATE, LO_SHIPMODE, C_NAME, C_ADDRESS, C_CITY, C_NATION, C_REGION, C_PHONE, C_MKTSEGMENT, S_NAME, S_ADDRESS, S_CITY, S_NATION, S_REGION, S_PHONE, P_NAME, P_MFGR, P_CATEGORY, P_BRAND, P_COLOR,P_TYPE,P_SIZE,P_CONTAINER) VALUES (19930101 [...]
+
+ sql """INSERT INTO lineorder_flat (LO_ORDERDATE, LO_ORDERKEY, LO_LINENUMBER, LO_CUSTKEY, LO_PARTKEY, LO_SUPPKEY, LO_ORDERPRIORITY, LO_SHIPPRIORITY, LO_QUANTITY, LO_EXTENDEDPRICE, LO_ORDTOTALPRICE, LO_DISCOUNT, LO_REVENUE, LO_SUPPLYCOST, LO_TAX, LO_COMMITDATE, LO_SHIPMODE,C_NAME,C_ADDRESS,C_CITY,C_NATION,C_REGION,C_PHONE,C_MKTSEGMENT,S_NAME,S_ADDRESS,S_CITY,S_NATION,S_REGION,S_PHONE,P_NAME,P_MFGR,P_CATEGORY,P_BRAND,P_COLOR,P_TYPE,P_SIZE,P_CONTAINER) VALUES (19930101 , 2 , 2 , 2 , 2 , [...]
+
+ sql """INSERT INTO lineorder_flat (LO_ORDERDATE, LO_ORDERKEY, LO_LINENUMBER, LO_CUSTKEY, LO_PARTKEY, LO_SUPPKEY, LO_ORDERPRIORITY, LO_SHIPPRIORITY, LO_QUANTITY, LO_EXTENDEDPRICE, LO_ORDTOTALPRICE, LO_DISCOUNT, LO_REVENUE, LO_SUPPLYCOST, LO_TAX, LO_COMMITDATE, LO_SHIPMODE, C_NAME, C_ADDRESS, C_CITY, C_NATION, C_REGION, C_PHONE, C_MKTSEGMENT, S_NAME, S_ADDRESS, S_CITY, S_NATION, S_REGION, S_PHONE, P_NAME, P_MFGR, P_CATEGORY, P_BRAND, P_COLOR,P_TYPE,P_SIZE,P_CONTAINER) VALUES (19920101 [...]
+
+ sql """INSERT INTO lineorder_flat (LO_ORDERDATE, LO_ORDERKEY, LO_LINENUMBER, LO_CUSTKEY, LO_PARTKEY, LO_SUPPKEY, LO_ORDERPRIORITY, LO_SHIPPRIORITY, LO_QUANTITY, LO_EXTENDEDPRICE, LO_ORDTOTALPRICE, LO_DISCOUNT, LO_REVENUE, LO_SUPPLYCOST, LO_TAX, LO_COMMITDATE, LO_SHIPMODE,C_NAME,C_ADDRESS,C_CITY,C_NATION,C_REGION,C_PHONE,C_MKTSEGMENT,S_NAME,S_ADDRESS,S_CITY,S_NATION,S_REGION,S_PHONE,P_NAME,P_MFGR,P_CATEGORY,P_BRAND,P_COLOR,P_TYPE,P_SIZE,P_CONTAINER) VALUES (19930101 , 2 , 2 , 2 , 2 , [...]
+
+ sql """INSERT INTO lineorder_flat (LO_ORDERDATE, LO_ORDERKEY, LO_LINENUMBER, LO_CUSTKEY, LO_PARTKEY, LO_SUPPKEY, LO_ORDERPRIORITY, LO_SHIPPRIORITY, LO_QUANTITY, LO_EXTENDEDPRICE, LO_ORDTOTALPRICE, LO_DISCOUNT, LO_REVENUE, LO_SUPPLYCOST, LO_TAX, LO_COMMITDATE, LO_SHIPMODE,C_NAME,C_ADDRESS,C_CITY,C_NATION,C_REGION,C_PHONE,C_MKTSEGMENT,S_NAME,S_ADDRESS,S_CITY,S_NATION,S_REGION,S_PHONE,P_NAME,P_MFGR,P_CATEGORY,P_BRAND,P_COLOR,P_TYPE,P_SIZE,P_CONTAINER) VALUES (2 , 2 , 2 , 2 , 2 , 2 ,'2', [...]
+
+ sql """INSERT INTO lineorder_flat (LO_ORDERDATE, LO_ORDERKEY, LO_LINENUMBER, LO_CUSTKEY, LO_PARTKEY, LO_SUPPKEY, LO_ORDERPRIORITY, LO_SHIPPRIORITY, LO_QUANTITY, LO_EXTENDEDPRICE, LO_ORDTOTALPRICE, LO_DISCOUNT, LO_REVENUE, LO_SUPPLYCOST, LO_TAX, LO_COMMITDATE, LO_SHIPMODE, C_NAME, C_ADDRESS, C_CITY, C_NATION, C_REGION, C_PHONE, C_MKTSEGMENT, S_NAME, S_ADDRESS, S_CITY, S_NATION, S_REGION, S_PHONE, P_NAME, P_MFGR, P_CATEGORY, P_BRAND, P_COLOR,P_TYPE,P_SIZE,P_CONTAINER) VALUES (1 , 1 , 1 [...]
+
+ qt_select_star "select * from lineorder_flat order by 1,2;"
+
+ explain {
+ sql("""SELECT SUM(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
+ FROM lineorder_flat
+ WHERE
+ LO_ORDERDATE >= 19930101
+ AND LO_ORDERDATE <= 19931231
+ AND LO_DISCOUNT BETWEEN 1 AND 3
+ AND LO_QUANTITY < 25;""")
+ contains "(lineorder_q_1_1)"
+ }
+ qt_select_q_1_1 """SELECT SUM(LO_EXTENDEDPRICE * LO_DISCOUNT) AS revenue
+ FROM lineorder_flat
+ WHERE
+ LO_ORDERDATE >= 19930101
+ AND LO_ORDERDATE <= 19931231
+ AND LO_DISCOUNT BETWEEN 1 AND 3
+ AND LO_QUANTITY < 25;"""
+
+ explain {
+ sql("""SELECT
+ SUM(LO_REVENUE), (LO_ORDERDATE DIV 10000) AS YEAR,
+ P_BRAND
+ FROM lineorder_flat
+ WHERE P_CATEGORY = 'MFGR#12' AND S_REGION = 'AMERICA'
+ GROUP BY (LO_ORDERDATE DIV 10000), P_BRAND
+ ORDER BY YEAR, P_BRAND;""")
+ contains "(lineorder_q_2_1)"
+ }
+ qt_select_q_2_1 """SELECT
+ SUM(LO_REVENUE), (LO_ORDERDATE DIV 10000) AS YEAR,
+ P_BRAND
+ FROM lineorder_flat
+ WHERE P_CATEGORY = 'MFGR#12' AND S_REGION = 'AMERICA'
+ GROUP BY YEAR, P_BRAND
+ ORDER BY YEAR, P_BRAND;"""
+
+ explain {
+ sql("""SELECT
+ C_NATION,
+ S_NATION, (LO_ORDERDATE DIV 10000) AS YEAR,
+ SUM(LO_REVENUE) AS revenue
+ FROM lineorder_flat
+ WHERE
+ C_REGION = 'ASIA'
+ AND S_REGION = 'ASIA'
+ AND LO_ORDERDATE >= 19920101
+ AND LO_ORDERDATE <= 19971231
+ GROUP BY C_NATION, S_NATION, YEAR
+ ORDER BY YEAR ASC, revenue DESC;""")
+ contains "(lineorder_q_3_1)"
+ }
+ qt_select_q_3_1 """SELECT
+ C_NATION,
+ S_NATION, (LO_ORDERDATE DIV 10000) AS YEAR,
+ SUM(LO_REVENUE) AS revenue
+ FROM lineorder_flat
+ WHERE
+ C_REGION = 'ASIA'
+ AND S_REGION = 'ASIA'
+ AND LO_ORDERDATE >= 19920101
+ AND LO_ORDERDATE <= 19971231
+ GROUP BY C_NATION, S_NATION, YEAR
+ ORDER BY YEAR ASC, revenue DESC;"""
+
+ explain {
+ sql("""SELECT (LO_ORDERDATE DIV 10000) AS YEAR,
+ C_NATION,
+ SUM(LO_REVENUE - LO_SUPPLYCOST) AS profit
+ FROM lineorder_flat
+ WHERE
+ C_REGION = 'AMERICA'
+ AND S_REGION = 'AMERICA'
+ AND P_MFGR IN ('MFGR#1', 'MFGR#2')
+ GROUP BY YEAR, C_NATION
+ ORDER BY YEAR ASC, C_NATION ASC;""")
+ contains "(lineorder_q_4_1)"
+ }
+ qt_select_q_4_1 """SELECT (LO_ORDERDATE DIV 10000) AS YEAR,
+ C_NATION,
+ SUM(LO_REVENUE - LO_SUPPLYCOST) AS profit
+ FROM lineorder_flat
+ WHERE
+ C_REGION = 'AMERICA'
+ AND S_REGION = 'AMERICA'
+ AND P_MFGR IN ('MFGR#1', 'MFGR#2')
+ GROUP BY YEAR, C_NATION
+ ORDER BY YEAR ASC, C_NATION ASC;"""
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 10/15: [Chore](sink) add index number check for table sink (#25461)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit fc37baaf53b773235a90799e492da5cbe022d0cf
Author: Pxl <px...@qq.com>
AuthorDate: Mon Oct 16 17:03:26 2023 +0800
[Chore](sink) add index number check for table sink (#25461)
add index number check for table sink
---
.../src/main/java/org/apache/doris/catalog/OlapTable.java | 4 ++++
.../src/main/java/org/apache/doris/planner/OlapTableSink.java | 10 ++++++++++
.../test/java/org/apache/doris/planner/OlapTableSinkTest.java | 6 ++++++
3 files changed, 20 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
index bc0073f7ef9..0f6ffc3cf6b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java
@@ -620,6 +620,10 @@ public class OlapTable extends Table {
return Status.OK;
}
+ public int getIndexNumber() {
+ return indexIdToMeta.size();
+ }
+
public Map<Long, MaterializedIndexMeta> getIndexIdToMeta() {
return indexIdToMeta;
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java
index ceee31a721d..d073f0b1394 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/OlapTableSink.java
@@ -166,6 +166,16 @@ public class OlapTableSink extends DataSink {
// must called after tupleDescriptor is computed
public void complete(Analyzer analyzer) throws UserException {
+ for (Long partitionId : partitionIds) {
+ Partition partition = dstTable.getPartition(partitionId);
+ if (dstTable.getIndexNumber() != partition.getMaterializedIndices(IndexExtState.ALL).size()) {
+ throw new UserException(
+ "table's index number not equal with partition's index number. table's index number="
+ + dstTable.getIndexIdToMeta().size() + ", partition's index number="
+ + partition.getMaterializedIndices(IndexExtState.ALL).size());
+ }
+ }
+
TOlapTableSink tSink = tDataSink.getOlapTableSink();
tSink.setTableId(dstTable.getId());
diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/OlapTableSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/OlapTableSinkTest.java
index dc98026a00e..0ebea0e3b35 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/planner/OlapTableSinkTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/planner/OlapTableSinkTest.java
@@ -92,6 +92,8 @@ public class OlapTableSinkTest {
new Expectations() {
{
+ dstTable.getIndexNumber();
+ result = 1;
dstTable.getId();
result = 1;
dstTable.getPartitionInfo();
@@ -128,6 +130,8 @@ public class OlapTableSinkTest {
new Expectations() {
{
+ dstTable.getIndexNumber();
+ result = 1;
dstTable.getId();
result = 1;
dstTable.getPartitionInfo();
@@ -190,6 +194,8 @@ public class OlapTableSinkTest {
new Expectations() {
{
+ dstTable.getIndexNumber();
+ result = 1;
dstTable.getId();
result = 1;
dstTable.getPartitionInfo();
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 03/15: [chore](build) Fix compilation errors reported by GCC-13 (#25439)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 6be4994e80e6647187c0dd03d9c4c8adfde31c08
Author: Adonis Ling <ad...@gmail.com>
AuthorDate: Sun Oct 15 07:57:36 2023 -0500
[chore](build) Fix compilation errors reported by GCC-13 (#25439)
1. Fix lots of compilation errors reported by GCC-13.
2. Fix the workflow BE UT (macOS).
---
.github/workflows/be-ut-mac.yml | 4 ++--
be/CMakeLists.txt | 3 ++-
be/src/olap/memtable.cpp | 2 +-
be/src/util/bitmap_value.h | 1 +
be/src/util/perf_counters.cpp | 2 +-
be/src/vec/aggregate_functions/aggregate_function_state_merge.h | 2 ++
be/src/vec/functions/uuid_numeric.cpp | 4 ++--
be/src/vec/runtime/shared_hash_table_controller.h | 2 +-
8 files changed, 12 insertions(+), 8 deletions(-)
diff --git a/.github/workflows/be-ut-mac.yml b/.github/workflows/be-ut-mac.yml
index 19e99d27b48..0df342bc697 100644
--- a/.github/workflows/be-ut-mac.yml
+++ b/.github/workflows/be-ut-mac.yml
@@ -29,7 +29,7 @@ concurrency:
jobs:
run-ut:
name: BE UT (macOS)
- runs-on: macos-12
+ runs-on: macos-13
steps:
- name: Checkout
uses: actions/checkout@v3
@@ -81,7 +81,7 @@ jobs:
'node'
'llvm@16'
)
- brew install "${cellars[@]}"
+ brew install "${cellars[@]}" || true
pushd thirdparty
branch="${{ github.base_ref }}"
diff --git a/be/CMakeLists.txt b/be/CMakeLists.txt
index d78d7bbc97b..030118b3268 100644
--- a/be/CMakeLists.txt
+++ b/be/CMakeLists.txt
@@ -259,7 +259,8 @@ if (COMPILER_GCC)
add_compile_options(-fdiagnostics-color=always
-Wno-nonnull
-Wno-stringop-overread
- -Wno-stringop-overflow)
+ -Wno-stringop-overflow
+ -Wno-array-bounds)
endif ()
if (COMPILER_CLANG)
diff --git a/be/src/olap/memtable.cpp b/be/src/olap/memtable.cpp
index e88be6b3ed8..2457c026224 100644
--- a/be/src/olap/memtable.cpp
+++ b/be/src/olap/memtable.cpp
@@ -374,7 +374,7 @@ void MemTable::_aggregate() {
auto& block_data = in_block.get_columns_with_type_and_name();
std::vector<RowInBlock*> temp_row_in_blocks;
temp_row_in_blocks.reserve(_last_sorted_pos);
- RowInBlock* prev_row;
+ RowInBlock* prev_row = nullptr;
int row_pos = -1;
//only init agg if needed
for (int i = 0; i < _row_in_blocks.size(); i++) {
diff --git a/be/src/util/bitmap_value.h b/be/src/util/bitmap_value.h
index f91ac507707..0db3997ce17 100644
--- a/be/src/util/bitmap_value.h
+++ b/be/src/util/bitmap_value.h
@@ -1335,6 +1335,7 @@ public:
return is_v1 ? BitmapTypeCode::type::BITMAP64 : BitmapTypeCode::type::BITMAP64_V2;
}
}
+ __builtin_unreachable();
}
template <typename T>
diff --git a/be/src/util/perf_counters.cpp b/be/src/util/perf_counters.cpp
index e1b4bb7de9b..74cae81783b 100644
--- a/be/src/util/perf_counters.cpp
+++ b/be/src/util/perf_counters.cpp
@@ -262,7 +262,7 @@ bool PerfCounters::init_proc_self_io_counter(Counter counter) {
}
bool PerfCounters::init_proc_self_status_counter(Counter counter) {
- CounterData data;
+ CounterData data {};
data.counter = counter;
data.source = PerfCounters::PROC_SELF_STATUS;
data.type = TUnit::BYTES;
diff --git a/be/src/vec/aggregate_functions/aggregate_function_state_merge.h b/be/src/vec/aggregate_functions/aggregate_function_state_merge.h
index 052f4e6e851..136eca0837a 100644
--- a/be/src/vec/aggregate_functions/aggregate_function_state_merge.h
+++ b/be/src/vec/aggregate_functions/aggregate_function_state_merge.h
@@ -24,6 +24,8 @@ const static std::string AGG_MERGE_SUFFIX = "_merge";
class AggregateStateMerge : public AggregateStateUnion {
public:
+ using AggregateStateUnion::create;
+
AggregateStateMerge(AggregateFunctionPtr function, const DataTypes& argument_types,
const DataTypePtr& return_type)
: AggregateStateUnion(function, argument_types, return_type) {}
diff --git a/be/src/vec/functions/uuid_numeric.cpp b/be/src/vec/functions/uuid_numeric.cpp
index d41912b20e8..2e71eee9c02 100644
--- a/be/src/vec/functions/uuid_numeric.cpp
+++ b/be/src/vec/functions/uuid_numeric.cpp
@@ -137,8 +137,8 @@ private:
/// It is guaranteed (by PaddedPODArray) that we can overwrite up to 15 bytes after end.
}
- UInt64 randomSeed() {
- struct timespec times;
+ UInt64 randomSeed() const {
+ struct timespec times {};
/// Not cryptographically secure as time, pid and stack address can be predictable.
diff --git a/be/src/vec/runtime/shared_hash_table_controller.h b/be/src/vec/runtime/shared_hash_table_controller.h
index 0f2517c46ca..33521898543 100644
--- a/be/src/vec/runtime/shared_hash_table_controller.h
+++ b/be/src/vec/runtime/shared_hash_table_controller.h
@@ -26,6 +26,7 @@
#include <vector>
#include "common/status.h"
+#include "vec/core/block.h"
namespace doris {
@@ -38,7 +39,6 @@ class BitmapFilterFuncBase;
namespace vectorized {
class Arena;
-class Block;
struct SharedRuntimeFilterContext {
std::shared_ptr<MinMaxFuncBase> minmax_func;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 11/15: [fix](planner)should call SlotRef'smaterializeSrcExpr() method if the slotRef is materialized (#25467)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit f2ba29652225a59f7e19043ac7daeccdba22c470
Author: starocean999 <40...@users.noreply.github.com>
AuthorDate: Mon Oct 16 19:42:12 2023 +0800
[fix](planner)should call SlotRef'smaterializeSrcExpr() method if the slotRef is materialized (#25467)
---
fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java | 1 +
1 file changed, 1 insertion(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java b/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java
index a460ea0f752..fe035d31050 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/planner/JoinNodeBase.java
@@ -585,6 +585,7 @@ public abstract class JoinNodeBase extends PlanNode {
rhsExpr = rhsExpr.substitute(tmpSmap);
vSrcToOutputSMap.getLhs().add(rhsExpr);
SlotRef slotRef = new SlotRef(slotDesc);
+ slotRef.materializeSrcExpr();
vSrcToOutputSMap.getRhs().add(slotRef);
newRhs.add(slotRef);
bSmapChanged = true;
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 06/15: [opt] use correct column label when execute query in FE (#25372)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 91fe09e3a28010a4de84bdb241b03cc063e04594
Author: morrySnow <10...@users.noreply.github.com>
AuthorDate: Mon Oct 16 15:03:33 2023 +0800
[opt] use correct column label when execute query in FE (#25372)
SET @a = '4';
SELECT @a;
previous:
+-----+
| '4' |
+-----+
| 4 |
+-----+
current:
+----+
| @a |
+----+
| 4 |
+----+
---
.../main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java | 2 +-
fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java | 3 ++-
.../doris/nereids/trees/expressions/functions/scalar/CurrentUser.java | 4 ++--
3 files changed, 5 insertions(+), 4 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java
index fb2231644ac..71647def182 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinScalarFunctions.java
@@ -461,7 +461,7 @@ public class BuiltinScalarFunctions implements FunctionHelper {
scalar(CurrentDate.class, "curdate", "current_date"),
scalar(CurrentTime.class, "curtime", "current_time"),
scalar(CurrentUser.class, "current_user"),
- scalar(Database.class, "database"),
+ scalar(Database.class, "database", "schema"),
scalar(Date.class, "date"),
scalar(DateDiff.class, "datediff"),
scalar(DateFormat.class, "date_format"),
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
index 99b5ee2dee1..7aba5058680 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/NereidsPlanner.java
@@ -468,10 +468,11 @@ public class NereidsPlanner extends Planner {
List<String> data = Lists.newArrayList();
for (int i = 0; i < physicalOneRowRelation.getProjects().size(); i++) {
NamedExpression item = physicalOneRowRelation.getProjects().get(i);
+ NamedExpression output = physicalPlan.getOutput().get(i);
Expression expr = item.child(0);
if (expr instanceof Literal) {
LiteralExpr legacyExpr = ((Literal) expr).toLegacyLiteral();
- columns.add(new Column(item.getName(), item.getDataType().toCatalogDataType()));
+ columns.add(new Column(output.getName(), output.getDataType().toCatalogDataType()));
super.handleLiteralInFe(legacyExpr, data);
} else {
return Optional.empty();
diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CurrentUser.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CurrentUser.java
index ec784689783..f36e4548ebc 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CurrentUser.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/CurrentUser.java
@@ -22,7 +22,7 @@ import org.apache.doris.nereids.trees.expressions.functions.AlwaysNotNullable;
import org.apache.doris.nereids.trees.expressions.functions.ExplicitlyCastableSignature;
import org.apache.doris.nereids.trees.expressions.shape.LeafExpression;
import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
-import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.StringType;
import com.google.common.collect.ImmutableList;
@@ -35,7 +35,7 @@ public class CurrentUser extends ScalarFunction
implements LeafExpression, ExplicitlyCastableSignature, AlwaysNotNullable {
public static final List<FunctionSignature> SIGNATURES = ImmutableList.of(
- FunctionSignature.ret(BigIntType.INSTANCE).args()
+ FunctionSignature.ret(StringType.INSTANCE).args()
);
public CurrentUser() {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 05/15: insert default when json array parse failed (#25447)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit feb1ebbe42b5e477dd9d0766b1c0e6d4674f90e3
Author: Pxl <px...@qq.com>
AuthorDate: Mon Oct 16 14:51:26 2023 +0800
insert default when json array parse failed (#25447)
insert default when json array parse failed
---
be/src/vec/exprs/table_function/vexplode_json_array.cpp | 2 +-
.../query_p0/sql_functions/table_function/explode_json_array.out | 4 ++++
.../query_p0/sql_functions/table_function/explode_json_array.groovy | 5 +++++
3 files changed, 10 insertions(+), 1 deletion(-)
diff --git a/be/src/vec/exprs/table_function/vexplode_json_array.cpp b/be/src/vec/exprs/table_function/vexplode_json_array.cpp
index 464cdecca4a..811d34ccdec 100644
--- a/be/src/vec/exprs/table_function/vexplode_json_array.cpp
+++ b/be/src/vec/exprs/table_function/vexplode_json_array.cpp
@@ -215,7 +215,7 @@ Status VExplodeJsonArrayTableFunction::process_close() {
}
void VExplodeJsonArrayTableFunction::get_value(MutableColumnPtr& column) {
- if (current_empty()) {
+ if (current_empty() || _parsed_data.get_value(_type, _cur_offset, true) == nullptr) {
column->insert_default();
} else {
column->insert_data((char*)_parsed_data.get_value(_type, _cur_offset, true),
diff --git a/regression-test/data/query_p0/sql_functions/table_function/explode_json_array.out b/regression-test/data/query_p0/sql_functions/table_function/explode_json_array.out
index d66887925ab..2300079d0a8 100644
--- a/regression-test/data/query_p0/sql_functions/table_function/explode_json_array.out
+++ b/regression-test/data/query_p0/sql_functions/table_function/explode_json_array.out
@@ -21,6 +21,10 @@
30 8
60 8
+-- !explode_json_array_8_invalid --
+0 8
+60 8
+
-- !explode_json_array9 --
-- !explode_json_array10 --
diff --git a/regression-test/suites/query_p0/sql_functions/table_function/explode_json_array.groovy b/regression-test/suites/query_p0/sql_functions/table_function/explode_json_array.groovy
index 7bc3dac1d62..cd4e018243e 100644
--- a/regression-test/suites/query_p0/sql_functions/table_function/explode_json_array.groovy
+++ b/regression-test/suites/query_p0/sql_functions/table_function/explode_json_array.groovy
@@ -45,6 +45,11 @@ suite("explode_json_array") {
LATERAL VIEW EXPLODE_JSON_ARRAY_INT('[40, 80]') t2 as d_age
GROUP BY c_age ORDER BY c_age """
+ qt_explode_json_array_8_invalid """ SELECT c_age, COUNT(1) FROM ${tableName}
+ LATERAL VIEW EXPLODE_JSON_ARRAY_INT('["1", 60]') t1 as c_age
+ LATERAL VIEW EXPLODE_JSON_ARRAY_INT('["b", "c"]') t2 as d_age
+ GROUP BY c_age ORDER BY c_age """
+
qt_explode_json_array9 """ SELECT * FROM ${tableName}
LATERAL VIEW EXPLODE_JSON_ARRAY_INT('[]') t1 AS c_age
ORDER BY id, c_age """
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[doris] 01/15: [Bug](delete) Use date as common type for date comparison (#25262)
Posted by kx...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
kxiao pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit d3f0a1856ece891e3ec0dfe8ee9e64f9692eb2cc
Author: Gabriel <ga...@gmail.com>
AuthorDate: Wed Oct 11 11:51:43 2023 +0800
[Bug](delete) Use date as common type for date comparison (#25262)
---
fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java | 2 ++
1 file changed, 2 insertions(+)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java b/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java
index f892b850bb5..bce4807357b 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/analysis/BinaryPredicate.java
@@ -403,6 +403,8 @@ public class BinaryPredicate extends Predicate implements Writable {
} else if (getChild(1).getType().isDate()
&& (getChild(0).getType().isStringType() && getChild(0) instanceof StringLiteral)) {
return ((StringLiteral) getChild(0)).canConvertToDateType(Type.DATE) ? Type.DATE : Type.DATETIME;
+ } else if (getChild(1).getType().isDate() && getChild(0).getType().isDate()) {
+ return Type.DATE;
} else {
return Type.DATETIME;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org