You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by da...@apache.org on 2024/07/09 16:20:50 UTC
(doris) branch branch-3.0 updated (2c35e89f247 -> e80f9864054)
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a change to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
from 2c35e89f247 bump to 3.0.0-rc09
new 95ce5f1d0a8 [branch-2.1](memory) Disable Arrow Jemalloc (#37529)
new a4dceaefc4f [fix](dynamic partition) drop partition exclude history_partition_num (#37539)
new fdd71aed86a [bugfix](paimon)adding dependencies for `clang` (#37512)
new 1d56a791a2f [fix](auth)fix fe can not restart when replay create row policy log (#37342)
new 5f44a77f1a5 [pipeline](fix) Set upstream operators always runnable once source operator closed (#37297)
new b7888834143 [fix](memory) Disable Arrow Jemalloc step 2
new e80f9864054 bump to 3.0.0-rc10
The 7 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/cmake/thirdparty.cmake | 1 -
be/src/pipeline/exec/operator.cpp | 5 +++++
cloud/cmake/thirdparty.cmake | 7 +++++--
fe/be-java-extensions/preload-extensions/pom.xml | 5 +++++
.../doris/clone/DynamicPartitionScheduler.java | 23 ++++++++++++++--------
.../main/java/org/apache/doris/policy/Policy.java | 3 ++-
.../java/org/apache/doris/policy/RowPolicy.java | 20 +++++++++++++++----
.../java/org/apache/doris/qe/ConnectProcessor.java | 2 +-
.../doris/catalog/DynamicPartitionTableTest.java | 6 +++++-
.../java/org/apache/doris/policy/PolicyTest.java | 21 +++++++++++++++++++-
gensrc/script/gen_build_version.sh | 2 +-
.../test_dynamic_partition_failed.groovy | 13 ++++++++----
thirdparty/build-thirdparty.sh | 6 ++----
13 files changed, 86 insertions(+), 28 deletions(-)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 02/07: [fix](dynamic partition) drop partition exclude history_partition_num (#37539)
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit a4dceaefc4fb3968cd571a3098823972d1527697
Author: yujun <yu...@gmail.com>
AuthorDate: Tue Jul 9 23:52:53 2024 +0800
[fix](dynamic partition) drop partition exclude history_partition_num (#37539)
FIX:
When dropping dynamic partition, PR #35778 will use math.max(start,
-history_partition_num) as the first partition, but it may delete users'
partitions if they specify both start and history_partition_num
inappropriately. For safety reason, revert this behavious changed, only
use start as the first partition when dropping partitions.
For those who had specified a very small start value, drop partitions
will catch an exception , and stop dropping this table's partition and
then record this error in dynamic info. Users can use command `SHOW
DYNAMIC PARTITION TABLES FROM DBXXX` to know this error. From this
error, it will give user hint to modify start if they really specify a
error start.
---------
Co-authored-by: Yongqiang YANG <98...@users.noreply.github.com>
---
.../doris/clone/DynamicPartitionScheduler.java | 23 ++++++++++++++--------
.../doris/catalog/DynamicPartitionTableTest.java | 6 +++++-
.../test_dynamic_partition_failed.groovy | 13 ++++++++----
3 files changed, 29 insertions(+), 13 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
index d17af1836fe..17a4d34aa38 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/clone/DynamicPartitionScheduler.java
@@ -448,8 +448,10 @@ public class DynamicPartitionScheduler extends MasterDaemon {
return dropPartitionClauses;
}
- int realStart = DynamicPartitionUtil.getRealStart(dynamicPartitionProperty.getStart(),
- dynamicPartitionProperty.getHistoryPartitionNum());
+ // drop partition only considering start, not considering history_partition_num.
+ // int realStart = DynamicPartitionUtil.getRealStart(dynamicPartitionProperty.getStart(),
+ // dynamicPartitionProperty.getHistoryPartitionNum());
+ int realStart = dynamicPartitionProperty.getStart();
ZonedDateTime now = ZonedDateTime.now(dynamicPartitionProperty.getTimeZone().toZoneId());
String lowerBorder = DynamicPartitionUtil.getPartitionRangeString(dynamicPartitionProperty,
now, realStart, partitionFormat);
@@ -464,9 +466,12 @@ public class DynamicPartitionScheduler extends MasterDaemon {
} catch (Exception e) {
// AnalysisException: keys.size is always equal to column.size, cannot reach this exception
// IllegalArgumentException: lb is greater than ub
- LOG.warn("Error in gen reservePartitionKeyRange. db: {}, table: {}",
- db.getFullName(), olapTable.getName(), e);
- recordDropPartitionFailedMsg(db.getFullName(), olapTable.getName(), e.getMessage(), olapTable.getId());
+ String hint = "'dynamic_partition.start' = " + dynamicPartitionProperty.getStart()
+ + ", maybe it's too small, can use alter table sql to increase it. ";
+ LOG.warn("Error in gen reservePartitionKeyRange. db: {}, table: {}. {}",
+ db.getFullName(), olapTable.getName(), hint, e);
+ recordDropPartitionFailedMsg(db.getFullName(), olapTable.getName(), hint + e.getMessage(),
+ olapTable.getId());
return dropPartitionClauses;
}
@@ -585,10 +590,12 @@ public class DynamicPartitionScheduler extends MasterDaemon {
addPartitionClauses = getAddPartitionClause(db, olapTable, partitionColumn, partitionFormat,
executeFirstTime);
}
+ clearDropPartitionFailedMsg(olapTable.getId());
dropPartitionClauses = getDropPartitionClause(db, olapTable, partitionColumn, partitionFormat);
tableName = olapTable.getName();
} catch (Exception e) {
- LOG.warn("has error", e);
+ LOG.warn("db [{}-{}], table [{}-{}]'s dynamic partition has error",
+ db.getId(), db.getName(), olapTable.getId(), olapTable.getName(), e);
if (executeFirstTime) {
throw new DdlException(e.getMessage());
}
@@ -602,10 +609,10 @@ public class DynamicPartitionScheduler extends MasterDaemon {
}
try {
Env.getCurrentEnv().dropPartition(db, olapTable, dropPartitionClause);
- clearDropPartitionFailedMsg(olapTable.getId());
} catch (Exception e) {
recordDropPartitionFailedMsg(db.getFullName(), tableName, e.getMessage(), olapTable.getId());
- LOG.warn("has error", e);
+ LOG.warn("db [{}-{}], table [{}-{}]'s dynamic partition has error",
+ db.getId(), db.getName(), olapTable.getId(), olapTable.getName(), e);
if (executeFirstTime) {
throw new DdlException(e.getMessage());
}
diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java
index c2f13837329..419707f7cee 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/DynamicPartitionTableTest.java
@@ -744,7 +744,11 @@ public class DynamicPartitionTableTest {
String alter5 = "alter table test.dynamic_partition4 set ('dynamic_partition.history_partition_num' = '3')";
ExceptionChecker.expectThrowsNoException(() -> alterTable(alter5));
Env.getCurrentEnv().getDynamicPartitionScheduler().executeDynamicPartitionFirstTime(db.getId(), tbl4.getId());
- Assert.assertEquals(7, tbl4.getPartitionNames().size());
+ Assert.assertEquals(9, tbl4.getPartitionNames().size());
+ String dropPartitionErr = Env.getCurrentEnv().getDynamicPartitionScheduler()
+ .getRuntimeInfo(tbl4.getId(), DynamicPartitionScheduler.DROP_PARTITION_MSG);
+ Assert.assertTrue(dropPartitionErr.contains("'dynamic_partition.start' = -99999999, maybe it's too small, "
+ + "can use alter table sql to increase it."));
}
@Test
diff --git a/regression-test/suites/partition_p0/dynamic_partition/test_dynamic_partition_failed.groovy b/regression-test/suites/partition_p0/dynamic_partition/test_dynamic_partition_failed.groovy
index a19efadf740..0a23d422a3f 100644
--- a/regression-test/suites/partition_p0/dynamic_partition/test_dynamic_partition_failed.groovy
+++ b/regression-test/suites/partition_p0/dynamic_partition/test_dynamic_partition_failed.groovy
@@ -18,8 +18,8 @@
suite('test_dynamic_partition_failed', 'nonConcurrent') {
def old_max_dynamic_partition_num = getFeConfig('max_dynamic_partition_num')
try {
- sql 'DROP TABLE IF EXISTS test_dynamic_partition_failed_1'
- sql '''CREATE TABLE test_dynamic_partition_failed_1
+ sql 'DROP TABLE IF EXISTS test_dynamic_partition_failed_ok1 FORCE'
+ sql '''CREATE TABLE test_dynamic_partition_failed_ok1
( `k1` datetime NULL )
PARTITION BY RANGE (k1)()
DISTRIBUTED BY HASH(`k1`) BUCKETS 1
@@ -36,8 +36,13 @@ suite('test_dynamic_partition_failed', 'nonConcurrent') {
"dynamic_partition.create_history_partition" = "true"
)'''
- def partitions = sql_return_maparray "SHOW PARTITIONS FROM test_dynamic_partition_failed_1"
+ def partitions = sql_return_maparray "SHOW PARTITIONS FROM test_dynamic_partition_failed_ok1"
assertEquals(9, partitions.size());
+ def dynamicInfo = sql_return_maparray("SHOW DYNAMIC PARTITION TABLES").find { it.TableName == 'test_dynamic_partition_failed_ok1' }
+ logger.info("table dynamic info: " + dynamicInfo)
+ assertNotNull(dynamicInfo)
+ assertTrue(dynamicInfo.LastDropPartitionMsg.contains("'dynamic_partition.start' = -99999999, maybe it's too small, "
+ + "can use alter table sql to increase it."))
setFeConfig('max_dynamic_partition_num', Integer.MAX_VALUE)
@@ -96,7 +101,7 @@ suite('test_dynamic_partition_failed', 'nonConcurrent') {
}
} finally {
setFeConfig('max_dynamic_partition_num', old_max_dynamic_partition_num)
- sql 'DROP TABLE IF EXISTS test_dynamic_partition_failed_1'
+ sql 'DROP TABLE IF EXISTS test_dynamic_partition_failed_ok1 FORCE'
sql 'DROP TABLE IF EXISTS test_dynamic_partition_failed_2'
sql 'DROP TABLE IF EXISTS test_dynamic_partition_failed_3'
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 01/07: [branch-2.1](memory) Disable Arrow Jemalloc (#37529)
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 95ce5f1d0a82df578ecf0501bd2b6616a6e8b69d
Author: Xinyi Zou <zo...@gmail.com>
AuthorDate: Tue Jul 9 23:48:27 2024 +0800
[branch-2.1](memory) Disable Arrow Jemalloc (#37529)
pick ##37528
---
thirdparty/build-thirdparty.sh | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index 4b9925c6e95..b90e4183dff 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -1006,7 +1006,6 @@ build_arrow() {
export ARROW_LZ4_URL="${TP_SOURCE_DIR}/${LZ4_NAME}"
export ARROW_FLATBUFFERS_URL="${TP_SOURCE_DIR}/${FLATBUFFERS_NAME}"
export ARROW_ZSTD_URL="${TP_SOURCE_DIR}/${ZSTD_NAME}"
- export ARROW_JEMALLOC_URL="${TP_SOURCE_DIR}/${JEMALLOC_ARROW_NAME}"
export ARROW_Thrift_URL="${TP_SOURCE_DIR}/${THRIFT_NAME}"
export ARROW_SNAPPY_URL="${TP_SOURCE_DIR}/${SNAPPY_NAME}"
export ARROW_ZLIB_URL="${TP_SOURCE_DIR}/${ZLIB_NAME}"
@@ -1058,7 +1057,8 @@ build_arrow() {
-DSnappy_LIB="${TP_INSTALL_DIR}/lib/libsnappy.a" -DSnappy_INCLUDE_DIR="${TP_INSTALL_DIR}/include" \
-DSnappy_SOURCE=SYSTEM \
-DBOOST_ROOT="${TP_INSTALL_DIR}" --no-warn-unused-cli \
- -Djemalloc_SOURCE=BUNDLED \
+ -DARROW_JEMALLOC=OFF -DARROW_MIMALLOC=OFF \
+ -DJEMALLOC_HOME="${TP_INSTALL_DIR}" \
-DARROW_THRIFT_USE_SHARED=OFF \
-DThrift_SOURCE=SYSTEM \
-DThrift_ROOT="${TP_INSTALL_DIR}" ..
@@ -1067,12 +1067,10 @@ build_arrow() {
"${BUILD_SYSTEM}" install
#copy dep libs
- cp -rf ./jemalloc_ep-prefix/src/jemalloc_ep/dist/lib/libjemalloc_pic.a "${TP_INSTALL_DIR}/lib64/libjemalloc_arrow.a"
cp -rf ./brotli_ep/src/brotli_ep-install/lib/libbrotlienc-static.a "${TP_INSTALL_DIR}/lib64/libbrotlienc.a"
cp -rf ./brotli_ep/src/brotli_ep-install/lib/libbrotlidec-static.a "${TP_INSTALL_DIR}/lib64/libbrotlidec.a"
cp -rf ./brotli_ep/src/brotli_ep-install/lib/libbrotlicommon-static.a "${TP_INSTALL_DIR}/lib64/libbrotlicommon.a"
strip_lib libarrow.a
- strip_lib libjemalloc_arrow.a
strip_lib libparquet.a
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 07/07: bump to 3.0.0-rc10
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit e80f986405460ec0d98f9a198eb3327640258537
Author: Yongqiang YANG <da...@gmail.com>
AuthorDate: Wed Jul 10 00:20:00 2024 +0800
bump to 3.0.0-rc10
---
gensrc/script/gen_build_version.sh | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/gensrc/script/gen_build_version.sh b/gensrc/script/gen_build_version.sh
index cb7277ecaa5..60df82e04fb 100755
--- a/gensrc/script/gen_build_version.sh
+++ b/gensrc/script/gen_build_version.sh
@@ -31,7 +31,7 @@ build_version_prefix="doris"
build_version_major=3
build_version_minor=0
build_version_patch=0
-build_version_rc_version="rc09"
+build_version_rc_version="rc10"
build_version="${build_version_prefix}-${build_version_major}.${build_version_minor}.${build_version_patch}-${build_version_rc_version}"
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 05/07: [pipeline](fix) Set upstream operators always runnable once source operator closed (#37297)
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 5f44a77f1a5e0a8d4b601bc79626006dfaa40f60
Author: Gabriel <ga...@gmail.com>
AuthorDate: Fri Jul 5 00:36:05 2024 +0800
[pipeline](fix) Set upstream operators always runnable once source operator closed (#37297)
Some kinds of source operators has a 1-1 relationship with a sink
operator (such as AnalyticOperator). We must ensure AnalyticSinkOperator
will not be blocked if AnalyticSourceOperator already closed.
---
be/src/pipeline/exec/operator.cpp | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/be/src/pipeline/exec/operator.cpp b/be/src/pipeline/exec/operator.cpp
index b944fcfe21a..141eb1eaf59 100644
--- a/be/src/pipeline/exec/operator.cpp
+++ b/be/src/pipeline/exec/operator.cpp
@@ -498,6 +498,11 @@ Status PipelineXLocalState<SharedStateArg>::close(RuntimeState* state) {
_peak_memory_usage_counter->set(_mem_tracker->peak_consumption());
}
_closed = true;
+ // Some kinds of source operators has a 1-1 relationship with a sink operator (such as AnalyticOperator).
+ // We must ensure AnalyticSinkOperator will not be blocked if AnalyticSourceOperator already closed.
+ if (_shared_state && _shared_state->sink_deps.size() == 1) {
+ _shared_state->sink_deps.front()->set_always_ready();
+ }
return Status::OK();
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 04/07: [fix](auth)fix fe can not restart when replay create row policy log (#37342)
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit 1d56a791a2fc5856da39f34beb9af3280f8cad16
Author: zhangdong <49...@qq.com>
AuthorDate: Tue Jul 9 12:20:14 2024 +0800
[fix](auth)fix fe can not restart when replay create row policy log (#37342)
When executing two SQL statements simultaneously,such as
```
DROP ROW POLICY IF EXISTS test_row_policy_3 on zd.user1;CREATE ROW POLICY test_row_policy_3 ON zd.user1 AS RESTRICTIVE TO role role1 USING (k1 in (1) or k2 in (2));
```
fe will can not restart
```
Caused by: java.lang.ClassCastException: org.apache.doris.analysis.DropPolicyStmt cannot be cast to org.apache.doris.analysis.CreatePolicyStmt
at org.apache.doris.policy.RowPolicy.gsonPostProcess(RowPolicy.java:169) ~[doris-fe.jar:1.2-SNAPSHOT]
at org.apache.doris.persist.gson.GsonUtils$PostProcessTypeAdapterFactory$1.read(GsonUtils.java:640) ~[doris-fe.jar:1.2-SNAPSHOT]
at com.google.gson.TypeAdapter.fromJsonTree(TypeAdapter.java:299) ~[gson-2.10.1.jar:?]
... 14 more
```
fix:
- setOrigStmt use split sql instead of full sql
- store index to cope with segmentation failure
---
.../main/java/org/apache/doris/policy/Policy.java | 3 ++-
.../java/org/apache/doris/policy/RowPolicy.java | 20 ++++++++++++++++----
.../java/org/apache/doris/qe/ConnectProcessor.java | 2 +-
.../java/org/apache/doris/policy/PolicyTest.java | 21 ++++++++++++++++++++-
4 files changed, 39 insertions(+), 7 deletions(-)
diff --git a/fe/fe-core/src/main/java/org/apache/doris/policy/Policy.java b/fe/fe-core/src/main/java/org/apache/doris/policy/Policy.java
index 81016e94826..01c5399d4ab 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/policy/Policy.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/policy/Policy.java
@@ -117,7 +117,8 @@ public abstract class Policy implements Writable, GsonPostProcessable {
}
return new RowPolicy(policyId, stmt.getPolicyName(), stmt.getTableName().getCtl(),
stmt.getTableName().getDb(), stmt.getTableName().getTbl(), userIdent, stmt.getRoleName(),
- stmt.getOrigStmt().originStmt, stmt.getFilterType(), stmt.getWherePredicate());
+ stmt.getOrigStmt().originStmt, stmt.getOrigStmt().idx, stmt.getFilterType(),
+ stmt.getWherePredicate());
default:
throw new AnalysisException("Unknown policy type: " + stmt.getType());
}
diff --git a/fe/fe-core/src/main/java/org/apache/doris/policy/RowPolicy.java b/fe/fe-core/src/main/java/org/apache/doris/policy/RowPolicy.java
index 2b52c03b7d5..2b8a492e82c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/policy/RowPolicy.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/policy/RowPolicy.java
@@ -102,6 +102,8 @@ public class RowPolicy extends Policy implements RowFilterPolicy {
**/
@SerializedName(value = "originStmt")
private String originStmt;
+ @SerializedName(value = "stmtIdx")
+ private int stmtIdx;
private Expr wherePredicate = null;
@@ -123,7 +125,7 @@ public class RowPolicy extends Policy implements RowFilterPolicy {
* @param wherePredicate where predicate
*/
public RowPolicy(long policyId, final String policyName, long dbId, UserIdentity user, String roleName,
- String originStmt,
+ String originStmt, int stmtIdx,
final long tableId, final FilterType filterType, final Expr wherePredicate) {
super(policyId, PolicyTypeEnum.ROW, policyName);
this.user = user;
@@ -132,12 +134,13 @@ public class RowPolicy extends Policy implements RowFilterPolicy {
this.tableId = tableId;
this.filterType = filterType;
this.originStmt = originStmt;
+ this.stmtIdx = stmtIdx;
this.wherePredicate = wherePredicate;
}
public RowPolicy(long policyId, final String policyName, String ctlName, String dbName, String tableName,
UserIdentity user, String roleName,
- String originStmt, final FilterType filterType, final Expr wherePredicate) {
+ String originStmt, int stmtIdx, final FilterType filterType, final Expr wherePredicate) {
super(policyId, PolicyTypeEnum.ROW, policyName);
this.user = user;
this.roleName = roleName;
@@ -146,6 +149,7 @@ public class RowPolicy extends Policy implements RowFilterPolicy {
this.tableName = tableName;
this.filterType = filterType;
this.originStmt = originStmt;
+ this.stmtIdx = stmtIdx;
this.wherePredicate = wherePredicate;
}
@@ -166,16 +170,20 @@ public class RowPolicy extends Policy implements RowFilterPolicy {
try {
SqlScanner input = new SqlScanner(new StringReader(originStmt), 0L);
SqlParser parser = new SqlParser(input);
- CreatePolicyStmt stmt = (CreatePolicyStmt) SqlParserUtils.getFirstStmt(parser);
+ CreatePolicyStmt stmt = (CreatePolicyStmt) SqlParserUtils.getStmt(parser, stmtIdx);
wherePredicate = stmt.getWherePredicate();
} catch (Exception e) {
- throw new IOException("table policy parse originStmt error", e);
+ String errorMsg = String.format("table policy parse originStmt error, originStmt: %s, stmtIdx: %s.",
+ originStmt, stmtIdx);
+ // Only print logs to avoid cluster failure to start
+ LOG.warn(errorMsg, e);
}
}
@Override
public RowPolicy clone() {
return new RowPolicy(this.id, this.policyName, this.dbId, this.user, this.roleName, this.originStmt,
+ this.stmtIdx,
this.tableId,
this.filterType, this.wherePredicate);
}
@@ -218,6 +226,10 @@ public class RowPolicy extends Policy implements RowFilterPolicy {
public Expression getFilterExpression() throws AnalysisException {
NereidsParser nereidsParser = new NereidsParser();
String sql = getOriginStmt();
+ if (getStmtIdx() != 0) {
+ // Under normal circumstances, the index will only be equal to 0
+ throw new AnalysisException("Invalid row policy [" + getPolicyIdent() + "], " + sql);
+ }
CreatePolicyCommand command = (CreatePolicyCommand) nereidsParser.parseSingle(sql);
Optional<Expression> wherePredicate = command.getWherePredicate();
if (!wherePredicate.isPresent()) {
diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
index d25f541a7af..164ed98a51c 100644
--- a/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
+++ b/fe/fe-core/src/main/java/org/apache/doris/qe/ConnectProcessor.java
@@ -374,7 +374,7 @@ public abstract class ConnectProcessor {
}
StatementBase parsedStmt = stmts.get(i);
- parsedStmt.setOrigStmt(new OriginStatement(convertedStmt, i));
+ parsedStmt.setOrigStmt(new OriginStatement(auditStmt, usingOrigSingleStmt ? 0 : i));
parsedStmt.setUserInfo(ctx.getCurrentUserIdentity());
executor = new StmtExecutor(ctx, parsedStmt);
executor.getProfile().getSummaryProfile().setParseSqlStartTime(parseSqlStartTime);
diff --git a/fe/fe-core/src/test/java/org/apache/doris/policy/PolicyTest.java b/fe/fe-core/src/test/java/org/apache/doris/policy/PolicyTest.java
index ddf0d68fd2b..8d6a2a48ae3 100644
--- a/fe/fe-core/src/test/java/org/apache/doris/policy/PolicyTest.java
+++ b/fe/fe-core/src/test/java/org/apache/doris/policy/PolicyTest.java
@@ -34,6 +34,7 @@ import org.apache.doris.common.Config;
import org.apache.doris.common.DdlException;
import org.apache.doris.common.ExceptionChecker;
import org.apache.doris.common.FeConstants;
+import org.apache.doris.persist.gson.GsonUtils;
import org.apache.doris.utframe.TestWithFeService;
import com.google.common.collect.Lists;
@@ -341,7 +342,7 @@ public class PolicyTest extends TestWithFeService {
FilterType filterType = FilterType.PERMISSIVE;
Expr wherePredicate = null;
- Policy rowPolicy = new RowPolicy(10000, policyName, dbId, user, null, originStmt, tableId, filterType,
+ Policy rowPolicy = new RowPolicy(10000, policyName, dbId, user, null, originStmt, 0, tableId, filterType,
wherePredicate);
ByteArrayOutputStream emptyOutputStream = new ByteArrayOutputStream();
@@ -364,4 +365,22 @@ public class PolicyTest extends TestWithFeService {
Assertions.assertEquals(tableId, newRowPolicy.getTableId());
Assertions.assertEquals(filterType, newRowPolicy.getFilterType());
}
+
+ @Test
+ public void testCompatibility() {
+ String s1 = "{\n"
+ + " \"clazz\": \"RowPolicy\",\n"
+ + " \"roleName\": \"role1\",\n"
+ + " \"dbId\": 2,\n"
+ + " \"tableId\": 2,\n"
+ + " \"filterType\": \"PERMISSIVE\",\n"
+ + " \"originStmt\": \"CREATE ROW POLICY test_row_policy ON test.table1 AS PERMISSIVE TO test_policy USING (k1 \\u003d 1)\",\n"
+ + " \"id\": 1,\n"
+ + " \"type\": \"ROW\",\n"
+ + " \"policyName\": \"cc\",\n"
+ + " \"version\": 0\n"
+ + "}";
+ RowPolicy rowPolicy = GsonUtils.GSON.fromJson(s1, RowPolicy.class);
+ Assertions.assertEquals(rowPolicy.getStmtIdx(), 0);
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 06/07: [fix](memory) Disable Arrow Jemalloc step 2
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit b7888834143cb3ae1a2483a4805d83501398d2d3
Author: Xinyi Zou <zo...@gmail.com>
AuthorDate: Tue Jul 9 15:44:49 2024 +0800
[fix](memory) Disable Arrow Jemalloc step 2
---
be/cmake/thirdparty.cmake | 1 -
cloud/cmake/thirdparty.cmake | 7 +++++--
2 files changed, 5 insertions(+), 3 deletions(-)
diff --git a/be/cmake/thirdparty.cmake b/be/cmake/thirdparty.cmake
index 24c0365da71..502a22bbdf4 100644
--- a/be/cmake/thirdparty.cmake
+++ b/be/cmake/thirdparty.cmake
@@ -81,7 +81,6 @@ if (USE_JEMALLOC)
else()
add_thirdparty(tcmalloc WHOLELIBPATH ${GPERFTOOLS_HOME}/lib/libtcmalloc.a NOTADD)
endif()
-add_thirdparty(jemalloc_arrow LIBNAME "lib/libjemalloc_arrow.a")
if (WITH_MYSQL)
add_thirdparty(mysql LIBNAME "lib/libmysqlclient.a")
diff --git a/cloud/cmake/thirdparty.cmake b/cloud/cmake/thirdparty.cmake
index bacd7d25b3d..2b35cba5363 100644
--- a/cloud/cmake/thirdparty.cmake
+++ b/cloud/cmake/thirdparty.cmake
@@ -55,13 +55,16 @@ add_thirdparty(glog)
add_thirdparty(gflags)
add_thirdparty(backtrace)
add_thirdparty(pprof WHOLELIBPATH ${GPERFTOOLS_HOME}/lib/libprofiler.a)
-add_thirdparty(tcmalloc WHOLELIBPATH ${GPERFTOOLS_HOME}/lib/libtcmalloc.a NOTADD)
add_thirdparty(protobuf)
add_thirdparty(thrift)
add_thirdparty(crypto)
add_thirdparty(openssl LIBNAME "lib/libssl.a")
add_thirdparty(jemalloc LIBNAME "lib/libjemalloc_doris.a")
-add_thirdparty(jemalloc_arrow LIBNAME "lib/libjemalloc_arrow.a")
+if (USE_JEMALLOC)
+ add_thirdparty(jemalloc LIBNAME "lib/libjemalloc_doris.a")
+else()
+ add_thirdparty(tcmalloc WHOLELIBPATH ${GPERFTOOLS_HOME}/lib/libtcmalloc.a NOTADD)
+endif()
add_thirdparty(leveldb) # Required by brpc
add_thirdparty(brpc LIB64)
add_thirdparty(rocksdb) # For local storage mocking
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
(doris) 03/07: [bugfix](paimon)adding dependencies for `clang` (#37512)
Posted by da...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/doris.git
commit fdd71aed86abd7ed3e80cb83dd9e94dbaaf87faa
Author: wuwenchi <wu...@hotmail.com>
AuthorDate: Tue Jul 9 22:27:35 2024 +0800
[bugfix](paimon)adding dependencies for `clang` (#37512)
## Proposed changes
When paimon uses the `hms` type catalog and reads data in jni mode, it
needs to use the `org.apache.commons.lang.StringUtils` class.
(This problem is not tested in the pipeline environment because the
pipeline environment automatically generates the
`java-udf-case-jar-with-dependencies.jar` for testing, which contains
the `lang` package.)
---
fe/be-java-extensions/preload-extensions/pom.xml | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/fe/be-java-extensions/preload-extensions/pom.xml b/fe/be-java-extensions/preload-extensions/pom.xml
index 235a3f270f9..7910cec0d58 100644
--- a/fe/be-java-extensions/preload-extensions/pom.xml
+++ b/fe/be-java-extensions/preload-extensions/pom.xml
@@ -207,6 +207,11 @@ under the License.
<artifactId>paimon-oss</artifactId>
<version>${paimon.version}</version>
</dependency>
+ <dependency>
+ <groupId>commons-lang</groupId>
+ <artifactId>commons-lang</artifactId>
+ <version>${commons-lang.version}</version>
+ </dependency>
</dependencies>
<build>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org