You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "nanfeng1999 (via GitHub)" <gi...@apache.org> on 2023/04/26 04:21:54 UTC
[GitHub] [doris] nanfeng1999 opened a new pull request, #19090: [feature](analysis) insert overwrite table
nanfeng1999 opened a new pull request, #19090:
URL: https://github.com/apache/doris/pull/19090
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] nanfeng1999 closed pull request #19090: [feature](analysis) insert overwrite table
Posted by "nanfeng1999 (via GitHub)" <gi...@apache.org>.
nanfeng1999 closed pull request #19090: [feature](analysis) insert overwrite table
URL: https://github.com/apache/doris/pull/19090
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #19090: [feature](analysis) insert overwrite table
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #19090:
URL: https://github.com/apache/doris/pull/19090#issuecomment-1539994668
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on a diff in pull request #19090: [feature](analysis) insert overwrite table
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #19090:
URL: https://github.com/apache/doris/pull/19090#discussion_r1181634672
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##########
@@ -1341,6 +1341,50 @@ public void replayCreateTable(String dbName, Table table) throws MetaNotFoundExc
}
}
+ public void addPartitionLike(Database db, String tableName, AddPartitionLikeClause addPartitionLikeClause)
+ throws DdlException {
+ try {
+ Table table = db.getTableOrDdlException(tableName);
+
+ if (table.getType() == TableType.VIEW) {
Review Comment:
```suggestion
if (table.getType() != TableType.OLAP) {
```
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##########
@@ -1341,6 +1341,50 @@ public void replayCreateTable(String dbName, Table table) throws MetaNotFoundExc
}
}
+ public void addPartitionLike(Database db, String tableName, AddPartitionLikeClause addPartitionLikeClause)
+ throws DdlException {
+ try {
+ Table table = db.getTableOrDdlException(tableName);
+
+ if (table.getType() == TableType.VIEW) {
+ throw new DdlException("Not support create partition from a View");
Review Comment:
```suggestion
throw new DdlException("Only support create partition from a OLAP table");
```
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##########
@@ -1341,6 +1341,50 @@ public void replayCreateTable(String dbName, Table table) throws MetaNotFoundExc
}
}
+ public void addPartitionLike(Database db, String tableName, AddPartitionLikeClause addPartitionLikeClause)
+ throws DdlException {
+ try {
+ Table table = db.getTableOrDdlException(tableName);
+
+ if (table.getType() == TableType.VIEW) {
+ throw new DdlException("Not support create partition from a View");
+ }
+
+ table.readLock();
+ try {
+ if (table.getType().equals(TableType.OLAP)) {
+ String partitionName = addPartitionLikeClause.getPartitionName();
+ String existedName = addPartitionLikeClause.getExistedPartitionName();
+ OlapTable olapTable = (OlapTable) table;
+ Partition part = olapTable.getPartition(existedName);
+ PartitionInfo partitionInfo = olapTable.getPartitionInfo();
+ PartitionDesc partitionDesc = partitionInfo.toPartitionDesc((OlapTable) table);
+ SinglePartitionDesc oldPartitionDesc = partitionDesc.getSinglePartitionDescByName(existedName);
+ if (oldPartitionDesc == null) {
+ throw new DdlException("Failed to ADD PARTITION" + partitionName + " LIKE "
+ + existedName + ". Reason: " + "partition " + existedName + "not exist");
+ }
+ DistributionDesc distributionDesc = part.getDistributionInfo().toDistributionDesc();
+ SinglePartitionDesc newPartitionDesc = new SinglePartitionDesc(false, partitionName,
+ oldPartitionDesc.getPartitionKeyDesc(), oldPartitionDesc.getProperties());
+ Map<String, String> properties = newPartitionDesc.getProperties();
+ AddPartitionClause clause = new AddPartitionClause(newPartitionDesc, distributionDesc,
+ properties, addPartitionLikeClause.getIsTempPartition());
+ table.readUnlock();
Review Comment:
This `table.readUnlock();` should be in a `finally` block.
You split the whole logic into 2 parts:
One is for creating an `AddPartitionClause`, the other is for calling `addPartition()`.
And refactor the current `try..catch`
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/InternalCatalog.java:
##########
@@ -1341,6 +1341,50 @@ public void replayCreateTable(String dbName, Table table) throws MetaNotFoundExc
}
}
+ public void addPartitionLike(Database db, String tableName, AddPartitionLikeClause addPartitionLikeClause)
+ throws DdlException {
+ try {
+ Table table = db.getTableOrDdlException(tableName);
+
+ if (table.getType() == TableType.VIEW) {
+ throw new DdlException("Not support create partition from a View");
+ }
+
+ table.readLock();
+ try {
+ if (table.getType().equals(TableType.OLAP)) {
Review Comment:
Remove this `if`
##########
fe/fe-core/src/main/java/org/apache/doris/qe/StmtExecutor.java:
##########
@@ -2145,6 +2163,160 @@ private void handleCtasRollback(TableName table) {
}
}
+ private void handleIotStmt() {
+ InsertOverwriteTableStmt iotStmt = (InsertOverwriteTableStmt) this.parsedStmt;
+ if (iotStmt.getPartitionNames().size() == 0) {
+ // insert overwrite table
+ handleOverwriteTable(iotStmt);
+ } else {
+ // insert overwrite table with partition
+ handleOverwritePartition(iotStmt);
+ }
+ }
+
+ private void handleOverwriteTable(InsertOverwriteTableStmt iotStmt) {
+ UUID uuid = UUID.randomUUID();
+ // to comply with naming rules
+ TableName tmpTableName = new TableName(null, iotStmt.getDb(), "tmp_table_" + uuid.toString().replace('-', '_'));
+ TableName targetTableName = new TableName(null, iotStmt.getDb(), iotStmt.getTbl());
+ try {
+ // create a tmp table with uuid
+ parsedStmt = new CreateTableLikeStmt(false, tmpTableName, targetTableName, null, false);
+ parsedStmt.setUserInfo(context.getCurrentUserIdentity());
+ execute();
+ } catch (Exception e) {
+ // Maybe our bug
+ LOG.warn("IOT create a tmp table error, stmt={}", originStmt.originStmt, e);
+ context.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, "Unexpected exception: " + e.getMessage());
+ return;
+ }
+ // after success create table insert data
+ try {
+ parsedStmt = new InsertStmt(tmpTableName, iotStmt.getQueryStmt());
+ parsedStmt.setUserInfo(context.getCurrentUserIdentity());
+ execute();
+ if (MysqlStateType.ERR.equals(context.getState().getStateType())) {
+ LOG.warn("IOT insert data error, stmt={}", parsedStmt.toSql());
+ handleIotRollback(tmpTableName);
+ return;
+ }
+ } catch (Exception e) {
+ LOG.warn("IOT insert data error, stmt={}", parsedStmt.toSql(), e);
+ context.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, "Unexpected exception: " + e.getMessage());
+ handleIotRollback(tmpTableName);
+ return;
+ }
+
+ // overwrite old table with tmp table
+ try {
+ List<AlterClause> ops = new ArrayList<>();
+ Map<String, String> properties = new HashMap<>();
+ properties.put("swap", "false");
+ ops.add(new ReplaceTableClause(tmpTableName.getTbl(), properties));
+ parsedStmt = new AlterTableStmt(targetTableName, ops);
+ parsedStmt.setUserInfo(context.getCurrentUserIdentity());
+ execute();
+ context.getState().setOk();
+ } catch (Exception e) {
+ // Maybe our bug
+ LOG.warn("IOT overwrite table error, stmt={}", parsedStmt.toSql(), e);
+ context.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, "Unexpected exception: " + e.getMessage());
+ handleIotRollback(tmpTableName);
+ }
+
+ }
+
+ private void handleOverwritePartition(InsertOverwriteTableStmt iotStmt) {
+ TableName targetTableName = new TableName(null, iotStmt.getDb(), iotStmt.getTbl());
+ List<String> partitionNames = iotStmt.getPartitionNames();
+ List<String> tempPartitionName = new ArrayList<>();
+ try {
+ // create tmp partitions with uuid
+ for (String partitionName : partitionNames) {
+ UUID uuid = UUID.randomUUID();
+ // to comply with naming rules
+ String tempPartName = "tmp_partition_" + uuid.toString().replace('-', '_');
+ tempPartitionName.add(tempPartName);
+ List<AlterClause> ops = new ArrayList<>();
+ ops.add(new AddPartitionLikeClause(tempPartName, partitionName, true));
+ parsedStmt = new AlterTableStmt(targetTableName, ops);
+ parsedStmt.setUserInfo(context.getCurrentUserIdentity());
+ execute();
+ }
+ } catch (Exception e) {
+ // Maybe our bug
+ LOG.warn("IOT create tmp table partitions error, stmt={}", originStmt.originStmt, e);
+ context.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, "Unexpected exception: " + e.getMessage());
+ return;
+ }
+
+ // after success add tmp partitions
+ try {
+ parsedStmt = new InsertStmt(targetTableName, new PartitionNames(true, tempPartitionName),
+ iotStmt.getQueryStmt());
+ parsedStmt.setUserInfo(context.getCurrentUserIdentity());
+ execute();
+ if (MysqlStateType.ERR.equals(context.getState().getStateType())) {
+ LOG.warn("IOT insert data error, stmt={}", parsedStmt.toSql());
+ handleIotPartitionRollback(targetTableName, tempPartitionName);
+ return;
+ }
+ } catch (Exception e) {
+ LOG.warn("IOT insert data error, stmt={}", parsedStmt.toSql(), e);
+ context.getState().setError(ErrorCode.ERR_UNKNOWN_ERROR, "Unexpected exception: " + e.getMessage());
+ handleIotPartitionRollback(targetTableName, tempPartitionName);
+ return;
+ }
+
+ // overwrite old table with tmp table
Review Comment:
```suggestion
// overwrite old partition with tmp partition
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morningman commented on pull request #19090: [feature](analysis) insert overwrite table
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #19090:
URL: https://github.com/apache/doris/pull/19090#issuecomment-1529822079
Need to add more test cases for these feature:
1. insert overwrite empty table/partition
2. insert overwrite with wrong table/partition name
3. insert overwrite table/partition with wrong data(data not in partition's range, or data is unqualified).
4. insert overwrite different partition type(range partition & list partition)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org