You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "hanyisong (via GitHub)" <gi...@apache.org> on 2023/02/27 15:22:15 UTC
[GitHub] [doris] hanyisong opened a new pull request, #17207: [Feature] Lightweight schema change of modify column with varchar length
hanyisong opened a new pull request, #17207:
URL: https://github.com/apache/doris/pull/17207
# Proposed changes
Issue Number: close #xxx
## Problem summary
Waiting to add test samples and modify principles.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SWJTU-ZhangLei commented on a diff in pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #17207:
URL: https://github.com/apache/doris/pull/17207#discussion_r1123056479
##########
fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java:
##########
@@ -869,7 +869,7 @@ public static void loadJournal(Env env, JournalEntity journal) {
env.getCatalogMgr().replayRefreshCatalog(log);
break;
}
- case OperationType.OP_MODIFY_TABLE_ADD_OR_DROP_COLUMNS: {
+ case OperationType.OP_MODIFY_TABLE_ADD_OR_DROP_OR_MODIFY_COLUMNS: {
Review Comment:
we cannot change OperationType.OP_MODIFY_TABLE_ADD_OR_DROP_COLUMNS, because the op code may write disk in previous doris version, this may cause compatibility problem
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SWJTU-ZhangLei commented on a diff in pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #17207:
URL: https://github.com/apache/doris/pull/17207#discussion_r1123108226
##########
fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java:
##########
@@ -869,7 +869,7 @@ public static void loadJournal(Env env, JournalEntity journal) {
env.getCatalogMgr().replayRefreshCatalog(log);
break;
}
- case OperationType.OP_MODIFY_TABLE_ADD_OR_DROP_COLUMNS: {
+ case OperationType.OP_MODIFY_TABLE_ADD_OR_DROP_OR_MODIFY_COLUMNS: {
Review Comment:
> I just changed the name, what do you think about the name OP_MODIFY_TABLE_LIGHT_SCHEMA_CHANGE?
1、you are right! Just changed the name will not cause compatibility problem.
2、 OP_MODIFY_TABLE_LIGHT_SCHEMA_CHANGE is a more comfortable name, the function name
how about replace 'modifyTableAddOrDropOrModifyColumns()' with 'modifyTableLighSchemaChange()'?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1451786026
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1483827571
run feut
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "hello-stephen (via GitHub)" <gi...@apache.org>.
hello-stephen commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1451984756
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 34.41 seconds
stream load tsv: 477 seconds loaded 74807831229 Bytes, about 149 MB/s
stream load json: 38 seconds loaded 2358488459 Bytes, about 59 MB/s
stream load orc: 68 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 27 seconds loaded 861443392 Bytes, about 30 MB/s
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20230302144512_clickbench_pr_107453.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on a diff in pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on code in PR #17207:
URL: https://github.com/apache/doris/pull/17207#discussion_r1123083104
##########
fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java:
##########
@@ -869,7 +869,7 @@ public static void loadJournal(Env env, JournalEntity journal) {
env.getCatalogMgr().replayRefreshCatalog(log);
break;
}
- case OperationType.OP_MODIFY_TABLE_ADD_OR_DROP_COLUMNS: {
+ case OperationType.OP_MODIFY_TABLE_ADD_OR_DROP_OR_MODIFY_COLUMNS: {
Review Comment:
I just changed the name, what do you think about the name OP_MODIFY_TABLE_LIGHT_SCHEMA_CHANGE?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1451957759
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on a diff in pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on code in PR #17207:
URL: https://github.com/apache/doris/pull/17207#discussion_r1123125237
##########
fe/fe-core/src/main/java/org/apache/doris/persist/EditLog.java:
##########
@@ -869,7 +869,7 @@ public static void loadJournal(Env env, JournalEntity journal) {
env.getCatalogMgr().replayRefreshCatalog(log);
break;
}
- case OperationType.OP_MODIFY_TABLE_ADD_OR_DROP_COLUMNS: {
+ case OperationType.OP_MODIFY_TABLE_ADD_OR_DROP_OR_MODIFY_COLUMNS: {
Review Comment:
Great idea, I accidentally left out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1475222118
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1475222518
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1457537077
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1483827603
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1457574659
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1480631237
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1457464909
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] dataroaring merged pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
Posted by "dataroaring (via GitHub)" <gi...@apache.org>.
dataroaring merged PR #17207:
URL: https://github.com/apache/doris/pull/17207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1480827097
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1480903062
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] SWJTU-ZhangLei commented on a diff in pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "SWJTU-ZhangLei (via GitHub)" <gi...@apache.org>.
SWJTU-ZhangLei commented on code in PR #17207:
URL: https://github.com/apache/doris/pull/17207#discussion_r1121308732
##########
fe/fe-core/src/main/java/org/apache/doris/alter/SchemaChangeHandler.java:
##########
@@ -1758,22 +1769,27 @@ public int getAsInt() {
}
} else if (alterClause instanceof ModifyColumnClause) {
// modify column
- processModifyColumn((ModifyColumnClause) alterClause, olapTable, indexSchemaMap);
+ boolean clauseCanLigthSchemaChange = processModifyColumn((ModifyColumnClause) alterClause, olapTable, indexSchemaMap);
lightSchemaChange = false;
+ if (clauseCanLigthSchemaChange == false) {
+ lightSchemaChangeWithVarcharLength = false;
+ }
} else if (alterClause instanceof ReorderColumnsClause) {
// reorder column
processReorderColumn((ReorderColumnsClause) alterClause, olapTable, indexSchemaMap);
lightSchemaChange = false;
+ lightSchemaChangeWithVarcharLength = false;
Review Comment:
can we reuse lightSchemaChange instead of add a new flag lightSchemaChangeWithVarcharLength?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature] Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1451874093
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Lchangliang commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
Posted by "Lchangliang (via GitHub)" <gi...@apache.org>.
Lchangliang commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1477697190
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hanyisong commented on pull request #17207: [Feature](schema change) Lightweight schema change of modify column with varchar length
Posted by "hanyisong (via GitHub)" <gi...@apache.org>.
hanyisong commented on PR #17207:
URL: https://github.com/apache/doris/pull/17207#issuecomment-1477290970
run p0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org