You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/07/22 10:37:08 UTC
[GitHub] [incubator-doris] ZhangYu0123 opened a new pull request #4143: Fix alter schema add key column bug in agg model
ZhangYu0123 opened a new pull request #4143:
URL: https://github.com/apache/incubator-doris/pull/4143
## Proposed changes
Fix the bug "alter schema add key column bug in agg model when using LinkedSchemaChange policy", the detail description #4142.
The change is when add key column in agg model , we set sc_directly = true to use SchemaChangeDirectly policy.
## Types of changes
What types of changes does your code introduce to Doris?
_Put an `x` in the boxes that apply_
- [x] Bugfix (non-breaking change which fixes an issue)
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._
- [x] I have create an issue on [Doris's issues](https://github.com/apache/incubator-doris/issues), and have described the bug/feature there in detail
- [x] Commit messages in my PR start with the related issues ID, like "#4071 Add pull request template to doris project"
- [x] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature works
- [ ] If this change need a document change, I have updated the document
- [x] Any dependent changes have been merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #4143: Fix alter schema add key column bug in agg model
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #4143:
URL: https://github.com/apache/incubator-doris/pull/4143#discussion_r459831200
##########
File path: be/src/olap/rowset/segment_group.cpp
##########
@@ -268,11 +265,14 @@ OLAPStatus SegmentGroup::add_zone_maps_for_linked_schema_change(
WrapperField* second = WrapperField::create(column);
DCHECK(second != NULL) << "failed to allocate memory for field: " << i;
- if (schema_mapping[i].ref_column == -1) {
+ // add column also can add zone map index
+ if (schema_mapping[i].ref_column == -1 || i >= zone_map_fields.size()) {
// ref_column == -1 means this is a new column.
// for new column, use default value to fill into column_statistics
- first->copy(schema_mapping[i].default_value);
- second->copy(schema_mapping[i].default_value);
+ if (schema_mapping[i].default_value != nullptr) {
+ first->copy(schema_mapping[i].default_value);
Review comment:
if the zone map set to default value， it may be filter data incorrectly
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #4143: Fix alter schema add key column bug in agg model
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #4143:
URL: https://github.com/apache/incubator-doris/pull/4143#discussion_r459832428
##########
File path: be/src/olap/rowset/segment_group.cpp
##########
@@ -268,11 +265,14 @@ OLAPStatus SegmentGroup::add_zone_maps_for_linked_schema_change(
WrapperField* second = WrapperField::create(column);
DCHECK(second != NULL) << "failed to allocate memory for field: " << i;
- if (schema_mapping[i].ref_column == -1) {
+ // add column also can add zone map index
+ if (schema_mapping[i].ref_column == -1 || i >= zone_map_fields.size()) {
// ref_column == -1 means this is a new column.
// for new column, use default value to fill into column_statistics
- first->copy(schema_mapping[i].default_value);
- second->copy(schema_mapping[i].default_value);
+ if (schema_mapping[i].default_value != nullptr) {
Review comment:
if you set it to default value， in duplicated table update from 0.11 to 0.12 may cause value clumn filtered unexpected
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on pull request #4143: Fix alter schema add key column bug in agg model
Posted by GitBox <gi...@apache.org>.
chaoyli commented on pull request #4143:
URL: https://github.com/apache/incubator-doris/pull/4143#issuecomment-663314921
Now, Add key column in prefix index will do directly schema change.
```
if (base_tablet->num_short_key_columns() != new_tablet->num_short_key_columns()) {
// the number of short_keys changed, can't do linked schema change
*sc_directly = true;
return OLAP_SUCCESS;
}
```
Another add key column will not change the order of record, it will be ok to do linked schema change.
So you should check why linked schema change will failed on zone_map.
You can reference the following function which is used to handle the zone map upon linked schema change.
```
add_rowset_for_linked_schema_change
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman merged pull request #4143: Fix alter schema add key column bug in agg model
Posted by GitBox <gi...@apache.org>.
morningman merged pull request #4143:
URL: https://github.com/apache/incubator-doris/pull/4143
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #4143: Fix alter schema add key column bug in agg model
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #4143:
URL: https://github.com/apache/incubator-doris/pull/4143#discussion_r459178509
##########
File path: be/src/olap/schema_change.cpp
##########
@@ -2111,6 +2111,11 @@ OLAPStatus SchemaChangeHandler::_parse_request(TabletSharedPtr base_tablet,
for (size_t i = 0; i < new_tablet->num_columns(); ++i) {
ColumnMapping* column_mapping = rb_changer->get_mutable_column_mapping(i);
if (column_mapping->ref_column < 0) {
+ // check agg model, add key-column operation use sc_directly to build index
+ if (new_tablet_schema.keys_type() == AGG_KEYS && new_tablet_schema.column(i).is_key()) {
+ *sc_directly = true;
+ return OLAP_SUCCESS;
+ }
Review comment:
this will disable all linked schema in when add key column in agg table, try some better way
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yangzhg commented on a change in pull request #4143: Fix alter schema add key column bug in agg model
Posted by GitBox <gi...@apache.org>.
yangzhg commented on a change in pull request #4143:
URL: https://github.com/apache/incubator-doris/pull/4143#discussion_r459831364
##########
File path: be/src/olap/rowset/segment_group.cpp
##########
@@ -268,11 +265,14 @@ OLAPStatus SegmentGroup::add_zone_maps_for_linked_schema_change(
WrapperField* second = WrapperField::create(column);
DCHECK(second != NULL) << "failed to allocate memory for field: " << i;
- if (schema_mapping[i].ref_column == -1) {
+ // add column also can add zone map index
Review comment:
better to make this comment more readable
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org