You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/03/06 02:14:53 UTC
[GitHub] [incubator-doris] chaoyli opened a new pull request #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 t…
chaoyli opened a new pull request #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 t…
URL: https://github.com/apache/incubator-doris/pull/3044
…o Doris-0.11
If delete predicate exists in meta in Doris-0.10, all of this predicates should
be remained. There is an confused place in Doris-0.10. The delete predicate
only exists in OLAPHeaderMessage and PPendingDelta, not in PDelta.
This trick results this bug.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3044:
Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to
Doris-0.11
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
URL: https://github.com/apache/incubator-doris/pull/3044#discussion_r388688841
##########
File path: be/src/olap/olap_snapshot_converter.h
##########
@@ -47,7 +47,7 @@ class OlapSnapshotConverter {
// convert olap header to tablet meta pb, convert delta to rowsetmetapb
// pending delta is not in tablet meta any more, so that convert pending delta to rowset and add it to pending rowsets
// as a return value
- OLAPStatus to_tablet_meta_pb(const OLAPHeaderMessage& olap_header, TabletMetaPB* tablet_meta_pb,
+ OLAPStatus to_tablet_meta_pb(OLAPHeaderMessage* olap_header, TabletMetaPB* tablet_meta_pb,
Review comment:
Originally, this function only converts the header to meta.
Can you implement another function to finish new added operation to make it less complicate?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3044: Write delete
predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
URL: https://github.com/apache/incubator-doris/pull/3044#issuecomment-595565586
#3045
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3044:
Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to
Doris-0.11
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
URL: https://github.com/apache/incubator-doris/pull/3044#discussion_r388731229
##########
File path: be/src/olap/olap_snapshot_converter.cpp
##########
@@ -150,7 +167,15 @@ OLAPStatus OlapSnapshotConverter::to_tablet_meta_pb(const OLAPHeaderMessage& ola
}
RowsetId next_id = StorageEngine::instance()->next_rowset_id();
RowsetMetaPB* rowset_meta = tablet_meta_pb->add_inc_rs_metas();
- convert_to_rowset_meta(inc_delta, next_id, olap_header.tablet_id(), olap_header.schema_hash(), rowset_meta);
+ PDelta temp_inc_delta = inc_delta;
+ for (auto& del_pred : delete_conditions) {
+ if (temp_inc_delta.start_version() == temp_inc_delta.end_version()
+ && temp_inc_delta.start_version() == del_pred.version()) {
+ DeletePredicatePB* delete_condition = temp_inc_delta.mutable_delete_condition();
+ *delete_condition = del_pred;
+ }
Review comment:
align
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request
#3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10
to Doris-0.11
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
URL: https://github.com/apache/incubator-doris/pull/3044#discussion_r388700694
##########
File path: be/src/olap/olap_snapshot_converter.cpp
##########
@@ -89,27 +93,27 @@ OLAPStatus OlapSnapshotConverter::to_olap_header(const TabletMetaPB& tablet_meta
return OLAP_SUCCESS;
}
-OLAPStatus OlapSnapshotConverter::to_tablet_meta_pb(const OLAPHeaderMessage& olap_header,
+OLAPStatus OlapSnapshotConverter::to_tablet_meta_pb(OLAPHeaderMessage* olap_header,
Review comment:
Because PDelta can not corresponds with RowsetMeta.
So before calling convert_to_rowset_meta, I add delete predicate to PDelta to store delete predicate.
OK, I use a PDelta temp variable to get rid of OLAPHeaderMessage.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on a change in pull request
#3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10
to Doris-0.11
Posted by GitBox <gi...@apache.org>.
chaoyli commented on a change in pull request #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
URL: https://github.com/apache/incubator-doris/pull/3044#discussion_r388700694
##########
File path: be/src/olap/olap_snapshot_converter.cpp
##########
@@ -89,27 +93,27 @@ OLAPStatus OlapSnapshotConverter::to_olap_header(const TabletMetaPB& tablet_meta
return OLAP_SUCCESS;
}
-OLAPStatus OlapSnapshotConverter::to_tablet_meta_pb(const OLAPHeaderMessage& olap_header,
+OLAPStatus OlapSnapshotConverter::to_tablet_meta_pb(OLAPHeaderMessage* olap_header,
Review comment:
Because PDelta can not corresponds with RowsetMeta.
So before calling convert_to_rowset_meta, I add delete predicate to PDelta to store delete predicate.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3044:
Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to
Doris-0.11
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
URL: https://github.com/apache/incubator-doris/pull/3044#discussion_r388730551
##########
File path: be/src/olap/olap_snapshot_converter.cpp
##########
@@ -131,11 +136,23 @@ OLAPStatus OlapSnapshotConverter::to_tablet_meta_pb(const OLAPHeaderMessage& ola
}
std::unordered_map<Version, RowsetMetaPB*, HashOfVersion> _rs_version_map;
+ const DelPredicateArray& delete_conditions = olap_header.delete_data_conditions();
for (auto& delta : olap_header.delta()) {
RowsetId next_id = StorageEngine::instance()->next_rowset_id();
RowsetMetaPB* rowset_meta = tablet_meta_pb->add_rs_metas();
- convert_to_rowset_meta(delta, next_id, olap_header.tablet_id(), olap_header.schema_hash(), rowset_meta);
- Version rowset_version = { delta.start_version(), delta.end_version() };
+ PDelta temp_delta = delta;
+ // PDelta is not corresponding with RowsetMeta in DeletePredicate
+ // Add delete predicate to PDelta from OLAPHeaderMessage.
+ // Only after this, convert from PDelta to RowsetMeta is valid.
+ for (auto& del_pred : delete_conditions) {
+ if (temp_delta.start_version() == temp_delta.end_version()
+ && temp_delta.start_version() == del_pred.version()) {
+ DeletePredicatePB* delete_condition = temp_delta.mutable_delete_condition();
+ *delete_condition = del_pred;
+ }
Review comment:
align
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3044:
Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to
Doris-0.11
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
URL: https://github.com/apache/incubator-doris/pull/3044#discussion_r388688340
##########
File path: be/src/olap/olap_snapshot_converter.cpp
##########
@@ -89,27 +93,27 @@ OLAPStatus OlapSnapshotConverter::to_olap_header(const TabletMetaPB& tablet_meta
return OLAP_SUCCESS;
}
-OLAPStatus OlapSnapshotConverter::to_tablet_meta_pb(const OLAPHeaderMessage& olap_header,
+OLAPStatus OlapSnapshotConverter::to_tablet_meta_pb(OLAPHeaderMessage* olap_header,
Review comment:
This function name indicates filling content from olap_header. It is not a good idea to change olap_header's content.
And comment about what will be done in this function is needed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli merged pull request #3044: Write delete
predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
Posted by GitBox <gi...@apache.org>.
chaoyli merged pull request #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
URL: https://github.com/apache/incubator-doris/pull/3044
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #3044:
Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to
Doris-0.11
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #3044: Write delete predicate into RowsetMeta upon upgrade from Doris-0.10 to Doris-0.11
URL: https://github.com/apache/incubator-doris/pull/3044#discussion_r388731769
##########
File path: be/src/olap/olap_snapshot_converter.cpp
##########
@@ -131,11 +136,23 @@ OLAPStatus OlapSnapshotConverter::to_tablet_meta_pb(const OLAPHeaderMessage& ola
}
std::unordered_map<Version, RowsetMetaPB*, HashOfVersion> _rs_version_map;
+ const DelPredicateArray& delete_conditions = olap_header.delete_data_conditions();
for (auto& delta : olap_header.delta()) {
RowsetId next_id = StorageEngine::instance()->next_rowset_id();
RowsetMetaPB* rowset_meta = tablet_meta_pb->add_rs_metas();
- convert_to_rowset_meta(delta, next_id, olap_header.tablet_id(), olap_header.schema_hash(), rowset_meta);
- Version rowset_version = { delta.start_version(), delta.end_version() };
+ PDelta temp_delta = delta;
+ // PDelta is not corresponding with RowsetMeta in DeletePredicate
+ // Add delete predicate to PDelta from OLAPHeaderMessage.
+ // Only after this, convert from PDelta to RowsetMeta is valid.
+ for (auto& del_pred : delete_conditions) {
+ if (temp_delta.start_version() == temp_delta.end_version()
+ && temp_delta.start_version() == del_pred.version()) {
+ DeletePredicatePB* delete_condition = temp_delta.mutable_delete_condition();
+ *delete_condition = del_pred;
+ }
Review comment:
```suggestion
if (temp_delta.start_version() == temp_delta.end_version()) {
for (auto& del_pred : delete_conditions) {
if (temp_delta.start_version() == del_pred.version()) {
DeletePredicatePB* delete_condition = temp_delta.mutable_delete_condition();
*delete_condition = del_pred;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org