You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/06/29 13:39:33 UTC
[GitHub] [incubator-doris] liutang123 opened a new issue #3976: Lost data when load data to partitions created by old version doris
liutang123 opened a new issue #3976:
URL: https://github.com/apache/incubator-doris/issues/3976
In #2358, we remove an important logic of `_check_added_rowset`:
```Version version = {rowset->start_version(), rowset->end_version()};
RowsetSharedPtr exist_rs = get_rowset_by_version(version);
// if there exist a rowset with version_hash == 0, should delete it
if (exist_rs != nullptr && exist_rs->version_hash() == 0) {
vector<RowsetSharedPtr> to_add;
vector<RowsetSharedPtr> to_delete;
to_delete.push_back(exist_rs);
RETURN_NOT_OK(modify_rowsets(to_add, to_delete));
}
```
This logic has a very important role to make BE compatible with 0.10.
In our cluster, there are many partitions created by 0.10, and the tablets have two initial RSs [(0-1), 0] and [(2-2), 0].
We updated the cluster to 0.11 and then to 0.12.
Now, we load data to these partitions. BE will create a new RS. When pubish version, FE will try to let BE to set this RS to [(2-2), XXX] and add it to tablet. But, tablet has a same version RS [(2-2), 0]. So, the new RS will not be added to the tablet and the data will **lost**.
Q1: Why we remove version_hash in 0.13?
Q2: How to make doris to be compatible with previous version in this case?
Q3: I use `/api/meta/header` to see these tablet and find that like follows:
```
{
"tablet_id": 218278919,
"schema_hash": 39606704,
"creation_time": 1576654229,
...
"rs_metas": [
{
"rowset_id": 0,
"tablet_id": 218278919,
"tablet_schema_hash": 39606704,
...
"empty": true,
"creation_time": 1576654229,
},
{
"rowset_id": 0,
"tablet_id": 218278919,
"tablet_schema_hash": 39606704,
...
"empty": true,
"creation_time": 1576654229,
}
}
```
Is it possible to use the `creation_time` to determine whether a RS is the initial RS?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3976: Lost data when load data to partitions created by old version doris
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3976:
URL: https://github.com/apache/incubator-doris/issues/3976#issuecomment-651476937
This code takes into effect in following Doris-0.13. And you upgrade to Doris-0.12, there is no problem.
I think you have already upgrade to master instead. In this situation, you can load one non-sense file to table to generate the [2-2, 0] version and then delete the data.
After this, the [3-3] version will be no problem.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on issue #3976: Lost data when load data to partitions created by old version doris
Posted by GitBox <gi...@apache.org>.
liutang123 commented on issue #3976:
URL: https://github.com/apache/incubator-doris/issues/3976#issuecomment-651129678
@chaoyli
cc @imay @gaodayue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 commented on issue #3976: Lost data when load data to partitions created by old version doris
Posted by GitBox <gi...@apache.org>.
liutang123 commented on issue #3976:
URL: https://github.com/apache/incubator-doris/issues/3976#issuecomment-651500229
> This code takes into effect in following Doris-0.13. And you upgrade to Doris-0.12, there is no problem.
> I think you have already upgrade to master instead. In this situation, you can load one non-sense file to table to generate the [2-2, 0] version and then delete the data.
> After this, the [3-3] version will be no problem.
Sure, we resolved this problem in our cluster by a very tricky method. But, I think this is a general problem and we should fix it before 0.13.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3976: Lost data when load data to partitions created by old version doris
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3976:
URL: https://github.com/apache/incubator-doris/issues/3976#issuecomment-651505901
Now Doris-0.13 is only exist in master and not be published, how to do you upgrade to it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] chaoyli commented on issue #3976: Lost data when load data to partitions created by old version doris
Posted by GitBox <gi...@apache.org>.
chaoyli commented on issue #3976:
URL: https://github.com/apache/incubator-doris/issues/3976#issuecomment-651476013
Previously, the version_hash is used to differentiate the same load with the same version.
In old days, FE will send version to BE before load finished. Assuming I have three BE(A, B, C) nodes.
If A succeed, but B, C failed. The version belongs to load will failed. So The version will be reused by another load.
Under this circumstance, version_hash is useful to differentiate the same version in Node A.
Now, The load is encapsulated as a transaction. It will use **two-phase commit** to complete the transaction.
The first phase is PREPARE, which only produce the data without version.
If all nodes are finished PREPARED, the FE will think the load is finished.
At this time, send a version will only once.
So after this refactor, the version_hash is surplus and to be removed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] liutang123 edited a comment on issue #3976: Lost data when load data to partitions created by old version doris
Posted by GitBox <gi...@apache.org>.
liutang123 edited a comment on issue #3976:
URL: https://github.com/apache/incubator-doris/issues/3976#issuecomment-651500229
> This code takes into effect in following Doris-0.13. And you upgrade to Doris-0.12, there is no problem.
> I think you have already upgrade to master instead. In this situation, you can load one non-sense file to table to generate the [2-2, 0] version and then delete the data.
> After this, the [3-3] version will be no problem.
Sure, we resolved this problem in our cluster by a very tricky method. But, I think this is a general problem and we should fix it before 0.13 by a general solution.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org