You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/03/07 16:57:50 UTC
[GitHub] [incubator-doris] WingsGo opened a new issue #3053: [Stream
Load]Invalid rollback for stream load
WingsGo opened a new issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053
**Describe the bug**
For #1998 solve the PUBLISH_TIMEOUT will cause BE rollback txn incorrect, but i found there is still some bug. As shown in following log, the txn committed succefully, but FE receive a rollback request
```
2020-03-07 20:57:31,450 INFO 7812 [GlobalTransactionMgr.commitTransaction():429] transaction:[TransactionState. transaction id: 48471692, label: xxx db id: 10005, coordinator: BE: xxx, transaction status: COMMITTED, error replicas num: 2, replica ids: 64461890,66187307, prepare time: 1583585844933, commit time: 1583585851450, finish time: -1, reason: ] successfully committed
2020-03-07 20:57:36,451 INFO 10956 [FrontendServiceImpl.loadTxnRollback():723] receive txn rollback request. db: xxx, tbl: xxx, txn id: 48471692, reason: transaction commit successfully, BUT data will be visible later, backend: xxx
```
BE's log:
```
W0307 20:55:19.606360 157168 stream_load_executor.cpp:192] commit transaction failed, errmsg=transaction commit successfully, BUT data will be visible laterid=a0481d6290973230-1fe841af6df5088b, job id=-1, txn id=48470900, label=xxx
I0307 21:25:42.506554 156973 txn_manager.cpp:496] find expire pending data. tablet_id=64461864 schema_hash=1937251931 tablet_uid=0841acc74eba0dee-576e37036f499486 transaction_id=48470900 exist_sec=1829
```
I found the reason is that the code in following should also add judge if status is PUBLISH_TIMEOUT, if it is, we shouldn't rollback the txn
```
Status status(result.status);
if (!status.ok()) {
LOG(WARNING) << "commit transaction failed, errmsg=" << status.get_error_msg()
<< ctx->brief();
return status;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo removed a comment on issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo removed a comment on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596169339
https://github.com/apache/incubator-doris/blob/c83729435f45aa6d2d93c644ba80aa10189dd962/be/src/runtime/stream_load/stream_load_executor.cpp#L177-L198
As in code, the warning log is `W0307 20:55:19.606360 157168 stream_load_executor.cpp:192] commit transaction failed, errmsg=transaction commit successfully, BUT data will be visible laterid=a0481d6290973230-1fe841af6df5088b, job id=-1, txn id=48470900, label=xxx
` I know that _handle() will check the staus' code, but i don't konw where to call the method `handle()` I found that the warning log means that the txn is committed successfully, but there is not a check if the txn is PUBLISH_TIMEOUT， so the warning log is printed and BE try to rollback txn. Maybe you can point that where the method `handle()` called to help me find out the true reason?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo commented on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596169113
https://github.com/apache/incubator-doris/blob/c83729435f45aa6d2d93c644ba80aa10189dd962/be/src/runtime/stream_load/stream_load_executor.cpp#L177-L198
As in code, the warning log is `W0307 20:55:19.606360 157168 stream_load_executor.cpp:192] commit transaction failed, errmsg=transaction commit successfully, BUT data will be visible laterid=a0481d6290973230-1fe841af6df5088b, job id=-1, txn id=48470900, label=xxx
` I know that _handle() will check the staus' code, but i don't konw where to call the method `handle()` I found that the warning log means that the txn is committed successfully, but there is not a check if the txn is PUBLISH_TIMEOUT， so the warning log is printed and BE try to rollback txn. I know that `handle()` method is called by http method, maybe you can point that how `commit_txn` method call the http api to help me find out the true reason?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo closed issue #3053: [Stream Load]Invalid
rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo closed issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo commented on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596169339
https://github.com/apache/incubator-doris/blob/c83729435f45aa6d2d93c644ba80aa10189dd962/be/src/runtime/stream_load/stream_load_executor.cpp#L177-L198
As in code, the warning log is `W0307 20:55:19.606360 157168 stream_load_executor.cpp:192] commit transaction failed, errmsg=transaction commit successfully, BUT data will be visible laterid=a0481d6290973230-1fe841af6df5088b, job id=-1, txn id=48470900, label=xxx
` I know that _handle() will check the staus' code, but i don't konw where to call the method `handle()` I found that the warning log means that the txn is committed successfully, but there is not a check if the txn is PUBLISH_TIMEOUT， so the warning log is printed and BE try to rollback txn. Maybe you can point that where the method `handle()` called to help me find out the true reason?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo edited a comment on issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo edited a comment on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596173655
I also have another question, why ` StreamLoadExecutor::execute_plan_fragment` is called in two method `_handle` and `_process_put` ? I know `_process_put` will execute the fragment plan by FE, but how `_handle` work?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo edited a comment on issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo edited a comment on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596173655
I also have another question, what's the meaning of `if the FE return a PUBLISH_TIMEOUT status, it will return this status directly to the caller(_handle() in stream_load.cpp).`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
morningman commented on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596161810
Are you sure the problem is here?
in `stream_load_executor->commit_txn()`, if the FE return a PUBLISH_TIMEOUT status, it return will this status directly to the caller(`_handle()` in `stream_load.cpp`).
And in `_handle()`, it will check the status' code, and only if it is NOT the PUBLISH_TIMEOUT, it will send a roll back request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo commented on issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo commented on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596173655
I also have another question, why ` StreamLoadExecutor::execute_plan_fragment` is called in two method `_handle` and `_process_put` ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman edited a comment on issue #3053:
[Stream Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
morningman edited a comment on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596161810
Are you sure the problem is here?
in `stream_load_executor->commit_txn()`, if the FE return a PUBLISH_TIMEOUT status, it will return this status directly to the caller(`_handle()` in `stream_load.cpp`).
And in `_handle()`, it will check the status' code, and only if it is NOT the PUBLISH_TIMEOUT, it will send a roll back request.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo edited a comment on issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo edited a comment on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596169113
https://github.com/apache/incubator-doris/blob/c83729435f45aa6d2d93c644ba80aa10189dd962/be/src/runtime/stream_load/stream_load_executor.cpp#L177-L198
@morningman As in code, the warning log is `W0307 20:55:19.606360 157168 stream_load_executor.cpp:192] commit transaction failed, errmsg=transaction commit successfully, BUT data will be visible laterid=a0481d6290973230-1fe841af6df5088b, job id=-1, txn id=48470900, label=xxx
` So, the return status should be PUBLISH_TIMEOUT. I know that _handle() will check the staus' code, but I think `_handle` will not called in method `stream_load_executor->commit_txn`, so it cannot caused by `_handle()` method
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo edited a comment on issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo edited a comment on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596169113
https://github.com/apache/incubator-doris/blob/c83729435f45aa6d2d93c644ba80aa10189dd962/be/src/runtime/stream_load/stream_load_executor.cpp#L177-L198
As in code, the warning log is `W0307 20:55:19.606360 157168 stream_load_executor.cpp:192] commit transaction failed, errmsg=transaction commit successfully, BUT data will be visible laterid=a0481d6290973230-1fe841af6df5088b, job id=-1, txn id=48470900, label=xxx
` So, the status should be PUBLISH_TIMEOUT. I know that _handle() will check the staus' code, but i don't konw where to call the method `handle()` I found that the warning log means that the txn is committed successfully, but there is not a check if the txn is PUBLISH_TIMEOUT， so the warning log is printed and BE try to rollback txn. I know that `handle()` method is called by http method, maybe you can point that how `commit_txn` method call the http api to help me find out the true reason?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo opened a new issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo opened a new issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053
**Describe the bug**
For #1998 solve the PUBLISH_TIMEOUT will cause BE rollback txn incorrect, but i found there is still some bug. As shown in following log, the txn committed succefully, but FE receive a rollback request
```
2020-03-07 20:57:31,450 INFO 7812 [GlobalTransactionMgr.commitTransaction():429] transaction:[TransactionState. transaction id: 48471692, label: xxx db id: 10005, coordinator: BE: xxx, transaction status: COMMITTED, error replicas num: 2, replica ids: 64461890,66187307, prepare time: 1583585844933, commit time: 1583585851450, finish time: -1, reason: ] successfully committed
2020-03-07 20:57:36,451 INFO 10956 [FrontendServiceImpl.loadTxnRollback():723] receive txn rollback request. db: xxx, tbl: xxx, txn id: 48471692, reason: transaction commit successfully, BUT data will be visible later, backend: xxx
```
BE's log:
```
W0307 20:55:19.606360 157168 stream_load_executor.cpp:192] commit transaction failed, errmsg=transaction commit successfully, BUT data will be visible laterid=a0481d6290973230-1fe841af6df5088b, job id=-1, txn id=48470900, label=xxx
I0307 21:25:42.506554 156973 txn_manager.cpp:496] find expire pending data. tablet_id=64461864 schema_hash=1937251931 tablet_uid=0841acc74eba0dee-576e37036f499486 transaction_id=48470900 exist_sec=1829
```
I found the reason is that the code in following should also add judge if status is PUBLISH_TIMEOUT, if it is, we shouldn't rollback the txn
```
Status status(result.status);
if (!status.ok()) {
LOG(WARNING) << "commit transaction failed, errmsg=" << status.get_error_msg()
<< ctx->brief();
return status;
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] WingsGo edited a comment on issue #3053: [Stream
Load]Invalid rollback for stream load
Posted by GitBox <gi...@apache.org>.
WingsGo edited a comment on issue #3053: [Stream Load]Invalid rollback for stream load
URL: https://github.com/apache/incubator-doris/issues/3053#issuecomment-596173655
I also have another question, how the `on_request` method in ev_http_server.cpp be called?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org