You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/04/07 07:56:47 UTC
[GitHub] [incubator-doris] gaodayue opened a new issue #3270: Query failed
by "failed to initialize storage reader, res=-230"
gaodayue opened a new issue #3270: Query failed by "failed to initialize storage reader, res=-230"
URL: https://github.com/apache/incubator-doris/issues/3270
When load frequency is high and query planning took several seconds, FE may query an already compacted tablet version which results in query failure.
The user facing error is like "failed to initialize storage reader. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, res=-230, backend=xx.xx.xx.xx".
We can replay related events from logs, listed below.
1) 11:51:37 BE updated the latest version of tablet 223602817 to 18647
```
I0325 11:51:37.129848 33983 engine_publish_version_task.cpp:116] publish version successfully on tablet. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, transaction_id=848571860, version=18647, res=0
```
2) 11:51:37 FE updated the read version for table's partition 223602812 to 18647
```
2020-03-25 11:51:37,176 INFO 33 [GlobalTransactionMgr.updateCatalogAfterVisible():1086] transaction state TransactionState. transaction id: 848571860, label: 0_table_name_1168497, db id: 126584253, coordinator: BE: xx.xx.xx.xx:56373, transaction status: VISIBLE, error replicas num: 0, replica ids: , prepare time: 1585108296144, commit time: 1585108297049, finish time: 1585108297175, reason: set partition 223602812's version to [18647] and version hash to [6755194206027783188]
```
3) 11:51:41 User executing a query which acquired version 18647 for tablet 223602817
4) 11:51:44 BE updated the latest version of tablet 223602817 to 18648
```
I0325 11:51:44.921618 33984 engine_publish_version_task.cpp:116] publish version successfully on tablet. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, transaction_id=848571890, version=18648, res=0
```
5) 11:51:45 the query in step 3 finished analyzing, begin to schedule
2020-03-25 11:51:45,428 INFO 943132 [QeProcessorImpl.registerQuery():68] register query id = f31d984bca9d40ba-9cf7e7544747b31a, job: -1
6) 11:51:52 BE compacted version 18546-18648
```
I0325 11:51:51.573143 33777 compaction.cpp:36] start cumulative compaction. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd
I0325 11:51:52.198297 33777 compaction.cpp:77] succeed to do cumulative compaction. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, output_version=18546-18648. elapsed time=6.25136s.
```
7) 11:51:52 BE received the fragment for query in step 3. The request failed because the queried version 18647 has been compacted
```
W0325 11:51:52.464841 46274 tablet.cpp:479] tablet:223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, version already has been merged. spec_version: 0-18647
W0325 11:51:52.479046 46274 olap_scanner.cpp:122] fail to init reader.res=-230
W0325 11:51:52.491998 46274 olap_scanner.cpp:63] OlapScanner preapre failed, status:failed to initialize storage reader. tablet=223602817.1265196163.de411743660db983-e1cfcb59e3e86abd, res=-230, backend=xx.xx.xx.xx
W0325 11:51:52.541859 46290 rowset_graph.cpp:194] fail to find path in version_graph. spec_version: 0-18647
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay closed issue #3270: Query failed by "failed
to initialize storage reader, res=-230"
Posted by GitBox <gi...@apache.org>.
imay closed issue #3270: Query failed by "failed to initialize storage reader, res=-230"
URL: https://github.com/apache/incubator-doris/issues/3270
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #3270: Query failed by
"failed to initialize storage reader, res=-230"
Posted by GitBox <gi...@apache.org>.
imay commented on issue #3270: Query failed by "failed to initialize storage reader, res=-230"
URL: https://github.com/apache/incubator-doris/issues/3270#issuecomment-610245714
Good analysis. Do you have a good idea to solve this problem?
By the way, it is too long to analysis a query in 4 seconds, do you know why?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on issue #3270: Query failed
by "failed to initialize storage reader, res=-230"
Posted by GitBox <gi...@apache.org>.
gaodayue commented on issue #3270: Query failed by "failed to initialize storage reader, res=-230"
URL: https://github.com/apache/incubator-doris/issues/3270#issuecomment-610250711
We use a straightforward approach to avoid compacting recently added version, which works well in practice.
> By the way, it is too long to analysis a query in 4 seconds, do you know why?
In our case, the query in trouble is a fairly complicated "insert into" query which scans dozens of tables, some of them without partition pruning.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] HangyuanLiu commented on issue #3270: Query
failed by "failed to initialize storage reader, res=-230"
Posted by GitBox <gi...@apache.org>.
HangyuanLiu commented on issue #3270: Query failed by "failed to initialize storage reader, res=-230"
URL: https://github.com/apache/incubator-doris/issues/3270#issuecomment-610241476
We had the same problem #2962 ，also because be load too much
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on issue #3270: Query failed by
"failed to initialize storage reader, res=-230"
Posted by GitBox <gi...@apache.org>.
imay commented on issue #3270: Query failed by "failed to initialize storage reader, res=-230"
URL: https://github.com/apache/incubator-doris/issues/3270#issuecomment-610278510
> We use a straightforward approach to avoid compacting recently added version, which works well in practice.
This sounds good for me. And actually, Doris keeps a hard link for recent loaded delta to do an incremental clone. However, these delta is not used when doing scan, it will be better if Doris can use them.
> In our case, the query in trouble is a fairly complicated "insert into" query which scans dozens of tables, some of them without partition pruning.
Anyway, 4 seconds seems too long. You can create another issue to track it if it is convenient to provide the insert SQL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org