You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/10/09 07:23:58 UTC
[GitHub] [druid] QingdongZeng3 opened a new issue #10497: Update metadata failed after indexing task empty running with few data out of time widnow
QingdongZeng3 opened a new issue #10497:
URL: https://github.com/apache/druid/issues/10497
Please provide a detailed title (e.g. "Broker crashes when using TopN query with Bound filter" instead of just "Broker crashes").
### Affected Version
0.16.0
The Druid version where the problem was encountered.
### Description
Please include as much detailed information about the problem as possible.
- Cluster size
12 nodes
- Configurations in use
indexer.runner.type = httpRemote
storage metadata with mysql , store segments in hdfs.
segmentGranularity = 1 Hour
TaskDuration = 1 Hour
lateMessageRejectionPeriod = 1800s
earlyMessageRejectionPeriod = 1800s
- Steps to reproduce the problem
1. Use my configurations ，run a datasource , produce some data , general some segments .(It's very nomal)
2 . In a whole index Task running lifecycle （start new index task --> finish index task，in my configuration,its 1 Hour），just
produce a few data ,all of those data are out of time .( they will be thrown away，and no new segments generated，so partation-offset metadata will not update in mysql，but update in overlord)
3. After task in step 2 finished，and new task is running.（New task will use new checkpoint from overlord to general segments ，but when it try to publish segment，it will faied because of begin-checkPoint not equals with mysq）
4.ERROR when update metadata
https://github.com/apache/druid/blob/master/server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java#L1135
- The error message or stack traces encountered. Providing more context, such as nearby log messages or even entire logs, can be helpful.
- Any debugging that you have already done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 edited a comment on issue #10497: Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 edited a comment on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-706018771
I‘m sorry but I can not upload picture of error log because of company secret protection.
In my opinion，this problem caused by "only update metadata in mysql when publish segment ,but update checkpoint in overlord in fixed time, so when task finished and checkpoint change，but no segment generated，this proben happened".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 commented on issue #10497: Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 commented on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-706018771
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 commented on issue #10497: [BUG] [Indexing Task 0.16.0] Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 commented on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-715210242
> Hi, @QingdongZeng3 . Have you tried using the [latest version](https://github.com/apache/druid/releases/latest) and will it still reproduce the problem?
oh，I didn't try it.But through code，this problem seems still exisit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 edited a comment on issue #10497: [BUG] [Indexing Task 0.16.0] Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 edited a comment on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-706980051
@jihoonson @leventov @gianm @FaxianZhao @QiuMM @jon-wei @liutang123 @kamaci @xiangqiao123 Hi, contributors of IndexerSQLMetadataStorageCoordinator.java , could you take a look at this problem ？Thanks！
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 commented on issue #10497: Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 commented on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-706018771
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 commented on issue #10497: [BUG] [Indexing Task 0.16.0] Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 commented on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-706980051
@jihoonson @leventov @gianm @FaxianZhao @QiuMM Hi, contributors of IndexerSQLMetadataStorageCoordinator.java , could you take a look at this problem ？Thanks！
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 edited a comment on issue #10497: Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 edited a comment on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-706018771
I‘m sorry but I can not upload picture of error log because of company secret protection.
In my opinion，this problem caused by "only update metadata in mysql when publish segment ,but update checkpoint in overlord in fixed time, so when task finished and checkpoint change，but no segment generated，this proben happened".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 commented on issue #10497: [BUG] [Indexing Task 0.16.0] Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 commented on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-715001727
阿西吧！没人啊！bug啊！
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 edited a comment on issue #10497: [BUG] [Indexing Task 0.16.0] Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 edited a comment on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-706018771
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 commented on issue #10497: [BUG] [Indexing Task 0.16.0] Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 commented on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-714271883
Hi all,
Is there someone take a look at this problem ？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 commented on issue #10497: [Indexing Task 0.16.0] Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 commented on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-706033702
To solve this problem，we colud check metadata after indexing task succeed ,even there is no new segment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] QingdongZeng3 edited a comment on issue #10497: [BUG] [Indexing Task 0.16.0] Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
QingdongZeng3 edited a comment on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-706980051
@jihoonson @leventov @gianm @FaxianZhao @QiuMM @jon-wei @liutang123 @kamaci @xiangqiao123 Hi, contributors of IndexerSQLMetadataStorageCoordinator.java , would you take a look at this problem ？Thanks！
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] asdf2014 commented on issue #10497: [BUG] [Indexing Task 0.16.0] Update metadata failed after indexing task empty running with few data out of time widnow
Posted by GitBox <gi...@apache.org>.
asdf2014 commented on issue #10497:
URL: https://github.com/apache/druid/issues/10497#issuecomment-715040455
Hi, @QingdongZeng3 . Have you tried using the [latest version](https://github.com/apache/druid/releases/latest) and will it still reproduce the problem?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org