You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2020/08/24 09:17:24 UTC
[GitHub] [carbondata] QiangCai opened a new pull request #3897: [CARBONDATA-3958] Fix CDC merge tasks can't finish issue
QiangCai opened a new pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897
### Why is this PR needed?
CDC merge tasks are blocked when data loading use CarbonTableOutputFormat.getRecordWriter method.
Because the poll method of the queue is time out in some cases.
### What changes were proposed in this PR?
If the output is not closed, it will poll a row batch in the loops till it gets a not null batch.
If the output is closed, it will break the loop.
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3897: [CARBONDATA-3958] Avoid blocking the loading task when the output queue poll timeout
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-680037073
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] ravipesala commented on pull request #3897: [CARBONDATA-3958] Fix CDC merge tasks can't finish issue
Posted by GitBox <gi...@apache.org>.
ravipesala commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-679098812
LGTM,
Anyway we are not using it in CDC as per the PR https://github.com/apache/carbondata/pull/3856.
This case not supposed to happen, it means source is not writing any data for more than 5 minutes. It should be some exceptional scenarios.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3897: [CARBONDATA-3958] Fix CDC merge tasks can't finish issue
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-679123761
@QiangCai as @ravipesala said since we are not using this, may be once after #3856 is merged, we can check and close this i think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3897: [CARBONDATA-3958] Fix CDC merge tasks can't finish issue
Posted by GitBox <gi...@apache.org>.
QiangCai commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-679445110
@ravipesala @akashrn5 we can merge it also. In the future, other cases can use CarbonTableOutputFormat and this issue will not happen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3897: [CARBONDATA-3958] Fix CDC merge tasks can't finish issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-679068984
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3847/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3897: [CARBONDATA-3958] Fix CDC merge tasks can't finish issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-679159418
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3851/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] asfgit closed pull request #3897: [CARBONDATA-3958] Avoid blocking the loading task when the output queue poll timeout
Posted by GitBox <gi...@apache.org>.
asfgit closed pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3897: [CARBONDATA-3958] Fix CDC merge tasks can't finish issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-679068472
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2106/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #3897: [CARBONDATA-3958] Fix CDC merge tasks can't finish issue
Posted by GitBox <gi...@apache.org>.
QiangCai commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-679096749
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3897: [CARBONDATA-3958] Avoid blocking the loading task when the output queue poll timeout
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-680108770
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2123/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3897: [CARBONDATA-3958] Fix CDC merge tasks can't finish issue
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-679159237
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/2110/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA1 commented on pull request #3897: [CARBONDATA-3958] Avoid blocking the loading task when the output queue poll timeout
Posted by GitBox <gi...@apache.org>.
CarbonDataQA1 commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-680106295
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/3864/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] akashrn5 commented on pull request #3897: [CARBONDATA-3958] Avoid blocking the loading task when the output queue poll timeout
Posted by GitBox <gi...@apache.org>.
akashrn5 commented on pull request #3897:
URL: https://github.com/apache/carbondata/pull/3897#issuecomment-680113725
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org