You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/09/27 04:37:26 UTC
[GitHub] [hive] zhengchenyu opened a new pull request #2674: HIVE-25561: Killed task should not commit file.
zhengchenyu opened a new pull request #2674:
URL: https://github.com/apache/hive/pull/2674
### What changes were proposed in this pull request?
We should set abort to true, when we catch any Exception.
### Why are the changes needed?
For tez engine in our cluster, I found some duplicate line, especially tez speculation is enabled. In partition dir, I found both 000002_0 and 000002_1 exist.
It's a very low probability event. HIVE-10429 has fix some bug about interrupt, but some exception was not caught.
In our cluster, Task receive SIGTERM, then ClientFinalizer(Hadoop Class) was called, hdfs client will close. Then will raise exception, but abort may not set to true.
Then removeTempOrDuplicateFiles may fail because of inconsistency, duplicate file will retain.
(Notes: Driver first list dir, then Task commit file, then Driver remove duplicate file. It is a inconsistency case)
### How was this patch tested?
Manual test in our cluster.
And I add some delay in our test code, then increase the problem's probability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on pull request #2674: HIVE-25561: Killed task should not commit file.
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on pull request #2674:
URL: https://github.com/apache/hive/pull/2674#issuecomment-979754088
thank you @zhengchenyu! I've merged your changes :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] abstractdog commented on pull request #2674: HIVE-25561: Killed task should not commit file.
Posted by GitBox <gi...@apache.org>.
abstractdog commented on pull request #2674:
URL: https://github.com/apache/hive/pull/2674#issuecomment-979775926
thanks @kgyrtkirk for the review and @zhengchenyu for the patch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk commented on pull request #2674: HIVE-25561: Killed task should not commit file.
Posted by GitBox <gi...@apache.org>.
kgyrtkirk commented on pull request #2674:
URL: https://github.com/apache/hive/pull/2674#issuecomment-979313436
@zhengchenyu : I was about to commit this - but I've stopped because of some minor issues.
Would you mind associating the email address(zheng...@ke.com) used to commit the patch to your github account? without that github adds some "Co-authored-by:" stuff..
May I also ask for your full name ? (or you could fill out "Full Name" in your jira or github profile) - however if you preffer "zhengchenyu001" that's fine as well - just let me know!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhengchenyu edited a comment on pull request #2674: HIVE-25561: Killed task should not commit file.
Posted by GitBox <gi...@apache.org>.
zhengchenyu edited a comment on pull request #2674:
URL: https://github.com/apache/hive/pull/2674#issuecomment-979684308
> Would you mind associating the email address([zheng...@ke.com](mailto:zheng...@ke.com)) used to commit the patch to your github account? without that github adds some "Co-authored-by:" stuff..
zhengchenyu001 is our company gitlab user name. My real fullname is zhengchenyu, just the username of my github and issues.apache.org account. To keep pace with github and issues.apache.org, you can just use "zhengchenyu" to commit this path.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhengchenyu edited a comment on pull request #2674: HIVE-25561: Killed task should not commit file.
Posted by GitBox <gi...@apache.org>.
zhengchenyu edited a comment on pull request #2674:
URL: https://github.com/apache/hive/pull/2674#issuecomment-979684308
> Would you mind associating the email address([zheng...@ke.com](mailto:zheng...@ke.com)) used to commit the patch to your github account? without that github adds some "Co-authored-by:" stuff..
zhengchenyu001 is our company gitlab user name. My real fullname is zhengchenyu, just the username of my github and issues.apache.org account. To keep pace with github and issues.apache.org, you can just use "zhengchenyu" to commit this patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhengchenyu edited a comment on pull request #2674: HIVE-25561: Killed task should not commit file.
Posted by GitBox <gi...@apache.org>.
zhengchenyu edited a comment on pull request #2674:
URL: https://github.com/apache/hive/pull/2674#issuecomment-928998346
@abstractdog @zabetak Can you help me review it, or give me some suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhengchenyu commented on pull request #2674: HIVE-25561: Killed task should not commit file.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on pull request #2674:
URL: https://github.com/apache/hive/pull/2674#issuecomment-928998346
@abstractdog Can you help me review it, or give me some suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhengchenyu commented on pull request #2674: HIVE-25561: Killed task should not commit file.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on pull request #2674:
URL: https://github.com/apache/hive/pull/2674#issuecomment-928998346
@abstractdog Can you help me review it, or give me some suggestion?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] zhengchenyu commented on pull request #2674: HIVE-25561: Killed task should not commit file.
Posted by GitBox <gi...@apache.org>.
zhengchenyu commented on pull request #2674:
URL: https://github.com/apache/hive/pull/2674#issuecomment-979684308
> Would you mind associating the email address([zheng...@ke.com](mailto:zheng...@ke.com)) used to commit the patch to your github account? without that github adds some "Co-authored-by:" stuff..
My fullname is zhengchenyu, just the username of github and issues.apache.org. you can just use "zhengchenyu".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] kgyrtkirk merged pull request #2674: HIVE-25561: Killed task should not commit file.
Posted by GitBox <gi...@apache.org>.
kgyrtkirk merged pull request #2674:
URL: https://github.com/apache/hive/pull/2674
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org