You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@celeborn.apache.org by "zwangsheng (via GitHub)" <gi...@apache.org> on 2024/12/09 12:38:25 UTC
[PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
zwangsheng opened a new pull request, #2985:
URL: https://github.com/apache/celeborn/pull/2985
<!--
Thanks for sending a pull request! Here are some tips for you:
- Make sure the PR title start w/ a JIRA ticket, e.g. '[CELEBORN-XXXX] Your PR title ...'.
- Be sure to keep the PR description updated to reflect all changes.
- Please write your PR title to summarize what this PR proposes.
- If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
Refactoring shuffleWriters, extract common methods as BasedShuffleWriter.
### Why are the changes needed?
Currently, HashBasedShuffleWriter and SortBasedShuffleWriter have a lot of the same methods, which makes it difficult for us to maintain the code. In many scenarios, we need to update multiple classes at the same time.
Therefore, we need to refactor Shuffle Writer, extract common methods without changing the specific method logic, and simplify the corresponding Shuffle Writer.
### Does this PR introduce _any_ user-facing change?
No, we do not change the logic code of apis or others.
### How was this patch tested?
UT and local env
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2985: [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods
URL: https://github.com/apache/celeborn/pull/2985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "turboFei (via GitHub)" <gi...@apache.org>.
turboFei commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2889018093
cc @zwangsheng any update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #2985: [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods
URL: https://github.com/apache/celeborn/pull/2985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2943020231
> cc @zwangsheng any update?
Wrong push branch, open a new PR #3306
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2871468316
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2528024318
> Cool. Do test this PR on the cluster, if something is missing, it will cause data correctness problems which are important.
Well, i'll test normal and abnormal cases in yarn cluster laster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2544701847
@zwangsheng Hi, How are your tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2545513341
> @zwangsheng Hi, How are your tests?
Tested on yarn cluster, passing normal Terasort tests, as well as abnormal random task failure tests.
But for the sake of correctness, I still need time to continue testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2796227437
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2568775618
> Test the normal case(with terasort test) and the abnormal case(RDD with task fail), seems this change is safe.
>
> @FMX plz take a review
OK, pls solve the conflicts. The review will be done soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2846669043
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng closed pull request #2985: [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods
URL: https://github.com/apache/celeborn/pull/2985
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2690042734
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2742678349
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2642251709
This issue was closed because it has been staled for 10 days with no activity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "FMX (via GitHub)" <gi...@apache.org>.
FMX commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2527956055
Cool. Do test this PR on the cluster, if something is missing, it will cause data correctness problems which are important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2615124613
This PR is stale because it has been open 20 days with no activity. Remove stale label or comment or this will be closed in 10 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [CELEBORN-1768][WRITER] Refactoring Shuffle Writer to extract common methods [celeborn]
Posted by "zwangsheng (via GitHub)" <gi...@apache.org>.
zwangsheng commented on PR #2985:
URL: https://github.com/apache/celeborn/pull/2985#issuecomment-2562178904
Test the normal case(with terasort test) and the abnormal case(RDD with task fail), seems this change is safe.
@FMX plz take a review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@celeborn.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org